Human Capital: Agencies Should More Fully Evaluate the Costs and Benefits of Executive Training
Released: 2014-03-04T00:00:00

Letter

Background

The federal government has established a policy to develop employees, including its SES, through training and development programs in order to improve public service, increase efficiency and economy, and build and retain a force of skilled and efficient employees, among other things. Under 5 U.S.C. § 3396, OPM is responsible for establishing training and development programs for SES and SES candidates, or for requiring agencies to establish such programs that meet OPM criteria. If agencies establish these training programs, 5 U.S.C. § 3396(b) states that OPM is required to assist agencies in the establishment of such programs and monitor their implementation. In accordance with this, OPM has issued regulations on training and management development. For example, in 2006 OPM issued a regulation requiring agencies to report training expenditures to OPM through the government-wide electronic reporting system. Executive Order No. 11348 (issued in 1967 and later amended by Executive Order No. 12107 in 1978) further requires OPM to assist agencies in developing sound programs and financial plans for training and to provide advice, information, and assistance to agencies on planning, programming, budgeting, operating, and evaluating training programs. OPM provides federal agencies with assistance through two primary mechanisms: guidance documents and technical assistance (see table 1 below).

Executive training courses are one part of a foundation of executive education programs that include coaching and rotational assignments to help executives understand leadership roles and responsibilities in their respective agencies. OPM reviews and certifies agencies’ SES candidate development programs (CDPs), which are designed to train potential SES members. OPM also convenes Qualifications Review Boards (QRBs), staffed by SES members across government, to provide an independent peer review of candidates proposed for initial career appointment to the SES.

Agencies Lack Complete and Reliable Data on the Cost of External Executive Training and Are Likely Underreporting Costs

Agencies Reported Spending Millions of Dollars and Most Used Multiple External Providers from Fiscal Years 2008 through 2012

The 26 CHCO Council agencies responding to our questionnaire reported that, from fiscal years 2008 through 2012, they spent almost $57 million (in constant 2012 dollars) for about 30,000 participants to attend executive training courses offered by external providers. On average, the agencies collectively reported spending between $9 and $15 million dollars a year (in constant dollars) on external executive training over the 5-year period.

The 26 CHCO agencies reported using at least 410 external providers from fiscal years 2008 to 2012 for executive training. Among the external providers most often used, CHCO agencies reported using OPM’s Federal Executive Institute (FEI) more than other external providers (see table 2).

OPM requires agencies to maintain records of training plans, expenditures, and activities, and to submit the recorded data to OPM. Since 2006, OPM has required agencies to report training data to OPM monthly via OPM’s EHRI system. OPM’s Guide to Human Resources Reporting states that these data should include training materials cost, per diem cost, travel cost, and tuition and fees. Further, federal financial accounting standards state that reliable information on the costs of federal programs or activities is crucial for effective management of government operations; federal financial accounting standards also state that Congress needs cost information to evaluate program performance, to make program authorization decisions, and to compare alternative courses of action.

External providers employed different approaches for executive training, which had different cost implications. Based on external providers’ views, the cost of individual executive education training courses can vary greatly among external providers due to a variety of factors, including location of training, length of training, and whether courses are tailored to meet the mission and goals of a particular agency. For example, VA’s Leading EDGE (Executives Driving Government Excellence) program offers half-day executive training workshops to all agencies where employees can participate either in person or online. The workshops can serve from about 300 to over 700 participants. According to Leading EDGE’s executive director, the annual cost for Leading EDGE is about $850 per participant without coaching and $2,100 per participant with coaching. Other external providers we contacted host residential training courses or programs which are off-site from participant’s work locations and last over a period of several days or weeks, including the following:

Leadership for a Democratic Society: FEI hosts this 22-day course over 28 days (includes weekends), which addresses (among other subjects) constitutional principles, personal leadership, organizational theory, and the policy framework in which governmental leadership occurs. In FY 2012, attending this course cost about $20,000.

Senior Executive Fellows Program: the Harvard Kennedy School of Government hosts this 22-day program over 27 days (includes weekends) and addresses (among other subjects) political management, effective implementation, and performance management. In FY 2012, attending this course cost about $18,000.

Leadership at the Peak: The Center for Creative Leadership offers this 5-day course which focuses on key leadership competencies such as communication, influence, validating and articulating a vision, and leading outwardly. The course incorporates a five-month learning experience including five days of classroom training, and other activities such as assessments and executive coaching sessions. In FY 2012, attending this course cost about $9,000.

Executive Leadership for America: The Brookings Institution conducts this 6-day course which addresses (among other subjects) the historical framework for constitutional government and how to create an organizational environment that is responsive to change. In FY 2012, attending this course cost about $6,000.

Agencies Experience Challenges in Collecting and Reporting Data

Agencies track some—but not all—costs of executive training administered by external providers. All 26 of the agency CHCOs that responded to our questionnaire provided data on executive training administered by external providers; however, half of the agencies reported that the data they provided were incomplete, and in some cases, had limitations. In addition, during our review, agency officials from 4 of the 26 agencies asked us to clarify the cost data we requested, noting that training cost data are not always readily available within their agencies. They explained that in order to respond to our data request, in some cases they would have to manually retrieve paper documents and tabulate the costs. Moreover, the agencies provided data that included different cost components. For example, one agency reported that it did not include costs for course materials. Two other agencies reported that they did not include costs for travel, lodging, or per diem. Other CHCO agencies cited additional problems with tracking and summarizing costs paid to external providers of executive education. For example, one agency reported experiencing challenges in tracking training costs agency-wide. According to agency officials, they were unable to completely (or 100 percent) accurately track training costs across the department because of the multiple, incompatible learning management systems that the department uses to track training throughout the agency. In addition, they stated that the agency’s practices for tracking training data are inconsistent within the agency because agency components do not always report training completed with external providers to the agency’s shared training database system. Another agency said its agency-wide training data were incomplete and unreliable because its internal training and financial tracking systems were incompatible with each other. In light of all these factors, agencies are likely underreporting the amount they spent on executive training.

In our interviews with Energy, Treasury, and VA, the CHCOs (or their representatives) reported that their agencies track and report training cost data to OPM in various formats, some automated and some manual. However, the three agencies reported facing challenges tracking cost data. Energy reported that executive training reporting is not consistent across the agency because many Energy components have direct funding streams from Congress and schedule their own training programs; thus, they do not always report training completed with external providers to Energy’s Office of Human Capital. Moreover, because Energy’s Office of Human Capital has no direct access to component organization records, the CHCO cannot ensure that all of Energy’s funded and administered training programs are tracked and reported to EHRI and OPM; in addition, Energy’s CHCO cannot identify whether executive training is reported to OPM by a component organization in a separate report or with other line item reporting of completed training. However, the cost and delivery of Energy’s SES candidate development program and its executive training provided centrally by Energy’s Office of Human Capital is normally tracked and reported on an annual basis although Energy has not had any recent centrally funded executive learning activities. Treasury and VA also reported difficulties in collecting and tracking reliable executive training data. Treasury reported that some of its individual bureaus were challenged by our request to provide executive training cost data because the costs had to be identified by a manual review of financial transaction listings. They stated that part of the reason the costs were difficult to pull together is that the training costs were not reported via OPM’s Standard Form 182—which is used to collect and document training information for inclusion in participants’ employee records and the EHRI database. Treasury officials explained that much of its reported executive training was done by contract and was not procured by either the individual or the individual offices that would normally complete the training form. As a result, Treasury enlisted multiple sources to pull the data together at each of the agency’s bureaus. Treasury officials reported that the agency is working internally to determine how to track this information at the department level. VA reported two issues with its electronic reporting of data to EHRI: inconsistent recording of overall training cost data in its training management system, and incompatibility between VA’s learning management system and EHRI.

We found discrepancies between the data for external executive training costs that agencies submitted to us and the data from OPM’s EHRI. In most instances where we identified discrepancies, the cost information that agencies provided to us was higher than the cost total from OPM’s EHRI. Table 3 provides illustrative examples of discrepancies in cost information.

OPM officials agreed that agencies have challenges collecting and reporting costs for executive training through the EHRI database, and as a result, executive training cost data in EHRI is probably lower than actual agency expenditures. OPM officials explained that although agencies have taken steps recently to improve their reporting of training data, some agencies continue to struggle with the reporting requirements, in part because these agencies’ learning management systems are not compatible with OPM’s EHRI. For these agencies, reports on training costs require manual uploading because some agencies cannot sufficiently aggregate training information at the sub-agency level. In September 2012, we reported on the weaknesses of training cost data that agencies report to OPM and found that several agencies were unable to collect reliable training cost data from components, due to multiple incompatible learning management systems and to inconsistent practices in tracking training cost data, such as the use of inconsistent coding schemes for tracking similar training activities. We recommended, among other things, that OPM (1) work with the CHCO Council to improve the reliability of agency training investment information, and (2) provide regular report summaries to agencies on EHRI training investment data and reliability. OPM agreed to work with agency CHCOs to continue to strengthen the use of the Standard Form 182 and to increase reliability of their training data; further, OPM stated that it would provide regular summaries to agencies on the training investment data submitted to OPM in order to improve transparency. However, OPM stated that these summaries will not directly lead to improved reliability of the data and explained that agencies must take action to improve the data to positively impact data reliability.

OPM officials told us that they provide assistance and guidance for federal training efforts but because current training data is unreliable, OPM officials said they have not used it to inform their training guidance and assistance to agencies. OPM officials stated that they do not believe they have a viable way to enforce reporting requirements for training data. However, OPM has avenues, such as interagency councils, that can help them assist agencies with fulfilling their reporting obligations. For example, during 2013, OPM officials reported that they, in collaboration with the Chief Learning Officers Council, created an interagency group to develop quality standards and desired reporting data elements to help agencies collect the right information and to make informed training decisions, but as of September 2012, the interagency group has not established any timeframes for completing this effort. OPM reported that as of October 2013, the working group is moving forward on a project to develop guidance on prioritizing and ranking federal training investments that will support a broader effort to develop a single standardized process for prioritizing training investments. In addition, OPM officials told us they have been meeting separately with CHCO Council agencies to discuss each agency’s training data reported to OPM and any deficiencies noted by OPM. OPM believes this will help rectify situations where agencies are unable to report training data—among other types of data—accurately, completely, and in a timely fashion. OPM officials said that they also plan to meet with smaller agencies for these same purposes.

However, OPM has not established interim milestones for the completion of these planned meetings with agencies to address training data deficiencies, nor has OPM set timeframes for agencies to have a plan in place for improving the completeness and reliability of agency data on executive training costs. One leading transformation practice is to establish milestones and timeframes to demonstrate progress towards achieving goals. Without interim milestones and timeframes for these efforts, OPM may be missing an opportunity to better position itself to hold agencies accountable for improving the data.

Most Agencies Reported Evaluating Executive Training but Few Reported Assessing Its Impact on Agency Missions

Twenty-two of twenty-six agency CHCOs reported to us that they systematically evaluate the effectiveness of executive training by assessing participant feedback. Most agencies reported they do this by using end-of-course evaluations. Seventeen of twenty-six agency CHCOs reported measuring the impact of training on participant performance. Five agencies reported using post-course surveys given to supervisors or participants to measure if what was learned affected the participants’ behavior. For example, one agency requested that managers and mentors of Senior Executive Service Candidate Development Program (SESCDP) candidates fill out a questionnaire about the impact and value of the program for their candidate; this questionnaire included to what degree the candidate applied lessons learned in the program toward their developmental assignments.

Twenty of the twenty-six agency CHCOs said they changed their executive training program as the result of evaluations. Six agencies reported they revised their program’s content and logistics as a result of feedback. For example, VA reported that it reorganized its training schedule to add subjects that its senior executives wanted included. Other agencies reported they expanded the program based on participant feedback. For example, one agency expanded the number of leadership development programs offered, while a second agency increased the number and types of training opportunities available for executives in order to increase participation in executive training. A third agency now offers more learning activities (e.g., virtual training events) that do not require travel, in order to better accommodate the executives’ schedules, as well as the current budget constraints within the agency.

A majority of agencies do not have a formal process for evaluating the impact of training on the agency’s performance goals and mission. Ten of twenty-six agency CHCOs – or less than half – said they measured the impact of training on the agency’s mission, while eight of twenty-six agency CHCOs said they measured the impact on agency performance goals.

Agencies are required by statute and OPM implementing regulations to evaluate how well training programs contribute to mission accomplishment and meet organizational performance goals. OPM’s Training Evaluation Field Guide provides information on methodologies for conducting training evaluations with the goal of identifying cost effective training initiatives that maximize mission accomplishments. The guide is based on the Kirkpatrick model of evaluation, a popular model of evaluation that includes measuring (1) participant reaction to the training program; (2) changes in employees’ skills, knowledge, or abilities; (3) changes in employees’ behavior or performance; and (4) the impact of the training on program or organizational results, which may be split into a fifth level that includes a return on investment assessment that compares training costs to derived benefits.

Similarly, our guidance on assessing training states that to measure the real impact of training, agencies need to move beyond data on inputs and outputs by developing additional indicators that help to determine how training contributes to the accomplishment of agency goals and objectives. The traditional approach of collecting and reporting data related to training and development often involves indicators that can be readily quantified, such as the number of employees trained, training hours per employee, and total training hours. While necessary, these kinds of measures do not fully provide agencies with the kind of information they need to determine how training efforts contributed to improved performance, to reduced costs, or to a greater capacity to meet new and emerging transformation challenges.

Two of the agencies we interviewed, Treasury and VA, were among those who reported in their questionnaires that they measured the impact of executive training on the agency’s performance goals and mission. VA uses various assessment measures to evaluate effectiveness against VA’s overall mission and the VA Secretary’s strategic objectives for accomplishing transformation: VA reported that these measures have led to better training investment decisions. For example, VA includes post- course requirements for its Strategic Leadership Course II – Basic, which is focused on providing its senior executives with tools to support strategic decision-making, critical thinking, and leading and driving change. Approximately 30 days after the training, class participants are asked to provide their “ah ha” moment from the training, with regard to their leadership style or techniques; approximately 90 days after the training, participants are asked how they have been able to apply the concepts from the executive training to their agency’s organization or mission. Sometimes specific questions of interest are substituted, such as recommendations on strategic challenge questions for future classes, or requests for programs or resources that they need to better accomplish their mission as strategic leaders. A second requirement may be assigned at the post 90-day training point. This requirement usually asks the participants to explain how they have applied lessons taken from the training to their organization or mission. Additionally, the agency provides assessment data to a team of program analysts at the VA Learning University who periodically provide the agency with feedback on the effectiveness of the program. According to VA, the evaluation tends to focus on the impact of training on agency mission and strategic goals, and also on return on investment; the evaluation also includes specific examples where senior executives applied what they gained at the training and applied it to their mission in the VA. Treasury officials explained that while it would be hard to isolate when training by itself contributes to an individual meeting an agency’s goal, by aligning all of its programs to the department’s strategic plan and to the executive core qualifications (ECQ), they are ensuring the training support the agency’s goals.

CHCO agencies recognized the importance of evaluating the impact of training on their mission but emphasized the difficulty in conducting these types of evaluations. For example, two agencies reported needing additional assistance from OPM for these types of activities, such as centralized evaluation information on the performance of external providers. In addition, we previously found that some agencies found it difficult to conduct these evaluations because their cost and time demands can be significant. OPM acknowledged that some agencies struggle with evaluating training programs. OPM reported that it has developed and published regulations related to training evaluation. According to OPM, these regulations were developed to enhance the impact of training on agency mission accomplishment. The regulations generally relate to training planning, execution, and demonstration of value. However, OPM is not sharing lessons learned from agencies that reported having assessed the impact of executive training on agency missions. Given that less than half of the CHCO agencies reported they are evaluating the impact of executive training on their missions, sharing lessons learned in its oversight role could help OPM better position the other CHCO agencies for conducting such evaluations.

We recognize that higher levels of evaluation (such as evaluating the impact on organization performance or the return on investment) can be challenging to conduct because of the difficulty and costs associated with data collection and the complexity of directly linking training and development programs to improved individual and organizational performance. However, eight agencies that responded to our questionnaire (and that are also members of the CHCO Council) reported that they have conducted these evaluations and may have lessons learned from which other agencies could benefit. According to GAO guidance on evaluating training, factors to consider when deciding the appropriate level of evaluation include estimated costs of the training effort, size of the training audience, management interest, program visibility, and the anticipated life span of the effort. The guidance also states that each agency consider the feasibility and cost-effectiveness of conducting these in-depth evaluations, along with budgetary and staffing circumstances that may limit the agency’s ability to complete such evaluations. Without conducting higher levels of evaluation, agencies are missing information that could help them make more effective training investment decisions; in addition, OPM is missing the opportunity to assist agencies in fully understanding the benefits of executive training on the continued development of senior executives and on the agency’s performance.

CHCOs Identified Opportunities for Agencies and OPM to Achieve Efficiencies in Executive Training

Opportunities Include Using Virtual Training and Centralizing Training

Twenty of twenty-six agency CHCOs responding to our questionnaire said they identified opportunities for achieving efficiencies in delivering executive training within their respective agencies. For instance, some agency CHCOs described steps agencies could take to increase participation in executive training programs within their respective organizations. For example, one CHCO said his agency could expand the base of eligible participants to include employees in other senior level positions outside the SES. Our guide for assessing strategic training and development efforts states that agencies should select employees for developmental programs, in part, based on the usefulness of training content given employees’ current duties and existing skills and competencies. Employees in other senior level positions could build their skills and competencies through participation in executive training; at the same time, agencies could increase participation in executive training courses by expanding participation eligibility to employees in other senior level positions outside the SES, thus reducing unit costs of training delivery. Another CHCO said his agency is considering using executives and senior leaders to serve as facilitators for leadership events. As we previously concluded, efforts to identify cost-effective and robust options for designing training and development programs may include use of in- house staff, such as senior executives, based on their capability to implement training.

Four CHCOs said they are considering implementing or expanding computer-based training, such as use of webinars, online books, virtual “executive summits” with developmental sessions and strategic meetings for executives, and other online training programs used for executive development. One CHCO also said his agency is considering use of blended learning activities—which incorporate classroom-based and computer-based training—to maximize impact and cost-effectiveness. This is consistent with leading practices for training investments. Specifically, one such practice calls for comparing the merits of different methods of delivery—such as classroom or computer-based training— and determining what mix of mechanisms to use to ensure efficient and cost-effective delivery. Another leading practice is for agencies to consider blended learning that combines different teaching methods (e.g., Web-based and instructor-led) within the same training effort and provides trainees with the flexibilities to choose among different training delivery methods, while leveraging resources in the most efficient way possible.

In our interviews with selected agencies, Treasury officials reported they are working to centralize training for all Treasury executives through the Treasury Executive Institute (TEI) in an attempt to enhance Treasury’s return on investment for organizational performance and to provide more consistency for mission-critical leadership training. Treasury officials said they could use TEI to leverage training resources throughout the department, with a goal of providing “just-in-time” training to senior executives, rather than requiring them to set aside their normal job responsibilities for a number of months to attend training. Treasury officials told us the agency is in the beginning stages of obtaining OPM approval for an agency-wide SESCDP, which would allow Treasury to train executives from all of its components and would help Treasury fill vacant SES positions agency-wide. Treasury officials said they are likely to operate the proposed agency-wide SESCDP alongside the Internal Revenue Service’s SESCDP—the only SESCDP currently in operation within Treasury. Treasury’s efforts to centralize SES candidate training is consistent with a leading practice related to identifying the most appropriate mix of centralized and decentralized approaches for an agency’s training and development programs. In particular, centralizing design can enhance consistency of training content and offer potential cost savings through standardized record keeping and simplified and more accurate reporting on courses, certifications, educational attainment, costs, or standards.

VA officials said they performed after action reviews on and made changes to their Strategic Leadership Course II – Basic course for career SES members within VA. Based on the reviews and subsequent changes to its executive training program, VA officials negotiated a new price structure from the course vendor to deliver the training at a lower cost in 2013 than if they had used the course’s 2012 price structure. We previously concluded that agencies can use evaluation feedback to identify problems and improve training and development programs as needed. Based on modifications to its executive training programs from after action reviews, VA was also able to reduce program costs, thus increasing the efficiency of training delivery.

Further, 20 of 26 agency CHCOs said they found opportunities for achieving efficiencies government-wide. For instance, some CHCOs said agencies can increase interagency cooperation on executive training by sharing space in training seminars with other agencies, and allowing admission of SES candidates from other federal agencies into training programs such as SESCDPs. Treasury officials we interviewed said they actively market shared training services to the rest of the government through TEI. Energy told us the agency produces a catalog on a semi- annual basis with real-time information on leadership development courses and programs located throughout the country; according to Energy officials, government officials could use the catalog to reduce travel and per diem costs for executive seminars conducted around the nation by identifying equivalent training that is available locally for senior executives. Additionally, several CHCOs said their agencies used VA’s Leading EDGE program to deliver executive training. One CHCO offered to share training facilities with other agencies. However, one CHCO noted that courses sponsored by OPM and VA raised the possibility of redundancy in executive training delivery provided by the two agencies. According to our guide for assessing strategic training and development efforts, agencies can try to achieve economies of scale and avoid duplication of effort when assessing training delivery options by taking advantage of existing course content or training, such as sharable online courseware or multiagency training programs. Specifically, interagency training can help address common developmental needs government- wide and promote cost-efficiency while taking advantage of existing resources rather than creating similar programs in multiple agencies.

CHCOs Reported OPM Can Help Agencies by Offering More Lower Cost or No Cost Training and by Centralizing Training Offerings

CHCOs from agencies responding to our questionnaire reported that OPM can take several steps to help both the federal government and individual agencies deliver executive training more efficiently. First, some agency CHCOs said that OPM should provide more training to senior executives government-wide that is free or lower cost to agencies. One CHCO noted that OPM could provide lower or no cost executive development opportunities to agencies by more effectively using technology to deliver virtual training. This would be consistent with leading training practices comparing the merits of different delivery vehicles, such as classroom or computer-based training, in order to assess what vehicles to use for ensuring efficient and cost-effective delivery. OPM officials said that agencies cannot avoid training costs, but OPM is trying to reduce agency training costs by bringing courses closer to agency offices and by using free space for training rather than renting sites. OPM has also developed free and fee-based training solutions for use government-wide, and hosts speaker series and meta-leadership courses for senior executives.

Second, agency CHCOs said that OPM could take steps to help centralize training offerings for senior executives, such as creating a centrally funded SESCDP, and could coordinate joint training efforts among agencies. For example, according to one CHCO, OPM could take steps to encourage agencies to fill paid-for, but unfilled seats for executive training programs across agency lines. CHCOs also said that OPM could assist agencies in locating executive training programs open to other agencies and departments, and provide a central source for listing executive development opportunities throughout the federal government. An example of this type of executive training program is the Leading EDGE program administered by VA. This program is operated by a senior executive within VA and offers executive training to agencies across the federal government. According to the program’s deputy director, 17 federal agencies are participating in Leading EDGE training. Treasury’s TEI is another example of a shared service model that presents an opportunity to leverage resources across the government. According to Treasury, TEI serves 9 other bureaus and agencies in addition to Treasury. Our guide for assessing training practices previously found that centralizing design of training programs can, among other things, offer potential cost savings.

OPM officials agree they could centralize training offerings for senior executives. For instance, OPM hosted the Office of the Director of National Intelligence’s (ODNI) suite of 12 courses detailing the full performance management cycle; in turn, OPM’s Human Resources University (HR University) made them available for use by other agencies, and OPM officials said they incurred savings of about $46 million in 11/2 years. In terms of coordinating joint training efforts among agencies, OPM officials said they collaborate with the CLO Council, the CHCO Council, and federal agencies to provide joint training. For example, OPM collaborates with VA to implement the Leading EDGE program for use by many agencies. OPM officials also suggested that federal agencies use additional FEI courses—and to reimburse FEI for implementing such courses through user fees—so it could become the federal government’s primary source for executive training. As mentioned earlier, sharing online courseware and implementing multiagency training programs can help agencies try to achieve economies of scale while avoiding duplication of effort in designing executive training courses that address similar content as existing courses.

OPM Reports Taking Additional Actions to Improve Training Efficiency

OPM officials noted they are taking additional steps to increase training efficiencies beyond what agency CHCOs suggested. Attributes of effective training include planning training that considers government- wide reforms and other targeted initiatives to improve management and performance, as well as administration priorities, legislative reforms, and major management challenges. In response to our September 2012 recommendation on federal training investment practices, OPM officials said the agency is working with the CLO Council to draft a government- wide mandatory training curriculum and finish plans to implement a pilot course for mandatory training. OPM officials also said the agency is working with CLO Council representatives to standardize elements of federal executive training programs by developing a framework for supervisory and managerial leadership development. The supervisory framework has been distributed to agencies. According to OPM officials, senior executives and other managers can use the supervisory training framework to identify learning objectives they need to achieve, and then target the right job assignments and training courses to address the objectives. OPM officials said agencies are not required to use the framework to develop their executive training programs; however, some agencies are using the framework as a “roadmap” for aligning their training programs.

In November 2012, OPM also released its Executive Development Best Practices guide, which includes key themes and recommendations OPM identified as hallmarks of successful executive development efforts across the public and private sectors. OPM has also developed a training and development webpage where agency officials can find and share products and courses for all areas of learning and development. However, the webpage does not describe examples of agency efforts to achieve efficiencies in delivering executive training.

OPM’s efforts so far to work with agencies to identify efficiencies for executive training have been relatively broad-based; while OPM has taken steps to increase efficiencies, it does not have a process for assessing lessons learned from agencies about potential efficiencies in executive training that other agencies could implement. Our questionnaire results suggest that OPM can use its oversight role to take additional steps to assess and promote ways agencies can achieve efficiencies in executive training that have government-wide application. By not taking such steps, OPM may be missing an opportunity to better position agencies to achieve such efficiencies.

Career SES and SES Candidates Said External Training is Valuable, but Training Programs Could be Improved

Career SES and SES Candidates Found Executive Training Useful in Building Organizational Knowledge, Developing Leadership Skills, and Expanding Networks of Colleagues

Focus group participants said that executive training helped them develop a better understanding of their roles and responsibilities in their respective agencies. Following visits to field offices during executive training, one participant said there is a significant amount of “interconnectivity and intertwining between different operations” within the participant’s agency. Participants also said they learned about key organizational issues and tools that they can use to address them. For instance, a group of SES candidates in one agency reviewed agency-wide telework implementation and developed a business case to overcome technical challenges related to phone systems.

Focus group participants from the three agencies we reviewed also said they developed or strengthened key leadership skills during executive training that align with ECQs and related competencies, such as

Leading and communicating with people. Participants we interviewed said they learned how to better motivate and communicate with staff. One participant noted that “how you interact with people is critical” – a lesson that was emphasized when the participant was implementing difficult changes in the participant’s first SES position. The participant also learned to be both sensitive and emphatic when dealing with issues raised by staff members. The participant said that executive training enabled the participant to help transform an organization with “primarily a bad reputation” into one of the top performing organizations within the participant’s agency. Leading People is one of five ECQs highlighted by OPM.

Problem-solving and decision-making. Executive training, according to participants, provided an environment where they could develop skills in problem solving. For instance, one participant described learning how to make more effective decisions by sharing best practices from different business units within an agency and noted that, as a result, the participant and agency colleagues “are making better decisions than if we were just kind of operating in our private little bubble.” Another participant noted that executive learning helps agency officials reach out to other areas in their organization and leverage others’ experiences in resolving problems, “instead of trying to reinvent the wheel” to identify solutions already considered within the agency. Problem-solving and decisiveness are two competencies identified under the Results Driven ECQ.

Strategic thinking. Executive training courses enabled some participants to step back from managing day-to-day operations and reconsider, from a strategic sense, what they wanted to accomplish in their position. One participant said the use of strategic thinking and leadership was necessary to create a vision for subordinates in conducting agency operations. A participant from another agency added that strategic thinking enables executives to consider the needs of stakeholders beyond one division and office, and to identify how any decisions the participant makes could affect an entire agency. Strategic thinking is a competency identified under the Leading Change ECQ.

Focus group participants also said that networking with colleagues and senior agency officials in executive training was beneficial during training courses and after re-entry into their positions. In particular, participants noted the following networking opportunities in executive training:

Participation develops personal relationships that boost confidence and build networks across and within federal agencies. One participant said relationships built with others helped make the executive feel more comfortable in an SES position, and validated that the participant was “on track” when facing difficult situations on the job. Another participant emphasized that a significant amount of work in the participant’s position involved relationships formed in small groups during executive training with people representing other agencies.

Participation builds accountability among executives within an organization. One SES candidate said that relationships formed during executive training extended into executive assignments following completion of training. The candidate said such relationships helped training participants hold each other accountable in their executive assignments, which can be a tremendous benefit to organizations.

Participants develop better responses to customer and stakeholder needs. Multiple participants said relationships they established in executive training helped them collaborate to resolve customer issues. For example, two participants from different components in one agency said they developed a relationship during executive training that enabled them to work together to provide financial relief to an agency customer without using additional agency resources. Another participant noted he completed a $400 million project with another agency before a scheduled deadline, based on trust and relationships he previously built with fellow participants in an SESCDP.

Participation promotes interaction and relationship building with senior leadership to address organizational issues. From interaction with the Postmaster General, one participant learned methods to motivate employees following “massive changes” within organizations, such as those experienced by the U.S. Postal Service. Another participant (who was addressing equal employment opportunity issues) established a connection with an agency undersecretary, who allowed the participant to send staff to the undersecretary’s office for shadowing assignments to better resolve equal employment opportunity issues within the participant’s office.

Participation facilitates brainstorming solutions to previously identified issues faced by career SES and SES candidates. Focus group participants in one agency discussed two cases where networking and brainstorming among executives helped solve problems and generated financial savings. One participant used a week-long training course to address implementation of a program involving overseas locations. Course members helped the participant develop ideas to adjust hiring practices that, according to the participant, saved the federal government money. Another participant noted that agency officials used executive training to determine steps needed to resolve backlogs of work within an agency; according to the participant, this work may result in cost savings for the federal government over the next 10 to 20 years.

Career SES and SES Candidates Said the Usefulness of Executive Training Depends, In Part, on Program Structures Used to Carry Out Training

Focus group participants found the use of cohorts—groups of career SES and SES candidates participating in the same training courses, and other types of groups—more conducive for problem-solving and knowledge sharing than use of large group settings. Participants from one agency, for example, supported the use of cohorts in executive training to foster connections and build a knowledge base outside their respective functions. Participants from Energy, Treasury, and VA said that participating in small groups and using learning tools—such as case studies and action learning projects—helped career SES and SES candidates enhance problem-solving skills and share ideas with each other. As an example, one participant said that members of small groups applied lessons learned from case studies towards real-life problems. Another participant noted that classmates from executive training regularly met in groups to discuss challenges and share best practices with each other.

Participants also found executive training to be more useful when conducted off-site from the participants’ offices and when participants could step back from day-to-day responsibilities. As described earlier, participants who were able to withdraw from managing day-to-day operations said they could incorporate more strategic thinking into considering organizational issues and accomplishments, and could better focus on learning needed skills to advance within their organization. Focus group members specifically praised the use of off-site training for allowing participants to develop ECQs, learn many of the skills needed to succeed as an executive, and refresh insights or skills they may not have used for a period of time, without distractions that may have hindered training at an on-site location.

In contrast, some participants did not find large gatherings to be useful in encouraging participation or integrating lessons learned. One focus group member noted that colleagues took advantage of large gatherings to complete training requirements without the need to pay attention to lessons learned from the gathering. Another focus group member who participated in a large gathering noted that attendees listened to speakers for several hours without collaborating with the speakers or each other, despite being informed they would be able to interact with people from other agencies.

Additionally, focus group participants said they found virtual training with computers, or with video or phone conferencing, to be less helpful than face-to-face training. For example, one participant said that virtual training tools, such as webinars, can provide effective training on technical or discrete subjects, yet are less effective for developing less tangible strategic, leadership, and executive skills. Participants said they found it difficult to build teams or individual relationships through virtual means, which limited networking and knowledge sharing. Other participants said they were multitasking and completing other tasks while taking virtual training; one participant noted that the training could be more interesting if delivered through other means and in a shorter time period.

Career SES and SES Candidates Believe Agencies Can Take Steps to Enhance Executive Training Programs While Improving Efficiencies of Training Delivery

Focus group participants also provided suggestions to improve the usefulness of such programs to career SES and SES candidates, while also increasing efficiencies in delivering training courses that could potentially reduce training costs for federal agencies.

As one example, focus group participants we interviewed suggested agencies could pool resources and consolidate executive training programs. One participant suggested the creation of a government-wide SESCDP through OPM or another agency; this step could standardize program financing and rules for program implementation government- wide. Likewise, focus group participants said executive training courses could be consolidated through the use of economies of scale which would acquire such training for multiple government agencies. Participants noted that the federal government—instead of individual agencies—could use its influence as a large-scale buyer to leverage taxpayer dollars and negotiate lower prices for training from external providers. Another participant suggested that the federal government could achieve cost savings by requiring external providers to compete against each other to provide executive training courses.

Focus group participants recommended the use of blended training courses—involving both face-to-face and virtual training—to increase efficiencies in training delivery. As mentioned earlier, participants described limitations in using virtual training such as limited interaction among participants and reduced attention paid to course material. However, participants said a blended learning model allows them to establish face-to-face relationships, with one participant noting that human interactions were a key part of learning. At the same time, use of computer-based technology and phone conferencing as part of a blended learning model enables participants to build upon such relationships and learn course material that can be effectively delivered through virtual means.

Additionally, some focus group participants suggested that OPM and other federal agencies could increase use of workshops and brief training sessions to deliver training more efficiently to executives. Specifically, use of one-day workshops and training sessions could increase participation in training by bringing together executives from multiple agencies to encourage cross-pollination of ideas and to learn improved leadership techniques. Finally, participants said that agencies could establish programs to increase selection of SESCDP graduates for open SES positions government-wide. These steps, according to participants, would allow agencies to more efficiently use training acquired by SESCDP graduates, and to achieve greater cost efficiencies by filling vacant agency SES positions with SESCDP graduates from other agencies.

Conclusions

Agencies are using their executive training programs to help develop the necessary leadership skills for executives to effectively manage the nation’s most pressing priorities. Agencies recognize that evaluating executive training activities yields useful information for improving investment decisions. Yet, agencies can only make decisions that improve the value of executive training when they have complete and reliable training cost data and some measure of how the training benefits the agency beyond just a measure of participant feedback. Over 7 years have elapsed since agencies were required to report training cost data to OPM, and many agencies are not yet accurately and completely reporting this information. For corrective actions OPM has already initiated, establishing milestones could help ensure that efforts progress as intended. Moreover, by not establishing a timeframe for improving the reliability of executive training cost data, OPM may be missing an opportunity to better position itself to hold agencies accountable for improving the data. Improving the overall reliability of the data would better position OPM and the agencies to make sound decisions on training investments.

Half of the CHCOs reported they do not have complete information on the cost of executive training provided by external training providers. Moreover, many agency CHCOs struggle to conduct evaluations that identify the impact of training on organizational performance and agency mission, although a few CHCO agencies reported conducting such evaluations. OPM has not used interagency councils to share lessons learned from agencies that have experience with assessing executive training’s impact on agency missions. Although OPM has developed and published guidance on evaluating training’s impact on agency mission and goals, over half of the CHCO agencies are not conducting these evaluations of executive training provided by external providers. Agency CHCOs reported that more assistance from OPM is needed. Additionally, many CHCOs believe there are opportunities for an increased role by OPM for improving efficiency in executive training and providing options for achieving efficiencies. By not assessing lessons learned and other potential efficiencies identified by agencies, OPM may be missing an opportunity to better position agencies to achieve these efficiencies in executive training. Given the fiscal challenges facing the nation, and the increasing complexity of policy issues managed by the SES, it is important for agency leaders and for OPM to take advantage of reasonable opportunities to share resources and improve the efficiency of training investments.

Recommendations for Executive Action

To help ensure that agencies track and report comparable and reliable cost data and perform evaluations that assess the impact of executive training on agency performance or missions, we recommend that the Director of OPM, in coordination with the CHCO Council, take the following actions:

Establish interim milestones for meeting with agencies in order to address training data deficiencies and to establish well-defined timeframes for improving the reliability of the data in its EHRI database.

Improve assistance to agencies regarding evaluating the impact of executive training on mission and goals, for example by sharing information and examples of how agencies could better conduct such evaluations.

To enhance the efficiency of executive training, we also recommend that the Director of OPM, in coordination with the CHCO Council

Assess potential efficiencies identified by agencies for possible government-wide implementation, and then take the steps necessary to implement these, such as updating the guidance governing executive training programs.

Agency Comments and Our Evaluation

We provided a draft of this report to the Director of OPM for review and comment. OPM provided written comments, which are reprinted in appendix II. OPM concurred with our recommendations and is engaged in a number of activities that would help to achieve the objectives of our recommendations. However, we believe that more steps can be taken.

In response to our first recommendation, OPM discussed actions it has initiated to address agencies’ training data deficiencies including launching the Federal Training Investments Work Group, a group of Chief Learning Officers, to help OPM define data elements that can and should be captured within agency automated learning management systems. OPM’s plan is to provide technical assistance to individual agencies during FY 2014 to help build agency capacity and to allow agencies to set milestones for improvement after assessing their training data using forthcoming standards. We believe OPM could work with agencies to set timeframes for agencies to have a plan in place for improving completeness and reliability of agency data on executive training costs. Working with agencies to mutually set timeframes would help OPM hold agencies accountable for improving training data, thus moving agencies towards meeting reporting requirements set in 2006.

In response to our second recommendation, OPM stated that it currently provides assistance to agencies on evaluating the impact of executive training through use of the OPM Training Evaluation Field Guide, which is included in our report, but did not indicate whether or how it measures the effectiveness of that guidance. We believe OPM could further strengthen its assistance by leveraging the experiences of agencies that have conducted these evaluations by sharing techniques and approaches with CHCO agencies that continue to struggle with evaluating executive training. OPM could also ask agencies to assess the effectiveness of its assistance.

In response to our third recommendation, OPM highlighted its Executive Development Best Practices Guide which contains a compilation of executive development best practices used by the public and private sector to support the continued learning of executives, as well as additional actions it has taken to help educate agencies on achieving efficiencies in executive training. In its oversight role, OPM, together with agencies, can develop and use criteria to assess efficiencies put in place by agencies that have potential government-wide application. For instance, OPM can assess agency initiatives—such as creating a centrally-funded SESCDP, or coordinating joint training efforts among agencies—to determine whether these alternative approaches are appropriate for wider application.

OPM also provided technical comments on the draft report, which we incorporated in the report as appropriate.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Director of OPM and other interested parties. In addition, the report will be available at no charge on the GAO website at www.gao.gov. If you have any questions about this report, please contact me at 202-512-2717 or jonesy@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

The objectives of this engagement were to review the costs of federal executive training programs and the value they provide to the federal workforce. Specifically, this report (1) describes what is known about how much Chief Human Capital Officers Council (CHCO Council) agencies have spent on executive training offered by external providers and assesses the reliability of available cost data; (2) assesses how CHCO Council agencies evaluate the effectiveness of the training; (3) compares opportunities identified by agency CHCOs to achieve efficiencies in executive training to leading training practices, and (4) provides views of career Senior Executive Service (SES) and SES candidates from selected agencies on the value of the training they received.

For the purposes of this review, we define the key terms “executive training,” “external provider,” “SES candidates,” and “career SES” as follows:

Executive training is classroom or computer-based instruction in leadership or development programs with a focus on the Office of Personnel Management’s (OPM) executive core qualifications (ECQ). Each individual training episode should be indicated by completion of Standard Form 182.

External provider is any organization (including educational institutions, professional societies, think tank organizations, or consulting firms) that provides training and is independent from the federal department or agency that employs the trainee. The external training provider is contracted to develop or provide training courses or programs. In this case, OPM’s Federal Executive Institute (FEI) and Management Development Centers are considered external providers.

SES candidates are GS-15, GS-14, and equivalent employees who are enrolled in either (1) an OPM-approved SES Candidate Development Program or (2) any agency-funded program that will train and develop employees for the SES or an equivalent.

Career SES or equivalents are those members with civil service status who are appointed competitively to SES positions or equivalents in positions below the top political appointees in the executive branch of government.

To obtain government-wide information on agency executive training programs, we developed a questionnaire on agency executive training practices and processes and administered it to 27 agencies represented on the CHCO Council. Through the questionnaire, we obtained high-level information on executive training programs, evaluation practices, and opportunities to achieve efficiencies in such programs, as identified by agencies. To the extent available, we also collected data on the amount agencies spent on executive training courses administered by external providers from fiscal years 2008 to 2012. We provided a standard set of questions to each CHCO to ensure we consistently captured their responses to our questions on their executive training practices. We received responses, supporting documentation, or both from 26 of 27 agencies. To analyze the information we received, one analyst inputted responses to quantitative questions and analyzed the responses for univariate statistics using Microsoft Excel; a second analyst checked the input and analysis of responses for accuracy. For the responses to open- ended questions, we conducted a content analysis to identify common themes among the 26 respondent agencies.

We assessed the reliability of executive training participant and cost data from OPM’s Enterprise Human Resources Integration (EHRI) system based on reviews of our prior work on federal training investments, testing of agency data from an OPM executive training dataset, and interviews conducted with OPM officials. We determined that the data provided in EHRI was reliable for the purposes of selecting case examples and for comparing participant and cost data in EHRI to similar data provided directly by agencies. We also assessed the reliability of executive training participant and cost data provided by agencies in response to our questionnaire. Specifically, we examined the data provided for obvious errors and inconsistencies and we also queried 10 agencies to better understand the data systems each agency used to collect and report executive training participant and cost data, as well as the quality of data entered into their system(s). We determined that aggregated participant and cost data for fiscal years 2008 to 2012 are sufficiently reliable for the purposes of the report; however, we qualified these data throughout the report due to potential underreporting from agencies.

We selected three agencies – the Department of Energy (Energy), Department of the Treasury (Treasury), and Department of Veterans Affairs (VA) – to obtain illustrative examples of how they approached executive training in their respective agencies and collected data related to costs; how they implemented and evaluated the results of their executive training programs identified in the questionnaire; and how they identified potential efficiencies and steps planned or already taken to achieve them. We selected these agencies based on workforce size, number of career SES, total training costs, and types of evaluations conducted to measure effectiveness of executive training, and whether agencies identified potential opportunities for achieving efficiencies in executive training. We interviewed agency officials from the three selected agencies with responsibility for administering their agency’s executive training programs. To supplement and support the views provided by agency officials, we also obtained and examined available documentation (such as agency executive training plans, evaluation plans, and other documents associated with executive training) from each of our selected agencies. The findings are not generalizeable to other agencies or to the federal government as a whole, because we designed our selection of agencies to examine the experiences, practices, and perspectives of a set of agencies that, in recent years, have utilized external providers to train executives and have reported that they (1) evaluate the benefits achieved through executive training, and (2) have identified opportunities to achieve efficiencies in executive training.

Additionally, we conducted focus groups of current or prior executive training participants from the three selected agencies in order to obtain their views on the usefulness and value of such training and their ideas for increasing efficiencies in executive training delivery within agencies and government-wide. For two of these agencies, we conducted two focus groups for each: one representing current or recent participants in SES candidate development programs, and one representing career SES members. For the third agency, which did not have recent SES candidates at the time we selected our focus group, we also conducted two focus groups: one representing early career SES members who completed a basic agency leadership course, and one representing mid- to late-career SES members who completed an advanced agency leadership course. The insights gained from the discussions were not generalizeable to all career SES and SES candidates within participants’ agencies or to the federal government as a whole.

We also collected documentation from and interviewed OPM officials involved in the agency’s oversight function for federal training activities in order to obtain information on their roles and responsibilities in monitoring agency training programs, on their perspectives on executive training government-wide, and on steps they have taken to improve training efficiencies. We examined how, if at all, OPM, the CHCO Council, and the Chief Learning Officers Council (made up of chief learning officers from the 27 agencies) share information on potential efficiencies by interviewing OPM officials and members of the CHCO Council and the Chief Learning Officers Council. We interviewed five external training providers who were most frequently used by CHCO Council agencies, to collect information on executive training course offerings, costs, and evaluation strategies; however, the collected information is not generalizeable to all external providers. We also interviewed officials from the Partnership for Public Service and Booz Allen Hamilton who had relevant expertise in federal leadership training activities in order to obtain information on prior studies of executive training, and on their perspectives about evaluating the effectiveness of executive training.

We conducted this performance audit from December 2012 to December 2013, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Office of Personnel Management

Appendix III: GAO Contacts and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Signora May (Assistant Director), Clifton G. Douglas Jr., Janelle House, Andrea Levine, Mark Ryan, Joseph Santiago, and Cynthia Saunders made key contributions to this report. Also contributing to this report were Sara Daleski, Deirdre Duffy, Rob Gebhart, and Rebecca Shea.