DHS Training: Improved Documentation, Resource Tracking, and Performance Measurement Could Strengthen Efforts
Released: 2014-09-10T00:00:00

Letter

Background

DHS Training and Development Roles and Responsibilities

DHS’s OCHCO is responsible, in broad terms, for the strategy, oversight, and planning of DHS employee training and development. At the same time, each DHS component, such as CBP and TSA, also has its own human capital office and training and development functions. In practice, DHS’s OCHCO focuses on department-wide efforts while each component focuses on ensuring its employees are trained and developed to meet its specific mission needs. In addition, FLETC, a component of DHS, offers and delivers law enforcement training to DHS components, including those in our review—CBP, ICE, TSA, and the Coast Guard. FLETC also serves as an interagency law enforcement training organization for more than 90 federal partner organizations, as well as state, local, tribal, and international entities. Table 1 provides a summary of training and development responsibilities at OCHCO and DHS components selected for our review.

Overview of Training Evaluation Requirements and Evaluation Models

In 2009, OPM developed and published regulations that require agencies to regularly evaluate training programs. Among other things, these regulations require agencies to evaluate their training programs annually to determine how well such plans and programs contribute to mission accomplishment and meet organizational performance goals.

The training and development process can loosely be segmented into four broad, interrelated elements: (1) planning/front-end analysis, (2) design/development, (3) implementation, and (4) evaluation. The four elements help to produce a strategic approach to federal agencies’ training and development efforts. One commonly accepted training evaluation model, which is endorsed by OPM in its training evaluation guidance, is known as the Kirkpatrick model. This model is commonly used in the federal government, including at DHS. The Kirkpatrick model consists of a four-level approach for soliciting feedback from training course participants and evaluating the impact the training had on individual development, among other things. The following is a description of what each level within the Kirkpatrick model is to accomplish:

Level 1: The first level measures the training participants’ reaction to, and satisfaction with, the training program. A level 1 evaluation could take the form of a course survey that a participant fills out immediately after completing the training.

Level 2: The second level measures the extent to which learning has occurred because of the training effort. A level 2 evaluation could take the form of a written exam that a participant takes during the course.

Level 3: The third level measures how training affects changes in behavior on the job. Such an evaluation could take the form of a survey sent to participants several months after they have completed the training to follow up on the impact of the training on the job.

Level 4: The fourth level measures the impact of the training program on the agency’s mission or organizational results. Such an evaluation could take the form of comparing operational data before and after a training modification was made. Figure 1 highlights the elements of the training development process, from the planning stage through the implementation and evaluation of training, and depicts how the Kirkpatrick model complements the training development process.

In addition to utilizing this training development process, agencies may also seek Federal Law Enforcement Training Accreditation (FLETA) for some or all of their training programs and academies. By attaining FLETA accreditation for their training academies or programs, agencies provide assurance that they have voluntarily submitted to a process of self- regulation and have successfully achieved compliance with a set of standards that demonstrate their adherence to quality, effectiveness, and integrity. FLETA accreditation also helps maintain training standards by ensuring that training programs are comprehensively evaluated, using Kirkpatrick levels 1 through 3 or an equivalent approach, within a 5-year period.

Overview of DHS-Specific Guidance for Developing and Evaluating Training and Development Efforts

In October 2010, DHS issued its DHS Learning Evaluation Guide. DHS created the guide to help the department’s learning and development community evaluate the effectiveness of training activities in a diverse organization with varied training needs. Among other things, the guidance gives an overview of best practices and provides components with tools they can use to implement the Kirkpatrick model in their training evaluations. In addition, the guide highlights the need for a training evaluation plan to identify and address (1) what is being evaluated, (2) how it is being evaluated, (3) when it is being evaluated, and (4) the factors involving stakeholder expectations, such as agency policies and procedures.

Overview of DHS’s Leader Development Program

In 2004, the Secretary of Homeland Security announced the “One DHS” policy, which identified the need for a common leadership competency framework across the department, as well as a unified training curriculum for current and future leaders. Accordingly, DHS established the LDP Office in May 2010 under the Office of the Chief Human Capital Officer to design, develop, and execute a department-wide leadership program that would strengthen leadership at all levels of the DHS workforce. Through the LDP, all DHS components are to invest in developing leaders with skills that transfer across the department, yet retain the agility to balance this with their own mission-focused leader development needs. In January 2011, DHS also developed the Leader Development Framework to serve as a 3- to 5-year strategic roadmap for implementing the LDP. This framework consists of five tiers of leader development programs for employees of different levels, such as the executive and supervisory levels.

In February 2013, DHS issued a directive—Directive 258-02: Leader Development—formally establishing responsibilities and policies related to leader development at DHS through the LDP, as well as instructions for implementing the directive. This directive specifies that the LDP is, among other things, to delineate requirements and activities to be implemented by components. The LDP is also to develop and manage centrally coordinated and high-potential programs for developing employees to fill future leader positions.

DHS Processes for Evaluating Training Programs Could Be Better Documented and More Reliably Capture Costs

DHS components have documented processes in place for evaluating their training programs and have used evaluation feedback to improve their training offerings; however, their documented processes varied on the extent to which selected attributes of an effective training evaluation process were included. Further, DHS identified opportunities for efficiencies and cost savings, but varying approaches for capturing training costs across the department affect DHS’s ability to reliably capture and track its training costs department-wide.

DHS Components Use Evaluations to Improve Training, but Documenting Selected Attributes of Their Evaluation Processes Could Improve Transparency and Consistency

The five DHS components in our review have a documented process in place for evaluating their training programs using the Kirkpatrick four-level model. However, their documented evaluation processes varied on the extent to which they included selected attributes of an effective evaluation process. Components use the results of their evaluations to make improvements to the training programs and assess training needs. For example, components used evaluation feedback to improve the delivery of training content, such as through additional hands-on training, and the use of e-learning. Table 2 provides such an example for each component.

OPM guidance on training evaluation, DHS’s learning evaluation guidance, and our prior work on effective training and development programs identify various attributes for effective training evaluations. Consistent with these criteria, the attributes of an effective training evaluation process include communicating (1) the goals the training programs are supposed to achieve, (2) which training programs will be evaluated, (3) the methodology for conducting the evaluations, (4) timeframes for conducting the evaluation, (5) roles and responsibilities for evaluation efforts, and (6) how the evaluation results will be used.

All DHS components in our review reflected a number of the attributes of an effective training evaluation process in their documentation. For example, all components included information on identifying goals that the training programs are to achieve, identifying which training programs are to be evaluated, and explaining how the evaluation results will be used. Table 3 presents information on the extent to which each component’s documented evaluation process includes these attributes, and additional details about component ratings are explained in appendix II.

However, components varied on the extent to which they included information in their documentation about evaluation methodologies, timeframes, and roles and responsibilities for evaluation. For example,

Evaluation methodologies: Each component’s documentation indicates that its training programs are to be evaluated using the Kirkpatrick model. However, only ICE’s and the Coast Guard’s documentation specify the methods to be used when performing each Kirkpatrick level of evaluation.

The other three components’ documentation does not specify how each Kirkpatrick level of evaluation is to be performed in practice.

Timeframes: ICE and FLETC’s documented evaluation processes fully presents timeframes for evaluations, but documentation for the remaining three components does not. For example, all of the components’ documentation identified the timeframes for the initial steps of completing evaluation surveys and collecting data. However, ICE and FLETC’s documentation also communicated timeframes for the subsequent steps for analyzing the results of the evaluation.

Roles and responsibilities: We found that three of five components did not consistently outline the roles and responsibilities for the evaluation efforts. For instance, two components, CBP and ICE, communicate information on roles and responsibilities for some parts of the process, but do not present this information for others. TSA’s documentation did not communicate information on the specific roles and responsibilities within the Office of Training and Workforce Engagement (OTWE) for evaluation activities.

Officials at one component, CBP, told us that since training at CBP is more decentralized through separate academies that follow their own processes, their documentation did not include some attributes of effective training and development programs, as their training standards were intended to be more of a “how-to guide” rather than a formal step- by-step methodology. According to TSA officials, they did not fully include certain attributes such as explaining the methodology to be used to conduct the evaluations and defining roles and responsibilities because their process is still under development and agreement on this information has not yet been reached internally. TSA officials stated that their documentation is to be finalized by the first quarter of fiscal year 2015. Officials at the Coast Guard told us that their documentation did not include information on the timeframes for analyzing the results, but they plan to rectify this with their current effort in fiscal year 2014 to revise their evaluation processes. Two of the five components–ICE, and FLETC— did not provide a reason for why their documentation did not include all of the attributes of effective training and development programs.

All DHS components agreed that having a documented training evaluation process provides benefits, such as helping to ensure consistency and transparency across the organization. Accordingly, as previously noted, TSA is working to finalize its training evaluation process. In addition, officials from two components, the Coast Guard and CBP, stated they plan to revise their documented processes in the near future. Specifically, according to Coast Guard officials, revisions to the training evaluation process are being made to more explicitly communicate their process, enhance standardization, and facilitate prioritizing its training evaluation efforts to focus on the most mission- critical training needs. As the process is under way, Coast Guard officials were not able to provide an estimate for when these revisions should be completed. Similarly, according to CBP officials, they plan to review and revise CBP’s training evaluation process to ensure consistency across the component. CBP believes this is necessary given that its training functions have become more decentralized since its last training evaluation process came into effect in 2008. According to CBP officials, the target completion of revisions is fiscal year 2015, after the reorganization of CBP’s Office of Training and Development is complete.

By ensuring that the components’ documented evaluation processes fully address attributes for effective training as they are drafted, updated or revised, DHS would have better assurance that the components have complete information to guide its efforts in conducting effective evaluations. Moreover, such documentation could help ensure that evaluation processes for assessing whether training programs appropriately support component and DHS needs can be repeated and implemented consistently. As components draft, update, and revise their documented evaluation processes, incorporating or more fully addressing the aforementioned attributes of effective training evaluations could help to ensure that components clearly communicate all aspects of their evaluation processes and that employees can consistently implement them.

DHS and Components Have Identified Opportunities for Efficiencies and Cost Savings, but Varying Approaches for Capturing Training Costs at DHS Affect Reliability

All DHS components reported reviewing the merits of different delivery mechanisms (e.g., classroom or computer-based training) to determine which mix would be the most efficient for at least one of their training programs. In addition, four of five components—CBP, the Coast Guard, ICE, and TSA—provided at least one illustrative example of how they used a mix of webinars, online learning, and classroom instructor-led training to develop a blended learning approach that improved cost- effectiveness. For example, in June 2013, TSA implemented webinar training for Sensitive Security Information (SSI), which helped TSA avoid travel costs and led to an estimated $855,026 in cost savings. In addition, ICE developed online training for Fourth Amendment instruction, which helped reduce the course length by 1 week, contributing to opportunity cost savings and savings in room and board totaling about $4.8 million over a 5-year period. The components we reviewed that used webinars or online learning stated that these delivery mechanisms did not adversely affect the quality of training offered in these instances. Furthermore, all DHS components reported that they have evaluated at least one training program to determine how to streamline or consolidate the training to make it more cost-effective. For example, FLETC adopted firearms simulation technology and more cost-effective ammunition, and according to our analysis of FLETC data, led to about $2.2 million in cost savings. See figure 2 for a photographic example of training using firearms simulation.

Similarly, at the department level, OCHCO has taken steps to streamline mandatory department-wide training requirements for counterintelligence and records management training. For example, based on review of legal requirements, OCHCO found that it could streamline mandatory records management training by consolidating multiple annual training requirements into a single course. According to OCHCO officials and our analysis of OCHCO data, this effort could lead to a potential cost saving of about $57.1 million over a 5-year period.

Table 4 provides illustrative examples of actions OCHCO and selected components have taken to improve the cost-effectiveness of training and the estimated cost savings from these actions over a 5-year period, according to our analysis of OCHCO and DHS components.

Starting in fiscal year 2014, CBP adopted a new approach to improve the identification of efficiencies in training, including regular reviews of training justification, cost, and prioritization of training needs. Though the process is not yet documented in policy, directives, or standard operating procedures, CBP officials report that it has allowed them to identify training cost discrepancies more consistently and efficiently. For example, according to CBP officials, their approach includes tracking key cost elements, such as travel, lodging, meal, rental vehicle requirements; duration of training; instructor costs; and contract costs, for all training courses under separate codes. Further, according to these officials, this allows CBP to better compare costs for course execution, such as the cost of course resources (e.g., delivery location, equipment, etc.) and related travel, if any, from year to year. In addition, CBP uses these data to challenge requests for training and identify possible alternatives for delivering existing courses at a reduced cost. The process also provides CBP with a vehicle for better projecting training costs. According to CBP officials, the new approach created requirements that internal offices provide more precise estimates of the number of participants attending training, which reportedly helped CBP more efficiently allocate about $5.8 million in fiscal year 2013. Before the new process was implemented, internal offices could not commit to filling training courses and slots, resulting in CBP’s Office of Training and Development overprojecting about $7 million in training costs in fiscal year 2012, which was returned to CBP. According to CBP officials, leadership change, overestimation of training funding needs, and spending reductions under sequestration in fiscal year 2013 were some of the key reasons for adopting this new approach. CBP officials reported that by refining cost projections, CBP improved their ability to approve more training within allocated budgetary resources. For example, based on the new approach in fiscal year 2014, CBP identified surplus training funds from unfilled training slots and class cancellations early enough to enhance training programs.

In addition, although DHS and components provided illustrative examples of efficiencies in training and cost savings, DHS uses different methods to capture training costs. DHS, through OCHCO, has worked to capture the cost and delivery of DHS’s training and development programs. However, at DHS headquarters and at the component level, there are inconsistencies in how training costs are captured across the department that have made it a challenge to accurately and reliably capture these costs across DHS. For example, OCHCO officials explained that the lack of a centralized funding source and disparate financial management systems used by components created challenges in reliably capturing training costs. Components also often capture training costs differently from one another, which can contribute to inconsistencies among training costs captured at DHS. Training costs may, for example, include expenses for instructional development, participant and instructor salary and benefits, equipment costs, and travel and per diem expenses. Accordingly, OCHCO officials report that some components include conferences as a training cost while others do not, and some components did not include mission-critical law enforcement training costs when they provided department-wide training costs.

In fiscal year 2012, the DHS Undersecretary for Management requested that OCHCO collect training cost data from components. During this process, OCHCO relied on senior-level data requests to retrieve annual training expenditure information from components. According to OCHCO officials, the senior-level data call process revealed that training cost data had limited reliability because some components were not consistent in determining the types of mission-critical training costs they provided, among other things. Given ongoing concerns about data reliability, OCHCO officials noted that it would be difficult to update and reliably aggregate department-wide training costs for fiscal year 2013. According to OCHCO officials, given budget constraints, it is difficult for OCHCO to make good investment decisions about training when they do not know how components spend their training dollars.

In addition, according to discussions with ICE officials, we found that the cost of ICE’s training and development programs may not be consistently and accurately captured. For example, ICE officials stated that participant and instructor salaries are consistently tracked as part of training costs, but travel expenses are less consistently tracked for all ICE internal training programs. Further, according to ICE officials, incomplete definitions of training and inconsistency in how costs are tracked also contribute to shortfalls in reliably capturing training costs. ICE officials reported, for example, that they cannot reliably capture training costs from one of ICE’s internal departments and, instead, need to rely on sporadic data calls to retrieve training budget and expenditure information from its departments. ICE officials reported concerns about the reliability of this process, partly because of concerns about inconsistent coding schemes for tracking similar training activities and the lack of third-party checks on the reliability of how training information is coded. As of August 2014, ICE officials report that their Office of the Chief Financial Officer is working to standardize its coding schemes—or object class reporting—across ICE programs and plans to implement the revised coding standards in fiscal year 2015.

OCHCO and ICE officials we met with acknowledged that the department has not identified all challenges that prevent DHS from accurately capturing training costs department-wide, but they have taken some preliminary steps toward more consistently defining training and capturing costs. For example, while DHS has not issued central guidance on what should be included in training costs, OCHCO officials noted that they provided a glossary of terms to components in December 2007 to help establish an initial definition of training department-wide. Although the glossary clarifies a number of training-related terms, it does not provide requirements for tracking training costs consistently across components. For example, the glossary notes that training program costs are calculated differently on a component basis. Further, according to ICE and OCHCO officials, DHS discussed the issue of accurately and reliably capturing training and development costs across the department as part of its Training Leaders Council in May 2014. OCHCO officials reported that the use of a standard form for requesting training, known as the federal government’s Standard Form 182, Authorization, Agreement, and Certification of Training, may be one method for improving the tracking of training costs. For example, the Form 182 may help provide for consistent definitions and methods for capturing certain training costs. However, while use of the standard Form 182 would be a positive step, it may not address certain reliability concerns associated with capturing training costs at DHS. For example, the approach may not prevent the duplicative capturing of procurement-related training costs or shortfalls in how training information is entered and captured in each component’s systems. According to the DHS Chief Learning Officer, requiring the use of the Form 182 DHS-wide is still in the preliminary stages of consideration and would require accompanying policy changes. As DHS has not yet made a decision on whether to require use of the Form 182, it does not yet have timeframes for implementing this proposal.

One leading training investment practice is that agencies should capture the cost and delivery of their training and development programs. Our prior work has also shown that agencies need reliable information on how much they spend on training and for what purposes. To capture the cost and delivery of training and development programs, agencies need credible and reliable data from learning management systems as well as accounting, financial, and performance reporting systems. To the extent possible, agencies also need to ensure data consistency across the organization (such as having data elements that are pulled from various systems representing the same type of information). Variations in the methods used to collect data can greatly affect the analysis of uniform, high-quality data on the cost and delivery of training and development programs. Given today’s budgetary constraints and the need to effectively utilize and account for all federal dollars, identifying existing challenges that prevent DHS from accurately capturing training costs department- wide and, to the extent that the benefits exceed the costs, implementing corrective measures to overcome these challenges, could enhance DHS’s resource stewardship.

DHS Is Implementing a Department-wide Leader Development Framework, but Could Strengthen Its Program Assessment

DHS’s Leader Development Program Office is in the process of implementing a department-wide, five-tier Leader Development Framework to build leadership skills across all staff levels. While DHS components generally stated that the LDP framework is beneficial, they raised concerns about its training requirements, which the LDP Office’s planned evaluation efforts may address. Further, the LDP office has developed a program-wide assessment approach to analyze the impact of the LDP that includes tracking 12 performance measures over time. However, the LDP Office could strengthen its performance measurement efforts by clearly identifying its program goals and better incorporating key attributes of successful performance measures we have previously identified.

DHS Has Implemented Portions of the Department-wide Leader Development Framework, and Components Deliver Additional Programs to Develop Their Leaders

DHS has implemented programs in support of two of five tiers within its department-wide Leader Development Framework, and the selected components in our review also deliver additional leader development programs for supervisors, managers, and executives. As previously discussed, DHS established the LDP Office in May 2010 to design, develop, and execute a department-wide program to strengthen leadership at all levels of the DHS workforce. In January 2011, DHS approved the Leader Development Framework as a 3- to 5-year strategic roadmap for implementing the LDP. This Leader Development Framework consists of five tiers that identify envisioned leader development programs for employees of different levels. These tiers, and the employees they include, are the following: executive (members of the Senior Executive Service, Coast Guard admirals, and selected other leaders), manager (nonexecutive employees who supervise other supervisors, lead through subordinate supervisors, and formally supervise at least one supervisory employee), supervisor (employees who accomplish work through, and are directly responsible for, the work of nonsupervisory employees, and who formally supervise only nonsupervisory employees), team lead (nonsupervisory employees formally designated as such or tasked to guide a group of people to results on a program, project, initiative, or task force), and team member (nonsupervisory DHS employees).

The LDP Office has implemented programs within two of the five Leader Development Framework tiers (supervisor and executive), initiated program development within two tiers (team lead and team member), and plans to begin program development within one tier during fiscal year 2014 (manager). According to the LDP Manager, the office prioritized implementation of the supervisor tier at the direction of the then deputy secretary, who identified supervisors as a critical nexus between strategic leadership and employee performance. The LDP Manager stated that the office also prioritized implementation of the executive tier because OCHCO officials were familiar with best practices for instruction for new executives and the then deputy secretary identified particular value in providing new executives with consistent instruction. Within the supervisor tier, the LDP Office has established the Cornerstone Program, which consists of a set of baseline requirements for new and seasoned supervisors at all levels. DHS components may fulfill Cornerstone Program requirements through new or existing training programs, cross- component programming, or a combination thereof. Within the executive tier, the LDP Office centrally administers the 3-week Capstone Cohort Program, which includes discussion forums, operational site visits, and learning activities intended to address real-world strategic issues. As the program is currently implemented, whereas components may elect to send participants to the Capstone Program, they are required by DHS to implement Cornerstone Program requirements.

In addition to implementing programs to support the supervisor and executive leader development framework tiers, the LDP Office has assumed responsibility for administering two preexisting DHS programs, the Senior Executive Service Candidate Development Program and DHS Fellows. These programs are not a part of any one tier, as their intended participants may span framework tiers. For example, the Senior Executive Service Candidate Development Program is designed for Senior Executive Service candidates who aspire to transition into the executive tier. Figure 3, an interactive graphic, describes the development and implementation status of the programs that support each tier as of August 2014. See appendix III for a print version of this figure. LDP Office officials anticipate fully implementing all five tiers of the Leader Development Framework before the end of fiscal year 2016.

Move mouse over the program names for more information. For a text version please see app. III.

Component program(s)

The five DHS components selected for our review have all participated in the LDP department-wide programs. In particular, according to LDP Office and component data and officials, all five components have programs in place, as required by DHS, intended to meet the Cornerstone Program requirements. For instance, data from the selected components demonstrate that the Fundamentals of DHS Leadership courses they delivered—one of four program segments of the Cornerstone Program— from fiscal year 2012 through fiscal year 2013 had more than 3,600 participants. According to the LDP Manager, the LDP Office has sought to avoid duplication of effort and costs in components’ implementation of the Cornerstone Program. For example, the LDP Office coordinated the development of instructional materials for all components to use to meet requirements for the Understanding the DHS Leadership Commitment segment of the Cornerstone Program, which is for individuals considering the supervisory path. In addition, information that a DHS working group collected from components during initial development of the Cornerstone Program indicated that most components could utilize existing programs to help meet program requirements. Specifically, six of the seven components that provided information to the working group indicated that they had an existing training program that they could use to provide instruction to first-time supervisors. For instance, FLETC uses two preexisting programs to meet Fundamentals of DHS Leadership requirements—the FLETC New Supervisor Training Program and the Law Enforcement Supervisor Leadership Training Program.

In addition to maintaining programs to meet Cornerstone Program requirements, each of the selected components has elected to participate in the three department-wide programs administered centrally by the LDP Office—the Capstone Cohort Program, Senior Executive Service Candidate Development Program, and DHS Fellows. In particular, these programs had a total of approximately 60 participants from the selected components from fiscal year 2012 through fiscal year 2013, according to LDP Office data. Table 5 summarizes approximate participation in Leader Development Framework programs that were provided by the selected components to meet department-wide requirements or centrally administered by the LDP from fiscal year 2012 through fiscal year 2013, according to LDP Office and component data.

In addition to the programs administered under the Leader Development Framework, the components in our review also deliver various additional leader development programs. For example, Coast Guard delivers a 1- week course for newly selected executives focused on change management issues and TSA delivers a program for supervisors over a period of 18 to 24 months that includes training, shadowing, mentoring, and other developmental leadership opportunities. Table 6 provides examples of leader development programs delivered by these components at the executive, supervisor, and manager levels.

As shown in table 7, according to data provided by the selected components, leader development programs they delivered for supervisors, managers, and executives, independent of department-wide programming, from fiscal year 2012 through fiscal year 2013 had a total of more than 10,000 participants.

Officials from the components in our review generally stated that it is beneficial for their components to provide leader development programs in addition to the LDP department-wide training because, whereas LDP training focuses on more general leadership skills and competencies, component-level training is tailored to the components’ unique missions. For example, TSA officials explained that TSA leader development programs focus on developing individuals for TSA’s mission-critical occupational areas (e.g., federal security directors and federal air marshals), which require proficiency in a set of leadership and technical competencies unique to TSA. In addition, TSA leader development programs utilize examples that are readily applicable to day-to-day TSA operations, according to these officials. Similarly, according to Coast Guard officials, their leader development programs afford the Coast Guard the opportunity to provide instruction using case studies and in- class discussions on how to lead a Coast Guard workforce in a Coast Guard context.

Component Officials Generally Stated the LDP Framework Is Beneficial, but Raised Concerns about LDP Requirements, Which LDP Assessment Efforts May Address

Component officials we spoke with generally agreed that the LDP is helpful in providing a common framework for leader development training. However, officials from three of the five components we met with raised concerns about the applicability or clarity of certain learning objectives the LDP requires they teach when implementing the Cornerstone Program.

Officials from one component stated that the LDP has established policy and procedures, which help to ensure all components are informed and have consistent definitions and policies related to leader development. Components also identified other benefits of the LDP, such as bringing focus across the department to leader development, allowing for collaboration on leader development activities, and having experienced staff who work solely on leader development issues and programs.

As previously discussed, the Cornerstone Program consists of requirements in four segments, one of which is a course on the fundamentals of DHS leadership that is required for all first-time federal supervisors. In order to fulfill LDP requirements for this course, all components must provide instruction on more than 200 learning objectives that identify content the course must cover. For example, these learning objectives include encouraging respect for individual differences and determining appropriate tasks to delegate. According to DHS guidance on the LDP, all DHS components are to develop leaders with skills that transfer across DHS, yet retain the agility to advance their own unique mission-focused leader development needs. However, officials from two components raised concerns about the applicability of certain objectives that the LDP requires them to teach. For example, officials from one component stated that the learning objective involving supervising a workforce of federal employees and contractors is not universally applicable because supervisors in their component do not supervise contractors. Officials from another component stated that one objective related to supervisors’ knowledge of the hiring process does not pertain to new supervisors within their component. According to these component officials, requiring them to teach objectives that are not pertinent to tasks supervisors must perform takes away from instructional time that they could use to teach more relevant content. According to the LDP Manager, the LDP Office established learning objectives in order to meet DHS’s direction to ensure sufficient consistency in leader development investment across components, but components may adapt the objectives, as appropriate. For example, TSA requested a waiver from teaching learning objectives focused on Title 5 of the U.S. Code, from which TSA is generally exempt. The LDP Manager granted the waiver, and suggested that TSA replace instruction focused on Title 5 with instruction on related subjects applicable to TSA. However, officials from the two components that raised concerns were not aware that Cornerstone Program requirements provided them with this flexibility.

Officials from two components raised concerns that some of the learning objectives required to be taught under the Fundamentals of DHS Leadership course do not clearly articulate what the training must cover, and that they are not written with standards that can be measured or observed. For example, according to officials from one component, some Fundamentals of DHS Leadership objectives are not consistent with their component’s standards, which require that performance objectives include condition, measurable performance behavior, and a standard that specifies the degree of quality expected in performance. For instance, one of the learning objectives that officials identified as not meeting these requirements states, “Recognize a recent study that reported 48 percent of workers surveyed responded to job pressure by performing illegal or unethical activities; 58 percent considered acting illegally or unethically.” A senior official from this component explained that this can result in implementation and evaluation challenges—if it is unclear what the outcome of an objective is supposed to be, it is difficult to know how to implement it or evaluate its implementation. This official also stated that officials from his component voiced their concerns about the clarity of learning objectives to DHS headquarters, but DHS did not change them. According to LDP Office officials, they solicited input from components on Cornerstone Program requirements and adopted selected changes. In particular, according to the LDP Manager, the LDP Office solicited input from components during four informal and two formal reviews of Cornerstone Program requirements.

The LDP Office has also awarded a contract for an assessment beginning in February 2014 that includes evaluation of the Fundamentals of DHS Leadership’s learning objectives. Scheduled for completion by September 2014, this assessment may help to address concerns raised by components. This assessment is to determine the Cornerstone Program’s overall implementation status, determine the effectiveness of the program’s products and elements, evaluate the efficacy of the Fundamentals of DHS Leadership’s learning objectives, and recommend specific tactical and strategic changes for improving program effectiveness.

Clearly Identifying Program Goals and Enhancing Performance Measures Could Strengthen LDP Assessment Efforts

The LDP has developed a program-wide assessment approach intended to analyze the impact of the LDP over time and assess whether the program is targeting the right things in the right way. However, the LDP Office could strengthen this assessment approach by more clearly identifying its program goals and ensuring its 12 performance measures incorporate key attributes of successful performance measures we have previously identified. The LDP’s assessment approach applies to all LDP program elements, including Capstone, Cornerstone, and other programs. The approach consists of (1) biannually collecting and analyzing completion rate data for all LDP programs implemented by components, (2) collecting and analyzing responses to six core evaluation questions immediately following each developmental activity and 6 months later, and (3) tracking 12 program performance measures. Table 8 provides some examples of these 12 measures.

For more detailed information about the LDP’s assessment approach, see appendix IV.

Developing this assessment approach is a positive step toward assessing the effectiveness of the LDP. However, the LDP Office has not clearly identified goals for the program, and the 12 measures that the office has developed to assess its performance do not consistently exhibit attributes we have previously identified as key for successful measurement. These key attributes include having linkage with division- and agency-wide goals, being clear, and having measurable targets. Table 9 presents definitions of these attributes along with potentially adverse consequences of not meeting them.

Performance measurement is the ongoing monitoring and reporting of program accomplishments, particularly progress toward preestablished goals. We have previously reported that performance measurement allows organizations to track progress in achieving their goals and gives managers crucial information to identify gaps in program performance and plan any needed improvements. In addition, according to Standards for Internal Control in the Federal Government, managers need to compare actual performance against planned or expected results and analyze significant differences. We observed the following when assessing the LDP’s performance measures against these selected key attributes:

Linkage: The LDP’s 12 performance measures do not clearly link with program goals and linkage is not clearly communicated throughout the organization. The LDP has identified working program goals, but they are disparately documented and not clearly identified as goals. When we asked LDP Office officials to identify the LDP’s goals, the LDP Manager referred us to statements in Directive 258-02: Leader Development, and provided us with a list of working program goals assembled from statements in various LDP materials. These working program goals included, for example, using best practices to maximize effectiveness and elevating the importance of developing leaders department-wide. However, neither the directive nor LDP materials clearly identify or refer to the statements the manager directed us to as goals. Given that the LDP’s identified working program goals are disparately documented and not clearly identified as goals, it is unclear whether the LDP’s 12 performance measures align with the statements and working program goals the LDP Manager identified. We tried to identify linkage between the LDP’s 12 performance measures and the informal goals the LDP Manager identified, but could not determine definitively how they relate.

According to the LDP Manager, the LDP Office did not clearly document the program’s goals in one place or record their linkage to the program’s performance measures because it established the goals as it developed programs to support the Leader Development Framework. In addition, the LDP Manager stated that DHS’s strategic human capital–related plans— which include goals—were under development when the office developed the measures. The LDP Manager explained that, in developing the performance measures, an LDP program official used DHS department- wide strategic plans, direction from DHS officials and stakeholders, and guidance from the DHS Learning Evaluation Guide—which provides guidance for evaluating the effectiveness of training activities—to determine categories for the performance measures and then developed measures pertaining to each category. In addition, DHS uses data that the LDP Office collects for its performance measures for strategic planning and reporting, according to the LDP Manager. For example, DHS uses data the LDP Office collects to measure progress against two goals established in the DHS Workforce Strategy for Fiscal Years 2011- 2016. We agree that data collected to track the measures may provide information for measuring progress against some department-wide goals established in strategy documents; however, it is not evident how these or other LDP performance measures link to goals specific to the LDP.

We have previously found that linkages between goals and measures are most effective when they are clearly communicated to all staff within the agency so that everyone understands what the organization is trying to achieve. Explicitly identifying program goals, creating an evident link between performance measures and program goals, and clearly communicating the linkage could help ensure that the behaviors and incentives created by the LDP’s 12 performance measures support the LDP’s intended outcomes, and that they are appropriate measures for the program.

Clarity: Not all LDP performance measures possess clarity because some measures include terms that are ambiguous and for which the LDP Office has not documented definitions. For example, one of the LDP’s measures is the percentage of developmental activities that fulfill LDP requirements delivered with shared resources. However, it is not clear what constitutes a “developmental activity” for the purpose of calculating this measure (e.g., a course or unit within a course), and the value could be different depending on the definition of “activity” used in the measure’s calculation. As a result, this measure could be confusing and misleading to users, such as DHS leadership and congressional constituents, by leading them to think that performance was better or worse than it actually was. According to the LDP Manager, the LDP Office has not documented definitions for terms used in its performance measures because components may elect to fulfill LDP requirements through varied approaches and this makes how terms such as “developmental activity” are defined contextually driven. While we recognize that components may use varied approaches to fulfilling LDP requirements, it is important that terms the LDP Office uses in its performances measures are clear so that users understand what the measures mean. LDP officials also stated that components are able to contact the LDP Office with questions about how to calculate the measures, and that the LDP Office will work with components to provide any requested clarification. Providing support to components is a positive step, but documenting definitions for ambiguous terms used in the measures could help ensure the meaning of their values is clear to stakeholders.

Targets: The LDP’s performance measures do not have measurable targets. According to the LDP Manager, the LDP has not set targets for its 12 program performance measures because it is too early in the process, as LDP officials have just established a baseline with fiscal year 2013 data. The LDP Manager anticipates developing LDP targets in the future and stated that program officials will consider doing so once they collect more data. The LDP Office does not have a definitive plan or time frame for setting targets, but according to the LDP Manager, expects to do so using fiscal year 2014 data. We agree that, consistent with key attributes of performance measures, developing measurable targets could help DHS determine whether the program’s performance is meeting expectations. To set appropriate targets, however, it will be important for the LDP Office to first clearly identify program goals and ensure its performance measures link to the goals.

DHS leadership has previously identified implementation of leader development programs as important to the department’s success and a means by which to improve its human capital management. For example, in February 2012, the then deputy secretary of homeland security stated that leader development is critical to DHS’s growth and long-term success and must be a strategic mission investment priority. In addition, DHS has identified implementation of the LDP among the actions it is taking to address DHS’s high-risk designation with respect to human capital management. By clearly identifying program goals and ensuring LDP performance measures include key attributes, such as linkage, clarity, and measurable targets, the LDP could strengthen its performance measurement, consequently producing actionable information for LDP management to use in identifying the need for, and making, program improvements.

Conclusions

As DHS faces increasingly complex national security challenges, it is important that it support employees with effective training and development programs to meet its mission requirements. Evaluating training and development programs is important for ensuring that such programs are cost-effective and continue to be relevant for the department. By updating DHS components’ documented training evaluation processes to more fully address key attributes for effective training evaluation, DHS components could have better assurance that the components have more complete information to guide their efforts in conducting effective evaluations. Such documentation can further help ensure that processes for assessing whether training programs support component and DHS needs are repeatable and consistently implemented. Further, given limited budgetary resources, by identifying existing challenges that prevent DHS from accurately capturing its training costs department-wide and, to the extent that the benefits exceed the costs, implementing corrective measures to overcome these challenges, DHS could improve its awareness about the actual costs of its training programs, and enhance its ability to consistently and reliably capture training costs DHS-wide, thereby enhancing its resource stewardship.

In addition, DHS is in the process of implementing a department-wide leader development program to build leadership skills across all staff levels. The effectiveness of this program is particularly important given that DHS leadership has identified leader development as critical to the department’s success. As DHS begins to assess the impact of the LDP program, clearly identifying LDP goals and ensuring that LDP performance measures possess key attributes, including (1) linkage to the program’s goals, (2) clarity, and (3) measurable targets by which to assess the measures could help provide DHS with the actionable information it needs to identify and make program improvements.

Recommendations for Executive Action

To ensure effective evaluation of federal training programs and enhance DHS’s stewardship of resources for federal training programs, we recommend that the Secretary of Homeland Security take the following two actions: direct DHS components to ensure that their documented training evaluation processes fully address attributes for effective training evaluation processes as they are drafted, updated, or revised and identify existing challenges that prevent DHS from accurately capturing training costs department-wide and, to the extent that the benefits of addressing those challenges exceed the costs, implement corrective measures to overcome these challenges.

In addition, to produce actionable information for DHS’s LDP management to use in identifying the need for, and making, program improvements, we recommend that the Secretary of Homeland Security direct the Chief Human Capital Officer to clearly identify LDP goals and ensure LDP performance measures include key attributes, such as linkage, clarity, and measurable targets.

Agency Comments and Our Evaluation

We provided a draft of this report to DHS for review and comment. DHS provided written comments, which are reprinted in appendix IV, and technical comments, which we incorporated as appropriate. DHS agreed with all three of the recommendations and outlined steps to address them. If fully implemented, these actions will address the intent of our recommendations.

With respect to the first recommendation, DHS agreed to ensure that effective training evaluation processes are documented and in place at components by incorporating a review of component training evaluation documents into the DHS Chief Human Capital Officer’s audit of human resource functions. DHS reports that a full review of components should be completed with the fiscal year 2019 audit cycle.

Regarding the second recommendation, DHS agreed to resolve the issue of capturing training costs Department-wide. For example, DHS plans to establish a team jointly chaired by the DHS Chief Human Capital Officer and the Chief Financial Officer, and comprised of representatives from both financial and training offices of each operational component and headquarters, that is to deliver a methodology to track and report training costs across DHS by June 30, 2015. DHS anticipates the new methodology will be implemented across all components by January 31, 2016.

In response to the third recommendation, DHS agreed to publish clear DHS Leader Development Program goals and performance measures that include key attributes, such as linkage, clarity and measurable targets on the DHS intranet website by December 31, 2014.

We are sending copies of this report to appropriate congressional committees and the Secretary of Homeland Security. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

Should you or your staff have any questions concerning this report, please contact me at (202) 512-9627 or by e-mail at maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VI.

David Maurer Director, Homeland Security and Justice .

Appendix I: Objectives, Scope, and Methodology

Our objectives for this report were to address the following questions: 1. To what extent does the Department of Homeland Security (DHS) have documented processes to evaluate training and development programs and reliably capture costs? 2. What leader development programs has DHS implemented, what are stakeholders’ perspectives on them, and to what extent does DHS measure program performance?

To understand training programs at DHS, we obtained information from the DHS Office of the Chief Human Capital Officer (OCHCO), and five selected components: the Federal Law Enforcement Training Center (FLETC), U.S. Customs and Border Protection (CBP), U.S. Immigration and Customs Enforcement (ICE), the Transportation Security Administration (TSA), and the United States Coast Guard. We selected these components to represent different DHS mission areas, workforce sizes, training costs, and number of career Senior Executive Service (SES) personnel. In addition, these components have a mix of new and more established training programs. When examining training programs at selected components, we reviewed component-level training evaluation and strategic plans when available; training budget requests; cost and expenditure documents; training procedures, policies, and organizational charts; and policies for identifying and prioritizing training programs; selected training course materials, and other relevant documents. To further our understanding of training at the component level, we also interviewed training officials at each of the selected components and identified these individuals based on their knowledge, experience, and leadership roles. We conducted our interviews at component headquarters located in the Washington, D.C. area, or field offices. In addition, as part of our review of DHS’s delivery of mission-critical law enforcement training across components, we observed training at FLETC’s Glynco, Georgia facilities. The perspectives DHS OCHCO and the selected components provided are not generalizable to all of DHS, but provided helpful insights into the selected components specific training and development programs at DHS.

To address the first question, regarding the extent to which DHS has documented processes to evaluate training and development programs, and ensure training costs are reliably captured, we reviewed DHS and component-specific documents and interviewed relevant officials at DHS OCHCO and each of the components. Specifically,

To determine the extent to which DHS has documented processes to evaluate its training and development programs, we reviewed policies and procedures related to the evaluation of training programs, such as component-specific standard operating procedures and training development standards. We then assessed the documented processes from each of the selected components against attributes of training evaluation processes identified by the Office of Personnel Management (OPM), DHS, and GAO to determine the extent to which the documents include selected attributes of evaluation processes. We selected the attributes for our analysis to include attributes that were consistently identified in relevant criteria documents related to training evaluation, such as the DHS Learning Evaluation Guide, the OPM Training Evaluation Field Guide, and GAO’s prior work on training and development, specifically the Guide for Strategic Training and Development Efforts in the Federal Government. These attributes also align with those identified in Standards for Internal Control in the Federal Government, which calls for agencies to document the plans, methods, and procedures used to achieve missions, goals, and objectives and support performance-based management practices. From these sources, we identified six attributes of a training evaluation process to conduct our analysis: (1) establishes goals about what the training program is supposed to achieve, (2) indicates which training programs are being evaluated, (3) explains the methodology used to conduct the evaluation, (4) presents time frames for conducting the evaluation, (5) presents roles and responsibilities for evaluation efforts, and (6) explains how the evaluation results will be used. We assessed each component’s documented evaluation process to determine the extent to which the attributes were included and gave a component a rating indicating that the attribute was fully met, a component partially met the attribute but did not fully or consistently meet all parts, or the component did not include any information to meet the attribute. We also conducted semistructured interviews with officials responsible for conducting training evaluation at each of the five components to understand the evaluation process that each component follows and how evaluation feedback is used.

To assess the extent to which DHS ensures training costs are reliably captured, we reviewed information and relevant documentation on processes and steps components took to examine available budget and cost information. We further reviewed documentation on the process of capturing training costs from each of our selected components, including financial audit reports. As part of our review of cost tracking at DHS, we observed methods components used for identifying efficiencies in training to identify cost savings and employ more cost-effective alternatives. We also conducted semistructured interviews with DHS and component officials responsible for administering training programs and tracking costs to understand how DHS and components identified and captured costs, and any challenges they may have in doing so in a reliable manner. Through our review of cost-saving documentation and interviews with DHS and component officials, we sought illustrative examples to understand how OCHCO and the selected DHS components identified potential efficiencies and steps planned or already taken to achieve them. Accordingly, OCHCO and DHS component officials identified examples of cost savings realized in selected training programs from fiscal year 2011 through fiscal year 2013, and we reviewed the reliability of the related cost-saving estimates. For example, we interviewed knowledgeable officials who provided cost estimates, reviewed the estimates related to cost savings, and replicated cost-saving calculations provided by components, including estimates for training equipment, salaries, and benefits. We determined through analysis of cost-saving estimates and interviews with knowledgeable officials at DHS and the selected components that the cost-saving data provided and reported for the illustrative examples in this product were sufficiently reliable for the purposes of illustrating the types of cost efficiencies that may be achieved. The cost-saving examples DHS OCHCO and components provided are not generalizable to all of DHS, but provided helpful insights into cost-saving efforts identified to date at DHS.

To address the second question, about leader development programs DHS has implemented, we reviewed program documentation, analyzed participant data, and interviewed officials from OCHCO and the selected components.

In particular, to determine what leader development programs DHS has implemented, we reviewed OCHCO Leader Development Program (LDP) curricula and requirements documentation, such as the Senior Executive Service Candidate Development Program Candidate Guide and The Cornerstone Program Requirements and Accountability Guide, and documentation of leader development programs provided by the selected components, such as program descriptions and evaluations. In addition, we obtained and analyzed data from OCHCO and the selected components on the number of participants in the leader development programs they provided during fiscal years 2012 and 2013. We assessed the reliability of these data by interviewing agency officials familiar with the sources of the data regarding internal controls built into the information systems and stand-alone documents in which they are stored and quality assurance steps performed after data are entered into the systems or documents. In addition, we compared participant completion data for the Fundamentals of DHS Leadership segment of the Cornerstone Program—one of DHS’s leader development programs—for similar time periods that components provided to us and had previously reported to the LDP Office. Where we identified discrepancies, we interviewed officials to determine their cause and the correct values. We determined that the data were sufficiently reliable for the purpose of reporting the approximate number of program participants. We also interviewed officials from OCHCO and the components regarding implemented and planned leader development programs.

To determine officials’ perspectives on DHS leader development programs, we obtained OCHCO and component officials’ views on the development and implementation of leader development programs and the programs’ strengths and weaknesses. The perspectives the interviewees provided are not generalizable to all DHS officials, but provided helpful insights into strengths and weaknesses of leader development programs.

To assess the extent to which DHS measures the performance of leader development programs, we reviewed program documentation from OCHCO and the selected components, including performance measurement requirements and guidance. In addition, we interviewed cognizant officials about what performance measurement information they collect and how they use the information. Through these efforts, we determined that the LDP Office uses 12 performance measures to assess the LDP’s impact. We assessed these measures against three of nine selected key attributes for performance measures identified in prior GAO work that we identified as relevant given the maturity level of the LDP. In particular, given that the LDP is a relatively new program, we focused our analysis on three attributes that we identified as foundational and—having linkage with division- and agency-wide goals, being clear, and having measurable targets. We selected linkage because aligning measures with division- and agency-wide goals and mission helps ensure that the behaviors and incentives created by the measures support the division- or agency-wide goals or mission. Once the measures’ relevance to a program is ensured through linkage, then assessment of more detailed aspects of the measures, such as reliability, is more relevant. Similarly, we selected having measurable targets because, without measurable targets, it may not be evident whether performance is meeting expectations. With regard to clarity, if a measure is not clearly stated and the name and definition are not consistent with the methodology used to calculate it, performance data could be confusing and misleading to users, such as department leadership and congressional constituents.

We conducted this performance audit from July 2013 to September 2014, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Presence of Effective Training Attributes in DHS’s Documented Training Evaluation Processes

Establishes goals about what the training program is supposed to achieve 

Component U.S. Customs and Border Protection (CBP) U.S. Immigration and Customs Enforcement (ICE) Coast Guard Transportation Security Administration (TSA)Federal Law Enforcement Training Center (FLETC)

Explains how the evaluation results will be used  . : The component’s documented evaluation processes fully included information to meet the attribute for all aspects of its evaluation process. : The component’s documented evaluation processes included some information to address a given attribute but did not include information to fully and consistently meet all parts of the attribute. This includes, for example, incomplete evaluation processes or incomplete information to address a given attribute for certain levels of the evaluation. : The component’s documented evaluation processes did not include information to address the attribute for the evaluation process.

CBP’s documentation presents ways CBP can implement the Kirkpatrick model but does not indicate the actual process that will be used. CBP’s documentation outlines when the various levels of evaluations are supposed to be administered, but does not present a timeframe for CBP to analyze the evaluation feedback. CBP’s documentation identifies some CBP entities that perform the evaluations and receive the evaluation information, but does not do so consistently for each level of evaluation. ICE’s documentation outlines who is responsible for the various level 1 evaluation activities and what ICE stakeholders should be involved in the process, but does not provide this information consistently for evaluation levels 2 and 3. The Coast Guard’s documentation provides guidance on when to administer the evaluation surveys, but it does not specify timeframes for the Coast Guard to analyze the evaluation data.

TSA’s documentation states that it will evaluate training programs using the Kirkpatrick model and discusses the process in a very general sense. However, the documentation does not indicate specifically how TSA will conduct each level of evaluation or the circumstances in which a certain approach will be used.

FLETC’s documentation indicates that it will evaluate its training programs using the four-level Kirkpatrick model. However, the documents do not consistently indicate how FLETC will develop, administer, and analyze the evaluation data for each level. For example, for level 3 evaluations, FLETC’s documentation includes some policies and procedures that govern the evaluations, but these procedures do not provide specifics on the process such as how the surveys are developed and deployed, and how the surveys are sent to a sample of students, among others.

Appendix III: Leader Development Framework Programs and Implementation Status

The following information appears as interactive content in figure 3 in the report body when viewed electronically.

Appendix IV: Summary of Leader Development Program Assessment Approach

The Department of Homeland Security’s (DHS) Leader Development Program (LDP) Office has developed a program-wide assessment approach intended to analyze the impact of the LDP over time and to assess whether the program is targeting the right things in the right way. This assessment approach, which applies to all LDP program elements— including Capstone, Cornerstone, and other programs—consists of (1) biannually collecting and analyzing completion rate data, (2) collecting and analyzing responses to six core evaluation questions, and (3) tracking 12 program performance measures. Table 11 provides more detailed information about this approach.

Appendix V: Comments from the Department of Homeland Security

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Joseph P. Cruz, Assistant Director; Chuck Bausell; Gary Bianchi; Gustavo Crosetto; Peter DelToro; Michele Fejfar; Eric Hauswirth; Adam Hoffman; Susan Hsu; Kirk Kiester; Tracey King; Taylor Matheson; Signora May; Linda Miller; Julia Vieweg; and Yee Wong made key contributions to this report.