DHS Training: Improved Documentation, Resource Tracking, and Performance Measurement Could Strengthen Efforts
Published: 2014-09-10T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: DHS is the third-largest cabinet-level department in the federal government, with over 230,000 employees doing diverse jobs. To fulfill its complex mission, DHS's workforce must have the necessary skills and expertise. GAO previously reported on DHS's hiring and recruiting efforts. GAO was asked to assess DHS's training practices.
This report addresses (1) the extent to which DHS has documented processes to evaluate training and reliably capture costs and (2) the extent to which DHS measures the performance of its leader development programs. To conduct its work, GAO reviewed documented training evaluation processes, training cost data from fiscal year 2011 through fiscal year 2013, and leadership training programs. GAO also interviewed training officials at the department level and at the five DHS components selected for this review about their varieties of training and development programs. Information from these components cannot be generalized to all of DHS, but provides insights.
section_title: What GAO Found
paragraphs: The Department of Homeland Security (DHS) has processes to evaluate training, track resources, and assess leader development. However, various actions could better position the department to maximize the impact of its training efforts.
Training evaluation: All five DHS components in GAO's review—U.S. Customs and Border Protection, U.S. Immigration and Customs Enforcement, the U.S. Coast Guard, the Transportation Security Administration, and the Federal Law Enforcement Training Center—have a documented process to evaluate their training programs. Their documented processes fully included three of six attributes of effective training evaluation processes identifying goals, programs to evaluate, and how results are to be used. However, the documented processes did not consistently include the other three attributes: methodology, timeframes, and roles and responsibilities (see table). By updating documentation to address these attributes, DHS components would have more complete information to guide its efforts in conducting effective evaluations.
Capturing training cost: DHS identified efficiencies and cost savings for delivering a number of training programs. However, different methods are used for capturing training costs across the department, which poses challenges for reliably capturing these costs across DHS. Components capture training costs differently, contributing to inconsistencies in training costs captured across DHS. Variation in methods used to collect data can affect the reliability and quality of DHS-wide training program costs. However, DHS has not identified all challenges that contribute to these inconsistencies. DHS could improve its awareness about the costs of training programs DHS-wide and thereby enhance its resource stewardship by identifying existing challenges that prevent DHS from accurately capturing training costs and implementing corrective measures.
Leader development: DHS's Leader Development Program (LDP) Office is in the process of implementing a department-wide framework to build leadership skills. However, the LDP Office has not clearly identified program goals and the measures it uses to assess program effectiveness do not exhibit some attributes that GAO previously identified as key for successful performance measurement. These include linkage of performance measures to the program's goals, clarity, and establishment of measurable targets to assess the measures. By clearly identifying program goals and incorporating key attributes, the LDP could better ensure actionable information for identifying and making program improvements.
section_title: What GAO Recommends
paragraphs: GAO recommends that DHS update its documentation to fully reflect key attributes of an effective evaluation, identify challenges to and corrective measures for capturing training costs department-wide, and clearly identify LDP goals and ensure that LDP performance measures reflect key attributes. DHS concurred and identified actions to address our recommendations.

section_title: Letter
paragraphs: The Department of Homeland Security (DHS) is the third-largest cabinet- level department in the federal government, with over 230,000 employees  doing diverse jobs in areas such as aviation, border security, emergency  response, cybersecurity analysis, and chemical facility inspection. To  address increasingly complex national security challenges, it is important  that DHS have a workforce with the skills and expertise to fulfill its  mission. Training and development programs are one way to help ensure  personnel have the necessary skills and to prevent competency gaps.  These programs can include a set of courses using a variety of  approaches, including classroom training, e-learning, webinars, coaching,  practical exercises, and rotational assignments. Effective training and  development programs for DHS’s mission-critical functions, such as law  enforcement, inspections, and screening, are important for enhancing  DHS’s ability to retain employees with the skills and competencies  needed to achieve results. According to DHS officials, DHS spent about  $1.1 billion on training and development programs in fiscal year 2012 and  about 7,000 staff are dedicated to training and development activities  across the department.
In addition, our work in identifying high-risk areas in the federal  government has identified DHS management, including the function of  human capital management, as a high-risk area. DHS’s management of  human capital has been on our high-risk list since 2003 because, among  other things, the department has not fully implemented a mechanism to  assess education, training, and other development programs and  opportunities to help employees build and acquire needed skills and  competencies. In addition, our high-risk work has also identified the need  for DHS to improve employees’ opinions of the quality of departmental  leadership as reflected in DHS’s scores on the Office of Personnel  Management’s (OPM) Federal Employee Viewpoint Survey. DHS uses  training as one of its tools for enhancing departmental leadership. As we  have reported since March 2004, using training evaluations to  demonstrate how training efforts help develop employees, improve  agencies’ performance, and inform decision making on investments in  training is a leading practice for ensuring agencies are being good  stewards of their training and development resources.
Given today’s fiscal realities and the need to deliver cost-effective training  and development programs without sacrificing quality or training  effectiveness, you asked us to evaluate DHS’s training practices, as well  as efforts to ensure training is efficient and effective in developing its next  cadre of leaders. This report will address the following two questions.  1.  To what extent does DHS have documented processes to evaluate  training and development programs and reliably capture costs?   2.  What leader development programs has DHS implemented, what are  stakeholders’ perspectives on them, and to what extent does DHS  measure program performance?
To understand training programs at DHS, we obtained information from  the DHS Office of the Chief Human Capital Officer (OCHCO), and five  selected components: the Federal Law Enforcement Training Center  (FLETC), U.S. Customs and Border Protection (CBP), U.S. Immigration  and Customs Enforcement (ICE), the Transportation Security  Administration (TSA), and the U.S. Coast Guard. We selected these  components to represent different DHS mission areas, workforce sizes,  training costs, and number of career Senior Executive Service (SES)  personnel. To further our understanding of training at the component  level, we also interviewed training officials at each of the selected  components and identified these individuals based on their knowledge,  experience, and leadership roles. The perspectives of DHS OCHCO and  the selected components provided are not generalizable to all training  programs at DHS, but provided helpful insights into the selected  components’ specific training and development programs at DHS.
To address the first question regarding the extent to which DHS has  documented processes to evaluate its training and development  programs, we reviewed documented policies and procedures related to  the evaluation of training programs for the five selected DHS  components, as well as completed training evaluations. We also  conducted semistructured interviews with officials responsible for  conducting training evaluation at each of the five components to  understand the evaluation process that each component follows and how  evaluation feedback is used. We then assessed the documented  processes from each of the selected components against attributes of  training evaluation processes identified by OPM, DHS, and GAO to  determine the extent to which the documents include select attributes of  evaluation processes. We selected the attributes for our analysis by  including six that were consistently identified in relevant criteria  documents related to training evaluation, such as the DHS Learning  Evaluation Guide, the OPM Training Evaluation Field Guide, and  GAO’s prior work on training and development. These attributes also  align with those identified in Standards for Internal Control in the Federal  Government for the plans, methods, and procedures used to accomplish  missions, goals, and objectives and support performance-based  management practices. The six training evaluation process attributes  include assessing whether each component’s documented process (1)  establishes goals about what the training program is supposed to  achieve, (2) indicates which training programs are being evaluated, (3)  explains the methodology used to conduct the evaluation, (4) presents  timeframes for conducting the evaluation, (5) presents roles and  responsibilities for evaluation efforts, and (6) explains how the evaluation  results will be used. We assessed each component’s documented  evaluation process to determine the extent to which the attributes were  included and gave a component a rating indicating that the attribute was  fully met, a component partially met the attribute but did not fully or  consistently meet all parts, or the component did not include any  information to meet the attribute.
To address the extent to which DHS ensures training costs are reliably  captured, we reviewed relevant documentation on processes and steps  taken to examine budget and cost documentation. As part of our review of  cost tracking at DHS, we observed methods OCHCO and the  components used for identifying efficiencies in training that were used to  identify cost savings. We also conducted semistructured interviews with  DHS and component officials responsible for administering training  programs and tracking costs to understand how DHS and components  identified and captured costs, and any challenges they may have in doing  so in a reliable manner. Accordingly, through our review of cost-saving  documentation and interviews with DHS and component officials, we  sought illustrative examples to understand how OCHCO and the selected  DHS components identified potential efficiencies and steps planned or  already taken to achieve them. We assessed the reliability of the reported  cost savings relevant to these illustrative examples and we replicated  cost-saving calculations provided by components, including estimates for  training equipment, salaries, and benefits. We determined through  analysis of cost-saving estimates and interviews with knowledgeable  officials at DHS and the select components that the cost-saving data  provided and reported in this product for the illustrative examples from  fiscal years 2011 through 2013 were sufficiently reliable for the purposes  of illustrating the types of cost efficiencies that may be achieved. The  cost-saving examples DHS and components provided are not  generalizable to all of DHS, but provided helpful insights into cost-saving  efforts identified to date at DHS.
To address the second question about leader development programs  DHS has implemented, we reviewed program documentation relevant to  leadership training programs. In addition, we obtained and analyzed data  from OCHCO and the selected components on the number of participants  in the leader development programs they provided during fiscal years  2012 and 2013. We assessed the reliability of these data by  interviewing agency officials familiar with the sources of the data  regarding internal controls built into the information systems and stand- alone spreadsheets in which the data are stored, and quality assurance  steps performed after data are entered into the systems or documents.  We determined that the data were sufficiently reliable for the purpose of  reporting the approximate number of program participants. We also  interviewed officials from OCHCO and the selected components  regarding implemented and planned leader development programs. To  assess the extent to which DHS measures the performance of leader  development programs, we reviewed program documentation from  OCHCO and the selected components, including performance  measurement requirements and guidance. In addition, we interviewed  cognizant officials about what performance measurement information  they collect and how they use the information. Through these efforts we  determined that the Leader Development Program (LDP) Office uses  performance measures to assess the LDP’s impact. We assessed these  measures against three of nine selected key attributes for performance  measures identified in prior GAO work that we identified as relevant given  the maturity level of the LDP. In particular, given that the LDP is a  relatively new program, we focused our analysis on three attributes that  we identified as foundational—having linkage between performance  measures and division- and agency-wide goals, being clear, and having  measurable targets. Additional details on our scope and methodology can  be found in appendix I.
We conducted this performance audit from July 2013 to September 2014,  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: DHS Training and Development Roles and Responsibilities
paragraphs: DHS’s OCHCO is responsible, in broad terms, for the strategy, oversight,  and planning of DHS employee training and development. At the same  time, each DHS component, such as CBP and TSA, also has its own  human capital office and training and development functions. In practice,  DHS’s OCHCO focuses on department-wide efforts while each  component focuses on ensuring its employees are trained and developed  to meet its specific mission needs. In addition, FLETC, a component of  DHS, offers and delivers law enforcement training to DHS components,  including those in our review—CBP, ICE, TSA, and the Coast Guard.  FLETC also serves as an interagency law enforcement training  organization for more than 90 federal partner organizations, as well as  state, local, tribal, and international entities. Table 1 provides a summary  of training and development responsibilities at OCHCO and DHS  components selected for our review.
section_title: Overview of Training Evaluation Requirements and Evaluation Models
paragraphs: In 2009, OPM developed and published regulations that require agencies  to regularly evaluate training programs. Among other things, these  regulations require agencies to evaluate their training programs annually  to determine how well such plans and programs contribute to mission  accomplishment and meet organizational performance goals.
The training and development process can loosely be segmented into  four broad, interrelated elements: (1) planning/front-end analysis, (2)  design/development, (3) implementation, and (4) evaluation. The four  elements help to produce a strategic approach to federal agencies’  training and development efforts. One commonly accepted training  evaluation model, which is endorsed by OPM in its training evaluation  guidance, is known as the Kirkpatrick model. This model is commonly  used in the federal government, including at DHS. The Kirkpatrick model  consists of a four-level approach for soliciting feedback from training  course participants and evaluating the impact the training had on  individual development, among other things. The following is a  description of what each level within the Kirkpatrick model is to  accomplish:
Level 1: The first level measures the training participants’ reaction to, and  satisfaction with, the training program. A level 1 evaluation could take the  form of a course survey that a participant fills out immediately after  completing the training.
Level 2: The second level measures the extent to which learning has  occurred because of the training effort. A level 2 evaluation could take  the form of a written exam that a participant takes during the course.
Level 3: The third level measures how training affects changes in  behavior on the job. Such an evaluation could take the form of a survey  sent to participants several months after they have completed the training  to follow up on the impact of the training on the job.
Level 4: The fourth level measures the impact of the training program on  the agency’s mission or organizational results. Such an evaluation could  take the form of comparing operational data before and after a training  modification was made. Figure 1 highlights the elements of the training  development process, from the planning stage through the  implementation and evaluation of training, and depicts how the  Kirkpatrick model complements the training development process.
In addition to utilizing this training development process, agencies may  also seek Federal Law Enforcement Training Accreditation (FLETA) for  some or all of their training programs and academies. By attaining FLETA  accreditation for their training academies or programs, agencies provide  assurance that they have voluntarily submitted to a process of self- regulation and have successfully achieved compliance with a set of  standards that demonstrate their adherence to quality, effectiveness, and  integrity. FLETA accreditation also helps maintain training standards by  ensuring that training programs are comprehensively evaluated, using  Kirkpatrick levels 1 through 3 or an equivalent approach, within a 5-year  period.
section_title: Overview of DHS-Specific Guidance for Developing and Evaluating Training and Development Efforts
paragraphs: In October 2010, DHS issued its DHS Learning Evaluation Guide. DHS  created the guide to help the department’s learning and development  community evaluate the effectiveness of training activities in a diverse  organization with varied training needs. Among other things, the guidance  gives an overview of best practices and provides components with tools  they can use to implement the Kirkpatrick model in their training  evaluations. In addition, the guide highlights the need for a training  evaluation plan to identify and address (1) what is being evaluated, (2)  how it is being evaluated, (3) when it is being evaluated, and (4) the  factors involving stakeholder expectations, such as agency policies and  procedures.
section_title: Overview of DHS’s Leader Development Program
paragraphs: In 2004, the Secretary of Homeland Security announced the “One DHS”  policy, which identified the need for a common leadership competency  framework across the department, as well as a unified training curriculum  for current and future leaders. Accordingly, DHS established the LDP  Office in May 2010 under the Office of the Chief Human Capital Officer to  design, develop, and execute a department-wide leadership program that  would strengthen leadership at all levels of the DHS workforce. Through  the LDP, all DHS components are to invest in developing leaders with  skills that transfer across the department, yet retain the agility to balance  this with their own mission-focused leader development needs. In  January 2011, DHS also developed the Leader Development Framework  to serve as a 3- to 5-year strategic roadmap for implementing the LDP.  This framework consists of five tiers of leader development programs for  employees of different levels, such as the executive and supervisory  levels.
In February 2013, DHS issued a directive—Directive 258-02: Leader  Development—formally establishing responsibilities and policies related  to leader development at DHS through the LDP, as well as instructions for  implementing the directive. This directive specifies that the LDP is, among  other things, to delineate requirements and activities to be implemented  by components. The LDP is also to develop and manage centrally  coordinated and high-potential programs for developing employees to fill  future leader positions.
section_title: DHS Processes for Evaluating Training Programs Could Be Better Documented and More Reliably Capture Costs
paragraphs: DHS components have documented processes in place for evaluating  their training programs and have used evaluation feedback to improve  their training offerings; however, their documented processes varied on  the extent to which selected attributes of an effective training evaluation  process were included. Further, DHS identified opportunities for  efficiencies and cost savings, but varying approaches for capturing  training costs across the department affect DHS’s ability to reliably  capture and track its training costs department-wide.
subsections: section_title: DHS Components Use Evaluations to Improve Training, but Documenting Selected Attributes of Their Evaluation Processes Could Improve Transparency and Consistency
paragraphs: The five DHS components in our review have a documented process in  place for evaluating their training programs using the Kirkpatrick four-level  model. However, their documented evaluation processes varied on the  extent to which they included selected attributes of an effective evaluation  process. Components use the results of their evaluations to make  improvements to the training programs and assess training needs. For  example, components used evaluation feedback to improve the delivery  of training content, such as through additional hands-on training, and the  use of e-learning. Table 2 provides such an example for each component.
OPM guidance on training evaluation, DHS’s learning evaluation  guidance, and our prior work on effective training and development  programs identify various attributes for effective training evaluations.  Consistent with these criteria, the attributes of an effective training  evaluation process include communicating (1) the goals the training  programs are supposed to achieve, (2) which training programs will be  evaluated, (3) the methodology for conducting the evaluations, (4)  timeframes for conducting the evaluation, (5) roles and responsibilities for  evaluation efforts, and (6) how the evaluation results will be used.
All DHS components in our review reflected a number of the attributes of  an effective training evaluation process in their documentation. For  example, all components included information on identifying goals that  the training programs are to achieve, identifying which training programs  are to be evaluated, and explaining how the evaluation results will be  used. Table 3 presents information on the extent to which each  component’s documented evaluation process includes these attributes,  and additional details about component ratings are explained in appendix  II.
However, components varied on the extent to which they included  information in their documentation about evaluation methodologies,  timeframes, and roles and responsibilities for evaluation. For example,
Evaluation methodologies: Each component’s documentation indicates  that its training programs are to be evaluated using the Kirkpatrick model.  However, only ICE’s and the Coast Guard’s documentation specify the  methods to be used when performing each Kirkpatrick level of evaluation.
The other three components’ documentation does not specify how each  Kirkpatrick level of evaluation is to be performed in practice.
Timeframes: ICE and FLETC’s documented evaluation processes fully  presents timeframes for evaluations, but documentation for the remaining  three components does not. For example, all of the components’  documentation identified the timeframes for the initial steps of completing  evaluation surveys and collecting data. However, ICE and FLETC’s  documentation also communicated timeframes for the subsequent steps  for analyzing the results of the evaluation.
Roles and responsibilities: We found that three of five components did  not consistently outline the roles and responsibilities for the evaluation  efforts. For instance, two components, CBP and ICE, communicate  information on roles and responsibilities for some parts of the process,  but do not present this information for others. TSA’s documentation did  not communicate information on the specific roles and responsibilities  within the Office of Training and Workforce Engagement (OTWE) for  evaluation activities.
Officials at one component, CBP, told us that since training at CBP is  more decentralized through separate academies that follow their own  processes, their documentation did not include some attributes of  effective training and development programs, as their training standards  were intended to be more of a “how-to guide” rather than a formal step- by-step methodology. According to TSA officials, they did not fully include  certain attributes such as explaining the methodology to be used to  conduct the evaluations and defining roles and responsibilities because  their process is still under development and agreement on this  information has not yet been reached internally. TSA officials stated that  their documentation is to be finalized by the first quarter of fiscal year  2015. Officials at the Coast Guard told us that their documentation did not  include information on the timeframes for analyzing the results, but they  plan to rectify this with their current effort in fiscal year 2014 to revise their  evaluation processes. Two of the five components–ICE, and FLETC— did not provide a reason for why their documentation did not include all of  the attributes of effective training and development programs.
All DHS components agreed that having a documented training  evaluation process provides benefits, such as helping to ensure  consistency and transparency across the organization. Accordingly, as  previously noted, TSA is working to finalize its training evaluation  process. In addition, officials from two components, the Coast Guard and  CBP, stated they plan to revise their documented processes in the near  future. Specifically, according to Coast Guard officials, revisions to the  training evaluation process are being made to more explicitly  communicate their process, enhance standardization, and facilitate  prioritizing its training evaluation efforts to focus on the most mission- critical training needs. As the process is under way, Coast Guard officials  were not able to provide an estimate for when these revisions should be  completed. Similarly, according to CBP officials, they plan to review and  revise CBP’s training evaluation process to ensure consistency across  the component. CBP believes this is necessary given that its training  functions have become more decentralized since its last training  evaluation process came into effect in 2008. According to CBP officials,  the target completion of revisions is fiscal year 2015, after the  reorganization of CBP’s Office of Training and Development is complete.
By ensuring that the components’ documented evaluation processes fully  address attributes for effective training as they are drafted, updated or  revised, DHS would have better assurance that the components have  complete information to guide its efforts in conducting effective  evaluations. Moreover, such documentation could help ensure that  evaluation processes for assessing whether training programs  appropriately support component and DHS needs can be repeated and  implemented consistently. As components draft, update, and revise their  documented evaluation processes, incorporating or more fully addressing  the aforementioned attributes of effective training evaluations could help  to ensure that components clearly communicate all aspects of their  evaluation processes and that employees can consistently implement  them.
section_title: DHS and Components Have Identified Opportunities for Efficiencies and Cost Savings, but Varying Approaches for Capturing Training Costs at DHS Affect Reliability
paragraphs: All DHS components reported reviewing the merits of different delivery  mechanisms (e.g., classroom or computer-based training) to determine  which mix would be the most efficient for at least one of their training  programs. In addition, four of five components—CBP, the Coast Guard,  ICE, and TSA—provided at least one illustrative example of how they  used a mix of webinars, online learning, and classroom instructor-led  training to develop a blended learning approach that improved cost- effectiveness. For example, in June 2013, TSA implemented webinar  training for Sensitive Security Information (SSI), which helped TSA avoid  travel costs and led to an estimated $855,026 in cost savings. In addition,  ICE developed online training for Fourth Amendment instruction, which  helped reduce the course length by 1 week, contributing to opportunity  cost savings and savings in room and board totaling about $4.8 million  over a 5-year period. The components we reviewed that used webinars  or online learning stated that these delivery mechanisms did not  adversely affect the quality of training offered in these instances.  Furthermore, all DHS components reported that they have evaluated at  least one training program to determine how to streamline or consolidate  the training to make it more cost-effective. For example, FLETC adopted  firearms simulation technology and more cost-effective ammunition, and  according to our analysis of FLETC data, led to about $2.2 million in cost  savings. See figure 2 for a photographic example of training using  firearms simulation.
Similarly, at the department level, OCHCO has taken steps to streamline  mandatory department-wide training requirements for counterintelligence  and records management training. For example, based on review of legal  requirements, OCHCO found that it could streamline mandatory records  management training by consolidating multiple annual training  requirements into a single course. According to OCHCO officials and our  analysis of OCHCO data, this effort could lead to a potential cost saving  of about $57.1 million over a 5-year period.
Table 4 provides illustrative examples of actions OCHCO and selected  components have taken to improve the cost-effectiveness of training and  the estimated cost savings from these actions over a 5-year period,  according to our analysis of OCHCO and DHS components.
Starting in fiscal year 2014, CBP adopted a new approach to improve the  identification of efficiencies in training, including regular reviews of  training justification, cost, and prioritization of training needs. Though the  process is not yet documented in policy, directives, or standard operating  procedures, CBP officials report that it has allowed them to identify  training cost discrepancies more consistently and efficiently. For example,  according to CBP officials, their approach includes tracking key cost  elements, such as travel, lodging, meal, rental vehicle requirements;  duration of training; instructor costs; and contract costs, for all training  courses under separate codes. Further, according to these officials, this  allows CBP to better compare costs for course execution, such as the  cost of course resources (e.g., delivery location, equipment, etc.) and  related travel, if any, from year to year. In addition, CBP uses these data  to challenge requests for training and identify possible alternatives for  delivering existing courses at a reduced cost. The process also provides  CBP with a vehicle for better projecting training costs. According to CBP  officials, the new approach created requirements that internal offices  provide more precise estimates of the number of participants attending  training, which reportedly helped CBP more efficiently allocate about $5.8  million in fiscal year 2013. Before the new process was implemented,  internal offices could not commit to filling training courses and slots,  resulting in CBP’s Office of Training and Development overprojecting  about $7 million in training costs in fiscal year 2012, which was returned  to CBP. According to CBP officials, leadership change, overestimation of  training funding needs, and spending reductions under sequestration in  fiscal year 2013 were some of the key reasons for adopting this new  approach. CBP officials reported that by refining cost projections, CBP  improved their ability to approve more training within allocated budgetary  resources. For example, based on the new approach in fiscal year 2014,  CBP identified surplus training funds from unfilled training slots and class  cancellations early enough to enhance training programs.
In addition, although DHS and components provided illustrative examples  of efficiencies in training and cost savings, DHS uses different methods to  capture training costs. DHS, through OCHCO, has worked to capture the  cost and delivery of DHS’s training and development programs. However,  at DHS headquarters and at the component level, there are  inconsistencies in how training costs are captured across the department  that have made it a challenge to accurately and reliably capture these  costs across DHS. For example, OCHCO officials explained that the lack  of a centralized funding source and disparate financial management  systems used by components created challenges in reliably capturing  training costs. Components also often capture training costs differently  from one another, which can contribute to inconsistencies among training  costs captured at DHS. Training costs may, for example, include  expenses for instructional development, participant and instructor salary  and benefits, equipment costs, and travel and per diem expenses.  Accordingly, OCHCO officials report that some components include  conferences as a training cost while others do not, and some components  did not include mission-critical law enforcement training costs when they  provided department-wide training costs.
In fiscal year 2012, the DHS Undersecretary for Management requested  that OCHCO collect training cost data from components. During this  process, OCHCO relied on senior-level data requests to retrieve annual  training expenditure information from components. According to OCHCO  officials, the senior-level data call process revealed that training cost data  had limited reliability because some components were not consistent in  determining the types of mission-critical training costs they provided,  among other things. Given ongoing concerns about data reliability,  OCHCO officials noted that it would be difficult to update and reliably  aggregate department-wide training costs for fiscal year 2013.  According to OCHCO officials, given budget constraints, it is difficult for  OCHCO to make good investment decisions about training when they do  not know how components spend their training dollars.
In addition, according to discussions with ICE officials, we found that the  cost of ICE’s training and development programs may not be consistently  and accurately captured. For example, ICE officials stated that participant  and instructor salaries are consistently tracked as part of training costs,  but travel expenses are less consistently tracked for all ICE internal  training programs. Further, according to ICE officials, incomplete  definitions of training and inconsistency in how costs are tracked also  contribute to shortfalls in reliably capturing training costs. ICE officials  reported, for example, that they cannot reliably capture training costs from  one of ICE’s internal departments and, instead, need to rely on sporadic  data calls to retrieve training budget and expenditure information from its  departments. ICE officials reported concerns about the reliability of this  process, partly because of concerns about inconsistent coding schemes  for tracking similar training activities and the lack of third-party checks on  the reliability of how training information is coded. As of August 2014, ICE  officials report that their Office of the Chief Financial Officer is working to  standardize its coding schemes—or object class reporting—across ICE  programs and plans to implement the revised coding standards in fiscal  year 2015.
OCHCO and ICE officials we met with acknowledged that the department  has not identified all challenges that prevent DHS from accurately  capturing training costs department-wide, but they have taken some  preliminary steps toward more consistently defining training and capturing  costs. For example, while DHS has not issued central guidance on what  should be included in training costs, OCHCO officials noted that they  provided a glossary of terms to components in December 2007 to help  establish an initial definition of training department-wide. Although the  glossary clarifies a number of training-related terms, it does not provide  requirements for tracking training costs consistently across components.  For example, the glossary notes that training program costs are  calculated differently on a component basis. Further, according to ICE  and OCHCO officials, DHS discussed the issue of accurately and reliably  capturing training and development costs across the department as part  of its Training Leaders Council in May 2014. OCHCO officials reported  that the use of a standard form for requesting training, known as the  federal government’s Standard Form 182, Authorization, Agreement, and  Certification of Training, may be one method for improving the tracking of  training costs. For example, the Form 182 may help provide for consistent  definitions and methods for capturing certain training costs. However,  while use of the standard Form 182 would be a positive step, it may not  address certain reliability concerns associated with capturing training  costs at DHS. For example, the approach may not prevent the duplicative  capturing of procurement-related training costs or shortfalls in how  training information is entered and captured in each component’s  systems. According to the DHS Chief Learning Officer, requiring the use  of the Form 182 DHS-wide is still in the preliminary stages of  consideration and would require accompanying policy changes. As DHS  has not yet made a decision on whether to require use of the Form 182, it  does not yet have timeframes for implementing this proposal.
One leading training investment practice is that agencies should capture  the cost and delivery of their training and development programs. Our  prior work has also shown that agencies need reliable information on how  much they spend on training and for what purposes. To capture the cost  and delivery of training and development programs, agencies need  credible and reliable data from learning management systems as well as  accounting, financial, and performance reporting systems. To the extent  possible, agencies also need to ensure data consistency across the  organization (such as having data elements that are pulled from various  systems representing the same type of information). Variations in the  methods used to collect data can greatly affect the analysis of uniform,  high-quality data on the cost and delivery of training and development  programs. Given today’s budgetary constraints and the need to effectively  utilize and account for all federal dollars, identifying existing challenges  that prevent DHS from accurately capturing training costs department- wide and, to the extent that the benefits exceed the costs, implementing  corrective measures to overcome these challenges, could enhance  DHS’s resource stewardship.
section_title: DHS Is Implementing a Department-wide Leader Development Framework, but Could Strengthen Its Program Assessment
paragraphs: DHS’s Leader Development Program Office is in the process of  implementing a department-wide, five-tier Leader Development  Framework to build leadership skills across all staff levels. While DHS  components generally stated that the LDP framework is beneficial, they  raised concerns about its training requirements, which the LDP Office’s  planned evaluation efforts may address. Further, the LDP office has  developed a program-wide assessment approach to analyze the impact  of the LDP that includes tracking 12 performance measures over time.  However, the LDP Office could strengthen its performance measurement  efforts by clearly identifying its program goals and better incorporating key  attributes of successful performance measures we have previously  identified.
subsections: section_title: DHS Has Implemented Portions of the Department-wide Leader Development Framework, and Components Deliver Additional Programs to Develop Their Leaders
paragraphs: DHS has implemented programs in support of two of five tiers within its  department-wide Leader Development Framework, and the selected  components in our review also deliver additional leader development  programs for supervisors, managers, and executives. As previously  discussed, DHS established the LDP Office in May 2010 to design,  develop, and execute a department-wide program to strengthen  leadership at all levels of the DHS workforce. In January 2011, DHS  approved the Leader Development Framework as a 3- to 5-year strategic  roadmap for implementing the LDP. This Leader Development  Framework consists of five tiers that identify envisioned leader  development programs for employees of different levels. These tiers, and  the employees they include, are the following: executive (members of the Senior Executive Service, Coast Guard  admirals, and selected other leaders), manager (nonexecutive employees who supervise other supervisors,  lead through subordinate supervisors, and formally supervise at least one  supervisory employee),  supervisor (employees who accomplish work through, and are directly  responsible for, the work of nonsupervisory employees, and who formally  supervise only nonsupervisory employees),  team lead (nonsupervisory employees formally designated as such or  tasked to guide a group of people to results on a program, project,  initiative, or task force), and  team member (nonsupervisory DHS employees).
The LDP Office has implemented programs within two of the five Leader  Development Framework tiers (supervisor and executive), initiated  program development within two tiers (team lead and team member), and  plans to begin program development within one tier during fiscal year  2014 (manager). According to the LDP Manager, the office prioritized  implementation of the supervisor tier at the direction of the then deputy  secretary, who identified supervisors as a critical nexus between strategic  leadership and employee performance. The LDP Manager stated that the  office also prioritized implementation of the executive tier because  OCHCO officials were familiar with best practices for instruction for new  executives and the then deputy secretary identified particular value in  providing new executives with consistent instruction. Within the  supervisor tier, the LDP Office has established the Cornerstone Program,  which consists of a set of baseline requirements for new and seasoned  supervisors at all levels. DHS components may fulfill Cornerstone  Program requirements through new or existing training programs, cross- component programming, or a combination thereof. Within the executive  tier, the LDP Office centrally administers the 3-week Capstone Cohort  Program, which includes discussion forums, operational site visits, and  learning activities intended to address real-world strategic issues. As the  program is currently implemented, whereas components may elect to  send participants to the Capstone Program, they are required by DHS to  implement Cornerstone Program requirements.
In addition to implementing programs to support the supervisor and  executive leader development framework tiers, the LDP Office has  assumed responsibility for administering two preexisting DHS programs,  the Senior Executive Service Candidate Development Program and DHS  Fellows. These programs are not a part of any one tier, as their intended  participants may span framework tiers. For example, the Senior  Executive Service Candidate Development Program is designed for  Senior Executive Service candidates who aspire to transition into the  executive tier. Figure 3, an interactive graphic, describes the development  and implementation status of the programs that support each tier as of  August 2014. See appendix III for a print version of this figure. LDP Office  officials anticipate fully implementing all five tiers of the Leader  Development Framework before the end of fiscal year 2016.
Move mouse over the program names for more information. For a text version please see app. III.
Component program(s)
The five DHS components selected for our review have all participated in  the LDP department-wide programs. In particular, according to LDP  Office and component data and officials, all five components have  programs in place, as required by DHS, intended to meet the Cornerstone  Program requirements. For instance, data from the selected components  demonstrate that the Fundamentals of DHS Leadership courses they  delivered—one of four program segments of the Cornerstone Program— from fiscal year 2012 through fiscal year 2013 had more than 3,600  participants. According to the LDP Manager, the LDP Office has sought  to avoid duplication of effort and costs in components’ implementation of  the Cornerstone Program. For example, the LDP Office coordinated the  development of instructional materials for all components to use to meet  requirements for the Understanding the DHS Leadership Commitment  segment of the Cornerstone Program, which is for individuals considering  the supervisory path. In addition, information that a DHS working group  collected from components during initial development of the Cornerstone  Program indicated that most components could utilize existing programs  to help meet program requirements. Specifically, six of the seven  components that provided information to the working group indicated that  they had an existing training program that they could use to provide  instruction to first-time supervisors. For instance, FLETC uses two  preexisting programs to meet Fundamentals of DHS Leadership  requirements—the FLETC New Supervisor Training Program and the  Law Enforcement Supervisor Leadership Training Program.
In addition to maintaining programs to meet Cornerstone Program  requirements, each of the selected components has elected to participate  in the three department-wide programs administered centrally by the LDP  Office—the Capstone Cohort Program, Senior Executive Service  Candidate Development Program, and DHS Fellows. In particular, these  programs had a total of approximately 60 participants from the selected  components from fiscal year 2012 through fiscal year 2013, according to  LDP Office data. Table 5 summarizes approximate participation in Leader  Development Framework programs that were provided by the selected  components to meet department-wide requirements or centrally  administered by the LDP from fiscal year 2012 through fiscal year 2013,  according to LDP Office and component data.
In addition to the programs administered under the Leader Development  Framework, the components in our review also deliver various additional  leader development programs. For example, Coast Guard delivers a 1- week course for newly selected executives focused on change  management issues and TSA delivers a program for supervisors over a  period of 18 to 24 months that includes training, shadowing, mentoring,  and other developmental leadership opportunities. Table 6 provides  examples of leader development programs delivered by these  components at the executive, supervisor, and manager levels.
As shown in table 7, according to data provided by the selected  components, leader development programs they delivered for  supervisors, managers, and executives, independent of department-wide  programming, from fiscal year 2012 through fiscal year 2013 had a total  of more than 10,000 participants.
Officials from the components in our review generally stated that it is  beneficial for their components to provide leader development programs  in addition to the LDP department-wide training because, whereas LDP  training focuses on more general leadership skills and competencies,  component-level training is tailored to the components’ unique missions.  For example, TSA officials explained that TSA leader development  programs focus on developing individuals for TSA’s mission-critical  occupational areas (e.g., federal security directors and federal air  marshals), which require proficiency in a set of leadership and technical  competencies unique to TSA. In addition, TSA leader development  programs utilize examples that are readily applicable to day-to-day TSA  operations, according to these officials. Similarly, according to Coast  Guard officials, their leader development programs afford the Coast  Guard the opportunity to provide instruction using case studies and in- class discussions on how to lead a Coast Guard workforce in a Coast  Guard context.
section_title: Component Officials Generally Stated the LDP Framework Is Beneficial, but Raised Concerns about LDP Requirements, Which LDP Assessment Efforts May Address
paragraphs: Component officials we spoke with generally agreed that the LDP is  helpful in providing a common framework for leader development training.  However, officials from three of the five components we met with raised  concerns about the applicability or clarity of certain learning objectives the  LDP requires they teach when implementing the Cornerstone Program.
Officials from one component stated that the LDP has established policy  and procedures, which help to ensure all components are informed and  have consistent definitions and policies related to leader development.  Components also identified other benefits of the LDP, such as bringing  focus across the department to leader development, allowing for  collaboration on leader development activities, and having experienced  staff who work solely on leader development issues and programs.
As previously discussed, the Cornerstone Program consists of  requirements in four segments, one of which is a course on the  fundamentals of DHS leadership that is required for all first-time federal  supervisors. In order to fulfill LDP requirements for this course, all  components must provide instruction on more than 200 learning  objectives that identify content the course must cover. For example,  these learning objectives include encouraging respect for individual  differences and determining appropriate tasks to delegate. According to  DHS guidance on the LDP, all DHS components are to develop leaders  with skills that transfer across DHS, yet retain the agility to advance their  own unique mission-focused leader development needs. However,  officials from two components raised concerns about the applicability of  certain objectives that the LDP requires them to teach. For example,  officials from one component stated that the learning objective involving  supervising a workforce of federal employees and contractors is not  universally applicable because supervisors in their component do not  supervise contractors. Officials from another component stated that one  objective related to supervisors’ knowledge of the hiring process does not  pertain to new supervisors within their component. According to these  component officials, requiring them to teach objectives that are not  pertinent to tasks supervisors must perform takes away from instructional  time that they could use to teach more relevant content. According to the  LDP Manager, the LDP Office established learning objectives in order to  meet DHS’s direction to ensure sufficient consistency in leader  development investment across components, but components may adapt  the objectives, as appropriate. For example, TSA requested a waiver from  teaching learning objectives focused on Title 5 of the U.S. Code, from  which TSA is generally exempt. The LDP Manager granted the waiver,  and suggested that TSA replace instruction focused on Title 5 with  instruction on related subjects applicable to TSA. However, officials from  the two components that raised concerns were not aware that  Cornerstone Program requirements provided them with this flexibility.
Officials from two components raised concerns that some of the learning  objectives required to be taught under the Fundamentals of DHS  Leadership course do not clearly articulate what the training must cover,  and that they are not written with standards that can be measured or  observed. For example, according to officials from one component, some  Fundamentals of DHS Leadership objectives are not consistent with their  component’s standards, which require that performance objectives  include condition, measurable performance behavior, and a standard that  specifies the degree of quality expected in performance. For instance,  one of the learning objectives that officials identified as not meeting these  requirements states, “Recognize a recent study that reported 48 percent  of workers surveyed responded to job pressure by performing illegal or  unethical activities; 58 percent considered acting illegally or unethically.”  A senior official from this component explained that this can result in  implementation and evaluation challenges—if it is unclear what the  outcome of an objective is supposed to be, it is difficult to know how to  implement it or evaluate its implementation. This official also stated that  officials from his component voiced their concerns about the clarity of  learning objectives to DHS headquarters, but DHS did not change them.  According to LDP Office officials, they solicited input from components on  Cornerstone Program requirements and adopted selected changes. In  particular, according to the LDP Manager, the LDP Office solicited input  from components during four informal and two formal reviews of  Cornerstone Program requirements.
The LDP Office has also awarded a contract for an assessment beginning  in February 2014 that includes evaluation of the Fundamentals of DHS  Leadership’s learning objectives. Scheduled for completion by September  2014, this assessment may help to address concerns raised by  components. This assessment is to determine the Cornerstone Program’s  overall implementation status, determine the effectiveness of the  program’s products and elements, evaluate the efficacy of the  Fundamentals of DHS Leadership’s learning objectives, and recommend  specific tactical and strategic changes for improving program  effectiveness.
section_title: Clearly Identifying Program Goals and Enhancing Performance Measures Could Strengthen LDP Assessment Efforts
paragraphs: The LDP has developed a program-wide assessment approach intended  to analyze the impact of the LDP over time and assess whether the  program is targeting the right things in the right way. However, the LDP  Office could strengthen this assessment approach by more clearly  identifying its program goals and ensuring its 12 performance measures  incorporate key attributes of successful performance measures we have  previously identified. The LDP’s assessment approach applies to all  LDP program elements, including Capstone, Cornerstone, and other  programs. The approach consists of (1) biannually collecting and  analyzing completion rate data for all LDP programs implemented by  components, (2) collecting and analyzing responses to six core evaluation  questions immediately following each developmental activity and 6  months later, and (3) tracking 12 program performance measures. Table  8 provides some examples of these 12 measures.
For more detailed information about the LDP’s assessment approach, see  appendix IV.
Developing this assessment approach is a positive step toward assessing  the effectiveness of the LDP. However, the LDP Office has not clearly  identified goals for the program, and the 12 measures that the office has  developed to assess its performance do not consistently exhibit attributes  we have previously identified as key for successful measurement. These  key attributes include having linkage with division- and agency-wide  goals, being clear, and having measurable targets. Table 9 presents  definitions of these attributes along with potentially adverse  consequences of not meeting them.
Performance measurement is the ongoing monitoring and reporting of  program accomplishments, particularly progress toward preestablished  goals. We have previously reported that performance measurement  allows organizations to track progress in achieving their goals and gives  managers crucial information to identify gaps in program performance  and plan any needed improvements. In addition, according to Standards  for Internal Control in the Federal Government, managers need to  compare actual performance against planned or expected results and  analyze significant differences. We observed the following when  assessing the LDP’s performance measures against these selected key  attributes:
Linkage: The LDP’s 12 performance measures do not clearly link with  program goals and linkage is not clearly communicated throughout the  organization. The LDP has identified working program goals, but they are  disparately documented and not clearly identified as goals. When we  asked LDP Office officials to identify the LDP’s goals, the LDP Manager  referred us to statements in Directive 258-02: Leader Development, and  provided us with a list of working program goals assembled from  statements in various LDP materials. These working program goals  included, for example, using best practices to maximize effectiveness  and elevating the importance of developing leaders department-wide.  However, neither the directive nor LDP materials clearly identify or refer  to the statements the manager directed us to as goals. Given that the  LDP’s identified working program goals are disparately documented and  not clearly identified as goals, it is unclear whether the LDP’s 12  performance measures align with the statements and working program  goals the LDP Manager identified. We tried to identify linkage between  the LDP’s 12 performance measures and the informal goals the LDP  Manager identified, but could not determine definitively how they relate.
According to the LDP Manager, the LDP Office did not clearly document  the program’s goals in one place or record their linkage to the program’s  performance measures because it established the goals as it developed  programs to support the Leader Development Framework. In addition, the  LDP Manager stated that DHS’s strategic human capital–related plans— which include goals—were under development when the office developed  the measures. The LDP Manager explained that, in developing the  performance measures, an LDP program official used DHS department- wide strategic plans, direction from DHS officials and stakeholders, and  guidance from the DHS Learning Evaluation Guide—which provides  guidance for evaluating the effectiveness of training activities—to  determine categories for the performance measures and then developed  measures pertaining to each category. In addition, DHS uses data that  the LDP Office collects for its performance measures for strategic  planning and reporting, according to the LDP Manager. For example,  DHS uses data the LDP Office collects to measure progress against two  goals established in the DHS Workforce Strategy for Fiscal Years 2011- 2016. We agree that data collected to track the measures may provide  information for measuring progress against some department-wide goals  established in strategy documents; however, it is not evident how these or  other LDP performance measures link to goals specific to the LDP.
We have previously found that linkages between goals and measures are  most effective when they are clearly communicated to all staff within the  agency so that everyone understands what the organization is trying to  achieve. Explicitly identifying program goals, creating an evident link  between performance measures and program goals, and clearly  communicating the linkage could help ensure that the behaviors and  incentives created by the LDP’s 12 performance measures support the  LDP’s intended outcomes, and that they are appropriate measures for the  program.
Clarity: Not all LDP performance measures possess clarity because  some measures include terms that are ambiguous and for which the LDP  Office has not documented definitions. For example, one of the LDP’s  measures is the percentage of developmental activities that fulfill LDP  requirements delivered with shared resources. However, it is not clear  what constitutes a “developmental activity” for the purpose of calculating  this measure (e.g., a course or unit within a course), and the value could  be different depending on the definition of “activity” used in the measure’s  calculation. As a result, this measure could be confusing and  misleading to users, such as DHS leadership and congressional  constituents, by leading them to think that performance was better or  worse than it actually was. According to the LDP Manager, the LDP  Office has not documented definitions for terms used in its performance  measures because components may elect to fulfill LDP requirements  through varied approaches and this makes how terms such as  “developmental activity” are defined contextually driven. While we  recognize that components may use varied approaches to fulfilling LDP  requirements, it is important that terms the LDP Office uses in its  performances measures are clear so that users understand what the  measures mean. LDP officials also stated that components are able to  contact the LDP Office with questions about how to calculate the  measures, and that the LDP Office will work with components to provide  any requested clarification. Providing support to components is a positive  step, but documenting definitions for ambiguous terms used in the  measures could help ensure the meaning of their values is clear to  stakeholders.
Targets: The LDP’s performance measures do not have measurable  targets. According to the LDP Manager, the LDP has not set targets for  its 12 program performance measures because it is too early in the  process, as LDP officials have just established a baseline with fiscal year  2013 data. The LDP Manager anticipates developing LDP targets in the  future and stated that program officials will consider doing so once they  collect more data. The LDP Office does not have a definitive plan or time  frame for setting targets, but according to the LDP Manager, expects to  do so using fiscal year 2014 data. We agree that, consistent with key  attributes of performance measures, developing measurable targets  could help DHS determine whether the program’s performance is  meeting expectations. To set appropriate targets, however, it will be  important for the LDP Office to first clearly identify program goals and  ensure its performance measures link to the goals.
DHS leadership has previously identified implementation of leader  development programs as important to the department’s success and a  means by which to improve its human capital management. For example,  in February 2012, the then deputy secretary of homeland security stated  that leader development is critical to DHS’s growth and long-term success  and must be a strategic mission investment priority. In addition, DHS has  identified implementation of the LDP among the actions it is taking to  address DHS’s high-risk designation with respect to human capital  management. By clearly identifying program goals and ensuring LDP  performance measures include key attributes, such as linkage, clarity,  and measurable targets, the LDP could strengthen its performance  measurement, consequently producing actionable information for LDP  management to use in identifying the need for, and making, program  improvements.
section_title: Conclusions
paragraphs: As DHS faces increasingly complex national security challenges, it is  important that it support employees with effective training and  development programs to meet its mission requirements. Evaluating  training and development programs is important for ensuring that such  programs are cost-effective and continue to be relevant for the  department. By updating DHS components’ documented training  evaluation processes to more fully address key attributes for effective  training evaluation, DHS components could have better assurance that  the components have more complete information to guide their efforts in  conducting effective evaluations. Such documentation can further help  ensure that processes for assessing whether training programs support  component and DHS needs are repeatable and consistently  implemented. Further, given limited budgetary resources, by identifying  existing challenges that prevent DHS from accurately capturing its  training costs department-wide and, to the extent that the benefits exceed  the costs, implementing corrective measures to overcome these  challenges, DHS could improve its awareness about the actual costs of  its training programs, and enhance its ability to consistently and reliably  capture training costs DHS-wide, thereby enhancing its resource  stewardship.
In addition, DHS is in the process of implementing a department-wide  leader development program to build leadership skills across all staff  levels. The effectiveness of this program is particularly important given  that DHS leadership has identified leader development as critical to the  department’s success. As DHS begins to assess the impact of the LDP  program, clearly identifying LDP goals and ensuring that LDP  performance measures possess key attributes, including (1) linkage to the  program’s goals, (2) clarity, and (3) measurable targets by which to  assess the measures could help provide DHS with the actionable  information it needs to identify and make program improvements.
section_title: Recommendations for Executive Action
paragraphs: To ensure effective evaluation of federal training programs and enhance  DHS’s stewardship of resources for federal training programs, we  recommend that the Secretary of Homeland Security take the following  two actions: direct DHS components to ensure that their documented training  evaluation processes fully address attributes for effective training  evaluation processes as they are drafted, updated, or revised and  identify existing challenges that prevent DHS from accurately capturing  training costs department-wide and, to the extent that the benefits of  addressing those challenges exceed the costs, implement corrective  measures to overcome these challenges.
In addition, to produce actionable information for DHS’s LDP  management to use in identifying the need for, and making, program  improvements, we recommend that the Secretary of Homeland Security  direct the Chief Human Capital Officer to clearly identify LDP goals and  ensure LDP performance measures include key attributes, such as  linkage, clarity, and measurable targets.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DHS for review and comment. DHS  provided written comments, which are reprinted in appendix IV, and  technical comments, which we incorporated as appropriate. DHS agreed  with all three of the recommendations and outlined steps to address  them. If fully implemented, these actions will address the intent of our  recommendations.
With respect to the first recommendation, DHS agreed to ensure that  effective training evaluation processes are documented and in place  at components by incorporating a review of component training  evaluation documents into the DHS Chief Human Capital Officer’s  audit of human resource functions. DHS reports that a full review of  components should be completed with the fiscal year 2019 audit  cycle.
Regarding the second recommendation, DHS agreed to resolve the  issue of capturing training costs Department-wide. For example, DHS  plans to establish a team jointly chaired by the DHS Chief Human  Capital Officer and the Chief Financial Officer, and comprised of  representatives from both financial and training offices of each  operational component and headquarters, that is to deliver a  methodology to track and report training costs across DHS by June  30, 2015. DHS anticipates the new methodology will be implemented  across all components by January 31, 2016.
In response to the third recommendation, DHS agreed to publish clear  DHS Leader Development Program goals and performance measures  that include key attributes, such as linkage, clarity and measurable  targets on the DHS intranet website by December 31, 2014.
We are sending copies of this report to appropriate congressional  committees and the Secretary of Homeland Security. In addition, the  report is available at no charge on the GAO website at  http://www.gao.gov.
Should you or your staff have any questions concerning this report,  please contact me at (202) 512-9627 or by e-mail at maurerd@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. Key contributors to  this report are listed in appendix VI.
David Maurer  Director, Homeland Security and Justice  .
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives for this report were to address the following questions:  1.  To what extent does the Department of Homeland Security (DHS)  have documented processes to evaluate training and development  programs and reliably capture costs?  2.  What leader development programs has DHS implemented, what are  stakeholders’ perspectives on them, and to what extent does DHS  measure program performance?
To understand training programs at DHS, we obtained information from  the DHS Office of the Chief Human Capital Officer (OCHCO), and five  selected components: the Federal Law Enforcement Training Center  (FLETC), U.S. Customs and Border Protection (CBP), U.S. Immigration  and Customs Enforcement (ICE), the Transportation Security  Administration (TSA), and the United States Coast Guard. We selected  these components to represent different DHS mission areas, workforce  sizes, training costs, and number of career Senior Executive Service  (SES) personnel. In addition, these components have a mix of new and  more established training programs. When examining training programs  at selected components, we reviewed component-level training evaluation  and strategic plans when available; training budget requests; cost and  expenditure documents; training procedures, policies, and organizational  charts; and policies for identifying and prioritizing training programs;  selected training course materials, and other relevant documents. To  further our understanding of training at the component level, we also  interviewed training officials at each of the selected components and  identified these individuals based on their knowledge, experience, and  leadership roles. We conducted our interviews at component  headquarters located in the Washington, D.C. area, or field offices. In  addition, as part of our review of DHS’s delivery of mission-critical law  enforcement training across components, we observed training at  FLETC’s Glynco, Georgia facilities. The perspectives DHS OCHCO and  the selected components provided are not generalizable to all of DHS, but  provided helpful insights into the selected components specific training  and development programs at DHS.
To address the first question, regarding the extent to which DHS has  documented processes to evaluate training and development programs,  and ensure training costs are reliably captured, we reviewed DHS and  component-specific documents and interviewed relevant officials at DHS  OCHCO and each of the components. Specifically,
To determine the extent to which DHS has documented processes to  evaluate its training and development programs, we reviewed policies  and procedures related to the evaluation of training programs, such as  component-specific standard operating procedures and training  development standards. We then assessed the documented processes  from each of the selected components against attributes of training  evaluation processes identified by the Office of Personnel Management  (OPM), DHS, and GAO to determine the extent to which the documents  include selected attributes of evaluation processes. We selected the  attributes for our analysis to include attributes that were consistently  identified in relevant criteria documents related to training evaluation,  such as the DHS Learning Evaluation Guide, the OPM Training  Evaluation Field Guide, and GAO’s prior work on training and  development, specifically the Guide for Strategic Training and  Development Efforts in the Federal Government. These attributes also  align with those identified in Standards for Internal Control in the Federal  Government, which calls for agencies to document the plans, methods,  and procedures used to achieve missions, goals, and objectives and  support performance-based management practices. From these  sources, we identified six attributes of a training evaluation process to  conduct our analysis: (1) establishes goals about what the training  program is supposed to achieve, (2) indicates which training programs  are being evaluated, (3) explains the methodology used to conduct the  evaluation, (4) presents time frames for conducting the evaluation, (5)  presents roles and responsibilities for evaluation efforts, and (6) explains  how the evaluation results will be used. We assessed each component’s  documented evaluation process to determine the extent to which the  attributes were included and gave a component a rating indicating that  the attribute was fully met, a component partially met the attribute but did  not fully or consistently meet all parts, or the component did not include  any information to meet the attribute. We also conducted semistructured  interviews with officials responsible for conducting training evaluation at  each of the five components to understand the evaluation process that  each component follows and how evaluation feedback is used.
To assess the extent to which DHS ensures training costs are reliably  captured, we reviewed information and relevant documentation on  processes and steps components took to examine available budget and  cost information. We further reviewed documentation on the process of  capturing training costs from each of our selected components, including  financial audit reports. As part of our review of cost tracking at DHS, we  observed methods components used for identifying efficiencies in training  to identify cost savings and employ more cost-effective alternatives. We  also conducted semistructured interviews with DHS and component  officials responsible for administering training programs and tracking  costs to understand how DHS and components identified and captured  costs, and any challenges they may have in doing so in a reliable  manner. Through our review of cost-saving documentation and interviews  with DHS and component officials, we sought illustrative examples to  understand how OCHCO and the selected DHS components identified  potential efficiencies and steps planned or already taken to achieve them.  Accordingly, OCHCO and DHS component officials identified examples  of cost savings realized in selected training programs from fiscal year  2011 through fiscal year 2013, and we reviewed the reliability of the  related cost-saving estimates. For example, we interviewed  knowledgeable officials who provided cost estimates, reviewed the  estimates related to cost savings, and replicated cost-saving calculations  provided by components, including estimates for training equipment,  salaries, and benefits. We determined through analysis of cost-saving  estimates and interviews with knowledgeable officials at DHS and the  selected components that the cost-saving data provided and reported for  the illustrative examples in this product were sufficiently reliable for the  purposes of illustrating the types of cost efficiencies that may be  achieved. The cost-saving examples DHS OCHCO and components  provided are not generalizable to all of DHS, but provided helpful insights  into cost-saving efforts identified to date at DHS.
To address the second question, about leader development programs  DHS has implemented, we reviewed program documentation, analyzed  participant data, and interviewed officials from OCHCO and the selected  components.
In particular, to determine what leader development programs DHS has  implemented, we reviewed OCHCO Leader Development Program (LDP)  curricula and requirements documentation, such as the Senior Executive  Service Candidate Development Program Candidate Guide and The  Cornerstone Program Requirements and Accountability Guide, and  documentation of leader development programs provided by the selected  components, such as program descriptions and evaluations. In addition,  we obtained and analyzed data from OCHCO and the selected  components on the number of participants in the leader development  programs they provided during fiscal years 2012 and 2013. We  assessed the reliability of these data by interviewing agency officials  familiar with the sources of the data regarding internal controls built into  the information systems and stand-alone documents in which they are  stored and quality assurance steps performed after data are entered into  the systems or documents. In addition, we compared participant  completion data for the Fundamentals of DHS Leadership segment of the  Cornerstone Program—one of DHS’s leader development programs—for  similar time periods that components provided to us and had previously  reported to the LDP Office. Where we identified discrepancies, we  interviewed officials to determine their cause and the correct values. We  determined that the data were sufficiently reliable for the purpose of  reporting the approximate number of program participants. We also  interviewed officials from OCHCO and the components regarding  implemented and planned leader development programs.
To determine officials’ perspectives on DHS leader development  programs, we obtained OCHCO and component officials’ views on the  development and implementation of leader development programs and  the programs’ strengths and weaknesses. The perspectives the  interviewees provided are not generalizable to all DHS officials, but  provided helpful insights into strengths and weaknesses of leader  development programs.
To assess the extent to which DHS measures the performance of leader  development programs, we reviewed program documentation from  OCHCO and the selected components, including performance  measurement requirements and guidance. In addition, we interviewed  cognizant officials about what performance measurement information  they collect and how they use the information. Through these efforts, we  determined that the LDP Office uses 12 performance measures to  assess the LDP’s impact. We assessed these measures against three of  nine selected key attributes for performance measures identified in prior  GAO work that we identified as relevant given the maturity level of the  LDP. In particular, given that the LDP is a relatively new program, we  focused our analysis on three attributes that we identified as foundational  and—having linkage with division- and agency-wide goals, being clear,  and having measurable targets. We selected linkage because aligning  measures with division- and agency-wide goals and mission helps ensure  that the behaviors and incentives created by the measures support the  division- or agency-wide goals or mission. Once the measures’ relevance  to a program is ensured through linkage, then assessment of more  detailed aspects of the measures, such as reliability, is more relevant.  Similarly, we selected having measurable targets because, without  measurable targets, it may not be evident whether performance is  meeting expectations. With regard to clarity, if a measure is not clearly  stated and the name and definition are not consistent with the  methodology used to calculate it, performance data could be confusing  and misleading to users, such as department leadership and  congressional constituents.
We conducted this performance audit from July 2013 to September 2014,  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Presence of Effective Training Attributes in DHS’s Documented Training Evaluation Processes
subsections: section_title: Establishes goals about what the training program is supposed to achieve 
paragraphs: Component  U.S. Customs and  Border Protection  (CBP)  U.S. Immigration and  Customs  Enforcement (ICE)  Coast Guard  Transportation  Security  Administration  (TSA)Federal Law  Enforcement  Training Center  (FLETC)
Explains how  the  evaluation  results will be  used     . : The component’s documented evaluation processes fully included information to meet the  attribute for all aspects of its evaluation process.   : The component’s documented evaluation processes included some information to address a given  attribute but did not include information to fully and consistently meet all parts of the attribute. This  includes, for example, incomplete evaluation processes or incomplete information to address a given  attribute for certain levels of the evaluation.  : The component’s documented evaluation processes did not include information to address the  attribute for the evaluation process.
CBP’s documentation presents ways CBP can implement the Kirkpatrick model but does not indicate  the actual process that will be used.   CBP’s documentation outlines when the various levels of evaluations are supposed to be  administered, but does not present a timeframe for CBP to analyze the evaluation feedback.  CBP’s documentation identifies some CBP entities that perform the evaluations and receive the  evaluation information, but does not do so consistently for each level of evaluation.  ICE’s documentation outlines who is responsible for the various level 1 evaluation activities and what  ICE stakeholders should be involved in the process, but does not provide this information consistently  for evaluation levels 2 and 3.  The Coast Guard’s documentation provides guidance on when to administer the evaluation surveys,  but it does not specify timeframes for the Coast Guard to analyze the evaluation data.
TSA’s documentation states that it will evaluate training programs using the Kirkpatrick model and  discusses the process in a very general sense. However, the documentation does not indicate  specifically how TSA will conduct each level of evaluation or the circumstances in which a certain  approach will be used.
FLETC’s documentation indicates that it will evaluate its training programs using the four-level  Kirkpatrick model. However, the documents do not consistently indicate how FLETC will develop,  administer, and analyze the evaluation data for each level. For example, for level 3 evaluations,  FLETC’s documentation includes some policies and procedures that govern the evaluations, but  these procedures do not provide specifics on the process such as how the surveys are developed  and deployed, and how the surveys are sent to a sample of students, among others.
section_title: Appendix III: Leader Development Framework Programs and Implementation Status
paragraphs: The following information appears as interactive content in figure 3 in the  report body when viewed electronically.
section_title: Appendix IV: Summary of Leader Development Program Assessment Approach
paragraphs: The Department of Homeland Security’s (DHS) Leader Development  Program (LDP) Office has developed a program-wide assessment  approach intended to analyze the impact of the LDP over time and to  assess whether the program is targeting the right things in the right way.  This assessment approach, which applies to all LDP program elements— including Capstone, Cornerstone, and other programs—consists of (1)  biannually collecting and analyzing completion rate data, (2) collecting  and analyzing responses to six core evaluation questions, and (3)  tracking 12 program performance measures. Table 11 provides more  detailed information about this approach.
section_title: Appendix V: Comments from the Department of Homeland Security
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Joseph P. Cruz, Assistant  Director; Chuck Bausell; Gary Bianchi; Gustavo Crosetto; Peter DelToro;  Michele Fejfar; Eric Hauswirth; Adam Hoffman; Susan Hsu; Kirk Kiester;  Tracey King; Taylor Matheson; Signora May; Linda Miller; Julia Vieweg;  and Yee Wong made key contributions to this report.