Human Capital: Selected Agencies' Experiences and Lessons Learned in Designing Training and Development Programs
Published: 2004-01-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Effective training and development programs are an integral part of a learning environment, helping improve federal workforce performance in achieving agency results. Therefore, in this report GAO was asked to identify examples of selected federal agencies' experiences and some of the key lessons they have learned in designing their training and development programs. This work focused on ways that these agencies (1) assessed agency skills gaps and identified training needs, (2) developed strategies and solutions for these training and development needs, and (3) determined methods to evaluate the effectiveness of training and development programs. GAO worked with five agencies to identify their experiences and lessons learned: the U.S. Army Corps of Engineers (USACE), Department of Defense; Fish and Wildlife Service (FWS), Department of the Interior (Interior); Internal Revenue Service (IRS), Department of the Treasury; the Office of Personnel Management (OPM); and Veterans Health Administration (VHA), Department of Veterans Affairs (VA). Agency officials provided information during interviews and furnished supporting documentation for analysis and review.
section_title: What GAO Found
paragraphs: GAO identified important lessons learned from five federal agencies' experiences in designing training and development programs for their employees that could be useful to other agencies facing similar challenges. These lessons learned are related to the following three areas. Assessing Agency Skill Requirements and Identifying Training Needs: The agencies used a variety of approaches to assess current and future skill requirements, such as implementing workforce planning models and conducting knowledge and skills inventories. Generally, the agencies are transitioning to more comprehensive approaches. One of the lessons learned was to involve stakeholders and benchmark with others to identify critical skills and competencies and related training needs. For example, IRS officials believed they needed a leadership competency model directly based on the work of their agency's business units. To develop a comprehensive model, they interviewed top IRS leaders and benchmarked with leading practices in the public and private sector. Developing Strategies and Solutions for Training Needs: The agencies considered a mixture of delivery mechanisms, as well as potential sources for training and development opportunities. However, projecting costs and benefits of proposed training and development programs presented challenges for them. The agencies usually developed broad information on anticipated benefits and expected costs of potential investments, although often without tying benefits to specific performance improvements or considering all costs. One of the lessons learned was to establish mechanisms to avoid duplication or inconsistencies. Education Service Representatives in each regional VHA network, for example, coordinate training and development programs with headquarters--sharing information about successful practices and identifying areas where coordination is needed. Determining Methods for Evaluating Training Programs: Overall, the agencies relied primarily on participants' end-of-course evaluations, but they are beginning to use more comprehensive evaluation approaches, including limited use of return-on-investment analysis techniques. One of the lessons learned is to plan for the use of multiple data types and sources in order to attain a balanced assessment once the course is implemented. For example, USACE's training center incorporated pre- and post-tests on over 90 percent of its courses, as well as approaches to collect participants' and course managers' feedback, as part of the design. Four of the five agencies provided comments on a draft of this report. Interior and VA said that they generally agreed with the report's findings regarding their respective agencies. IRS and OPM said that they appreciated the opportunity to be included in the report and to share information on training activities. USACE provided no comments on the draft report.

section_title: Letter
paragraphs: To effectively address the nation’s most urgent priorities and take  advantage of emerging opportunities, federal agencies need to continue to  build their fundamental capabilities to carry our their work in an  environment that is increasingly complex and rapidly changing.  To build  their capacity, agencies should invest resources wisely to ensure that their  employees have the information, skills, and competencies they need to  succeed.  As you are well aware, these investments must include training  and development efforts to continuously enhance the skills and  competencies of the federal workforce and improve the quality of agencies’  results.
As agreed with your office, this report provides information on selected  federal agencies’ experiences and lessons learned in key aspects of  designing training and development programs for their employees.   Specifically, we focused on the agencies’ experiences and lessons learned  related to   assessing current and future agency skill and competency requirements  and identifying related training and development needs,  developing strategies and solutions for training and development needs,   determining methods to evaluate the effectiveness of training and  development programs.
For this review, lessons learned were defined as knowledge that could be  applied in the future that the agencies gained through either positive or  negative experiences.  The experiences and lessons learned from the five  agencies we reviewed may well provide some valuable ideas and useful  approaches that could be adopted by other federal agencies as they attempt  to address ongoing training and development challenges—particularly  those related to the elements within the training process that relate to  design and development.
To address these issues and as agreed with your office, we focused our  review on five federal agencies:  U.S. Army Corps of Engineers (USACE),  Department of Defense; Fish and Wildlife Service (FWS), Department of  the Interior (Interior); Internal Revenue Service (IRS), Department of the  Treasury; Office of Personnel Management (OPM); and Veterans Health  Administration (VHA), Department of Veterans Affairs (VA).  At the  headquarters of the five agencies—and in some field locations—we  collected and reviewed documents on the agencies’ training and  development efforts and interviewed officials from the agencies’ human  capital and training organizations, as well as assorted program offices.  We  selected the five agencies for various reasons, including your office’s  interests, the diversity of employee occupations within the agency, and  reported innovative approaches for training and developing their  employees.  This agency selection process was not designed to identify  examples that could be considered representative of all training and  development efforts at the five agencies or the federal government as a  whole.  Furthermore, in citing examples that relate to the lessons learned  on the design of training, we did not assess the effectiveness of these  training programs and practices.  Rather, we attempted to highlight some of  the experiences and lessons that the agencies found helped them move  forward in improving their training and development programs.  Federal  agencies’ training and development strategies, and how they are designed  to operate in conjunction with other strategies to improve individual and  organizational performance, continue to change and evolve.
We conducted our review from August 2002 through November 2003 in  accordance with generally accepted government auditing standards.  See  appendix I for additional information on our objective, scope, and  methodology.
subsections: section_title: Background
paragraphs: We recently issued an exposure draft of an assessment guide that  introduces a framework for evaluating a federal agency’s training and  development efforts.  This assessment guide consists of a set of principles  and key questions that federal agencies can use to ensure that their training  and development investments are targeted strategically and are not wasted  on efforts that are irrelevant, duplicative, or ineffective.  As detailed in our  assessment guide, the training and development process can loosely be  segmented into four broad, interrelated components: (1) planning/front- end analysis, (2) design/development, (3) implementation, and   (4) evaluation.  Figure 1 depicts an overview of this process along with the  general relationships between the four components that help to produce a  strategic approach to federal agencies’ training and development efforts.   Although these components can be discussed separately, they are not  mutually exclusive and encompass subcomponents that may blend with  one another.  For instance, evaluation is an integral part of the  planning/front-end analysis as agencies strive to reach agreement up front  on how the success of various strategies to improve performance,  including training and development efforts, will be assessed.  As noted in  the assessment guide, agencies can build on lessons learned and  performance data and feedback from previous experiences.  This report  can provide a starting point for agencies to use to build on the experiences  and lessons learned by the five agencies we reviewed as part of their efforts  to design and develop training and development programs.  (See app. II for  a description of the five agencies included in this study.)
Our assessment guide also summarizes our observations on the core  characteristics that make a training and development process effective and  strategically focused on achieving results.  These eight core characteristics  are described in more detail in appendix III, and include  leadership commitment and communication,   partnerships and learning from others,   data quality assurance, and   continuous performance improvement.
A concerted effort to integrate these core characteristics can further an  agency's efforts to continually improve its training and development  process.
section_title: Agencies Used Varied Approaches in Assessing Skills and Competencies and Identifying Related Training Needs
paragraphs: Federal agencies face diverse challenges in their efforts to identify and  measure the skills and competencies that their employees must possess to  support missions and goals.  Officials from the five agencies in our review  recognized the importance of assessing the need for specific skills and  competencies now and in the future in order to identify related workforce  training needs.  These agencies generally focused on the desired  performance of the agency and its employees, determined the difference  between the desired and actual skill levels, and attempted to identify the  key factors contributing to performance, including the need for enhanced  workforce competencies.  Officials used a variety of approaches and tools  to assist in determining the human capital skills and competencies that are  critical to achieving their long-term goals.
An agency’s ultimate goal in undertaking training and development efforts  is, of course, to optimize employee and organizational performance.  To  help ensure that each training program is linked to improving individual  and agency performance, agencies first need to analyze their strategic and  performance goals so that they can determine where training and  development can most effectively enhance goal achievement.
Organizations can evaluate the extent to which human capital approaches  support their accomplishment of current, emerging, and future strategic  goals through the use of workforce planning.  Workforce planning focuses  on determining the skills and competencies needed now and in the future  to meet the agency’s goals; identifying the current and projected level of the  skills and competencies of the workforce; and crafting strategies for  acquiring, developing, and retaining people to address any identified needs.   These needs include the knowledge, skills, and abilities needed for the  agency to pursue its current and future mission as well as the size of the  workforce and its deployment across the organization.  After identifying  the skills and competencies that employees need now or in the future,  agencies must tackle the challenge of determining what combination of  strategies to use, such as hiring new employees with needed skills and  competencies, relying on outsourcing, and/or enhancing employees’ skills  and competencies through training and development. While agencies’  approaches to workforce planning will vary, we have identified the need for  a strategic workforce planning process to ensure that each agency’s human  capital program capitalizes on its workforce’s strengths and addresses  related challenges in a manner that is clearly linked to achieving the  agency’s missions and goals.  Figure 2 presents a model of this strategic  workforce planning process.
The focus in this report is on the workforce strategies that involve the  design of training and development programs.
subsections: section_title: Agencies' Experiences in Assessing Skill and Competency Requirements and Identifying Training Needs
paragraphs: The five agencies used several approaches to design training and  development programs that focused on skills and competencies their  assessments indicated needed enhanced attention.  One common approach  that officials used to help identify training needs was interviewing or  surveying managers, supervisors, and employees.  Agencies also  established councils and held conferences, made comparisons with leading  organizations through benchmarking, and analyzed workforce data and  trends.
OPM’s 2001 skills assessment, for example, relied on a survey of agency  managers and supervisors.  Officials used the survey results to identify the  most important occupational competencies needed to achieve OPM’s  mission, the level at which employees possessed those competencies, and  the level at which they would be needed in the future.  Their analysis  identified minor gaps in the level of competencies needed for both current  and future work in mission critical occupations.  In addition, the analysis  pointed out more serious developmental needs for OPM’s retirement and  insurance benefits specialists.  These needs were related to the changing  role of these specialists, who increasingly need to work more closely with  clients in responding to complex issues.
Also in 2001, IRS established a workforce planning council consisting of  senior management representatives from each of the agency’s operating  divisions.  IRS officials told us that this council has become the primary  vehicle for communicating workforce planning information among IRS’s  four operating divisions.  At FWS, the human resources office hosted a 3- day workforce planning conference in 2002 to draw on the experience and  expertise of agency personnel in identifying critical workforce issues for  the agency for the next 3 to 5 years.  Managers and program experts  representing all eight major FWS program offices, seven field regions, and  headquarters offices participated.
Agencies also compared their performance and needed skills and  competencies with leading organizations through benchmarking.  In  August 2000, VHA commissioned an internal task force charged with  developing a well-defined, comprehensive succession plan for the agency.   The ideas garnered from benchmarking led to VHA establishing an  expectation for agency leaders to help identify and train their successors.   The task force’s December 2001 report presented a comprehensive  succession plan for VHA, and implementing a comprehensive leadership  development program was one of the six major components of this plan.
Agency officials also analyzed workforce data to assess skills and help  identify training needs.  Generally, they collected information on employee  demographics and retirement eligibility and used these data to project  attrition and retirement rates. OPM officials, for example, collected and  analyzed attrition and turnover data on the agency’s senior executives  along with the distribution of current executives by unit and projected  retirements through 2010.  They also collected and analyzed data on hires,  separations, and workforce diversity across the agency.  As part of this  analysis, officials assessed the agency’s use of contractors and considered  how sourcing alternatives could affect OPM’s plans for hiring, training, and  development.
Officials from the five agencies told us that they used a wide range of  resources and tools to assess skills and competencies as part of identifying  and designing needed training and development programs.  They used  workforce planning models; assessed the workforce in view of  organizational, occupational, and unit-based competency standards;  conducted knowledge and skills inventories; and evaluated job  performance appraisals and information from individual development  plans (IDP).  To identify needed executive competencies, for example,  OPM used a five-step workforce planning model that it had developed in its  role of providing human capital tools for use by other federal agencies (see  fig. 3).  The resulting analysis called for enhancing leadership development  within OPM.
Officials said that their agencies have transitioned, or are in the process of  transitioning, to more comprehensive, consistent planning approaches.   This transition is coming about as agencies attempt to institutionalize their  workforce planning efforts as part of their ongoing strategic planning and  budgeting processes.  In 2002, FWS conducted its first formal, agencywide  workforce planning process.  Although FWS initially employed a contractor  to help develop a permanent workforce planning process, it plans to  continue to manage it in-house.  This workforce planning process is to be  implemented on a 2-year cycle that is integrated with the agency’s strategic  planning and budgeting processes.
section_title: Agencies’ Lessons Learned in Assessing Skill and Competency Requirements and Identifying Training Needs
paragraphs: By considering the viewpoints of a range of stakeholders and candidly and  openly assessing progress toward meeting their goals, agencies can help  ensure that their strategic and annual performance planning processes  adequately reflect current ideas, policies, and practices in the field.   Agencies continue to integrate workforce planning into these other  planning processes.  It is important to note that a wide variety of strategies  other than training and development are also available to agency leaders as  they attempt to transform their cultures and operations.  Training and  development is not always the best solution—reengineering processes or  other actions may be needed to build an environment that effectively  supports performance.  In addition, training and development strategies  frequently need to be implemented in conjunction with other initiatives,  given that the day-to-day environment and organizational culture may also  need to change to enable employees to successfully use new skills or  competencies on the job.
Our review identified four lessons learned by the agencies related to  assessing skills and competencies to identify, focus, and prioritize training  needs.
Lesson learned:  Involve key stakeholders and benchmark with  other organizations when identifying skills and competencies to  help ensure that training and development programs are aligned  with current and emerging needs and business practices.
Organizations in the private and public sectors have increasingly turned to  developing competency models that outline behaviorally defined skills and  competencies employees should possess and that can be tied directly to  training and development plans and programs.  We have found that an  effective performance management system uses competencies to provide a  fuller assessment of performance.
IRS involved key stakeholders and benchmarked with other organizations  in developing its leadership competency model In working to build its leadership development program in the wake of the  IRS Restructuring and Reform Act of 1998, IRS officials believed they  needed a leadership competency model that was based directly on the  work of IRS's business units.  To identify the essential characteristics that  enable IRS employees to function as effective leaders in the newly  modernized agency, human capital specialists at IRS conducted behavioral  interviews with 35 top IRS leaders in 1999, asking them to identify major  successes and challenges during their careers.  Using information gathered  from these interviews, officials identified core management responsibilities  and corresponding competencies required for leaders in IRS.  With the  assistance of a contractor, IRS validated the leadership competency model  by comparing it against leading practices in the public and private sector  and linking it to the mission and goals of the agency.  The resulting  competency model now forms the basis for IRS’s leadership development  efforts, as well as how IRS selects, evaluates, and recognizes its leaders.   Figure 4 shows a listing of IRS’s five core management responsibilities and  the 21 corresponding leadership competencies.
VHA benchmarked with other organizations and used a pilot test in  developing its high performance development model   VHA conducted an extensive literature search and benchmarked with  several leading private sector firms (including Bell South, Coca-Cola, and  Motorola) to provide a foundation for its effort to create a new competency  model for VHA employees.  VHA used this information in developing its  high performance development model, which was implemented throughout  the department in 2002.  This model consists of eight core competencies  and related performance tools that represent the major skills and  competencies that employees need to fulfill VHA’s mission.  The model was  designed to serve as a framework for identifying and developing future  leaders, as well as to enhance development of VHA’s entire workforce.   VHA said that using this model helped more effectively align training and  development programs with agency priorities.  According to VHA, the fact  that the core competencies apply to all levels and functions within the  agency helps ensure alignment within and between organizational units  and is a key component in motivating sustained and improved  performance.  VHA also uses the model on an individual employee basis as  a process for identifying specific developmental needs.  Figure 5 lists the  eight core competencies in VHA’s high performance development model.
Lesson learned:  Analyze existing agency data on individual  employee’s skills and competencies and information from  performance appraisals to help identify skills and competencies  that need to be addressed throughout the agency as well as on an  individual basis.
To obtain a unit- or agencywide perspective of skills and competencies,  some agencies such as USACE and IRS have explored new ways of  aggregating data from tools that are primarily focused on individual  employees, such as IDPs, performance assessments, and 360-degree  feedback instruments.  Officials from these agencies told us that this  information helped them discern a clearer picture of the overall strengths  and weaknesses of their employees and offered direction in planning and  designing training and development programs to help focus efforts to  enhance skills and competencies throughout the agency.
USACE’s automated training management program provides a Web- enabled integrated database   According to USACE officials, using an automated training management  program has allowed managers to identify divisionwide gaps in workforce  skills and competencies.  Using this system (currently in four of USACE’s  eight divisions) employees prepare an IDP assessing their knowledge,  skills, and abilities in relation to a series of mission essential tasks.  With  supervisory guidance, each task is identified as critical, important, or  beneficial and employees indicate whether they have received adequate,  partial, or no training in that area.  With this assessment as a guide, the  supervisor and employee can consult the system’s built-in course catalog to  select internal or external training to enhance the employee’s development.   In addition, the system also has the capability of aggregating data.  USACE  officials said that this capability provides a simple method for division  managers to obtain a picture of the level of skills and competencies in their  workforce.  This information informs decisions on training priorities and  helps managers determine the most efficient use of available resources.
IRS aggregated data from 360-degree feedback instruments to help  identify training needs   To assess the progress and developmental needs of leaders within IRS, the  agency’s leadership development office recently aggregated and analyzed  multiyear data from the 360-degree performance assessments of IRS  managers.  This analysis helped to show areas of strength and weakness in  skills and competencies across the agency’s managerial ranks.  The  director of leadership and organizational effectiveness at IRS said that, in  the past, the agency did not sufficiently assist managers in effectively using  360-degree feedback they received.  However, he said that IRS now  emphasizes the importance of using 360-degree feedback data, both on an  organizational and individual basis, to focus on strengths in developing key  leadership competencies.  IRS’s Extraordinary Leader Program involves  designing unique developmental approaches to help managers become  more effective leaders.  Using results of the manager’s 360-degree  assessment, IRS creates a customized leadership development program  focusing first on correcting any “fatal flaw” weaknesses and then building  on the manager’s demonstrated strengths in areas that IRS has identified as  key to providing effective leadership within its organizational culture and  operating environment.
Lesson learned:  Link the agency’s workforce planning efforts with  training needs assessments to ensure consistency and enhance  strategic alignment.
The agencies’ training and development organizations had a range of  responsibilities, including designing training and development programs  based on strategic initiatives, soliciting input from stakeholders, and  prioritizing and scheduling training based on strategic initiatives and  stakeholder input.  Generally, the training organization and agency  stakeholders can work together more effectively when they better  understand how each office or function within the agency contributes to  achieving business goals.  In some cases, this included efforts to link  training needs assessments with the agency’s overall workforce planning  efforts.  Officials told us that this linkage helped ensure that workforce  plans developed by the agency’s human capital office were consistent with  training needs assessments done by the agency’s training and development  organization.
FWS involved key internal stakeholders in its planning processes   When assessing workforce skills and competencies, FWS officials worked  to ensure that the agency’s workforce plan was linked with a training needs  assessment done by its training center.  The workforce planning effort  identified broad competencies needed across the agency’s workforce while  the training needs assessment identified the types of training courses to  develop skills and competencies within agency units and occupations.   Officials said that they viewed the training needs assessment as a tool that  was useful in refining the agency’s workforce plan and in prioritizing and  budgeting for the development and delivery of training.  Both efforts  involved key stakeholders from the human capital and training offices as  well as other FWS units.  In the future, officials said they will rely on the  results of the agency’s workforce planning efforts to directly serve as the  agency’s training needs assessment.
USACE relies on its Learning Advisory Board and automated training  management program to effectively link planning efforts USACE relies chiefly on the coordination activities of its Learning Advisory  Board to ensure its workforce planning efforts and training needs  assessments are effectively linked.  USACE in 2001 formed the Learning  Advisory Board, comprised of senior managers from across the agency, to  review the adequacy of USACE’s training and development and ensure that  training is properly aligned with the agency’s missions, goals, and plans.  In  addition, the four divisions that use the automated training management  program can also rely on data from that system to assess training needs.   This system allows managers to compare information on individuals’ skills  and competencies with workforce planning results from within the division  and across the agency.  According to USACE officials, this systematic  comparison more closely links workforce planning and training needs  assessments to the essential mission-related operations.
Lesson learned:  Consider the training needs of staff from other  organizations that will likely use the agency’s training programs or  facilities to effectively leverage training investments and meet  diverse needs.
When planning and designing training programs for its employees, FWS  and USACE officials told us that they gained insight into the assessment of  potential training solutions by considering the possible involvement of  trainees from other organizations.  In some instances, for example, it would  not have been cost-effective to design, develop, and deliver a training effort  for a small number of employees or occupations.  However, officials’  determinations that other agencies or organizations also needed similar  training provided the critical mass needed to move forward.  Officials said  they found that partnering with other organizations helped make training  efforts more cost effective to design, develop, and deliver.
FWS’s training center assessed training needs of possible participants  from other Interior components   Officials from FWS’s training center said that although they focus primarily  on meeting the needs of FWS employees when planning and designing  training, they also look at the training needs of other organizations,  particularly other agencies within Interior.  To aid in communication and  coordination, agencies within the department designate employees to serve  as liaisons between each of the Interior agencies and FWS’s training center.   These liaisons facilitate efforts to incorporate their agency’s needs into the  training center’s plans and the design and delivery of training and  development programs.  According to agency officials, these liaisons help  ensure that the center’s courses remain current because they facilitate a  dialogue between the various agencies.  For example, a recent FWS course  on wetland plant identification involved participants from FWS; U.S.  Geological Survey, which is another Interior component; Natural  Resources Conservation Service, an agency of the U.S. Department of  Agriculture; USACE; and three private sector firms.
USACE’s training center considers training needs of staff from other  federal and state organizations  Officials from USACE’s training center told us that during the agency’s  annual training needs assessment, they assess the possible training demand  from other organizations’ employees in addition to identifying workforce  development needs of units and offices within USACE.  According to  USACE, approximately 2,500 people each year—about 25 percent  of the  participants in the agency’s training center programs—are from other  federal and state organizations.  Officials noted that the training center  offers courses needed to obtain certifications for certain professional  requirements.  It offers courses accredited by several professional  associations, including the National Society for Professional Engineers, the  American Institute of Architects, and the International Association for  Continuing Education and Training.  USACE officials said that they would  like to offer training to a greater number of employees from private sector  firms; however, current law requires receipts for services provided to  private individuals and organizations to be deposited into the general  treasury as miscellaneous receipts.  USACE officials said that this  requirement hampers the ability of USACE’s training center to keep funds it  could generate to further invest in its training programs.  However, when  agencies are required to return receipts for services to the Treasury, the  Congress is preserving its oversight and control over the programs  generating the fees.
section_title: Agencies Developed Strategies and Solutions for Their Training Needs
paragraphs: Agencies carry out their training and development efforts on the basis of  estimated needs, priorities, and available resources and recognize that  adequate planning allows them to establish priorities and determine the  best ways to leverage investments to improve performance.  The five  agencies we reviewed set priorities for training and development on the  basis of various factors, such as the results from skill and competency  assessments, the availability of resources, and the interests of agency  leaders.  They usually relied on training officials, agency managers, and  subject matter experts to assist in developing strategies and approaches for  addressing training needs.  Although the agencies designed and delivered  training using both centralized and decentralized approaches, we found  that leadership development programs were more highly centralized and  managed at headquarters.  Agency officials acknowledged that they found  projecting costs and benefits of proposed training and development  programs to be very challenging.  Although they sometimes developed  broad information on anticipated benefits and expected costs, this often  did not involve tying anticipated benefits to specific performance  improvements or considering all related costs.
As outlined in figure 6, agencies can plan and establish priorities by  developing an annual training plan to target developmental areas of  greatest need and outline the most cost-effective training approaches to  address those areas.  Considerations involved in assessing investment  opportunities for the training plan include balancing the competing  demands confronting the agency and the amount of resources available in  order to determine how those demands can best be met with available  resources.  It is also important to consider how to effectively integrate all  of the strategies the agency plans to use to improve performance and meet  emerging demands.  When training is identified as a solution to improve  performance, agencies can compare various training strategies by weighing  their estimated costs and anticipated benefits to build a convincing  business case that supports the selected training strategy.  Developing a  business case that sets forth the expected costs and benefits of the  performance improvement investment provides decision makers with  essential information for allocating necessary resources.
subsections: section_title: Agencies’ Experiences in Developing Strategies and Solutions for Training Needs
paragraphs: The agencies we reviewed used a wide range of strategies and solutions to  improve performance through designing training and development  programs for their employees.  Officials told us they considered a mixture  of both on-the-job and other developmental programs, contemplated an  assortment of mechanisms for delivering the training, and assessed  potential sources to meet their learning needs.  For example, USACE’s  leadership development program for midlevel engineers and scientists  involved formal classroom training, mentoring, and a 6-month  developmental assignment.  VHA employees can access a wide variety of  informational and educational content through the VA Knowledge  Network, a satellite-based system of live and on-demand programming  delivered directly to employees’ desktops.  IRS’s training unit developed an  automated ROI workbook tool that the agency’s business units can use to  assess whether proposed training programs should be delivered in a  classroom or by an e-learning approach.  OPM partnered with an employee  union to offer a midcareer development program that provided an  opportunity for current OPM employees to enhance existing skills, explore  new career fields, and gain practical experience.  Officials from FWS’s  training center cited courses in negotiation, communication, and  interpersonal skills as examples of vendor-provided courses.  They said the  center decided not to invest in designing these courses since they do not  require field experience and expertise, which are critical in designing other  FWS courses.
Projecting costs and benefits of proposed training and development  programs was a challenge for the five agencies.  They usually developed  broad information on anticipated benefits and expected costs, often  without tying anticipated benefits to specific performance improvements  or considering all costs related to the training program.  For example, VHA  officials told us that the agency’s assessments of anticipated benefits and  expected costs of proposed training are generally unsystematic.  One VHA  office or field location may not have analyzed the relative costs or benefits  of proposed training while another office or location may have considered  anticipated benefits and developed estimates of costs and savings using  different training approaches.  At FWS, some proposed training programs,  such as its Advanced Leadership Development Program, involved detailed  estimates of costs, both for the training center and for participating FWS  field offices, as well as the identification of specific competencies to be  developed in the program.  Other proposed training programs at FWS did  not have documented and detailed estimates of expected costs and  benefits.  FWS officials said that they assess the anticipated costs and  benefits of all their proposed training and development programs but that  the extent of these assessments and the amount of documentation  supporting the assessments vary, depending on many factors, such as the  content, delivery mechanism, and uniqueness of the proposed training.
According to agency officials we interviewed, limited funding sometimes  affected agencies’ abilities to design and deliver training and development  programs that officials believed were needed.  USACE officials said, for  example, that in a recent survey, about two-thirds of agency supervisors  and one-half of agency executives believed that the agency had less funding  for training civilian employees than is needed.  Funding and resource  limitations sometimes forced the agencies to think of new and practical  ways to ensure that their employees had the knowledge and skills needed  to carry out their work.  For example, because of an unexpected decrease  in available travel funds, VHA officials canceled plans for a large national  conference on the use on an automated managerial cost accounting  system.  This system was designed to provide VHA managers with data  important in making clinical decisions, managing workload, and controlling  medical care costs.  Instead, to enable employees to obtain the information  that would have been presented at the conference, VHA officials provided  the content via a satellite broadcast along with a series of audio  conferences.
At FWS, the number of people that the training center can train onsite, of  course, is limited by the current capacity of the facility classrooms and  residences.  According to FWS officials, incorporating a blended learning  approach into its curriculum has enabled trainees to perform part of the  course work outside class, thus allowing instructors to focus on those  topics that require special attention.  Training center officials also said that  the agency has increased its use of e-learning and other mechanisms to  develop employees where they work instead of coming to the training  center.  Officials told us that FWS has reduced training costs by offering  more training online and using CD ROMs to provide field offices with  course material that previously had been offered only on location at the  training center.
section_title: Agencies’ Lessons Learned in Developing Strategies and Solutions for Their Training Needs
paragraphs: Agency officials have encountered a variety of challenges in their efforts to  design training programs to meet the developmental needs of their  employees.  How agencies respond to these challenges can greatly affect  their success in aligning priorities with strategic direction, assessing  strategies, identifying alternative sources and methods, weighing potential  costs and anticipated benefits, and assessing how other performance  improvement initiatives might complement training efforts.  It is also  important to consider evaluation feedback on an ongoing basis.  We  identified eight lessons learned related to the five agencies’ efforts to  develop strategies and solutions for their training and development needs.
Lesson learned:  Incorporate information on employees’ various  competency levels and job needs into the design of training and  development programs to increase their relevancy and timeliness.
When designing effective training and development programs, the way the  work is actually to be done on the job and the developmental needs of the  expected trainees are key considerations.  Analyzing the tasks of specific  jobs and occupations can help ensure that training accurately reflects the  way employees are expected to perform on a day-to-day basis.  To help  ensure that the training effectively addresses employees’ developmental  needs, agencies can determine the workforce’s level of proficiency in  mission-critical skills and competencies by conducting skills assessments  and using information obtained through interviews or surveys of  employees and their supervisors.  The increased information and insight  provided by these approaches can allow agencies to incorporate  information on employees’ various competency levels and job needs into  the design of training and development programs--increasing both the  relevancy and timeliness of the learning.
FWS targeted training to employees to leverage specific knowledge and  experiences FWS’s training center recognized employees’ various competency levels  and job needs when developing courses on the use of geographic  information systems (GIS), which are becoming increasingly important  mapping and information analysis tools for natural resources agencies,  according to FWS officials.  Rather than providing instruction focused  solely on software features and functions of GIS, the training center  designed its curriculum to teach the application of GIS to employees based  on their roles in natural resources management.  With a focus on these  varied roles and related needs, the center developed separate GIS training  courses for “explorers” (natural resource managers or others just wanting  to know about GIS), “users” (biologists and other personnel using GIS in  their daily job) and “developers” (those individuals designing and  developing a natural resources GIS for use by others).  Officials at FWS’s  training center said that with the training targeted to the specific  background and needs of employees, FWS can minimize the time spent  teaching participants information that they already know or do not need to  know in carrying out their job responsibilities.  The training center’s  curriculum consists of 15 separate GIS courses, with 3 additional courses  under consideration or development.  According to FWS officials, about  1,200 FWS employees at over 400 offices use GIS software in their jobs.
IRS used skills assessments to focus on developmental needs To assist in identifying employees’ competency levels and incorporating job  needs into training, IRS developed and used technical assessment batteries  for the agency’s field assistance personnel and customer service  representatives.  These multiple-choice instruments were designed to  assess the key technical knowledge that the employee needs in order to  carry out his or her job.  On the basis of each employee’s test results, the  agency will recommend specific training and other appropriate  interventions, such as mentoring, to improve performance.  IRS officials  told us that in some cases where the assessments showed that individuals  were already knowledgeable in a particular area, employees still wanted to  take the related training because they viewed training as a job benefit as  well as a way to improve knowledge and skills for their jobs.  This provides  an example of how important agencies’ considerations of the  organizational culture and working environment when designing training  and development programs are in preparing for and addressing issues that  may arise during implementation.  As we point out in our training guide,  employees need to not only understand the goals of agencies’ training and  development efforts, but also to accept responsibility for developing their  competencies and careers, as well as for improving their organization’s  performance.
Lesson learned:  Assess options for using other organizations’  course content, staff, services, or facilities when designing a new  training and development program in order to develop efficient and  cost-effective strategies.
When thinking about strategies and sources for the design of a new training  and development program, officials can potentially discover more efficient  and cost-effective approaches through the use of other organizations’  course content, staff, services, and/or facilities.  Adequate planning can  help an agency in meeting the developmental needs of trainees without  overburdening the agency’s training capacity or creating excess capacity.   Obtaining reasonable estimates of likely costs and identifying potential  obstacles of using others’ training resources can help agencies develop  more informed perspectives on ways to effectively leverage resources.
IRS considered several options in designing a course for senior  managers In evaluating options for designing and delivering a new training course for  its senior managers, IRS considered various sources for such training,  including internal resources, contractor support, and partnerships.  IRS  officials said that they considered internal resources to design the training  but quickly realized that the agency did not have sufficient expertise.  The  officials also considered using a contractor but concluded that the costs  would be too high.  Instead, IRS decided to partner with the Federal  Executive Institute, an OPM-sponsored training facility in Charlottesville,  Virginia, that provides training to senior employees from across the federal  government.  IRS officials found that partnering IRS design and subject  matter experts with institute and other renowned leaders in the field  delivered the most cost-effective approach and yielded the best results.   The design team produced a course called “Learning Through Others,”  delivered on the Charlottesville campus.  According to IRS officials, this  course surpassed agency needs and expectations and was less expensive  than a direct contracting arrangement with an outside vendor.  They added  that participants in the training program could learn public service values  through lessons and encounter competency-based experiential learning,  business-related challenges, and a capstone simulation.  IRS officials  characterized the course as high quality and said the prestige associated  with studying at the Federal Executive Institute provided an additional  benefit for IRS participants.
Lesson learned:  Establish mechanisms and controls to avoid  unnecessary duplication or inconsistency within and across  agencies’ training efforts.
The agencies used both centralized and decentralized approaches by, for  example, centrally managing reporting and record keeping while allowing  some localized management of training content.  Whatever mix of  centralized and decentralized approaches is used, agencies recognize that  it is important to limit overlap and duplication and ensure the delivery of an  integrated message when appropriate.  VHA and FWS officials found that  establishing mechanisms and controls is important to limit duplication or  inconsistencies within an agency, across component organizations within a  department, or across the federal government as a whole.
VHA’s Employee Education System helped limit duplication of effort VHA’s Employee Education System, which serves as an internal training  consulting team within VHA, assists the agency’s 21 regional networks in  designing and implementing programs to develop general and specific  skills for VHA employees.  Within each of the 21 regional networks, an  Education Service Representative acts as a liaison in coordinating  numerous developmental programs with VHA headquarters—sharing  information with their counterparts about effective practices and  identifying areas of possible duplication or inconsistency across VHA.   According to VHA officials, the coordination and communication achieved  through this organizational structure has helped ensure consistency in  implementing the agency’s national training priorities.  For example,  officials said that the consulting team assisted in implementing changes to  VHA’s processes for collecting third-party insurance reimbursements in the  wake of legislation that required VA to make greater efforts to collect  unpaid debts from veterans.  They told us that these legislative changes  enabled local VHA facilities to receive these reimbursements, but also  overwhelmed the local billing and debt collection processes.  To address  the problem and help ensure consistency across the agency, the training  consulting team participated in redesigning the processes for coding,  billing, and debt collection; trained the employees responsible for billing  and debt collection in the new processes; and created graphical  representations of the new processes and posted them throughout the  agency’s facilities to aid employees in learning.
Interior’s Training Directors Council facilitated communication across  departmental components Interior used its Training Directors Council to facilitate communication  across the department’s different bureaus, thus helping to minimize  duplicative training and development efforts.  This council provides  opportunities—through formal meetings and informal communications— for training managers from Interior’s various bureaus to share curriculum  and related training ideas with their colleagues.  The director of the training  center at FWS, who chairs the council, told us that on more than one  occasion he has discovered through council business that other Interior  components had developed strategies or solutions to address emerging or  existing needs that FWS’s training center had also identified.  For instance,  the training center at FWS was considering whether to add a new course on  grants management to its curriculum.  Through its participation in the  council, FWS determined that another component agency, the Bureau of  Land Management, already offered grants management courses through its  National Training Center in Phoenix, Arizona.  After reviewing the content  of these Bureau of Land Management courses, officials at FWS’s training  center determined that they did not need to design and develop a separate  grants management course.  Instead, FWS and the Bureau of Land  Management now jointly manage the delivery of this training course.
Lesson learned:  Develop and use criteria for determining the  optimal mix of delivery mechanisms to use in order to select the  most effective approaches given each learning situation.
In response to emerging demands and the increasing availability of new  technologies, agencies are faced with the challenge of choosing the optimal  mix of training delivery mechanisms to design training that is as effective  and efficient as possible.  Agency officials consider a wide variety of  instructional approaches to achieve learning—in the classroom, through  distance learning, or through structured on-the-job experiences.  Officials  also took other factors into account, including whether to provide  individualized instruction or team-based training and when to use blended  learning that combines different teaching methods (e.g., Web-based and  instructor-led) within the same training program.  USACE found that  identifying and systematically using criteria to help select effective delivery  mechanisms assisted in building well-supported justifications for the  design of training and development programs.
USACE used criteria to select media and method of instruction   To select the appropriate media and method of instruction for its training  programs, USACE uses criteria contained in the Corps of Engineers  Systems Approach to Training, the agency’s documented process for  developing training programs.  The criteria include issues such as the  expected frequency of changes to the training content, the size and  diversity of the target population, and the degree of student interaction  required.  USACE officials told us that, using these criteria, course  managers from USACE’s training center coordinate with relevant agency  program offices and subject matter experts to decide on the appropriate  mode for training delivery.  While most of the training center’s courses  occur in a conventional classroom setting, agency decision makers have  focused on trying to identify courses (or modules of courses) to convert  from classroom training to more economical modes of delivery, such as  distance learning, computer-assisted instruction, computer-based  instruction, or a combination of such approaches.  USACE officials said  that many of their courses now incorporate CD ROM and Internet-based  materials as prework assignments before attending classroom training and  for reference use during and after the training events.
VHA used a profiling tool to help in selecting delivery approaches VHA’s internal training consulting team used a training delivery strategy  tool that consists of a series of questions structured to guide users through  the process of selecting an appropriate delivery approach for a proposed  training effort.  This team designed the training delivery strategy tool to  help staff plan, analyze, develop, and deliver training and development  activities.  To aid in decision making, the tool includes factors such as  audience composition, course goals and objectives, course modules, any  prerequisites, participant preparation, and course follow-up and  evaluation.  VHA officials said that applying this tool and analyzing the  resultant profile helps maximize learner understanding, retention, and  application.
Lesson learned:  Ensure that employees have the needed equipment  and technologies so that they can take maximum advantage of  learning opportunities.
Many organizations are taking advantage of more flexible design and  delivery methods made possible by technology to, for example, deliver  training to the user's desktop, thereby making training more accessible and  cost effective.  As agencies move forward in using new approaches, it is  important to ensure that employees have the needed equipment and  technology to take maximum advantage of learning opportunities.
IRS converted mandatory training courses to an online format   IRS recently converted a series of mandatory training courses from face-to- face group briefings to an online format in order to more efficiently provide  this training to its employees.  Through these online mandatory training  programs, which include computer security awareness, ethics issues, and  prevention of sexual harassment, IRS wanted to (1) reduce the burden of  managers who previously had to prepare for and deliver the training,   (2) provide ready access of the information to line employees when and  where they need it, and (3) lower the costs associated with the group  briefings.  IRS officials said that online delivery lessened employee time in  taking the training from approximately 6 to 2 hours and in some cases  eliminated the need for travel.  To ensure the accessibility and usability of  these online briefings, IRS worked to resolve various challenges in the  conversion, such as designing the online product for the lowest computing  capabilities of the bulk of the trainee population and providing alternative  delivery mechanisms for individuals who were without a computer or  Intranet access.  The officials said that they also learned it is important to  design the online briefings based on a common template and style guide to  standardize their look and feel, provide online text-only versions of the  training for persons who are visually impaired, and test the usability of the  online briefings with end users on a range of equipment and allow  sufficient time for needed revisions.
OPM initiated a pilot program for its employees to use online courseware   To explore opportunities for increased use of e-learning approaches, OPM  established a pilot program that allowed about 250 of its employees access  to approximately 1,800 online courses through the Department of  Transportation’s  Transportation Virtual University.  OPM’s training unit  worked with offices throughout OPM to identify employees to participate  in the pilot program.  OPM officials said that all employees selected for the  pilot program had access to the equipment and technology needed to make  use of the University’s online training, by using a computer either at the  employee’s desktop or at some central location.  As a result of this pilot,  OPM enhanced its offerings of online training for its employees by  becoming a partner in the GoLearn.com initiative, a governmentwide  online training center for federal employees.  OPM officials said the pilot  program also showed that not all employees function effectively in an  online training environment and that some employees need a more  structured format in order to learn.
Lesson learned:  Plan early when developing integrated solutions  that complement other planned and ongoing strategies to improve  performance so that when implemented the strategies work  effectively and are aligned to achieve agency goals.
When designing training and development programs, the agencies  sometimes considered how they could integrate them with other strategies  to improve performance and meet emerging demands.  If the work  environment is not conducive to providing opportunities to use new skills  or work in different ways, no matter how good the training program is it  may not be effective or successful in terms of changing on-the-job  performance.  In addition, training and development programs represent a  significant investment of resources (including time and money) and may not always be part of an appropriate solution.  The agencies developed  integrated solutions that included developing and using job aids,   performance support tools, and other approaches to enhance knowledge  management and to aid employees on the job as a complement to training.
OPM planned for an electronic support tool to aid agency employees in  using a new computer system to process retirement claims  As part of its effort to reengineer and modernize its processing and support  of federal employees’ retirement claims, OPM is developing plans for an  integrated Electronic Performance Support System to aid the agency’s  benefits specialists in using a new computer system.  Procedural and  information job aids are to be built directly into the software to provide  documentation and guidance, “just in time” assistance, and error detection.   This is intended to be an integrated system to permit coordination between  different modes of training and enhance the learning and performance of  the OPM employees working with reengineered business processes and the  new computer system.  OPM officials said, for example, that this system  would assist employees in completing steps using actual data and  circumstances of a particular case they were working on rather than  consulting a manual or using data put together just for training.  According  to OPM officials, as the focus under modernization shifts from processing  claims to providing customer service, this system will help employees  working in OPM’s retirement program to interact more directly with  program participants to answer questions and solve problems about  retirement issues.
USACE identified online solutions to help enhance and integrate  training efforts As a complement to the training and development programs it offers to its  employees, USACE recently entered into a joint project with the  Department of Labor to use an online knowledge management system  called Workforce Connections.  This system, which resulted from a  memorandum of understanding promoting cooperative efforts between the  departments of Defense and Labor, will provide the USACE workforce with  on-demand, online access to job aids, performance support materials, and course content 7 days a week, 24 hours daily.  The system will feature  development and maintenance of online communities of practice to  support knowledge management of USACE’s Learning Network, which is  USACE’s overall platform for delivering a wide variety of learning  resources to agency employees.  Another part of the learning network is  USACE’s Virtual Campus, a distance learning site that allows employees  access to Web-based courses and training events.  Another component of  the learning network includes electronic performance support tools, such  as job aids and other information resources.  USACE officials said that they  consider the systems in the learning network to comprise a powerful  solution that effectively integrates the agency’s training efforts.
Lesson learned:  Plan for the direct participation of senior agency  leaders and experienced staff in the delivery of training and  development programs to increase buy-in and build support for  organizational change.
Internal resources, such as subject matter experts and high performers, can  often provide valuable insight into training design because of their  familiarity with the agency's policies, programs, and corporate culture.  To  increase buy-in, help establish greater credibility, and build support for  organizational change, the agencies have learned the value of planning for  the direct involvement of senior managers in the training program.
IRS and OPM involved executives and managers IRS officials told us that a key feature of the agency’s frontline managers  course is that it was designed to use senior managers and experienced  frontline managers drawn from the agency’s business units to teach the  course.  In addition, executives participate in course modules that focus on  emerging issues facing the agency.  The deputy commissioner of IRS’s Wage  and Investment business unit served as the executive sponsor of the  training program and participated in course modules featuring executives.   The officials also told us that IRS executives partner with outside vendors  to serve as an instructor team to deliver all courses designed for senior  managers.  Using business unit executives and managers as course  instructors helped ensure that the course’s content and emphasis related to  the mission, goals, and guiding principles of the agency.
At OPM, agency managers have been the first to take special initiative  training—such as courses on prohibited personnel practices,  whistleblower procedures, and information technology security—before  they are offered agencywide.  OPM officials said that teaching the material  to agency managers in advance of line employees enables the managers to  model desired behaviors and learning for their employees and convincingly  convey how they personally benefited from the training.
FWS and VHA relied on in-house experts FWS’s training center brings in FWS field office personnel when building a  cadre of senior, in-house instructors.  Training center officials said that  involving trainers from the field helps to build trust with trainees and  provides an added level of credibility that neither academics nor other  subject matter experts who lack field experience can easily replicate.   According to these officials, many expert employees come from the field  and stay to teach at the center for 3 or 4 years.  Some, however, teach only  one or two courses or get involved for a short duration before returning to  their positions in the field.  The director of the training center said that he  views this passing-on of information from seasoned veterans to less  experienced employees as crucial for maintaining the unique knowledge  base of the agency.
VHA used “super users” to teach medical center personnel to use its  computerized patient record system, a computer interface that allows  hospital personnel to keep more comprehensive patient records and  enables clinicians, managers, and other staff to review and analyze data  gathered on any patient.  The super users—VHA employees with other job- related duties and responsibilities—were trained to be thoroughly  knowledgeable about the system so they could demonstrate its capabilities  and directly relate the training to employees’ work.  VHA initiated this  strategy when the agency began rolling out the application in 1997.  VHA  officials said that planning to build on the direct involvement of these super  users was successful because they served as first-line resources for  employees’ questions about the new system and helped the agency to build  organizational support for the system.  According to VHA, the agency  developed a cadre of more than 2,500 super users, and about 180,000 VHA  employees use the patient record system.
section_title: Agencies are Considering More Sophisticated Evaluation Approaches As Part of Designing their Training and Development Programs
paragraphs: Without evaluation of training programs, participants may take ineffective  courses that do not provide the necessary learning experience or that do  not translate to improved performance on the job.  Overall, the five  agencies in our review relied primarily on standard end-of-course  evaluations to obtain the participants’ reaction to, and satisfaction with, a  specific training course or learning opportunity.  Although the agencies  encountered challenges given some of the difficulties associated with  measuring the impact of training on individual and organizational  performance, they have begun or are planning to use more comprehensive  and sophisticated evaluation techniques for assessing their training and  development efforts.  Such techniques include the use of pre- and post- testing to determine the extent of learning accomplished, tracking the  performance or advancement of individuals and work units before and  after training is completed to assess professional growth and  improvements in organizational performance, and limited use of ROI  analyses to compare the benefits (quantified in dollars) with the costs of a  training and development program.
To help determine whether the objectives of training and development are  achieved, agencies can begin by incorporating measures of effectiveness  into the design of training and development programs.  Defining objectives  in a measurable way enables agency officials to offer a more convincing  business case and contributes to improving the quality of feedback.  Whenever possible, training goals should measure the individual and  organizational results achieved rather than the training inputs or outputs  (e.g., number of available courses or people trained).  Figure 7 depicts  some of the steps involved in determining the evaluation methods to use in  designing training and development programs.
subsections: section_title: Agencies’ Experiences in Determining Methods for Evaluating Their Training and Development Programs
paragraphs: Agency officials recognized the importance of determining during design  how they planned to evaluate the effectiveness of their training and  development programs.  To collect information on participants’ reaction to  and satisfaction with the training program, for example, VHA uses a  standard evaluation form with questions related to program design,  delivery, outcomes, overall satisfaction, and logistics.  In completing this  survey, training participants evaluate their success in completing learning  objectives and the performance of the faculty.  Other agencies also  obtained participant feedback though interviews or focus groups.  OPM  conducted exit interviews with individuals who participated in that  agency’s Presidential Management Intern program, a 2-year developmental  program for individuals from a wide variety of academic disciplines who  wish to enter the federal service.  These exit interviews were designed to  obtain feedback from the participants about their overall satisfaction and  reaction to the Presidential Management Intern program, including their  suggestions for enhancing the training provided.
The five agencies have begun to use, or are planning to use, more  comprehensive and sophisticated evaluation techniques to assess the  extent to which training and development programs increased employees’  knowledge and skills or enhanced individual and organizational  performance.  One of these more sophisticated evaluation techniques is the  use of pre- and post-testing to determine the extent of learning during the  training program.  USACE’s training center conducts pre- and post-tests on  over 90 percent of the courses it offers and is working toward the goal of  using such tests for all courses.  The agencies also tracked job performance  and the advancement or movement of personnel to assess the potential  effectiveness of training.  FWS officials told us they track participants’  career advancement to determine the extent to which participation in the  leadership development program for midlevel employees contributed to  increased mobility into more responsible leadership positions in the  agency.  According to training center officials, about 37 percent of the  program graduates have taken either promotions or new lateral  assignments since the program’s inception in January 2002.
In addition, some agencies attempted to conduct ROI analyses to compare  the benefits (quantified in dollars) to the costs of a particular training and  development program.  VHA officials pointed to concerted efforts to  conduct ROI analyses on several training and development programs,  including customer service, leadership development, and computer-based  training.  IRS officials, on the other hand, have decided that the challenges  and difficulties in conducting such analyses are not worth the effort for the  resultant information—given the challenge of isolating the performance  improvements that might result from a specific training activity and the  difficulty in monetizing identified benefits in order to calculate the ROI.   IRS instead uses the concept of “time to capability” to determine whether  and to what extent a training course, program, or other training  intervention has improved the organization's ability to perform its mission  successfully.  IRS defines time to capability as the validated accumulation  over time of employees who have been trained in specific competencies  deemed critical to the success of an organizational unit.  Under this  approach, when IRS has trained a predetermined number of employees,  officials consider that the agency has achieved the goal of training a critical  mass within its workforce and conclude that the agency has an  organizationwide capability in the specific competencies.
section_title: Agencies’ Lessons Learned in Determining Methods for Evaluating Their Training and Development Programs
paragraphs: Agencies’ training and development efforts involve a continuous effort  throughout planning, design, implementation, and evaluation.  Therefore, it  is important to recognize that evaluation is not a static requirement to be  carried out after the fact.  When undertaking design and development of  training, agencies can rely on evaluations and benchmarking to determine  what approaches work best given all the related elements, such as the  proposed audience for the training program, the material to be covered,  and possible delivery mechanisms that could be employed.  Determining  methods for evaluating training programs as part of their design can help  identify and remove obstacles to successful implementation.  For example,  an agency officials said that catching potential problems early on saved  valuable time and resources that a major redesign of training later on likely  would have entailed.  On the basis of our review at the five agencies, we  identified four lessons learned regarding the agencies’ efforts to determine  methods for evaluating their training and development programs.
Lesson learned:  Incorporate appropriate aspects of the evaluation  approach when designing training and development programs by  specifying what results are expected to better ensure the  availability and use of quality performance data.
In assessing how and to what degree performance could be improved with  a specific training program, agencies should try to establish a targeted level  of improved performance as well as assess the possible consequences if the  training were not to occur.  Determining a target level for improved  performance can aid agencies in assessing whether the expected costs  associated with the proposed training are worth the anticipated benefits.   Agreeing upon the planned evaluation approach in the design clearly sets  forth the results the agency expects to achieve through the training.  In  addition, planning ahead helps ensure the availability and use of sound and  relevant performance data.
VHA decided to evaluate a customer service training program after it  was implemented  In response to low scores on customer satisfaction surveys, a VHA regional  network office pursued various initiatives to improve customer service,  including the design, development, and implementation of a new training  program focused on creating a more customer-service-oriented culture and  improving employee morale and collaboration to better meet customer  needs.  After delivery of this new training program, called “The Customer,”  VHA selected a contractor to evaluate the effectiveness of the training  effort.  In its report, the contractor stated that because the opportunity to  conduct the evaluation did not occur until after the training program had  been delivered, the use of preferred evaluation methods was not possible.   The contractor reported that this lack of preprogram planning had also  been experienced in some other VHA network offices.  Nevertheless, the  contractor evaluated the customer service training program by surveying  participating employees and their supervisors.  From these survey results,  the contractor concluded that the training program was viewed as  successful for those who needed it, but that the training did not receive a  sufficient level of support from supervisors.  The contractor recommended  that VHA obtain additional feedback from supervisors as well as from an  individual hired to telephone or visit the VHA facilities in the network  office to observe customer service activities.
Lesson learned:  Consider new approaches for collecting and  analyzing performance data with the aim of increasing the quality  and quantity of training evaluation feedback.
The agencies faced various challenges in obtaining a high quality and  quantity of feedback needed to evaluate their training and development  programs.  We previously reported that low participation on the part of  employees and managers in surveys and focus groups may limit an agency’s  access to the data needed to complete valid and useful evaluations of  training programs.  With strong agency support and proper planning,  stakeholders, including training participants, supervisors, managers, and  trainers, are more likely to provide the information and feedback needed to  successfully and effectively evaluate agency training and development  programs.  USACE, for example, recognized that it needed to ensure that it  incorporated a wider variety of stakeholder perspectives in assessing the  impact of training on employee and agency performance.  Stakeholders'  perspectives can be obtained through surveys and questionnaires,  individual or group interviews, or communication with more formal  multidisciplinary bodies such as advisory or education councils.  Valuable  sources of information include the training participants; training designers,  developers, and facilitators; agency leaders, managers, supervisors,  subordinates, and coworkers; employee organizations; internal and  external customers; and functional and subject matter experts.
USACE’s training center altered its approach to obtaining supervisory  feedback on its training efforts USACE’s training center has been attempting to obtain more sophisticated  evaluation feedback to determine if its training courses affected  employees’ behavior on the job.  Training center officials told us that they  originally had planned to evaluate all of the center’s training courses by  obtaining feedback from employees and supervisors 6 months after the  training course on the extent to which employee on-the-job behavior had  changed.  In a test of this approach, the training center sent out hard copies  of two generic survey forms:  one for the employee and one for the  supervisor.  Because the training center had locator information for  trainees but not for their supervisors, both surveys were mailed to the  employee, who was then asked to forward one to his or her supervisor.   Training center officials said the response to this survey effort was  disappointing.  For one 35-person class, for example, the center received 3  employee surveys and 1 supervisor survey.  To increase the level of  feedback, officials told us that the center is transitioning to an electronic  process whereby the survey forms are sent via e-mail to the employee, who  is then requested to forward the survey via e-mail to his or her supervisor.   Training center officials said that in a recent test of this new approach on  one course, the center received a 67 percent response rate from employees  and a 36 percent response rate from supervisors.  The officials told us that  they are working to develop unique evaluation forms for each course in  order to obtain feedback on specific learning objectives rather than rely on  a generic survey form for all courses.
Lesson learned:  Plan for the use of multiple data types and sources  to provide a balanced approach in assessing the effectiveness of  training and development programs.
Successful organizations typically develop and implement human capital  approaches based on a thorough assessment of the organizations' specific  needs and capabilities.  Valid and reliable data are the starting point for  such assessments.  To assess the results achieved through training and  development, agencies can rely upon hard (quantitative) data, such as  productivity/output, quality, costs, and time, or soft (qualitative) data, such  as feedback on how well a training program satisfied employees'  expectations.  By taking steps to agree on measures of success up front,  agency officials can decide on the objectives for each training and  development program.  Using a balanced approach that reflects feedback  from customers and employees, as well as organizational results, is  particularly important as agencies transform their cultures and operations.   In addition, because the work of federal employees can be complex and  often cannot be reduced to a single task, a balanced approach to both the  types and sources of data helps to strengthen the linkages between training  and development programs and improved performance.
USACE conducts evaluations of some training courses In addition to obtaining end-of-course participant feedback and  administering pre- and post-tests for many of its courses, USACE’s training  center also conducts course evaluations of its offerings.  To conduct this  evaluation, the designated course manager from USACE’s training center is  responsible for observing the full course and assessing various aspects of  its design and delivery, including the training content, materials, and  instructors.  Training center officials said that various factors can trigger  the decision to conduct an overall course evaluation, including a significant  decline in trainees’ overall satisfaction with the course, the introduction of  new instructors, the use of contractor assistance, or a specific  recommendation from an agency office or unit.  According to training  center officials, the results of this course evaluation are assembled with the  end-of-course participant survey feedback and pre- and post-testing results  to present a comprehensive and balanced view of the effectiveness of the  training program.
Lesson learned:  Take into account all relevant factors for  determining the costs of a training and development program to  better ascertain whether it is cost-effective in relation to benefits  achieved.
Calculating the ROI for a training program involves identifying and  monetizing the program’s benefits and then dividing this by a full tabulation  of the program’s costs.  These costs should usually include the cost of  program materials provided to each participant; the cost of the facilities;  the costs of the facilitator or instructor, including time for both preparation  and delivery; any travel-related expenses for participants; salaries and  benefits costs of the participants for the time they attend the program; and  an allocation of relevant administrative and overhead costs.
VHA’s evaluation of a leadership development program did not include all  costs In cooperation with VHA’s internal training consulting team, one of VHA’s  regional network offices designed, developed, and implemented a  networkwide leadership development program called “Competency  Development for Leaders in the 21st Century.”  According to information  we gathered during our review, the costs incurred for the consulting team’s  efforts on this training program were not included in the ROI calculation  even though it contributed substantially toward developing and  implementing the program.  VHA officials told us that these costs were not  included in the analysis because the course designers only wanted to  determine the return on the network’s investment, not the agency’s overall  investment.  Agency officials said that one of the main goals of involving  the consulting team was to help the network develop the capability to use  the ROI process to evaluate training.  Although these costs were not  included in the ROI cost tabulation, VHA did include the salaries (plus  employee benefits) of the participants for the time they attended the  developmental program.  As we noted in our recently issued assessment  guide, agencies might overlook the costs of participant attendance when  calculating the total costs of a training program.
section_title: Conclusions and Observations
paragraphs: Federal agencies are faced with the need to invest resources wisely to  ensure that their employees possess the information, skills, and  competencies required to carry out their work successfully.  The examples  provided in this report may help to address this need by describing some of  the experiences and lessons learned that other agencies might find  applicable or adaptable to their unique situations.  This information is  intended to both provide a realistic perspective on how agencies have  approached designing their training and development programs to date as  well as to take a more detailed look at some of the concepts explored in  our recently issued assessment guide focused on strategic training and  development efforts in the federal government.
Our work reviewing the selected agencies’ efforts to design training and  development programs reinforces the significance of good planning and  design of these programs to ensure their successful implementation and  evaluation.  The experiences and lessons learned we identified also  demonstrate how effective design efforts—as part of a strategic training  and development process—rely on the eight core characteristics that we  identified in our earlier work:  (1) strategic alignment, (2) leadership  commitment and communication, (3) stakeholder involvement,   (4) accountability and recognition, (5) effective resource allocation,   (6) partnerships and learning from others, (7) data quality assurance, and   (8) continuous performance improvement.  Indeed, by focusing on these  eight core characteristics, agencies can improve not only the design of their  training and development efforts but also the planning, implementation,  and evaluation of their programs to better ensure that their employees have  the information, skills, and competencies needed to carry out their work  successfully.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report on December 12, 2003, to the Secretary of  Defense, the Secretary of the Interior, the Commissioner of Internal  Revenue, the Director of OPM, and the Secretary of VA.  Interior, IRS, and  OPM provided written comments on the draft report.  In his written  comments (see app. IV), Interior’s Assistant Secretary for Fish and Wildlife  and Parks generally agreed with the report’s findings regarding the  Department and the FWS.  He said that the report provides important  examples that can help the Department continue to move forward with  additional confidence in its actions.  In his written comments (see app. V),  the Commissioner of Internal Revenue said that IRS was honored to share  some of its lessons learned with us for governmentwide dissemination.  He  said that our review also provides IRS with practices from other agencies  to assist IRS in its efforts to continually improve its programs.  In her  written comments (see app. VI), the Director of OPM said that she  appreciated the opportunity for OPM to be included in the report and to  share information on OPM’s training and development activities and  programs.  Interior, IRS, and OPM also provided technical comments to  clarify specific points regarding the information presented in the draft  report, which we have incorporated as appropriate.  In comments by E-mail  through its GAO liaison, VA agreed with the information presented  regarding the Department and had no additional comments on the draft  report.  USACE informed us that it had no comments on the draft report.
As agreed with your office we are sending copies of this report to the  Ranking Minority Member, Subcommittee on Oversight of Government  Management, the Federal Workforce and the District of Columbia, Senate  Committee on Governmental Affairs; the Chairmen and Ranking Minority  Members of the Senate Committee on Governmental Affairs and the House  Committee on Government Reform; and other interested congressional  parties.  We are also providing copies to the Secretary of the Army, the  Secretary of the Interior, the Secretary of the Treasury, the Director of  OPM, and the Secretary of Veterans Affairs.  This report is available to  others upon request.  In addition, the report is available at no charge on the  GAO Web site at http://www.gao.gov.
If you have any questions about this report, please contact me on (202) 512- 6806.  Key contributors to this report are listed in appendix VII.
section_title: Objective, Scope and Methodology
paragraphs: The objective of this review was to provide information on selected federal  agencies’ experiences and lessons learned in key aspects of designing  training and development programs for their employees.  Specifically, we  focused on the agencies’ experiences and lessons learned related to   assessing current and future agency skill and competency requirements  and identifying related training and development needs,  developing strategies and solutions for training and development needs,   determining methods to evaluate the effectiveness of training and  development programs.
For this review, lessons learned were defined as knowledge that could be  applied in the future that the agencies gained through either positive or  negative experiences.
To address this objective we focused on five agencies: the U.S. Army Corps  of Engineers (USACE), Department of Defense; the U.S. Fish & Wildlife  Service (FWS), Department of the Interior; the Internal Revenue Service  (IRS), Department of the Treasury; the U.S. Office of Personnel  Management (OPM); and the Veterans Health Administration (VHA),  Department of Veterans Affairs.  We chose the five agencies for a variety of  reasons, including the diversity of employee occupations within the  agencies, reported innovative approaches for training and developing their  employees, and congressional requester interest.  We selected USACE to  obtain a Department of Defense perspective in light of that department’s  reputation as a leader in the area of training and developing military  personnel.  We included FWS and VHA to obtain information related to a  broad mix of employee occupations.  We selected IRS because of reported  innovative approaches to training and development and included OPM  because of its role as the federal government’s human capital agency.
To obtain information and related documentation, we visited the following  locations:    USACE’s headquarters human resources directorate in Washington,  D.C., and its Professional Development Support Center in Huntsville,  Alabama.
FWS’s headquarters human resources division in Arlington, Virginia, and  its National Conservation Training Center in Shepherdstown, West  Virginia.
IRS’s strategic human resources division in Arlington, Virginia, and the  small business and self-employed business line.
OPM’s headquarters human resources office and the career  development branch of the center for retirement and insurance service.
VHA’s Employee Education System headquarters in Washington, D.C.,  and the medical center and local network offices in Durham, North  Carolina.
It is important to note that our methodology was not designed to identify  examples that would be representative of all training and development  efforts at the five agencies in our review or of the government as a whole.   We did not verify the accuracy and reliability of the data provided to us or  the systems used to produce the information.  Further, in citing training and  development programs as examples in connection with lessons learned, we  did not assess the effectiveness of the training programs and practices.   Rather, our intent was to highlight and briefly describe some experiences  and lessons learned that agency officials believed helped each agency  improve or enhance its training and development programs.
To obtain information about the five agencies’ experiences and lessons  learned related to designing training and development programs, we   Interviewed agency human capital and training officials and subject  matter experts responsible for agency training, performance, and other  initiatives; and  Reviewed and analyzed agency documents such as workforce plans,  analyses, and reports; strategic, performance, and succession plans and  reports; organizational, occupational, and unit-based competency  standards; knowledge and skills inventories; skills gaps assessments;  competency and skill assessments; surveys of agency employees;  training plans and proposals; workforce demographic data; budget data;  evaluation plans and reports; and performance measures.
The lessons learned we identified for inclusion in this report were based on  (1) their linkages with one or more of the eight core characteristics of a  strategic training and development process, which we had identified in our  previous work (see app. III) and (2) sufficient evidence from the agency to  support the experiences that they relayed to us.  At the exit conferences for  the five agencies, we presented agency officials with the list of lessons  learned that we had identified and wished to attribute to their experiences.   At that time, we also informed each agency of the specific examples from  their experiences that we would likely attribute to these lessons learned.   In these meetings, agency officials expressed no objections to the lessons  learned we had identified and in some cases provided additional  information to support specific examples from their experiences that we  proposed to use for this report.
We conducted our audit work between August 2002 and November 2003 in  accordance with generally accepted government auditing standards.
section_title: Background on Selected Agencies and Their Training and Development Functions
paragraphs: The following summarizes key information on the five agencies included in  this review.  These summaries include information on the agencies’  missions, organizational structures, and training and development  functions.
subsections: section_title: U.S. Army Corps of Engineers
paragraphs: The U.S. Army Corps of Engineers (USACE), part of the Department of the  Army within the Department of Defense, is comprised of approximately  34,600 civilian and 650 military men and women.  USACE has a diverse  workforce consisting of military and civilian engineers, biologists,  geologists, hydrologists, natural resource managers, and other specialists  who work in engineering and environmental matters.  USACE’s mission  is to provide engineering services to the nation: (1) planning, designing,  building and operating water resources and other civil works projects, (2)  designing and managing the construction of military facilities for the Army  and Air Force, and (3) providing design and construction management  support for other Department of Defense and federal agencies.  USACE  headquarters office is located in Washington, D.C.
USACE’s Professional Development Support Center, located in Huntsville,  Alabama, serves as the center of learning and training for the agency.  The  training center manages and implements the Proponent-Sponsored  Engineer Corps Training program, which provides job-related training  through technical, professional, managerial, and leadership courses for  USACE and other government agencies.  USACE’s training center offers  more than 200 courses covering topics that support the agency’s mission.
section_title: U.S. Fish and Wildlife Service
paragraphs: The mission of the U.S. Fish and Wildlife Service (FWS), a component of  the Department of the Interior, is working with others to conserve, protect,  and enhance fish, wildlife, and plants and their habitats for the continuing  benefit of the American people.  FWS’s headquarters is located in  Washington D.C., while its field units are located throughout the United  States.  FWS employs more than 9,600 people and is supported by a  volunteer force of 29,000.  Nearly 90 percent of FWS employees work in  field locations.
The National Conservation Training Center, located in Shepherdstown,  West Virginia, is FWS’s training center and is responsible for training a wide  range of employees in the conservation community and serves as a  gathering place where conservation professionals from government,  nonprofit organizations, and corporations work toward common goals.
Training for FWS’s law enforcement personnel is primarily conducted  through the Federal Law Enforcement Training Center, an interagency law  enforcement training organization headquartered in Glynco, Georgia.
section_title: Internal Revenue Service
paragraphs: The Internal Revenue Service (IRS) is a branch of the U.S. Department of  the Treasury.  IRS’s mission is to provide America's taxpayers top quality  service by helping them understand and meet their tax responsibilities and  by applying the tax law with integrity and fairness to all.  IRS’s  organizational structure includes the following business units: four  operating divisions organized around four major customer segments (Wage  and Investment, Small Business/Self-Employed, Large and Mid-Size  Business, and Tax Exempt and Government Entities); four functional  divisions (National Taxpayer Advocate, Appeals, Criminal Investigation,  and Communications and Liaison); and two shared services/support  divisions (Agency-Wide Shared Services and Modernization and  Information Technology Services).  As of March 2003, IRS had about  116,300 employees.
IRS takes a decentralized approach to training and developing its  workforce.  Each business unit has an embedded human resources  component that provides advice and analysis on related policies and issues  and formulates strategies, procedures, and practices to address the unit's  human capital needs.  Learning and Education, one of eight major divisions  comprising IRS’s Office of Strategic Human Resources, provides guidance  and sets policy and standards on training and development for the agency’s  business units and headquarters offices.
section_title: U.S. Office of Personnel Management
paragraphs: The U.S. Office of Personnel Management (OPM), the federal government’s  human capital agency, provides human resources policy leadership,  technical advice and assistance, and products and services to federal  agencies, employees, annuitants, and job seekers.  It also oversees  governmentwide compensation and performance management systems,  and provides retirement, health benefit, and other insurance services to  federal employees, annuitants, other beneficiaries, and agencies.  In March  2003, OPM completed a major restructuring process through which it  consolidated various agency functions.  As of March 2003, OPM employed  approximately 3,500 people, many of them stationed in agency  headquarters in Washington, D.C.  OPM has a field presence in 16 major  U.S. cities as well as operating centers in Pennsylvania and Georgia.
OPM’s training and development efforts are largely decentralized to the  agency’s various program and staff offices.  The employee training and  development unit within the agency’s human capital management office is  responsible for setting overall strategy and for planning and implementing  agencywide training such as leadership development programs and various  mandatory training programs.  According to OPM, the agency’s newly  established Chief Human Capital Officer plays a significant role in advising  the OPM Director on overall employee training and development initiatives  and programs, as well as the establishment of the agency’s training budget.
section_title: Veterans Health Administration
paragraphs: The Veterans Health Administration (VHA), one of three major  administrations within the Department of Veterans Affairs (VA), is  responsible for providing primary care, specialized care, and related  medical and social support services to veterans through an integrated  health care system.  VHA administers its functions through a group of 21  regional network offices located around the United States.  As of March  2003, VHA employed about 203,500 people out of a total VA workforce of  about 225,000 employees.
VA takes a decentralized approach to training and development operations.   VA’s human resources office provides advice and guidance on training to  VHA and the other departmental components but delegates training and  development operations to each component.  VHA’s organization includes  the Employee Education System, which is an internal training consulting  group that provides educational services that support the workforce  development and continuing education needs for VHA employees.  This  internal consulting group of about 300 individuals primarily helps to assess  agency training needs at the national level, while VHA network offices and  medical centers take lead responsibility for assessing their own local  needs.  These internal training consultants are available to assist VHA  network offices and medical centers in designing, developing,  implementing, and evaluating training and development programs to meet  these local needs.
section_title: Core Characteristics of a Strategic Training and Development Process
paragraphs: The following summarizes the eight core characteristics that make a  training and development process effective and strategically focused on  achieving results.  We identified these core characteristics as part of our  recent work in developing an assessment guide to assist federal agencies in  evaluating their training and development efforts.
Strategic alignment. Clear linkages exist between the agency's  mission, goals, and culture and its training and development efforts. The  agency's mission and goals drive a strategic training and development  approach and help ensure that the agency takes full advantage of an  optimal mix of strategies to improve performance and enhance capacity  to meet new and emerging challenges.
Leadership commitment and communication. Agency leaders and  managers consistently demonstrate that they support and value  continuous learning, are receptive to and use feedback from employees  on developmental needs and training results, and set the expectation  that fair and effective training and development practices will improve  individual and organizational performance.
Stakeholder involvement. Agency stakeholders are involved  throughout the training and development process to help ensure that  different perspectives are taken into account and contribute to effective  training and development programs. Stakeholders' views are  incorporated in identifying needed performance enhancements,  developing and effectively implementing well-thought-out strategies,  and helping to conceptualize and use balanced measures that accurately  reflect the extent to which training and development efforts contribute  toward achieving results.
Accountability and recognition. Appropriate accountability  mechanisms, such as performance management systems, are in place to  hold managers and employees responsible for learning and working in  new ways. Appropriate rewards and incentives exist and are used fairly  and equitably to encourage innovation, reinforce changed behaviors,  and enhance performance.
Effective resource allocation. The agency provides an appropriate  level of funding and other tools and resources—along with external  expertise and assistance when needed—to ensure that its training and  development programs reflect the importance of its investment in  human capital to achieving its mission and goals.
Partnerships and learning from others. Coordination within and  among agencies achieves economies of scale and limits duplication of  efforts. In addition to benchmarking high-performing organizations,  these efforts allow an agency to keep abreast of current practices,  enhance efficiency, and increase the effectiveness of its training and  development programs.
Data quality assurance. The agency has established policies and  procedures that recognize and support the importance of quality data  and of evaluating the quality and effectiveness of training and  development efforts. It establishes valid measures and validated  systems to provide reliable and relevant information that is useful in  improving the agency's training and development efforts.
Continuous performance improvement. Agency practices and  policies foster a culture of continuous improvement and optimal  organizational performance regarding training and other activities.  Stakeholders rely on and use program performance information and  other data to assess and refine ongoing training and development  efforts; target new initiatives to improve performance; and design,  develop, and implement new approaches to train and develop  employees.
section_title: Comments from the Department of the Interior
paragraphs: The following is GAO’s comment on the Department of the Interior’s letter  dated January 23, 2004.
subsections: section_title: GAO Comment
paragraphs: We have clarified the mission statement of the Fish and Wildlife Service to  note its collaboration with others to accomplish its mission.
section_title: Comments from the Internal Revenue Service
section_title: Comments from the Office of Personnel Management
section_title: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Acknowledgments
paragraphs: In addition to the persons named above, K. Scott Derrick, Gerard Burke,  T.J. Thomson, and Thomas Davies, Jr. made key contributions to this  report.
section_title: GAO’s Mission
paragraphs: The General Accounting Office, the audit, evaluation and investigative arm of  Congress, exists to support Congress in meeting its constitutional responsibilities  and to help improve the performance and accountability of the federal government  for the American people. GAO examines the use of public funds; evaluates federal  programs and policies; and provides analyses, recommendations, and other  assistance to help Congress make informed oversight, policy, and funding  decisions. GAO’s commitment to good government is reflected in its core values of  accountability, integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents at no cost is  through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older  products. The Web site features a search engine to help you locate documents  using key words and phrases. You can print these documents in their entirety,  including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and  correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site  daily. The list contains links to the full-text document files. To have GAO e-mail this  list to you every afternoon, go to www.gao.gov and select “Subscribe to   e-mail alerts” under the “Order GAO Products” heading.
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Public Affairs