Tax Administration: IRS Needs Better Strategic Planning and Evaluation of Taxpayer Assistance Training
Released: 2005-08-11T00:00:00

Letter

Background

IRS provides tax law assistance to taxpayers through IRS’s toll-free telephone lines and tax law and return preparation assistance in person at IRS taxpayer assistance centers (formerly known as “walk-in sites”) nationwide, in addition to other means. The four taxpayer assistance programs we reviewed were: W&I Customer Accounts Services (CAS) general tax law assistance by telephone. Many taxpayers contact IRS by calling its toll-free telephone number which is operated and staffed by CAS. According to IRS fiscal year 2004 data, 3,420 CAS staff handled more than 8.7 million telephone calls from taxpayers with general, less complex tax law questions. We did not verify these IRS data or other IRS data on taxpayer assistance programs’ workload and staffing.

SB/SE Taxpayer Education Communication (TEC) complex tax law assistance by phone. According to IRS officials, during the 2005 filing season and in previous years, TEC staff supported telephone service by answering taxpayers’ questions on selected, more complex tax topics. When taxpayers called IRS’s toll-free number about these topics, W&I CAS staff recorded the taxpayers’ contact information and questions so that TEC staff could call the taxpayers back later to provide answers. According to IRS, in the 2004 filing season, TEC handled about 320,000 telephone calls for complex tax law assistance. According to IRS, TEC trained approximately 400 staff and used 272,212 staff hours to provide telephone assistance in 2004. In 2006 these calls will be handled by W&I CAS staff.

W&I Customer Assistance, Relationships, and Education (CARE) walk-in tax law assistance. According to IRS, about 1.4 million staff hours were devoted to providing walk-in assistance in fiscal year 2004. Also according to IRS, 1,654 staff were trained to provide tax law assistance in 2004. In 2004, there were about 7.7 million contacts with taxpayers at IRS’s approximately 400 taxpayer assistance centers. IRS did not have information on how many of these contacts were for tax law assistance. Many of the contacts were for other services, such as tax forms, publications, or accounts issues.

W&I Customer Assistance, Relationships, and Education (CARE) walk-in return preparation assistance. According to IRS data, IRS staff prepared 476,813 tax returns in fiscal year 2004. IRS did not have data on the number of staff that prepared tax returns at taxpayer assistance centers.

IRS does not have data on the amount of time its assistors spend annually being trained on tax law. However, all staff providing tax law assistance receive some training each year. IRS has continuing professional education and refresher training requirements for all its taxpayer assistance staff.

Tax law assistance staff receive a variety of training. According to an IRS official, the staff get primarily classroom training but also receive computer-based training. On-the-job training, managerial coaching, and workshops are also part of training. Taxpayer assistance staff receive training on technical tax law topics, how to use IRS systems and guidance to answer questions, and communication.

Responsibility for training and developing IRS’s tax law assistance staff is decentralized. IRS’s Human Capital Office provides guidance and sets policy and standards on training and development for the agency. W&I and SB/SE each have a human resources office with Learning and Education (L&E) staff who are assigned to the taxpayer assistance programs. L&E staff provide program staff advice and analysis on related policies and issues and formulate strategies, procedures, and practices to address the programs’ human capital needs, including training. Generally speaking, the taxpayer assistance program offices identify training needs, L&E staff work with program staff to develop and fund annual training plans, and the program offices administer training. According to IRS policy, L&E staff are responsible for evaluating training.

According to fiscal year 2004 IRS data, IRS invested about $7 million in training W&I CAS and CARE staff to provide taxpayer assistance, including such expenses as travel, supplies, contractor fees, and development costs. IRS could not separate these costs into amounts spent on tax law training and other topics. According to SB/SE TEC fiscal year 2004 data, training costs were $325,072 in student and instructor travel. However, the cost data IRS provided did not include the costs of assistors’ time associated with training. As with the staffing and workload data, we did not verify the accuracy of IRS’s training cost data.

In March 2004, we issued an assessment guide that introduced a framework for evaluating a federal agency’s training and development efforts. This assessment guide consists of a set of principles and key questions that federal agencies can use to ensure that their training and development investments are targeted strategically and are not wasted on efforts that are irrelevant, duplicative, or ineffective. As detailed in our assessment guide, the training and development process can loosely be segmented into four broad, interrelated components: (1) planning/front- end analysis, (2) design/development, (3) implementation, and (4) evaluation. Figure 1 depicts the general relationships between the four components, including the feedback loop between evaluation and planning. Planning and evaluation are highlighted because they are the focus of this report. Although these components can be discussed separately, they are not mutually exclusive and encompass subcomponents that may blend with one another. For instance, evaluation is part of the planning as organizations should reach agreement up front on how the success of training and development efforts will be assessed.

TIGTA is conducting a review that covers some aspects of the design and implementation of training.

Scope and Methodology

Our work examined the training and development of employees who provide tax law and return preparation assistance to taxpayers over the telephone and at walk-in centers, and covered both seasonal and full-time employees in IRS’s W&I and SB/SE divisions. Our assessment of IRS’s training and development program for taxpayer assistance employees was based on analyses of IRS data and interviews with IRS officials. We also obtained background information from sources outside IRS such as TIGTA, the IRS Oversight Board, and the IRS National Taxpayer Advocate.

To assess how IRS plans for and evaluates the training and development of IRS employees who provide tax law and return preparation assistance to taxpayers over the telephone and at walk-in centers, we used our guide for assessing training and development programs in the federal government as a framework.

We used the parts of the GAO guide dealing with the planning and evaluation components, along with supplemental guidance from the Office of Personnel Management, to identify the detailed strategic training and development criteria applicable to IRS and organized them into chronological phases. We focused our criteria on strategic planning and evaluation, that is, planning and evaluation intended to help achieve IRS’s accuracy goals. We developed 27 criteria, listed in appendices I and II. We also developed examples of evidence that would demonstrate conformance with each of the criteria. For example, for the criterion “conduct a knowledge and skills inventory to identify employees’ current skills and knowledge,” we looked for evidence such as a completed knowledge and skills surveys and proficiency tests (see app. I for the planning phase criteria and further evidence examples and app. II for the same information on evaluation.) We shared the criteria with officials in IRS’s Human Capital Office and the taxpayer assistance programs and assigned L&E staff. The officials from the Human Capital Office said the criteria are appropriate and consistent with their policy guidance. Some program and L&E officials expressed concerns about their need and ability to satisfy the criteria. Their concerns are discussed in the planning and evaluation sections of this report.

In applying the criteria to IRS, we collected documents describing IRS’s planning and evaluation processes for the four types of assistance. We also interviewed officials responsible for the taxpayer assistance programs in W&I CAS and CARE units and SB/SE TEC unit and associated L&E staff responsible for training planning and evaluation, to get more details where needed. We reviewed all the evidence and made a judgment about the extent to which IRS’s practices conformed to the criteria. We then discussed our initial assessments with IRS officials responsible for planning and evaluating the taxpayer assistance training and development programs who, for some of our preliminary assessments, provided additional written evidence for us to consider in making our final assessments. We then revised our initial assessments based on the evidence they provided. We made our assessments in two steps. First, an analyst reviewed all the evidence and made a judgment about the extent to which it conformed to the criteria. Then a second analyst independently reviewed the assessments. The evidence supporting our assessments is provided in appendices III through X.

Much of the information we relied on was descriptive. We determined that the information was reliable for our purposes. To the extent possible we corroborated interview evidence with documentation. Where not possible, the description is attributable to IRS officials. Where relevant we corroborated that policy guidance, such as Internal Revenue Manual guidance, was being implemented by collecting documentation and reports showing implementation. With respect to controls over databases, we reviewed documentation of the controls but did not assess their adequacy or test data in the databases.

We conducted our work at the Wage and Investment Division headquarters in Atlanta, Ga.; the Small Business/Self-Employed Division offices in Philadelphia, Pa.; and IRS headquarters Human Capital Office in Washington, D.C. from May 2004 through May 2005 in accordance with generally accepted government auditing standards.

Long-term Goals and Analyses of Needs and Impacts Could Improve IRS’s Planning for Training and Development

Summarizing table 1, IRS’s planning for taxpayer assistance training primarily focuses on meeting short-term needs, such as the challenge of training staff about tax law changes in preparation for the next year’s filing season. Planning could be enhanced by long-term goals and analyses of the relative importance of the factors that affect accuracy, other organizations’ experiences, and gaps between long-term needs and existing skills. Table 1 shows our assessments of planning for training by IRS’s four primary taxpayer assistance programs for each phase of the planning process: goals and priorities, information gathering and assessments, skills and knowledge analysis, and strategy development and selection. The evidence supporting our assessments is shown in appendices III through VI.

Goals and Priorities

In the goals and priorities phase, the four units were relatively strong in involving key stakeholders and communicating the importance of training. The programs involved key stakeholders in annual training planning decisions, shown by half-circles in table 1. However, they did not have long-term planning processes in which to involve stakeholders. All four programs had some upper-level management efforts to communicate to staff the importance of training and development in improving accuracy. CARE’s was a model of communicating to all levels of staff the importance of training. For example, CARE’s management guidance included managers responsible, communication vehicles, key dates, and the message to be conveyed. The other programs communicated with managers, but did not communicate more widely through the organization.

With respect to goals, IRS does not have long-term goals, as opposed to annual goals, for accuracy; nor do the four programs have goals for training and development or measures of the impact of training on accuracy. This observation about the lack of goals for accuracy is consistent with other recent reports where we cited a lack of IRS long-term goals, as well as reports and other assessments by OMB and TIGTA. Two of the programs had annual goals, shown by half-circles.

Some taxpayer assistance officials that we talked with questioned the need for long-term accuracy goals or training and development goals and measures. For example, W&I CAS telephone assistance officials said setting long-term accuracy goals is not necessary because their annual accuracy goals would move IRS toward improved performance. Program officials also said that since staff training is considered a part of doing business and is not managed as a program, training goals are unnecessary. Further, some said that, given the number of factors that affect accuracy, developing measures of training effectiveness would not be possible.

However, without long-term accuracy goals, IRS, Congress, and others are hampered in evaluating whether IRS is making satisfactory progress improving taxpayers’ service. As we said in a prior report on IRS’s telephone assistance program, a long-term, results-oriented goal would provide a meaningful sense of direction as well as a yardstick for measuring the results of operations and evaluating the extent of improvements resulting from changes in resources, new technology, or management of human capital. Similarly, goals and performance measures for training would provide direction and help measure progress. We recognize that collecting performance data can sometimes be challenging, as discussed in the evaluation section. One IRS taxpayer assistance program has been collecting performance data suitable for determining the effectiveness of training.

Information Gathering and Assessments

In this phase, the four programs determined and tracked selected factors that affect accuracy, but would benefit from a more complete analysis as well as benchmarking. IRS has identified, and tracked on an individual basis, selected factors that affect accuracy. One nontraining factor that IRS has tracked and analyzed is the use and quality of the Probe and Response Guide, a manual provided to taxpayer assistance staff intended to guide them in responding to taxpayers’ questions. IRS blamed decreases in accuracy in 2003 and 2004 on problems with the Probe and Response Guide. According to IRS officials, they used their analysis of the Probe and Response Guide to make the guidance more usable by taxpayer assistance staff. Officials attribute improvements in accuracy in 2005, at least in part, to this effort.

However, IRS has not conducted an analysis of the factors that affect accuracy, including training—shown by half-circles in Table 1. Specifically, IRS has not done an analysis to determine the relative importance of the various factors, including training, that affect accuracy. Determining the relative importance of the factors that affect accuracy is important because there are multiple factors. Some factors that affect accuracy are strategic and outside of IRS’s direct control such as tax law changes. Other factors are operational and subject to IRS control such as manuals, information systems, and training. Without an understanding of the relative importance each factor has on accuracy, it is difficult for IRS to make informed decisions about a strategy for improving accuracy, including training’s role in that strategy. Because of the purpose of the analysis and the number of factors involved, the analysis might not give a precise measure of each factor’s impact. Furthermore, such analyses may be conducted on an occasional basis.

None of the taxpayer assistance programs collected another type of information—best practices of other organizations learned through benchmarking. Some officials told us it would not be possible to benchmark training programs because IRS has a unique mission. However, the telephone taxpayer assistance programs have benchmarked other aspects of their operations such as performance measures. Also, as we have stated in another report, many processes that seem unique to the government actually have counterparts in the private sector. Looking at processes in dissimilar organizations can lead to rewarding improvements because it stimulates new thinking about traditional approaches to doing work. For example, in the report cited above, we noted that Xerox benchmarked L.L. Bean to improve order fulfillment.

Still another type of information useful for planning can be obtained from evaluations of training efforts. As we discuss in the background, evaluation can provide useful feedback about the impact of existing training to help plan improvements to training. Evaluation of IRS’s training efforts is the subject of our second section.

Skills and Knowledge Analysis

All four assistance taxpayer assistance programs conducted annual needs assessments. These assessments identified the knowledge and skills assistors needed for the following year. None of them had longer term analyses to project future needs. Determining skills and knowledge needed is challenging for IRS, especially when IRS must react quickly to tax law changes. For example, Congress passed a law in early January 2005 allowing taxpayers to deduct on either their 2004 or 2005 tax returns contributions made during January 2005 for relief of the victims of the December 2004 Indian Ocean Tsunami. According to an IRS official, IRS had to react to this change, which took place after the filing season had already begun, by quickly alerting staff and providing them the information necessary to assist taxpayers who had questions about the deduction.

However, none of the four programs had analyses of long-term needs, such as improved proficiency in doing research, which is the basis for the half- circles in table 1. Longer-term analyses of needs could help the four programs better plan strategies to meet future needs. Such planning could help ensure that the programs acquire, train, and retain the needed staff.

Three of the four programs had knowledge and skills inventories based on actual testing to determine employees’ proficiency and knowledge levels. TEC was the exception. TEC assumes the employees have an underlying technical foundation because of their background in IRS. However, in a recent TEC survey of 133 employees, the percentage of employees who self-reported having pre-existing skills or being fully proficient in several technical categories, such as depreciation, sale of property, and trusts and fiduciaries, ranged from 31 percent to 62 percent.

Three of the four programs performed gap analyses to determine the difference between current and needed skills and knowledge. Because of the above limitations in needs analyses, the gap analysis was necessarily annual not long term. Again, TEC was the exception. Because TEC had not done a needs analysis, it could not do a gap analysis.

Strategy Development and Selection

Weaknesses in analyses, described in the previous sections, hampered the four programs’ ability to develop criteria for when to use training, as opposed to other strategies, for filling knowledge and skills gaps, such as hiring and retention efforts. In addition, the four programs lacked information about the benefits and costs of their training efforts. As noted in the background section, IRS trains several thousand tax law assistance staff annually but does not have data on the cost of this significant effort. Without an understanding of the usefulness of other strategies, and without an understanding of the benefits and costs of training, IRS lacked information that would be useful for making resource allocation decisions. This matters because the resources devoted to training are significant. With better information to select and develop a strategy, IRS might be able to improve accuracy and save resources.

Evaluating the Impact of Training on Accuracy Could Provide a Basis for Future Improvements

Given the importance of accurate answers to taxpayers’ questions and the resources spent on training, the four assistance programs would benefit from more sophisticated evaluations of the effectiveness of training. The four programs all conduct some evaluations of their training efforts. However, only TEC attempted an evaluation of the impact of training on accuracy. In table 2, the lack of evaluation plans in the other three programs had a cascading effect that left the programs generally unable to fully satisfy our assessment criteria in subsequent evaluation phases—data collection, data analysis, and application of evaluation results. Appendices VII through X show the evidence supporting our assessments in table 2.

Evaluation Plan Phase

IRS adopted a four-level model based on the widely accepted Kirkpatrick model for evaluating training. Under this model, the sophistication of analysis increases as the numerical level of analysis increases: Level 1 — Reaction. The goal is to measure participants’ reaction to training, usually through questionnaires.

Level 2 – Learning. The goal is to determine what the training participants learned through various kinds of tests administered immediately after the training is completed.

Level 3 – Behavior. The goal is to determine if the job performance of the training participants changed in the aftermath of the training. The most common means for making this determination is the administration of a survey of trained staff and their supervisors 3 months, on average, after training is complete.

Level 4 – Results. The goal is to determine if the training led to the desired results, in this case, improved accuracy of taxpayer assistance.

Options for level 4 analyses might include statistical correlations between training and accuracy and controlled experiments where some staff receive new training and others do not. Such ROI analyses are not a part of IRS’s evaluation approach. This level is sometimes split into two levels with the fifth level—often referred to as return on investment (ROI)— representing a comparison of costs and benefits quantified in dollars.

Not all training and development programs are suitable for evaluations of their effect on organizational results. The difficulty and costs of analyzing the impact of training on accuracy need to be weighed against the benefits of such an evaluation.

However, the more significant the activity targeted by the training and development program, the greater the need for level 4 analysis. Factors to consider when deciding the appropriate level of evaluation include estimated costs of the training effort, size of the training audience, management interest, program visibility, and anticipated life span of the effort. As noted in the background, IRS devotes significant resources to assisting taxpayers and training assistance staff. Congress has also expressed great interest in improving taxpayer service, such as accuracy.

For example, much of the focus of the Internal Revenue Service Reform and Restructuring Act of 1998 was on in improving taxpayer service.

IRS’s Human Capital Office recognizes the need to do level 4 evaluations. The office has a policy statement stating that level 4 evaluations should be done for all mission-critical training.

Despite the large investment of resources, significant congressional attention, and Human Capital Office guidance, IRS officials involved in three of the four taxpayer assistance training programs had not agreed on whether to conduct a level 4 evaluation. They conducted analyses only at lower levels, usually levels 1 or 2. By contrast, officials at L&E and TEC agreed to conduct a pilot test of a level 4 analysis in 2004 for more complex tax law questions answered by telephone. Although the pilot test was not successful in 2004—for reasons discussed below in the data analysis phase—L&E and TEC intend to use the data collection and analysis plan created for 2004 to conduct a similar evaluation in 2005.

Data Collection Phase

For three of the four taxpayer assistance programs, as shown in table 2, the lack of level 4 analyses had a cascading effect that left the programs unable to fully satisfy our assessment criteria in the subsequent evaluation phases. All four programs had controls in place to help ensure systematic, timely data collection for level 1 and level 3 evaluations. TEC also had such controls in place for its level 4 evaluation. However, the database used to store level 2 data—the Administrative Corporate Education System (ACES)—is no longer in place, and there is no replacement system planned. With the exception of TEC, the programs did not attempt to collect level 4 data. Database controls and data collection plans for TEC’s level 4 pilot evaluation helped ensure the systematic and timely collection of data.

All four units had controls in place to help ensure that level 1 and level 3 data were accurate, valid, reliable, and complete. Because of the lack of a level 2 database, none of the four units had controls in place for that data.

TEC was concerned that the accuracy data, which were based on test telephone calls, did not reflect the types of calls that taxpayers actually made. TEC and IRS’s quality review staff are working together to try to address these concerns.

Data Analysis Phase

All four assistance programs conducted some analyses of their training efforts and analyzed stakeholders’ assessments of training to identify potential improvements to individual courses. A level 1 analysis, usually surveys of trainees’ opinions about a course, was conducted for all courses. The officials also said level 2 analyses, often testing, are generally done while level 3 analyses are infrequently done.

None of the four assistance programs successfully completed a level 4 evaluation of the impact of training on accuracy, compared benefits to costs, benchmarked training evaluation, or analyzed stakeholders’ assessments of the impact of training on accuracy. The 2004 TEC level 4 pilot test was not completed because the level of proficiency of TEC staff in the five technical categories reviewed was insufficient.

Application of Results Phase

All the business units applied the results of their data analysis to individual training courses, not their training efforts as a whole. For example, they used the results of their level 1 and level 3 analyses to make improvements in individual training courses and to identify training needs for the upcoming year. However, because there had been no successful evaluations of the impact of training on accuracy, evaluation results could not be used to plan a strategy to improve accuracy.

Conclusions

Training has the potential to improve the service received from IRS by millions of taxpayers. To its credit, IRS has planning processes in place to address the challenges of training staff for each year’s filing season. The challenges include training staff to answer questions about annual tax law changes—changes that are often very complex. Although we do not know how much training has contributed, IRS’s taxpayer assistance accuracy has improved in recent years. On the other hand, IRS’s current level of accuracy remains a concern, especially the accuracy of walk-in assistance and return preparation. In addition, IRS devotes significant resources to training its tax law assistance staff.

A more strategic approach to planning and evaluation would have several benefits. Strategic planning could help managers better understand the extent to which training, as opposed to other factors that affect accuracy, could be used to improve accuracy. Evaluations of training’s impact on accuracy could help managers better understand which specific training techniques are effective and which are not. While potentially difficult to design and costly to conduct, IRS’s Human Capital Office has policy guidance urging more such analyses. In addition, one taxpayer assistance program is pilot testing such an analysis. Gap analyses, evaluations, and cost-benefit comparisons might also contribute to providing training at lower cost by distinguishing between effective and ineffective training.

Recommendations

We recommend that the Commissioner of Internal Revenue take appropriate action to ensure that IRS’s planning and evaluations of its taxpayer assistance training and development efforts better conform to guidance for strategically planning and evaluating the training efforts.

Specifically, in the area of planning, we recommend that IRS: Establish a long-term goal for the accuracy of taxpayer assistance. Establish goals and measures for training and development logically linked to accuracy.

Determine and track the relative importance of the various factors, including training, that affect accuracy.

Benchmark training and development programs against high-performing organizations.

Conduct skills and knowledge gap analyses for all taxpayer assistance programs, to include identifying and comparing current skills to long-term skill needs.

Consider costs, benefits, ways to mitigate risks, and the appropriate level of investment for training and development efforts.

In the area of evaluation, we recommend that IRS continue to pursue the level 4 pilot in TEC and, if that analysis is shown to be feasible, conduct level 4 evaluations for its other taxpayer assistance training and development programs. The evaluations should include the following: an analysis of the feasibility and cost effectiveness of alternative level 4 methodologies and a data collection and analysis plan, a comparison of the accuracy benefits to the costs of the training, benchmarking of the analytical methods and the results of the data analysis against high-performing organizations, and an analysis of stakeholder assessments of the impact of training on accuracy.

We also recommend that IRS replace the defunct ACES database, which had been used to store level 2 data, with another database for this purpose.

Agency Comments and Our Evaluation

The Commissioner of Internal Revenue provided written comments on a draft of this report in a letter dated July 6, 2005 (see app. XII). He said the report offers valuable insight, is timely, and has been shared with the project manager for IRS’s recently-initiated effort to reengineer its learning and education processes. Of the eight recommendations we made, the Commissioner agreed with five recommendations: (1) establish long-term accuracy goals, (2) determine and track the relative importance of factors that affect accuracy, (3) benchmark training and development programs, (4) conduct level 4 evaluations, and (5) replace IRS’s system for storing level 2 evaluation data.

The Commissioner partially responded to the remaining three recommendations. In commenting on our recommendation to establish training goals and measures linked to accuracy, the Commissioner focused on evaluation. Although related to after-the-fact evaluation, setting clear training goals and measures to ascertain progress toward those goals— consistent with agency goals which, in the case of taxpayer assistance, would include accuracy—is an important up-front planning step by which key stakeholders should agree on what training success is and how it will be assessed.

The Commissioner said that IRS recognizes the value of conducting skills and knowledge gap analyses. He summarized the programs’ efforts to identify short-term skills needs, which we recognized in this report. However, he did not identify how short-term gap analyses would be conducted for the program for responding to more complex tax questions. Nor did the Commissioner discuss gap analyses to identify long-term skills needed to reach accuracy goals. Failing to conduct gap analyses, including analyses of strategic changes—such as economic, technological, and demographic developments—can hinder performance and the development of strategies that integrate new capabilities and provide flexibility to meet new challenges and improve service.

In commenting on our recommendation that IRS consider the costs and benefits of training efforts, the Commissioner’s comments did not specifically mention benefits and costs but did mention unfunded needs. Given the resources dedicated to training staff and providing taxpayer assistance, and the impact that assistance accuracy can have on taxpayers, the taxpayer assistance programs would benefit from analytically-based assurance that training efforts focus in a cost beneficial way on achieving accuracy goals.

As we agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution of it until 30 days from the date of this letter. We will then send copies of this report to the Chairman of the Senate Committee on Finance, the Chairman and Ranking Minority Member of the House Committee on Ways and Means, and the Chairman and Ranking Minority Member, Subcommittee on Oversight, House Committee on Ways and Means. We will also send copies to the Secretary of the Treasury; the Commissioner of Internal Revenue; the Director, Office of Management and Budget; and other interested parties. We will also make copies available to others on request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-9110 or whitej@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix XIII.

Appendix I: What GAO Looked For: Examples of Planning Practices That Would Conform to Strategic Guidance for Training

Establish quantitative long-term accuracy goals that link to IRS’s strategic goals

As part of its process for deciding which training and development programs to implement, steps to ensure efforts will target needed improvements and enhance needed skills, which would yield such items as A logical or explicit link between the training and development programs offered and (1) the knowledge and skills identified in the skill needs assessment or (2) the strategic and operational factors identified as affecting accuracy (to the extent that the needs and the identified factors could be addressed through training)

Appendix II: What GAO Looked For: Examples of Evaluation Practices That Would Conform to Strategic Guidance

Establish an overall approach for evaluating the impact of training on accuracy (a level 4 evaluation)

Steps to ensure that analyses will help determine the impact of training and development on accuracy performance, such as Analyses have considered and/or used an array of approaches (both qualitative and quantitative)

Analyses that separate training from other strategic and operational factors that might affect accuracy (to the extent that it is cost-effective and/or feasible)

Appendix III: Assessments of Planning: Less Complex Tax Law Telephone Service by W&I CAS

Establish quantitative long-term accuracy goals that link to IRS’s strategic goals

CAS reported an annual accuracy goal linked to IRS’s strategic goal of improving taxpayer service. The accuracy goal was not long-term. CAS had accuracy projections through 2010, but according to IRS officials, these projections were only internal and subject to change. In addition, OMB stated that IRS did not have long-term goals.

CAS involved stakeholders in its annual planning decisions. For example, CAS sought input from employees and managers on training priorities and needed changes to training. However, CAS did not have a long-term training planning process for involving key stakeholders in such strategic issues as establishing training program goals and long-term employee developmental needs.

CAS had no such goals or measures.

CAS management had efforts that conveyed the importance of training, such as an annual letter sent to field office directors that referred to the need for specific types of training. However, management did not communicate more widely throughout the organization the strategic value of training and development and its importance in achieving long-term accuracy goals.

CAS collected and analyzed accuracy data by using a quality review system to identify operational factors affecting accuracy, such as the proper use of IRS guidance to assistors, and the use of a pareto analysis to determine the correlation between the causes and effects of key errors. However, CAS did not analyze and track the strategic factors such as tax law changes. As a result, CAS did not have information on the impact of training on accuracy, while holding other factors such as the quality of assistors’ guidance or tax law changes, constant.

For our assessment, see “Application of Evaluation Results” phase in appendix VII.

CAS had not benchmarked its training and development program. IRS had benchmarked other nontraining practices related to customer service such as performance measures.

It was too early for an assessment because the Enterprise Learning Management System (ELMS), which would be CAS’s primary system used for providing human resource information, is being implemented in fiscal year 2005.

CAS identified the knowledge and skills assistors needed for the following year. For example, CAS projected the number of questions expected by tax topic and took into account recent tax law changes. However, CAS had not identified the specific knowledge and skills needed to achieve long-term accuracy goals.

CAS had an inventory of assistors’ current knowledge and skills based on testing.

CAS did an annual gap analysis, but because CAS had not performed a needs assessment for a period that extended beyond the next year, CAS was unable to determine the differences between current skills and knowledge and skills and knowledge needed in the future.

CAS had no criteria to guide decisions on using training and development strategies, as opposed to other strategies, such as improving guidance or hiring practices, to address skills and knowledge gaps.

CAS had practices such as employee testing, and studies conducted to identify the most frequent errors made on an annual basis, that allowed CAS to link training efforts to identified short-term skills and knowledge gaps. However, because of limitations in long-term gap analyses, CAS had limited ability to ensure training and development efforts were linked to long-term needed skills and knowledge.

CAS did not have these analyses.

For our assessment, see “Evaluation Plan” phase in appendix VII.

CAS had efforts to anticipate and react to potential short-term changes that could affect training, such as tax law changes or lack of experienced assistors due to staff turnover. However, CAS did not have a training planning process to identify potential long-term changes such as technological innovations or changes in the economy that might impact training and development.

Appendix IV: Assessments of Planning: More Complex Tax Law Telephone Service by SB/SE1 TEC

Establish quantitative long-term accuracy goals that link to IRS’s strategic goals

TEC did not have such goals.

TEC involved stakeholders in its annual training planning. For example, TEC surveyed employees to obtain input on training needs for the upcoming year. However, TEC did not have a long- term training planning process for involving key stakeholders in such strategic issues as establishing training program goals and long-term employee developmental needs.

TEC had no such goals or measures.

TEC management had efforts that conveyed the importance of training, such as an annual letter sent to field office directors that referred to the need for specific types of training. However, management did not communicate more widely throughout the organization the strategic value of training and development and its importance in achieving long-term accuracy goals.

TEC collected and analyzed accuracy data to identify factors, such as an analysis of quality review system data. However, TEC did not analyze and track the strategic factors such as changes to tax law. As a result, TEC did not have information on the impact of training on accuracy, while holding other factors such as the quality of assistors’ guidance or tax law changes, constant.

For our assessment, see “Application of Evaluation Results” phase in appendix VIII.

TEC had not benchmarked its training and development program.

It was too early for an assessment because the Enterprise Learning Management System (ELMS), which would be TEC’s primary system used for providing human resource information, is being implemented in fiscal year 2005.

Annually, TEC identified the types of tax law topics it would be handling and identified the knowledge and skills assistors needed. This process included taking account of recent tax law changes. However, TEC had not identified the specific knowledge and skills needed to achieve long-term accuracy goals.

In 2006, W&I will be handling these calls and training their staff to provide answers to these more complex tax law questions.

TEC did not conduct an inventory of its employees’ current skills and knowledge. Instead, TEC assumed that its employees had an underlying technical foundation because of their prior IRS experience, and the annual training they received was to supplement previously acquired skills.

Because TEC had not done a knowledge and skills inventory, a gap analysis could not be done.

TEC had no criteria to guide decisions on using training and development strategies, as opposed to other strategies such as improving guidance or hiring practices, to address skills and knowledge gaps.

In 2004, TEC matched its training courses to the general skills and knowledge assistors needed. However, because of limitations in long-term gap analyses, TEC had limited ability to ensure training and development efforts were linked to long-term needed skills and knowledge.

TEC did not have these analyses.

For our assessment, see “Evaluation Plan” phase in appendix VIIl.

TEC had informal processes for anticipating and reacting to potential short-term changes that could affect training. However, TEC did not have a training planning process to identify potential long-term changes such as technological innovations or changes in the economy that might impact training and development.

Appendix V: Assessments of Planning: Tax Law Questions Walk-In Service by W&I CARE

Appendix V: Assessments of Planning: Tax Law Questions Walk-In Service by W&I CARE CARE reported an annual accuracy goal linked to IRS’s strategic goal of improving taxpayer service. The goal was not long-term. CARE had accuracy projections through 2010, but according to IRS officials, these projections were internal and subject to change. In addition, OMB stated that IRS did not have long-term goals.

CARE involved stakeholders in its annual planning. For example, human capital training staff developed an annual training plan based on needs identified by program staff. However, CARE did not have a long-term training planning process in which to involve stakeholders in such strategic issues as identifying measures to assess training progress or identifying long-term employee developmental needs.

CARE had no such goals or measures.

CARE implemented a plan for communicating to various levels of staff, including front-line employees, management’s vision for training. The plan included the vehicle and product to be used to communicate this message, along with key dates activities should occur to deliver the message of how training and development plays a role in providing quality taxpayer service and enhancing employee development to achieve future goals.

CARE collected and analyzed data, including quality review system data, to identify operational factors affecting accuracy to identify the most frequent errors. CARE also had studies to determine why those errors occurred, such as why staff did not properly use IRS guidance to answer questions. However, CARE did not analyze and track the strategic factors such as changes to the tax law or attrition. As a result, CARE did not have information on the impact these factors had on accuracy holding other factors such as changes in the tax law or change in guidance constant. ___ For our assessment see the “Application of evaluation results” phase in appendix IX.

CARE had not benchmarked its training and development program. IRS had benchmarked other nontraining practices related to customer service such as performance measures.

It was too early for an assessment because the Enterprise Learning Management System (ELMS), which would be CARE’s primary system used for providing human resource information, is being implemented in fiscal year 2005.

CARE had identified the knowledge and skills assistors needed for the following year. One example is ensuring that assistors have the skills and knowledge needed to respond to questions about recent tax law changes. However, CARE had not identified the specific knowledge and skills needed to achieve long-term accuracy goals.

CARE had an inventory of assistors’ current knowledge and skills based on a prescreening testing process.

CARE did an annual gap analysis, but because CARE had not performed a needs assessment for a period that extended beyond the next year, it had not performed a gap analysis to determine the differences between current skills and knowledge and skills and knowledge needed in the future.

CARE had no criteria to guide decisions on using training and development strategies, as opposed to other strategies, such as improving guidance, or hiring practices to address skill gaps.

CARE has practices such as employee testing, and studies conducted to identify the most frequent errors made on an annual basis, that allow CARE to link training efforts to identified short-term skills and knowledge gaps.

However because of limitations in long term gap analyses, staff have limited ability to ensure training and development efforts are linked to long term needed skills and knowledge.

CARE did not have these analyses.

For our assessment, see “Evaluation Plan” phase in appendix IX.

CARE had efforts to anticipate and react to potential short-term changes that could affect training, such as tax law changes or lack of experienced assistors due to staff turnover. However, CARE did not have a training planning process to identify potential long-term changes, such as technological innovations or changes in the economy, that might impact training and development.

Appendix VI: Assessments of Planning: Return Preparation Walk-In Service by W&I CARE

Establish quantitative long-term accuracy goals that link to IRS’s strategic goals

CARE did not have quantitative long-term accuracy goals for return preparation. Fiscal year 2005 will be the baseline year for a new measure of tax preparation accuracy, so therefore there are no annual or long-term goals.

CARE involves stakeholders in annual planning. For example, human capital training staff developed an annual training plan based on needs identified by program staff. However, CARE did not have a long-term training planning process in which to involve stakeholders in such strategic issues as identifying measures to assess training progress or identifying long-term employee developmental needs.

CARE had no such goals or measures.

CARE collected and analyzed accuracy data, including quality review system data, to identify operational factors affecting accuracy to identify the most frequent errors. CARE also had studies to determine why those errors occurred, such as why staff did not properly use IRS guidance to answer questions. However, CARE did not analyze and track the strategic factors such as changes to the tax law or attrition. As a result, CARE did not have information on the impact these factors will have on accuracy holding other factors such as changes in the tax law or change in guidance constant.

For our assessment see the “Application of evaluation results” phase in appendix X.

CARE had not benchmarked its training and development program. IRS had benchmarked other nontraining practices related to customer service such as performance measures.

It was too early for an assessment because the Enterprise Learning Management System (ELMS), which would be CARE’s primary system used for providing human resource information, is being implemented in fiscal year 2005.

CARE identified the knowledge and skills assistors needed for the following year. One example is ensuring that assistors have the skills and knowledge needed to respond to questions about recent tax law changes. However, CARE had not identified the specific knowledge and skills enhancements needed to achieve long-term accuracy goals.

CARE had an inventory of assistors’ current knowledge and skills based on a prescreening testing process.

CARE did an annual gap analysis, but because CARE had not performed a needs assessment for a period that extended beyond the next year, it had not performed a gap analysis to determine the differences between current skills and knowledge and skills and knowledge needed in the future.

CARE had no criteria to guide decisions on using training and development strategies, as opposed to other strategies, such as improving guidance, or hiring practices to address skill gaps.

CARE had practices such as employee testing, and studies conducted to identify the most frequent errors made on an annual basis, that allowed CARE to link training efforts to identified short-term skills and knowledge gaps.

However, because of limitations in long-term gap analyses, CARE had limited ability to ensure training and development efforts were linked to long-term needed skills and knowledge.

CARE did not have these analyses.

For our assessment, see “Evaluation Plan” phase in appendix X.

CARE had efforts to anticipate and react to potential short-term changes that could affect training, such as tax law changes or lack of experienced assistors due to staff turnover. However, CARE did not have a training process to identify potential long-term changes such as technological innovations or changes in the economy that might impact training and development.

Appendix VII: Assessments of Evaluation: Less Complex Tax Law Telephone Service by W&I CARE

Establish an overall approach for evaluating the impact of training on accuracy (a level 4 evaluation)

L&E had adopted a four-level model for evaluating training, the Evaluation Monitoring System-Integrated Training Evaluation and Measurement Services (EMS-ITEMS), which was based on the widely- accepted Kirkpatrick model. IRS’s Human Capital Office L&E officials concluded that level 4 evaluations were appropriate. However, CAS and L&E officials had not agreed on whether to conduct level 4 evaluations. In addition, L&E officials had not documented an analysis of what level of evaluation was appropriate.

Because L&E and CAS officials had not agreed to do a level 4 evaluation, L&E had not analyzed the feasibility and cost effectiveness of alternative level 4 methodologies.

Because L&E and CAS officials had not agreed to do a level 4 evaluation, L&E did not have a level 4 data analysis plan. However, EMS-ITEMS did provide general guidance for different levels of evaluation.

EMS-ITEMS had controls to help ensure systematic and timely collection of data for level 1 and 3 evaluations. EMS-ITEMS had guidance on the collection of data including responsible parties and timing. However, the Administrative Corporate Education System (ACES) database used to collect level 2 data was no longer in place, and as a result, there was no system to consistently collect and store level 2 data. In addition, because L&E and CAS officials had not agreed to do a level 4 evaluation, L&E had not collected the data necessary to do a level 4 evaluation.

EMS-ITEMS had level 1 and some level 3 data and the system had controls in place to help ensure that data were accurate, valid, and reliable. However, because L&E and CAS had not agreed to do a level 4 evaluation, L&E did not collect level 4 data. The ACES database used to collect level 2 data no longer exists and there is no replacement system planned.

L&E conducted some level 1, 2, and 3 evaluations of participants’ opinions, learning, and subsequent job performance. For example, L&E analyzed course evaluations, tests, and supervisory evaluations after employees completed coursework to identify needed improvements to training. However, L&E had done no level 4 evaluations.

Because L&E and CAS had not agreed to do a level 4 evaluation, L&E did not do a comparison of benefits to cost.

None.

L&E and CAS analyzed stakeholder assessments of training to identify needed changes. For example, management councils and focus groups were used to determine if changes were needed to improve course material, training environment, timing, and objectives of training. However, because L&E and CAS had not agreed to do a level 4 evaluation, there were no efforts to collect and analyze assessments from internal and external stakeholders to assess training in terms of its impact on accuracy.

Evaluations were for individual courses, not for the program as a whole.

L&E and CAS reported using evaluation results in planning to make improvements to training courses or to identify training needs for the upcoming year. L&E and CAS used the available level 1 through 3 evaluations to make planned improvements on individual courses. However, because L&E and CAS had not agreed to do a level 4 evaluation or cost-benefit comparisons, L&E and CAS’s ability to make informed decisions on improving training to improve accuracy was limited.

Appendix VIII: Assessments of Evaluation: More Complex Tax Law Telephone Service by SB/SE TEC

Establish an overall approach for evaluating the impact of training on accuracy (a level 4 evaluation)

L&E had adopted a four-level model for evaluating training (EMS-ITEMS) that was based on the widely accepted Kirkpatrick model. In 2004, L&E and TEC planned a pilot test of a level 4 evaluation. L&E and TEC officials stated in an official document that they intend to conduct a level 4 evaluation in 2005.

L&E had not analyzed the feasibility and cost effectiveness of alternative methodologies.

L&E had a plan to conduct a level 4 evaluation in 2004. L&E planned to conduct a level 4 evaluation in 2005 similar to that done in 2004. Officials said they would use the 2004 plan as the basis for the 2005 evaluation effort.

L&E had controls to ensure data were collected systematically and in a timely manner. Data collection practices for all four levels of evaluation in the level 4 pilot test evaluation included specific steps to be accomplished, status updates, and due dates. In addition, for the 2005 pilot, data on TEC accuracy in five technical categories and staff proficiency in those technical categories were available before the end of 2004.

In 2004, TEC was concerned that the accuracy data in the five technical categories, based on test calls, did not reflect the types of calls taxpayers actually made. TEC and IRS’s quality review staff subsequently worked together in an effort to fix the problem. EMS-ITEMS had level 1 and some level 3 data, and the system had controls in place to ensure that data were accurate, valid, and complete. However, the database used to collect level 2 data no longer existed, and there was no replacement system planned.

In the 2004 pilot level 4 evaluation, L&E did an analysis but was unable to assess the impact of training on accuracy, because the level of proficiency of TEC staff in the five technical categories (capital gains, depreciation and sale of business property, rentals, trust and fiduciaries, and international and alien) was insufficient. L&E decided not to measure the impact of training on accuracy until 80 percent of the employees were proficient in the five technical categories. In a survey of 133 TEC employees, the percent reported having preexisting skills or being fully proficient in the five technical categories ranged from 31 to 62 percent.

Comparing the value of training to the costs of training was to be the third phase of the uncompleted 2004 pilot test.

None.

L&E and TEC analyzed stakeholder assessments of training to identify needed changes. For example, TEC used information from employee surveys to target training materials on the types of calls assistors reported receiving. However, they did not successfully collect and analyze assessments from internal and external stakeholders to assess training in terms of its impact on accuracy.

As discussed above, although L&E attempted to do a pilot test of a level 4 evaluation of the training program as a whole, the evaluation was unsuccessful.

L&E and TEC reported using evaluation results in planning to make improvements to training courses or to identify training needs for the upcoming year. However, because there was no successful level 4 evaluation or benefit cost comparison, L&E and TEC’s ability to make informed decisions on improving training to improve accuracy was limited.

Appendix IX: Assessments of Evaluation: Tax Law Walk-in Service by W&I CARE

Establish an overall approach for evaluating the impact of training on accuracy (a level 4 evaluation)

L&E had adopted a four-level model for evaluating training, the Evaluation Monitoring System-Integrated Training Evaluation and Measurement Services (EMS-ITEMS), based on the widely-accepted Kirkpatrick model. IRS’s Human Capital Office L&E officials concluded that level 4 evaluations were appropriate. However, CARE and L&E officials had not agreed on whether to conduct level 4 evaluations. In addition, L&E officials had not documented an analysis of what level of evaluation was appropriate.

Because L&E and CARE officials had not agreed to do a level 4 evaluation, L&E had not analyzed the feasibility and cost effectiveness of alternative level 4 methodologies. In addition, L&E and CARE officials did not look at methodologies specific to questions on tax law assistance. The approach for training and evaluating staff who provided assistance by answering walk-in customers’ tax law questions and staff preparing returns were the same.

Because L&E and CARE officials had not agreed to do a level 4 evaluation, L&E did not have a level 4 data analysis plan. However, EMS-ITEMS did provide general guidance for different levels of evaluation.

EMS-ITEMS had controls to help ensure systematic and timely collection of data for level 1 and 3 evaluations. EMS-ITEMS has guidance on the collection of data including responsible parties and timing. However, the Administrative Corporate Education System (ACES) database used to collect level 2 data was no longer in place, and as a result, there was no system to consistently collect and store level 2 data. In addition, because L&E and CARE officials had not agreed to do a level 4 evaluation, L&E had not collected the data necessary to do a level 4 evaluation.

EMS-ITEMS had level 1 and some level 3 data and the system had controls in place to help ensure that data were accurate, valid, and reliable. However, because L&E and CARE had not agreed to do a level 4 evaluation, L&E did not collect level 4 data. The ACES database used to collect level 2 data no longer exists and there was no replacement system planned.

L&E conducted some level 1, 2, and 3 evaluations of participants’ opinions, learning, and subsequent job performance. For example, L&E analyzed course evaluations, tests, and supervisory evaluations after employees completed coursework to identify needed improvements to training. However, L&E had done no level 4 evaluations. In addition, there was no distinction between analysis of walk-in tax law and return preparation assistance.

Because L&E and CARE had not agreed to do a level 4 evaluation, L&E did not do a comparison of benefits to cost.

None.

L&E and CARE analyzed stakeholder assessments of training to identify needed changes. For example, focus groups were used to determine if changes were needed to improve course material, training environment, timing, and objectives of training. However, because L&E and CARE had not agreed to do a level 4 evaluation, there were no efforts to collect and analyze assessments from internal and external stakeholders to assess training in terms of its impact on accuracy.

Evaluations were for individual courses, not for the program as a whole.

L&E and CARE reported using evaluation results in planning to make improvements to training courses or to identify training needs for the upcoming year. L&E and CARE used the available level 1 through 3 evaluations to make planned improvements on individual courses. However, because L&E and CARE had not agreed to do a level 4 evaluation or cost-benefit comparisons, L&E and CARE’s ability to make informed decisions on improving training to improve accuracy was limited.

Appendix X: Assessments of Evaluation: Return Preparation Walk-in Service by W&I CARE

Establish an overall approach for evaluating the impact of training on accuracy (a level 4 evaluation)

L&E has adopted a four-level model for evaluating training, the Evaluation Monitoring System-Integrated Training Evaluation and Measurement Services (EMS-ITEMS), based on the widely-accepted Kirkpatrick model. IRS’s Human Capital Office L&E officials had concluded that level 4 evaluations were appropriate. However, CARE and L&E officials had not agreed on whether to conduct a level 4 evaluation. In addition, L&E officials had not documented an analysis of what level of evaluation was appropriate.

Because L&E and CARE officials had not agreed to do a level 4 evaluation, L&E had not analyzed the feasibility and cost effectiveness of alternative level 4 methodologies. In addition, L&E and CARE officials did not look at methodologies specific to questions on return preparation. The approaches for training and evaluating staff that provided assistance by preparing returns and staff answering walk-in customers’ tax law questions were the same.

Because L&E and CARE officials had not agreed to do a level 4 evaluation, L&E did not have a level 4 data analysis plan. However, EMS-ITEMS did provide general guidance for different levels of evaluation.

EMS-ITEMS had controls to help ensure systematic and timely collection of data for level 1 and 3 evaluations. EMS-ITEMS had guidance on the collection of data including responsible parties and timing. However, the Administrative Corporate Education System (ACES) database used to collect level 2 data was no longer in place, and as a result, there was no system to consistently collect and store level 2 data. In addition, because L&E and CARE officials had not agreed to do a level 4 evaluation, L&E had not collected the data necessary to do a level 4 evaluation.

EMS-ITEMS had level 1 and some level 3 data and the system had controls in place to help ensure that data were accurate, valid, and reliable. However, because L&E and CARE had not agreed to do a level 4 evaluation, L&E did not collect level 4 data. The ACES database used to collect level 2 data no longer existed and there was no replacement system planned.

L&E conducted some level 1, 2, and 3 evaluations of participants’ opinions, learning, and subsequent job performance. For example, L&E analyzed course evaluations, tests, and supervisory evaluations after employees completed coursework to identify needed improvements to training. However, L&E had done no level 4 evaluations. In addition, there was no distinction between analysis of walk-in tax law and return preparation assistance.

Because L&E and CARE had not agreed to do a level 4 evaluation, L&E did not do a comparison of benefits to cost.

None.

Analyze internal and external stakeholders’ assessments of training to include impact on accuracy

L&E and CARE analyzed stakeholder assessments of training to identify needed changes. For example, focus groups were used to determine if changes were needed to improve course material, training environment, timing, and objectives of training. However, because L&E and CARE had not agreed to do a level 4 evaluation, there were no efforts to collect and analyze assessments from internal and external stakeholders to assess training in terms of its impact on accuracy.

Evaluations were for individual courses, not for the program as a whole.

L&E and CARE reported using evaluation results in planning to make improvements to training courses or to identify training needs for the upcoming year. L&E and CARE used the available levels 1 through 3 evaluations to make planned improvements on individual courses. However, because L&E and CARE had not agreed to do a level 4 evaluation or cost-benefit comparisons, L&E and CARE’s ability to make informed decisions on improving training to improve accuracy was limited.

Appendix XI: Information on the Kirkpatrick Model of Training and Development Evaluation

In recent years, a growing number of organizations have adopted a balanced, multilevel approach to evaluating their training and development efforts. Such an approach can help provide varied data and perspectives on the effect that training efforts have on the organization. One commonly accepted model consists of four levels of assessment. The first level measures the training participants’ reaction to, and satisfaction with, the training program or planned actions to use new or enhanced competencies. The second level measures the extent to which learning has occurred because of the training effort. The third level measures the application of this learning to the work environment through changes in behavior that trainees exhibit on the job because of the training or development program. The fourth level measures the impact of the training program on the agency’s program or organizational results. The fourth level is sometimes split into two levels with the fifth level representing a comparison of costs and benefits quantified in dollars. This fifth level—often referred to as return on investment (ROI)—compares the benefits (quantified in dollars) to the costs of the training and development program.

Not all training and development programs require, or are suitable for, higher levels of evaluation. Indeed, higher levels of evaluation can be challenging to conduct because of the difficulty and costs associated with data collection and the complexity in directly linking training and development programs to improved individual and organizational performance. Figure 2 depicts an example gradation of the extent to which an agency could use the various levels of evaluation to assess its training and development programs. For example, an agency may decide to evaluate participants’ reactions for all (100 percent) of its programs, while conducting an ROI analysis for 5 percent of its programs.

Appendix XII: Comments from IRS

Appendix XIII: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, Charlie Daniel, David Dornisch, Chad Gorman, Jason Jackson, Ronald Jones, Veronica Mayhand, and Katrina Taylor made key contributions to this report.