Report to the Chairman, Subcommittee

on Oversight, Committee on Ways and

Means, House of Representatives

United States General Accounting Office

GAO

December 2001 IRS TELEPHONE

ASSISTANCE

Limited Progress and

Missed Opportunities

to Analyze

Performance in the

2001 Filing Season

GAO-02-212
Page i GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

Letter 1

Results in Brief 2

Background 3

Scope and Methodology 5

IRS Made Limited Progress Toward Providing World-Class

Telephone Service 6

IRS Missed Some Opportunities to Better Understand Telephone

Performance 8

Conclusions 15

Recommendation for Executive Action 15

Agency Comments and Our Evaluation 16

Appendix I Descriptions and Formulas for Measures of IRS

Telephone Assistance Accessibility and Accuracy 18

Appendix II Comments From the Internal Revenue Service 19

Tables

Table 1: IRS Tax Filing Season Telephone Assistance Performance 6

Table 2: Descriptions and Formulas for Measures 18

Abbreviations

CQRS Centralized Quality Review System

GPRA Government Performance and Results Act

IRS Internal Revenue Service

Contents
Page 1 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

December 7, 2001

The Honorable Amo Houghton

Chairman, Subcommittee on Oversight

Committee on Ways and Means

House of Representatives

Dear Mr. Chairman:

For years, Congress has been concerned about the quality of service

taxpayers received when calling the Internal Revenue Service (IRS) for

help in understanding and meeting their tax obligations. IRS’ goal is to

make its telephone operation a “world-class customer service

organization” that provides taxpayers with accessible and accurate

assistance comparable to the best practices in the private and public

sectors. IRS has taken steps intended to improve how it responds to the

tens of millions of telephone calls received each year, such as expanding

the hours of service and increasing the use of automation. However, as we

previously reported to you, IRS has continued to struggle to provide

accessible and accurate telephone assistance. In the 2000 tax filing

season,1

 the quality of telephone assistance was mixed and below IRS’

long-term goal of providing world-class service.2

Because of your continuing interest in the quality of IRS’ telephone

assistance, you asked us to assess the performance and management of

IRS’ telephone operations in the 2001 tax filing season. More specifically,

our objectives were to (1) compare IRS’ performance in providing

accessible and accurate telephone assistance in the 2001 tax filing season

with the 2000 tax filing season and 2001 performance targets and (2)

assess IRS’ efforts to determine the factors that affected performance in

the 2001 tax filing season and its plans to evaluate the actions it took to

improve performance.

To address these objectives, we collected and analyzed data on eight

telephone assistance performance measures: four on taxpayers’ ability to

 1

According to IRS officials, the tax filing season generally begins January 1 and ends

around July 15 each year.

2

IRS Telephone Assistance: Quality of Service Mixed in the 2000 Filing Season and

Below IRS’ Long-Term Goal (GAO-01-189, Apr. 6, 2001).

United States General Accounting Office

Washington, DC 20548
Page 2 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

gain access to IRS and four on the accuracy of IRS’s response to

taxpayers. We also interviewed IRS officials involved in managing

telephone operations and obtained and analyzed supporting

documentation. The interviews included surveys of the 10 field directors

responsible for managing IRS’ 26 call sites, which employ about 10,000

telephone assistors. Our scope and methodology are discussed in greater

detail in a separate section of this report.

Overall, IRS made limited progress in the 2001 tax filing season toward its

goal of providing world-class telephone service. When compared with the

2000 tax filing season, access and accuracy performance for 2001

improved considerably in two of six comparable measures, declined in

one, and changed 2 percentage points or less in the others. In addition,

performance did not meet six of the eight 2001 targets IRS set to help it

move IRS toward its goal of world-class service. More specifically, when

access measures were compared with the 2000 tax filing season, there was

a 4-percentage-point decline in callers that hung up while waiting to speak

with an assistor, but they waited 15-percent longer to speak with an

assistor. The other two comparable measures of access showed little

change. For the accuracy measures, the quality of service when

responding to account inquiries increased by 10 percentage points, but the

quality of service when responding to tax law questions showed little

change. When compared with 2001 performance targets, IRS did not meet

any of its access goals, falling considerably below its target to shorten the

time taxpayers spend waiting to speak with an assistor. Although assistors

exceeded quality-of-service targets when responding to taxpayer questions

about their accounts and tax law, they did not meet higher targets for

providing taxpayers with correct answers and account adjustments.

IRS officials missed some opportunities to analyze data to better

understand the factors affecting telephone performance, including the

actions it took to improve performance. IRS collected and analyzed a

variety of data about the key factors affecting access and accuracy.

However, IRS managers sometimes reached conclusions about these key

factors without conducting analyses to test their conclusions. IRS officials

also missed opportunities to plan evaluations to determine the

effectiveness of the actions IRS took to improve access and accuracy. For

example, field directors disagreed about whether the new Accounts

Resolution Guide, a computer-based guide to help assistors resolve

Results in Brief
Page 3 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

taxpayers’ questions, improved accuracy in 2001. At the time the guide was

implemented, IRS did not have a plan to evaluate it, and no evaluation was

done. The Government Performance and Results Act of 1993 (GPRA)3 and

IRS guidance stress the need for conducting analyses to help management

understand how factors affect performance so that corrective actions can

be targeted toward those factors. We recognize that some data analyses

can be costly; however, some of the missed opportunities involve key

factors affecting performance. Also, since IRS devotes significant

resources—10,000 assistors—to telephone assistance, the benefit of more

effectively using the resources to improve service could be substantial.

Because of the limited progress made in the 2001 filing season, IRS cannot

afford to miss opportunities to analyze performance data and evaluate

improvement actions. We are recommending that the Commissioner

ensure that managers at the appropriate levels of IRS follow existing

guidance to plan and conduct analyses of the factors affecting

performance and to evaluate improvement actions.

In a letter dated December 3, 2001, the IRS Commissioner agreed with our

recommendations. (See p. 16 for a discussion of agency comments, which

are reprinted in app. II.)

In the 2001 tax filing season, IRS received more than 70.7 million calls on

its three toll-free assistance numbers and answered over 50.5 million

calls—assistors answered 22.7 million calls and automated systems

answered 27.8 million calls.4

 As in previous years, IRS had three toll-free

telephone numbers that taxpayers could call with questions about tax law,

taxpayer accounts, and refunds.5

 Located at 26 call sites, IRS has about

10,000 assistors that help taxpayers with a variety of questions ranging

from the applicability of tax laws to the status of their accounts. IRS’ call

 3

P.L. 103-62.

4

Automated systems include the Telephone Routing and Interactive System (TRIS) and

TeleTax interactive applications that allow taxpayers with certain questions to obtain the

information, such as the status of their refunds, or service they need without speaking to an

assistor. The three telephone numbers are part of TRIS, while TeleTax is a separate

number that also provides recorded information on about 150 tax topics. Taxpayers who

called a TRIS number about refund status were given the option of being routed to TeleTax

in the 2001 tax filing season.

5

IRS’s published telephone number for providing taxpayer assistance is 800-829-1040. IRS

lists additional toll-free telephone numbers on the notices it sends to taxpayers asking

them to call about an account or refund issue.

Background
Page 4 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

sites are supervised by 10 field directors, each of whom oversees two to

three sites.

IRS has four measures to evaluate the extent to which taxpayers are

provided with accessible telephone assistance, and four to evaluate the

extent to which taxpayers are provided with accurate telephone

assistance. (For more information on IRS’ telephone assistance access and

accuracy measures see app. I.) IRS’ measures of access are based on

actual counts of calls using data collected by IRS’ telephone system. IRS’

measures of the accuracy of assistance, the quality and correct response

measures, are estimates based on representative samples of nationwide

calls that quality assurance staff monitor and score for accuracy. IRS

began collecting data on correct responses in June 2000, so there are no

data for the 2000 tax filing season to compare with 2001.

Over the years, IRS has studied its telephone performance and made

changes designed to improve it. For example, in 1999, IRS extended its

hours of service to 24 hours a day, 7 days a week. By providing around-theclock service, IRS expected to distribute demand more evenly and thus

improve taxpayers’ access to service. With the increased use of callrouting technology in 1999, IRS began to manage its telephone operations

centrally at the Joint Operations Center in Atlanta. Routing calls to the

first available assistor who had the necessary skills to answer the

taxpayer’s question was expected to improve taxpayers’ access to service

and lessen the disparity in the level of service across sites. However, the

level of service declined in 1999,6

 and the quality of service was mixed in

the 2000 tax filing season and below IRS’ long-term goal of providing

world-class customer service.

According to IRS, some of the key factors that affected performance in the

2000 tax filing season were the demand for assistance, staffing levels,

assistor productivity, assistor skills, and IRS’ guidance for assistors. As we

discussed in a previous report,7

 IRS’ analyses did not cover all key

management decisions or other key factors that could have affected

telephone performance. Additionally, determining how each factor

affected performance was made even more difficult because many of the

factors are interrelated; changes in one can affect another. The IRS

Commissioner has recognized the complex interrelationships within the

 6

See Tax Administration: IRS’ 1999 Tax Filing Season (GAO/GGD-00-37, Dec. 15, 1999).

7

GAO-01-189, Apr. 6, 2001.
Page 5 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

telephone-operating environment and has stated that years of sustained

effort will be required for IRS to achieve its goal of providing world-class

telephone service.

To address our objectives, we interviewed IRS officials involved in

managing toll-free telephone operations and obtained and analyzed

supporting documentation as follows:

• To assess IRS’ performance in responding to calls on the three main

telephone assistance toll-free numbers, we compared the 2001 tax filing

season performance for accessibility and accuracy measures with IRS’

performance in the 2000 tax filing season and its 2001 performance

targets.8

• To assess IRS’ efforts to determine the factors that affected performance

in the 2001 tax filing season, including actions it took to improve

performance, we used as criteria GPRA and IRS’ own guidance on

analyzing performance data. We interviewed IRS officials in the Wage and

Investment and Small Business and Self-Employed Divisions,9

 and the

Joint Operations Center. We also analyzed various documents, including

reports on IRS’ efforts to determine the factors that affect telephone

performance and the results of actions to improve performance. In

addition, we used a questionnaire to obtain information from the 10 field

directors about their efforts to identify the factors that affected

performance and assess the effectiveness of actions taken to improve

performance.

While we did not independently assess the accuracy of IRS’ performance

data, we verified that IRS had procedures in place intended to ensure data

reliability. We did our work from February 2001 through October 2001 in

accordance with generally accepted government auditing standards.

 8

Accessibility measures are based on comparable data for the weeks beginning January 1

and ending July 14. Tax law and accounts accuracy rates are based on data for the months

of January through June.

9

IRS restructured its operation into four divisions: Wage and Investment Income (W&I),

Small Business and Self-Employed (SB/SE), Large and Mid-Size Business, and Tax-Exempt

and Government Entities. The W&I and SB/SE Divisions provide IRS’ toll-free assistance.

Scope and

Methodology
Page 6 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

IRS made limited progress in the 2001 tax filing season toward its longterm goal of providing world-class telephone service. When compared

with the 2000 tax filing season, access and accuracy performance

improved by 2 percentage points or less in three of the six comparable

measures. The quality of responses to account inquiries increased 10

percentage points, and there was a 4 percentage point decline in callers

who hung up while waiting to speak with an assistor; however, taxpayers

waited 15-percent longer to speak with an assistor. When compared with

2001 performance targets, IRS did not meet any of its accessibility goals.

These targets were intended to move IRS toward its goal of providing

world-class service. Although it met or exceeded the 2001 quality targets,

IRS did not meet the current year’s higher targets for providing taxpayers

with correct responses.

Table 1 compares IRS’ actual 2000 performance levels with its 2001

performance levels and targets. (See app. I for more information on the

measures.)

Table 1: IRS Tax Filing Season Telephone Assistance Performance

2000 2001

Accessibility measuresaActual Actual Target

 Assistor level of service 60% 61% 69%

 Assistor response level 39 41 49

 Abandon rate 22 18 12

 Average speed of answer 237 seconds 273 seconds 132 seconds

Accuracy measuresa

 Tax law quality rate 73% +/- 2%b75% +/- 1%b74%

 Accounts quality rate 59 +/- 2% b 69 +/- 1% b 63

 Tax law correct response rate c79 +/- 1% b 82

 Accounts correct response rate c88 +/- 1% b 91

a

Accessibility measures are based on actual counts. Accuracy measures are based on representative

samples.

b

Actual values are estimated at the 90-percent confidence level.

c

Comparable data do not exist.

Source: IRS data.

According to IRS officials, the access measures are similar to those

commonly used by world-class customer service organizations. They are

designed to focus efforts on enhancing taxpayers’ experience in getting

IRS Made Limited

Progress Toward

Providing World-Class

Telephone Service
Page 7 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

access to assistance. For example, the “assistor level of service” measure10

is intended to show IRS’ effectiveness in providing callers with access to

an assistor.11 The “assistor response level” is to measure the percentage of

taxpayers that waited 30 seconds or less to speak with an assistor. The

“abandon rate” measure is to show the percentage of taxpayers who hang

up while waiting to speak with an assistor, while the “average speed of

answer” measure is to show the average number of seconds taxpayers

wait to speak to an assistor.

IRS’ accuracy measures are designed to gauge the taxpayers’ experience in

getting accurate assistance. The “quality” measures are to show, for a

representative sample of calls, the percentage for which assistors followed

all procedures, such as properly identifying themselves at the beginning of

calls, doing appropriate research on taxpayers’ accounts, and providing

accurate information to taxpayers. The new “correct response rate”

measures are intended to show the percentage of calls for which IRS

assistors provided correct responses to inquiries without taking into

account procedural errors that would not affect the accuracy of the

information given the taxpayer. IRS began collecting these data in June

2000, so there were no data for the 2000 filing season to compare with

2001.

IRS also measures its performance in answering calls through the use of

automation. However, we did not consider this measure—“automated

service completion rate”—in assessing IRS’ performance because it

assumes that callers who get through to TeleTax are served. The TeleTax

system does not have data on how many callers hung up before

completing an automated service. Although IRS officials recognize that the

measure had limitations, according to them, routing refund status calls to

TeleTax allowed IRS to answer about 11.3 million more calls made to its

three main toll-free assistance numbers as compared with the 2000 tax

filing season. We are continuing to assess many of IRS’ new performance

measures, including those used to evaluate telephone assistance.

 10This measure includes callers who received automated service when the assistor service

queue was full.

11IRS’ previous access measure, level of service, was calculated by dividing the total calls

answered by total call attempts on IRS’ toll-free assistance lines. The Commissioner of IRS

discontinued use of this measure in February 2001, replacing it with the assistor level of

service measure.
Page 8 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

IRS missed some opportunities to better understand the factors that

affected performance and to plan evaluations of the actions it took to

improve performance. GPRA and IRS guidance outline the benefits of first

gathering and then analyzing data to help managers understand the

reasons for performance. To this end, IRS collected performance data on

access and conducted some analyses. Even so, IRS missed opportunities

to do other analyses of the factors affecting performance including actions

to improvement it. Contributing to the missed opportunities was a lack of

planning for the evaluation of those actions.

IRS has a variety of systems in place to make data on the access and

accuracy of telephone assistance available to managers. Two of these

systems are its Joint Operations Center in Atlanta and Centralized Quality

Review System (CQRS) in Philadelphia. Managers at call sites also collect

data on factors affecting access and accuracy performance.

IRS’ Joint Operations Center in Atlanta manages the activities of the 26 call

sites, including monitoring access data and routing calls to the next

available assistor anywhere in the country. The Center collects data on

various accessibility measures and makes those data available daily to IRS

managers through an internal Web site. According to IRS officials, IRS

improved the collection of performance data in the 2001 tax filing season.

For example, IRS implemented the Enterprise Telephone Database to

provide a central call information database. The database was designed to

provide IRS analysts and management with the most accurate information

for analysis and program decision-making by centralizing data collection

and producing a standard set of management reports.

IRS’ CQRS staff in Philadelphia are responsible for collecting data on the

accuracy of telephone assistance. CQRS provides call-site officials with

daily access through its internal Web site on the results of the sample of

calls answered by their sites. It also provides weekly and monthly reports

on the quality of sites’ responses to taxpayers’ questions about tax law or

about their accounts—two of the accuracy performance measures. These

data show the call sites what errors assistors are making so site managers

can quickly take action to reduce these errors. IRS officials told us that

they made better use of the data in the 2001 filing season. They said that

Wage and Investment Division and site officials developed strategies to

reduce assistor errors based on CQRS reports.

IRS call sites collect data on factors affecting access and accuracy in

various ways. For example, supervisors use real-time data and historical

IRS Missed Some

Opportunities to

Better Understand

Telephone

Performance

IRS Collected Performance

Data on Access and

Accuracy
Page 9 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

reports available at the call sites on how assistors spent their time,

including average handle time—the time an assistor spends talking with

the taxpayer, keeping the taxpayer on hold, and finishing the call and

indicating readiness to receive another call. Also, local staff monitor calls

to provide more detailed information on what errors assistors are making

and in what units the errors are being made.

IRS, both at the national and call-site levels, conducted some analyses of

performance data intended to determine the factors affecting

performance. GPRA and IRS guidance stress that analysis is a key part of

understanding performance and identifying improvement options. Analysis

of performance data is intended to help managers understand changes in

performance, determine root causes, and identify improvement options.

We identified several examples of analytical efforts to determine the

factors affecting performance at both the national and call-site levels. In

one example with regard to access, IRS officials analyzed data provided by

the Joint Operations Center to determine the reasons for the lower-thanexpected level of service in the first 3 months of fiscal year 2001. They

concluded that declining assistors’ productivity, as measured by average

handle time, was the major reason for the decline in access. Officials from

the operating divisions and the call sites conducted a series of

assessments to determine the underlying reasons for the increase in

average handle time. The assessments included focus groups with

managers and employees to solicit their views on productivity and

monitoring of telephone calls to determine how assistors use the time

between calls. The assessments identified three major categories of

factors that had negatively affected average handle time: management

practices, work processes, and computer systems. According to IRS

officials, some management practices adversely affected the level of

service because managers did not take actions to improve assistors’ use of

time between calls, the primary factor that increased average handle time.

IRS officials said that they took immediate corrective actions, such as

briefing assistors and supervisors and eliminating unnecessary data entry

and taxpayer notification requirements. They also organized teams to

further evaluate and resolve the more complicated work process and

computer systems issues.

According to IRS officials, IRS improved the analysis of Joint Operations

Center performance data in the 2001 tax filing season. For example,

analysts studied the factors that affected the demand for live assistance

regarding refunds, including the impact of increased electronic filing.

IRS Did Some Analyses to

Determine the Factors

Affecting Performance
Page 10 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

Center analysts also began developing quantitative models of the time

taxpayers and IRS spend on telephone questions, with the intent to better

match IRS’ resources with taxpayer needs.

Regarding accuracy, CQRS staff analyzed assistor errors and made

nationwide and individual site suggestions for addressing the causes of the

errors. The suggestions included changes to the assistors’ training and

guidance. Also, IRS field directors conducted some analyses to determine

the factors that affected access and accuracy. For example, one director

said analysis staff at one of her sites was doing a study to determine if the

site’s extensive use of faxing negatively affected access. She said she

believed the site’s average handle time was longer than others owing to the

site’s policy to keep the taxpayer on the line until the accounts issue was

resolved, even while the taxpayer faxed documents. Another director said

he monitored assistors to determine whether the computer-based research

tools assistors used to answer taxpayers’ questions met assistors’ needs.

Although IRS conducted some analyses of performance data, it missed

opportunities to do other analyses at the field level that could have

provided a better understanding of the factors affecting telephone

assistance performance, including the actions it took to improve

performance. Identifying the key factors that most affect performance is

important, yet difficult because those factors that can affect telephone

access and accuracy are often numerous and interrelated. IRS guidance

recognizes that there are a variety of approaches to conducting analyses,

such as hypothesis testing, which involves forming a tentative conclusion

that is tested using the data. We recognize that some analyses can be

costly, but as already noted, GPRA and IRS guidance stress that analysis is

a key part of identifying improvement options.

Field directors sometimes reached conclusions about the factors affecting

access and accuracy without conducting analyses to test their conclusions.

Seven of 10 directors said that the relative inexperience of assistors,

caused primarily by higher-than-usual attrition, was a key factor affecting

performance in the 2001 tax filing season. They said many experienced

seasonal assistors had taken permanent positions in other parts of IRS,

and the new hires who replaced them tended to take longer on calls and

make more errors. The second most common factor they cited was

problems with the computer-based research tools that assistors used to

answer taxpayers’ questions. Five of 10 directors cited such problems,

including difficulties in using the Servicewide Electronic Research Project

to search the Internal Revenue Manual, the assistors’ primary guidance for

IRS Missed Some

Opportunities to Analyze

Performance
Page 11 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

handling calls regarding taxpayers’ accounts. Some directors said the

computer systems were cumbersome and difficult to navigate, causing

assistors to take longer on calls and make errors, and some said computer

systems often failed and thus hampered assistors’ ability to research

questions.

Although directors cited high attrition and computer problems as key

factors affecting performance, only two directors identified a specific

analysis to support their conclusions. These directors said that focused

monitoring was done at their sites that confirmed the limitations of a

computer system assistors used to answer taxpayers’ questions. When we

asked other directors whether they or their staff had conducted analyses

to confirm or refute their conclusions about the factors that affected

performance, they acknowledged that they had not.

We identified several opportunities to conduct analyses of performance.

One way directors could have analyzed the impact of high attrition on

access and accuracy would have been to monitor a sample of calls

handled by experienced and inexperienced assistors to compare error

rates and average handle time. One director acknowledged that her

analysis staff could have done more to learn about how accepting

additional calls from businesses affected performance, such as comparing

the handle time for business taxpayer calls with individual taxpayer calls.

In another example, the program manager for the new Accounts

Resolution Guide, a computer-based, step-by-step guide on how to resolve

an account-related telephone call, agreed that more could have been done

to evaluate the guide’s effectiveness. For example, local managers could

have observed and compared assistors who used and did not use the

guide. According to IRS officials, additional analyses such as these could

have been done at relatively low cost.

Analyzing performance data can be important for several reasons. First,

there can be disagreement about which actions improve performance. For

example, some directors cited the Accounts Resolution Guide as a reason

for the significant improvement in their accounts quality rate. However,

another director said that the guide actually had a negative effect on

accounts quality, saying that because the guide was new to some assistors,

the “learning curve” to become proficient in using the guide caused

assistors to make errors.

Second, when multiple factors affect performance, knowing the extent to

which each factor has an impact can help managers decide where to focus

scarce managerial attention. For example, the solutions for addressing
Page 12 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

high attrition and computer problems are likely to be different.

Understanding the relative importance of high attrition and computer

problems could help prioritize improvement actions. In addition, in the

case of multiple factors, IRS’ use of performance measures to determine

the effect of one factor without controlling for other factors can be

misleading. In one case, a field director noted the risks of using average

handle time as an indicator of the effectiveness of actions taken to

improve the productivity of telephone assistors. The director noted that

other factors, such as the complexity of calls handled, could also affect

average handle time.

Third, the interrelationship among factors makes it difficult to determine

which factors most affect performance. For example, as we reported last

year, the quality of guidance assistors use can affect not only the accuracy

but also the accessibility of telephone assistance.12 Although step-by-step

guidance on how to respond to questions would likely improve accuracy,

it could also cause assistors to take more time answering calls, thereby

negatively affecting taxpayers’ access to service.

As we previously reported, conducting systematic analyses of program

performance is important for determining the factors affecting

performance and identifying opportunities for improvement.13 IRS

guidance states that analysis to understand the underlying factors

influencing the performance reflected in the balanced measures is

necessary to determine how to improve performance and warns that

managers should not “jump to conclusions” about the causes of

performance problems.14 As we said in a report on management reform,

“an organization cannot improve performance and customer satisfaction if

it does not know what it does that causes current levels of performance

and customer satisfaction.”

15 Because the factors affecting telephone

performance are numerous and are often interrelated, conducting analyses

is essential to determining the factors that have the most effect on

 12GAO-01-189, Apr. 6, 2001.

13Managing for Results: Challenges Agencies Face in Producing Credible Performance

Information (GAO/GGD-00-52, Feb. 4, 2000).

14Internal Revenue Manual 105.4.1.

15Management Reform: Using the Results Act in Quality Management to Improve Federal

Performance (GGD/T-99-151, July 29, 1999).
Page 13 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

performance so that corrective actions can be targeted toward those

factors.

We recognize that some analysis can be costly. Consequently, the costs

need to be balanced against the benefits. Considering that IRS devotes

significant resources (about 10,000 assistors) to telephone assistance, the

benefits of analysis—identifying ways to more effectively use resources

and improve service—could be substantial.

IRS missed opportunities to plan evaluations to determine the

effectiveness of actions it took to improve the access and accuracy of its

telephone assistance. IRS guidance presents a seven-step process designed

to guide data collection and analysis to identify ways to improve

performance. The last step states that managers should establish a plan

that tracks the effectiveness of actions taken to improve performance.

Without such a plan, IRS may not collect the data needed to judge the

action’s effectiveness. Additionally, planning to collect the data before the

improvement action is implemented may be less costly than developing

the data and evaluating the action later.

IRS field directors cited several different actions they took to address

factors that negatively affected access and accuracy in the prior filing

season:

• Assistor skill gaps (the difference between the skills assistors had and

the skills needed by IRS). To address skill gaps, field directors most

frequently cited training as the action taken, with all 10 directors referring

to training as the primary, and most often only, action taken. Although

training was designed at the division level, field directors and managers

were responsible for implementing it in the field, such as selecting the

trainers and determining which assistors need to be trained.

• Errors caused by flaws in the guidance assistors used to respond to

taxpayers’ account questions. To address the flaws in assistors’ guidance

for answering taxpayer calls, 5 of the 10 field directors cited the

implementation and use of computer-based tools to improve guidance,

including the Accounts Resolution Guide.

IRS Missed Opportunities

to Plan Evaluations of

Actions to Improve Access

and Accuracy
Page 14 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

• Declining assistor productivity. All 10 field directors said that the

primary actions taken to address assistor productivity declines were

nationwide managers’ training and employee briefings.16

Although field directors said all three of these actions were key to

improving telephone assistance performance this year, none of the field

directors cited specific evaluations or plans for assessing the effectiveness

of these actions. Instead, field directors based their assessment of actions

on performance trends, not taking into account the multiple factors or the

interrelatedness of factors that can affect a performance measure.

One example of a missed opportunity to plan an evaluation of an

improvement action on the national level is the lack of a systematic plan to

assess the impact the Accounts Resolution Guide had on access and

accuracy. The program manager for the guide said that IRS did not

develop such a plan because assistors were not required to use the guide

and IRS’ remote monitoring system was unable to determine when the

guide was used. The program manager agreed that local managers could

have done more to evaluate its effectiveness because they could have

observed and compared the results of assistors who did or did not use the

guide. Having an evaluation plan when the new Accounts Resolution

Guide was distributed to the field would have provided local managers

with guidance on the type of data to collect.

Another example of a missed opportunity is the lack of evaluation plans in

filing season readiness plans. IRS field directors complete a standard plan

each year, adding any items unique to their sites, to ensure that the sites

have taken all the necessary steps to provide phone assistance in the tax

filing season. Such steps include providing appropriate training and having

the equipment and guidance assistors need to respond to taxpayer calls.

The readiness plans we reviewed, however, did not include steps to ensure

that sites collect and analyze data to evaluate the effectiveness of any

improvement actions.

IRS officials noted that since some of the improvement actions were

national in scope, field directors would not have been individually

responsible for evaluating the effectiveness of the actions. We recognize

 16Management training was designed to improve managers’ skills in addressing productivity

issues, and the briefings were designed to increase assistor awareness of the increase in

average handle time and its impact on access.
Page 15 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

that evaluations of national improvement actions, such as the Accounts

Resolution Guide and managers’ training and employee briefings, to

address productivity may involve the higher levels in IRS that are

responsible for the action. Accordingly, as noted above, we discussed the

Guide with its national program manager and were told that no systematic

evaluation was done. We also discussed the actions to address

productivity with division-level officials. Similar to the field directors,

division officials evaluated the actions by monitoring trends in average

handle time and comparing average handle time with previous

performance.

IRS made limited progress in the 2001 tax filing season toward its longterm goal of providing world-class customer service. To speed progress

toward its long-term goal, IRS managers need to identify the causes for

performance, plan strategies to improve performance, and evaluate how

well those strategies worked. Unfortunately, IRS sometimes missed

opportunities to conduct analysis to help managers understand the

reasons for performance and to evaluate actions taken to improve

performance. The decision on the type of analysis to be done and who will

do it should consider the costs and benefits of the analysis and which

organizational levels are most responsible for the factor or improvement

action being analyzed. We recognize that some analyses can be costly;

however, some of the missed opportunities were low-cost and some

involved key factors affecting actions taken to improve performance. In

addition, there are costs in not using the 10,000 assistors as effectively as

possible. Considering IRS’ limited progress, it cannot afford to miss

opportunities without determining the most effective use of its resources

to improve performance.

We recommend that the Commissioner ensure that managers follow IRS

guidance on analyzing the factors that affect performance and evaluating

improvement actions. Specifically, we recommend that (1) field directors

be required to develop and follow written plans to collect and analyze data

to test their conclusions about the key local factors affecting performance

and, when appropriate, evaluate local improvement actions, such as

actions involving training; (2) field directors include in filing season

readiness plans a step to ensure that site managers have plans to evaluate

the effectiveness of any local improvement actions; and (3) program

managers and other appropriate national officials be required to develop

and follow written plans to evaluate the effectiveness of key national

improvement actions, such as the Accounts Resolution Guide.

Conclusions

Recommendation for

Executive Action
Page 16 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

The Commissioner of Internal Revenue provided written comments on a

draft of this report in a December 3, 2001, letter, which is reprinted in

appendix II. The Commissioner stated that while the report categorized

IRS’ progress toward providing world-class telephone assistance as

limited, he is confident that IRS is moving in the right direction. He noted

that IRS had initiated a number of strategies to improve telephone

assistance and agreed with our recommendation. Specifically, he agreed

that IRS needs “better testing, documentation, and analytical activities to

determine the factors that affect performance and assess the results of our

improvement actions.”

In his comments, the Commissioner noted that the report focuses on

accessibility to telephone assistors and stated that IRS also assists

taxpayers through automated telephone services and other means, such as

IRS’ Internet Web site and walk-in Tax Assistance Centers. As noted in the

report, we did not assess IRS’ automated telephone services because IRS’

method of measuring its performance in providing automated services had

limitations. The measure assumed that all callers that go through one of its

automated systems—TeleTax—were served because the TeleTax system

does not have data on how many taxpayers hung up before completing an

automated service. Other taxpayer services, such as walk-in assistance,

are to be addressed in our upcoming report on various aspects of the 2001

tax filing season.

As agreed with your staff, unless you publicly release its contents earlier,

we will make no further distribution of this report until 30 days after its

issue date. At that time, we will send copies of this report to the Chairmen

and Ranking Minority Members of the Senate Committee on Finance and

the House Committee on Ways and Means and the Ranking Minority

Member of the Subcommittee. We will also send copies to the Secretary of

the Treasury; the Commissioner of Internal Revenue; the Director, Office

of Management and Budget; and other interested parties. We will make

copies available to others on request.

Agency Comments

and Our Evaluation
Page 17 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

If you have any questions or would like additional information, please call

me at (202) 512-9110 or Carl Harris at (404) 679-1900. Key contributors to

this report are Ronald W. Jones and Ronald J. Heisterkamp.

James R. White

Director, Tax Issues
Appendix I: Descriptions and Formulas for

Measures of IRS Telephone Assistance

Accessibility and Accuracy

Page 18 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

Table 2: Descriptions and Formulas for Measures

Measure Description Formula

Assistor level of service The percentage of callers who were

routed to an assistor or who call when

the assistor queue is full and who

reached an assistor or completed an

automated service.

The number of calls answered by assistors, and by

automated services, when the waiting queue for speaking

with an assistor was full, divided by the total of these calls

plus the number of calls that hung up while waiting to

speak with an assistor or when the queue for speaking to

assistor was full, plus busy signals.

Assistor response level The percentage of callers who waited

30 seconds or less to speak with an

assistor.

Total number of services provided that were delivered

within the threshold of 30 seconds divided by the total

services handled.

Abandon rate The percentage of callers who hung

up while waiting to speak with an

assistor.

Total calls that hang up after waiting in line for an assistor

divided by the total call attempts offered to assistors.

Average speed of answer The average number of seconds

callers waited before an assistor took

their call.

Total number of seconds that customers waited in line for

service by assistor divided by the total services provided.

Tax law quality rate An estimate of the percentage of calls

dealing with tax law in which assistors

followed all required procedures.

The estimate is calculated by taking the site quality rate

from the Centralized Quality Review System (CQRS) and

weighting it using total site volume and number reviewed.

Accounts quality rate An estimate of the percentage of calls

dealing with accounts issues in which

assistors followed all required

procedures.

The estimate calculated by taking the site quality rate from

CQRS and weighting it using total site volume and number

reviewed.

Tax law correct response rate An estimate of the percentage of

callers who received accurate

responses to tax law inquiries.

The estimate calculated by taking the site rate from CQRS

and weighting it using total site volume and number

reviewed.

Accounts correct response rate An estimate of the percentage of

callers who received accurate

responses to account inquiries.

The estimate calculated by taking the site rate from the

Centralized Quality Review System and weighting it using

total site volume and number reviewed.

Appendix I: Descriptions and Formulas for

Measures of IRS Telephone Assistance

Accessibility and Accuracy
Appendix II: Comments From the Internal

Revenue Service

Page 19 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance

Appendix II: Comments From the Internal

Revenue Service
Appendix II: Comments From the Internal

Revenue Service

Page 20 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance
Appendix II: Comments From the Internal

Revenue Service

Page 21 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance
Appendix II: Comments From the Internal

Revenue Service

Page 22 GAO-02-212 IRS' 2001 Filing Season Telephone Assistance (440050)
The General Accounting Office, the investigative arm of Congress, exists to

support Congress in meeting its constitutional responsibilities and to help

improve the performance and accountability of the federal government for the

American people. GAO examines the use of public funds; evaluates federal

programs and policies; and provides analyses, recommendations, and other

assistance to help Congress make informed oversight, policy, and funding

decisions. GAO’s commitment to good government is reflected in its core values

of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents is through the

Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-text files of

current reports and testimony and an expanding archive of older products. The

Web site features a search engine to help you locate documents using key words

and phrases. You can print these documents in their entirety, including charts and

other graphics.

Each day, GAO issues a list of newly released reports, testimony, and

correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site

daily. The list contains links to the full-text document files. To have GAO E-mail

this list to you every afternoon, go to our home page and complete the easy-to-use

electronic order form found under “To Order GAO Products.”

The first copy of each printed report is free. Additional copies are $2 each. A

check or money order should be made out to the Superintendent of Documents.

GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a

single address are discounted 25 percent. Orders should be sent to:

U.S. General Accounting Office

P.O. Box 37050

Washington, D.C. 20013

To order by phone: Voice: (202) 512-6000

TDD: (301) 413-0006

Fax: (202) 258-4066

GAO Building

Room 1100, 700 4th Street, NW (corner of 4th and G Streets, NW)

Washington, D.C. 20013

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm,

E-mail: fraudnet@gao.gov, or

1-800-424-5454 (automated answering system).

Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800

U.S. General Accounting Office, 441 G. Street NW, Room 7149,

Washington, D.C. 20548

GAO’s Mission

Obtaining Copies of

GAO Reports and

Testimony

Order by Mail or Phone

Visit GAO’s Document

Distribution Center

To Report Fraud,

Waste, and Abuse in

Federal Programs

Public Affairs