GAO-01-189

Report to the Chairman, Subcommittee

on Oversight, Committee on Ways and

Means, House of Representatives

United States General Accounting Office

GAO

April 2001 IRS TELEPHONE

ASSISTANCE

Quality of Service

Mixed in the 2000

Filing Season and

Below IRS’ Long-Term

Goal
Page i GAO-01-189 Quality of Service Mixed in 2000 Filing Season

Letter 1

Results in Brief 2

Background 4

Scope, and Methodology 5

Telephone Assistance Results Were Mixed and Fell Short of WorldClass Service 6

Interrelated Factors Influenced IRS’ Telephone Performance 8

IRS’ Analysis of Its Performance Was Incomplete 15

IRS’ Performance Management System Restricts the Use of

Productivity Data 17

Conclusion 18

Recommendation for Executive Action 18

Agency Comments and Our Evaluation 18

Appendix I Comments From the Internal Revenue Service 21

Table

Table 1: IRS Filing Season Telephone Assistance Performance 7

Contents
Page 1 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

April 6, 2001

The Honorable Amo Houghton

Chairman, Subcommittee on Oversight

Committee on Ways and Means

House of Representatives

Dear Mr. Chairman:

IRS receives tens of millions of telephone calls each year from taxpayers

needing assistance in understanding and meeting their tax obligations.

Taxpayers call with a variety of issues, including questions about filing

returns, tax law, refunds, and notices IRS sends when it detects errors on

returns. Taxpayers often have been unable to reach IRS, especially during

the filing season when demand for assistance is greatest. Moreover, when

taxpayers reached IRS, they may have received inaccurate information.

IRS has pledged to take steps to make its telephone operation a “worldclass customer service organization” that would provide taxpayer

assistance that is comparable to the best practices in the private and

public sectors.

Because of your interest in the quality of IRS’ telephone service, you asked

us to address various aspects of IRS’ telephone assistance program and

other telephone customer service organizations. This is our third report in

response to your request. Our first report, issued in August 2000, presented

information on the human capital management practices of selected public

and private call centers.1

 Our second report, issued in January 2001,

evaluated IRS’ human capital management practices in identifying human

capital needs, implementing human capital practices to achieve IRS’ goals,

and evaluating those practices to continuously improve.2

 This report

discusses the performance of IRS’ telephone assistance in the 2000 filing

season and how IRS identified lessons learned to improve performance.

More specifically, our objectives were to (1) assess IRS’ performance in

the 2000 filing season using three telephone program performance

 1

Customer Service: Human Capital Management at Selected Public and Private Call Centers

(GAO/GGD-00-161, Aug. 22, 2000).

2

IRS Telephone Assistance: Opportunities to Improve Human Capital Management (GAO01-144, Jan. 30, 2001).

United States General Accounting Office

Washington, DC 20548
Page 2 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

measures: level of service, tax law accuracy, and account accuracy;3 (2)

identify the key factors and describe how they affected performance; and

(3) assess IRS’ process for analyzing its performance in the 1999 and 2000

filing seasons in order to make improvements. In the course of our audit

work, we learned that IRS restricted supervisors from using productivity

data to evaluate assistors’ performance. Some IRS officials we talked to

were uncertain about the basis for the restriction, for example, some

thought that it was mandated by the Internal Revenue Service

Restructuring and Reform Act of 1998.4

 After discussion with your staff,

we agreed to determine IRS’ basis for this restriction, as a fourth objective.

IRS telephone assistance showed mixed results in the 2000 filing season.

IRS answered 59 percent of calls to the toll-free taxpayer assistance lines

in 2000, better than in the 50 percent achieved in 1999, but below the 69

percent level of service achieved in the 1998 filing season. For tax law and

account questions, IRS estimated it provided accurate answers 73 percent

and 59 percent of the time, respectively.5

 Tax law accuracy was

comparable to 1999 performance, but lower than the 2000 target of 80

percent. Account accuracy was slightly lower than the 2000 target of 63

percent. Although IRS had not defined world-class service in terms of

specific measures and goals, IRS officials acknowledged performance fell

short of its long-term goal of providing assistance comparable to that

provided by leading public and private telephone customer service

organizations.

A number of interrelated factors influenced IRS’ performance in providing

telephone assistance, such as the demand for assistance, number of staff

IRS devoted to providing assistance, assistors’ productivity, and their skill

level. IRS did not have complete information on how these and other

factors affected level of service and accuracy in the 2000 filing season.

However, according to IRS officials,

 3

Level of service is the rate at which taxpayers who called IRS actually got through and

received service. Tax law and account accuracy rates are the percentages of a sample of

calls in which telephone assistors provided accurate answers and fully adhered to IRS

guidance for assisting taxpayers with those types of questions.

4

P.L. 105-206, July 22, 1998.

5

The sample of tax law and account calls IRS monitored allowed IRS to be 90 percent

confident that the telephone assistors’ true accuracy rate was between 72 and 74 percent in

responding to tax law questions, and between 58 and 60 in responding to account

questions.

Results in Brief
Page 3 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

• the increase in level of service was due primarily to a 25-percent decrease

in demand, partly because IRS sent taxpayers fewer notices and processed

returns more quickly;

• the level of service might have been higher had IRS not reduced staffing of

telephone assistance by 8 percent compared to 1999 in an effort to better

balance overall workload and staffing among customer service programs;

• assistor productivity, in terms of the average length of a call, declined

about 17 percent, partly because assistors received a greater percentage of

the types of calls that took longer to answer; and

• the level of productivity continued to be affected by policy changes IRS

made before the 1999 filing season, such as discontinuing the automatic

routing of another call to assistors immediately upon their completion of a

call.

In addition, according to IRS officials, gaps between the skills that

assistors had and those IRS needed negatively affected the accuracy of

IRS’ responses to taxpayer inquiries. Also, flaws in the guidance assistors

used and frequent changes to it caused inaccurate responses to taxpayers’

account questions.

IRS’ analysis of its telephone assistance performance in the 1999 and 2000

filing seasons was incomplete. Although IRS collected various data and

conducted several analyses of performance, the approach either did not

assess or assessed incompletely some of the key management decisions

and other factors that affected performance. As a consequence, IRS

management did not have some significant information that could have

been used to make decisions intended to improve future performance. For

example, in its studies of productivity, IRS did not study all the segments

of assistors’ time that would affect overall productivity.

Because IRS’ analysis of performance in recent filing seasons was

incomplete, we recommend that the IRS Commissioner ensure that IRS

analyzes all the key management decisions and other key factors affecting

telephone performance each filing season to determine their impact on the

quality of service and to make improvements. The Commissioner agreed

with our assessment of IRS’ telephone performance during the 2000 filing

season and with our recommendation.

IRS’ “balanced measures” performance management system, and not the

IRS Restructuring and Reform Act of 1998, was the basis for restricting the
Page 4 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

use of productivity data to evaluate employee performance.6 When

designing and implementing the balanced measures system, IRS

management decided to prohibit telephone assistance supervisors from

using productivity data when evaluating all assistors. The prohibition was

intended to promote a more balanced focus by assistors on efficiency,

quality, and service.

IRS’ telephone assistors are located at 25 call sites around the country. In

the 1999 filing season, IRS made major changes to its telephone customer

service program. For example, IRS extended its hours of service to 24

hours a day, 7 days a week. IRS officials said they believed around-theclock assistance would improve the level of service by distributing

demand more evenly and support IRS’ efforts to provide world-class

service by making assistance available anytime.

Also in 1999, IRS began managing its telephone operations centrally at the

Customer Service Operations Center in Atlanta by using new call-routing

technology. IRS’ call router was designed to improve the overall level of

service, as well as lessen disparities in the level of service across sites by

sending each call to the first available assistor nationwide who had the

necessary skills to answer the taxpayer’s question. As part of this

centralized management, IRS developed its first national call schedule that

projected the volume of calls, for each half-hour, at each of IRS’ 25 call

sites, and the staff resources necessary to handle that volume.

As in previous years, in the 2000 filing season, IRS had three toll-free

telephone numbers taxpayers could call with questions about tax law,

taxpayer accounts, and refunds. The three primary measures IRS used to

evaluate its telephone performance were level of service, tax law

accuracy, and account accuracy.

IRS measures its level of service by determining the rate at which

taxpayers that call IRS actually get through and receive assistance. Level

of service is calculated by dividing the number of calls answered by the

total call attempts. Calls answered is defined as calls that received service,

 6

IRS’ balanced measure system is its approach to evaluating employee and organizational

performance; it is designed to balance customer satisfaction, employee satisfaction, and

business results.

Background
Page 5 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

either from assistors or telephone interactive applications.7 Total call

attempts includes repeat calls and is the sum of calls answered, calls

abandoned by the caller before receiving assistance, and calls that

received a busy signal. IRS’ tax law accuracy and account accuracy rates

are based on a sample of nationwide calls that quality assurance staff

listen in on and score for accuracy. Using IRS’ Centralized Quality Review

System, staff in Philadelphia listen to sample calls from beginning to end

and determine whether the assistors provide accurate answers, follow

procedural guidance to ensure a complete response, and are courteous to

the taxpayers. If the assistors fail to adhere to any part of the guidance, or

are not courteous to the taxpayers, the calls are counted as inaccurate. IRS

began centrally monitoring calls to measure tax law accuracy in fiscal

year1999 and account accuracy in fiscal year 2000.

To address our objectives, we examined documents and interviewed IRS

officials. Specifically:

• to assess IRS’ performance in the three main telephone assistance toll-free

numbers, we compared its 2000 filing season level of service, tax law

accuracy, and account accuracy with its performance in the 1998 and 1999

filing seasons and its performance targets, and discussed with IRS officials

how its performance compared with world-class customer service; 8

• to identify the key factors and describe how they affected performance in

the 1999 and 2000 filing seasons we interviewed IRS officials, including

executives, division chiefs, and first-line supervisors in Customer Service

 7

Interactive applications are designed to allow taxpayers with certain questions to obtain

the information or service they need without speaking to an assistor, such as determining

the status of their refunds.

8

According to IRS officials, the filing season generally begins January 1 and ends around

July 15 each year. In previous years, GAO's review of telephone performance generally

covered the period January 1 through the April 15 filing deadline. However, demand for

telephone assistance continues past the deadline because taxpayers call about such issues

as the status of refunds or notices IRS sends to taxpayers related to returns they filed.

Level of service rates are based on comparable data for weeks beginning January 1 and

ending around July 15. Tax law and accounts accuracy rates are based on data for the

months of January through June, except for tax law accuracy in 1999, which was based on

data for January through April due to database limitations. According to IRS, account

accuracy in the 2000 filing season should not be compared to accuracy rates in previous

years, nor should tax law accuracy in the 1999 or 2000 filing season be compared to the

accuracy rate in 1998, because IRS changed the way accuracy was measured. Instead of

conducting test calls that measured limited adherence to guidance, IRS monitored actual

calls and more strictly measured assistors’ adherence to guidance.

Scope, and

Methodology
Page 6 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

Field Operations and at call sites; and analyzed documents, including

various reports that described and analyzed the factors that affected IRS’

performance;

• to assess IRS’ process for analyzing its performance in the 1999 and 2000

filing seasons in order to make improvements, we interviewed IRS

officials, including National Office and Customer Service Field Operations

officials responsible for collecting and analyzing data on IRS performance;

and analyzed documents, including various reports related to the process,

such as the 1999 National Office business review and statistical analyses of

2000 filing season performance; and

• to determine the basis for restricting supervisors from using productivity

data to evaluate or discuss telephone assistor performance, we

interviewed IRS officials, including officials in the Organizational

Performance Division and Customer Service Field Operations; and

analyzed documents related to the restriction, including the Internal

Revenue Manual and materials used to train supervisors on the use of

statistics.

We performed our work at IRS’ National Office in Washington, D.C.; Office

of the Chief, Customer Service Field Operations, and Customer Service

Operations Center in Atlanta; and the telephone assistance call sites in

Atlanta, Dallas, and Kansas City, KS. We chose these three sites in order to

include sites of various sizes, hours of operation, and work. We did not

independently assess the accuracy of IRS’ performance data, however, we

verified that IRS had procedures in place intended to ensure data

reliability. We did our work from January 2000 through February 2001 in

accordance with generally accepted government auditing standards.

We obtained written comments on a draft of this report from the

Commissioner of Internal Revenue in a letter dated April 2, 2001. The

comments are discussed at the end of this report and reprinted in

appendix I.

IRS telephone assistance showed mixed results in the 2000 filing season.

Performance improved somewhat in the 2000 filing season as compared

with 1999, but according to IRS officials, fell short of IRS’ long-term goal to

provide world-class customer service. While IRS had not established

specific measures and goals for world-class service, it was considering

adopting some of those used by leading telephone customer service

organizations.

Telephone Assistance

Results Were Mixed

and Fell Short of

World-Class Service
Page 7 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

In the 2000 filing season, IRS answered 36.1 million of the 61 million calls

taxpayers made, resulting in a 59-percent level of service—better than the

50 percent IRS achieved in the 1999 filing season and its target of 58

percent, but short of the 69 percent IRS achieved in the 1998 filing season.

IRS provided accurate responses in 73 percent of the tax law calls it

answered—unchanged from 1999 and lower than its 2000 target of 80

percent. Account accuracy in the 2000 filing season was slightly lower

than IRS’ target of 63 percent. Table 1 shows IRS’ performance during the

1998-2000 filing seasons.

Table 1: IRS Filing Season Telephone Assistance Performance

1998 1999 2000

Measure Actual Actual Target Actual

Level of service 69% 50% 58% 59%

Tax law accuracy a 73%b 80% 73%b

Accounts accuracy a a 63% 59%b

a

Comparable data does not exist.

b

Actual values are estimates, with 90 percent confidence that the true value is never more than plus

or minus 2 percentage points.

Source: IRS data.

IRS officials in National Office and Customer Service Field Operations

recognized that telephone performance in the 2000 filing season fell short

of its long-term goal of providing world-class customer service--assistance

comparable to that provided by leading public and private telephone

customer service organizations. IRS has not defined world-class service in

terms of specific measures and goals.9

 However, IRS officials have

acknowledged the need to change their performance measures to be more

consistent with leading telephone customer service organizations. IRS’

level of service measures the percentage of call attempts that receive

assistance, with no consideration of how long callers wait for it. Some

leading organizations measure service level as the percentage of calls

answered within a specified period of time, such as answering 90 percent

of calls within 30 seconds.10 IRS was considering adopting a similar

measure and goal. However, IRS’ performance in fiscal year 2000 fell

 9

In responding to our earlier report (GAO-01-144, January 30, 2001), the Commissioner said

that IRS’ short-term level-of-service goal for fiscal year 2002 would be 74 percent, with a

goal of reaching 85 to 90 percent by fiscal year 2003.

10GAO/GGD-00-161, August 22, 2000.
Page 8 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

substantially short of this level, with only 31 percent of calls being

answered within 30 seconds.

A number of interrelated factors influenced IRS’ telephone assistance

performance in the 2000 filing season. According to IRS, some of the key

factors were the demand for assistance, staffing levels, assistor

productivity, assistor skills, and IRS’ guidance for assistors. Additionally,

many of the factors were interrelated—changes in one factor could cause

changes in others.

According to an analysis by Customer Service Field Operations officials,

IRS was able to answer a greater percentage of calls in the 2000 filing

season compared with 1999 because demand for service substantially

decreased. IRS measured demand in two ways: total call attempts and

unique telephone number attempts. Total call attempts includes repeat

calls and is the sum of calls answered, calls abandoned by the caller

before receiving assistance, and calls that received a busy signal. The

unique telephone number measure is designed to count the number of

taxpayers who called, rather than the number of calls. It measures the

number of calls from identifiable telephone numbers, and counts all call

attempts from each telephone number as one call until it reaches IRS and

is served, or until a 1-week window expires. Total call attempts decreased

from 83.5 million in 1999 to 62.8 million, a 25-percent decrease, while

unique number attempts decreased from 33.2 million to 25.9 million, a 22-

percent decrease.11 According to IRS, demand declined partly because IRS

issued 1.8 million fewer notices to taxpayers asking them to call IRS about

such issues as math errors IRS detected while processing returns. Also,

fewer taxpayers called about the status of their refunds because IRS

processed returns more quickly.

Additionally, timing of notices IRS sends taxpayers influences demand for

assistance. For example, as we previously reported, in the 2000 filing

season, because of contract delays, a contractor mailed the bulk of over 1

million notices to taxpayers over a 2-week period, rather than over a 7-

 11Total call attempts are based on data from October 1, 1999, through April 22, 2000, for all

of IRS’ toll-free lines, which include, among others, IRS telephone lines for collecting taxes

and taking criminal investigation calls. Unique call attempts are based on data from

October 1, 1999, through April 8, 2000, for the three assistance toll-free lines.

Interrelated Factors

Influenced IRS’

Telephone

Performance

Demand
Page 9 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

week period as intended.12 When taxpayers called about the notices, IRS

was unprepared to answer the unexpected increase in the number of

telephone calls, which caused level of service to decline during this period.

According to IRS officials, a factor that may have prevented the level of

service from being higher in the 2000 filing season was IRS’ decision to

reduce the staff dedicated to telephone assistance as compared with 1999.

Specifically, in the 2000 filing season, IRS dedicated 4,912 staff years to

telephone assistance as compared to 5,339 staff years in 1999, an 8-percent

decline.13 According to IRS officials, IRS dedicated fewer resources to

telephone assistance to increase staffing in other programs, including the

telephone collection system, adjustments, and service center compliance.

IRS managers were concerned that in 1999, when IRS redirected resources

from these other programs to telephone assistance, the backlog in these

programs increased to unacceptable levels, causing uneven service and a

decline in collection revenues.

Assistor productivity is another factor that affects the level of service

taxpayers receive from IRS. According to IRS officials, the level of service

would have been higher had assistor productivity not declined in the 2000

filing season. This decline was in addition to a productivity decline that

occurred in the 1999 filing season.

According to analysts and officials in Customer Service Field Operations, a

key indicator of productivity is the average time for an assistor to handle a

call. Handle time is the total of the time

• an assistor spends talking to the taxpayer,

• the taxpayer is on hold, and

 12 Tax Administration: IRS' 2000 Tax Filing Season and 2001 Budget Request (GAO/TGGD/AIMD-00-133, Mar. 28, 2000).

13A staff year is equivalent to one full-time employee, also known as a full-time equivalent.

It includes direct and overhead time (both regular and overtime) expended by permanent

and seasonal Customer Service and Compliance employees related to answering telephone

calls on the three assistance lines, Criminal Investigation, and Problem Resolution Program

toll-free telephone lines. In October 2000, IRS had 9,865 customer service representatives,

with about 40 percent of them being seasonal employees who worked less than 12 months

per year.

Staffing Level

Productivity
Page 10 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

• the assistor spends in “wrap status”, which is the time between hanging up

at the end of a call and indicating readiness to receive another call.

An IRS analysis showed that the average handle time increased from 318.5

seconds in the 1999 filing season to 371.5 seconds in the 2000 filing season,

or about a 17-percent decline in productivity.

According to a Treasury Inspector General for Tax Administration report,

an increase in the number of calls an assistor handles has a profound

effect on level of service. For example, if assistors had handled one more

call per hour, IRS would have answered more than 8.5 million additional

calls during the first 6 months of fiscal year 1999.14

While IRS had not determined all the causes of the decline in productivity

since 1998, according to a July 2000 IRS study, approximately 58 percent

of the productivity decline from 1999 to 2000 was due to assistors’

receiving a greater percentage of calls that took longer to handle. For

example, screening calls, in which the assistor talked with the taxpayer for

only a short time to determine the taxpayer’s question and where the call

should be routed, decreased from 35 percent of the calls assistors handled

in 1999 to 21 percent in 2000. The study concluded that assistors likely

handled fewer of these calls because IRS changed its telephone message

to discourage callers from posing as rotary dialers without a touch-tone

telephone, allowing them to bypass the menu system and go directly to an

assistor. This study did not identify what caused the remaining 42-percent

decline in productivity in 2000.

According to IRS officials, four policy changes that lowered productivity in

the 1999 filing season continued to adversely affect productivity in the

2000 filing season. Specifically, in 1999, IRS

• discontinued automatically routing another call to an assistor immediately

upon completion of a call;

• increased restrictions on using productivity data when evaluating

assistors’ performance;

 14Treasury Inspector General for Tax Administration, Toll-Free Telephone Service Levels

Declined in 1999, Despite Costly Efforts to Achieve World-Class Performance (Ref. No.

2000-30-062, March 2000), p. 29.
Page 11 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

• disproportionately diverted staff from the peak demand shifts to shifts

when fewer taxpayers call when it implemented its 24-hour-a-day, 7-day-aweek assistance; and

• discontinued measuring productivity of individual call sites.

First, as part of its November 1998 agreement with the National Treasury

Employees Union, IRS discontinued using a call management tool—”autoavailable”—that automatically routed another telephone call to an assistor

as soon as a call was completed. Instead, assistors were placed in “wrap

status” after each call and were unavailable until they pressed a keyboard

button that made them available. Wrap status was designed to allow

assistors time to document the results of a call or to allow them to take a

momentary break after a stressful call. According to IRS officials, allowing

assistors to determine when they were ready to take another call added

time to each call, causing other callers to wait longer for service. With

longer wait times, many taxpayers hung up before reaching an assistor,

thereby reducing level of service. According to IRS statistics, for its tax

law, account, and refund assistance lines, the average wrap times

increased 94, 204, and 176 percent, respectively, from 1998 to 1999.

Second, 1999 was the first filing season with increased restrictions on

supervisors using productivity data to evaluate assistors’ performance or

discuss their performance. Some IRS studies of the 1999 filing season

concluded that the restrictions negatively affected productivity. For

example, one IRS study found that many site managers were concerned

about their inability to properly manage assistors’ use of wrap time

without using productivity data. Five of the seven supervisors we spoke to

about the 2000 filing season said they were dissatisfied with the

restrictions. They said assistors know supervisors are restricted from

using productivity data to evaluate employees’ performance and that

supervisors do not have adequate time to devote to monitoring and

physical observation. Therefore, they said assistors are free to spend more

time than necessary in wrap status.

Our conversations with IRS officials, including supervisors at call sites and

officials in the Organizational Performance Division, and review of related

documents indicated officials were uncertain about the basis for the

restriction, and some thought that it was mandated by the IRS

Restructuring and Reform Act. We discuss this issue near the end of this

report.

Third, increasing the hours of telephone assistance to 24 hours a day, 7

days a week for the 1999 filing season may have decreased overall
Page 12 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

productivity because IRS disproportionately shifted staffing away from the

hours when most taxpayers call. According to an IRS review, the diversion

of staff away from hours when most taxpayers called resulted in a lower

level of service because taxpayers waited longer for assistance, more

taxpayers hung up while waiting, and demand increased because

taxpayers redialed more. Limited data from a week in the 2000 filing

season indicated that IRS continued to overstaff the night shift when

compared to the other shifts. For example, for the week of April 2, 2000,

through April 8, 2000, assistors working the night shift spent, on average,

44 percent of their time waiting to receive a call, whereas assistors

working the day and evening shift spent 15 percent of their time waiting to

receive a call.

An IRS Customer Service Field Operations official responsible for

scheduling staff said assistors spent more time waiting for calls at night

because, when compared with the demand for assistance, IRS scheduled

disproportionately more assistors during the night shift than other shifts.

Assistors working nights generally had fewer skills, which required a

disproportionate level of staffing to ensure that all needed skills were

available. According to the official, IRS’ attempts to attract more skilled

assistors to work off-peak hours were unsuccessful. To counter the

negative effects of staffing the extended hours, for fiscal year 2000, IRS

limited its staffing of tax law assistance to 16 hours a day, 6 days a week

after the filing deadline, when fewer taxpayers call with tax law questions.

Fourth, beginning in 1999, IRS no longer had a performance measure that

held sites accountable for productivity. Instead of measuring level of

service as it had in the past, IRS measured a site’s performance on the

number of assistors assigned to answer telephone calls each half-hour as

compared to the number of assistors specified in the site’s half-hour work

schedule. IRS made this change, in part, because the sites were no longer

responsible for predicting and meeting demand. According to an IRS

assessment of the 1999 filing season, replacing the site level of service

measure with the measure of assistor presence diminished the focus on

productivity and the extent to which sites sought opportunities to improve

productivity.

IRS Customer Service Field Operations officials added that, despite the

decline in productivity, taxpayers might have received better service

overall if assistors took the time needed to fully resolve each taxpayer’s

call, rather than being concerned about the number of calls answered.

However, IRS had not determined if the decline in productivity had

improved the quality of service.
Page 13 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

According to IRS officials, including the Commissioner, Customer Service

Field Operations officials, and supervisors at call sites, the accuracy rates

IRS achieved in the 2000 filing season continued to be adversely affected

by assistor skill gaps—the difference between the skills assistors had and

the skills needed by IRS. Skill gaps were caused, in part, when IRS

implemented its new call router in 1999.

With the call router, individual assistors were required to answer calls on a

broader range of topics, often without adequate training or experience.

Before the 1999 filing season, each call site decided how it would group

topics for routing and assistor specialization. According to a cognizant

official, the number of topic groups at sites ranged from 40 to 125, which

allowed assistors to typically specialize in only one or two topics. Because

the new call router could not handle differences in topic groups among

call sites, nor efficiently route calls to that many groups, the topic groups

had to be standardized and were reduced to 31. This increased the number

of topics in each group, which typically required an assistor to answer

calls on five or more tax law topics, creating a skill gap. IRS officials

recognized that assistors had struggled with the amount of information

they were required to know in 1999, so for the 2000 filing season IRS

increased the number of topic groups to 46, which decreased the number

of topics in each group. However, according to IRS officials, the loss of

specialization continued to affect accuracy in the 2000 filing season.

IRS officials said they were aware of how skill gaps had negatively

affected the accuracy of the assistance taxpayers received in 1999 and, in

August 1999, IRS began to revise its training materials to better prepare

assistors to answer questions in their assigned topic groups. However,

according to IRS officials, much of the new training material was not

developed in time for the 2000 filing season. Furthermore, a cognizant IRS

official said the first attempt to revise the training did not separate each

topic into a self-contained course. For the 2001 filing season, IRS revised

its training material so that each course contained only one topic, enabling

IRS to provide assistors with just-in-time training on the specific topics

they were assigned to work.

IRS officials said organizational changes are needed to further reduce the

number of topics assistors are expected to know. In a May 2000 memo, the

Commissioner cited low accuracy scores and employee survey comments

as evidence that IRS was expecting its assistors and managers to have

knowledge in areas that are far too broad and that IRS was “attempting the

impossible” by trying to fill skill gaps solely with training. IRS officials said

IRS’ reorganization would allow specialization by taxpayer group, but that

Assistors’ Skills
Page 14 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

even greater levels of specialization were needed. Accordingly, as part of

its restructuring efforts, in June 2000, IRS began long-term planning efforts

to create greater specialization at both the call site and assistor levels.

The quality of the guidance assistors used also affected whether they

provided accurate assistance. IRS officials at National Office and call sites

said the guidance assistors used in the 2000 filing season to respond to

account questions was confusing and difficult to use, causing assistors to

make mistakes, thereby lowering the accuracy rate.

IRS officials said that over the years, the Internal Revenue Manual—the

assistors’ guide for account questions—had grown from a collection of

handbooks to a large, unwieldy document with duplicative and erroneous

information. According to IRS officials, errors in the Manual had long been

a problem for which sites had developed local “workaround” procedures.

IRS established a task force to correct these problems, and issued a new

draft version at the end of the 1999 filing season. While the draft Manual

was smaller and contained less duplicative and erroneous information, it

was missing some needed information and cross-references. However, IRS

did not realize the extent of the problems with the Manual until October

1999, when it began holding assistors accountable for strictly adhering to

the Manual as part of its central monitoring of account accuracy. As a

result, the draft was recalled, and the task force continued to make

corrections to the Manual throughout the filing season. The task force

issued two new versions in February 2000 and May 2000. According to IRS

officials, the frequent changes in the Manual made it difficult for assistors

to know which version to use, sometimes leading to inaccurate answers.

According to IRS officials responsible for Manual revision, as October 1,

2000, the task force had corrected problems with the Manual and related

training material in time for the 2001 filing season. Additionally, IRS

officials said they implemented a new guide in October 2000 to make it

easier for assistors to follow the proper steps and provide accurate

assistance to taxpayers with account questions.

Determining how each factor affects level of service and accuracy is made

even more difficult because many of the factors are interrelated; changes

in one can affect another. For example, the demand for assistance, or the

number of call attempts, is influenced by the level of productivity. Fewer

incoming calls make it easier for a given number of assistors to answer a

greater percentage of incoming calls. Answering a greater percentage of

Assistors’ Guidance

Interrelationships Make

Identifying Key Factors

Difficult
Page 15 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

incoming calls—a higher productivity level—reduces the number of repeat

calls, which reduces the number of calls overall. Similarly, the quality of

guidance assistors use affects not only accuracy, but also demand. While

step-by-step guidance on how to respond to questions would likely

improve accuracy levels and service for some taxpayers, it could also

cause assistors to take more time answering the call, lower productivity,

and increase the number of taxpayers who are unable to get through,

causing them to redial, and thereby increase demand.

IRS’ analysis of its telephone assistance performance in the 1999 and 2000

filing seasons was incomplete. Although IRS collected various data and

conducted several analyses of performance, the approach either did not

assess or assessed incompletely some of the key management decisions

and other factors that affected performance. As a consequence, IRS

management had less information than it could have on which to make

decisions intended to improve future performance.

IRS undertook many efforts in 1999 and 2000 intended to identify factors

that affected performance. For example, IRS

• conducted a best practices productivity study in 1999 to identify best

practices among IRS call sites and why productivity varied among them;

• reviewed its implementation of 24-hour–a-day, 7-day-a-week assistance to

determine its effects on such things as costs and quality of assistance;

• conducted local and centralized monitoring of telephone calls to

determine what errors assistors made and why;

• conducted a study in 2000 to determine why productivity had declined;

• established a filing season critique program in 2000 to solicit information

from field staff about their problems and successes during the filing

season; and

• conducted a 1999 fiscal year business review that addressed many of the

factors that affected telephone performance.

In some of its efforts, IRS began analyzing the data made available through

management information systems at its Customer Service Operations

Center, which opened in December 1998. For example, as a part of the

2000 productivity study noted above, IRS used statistical analysis to assess

how productivity was affected by such factors as the complexity of calls

handled and assistor experience and education. In a similar analysis, IRS

assessed how call demand was affected by such factors as returns filed,

notices issued, refunds issued, refund cycle times, and electronic filing

IRS’ Analysis of Its

Performance Was

Incomplete
Page 16 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

return rates. Although IRS now has better quantitative data to assess its

performance and make decisions about ways to improve performance, IRS

officials said much work still needs to be done to understand the factors

that affect performance.

Other leading telephone customer service organizations we studied see the

importance of continuous evaluation and incorporating evaluation results

to make improvements.15 As we said in a recent report on management

reform, “an organization cannot improve performance and customer

satisfaction if it does not know what it does that causes current levels of

performance and customer satisfaction.”

16

IRS’ efforts to evaluate the factors affecting telephone assistance were

incomplete and failed to provide IRS management with some significant

information that could have been used to improve performance. For

example, while IRS did several studies of productivity, the studies relied

on handle time as the measure of productivity. Other segments of

assistors’ time that would affect overall productivity, including time spent

waiting to receive a call, time spent away from the telephone (in meetings,

breaks, and training), and time assistors were not assigned to answer calls,

were not studied. In another example, the most extensive single review of

the factors that affected performance—the 1999 National Office business

review—did not assess how extending the hours of service to 24 hours, 7

days a week affected level of service. Earlier, we described how IRS’

disproportionate move of assistors to the night shift created differentials

between shifts in the time spent waiting for a call. Furthermore, while the

National Office review examined the effects of demand on service, it did

not examine why demand increased in 1999. Also, IRS did not evaluate the

effectiveness of its management decision not to automatically route calls

to assistors as soon as they completed a call, or the several other policy

changes noted above, even though they were intended to significantly

improve overall performance.

The gaps in IRS’ information about the factors affecting past performance

impaired IRS’ efforts to improve performance. An important example is

the decline in productivity, as measured by handle time. As discussed

earlier, some IRS officials believe that taxpayers may have received better

 15GAO/GGD-00-161, August 22, 2000.

16Management Reform: Using the Results Act in Quality Management to Improve Federal

Performance (GGD/T-99-151, July 29, 1999).
Page 17 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

service overall if assistors took the time needed to fully resolve taxpayers’

calls. However, IRS had not determined whether overall service improved

as a result of increased handle time. Also discussed earlier was the quality

of guidance provided assistors. IRS did not realize until October 1999 the

extent of problems in the Internal Revenue Manual, too late for fixes to be

made for the 2000 filing season and sometimes leading to inaccurate

answers for taxpayers.

IRS’ “balanced measures” performance management system and not the

IRS Restructuring and Reform Act of 1998 was the basis for IRS restricting

the use of productivity data to evaluate employee performance.17 The Act,

and subsequent regulation, prohibited supervisors from using records of

tax enforcement results, or other quantity measures, to impose production

quotas on or evaluate employees that make judgments about the

enforcement of tax laws.18 When designing and implementing the balanced

measures system, IRS management decided to prohibit telephone

assistance supervisors from using productivity data when evaluating all

assistors, even those that do not make tax enforcement judgments. The

prohibition was intended to promote a more balanced focus by assistors

on efficiency, quality, and service.

According to Organizational Performance Division officials, the balanced

measures system does not prohibit supervisors from using productivity

data to monitor employee performance. However, it requires supervisors

to “get behind the numbers” and base discussions and evaluations of

employee performance solely on the direct review of employees’ work.

Officials said IRS’ design of the balanced measures system was heavily

influenced by IRS’ environment in 1997 and 1998, during which IRS was

under intense pressure from Congress, the administration, and

stakeholders to improve service to taxpayers. The National Performance

Review Customer Service Task Force and National Commission on

Restructuring the IRS had found that IRS’ overall environment and

performance measurement focused on productivity to the detriment of

 17IRS’ balanced measure system is its approach to evaluating employee and organizational

performance; it is designed to balance customer satisfaction, employee satisfaction, and

business results.

18Records of tax enforcement results include such things as number of dollars collected or

number of seizures conducted, while quantity measures are outcome-neutral and include

such things as number of cases closed, time spent assisting taxpayers, or time spent per

case.

IRS’ Performance

Management System

Restricts the Use of

Productivity Data
Page 18 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

service to taxpayers, making employees strive to meet short-term

performance and efficiency goals rather than have a balanced focus on

efficiency, quality, and taxpayer service. IRS officials said the

overemphasis on level of service and other productivity measures had

resulted in employees perceiving that productivity was more important

than quality, so assistors hurried through telephone calls and served

taxpayers poorly, rather than taking the time necessary to give the

taxpayer full, quality service. Also, officials said supervisors tended to

consider measures as ends in themselves, rather than determining the

causes behind employee performance and taking action to improve

performance.

IRS must significantly improve telephone assistance if it is to meet its longterm goal of providing world-class customer service to the tens of millions

of taxpayers that call. While IRS has undertaken efforts to analyze its

performance and identify ways to improve, these efforts have been

incomplete. IRS’ analyses did not cover all of the key management

decisions and other key factors that affect telephone performance.

Designing and conducting a comprehensive analysis of the key

management decisions and other key factors that affect telephone

performance in each filing season will be a difficult task because the

factors that affect performance are multiple and interrelated. However,

without a more comprehensive analysis of the factors that affect

performance, IRS management lacks the information it needs to make

decisions to improve performance.

We recommend that the IRS Commissioner ensure, as part of its analysis

of telephone assistance performance each filing season, that IRS take into

account all key management decisions and other key factors that can

affect performance, such as implementing 24-hour, 7-day assistance and

the decline in assistor productivity, to determine their impact on the

quality of service and to make improvements.

The Commissioner of Internal Revenue provided written comments on a

draft of this report in an April 2, 2001, letter, which is reprinted in

appendix I. The Commissioner agreed with our assessment of IRS’

telephone performance during the 2000 filing season and with our

recommendation. The Commissioner stated that the assessment of key

management decisions and direction should be fully integrated into both

the planning process and performance review. He recognized that IRS

Conclusion

Recommendation for

Executive Action

Agency Comments

and Our Evaluation
Page 19 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

needed to improve its performance analysis to take into account all key

management decisions and other factors that can affect performance. He

stated that this would be done as a part of IRS’ annual filing season

evaluation.

The Commissioner again expressed concern with our comparison of IRS'

performance in the 2000 filing season with its performance in the 1998

filing season, commenting that “comparisons to 1998 are not valid due to

the changes made to accommodate our technological advance to a

national networked system.” As stated in our evaluation of the

Commissioner’s comments on our earlier report,19 we believe it is

appropriate to compare IRS’ performance before and after such

operational changes. The changes made after 1998 were intended to

improve IRS’ telephone service. The only way to tell if service improved is

to compare performance levels before and after the changes.

As agreed with your office, unless you publicly announce its contents

earlier, we plan no further distribution of this report until 30 days from the

date of this letter. At that time, we will send copies to Representative

William J. Coyne, Ranking Minority Member of the Subcommittee;

Representative William Thomas, Chairman, and Representative Charles B.

Rangel, Ranking Minority Member, Committee on Ways and Means; the

Honorable Paul H. O’Neill, Secretary of the Treasury; the Honorable

Charles O. Rossotti, Commissioner of Internal Revenue; and the

Honorable Mitchell E. Daniels, Jr., Director, Office of Management and

Budget. We will also make copies available to others upon request.

 19Tax Administration: Assessment of IRS' 2000 Tax Filing Season (GAO-01-158, Dec. 22,

2000).
Page 20 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

If you have any questions or would like additional information, please call

James R. White at (202) 512-9110 or Carl Harris at (404) 679-1900. Key

contributors to this report are Ronald W. Jones, Julie Schneiberg, and

Sally Gilley.

Sincerely yours,

James R. White

Director, Tax Issues
Appendix I: Comments From the Internal

Revenue Service

Page 21 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

Appendix I: Comments From the Internal

Revenue Service
Appendix I: Comments From the Internal

Revenue Service

Page 22 GAO-01-189 Quality of Service Mixed in 2000 Filing Season
Appendix I: Comments From the Internal

Revenue Service

(268906) Page 23 GAO-01-189 Quality of Service Mixed in 2000 Filing Season

The first copy of each GAO report is free. Additional copies of reports are

$2 each. A check or money order should be made out to the

Superintendent of Documents. VISA and MasterCard credit cards are also

accepted.

Orders for 100 or more copies to be mailed to a single address are

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

Orders by visiting:

Room 1100

700 4th St., NW (corner of 4th and G Sts. NW)

Washington, DC 20013

Orders by phone:

(202) 512-6000

fax: (202) 512-6061

TDD (202) 512-2537

Each day, GAO issues a list of newly available reports and testimony. To

receive facsimile copies of the daily list or any list from the past 30 days,

please call (202) 512-6000 using a touchtone phone. A recorded menu will

provide information on how to obtain these lists.

Orders by Internet

For information on how to access GAO reports on the Internet, send an email message with “info” in the body to:

Info@www.gao.gov

or visit GAO’s World Wide Web home page at:

http://www.gao.gov

Contact one:

• Web site: http://www.gao.gov/fraudnet/fraudnet.htm

• E-mail: fraudnet@gao.gov

• 1-800-424-5454 (automated answering system)

 (268906)

Ordering Information

To Report Fraud,

Waste, and Abuse in

Federal Programs