Report to the Chairman, Subcommittee

on Oversight, Committee on Ways and

Means, House of Representatives

United States General Accounting Office

GAO

November 2002 TAX

ADMINISTRATION

IRS Needs to Further

Refine Its Tax Filing

Season Performance

Measures

GAO-03-143
Why GAO Did This Study

The tax filing season, roughly

January 1 through April 15, is

when most taxpayers file their

returns, receive refunds, and call

or visit IRS offices or the IRS

Web site with questions. To

provide better information about

the quality of filing season

services, IRS is revamping its

suite of filing season

performance measures. Because

the new measures are part of a

strategy to improve service and

because filing season service

affects so many taxpayers, GAO

was asked to assess whether the

new measures have the four

characteristics of successful

performance measures

graphically depicted below.

November 2002

TAX ADMINISTRATION

IRS Needs to Further Refine Its Tax Filing

Season Performance Measures

The full report, including GAO's objectives, scope, methodology, and analysis is available at www.gao.gov/cgi-bin/getrpt?GAO-03-143. For additional

information about the report, contact James White, 202-512-9110 or WhiteJ@gao.gov.

Highlights of GAO- 03-143, a report to the Subcommittee on Oversight, Committee on Ways and

Means, House of Representatives

What GAO Recommends

GAO is making recommendations

to the Commissioner of Internal

Revenue directed at taking

actions to better ensure that IRS

validates the accuracy of data

collection methods for several

measures; modifies the formulas

used to compute various

measures; and adds certain

measures, such as cost of service,

to its suite of measures.

Of GAO’s 18 recommendations,

IRS agreed with 12 and discussed

actions that had been taken or

would be taken to implement

them. For 2 of those 12, the

actions discussed by IRS did not

fully address GAO’s concerns. IRS

did not agree with the other 6

recommendations.

United States General Accounting Office

What GAO Found

In assessing 53 performance measures across IRS’s four program areas,

GAO found that IRS has made significant efforts to improve its

performance measurement system. Many of the measures satisfied some

of the four key characteristics of successful performance measures

established in earlier GAO work. Although improvements are ongoing,

GAO identified instances where measures showed weaknesses including

the following: (1) The objectivity and reliability of some measures could

be improved so that they will be reasonably free from significant bias and

produce the same result under similar circumstances. For example,

survey administrators may notify Telephone Assistance’s customer

service representatives (CSR) too soon that their call was selected to

participate in the customer satisfaction survey, which could bias CSR

behavior towards taxpayers and adversely affect the measure’s

objectivity. In addition, the measure Electronic Filing and Assistance

uses to determine the number of Web site hits was not reliable because it

did not represent the actual number of times the Web site is accessed.

(2) The clarity of some performance information was affected when that

measure’s definition and formula were not consistent. For example, the

definition for “CSR response level” measure is the percentage of callers

who receive service from a CSR within a specified period of time, but the

measure did not include callers who received a busy signal or hung up.

(3) Some suites of measures did not cover governmentwide priorities

such as quality, timeliness, and cost of service. For example, Field

Assistance was missing measures for timeliness and cost of service.

Performance Measures Should Have Four Characteristics

Field assistance

Telephone 

assistance

Submission

processing

Electronic filing

 and assistance

www.irs.gov

Performance measures should

Key tax filing

season programs

Cover multiple priorities

Demonstrate results

Provide useful information

for decision making

Be limited to the vital few

1040

Source: GAO.

G A O

Accountability Integrity Reliability

Highlights
Page i GAO-03-143 Tax Filing Performance Measures

Letter 1

Results in Brief 3

Background 6

Scope and Methodology 12

Filing Season Performance Measures Have Many of the Attributes

of Successful Measures, but Further Enhancements Are Possible 14

Conclusions 36

Recommendations for Executive Action 37

Agency Comments and Our Evaluation 40

Appendix I Expanded Explanation of Our Attributes and

Methodology for Assessing IRS’s Performance

Measures 45

Appendix II The 53 IRS Performance Measures Reviewed 54

Appendix III Comments from the Internal Revenue Service 71

GAO Comments 80

Appendix IV GAO Contacts and Staff Acknowledgments 81

GAO Contacts 81

Acknowledgments 81

Bibliography 82

Related Products 83

Tables

Table 1. Key Attributes of Successful Performance Measures 3

Table 2: Overview of Our Assessment of Telephone Assistance

Measures 16

Table 3: Overview of Our Assessment of Electronic Filing and

Assistance Measures 22

Contents
Page ii GAO-03-143 Tax Filing Performance Measures

Table 4: Overview of Our Assessment of Field Assistance Measures 27

Table 5: Overview of Our Assessment of Submission Processing

Measures 33

Table 6: Telephone Assistance Performance Measures 54

Table 7: Electronic Filing and Assistance Performance Measures 61

Table 8: Field Assistance Performance Measures 64

Table 9: Submission Processing Performance Measures 68

Figures

Figure 1: IRS’s Mission and the Link between Its Strategic Goals

and the Elements of Its Balanced Measurement System 7

Figure 2: Linkage from IRS Mission to Operating Unit Measure and

Target 9

Figure 3: Performance Measures Should Have Four Characteristics 10

Figure 4: Example of Relationship among Field Assistance Goals

and Measures 28

Abbreviations

CQRS Centralized Quality Review Site

CSR customer service representative

GPRA Government Performance and Results Act of 1993

IRS Internal Revenue Service

Q-Matic Queuing Management System

TAC Taxpayer Assistance Center

W&I Wage and Investment
Page 1 GAO-03-143 Tax Filing Performance Measures

November 22, 2002

The Honorable Amo Houghton

Chairman, Subcommittee on Oversight

Committee on Ways and Means

House of Representatives

Dear Mr. Chairman:

For most taxpayers, their only contacts with the Internal Revenue Service

(IRS) are associated with the filing of their individual income tax returns.

Most taxpayers file their returns between January 1 and April 15, which is

generally referred to as the “filing season.”1

 In addition to the filing itself,

which can be on paper or electronic, these contacts generally involve

millions of taxpayers seeking help from IRS by calling one of IRS’s toll-free

telephone numbers, visiting one of IRS’s field assistance centers, or

accessing IRS’s Web site on the Internet (www.irs.gov). Between January

1 and July 13, 2002, for example, IRS received about 105 million calls for

assistance over its toll-free telephone lines.2

As part of a much larger effort to modernize and become more responsive

to taxpayers, IRS is revamping how it measures and reports its filing

season performance. The new filing season performance measures are to

balance customer satisfaction, employee satisfaction, and business results,

such as the quality of answers to taxpayer inquiries and the timeliness of

refund issuance. IRS intends to use the balanced measures to make

managers and frontline staff more accountable for improving filing season

performance.

Because so many taxpayers are affected by IRS’s performance during the

filing season and because the revamped measures are part of a strategy to

improve performance, you asked us to review IRS’s new set of filing

 1

Although April 15 is generally considered the end of the filing season, millions of taxpayers

get extensions from IRS that allow them to delay filing until as late as October 15.

2

IRS tracks its performance in providing filing season-related telephone service through

mid-July instead of April because it receives many filing season-related calls after April 15

from taxpayers who are inquiring about the status of their refunds or responding to notices

they received from IRS related to returns they filed.

United States General Accounting Office

Washington, DC 20548
Page 2 GAO-03-143 Tax Filing Performance Measures

season performance measures. Those measures belong to the four

program areas critical to a successful filing season: telephone assistance;

electronic filing and assistance; field assistance; and the processing of

returns, refunds, and remittances (referred to as “submission processing”).

Specifically, our objective was to assess whether the key performance

measures IRS uses to hold managers accountable in the four program

areas had the characteristics of a successful performance measurement

system.

Previous GAO work indicated agencies successful in measuring

performance had performance measures that demonstrate results, are

limited to the vital few, cover multiple priorities, and provide useful

information for decision making.3

 To determine whether IRS’s filing

season performance measures satisfy these four general characteristics,

we assessed the measures using nine specific attributes.4

 Earlier GAO

work cited these specific attributes as key to successful performance

measures. Table 1 is a summary of the nine attributes, including the

potentially adverse consequences if they are missing. All attributes are not

equal and failure to have a particular attribute does not necessarily

indicate that there is a weakness in that area or that the measure is not

useful; rather, it may indicate an opportunity for further refinement. An

expanded explanation of the nine attributes is included in appendix I.

 3

Some earlier work includes U.S. General Accounting Office, Executive Guide: Effectively

Implementing the Government Performance and Results Act, GAO/GGD-96-118

(Washington, D.C.: June 1996) and U.S. General Accounting Office, The Results Act: An

Evaluator’s Guide to Assessing Agency Annual Performance Plans, GAO/GGD-10.1.20

(Washington, D.C.: Apr. 1998).

4

The four characteristics are overarching, thus there is not necessarily a direct link

between any one attribute and any one characteristic.
Page 3 GAO-03-143 Tax Filing Performance Measures

Table 1. Key Attributes of Successful Performance Measures

Source: Summary of information in appendix I.

We shared these attributes with various IRS officials, who generally agreed

with their relevance. As discussed in greater detail in the separate scope

and methodology section of this report, we took many steps to validate

and ensure consistency in our application of the attributes.

We testified before the Subcommittee on Oversight on some of the interim

results of our assessment in April 2002.5

In assessing 53 performance measures across four of IRS’s key filing

season program areas, we found that the measures satisfied many of the

nine attributes of successful performance measures previously listed in

table 1. As part of its agencywide reorganization, IRS has made significant

efforts to improve its performance measurement system, which is to

 5

U.S. General Accounting Office, Internal Revenue Service: Assessment of Budget Request

for Fiscal Year 2003 and Interim Results of 2002 Tax Filing Season, GAO-02-580T

(Washington, D.C.: Apr. 9, 2002).

Results in Brief

Attributes

Linkage

Clarity

Measurable target

Objectivity

Reliability

Core program activities

Limited overlap

Balance

Governmentwide

priorities

Measure is aligned with division and agencywide goals and 

mission and clearly communicated throughout the organization.

Measure is clearly stated and the name and definition are

consistent with the methodology used to calculate it.

Measure has a numerical goal.

Measure is reasonably free from significant bias or manipulation.

Measure produces the same result under similar conditions.

Measures cover the activities that an entity is expected to

perform to support the intent of the program.

Measure should provide new information beyond that

provided by other measures.

Balance exists when a suite of measures ensures that

an organization's various priorities are covered.

Each measure should cover a priority such as quality, 

timeliness, and cost of service.

Behaviors and incentives created by measures do not support 

achieving division or agencywide goals or mission.

Data could be confusing and misleading to users.

Can not tell whether performance is meeting expectations.

Performance assessments may be systematically over- or 

understated.

Reported performance data is inconsistent and adds 

uncertainty.

Not enough information available in core program areas to

managers and stakeholders.

Manager may have to sort through redundant, costly information

that does not add value. 

Lack of balance could create skewed incentives when measures

over-emphasize some goals.

A program's overall success is at risk if all priorities are not

addressed.

Definitions Potentially adverse consequences of not meeting attribute
Page 4 GAO-03-143 Tax Filing Performance Measures

provide useful information about how well IRS performed in achieving its

goals. The improvement of this system is an ongoing process where, in

some cases, IRS is only beginning to collect baseline information on which

to form targets and develop other measures that would provide better

information to evaluate performance results. Despite IRS’s progress, we

identified instances in all four program areas where the individual

measures or suites of measures did not meet some of our nine attributes.

Some of these instances represent opportunities for IRS to further refine

its measures.

All of the 15 telephone assistance measures had some of the attributes of

successful performance measures. Of the more significant problems, five

measures had either clarity or reliability problems and one had an

objectivity problem. For example,

• five measures did not provide managers and other stakeholders with

clear information about the program’s performance. For example, the

definition for “customer service representative (CSR) response level” is

the percentage of callers who receive service from a CSR within a

specified period of time, but the formula did not include callers who

received a busy signal or hung up; this limitation could lead managers

and other stakeholders to conclude that IRS is providing significantly

better service than it is.

All of the 13 electronic filing and assistance performance measures

fulfilled some of the 9 attributes. The most significant problems involved

changing targets, objectivity, and missing measures. For example,

• electronic filing and assistance changed the targets for two of its

measures during fiscal year 2001, which could distort the assessment of

performance because what was to be observed changed. For example,

it changed the target for the “number of 1040 series returns filed

electronically” from 42 million to 40 million because midyear data

indicated that 42 million 1040 series returns were not going to be filed

electronically. Because of the subjective considerations involved,

changing the target in this situation also affected the measure’s

objectivity.

All of field assistance’s 14 performance measures satisfied some of the

attributes. Many of the more important problems involved clarity and

reliability. In addition, some measures were missing, which could cause an

emphasis on some program goals at the expense of a balance among all

goals. For example,
Page 5 GAO-03-143 Tax Filing Performance Measures

• the methods used to track workload volume and staff hours expended

required manual input that is subject to errors and inconsistencies,

which could affect data accuracy and thus the reliability of 8 of field

assistance’s 14 measures.

• Field assistance did not have timeliness, efficiency, or cost of service

measures.

Many of the 11 submission processing measures had the attributes of

successful performance measures. Some of the more significant problems

related to clarity and reliability. For example,

• one measure—“productivity”—was unclear because it is a compilation

of different types of work IRS performs in processing returns,

remittances, and refunds and issuing notices and letters. Managers told

us that they needed specific information related to their own

operations and that the measure’s methodology was difficult to

understand.

In all four program areas, we were unable, because of documentation

limitations, to verify the linkages among IRS’s goals and measures. Among

other things, such linkages provide managers and staff with a road map

that shows how their day-to-day activities contribute to attaining

agencywide goals.

We are making recommendations to the Commissioner of Internal

Revenue directed at taking actions to better ensure that IRS’s filing season

measures have the four characteristics of successful performance

measures. For example, we are recommending that IRS modify the

formulas used to compute various measures; validate the accuracy of data

collection methods for several measures; and add certain measures such

as cost of service, to its suite of measures.

We requested comments on a draft of this report from the Commissioner

of Internal Revenue. We received written comments, which are reprinted

in appendix III. In his comments, the Commissioner agreed that there were

opportunities to refine some performance measures and said that our

observation about the ongoing nature of the performance measurement

process was on target. The Commissioner agreed with 12 of our 18

recommendations and discussed actions that had been taken or would be

taken to implement them. In 2 of those cases, the actions discussed by IRS

did not fully address our concerns. The Commissioner disagreed with the
Page 6 GAO-03-143 Tax Filing Performance Measures

other 6 recommendations. We discuss the Commissioner’s comments in

the “Agency Comments and Our Evaluation” section of the report.

In keeping with the Government Performance and Results Act of

1993 (GPRA),6

 IRS revamped its set of filing season performance measures

as part of a massive, ongoing modernization effort. Congress mandated the

modernization effort in the IRS Restructuring and Reform Act of 19987

 and

intended that IRS would better balance service to taxpayers with

enforcement of the tax laws. To implement the modernization mandate,

the Commissioner of Internal Revenue developed a strategy composed of

five interdependent components. One of those components is the

development of balanced performance measures.8

Balanced measures are to emphasize accountability for achieving specific

results and to reflect IRS’s priorities, which are articulated in its mission

and its three strategic goals—top quality service to all taxpayers through

fair and uniform application of the law, top quality service to each

taxpayer in every interaction, and productivity through a quality work

environment. IRS has defined three elements of balanced measures—

(1) customer satisfaction, (2) employee satisfaction, and (3) business

results (quality and quantity measures)—to ensure balance among its

priorities. Figure 1 shows IRS’s mission and the link between its strategic

goals and the three elements of IRS’s balanced measurement system.

 6

GPRA, P.L. 103-62, was enacted to hold federal agencies accountable for achieving

program results. IRS’s balanced measurement system is consistent with the intent of

GPRA.

7

IRS’s Restructuring and Reform Act of 1998, P.L. 105-206, was enacted on July 22, 1998,

and calls for broad reforms in areas such as the structure and management of IRS,

electronic filing, and taxpayer protection and rights.

8

The other components include revamped business practices, customer-focused operating

divisions, management roles with clear responsibility, and new technology.

Background
Page 7 GAO-03-143 Tax Filing Performance Measures

Figure 1: IRS’s Mission and the Link between Its Strategic Goals and the Elements

of Its Balanced Measurement System

Source: GAO depiction of information in IRS Publication 3561 and IRS’s Progress Report (December

2001).

IRS intends to use the balanced measures to make managers and frontline

staff more accountable for improving filing season performance. We

reviewed the performance measures in the four programs areas that

interact with taxpayers the most during the filing season—telephone

assistance, electronic filing and assistance, field assistance, and

submission processing. Each of these program areas is part of IRS’s Wage

and Investment (W&I) operating division, which generally serves

taxpayers whose only income is from wages and investments.9

 Although

IRS had measures of performance prior to the reorganization, IRS

managers have spent much effort to revamp the filing season performance

measures since that time.

An important aspect of IRS’s progress in the challenging task of improving

its performance measures was the development of a new Strategic

Planning, Budgeting, and Performance Management process in 2000. As

part of that process, IRS prepares an annual Strategy and Program Plan

 9

As part of IRS’s reorganization that took effect in October 2000, IRS established four

operating divisions that serve specific groups of taxpayers. The four divisions are (1) Wage

and Investment, (2) Small Business and Self-Employed, (3) Large and Mid-Size Businesses,

and (4) Tax Exempt and Government Entities.
Page 8 GAO-03-143 Tax Filing Performance Measures

that communicates some of the various levels of IRS’s goals (e.g., strategic

goals, operating division goals) and many performance measures.10

Although the Strategy and Program Plan does not document all the

linkages among the various goals and performance measures, figure 2 is an

example we developed to demonstrate the complete relationship from the

agency level mission down to the operating unit’s measures and targets.

 10The Strategy and Program Plans we used in our analysis had actual performance

information for part of the current fiscal year and planning information for the current and

two subsequent fiscal years. An IRS manager said the agency plans to stop including actual

information in Strategy and Program Plans prepared after fiscal year 2002.
Page 9 GAO-03-143 Tax Filing Performance Measures

Figure 2: Linkage from IRS Mission to Operating Unit Measure and Target

Source: GAO Analysis of IRS’s Strategy and Program Plan (October 29, 2001), the W&I Business

Performance Review (January 2002), IRS’s Progress Report (December 2001) and IRS Publication

3561.

The Strategy and Program Plan is an important document because the

Commissioner holds IRS managers accountable for the results of the

performance measures contained within it. In addition, many of the

measures within the document are presented to outside stakeholders, such

as Congress and the public, as key indicators of IRS’s performance. The

Strategy and Program Plan is the source of the 53 measures we reviewed

in the four programs.

To meet taxpayer demands for timely, accurate, and efficient services

Hire, train, and organize customer service representatives

Top quality service to each taxpayer in every interaction

To provide America's taxpayers top quality service by helping them understand and meet their 

tax responsibilities by applying the tax law with integrity and fairness to all

IRS Mission

IRS Strategic Goal

Telephone Operational Priority

Wage and Investment Operating Division Goal

Track hiring and recruitment and report to headquarters monthly during the filing season

Telephone Improvement Project

Toll-free quality (accounts and tax law)

Accounts target (2001) 67% accuracy Tax law target (2001) 74% accuracy

Telephone Performance Measure and Target
Page 10 GAO-03-143 Tax Filing Performance Measures

As we discussed in our June 1996 guide on implementing GPRA,11 agencies

that were successful in measuring performance strived to establish

performance measures that were based on four general characteristics.

Those four characteristics are shown in figure 3 as applicable to the four

filing season programs we reviewed and are described in more detail

following the figure.

Figure 3: Performance Measures Should Have Four Characteristics

Source: GAO.

 11GAO/GGD-96-118.

Field assistance

Telephone 

assistance

Submission

processing

Electronic filing

 and assistance

www.irs.gov

Performance measures should

Key tax filing

season programs

Cover multiple priorities

Demonstrate results

Provide

useful information

for decision making

Be limited to the vital few

1040
Page 11 GAO-03-143 Tax Filing Performance Measures

Demonstrate results. Performance measures should show an

organization’s progress towards achieving an intended level of

performance or results. Specifically, performance goals establish intended

performance, and measures can be used to assess progress towards

achieving those goals.

Be limited to the vital few. Limiting measures to core program activities

enables managers and other stakeholders to assess accomplishments,

make decisions, realign processes, and assign accountability without

having an excess of data that could obscure rather than clarify

performance issues.

Cover multiple priorities. Performance measures should cover many

governmentwide priorities, such as quality, timeliness, cost of service,

customer satisfaction, employee satisfaction, and outcomes. Performance

measurement systems need to include incentives for managers to strike

the difficult balance among competing interests. One or two priorities

should not be overemphasized at the expense of others. IRS’s history

shows why this balance is important. Because of its emphasis on achieving

certain numeric targets, such as the amount of dollars collected, IRS failed

to adequately consider other priorities, such as the fair treatment of

taxpayers.

Provide Useful Information for Decision Making. Performance measures

should provide managers and other stakeholders timely, action-oriented

information in a format that helps them make decisions that improve

program performance. Measures that do not provide managers with useful

information will not alert managers and other stakeholders to the

existence of problems nor help them respond when problems arise.

On the basis of these four characteristics of successful performance

measures, we used various performance management literature to develop

a set of nine specific attributes that we used as criteria for assessing IRS’s

filing season performance measures. The nine attributes are linkage,

clarity, measurable target, objectivity, reliability, core program activities,

limited overlap, balance, and governmentwide priorities. Appendix I

describes these attributes in more detail.
Page 12 GAO-03-143 Tax Filing Performance Measures

As previously mentioned, we focused our work on four key filing season

programs—telephone assistance, electronic filing and assistance, field

assistance, and submission processing—within W&I. IRS officials

identified the performance measures in the Strategy and Program Plan to

be the highest, most comprehensive level of measures for which they are

accountable. After discussions with IRS, we decided to review all

53 measures in the Strategy and Program Plan relating to the four filing

season programs. We used W&I’s draft fiscal year 2001 – 2003 Strategy and

Program Plan (dated July 25, 2001) to conduct our review and updated

relevant information with the final plan (dated October 29, 2001).

Appendix II describes each measure we reviewed in the four program

areas and provides other relevant information, such as targets and

potential weaknesses.

Our review focused on whether IRS’s new set of filing season performance

measures had the characteristics of a successful performance

measurement system (i.e., demonstrated results, were limited to the vital

few, covered multiple priorities, and provided useful information for

decision making). For use as criteria in assessing the measures, and as

detailed in appendix I, we identified nine attributes of performance

measures from various sources, such as earlier GAO work, Office of

Management and Budget Circular No. A-11,12 GPRA, and IRS’s handbook

on Managing Statistics in a Balanced Measures System.13 We shared our

attributes with IRS officials from various organizations that have a role in

developing or monitoring performance measures. Those units included

IRS’s Organizational Performance Division and several W&I units, such as

Strategy and Finance; Planning and Analysis; Customer Account Services;

and Communications, Assistance, Research, and Education. Officials in

these units generally agreed with the relevance of our attributes and our

assessment approach.

We applied the 9 attributes to the 53 filing season measures in a systematic

manner, but some judgment was required. To ensure consistency and

reliability in our application of the attributes, we had one staff person

responsible for each of the four areas. That staff person prepared the

 12Office of Management and Budget, Preparation and Submission of Budget Estimates,

Circular No. A-11, Revised. Transmittal Memorandum No. 72 (Washington, D.C.:

July 12, 1999).

13IRS, Managing Statistics in a Balanced Measures System, Handbook 105.4 (Washington,

D.C.: Oct. 1, 2000).

Scope and

Methodology
Page 13 GAO-03-143 Tax Filing Performance Measures

initial analysis and at least two other staff reviewed those detailed results.

Several staff reviewed the results for all four areas. We did not do a

detailed assessment of IRS’s methodology for calculating the measures,

but looked only at methodological issues as necessary to assess whether a

particular measure met the overall characteristics of a successful

performance measure.

In applying the attributes, we analyzed numerous pieces of

documentation, such as IRS’s Congressional Budget Justification, Annual

Performance Plan, and data dictionary,14 and many other reports and

documents dealing with the four IRS programs, goals, performance

measures, and improvement initiatives. We interviewed IRS officials at

various levels within telephone assistance, electronic filing and assistance,

field assistance, and submission processing to understand the measures,

their methodology, and their relationship to goals, among other things. We

also interviewed officials from various IRS organizations that are involved

in managing, collecting, and/or using performance data, such as the

Organizational Performance Division; Strategy and Finance; Customer

Account Services; Statistics of Income; and the Centralized Quality Review

Site; and a representative of an IRS contractor, Pacific Consulting Group,

responsible for analyzing and reporting the results of telephone

assistance’s customer satisfaction survey. Appendix I provides more detail

on the nine attributes we used, including explanations and examples of

each attribute and information on our methodology for assessing each

attribute.

We conducted our review in Atlanta, Ga.; Washington, D.C.; Cincinnati,

Ohio; and Memphis, Tenn. from September 2001 to September 2002 in

accordance with generally accepted government auditing standards.

 14The data dictionary is an IRS document that provides information on performance

measures, such as the measure’s name, description, and methodology.
Page 14 GAO-03-143 Tax Filing Performance Measures

The 53 filing season performance measures included in our review have

many of the attributes of successful performance measures, as detailed in

appendix I. For example, in all four of the program areas we reviewed,

most measures covered the core activities of each program and had

targets in place. In addition, IRS had several on-going initiatives aimed at

improving its measures, such as telephone assistance’s efforts to revamp

all aspects of its quality measures.

At the same time, however, the measures did not satisfy all the attributes,

indicating the potential for further enhancements. The nine attributes we

used to assess each measure are not equal and failure to have a particular

attribute does not necessarily indicate that there is a weakness in that

area. In some cases, for example, a measure may not have a particular

attribute because benchmarking data are being collected or a measure is

being revised. Likewise, a noted weakness, such as a measure not having

clarity or being reliable, does not mean that the measure is not useful. For

example, telephone assistance’s “CSR level of service” measure does not

meet our clarity attribute because its name and definition indicate that

only calls answered by CSRs are included, but its formula includes some

calls answered by automation. This defect currently does not impair the

measure’s usefulness because the number of automated calls is fairly

insignificant. Other weaknesses, however, could lead managers or other

stakeholders to draw the wrong conclusions, overlook the existence of

problems, or delay resolving problems. For example, electronic filing and

assistance’s “number of IRS digital daily Web site hits” measure was not

considered clear or reliable because it systematically overstates the

number of times the Web site is accessed. In total, therefore, the

weaknesses identified should be considered areas for further refinement.

Such refinements are not expected to be costly or involve significant

additional effort on the part of IRS because in many instances our

recommendations only include modifications or increased rigor to

procedures or processes already in place.

The rest of this report discusses the results of our analysis for each of the

four program areas—telephone assistance, electronic filing and assistance,

field assistance, and submission processing.

Filing Season

Performance

Measures Have Many

of the Attributes of

Successful Measures,

but Further

Enhancements Are

Possible
Page 15 GAO-03-143 Tax Filing Performance Measures

As shown in table 2, all 15 of IRS’s telephone performance measures have

some of the attributes of successful performance measures.15 However, as

summarized in this section, the measures have several shortcomings. For

example, we identified opportunities to improve the clarity of five

measures and the reliability of five other measures. Table 6 in appendix II

has more detailed information about each telephone measure, including

any weaknesses we identified and any recommendations for improvement.

 15IRS deleted its “automated completion rate” measure in the 2002 Strategy and Program

Plan and now has 14 telephone measures. However, IRS still tracks that measure.

Telephone Assistance

Measures
Page 16 GAO-03-143 Tax Filing Performance Measures

Table 2: Overview of Our Assessment of Telephone Assistance Measures

Note: A check mark denotes that the measure has the attribute.

a

We were unable to verify the linkages between goals and measures because of insufficient

documentation.

b

Core program activities of telephone assistance are to provide timely and accurate assistance to

taxpayers with inquiries about the tax law and their accounts.

c

IRS also refers to CSRs as assistors.

d

IRS considers that these measures are balanced because they address priorities, such as customer

and employee satisfaction and business results. However, including measures, such as cost of

service, could improve the balance of telephone assistance’s program priorities.

Source: GAO analysis.

Although telephone assistance management stated that their goals and

measures generally aligned, we were unable to verify this because no

documentation shows the complete relationship. For example, some

documentation may show a link from a measure to an agencywide goal,

but the operating division level goals were omitted. When we attempted to

create the linkage ourselves, we found it difficult to determine how some

measures related to the different agencywide and operating division goals.

When we asked some IRS officials to describe the complete link, they too

had a difficult time and were uncertain of some connections.

Telephone assistance managers stated that staff received performance

management training that should help them to understand their role in

No Documentation Shows the

Complete Linkage between

Agencywide Goals and

Telephone Measures

Total automated calls answered

Customer Service Representative 

(CSR)c calls answered

CSR level of service

Toll-free customer satisfaction

Toll-free tax law quality

Toll-free accounts quality

Average handle time

Automated completion rate

CSR services provided

Toll-free tax law correct response rate

Toll-free account correct response rate

Toll-free timeliness

Toll-free employee satisfaction

CSR response level

Average speed of answer

Linkagea Clarity

Measurable 

target Objectivity Reliability

Core

program

activitiesb

Limited

overlap

Governmentwide

priorities

Telephone assistance attributes

Telephone 

assistance measures Balance

This attribute 

applies to 

the overall 

suite of 

measures 

rather than 

to the 

measures

individually.d
Page 17 GAO-03-143 Tax Filing Performance Measures

helping the organization achieve its goals. However, having clear and

complete documentation would provide evidence that linkages exist and

help prevent misunderstandings. When employees do not understand the

relationship between goals and measures, they may not understand how

their work contributes to agencywide efforts and, thus, goals may not be

achieved.

Ten of the 15 measures have clarity (e.g., “automated calls answered”

clearly describes the count of all toll-free calls answered at customer

service sites by automated service). However, five measures contain or

omit certain data elements that can cause managers or other stakeholders

to misunderstand the level of performance. For example, the “CSR

response level,” is defined as the percentage of callers who started

receiving service from a CSR within a specified period of time. However,

this may not reflect the real customer experience at IRS because the

formula for computing the measure does not include callers who tried to

reach a CSR but did not, such as callers who (1) hung up while waiting to

speak to a CSR, (2) were provided access only to automated services and

hung-up, and (3) received a busy signal.16 (The other four measures, as

noted in table 6 in appendix II, are “CSR level of service,” “automated

completion rate,” “CSR service provided,” and “toll-free customer

satisfaction.”)

Measures that do not provide clear information about program

performance may affect the validity of managers’ and stakeholders’

assessments of IRS’s performance, possibly leading to a misinterpretation

of results or a failure to take proper action to resolve performance

problems.

Eleven of the 15 measures have numerical targets that facilitate the future

assessment of whether overall goals and objectives were achieved. Of the

four measures with no targets, three were measures for which IRS was

collecting data for use in developing first-time targets and one was a

measure (“automated completion rate”) that IRS was no longer tracking in

the Strategy and Program Plan. Although we generally disagree with the

removal of the “automated completion rate” measure from the Strategy

and Program Plan, as described in an upcoming section, not having targets

in these instances is reasonable.

 16There were about 30 million of these calls during in fiscal year 2001, which can have a

significant impact on the “CSR response level” measure.

Most Telephone Measures Have

Clarity

Most Telephone Measures Have

Targets
Page 18 GAO-03-143 Tax Filing Performance Measures

IRS determines customer satisfaction with its toll-free telephone

assistance through a survey administered to taxpayers who speak with a

CSR.17 We observed survey collection methods in Atlanta that were not

always objective; that is, the administrators did not always follow

prescribed procedures for selecting calls to participate in the survey. Not

following prescribed procedures produces a systematic bias that could

compromise the randomness of the sample. Also, IRS procedures do not

require that administrators listen to the entire call. Although

administrators are instructed to notify the CSR towards the end of a call

that the call was selected for the survey, this may not occur. If an

administrator begins listening to a call after it has started, it can be

difficult to determine the full nature of the taxpayer’s question and thus

whether the conversation is about to end. As a result, an administrator

could prematurely notify a CSR that the call was selected for the survey,

which could change the CSR’s behavior towards the taxpayer and affect

the results of the survey and the measure. In addition, administrators may

not be able to correctly answer certain questions on the survey, which

could impair any analysis of those answers. We discussed these issues

with a representative of the IRS contractor (Pacific Consulting Group)

responsible for analyzing and reporting the survey results who said that

(1) he was aware of these problems and (2) the same problems existed at

other locations.

IRS has taken corrective action on one of these weaknesses. Because

management decided that the procedures for selecting calls to participate

in the customer satisfaction survey were too difficult to follow, it revised

them. Sites began using the revised sampling procedures in July 2002.

The reliability of telephone assistance’s five quality measures (“toll-free

tax law quality,” “toll-free accounts quality,” “toll-free tax law correct

response rate,” “toll-free account correct response rate,” and “toll-free

timeliness”) is suspect because of potential inconsistencies in data

collection that arise due to differences among individual reviewer’s

judgment and perceptions.18 Although it is not certain how much variation

among reviewers exists, errors could occur throughout data collection and

could affect the results of the measures and conclusions about the extent

to which performance goals have been achieved.

 17CSRs answer about 24 percent of all incoming calls.

18As of January 2002, there were 53 quality reviewers in the Centralized Quality Review

Site: 26 for tax law inquiries, 20 for account inquiries, and 7 others.

Data Collection Methods for

Telephone Assistance’s

Customer Satisfaction Measure

Are Not Always Objective

Reliability of Five Telephone

Quality Measures Is Suspect
Page 19 GAO-03-143 Tax Filing Performance Measures

Reliability and credibility increase when performance data are checked or

tested for significant errors. IRS has conducted consistency reviews in the

past and found problems. It has taken steps to improve consistency, the

most important of which was the establishment of the Centralized Quality

Review Site (CQRS).19 Among other controls within CQRS that are

designed to enhance consistency, reviewers are to receive the same

training and gather to discuss cases where the guidance is not clear. IRS

has conducted one review to determine the effectiveness of CQRS and its

efforts to improve consistency since IRS’s October 2000 reorganization

and continues to find some problems.

At the time of our review, IRS was reviewing the five quality measures as

part of an ongoing improvement initiative. Since that time, it redesigned

many aspects of the measures, including what is measured, how the

measures are calculated, how data are collected, and how people are held

accountable for quality.20 Changes emanating from this initiative may

further enhance consistency.

Telephone assistance’s core program activities are to provide timely and

accurate assistance to taxpayers with inquiries about the tax law and their

accounts. IRS has at least one measure that directly addresses each of

these core activities. For example, “toll-free accounts quality” is a measure

that shows the percentage of accurate responses to taxpayers’ account

related questions.

The amount of overlap that exists between measures is a managerial

decision. Of the 15 telephone measures we reviewed, 10 have at least

partial overlap. For example, both the “CSR response level” and “average

speed of answer” measures attempt to show how long a taxpayer waited

before receiving service, except that the former shows the number of

 19CQRS is responsible for monitoring the accuracy of telephone assistance. It produces

various reports that show call sites what errors CSRs are making so site managers can take

action to reduce those errors.

20IRS significantly modified its five quality measures beginning in October 2002 based on

the results of its initiative, which was aimed at redesigning the way IRS measures quality to

better capture the taxpayer’s experience. Specifically, IRS renamed the toll-free correct

response rate measures for tax law and account inquiries to “customer accuracy” for tax

law or account inquiries. Plans call for the tax quality measures for tax law and account

inquiries to be discontinued, but reported in fiscal year 2003 for trending and comparative

purposes. IRS also eliminated the “toll-free timeliness” measure and replaced it with a new

“quality timeliness” measure. Finally, IRS implemented a new measure called

“professionalism.”

Telephone Measures Cover

Core Program Activities

Some Overlap Exists between

Telephone Assistance

Measures
Page 20 GAO-03-143 Tax Filing Performance Measures

taxpayers receiving service within 30 seconds while the latter shows the

average wait time for all taxpayers. (Table 6 in appendix II has information

on other overlapping measures.)

IRS officials said that overlapping measures can add value to

management’s decision-making process because each measure provides a

nuance that can be missed if both measures were not present. For

example, the “CSR calls answered” measure shows the number of

taxpayer calls answered while the “CSR services provided” measure

attempts to account for situations in which more than one CSR was

involved in handling a single call. At the same time, however, overlapping

measures (1) leave managers to sift through redundant, sometimes costly,

information to determine goal achievement and (2) could confuse outside

stakeholders, such as Congress.

Although we are not suggesting that IRS stop tracking or reporting any of

the overlapping measures, we question whether IRS has limited the

telephone measures included in the Strategy and Program Plan to the vital

few. Telephone officials agreed with this assessment and stated that some

of the overlapping measures will be removed from future Strategy and

Program Plans.

When considering governmentwide priorities, such as quality, timeliness,

cost of service, and customer and employee satisfaction, telephone

assistance is missing two measures—(1) cost of service and (2) a measure

of customer satisfaction for automated services, as described below.

• Cost of Service. According to key legislation21 and accounting

standards,22 agencies should develop and report cost information.

Besides showing financial accountability in the use of taxpayer dollars,

the cost information called for can be used for various purposes, such

as authorizing and modifying programs and evaluating program

performance. IRS does not report the average cost to answer a

taxpayer’s inquiry by telephone. A cost-per-call analysis could provide a

 21The Chief Financial Officer’s Act, P.L. 101-576,underscores the importance of improving

financial management in the federal government. Among other things, it calls for

developing and reporting cost information.

22Statement of Federal Financial Accounting Standard Number 4, “Managerial Cost

Accounting Concepts and Standards for the Federal Government,” is aimed at providing

reliable and timely information on the full cost of federal programs, their activities, and

outputs.

Telephone Measures Do Not

Fully Cover Governmentwide

Priorities
Page 21 GAO-03-143 Tax Filing Performance Measures

link between program goals and costs, as required by GPRA, and help

IRS management and Congress decide about future investments in

telephone assistance. IRS officials said they would like to develop a

cost of services measures and are trying to determine what information

would be meaningful to include or exclude in the calculation.

• Customer Satisfaction for Automated Services. Although IRS

projections show that about 70 percent of its fiscal year 2002 calls

would be handled by automation, it has no survey mechanism in place

to determine taxpayers’ satisfaction with these automated services. IRS

officials agreed this would be a meaningful measure and want to

develop one for the future, but no implementation plans have been

established.

Also, as previously mentioned, IRS has removed the “automated

completion rate” measure from its Strategy and Program Plan. We realize,

as noted in table 6 in appendix II, that this measure has limitations that

need to be addressed. However, because such a large percentage of calls

are handled by automation and because IRS plans to serve even more calls

with automation in the future, re-inclusion of that measure in the Strategy

and Program Plan may be warranted if the associated problems can be

resolved.

Telephone assistance has measures in place for customer satisfaction,

employee satisfaction, and business results and, therefore, IRS considers

the measures balanced. However, including other measures, such as a cost

of service measure, as previously described, could further enhance the

balance of program priorities.

As shown in table 3, all 13 of electronic filing and assistance’s performance

measures have some of the attributes of successful performance

measures. However, as summarized in this section, the measures have

some shortcomings. For example, several of the measures had some

overlap and two measures had shortcomings related to the changing of

targets during the fiscal year. Table 7 in appendix II has more detailed

information about each electronic filing and assistance measure, including

any weaknesses we identified and any recommendations for improvement.

Telephone Measures Are

Balanced

Electronic Filing and

Assistance Measures
Page 22 GAO-03-143 Tax Filing Performance Measures

Table 3: Overview of Our Assessment of Electronic Filing and Assistance Measures

Note: A check mark denotes that the measure has the attribute.

a

We were unable to verify the linkages between goals and measures because of insufficient

documentation.

b

Electronic filing and assistance’s core program activities are to provide individual and business

taxpayers with the capability to transact and communicate electronically with IRS.

c

Electronic filing and assistance measures address most governmentwide priorities, such as quantity,

customer satisfaction, and employee satisfaction; however, they do not cover two important

priorities—quality and cost of service.

Source: GAO analysis.

Electronic filing and assistance’s 13 performance measures are aligned

with IRS’s overall mission and IRS’s strategic goals. However, we were

unable to validate whether the lower level goals, such as electronic filing

and assistance’s operational goals and improvement projects, are linked to

the agencywide strategic level goals and operating division performance

measures because there is not complete documentation available to show

that linkage.

Electronic filing and assistance’s managers stated that goals and measures

generally align and that employee briefings were held to communicate

their goals to the organization. It is essential that all staff be familiar with

IRS’s mission and goals, electronic filing and assistance’s goals and

performance measures, and how electronic filing and assistance

Overall Alignment of Electronic

Filing and Assistance’s Goals

and Measures Not Fully

Documented

Linkagea Clarity

Measurable 

target Objectivity Reliability

Core

program

activitiesb

Limited

overlap

Governmentwide

priorities

Electronic filing and assistance attributes

Balance

Number of 1040 series returns electronically filed

Number of business returns electronically filed

Total number of electronically filed returns

Number of information returns electronically filed

Percent of information returns electronically filed

Percent of individual returns electronically filed

Number of payments received electronically

Percent of payments received

Number of electronic funds withdrawls/credit card 

transactions

Number of IRS digital daily Web site hits 

Number of downloads from "IRS.GOV"

Customer satisfaction-individual taxpayers

Employee satisfaction-electronic filing and 

assistance

Electronic filing and

assistance measures

This attribute 

applies to 

the overall 

suite of 

measures 

rather than 

to the 

measures

individually.c
Page 23 GAO-03-143 Tax Filing Performance Measures

determines whether it is achieving its goals so that staff know how their

day-to-day activities contribute to the goals and IRS’s overall mission.

When this is lacking, priorities may not be clear and staff efforts may not

be tied to goal achievement.

All but one of electronic filing and assistance’s 13 performance measures

had clarity. The “number of IRS digital daily Web site hits” measure, which

is defined as the number of “hits” to IRS’s Web site, is not clear because its

formula counts multiple hits every time a user accesses the site’s home

page and counts a hit every time a user moves to another page on the Web

site. The formula is not consistent with the definition because it does not

represent the actual number of times the Web site is accessed.

In its fiscal year 2003 Annual Performance Plan,23 IRS acknowledged

limitations with this measure as follows.

“ . . . changes in the IRS Web design may cause a decrease in the number of ‘hits’ recorded

in both [fiscal years] 2002 and 2003. This decrease will be due to improved Web site

navigation and search functions, which may reduce the amount of random exploration by

users to find content. The decrease will also be due to better design of the Web pages

themselves that will reduce the number of graphics and other items that are used to create

the Web page, all of which are counted as ‘hits’ when a page is accessed.”

In our report on IRS’s 2001 tax filing season, we recommended that IRS

either discontinue the use of “hits” as a measure of the performance of its

Web site or revise the way “hits” are calculated so that the measure more

accurately reflects usage.24 IRS responded that it should continue to count

“hits” as a measure of the Web site’s performance because “hits” indicate

site traffic and can be used to measure system performance and estimate

system needs. However, officials stated that they could improve their

method of counting “hits” once they had implemented a more

sophisticated, comprehensive Web analytical program. According to

electronic filing and assistance officials, IRS introduced its redesigned

Web site in January 2001 and implemented a new analytical program, but

“hits” are still being calculated the same way.

 23The Annual Performance Plan is a key document IRS produces each year to comply with

the requirements of GPRA. It highlights a limited number of IRS performance measures.

24U.S. General Accounting Office, Tax Administration: Assessment of IRS’s 2001 Tax

Filing Season, GAO-02-144 (Washington, D.C.: Dec. 21, 2001).

Most Electronic Filing and

Assistance Measures Have

Clarity
Page 24 GAO-03-143 Tax Filing Performance Measures

Electronic filing and assistance changed the targets for two measures—

“number of 1040 series returns filed electronically”25 and “total number of

returns electronically filed”—during fiscal year 2001. Changing targets

could distort the assessment of performance because what was to be

observed changed. No major event (such as legislation that affected the

ability of many taxpayers to file electronically) happened that warranted

changing the targets in the strategic plan. Instead, electronic filing and

assistance changed the target for the first of those measures from

42 million returns to 40 million returns because IRS’s Research Division’s

midyear data indicated that 42 million 1040 series returns were not going

to be filed electronically. Because the number of 1040 series returns filed

electronically is a subset of the total number of returns filed electronically,

electronic filing and assistance also reduced the target for total electronic

filings. Because of these subjective considerations, changing the targets in

this situation also affected the objectivity of these measures.

Of electronic filing and assistance’s 13 performance measures, we

considered 12 to be reliable because the data on performance comes from

sources, such as IRS’s masterfile26 and computer program runs, that are

subject to validity checks. The one measure we did not consider reliable

was the “number of IRS digital daily Web site hits,” because it does not

represent the actual number of times the Web site is accessed, as

previously described.

Electronic filing and assistance’s core program activities are to provide

individual and business taxpayers the capability to transact and

communicate electronically with IRS. Electronic filing and assistance

focuses on taxpayers’ ability to file their returns, pay their taxes, receive

assistance, and obtain information electronically. These core activities are

all covered by the 13 performance measures.

Seven of the 13 electronic filing and assistance measures had partial

overlap. For example, the “number of 1040 series returns electronically

filed” and “percent of individual returns electronically filed” measures

provide related information on a key program activity. The difference is

that the former is a count of the number filed electronically while the

latter is the percentage of total individual tax returns filed electronically.

 251040 series returns are individual income tax returns filed on Forms 1040, 1040A, and

1040EZ.

26The masterfile is the system where most of IRS’s taxpayer data resides.

Two Electronic Filing and

Assistance Measures Had

Targets Changed and Lack

Objectivity

Electronic Filing and

Assistance Measures Are

Reliable, with One Exception

Measures Cover Electronic

Filing and Assistance’s Core

Program Activities

Overlap Exists among

Electronic Filing and

Assistance Measures
Page 25 GAO-03-143 Tax Filing Performance Measures

(Table 7 in appendix II has information on other overlapping electronic

filing and assistance measures.)

The amount of overlap to tolerate among measures is management’s

judgment. Electronic filing and assistance officials told us that each of the

overlapping measures we identified provides additional information to

managers. For example, the “number of 1040 series returns electronically

filed” provides managers with information on the size of the electronic

return workload whereas the “percent of individual returns electronically

filed” tells them how they are doing in relation to IRS’s long-term strategic

goal of 80 percent. IRS officials also pointed out that both number and

percent performance measures exist because external customers, such as

the press, like to use the measures for reporting purposes.

Although electronic filing and assistance’s measures address several

governmentwide priorities, such as quantity, customer satisfaction, and

employee satisfaction, they do not cover two important priorities—quality

and cost of service. As a result, its performance measurement system is

not fully balanced.

Electronic filing and assistance classifies four of its performance measures

as quality measures, but the measures are merely counts of certain types

of electronic transactions (such as “number of payments received

electronically”). On the other hand, it tracks what we consider to be

quality measures (i.e., “processing accuracy”27 and “refund timeliness,

electronically filed”)28 but those measures are not in the Strategy and

Program Plan. These quality measures and others, such as one that tracks

the number of electronic returns rejected,29 could be important indicators

of program success or failure. For example, IRS data indicate that many

electronic tax returns are rejected; a measure that captures the volume of

rejects could help to focus management’s attention on the cause of those

rejects.

 27“Processing accuracy” refers to the total number of returns that do not go to the error

resolution system. Transactions that fail validity checks during processing are corrected

through the error resolution system.

28“Refund timeliness, electronically filed” is the amount of time it takes for taxpayers to

receive their refunds when filing electronically.

29Electronic returns can be rejected, for example, if taxpayers fail to include required Social

Security numbers. IRS requires taxpayers to correct such errors before it will accept their

electronic returns.

Electronic Filing and

Assistance’s Measures Do Not

Cover Some Governmentwide

Priorities, Thus Hindering

Balance
Page 26 GAO-03-143 Tax Filing Performance Measures

Also, similar to our discussion of a cost of service measure in the

telephone section, a “cost-per-electronically filed return” could provide a

link between program goals and costs, as required by GPRA, and help IRS

management and Congress decide about future investments in electronic

filing and assistance.

As shown in table 4, all 14 of field assistance’s performance measures have

some of the attributes of successful performance measures. However, as

summarized in this section, the measures have several shortcomings,

primarily with respect to clarity, reliability, and balance. Table 8 in

appendix II has more detailed information about each field assistance

measure, including any weaknesses we identified and any

recommendations for improvement.

Field Assistance Measures
Page 27 GAO-03-143 Tax Filing Performance Measures

Table 4: Overview of Our Assessment of Field Assistance Measures

Note: A check mark denotes that the measure has the attribute.

a

We were unable to verify the linkages between goals and measures because of insufficient

documentation.

b

Core program activities of field assistance are to provide face-to-face assistance, education, and

compliance services.

c

Although field assistance continues to develop its suite of performance measures, important

measures of timeliness, efficiency or productivity, and cost of service are missing and impair balance.

Source: GAO analysis.

Field assistance recognizes the importance of creating a clear relationship

between goals and measures and has developed a template that shows

some of that relationship. Figure 4 is an excerpt of the template, with the

completed portions, as of October 2002, shown in gray.

Relationship between Goals

and Field Assistance Measures

Not Complete

Customer satisfaction

Return preparation contacts

Geographic coverage

Return preparation units

Tax Assistance Centers (TAC) total contacts 

(includes return preparation contacts)

Forms contacts

Tax law contacts

Account contacts

Other contacts

Tax law accuracy

Accounts/notices accuracy

Return preparation accuracy

Employee satisfaction

Alternate contacts

Field assistance 

measures Linkagea Clarity

Measurable 

target Objectivity Reliability

Core

program

activitiesb

Limited

overlap

Governmentwide

priorities

Field assistance attributes

Balance

This attribute 

applies to 

the overall 

suite of 

measures 

rather than 

to the 

measures

individually.c
Page 28 GAO-03-143 Tax Filing Performance Measures

Figure 4: Example of Relationship among Field Assistance Goals and Measures

Source: GAO’s analysis of field assistance’s business plan template.

Although the template demonstrates a noteworthy effort to show a clear

link between goals and measures, it omits the link to IRS’s mission, IRS’s

strategic goals, and field assistance’s improvement projects. These links

are important because they serve as the bridge between long-term

strategic goals and short-term daily operational goals, which can, among

other things, be used for holding IRS and the field assistance program

accountable for achieving those goals. Also, officials told us that the

completed template would only cite the type of performance measure—

employee satisfaction, customer satisfaction, or business results—not the

specific measure and target. The link to the specific measure provides

additional information needed to clearly communicate the alignment of

goals and measures throughout the agency, and the target communicates

the level of performance the operating division hopes to achieve.

Many of field assistance’s measures lack clarity. For example, the

“geographic coverage” measure is unclear, even to IRS officials, because it

is not evident by its name or definition what is or is not included in the

measure’s formula. Specifically, officials debated whether or not the

measure included alternate sites30 and kiosks.31 Similarly, the formula only

considers the location of Taxpayer Assistance Centers (TAC), not their

hours of operation or services provided. Although we saw no evidence

that this lack of clarity led to adverse consequences, it could. For example,

 30Alternate sites are staffed with field assistance employees and offer limited face-to-face

services, such as preparing returns and distributing forms. Field assistance has about

50 alternate sites, such as temporary sites in shopping malls and libraries. Alternate sites

are currently not included in the “geographic coverage” measure.

31Kiosks are automated machines that taxpayers can use to obtain certain forms, answers

to frequently asked questions, and general IRS information in English and Spanish. Kiosks

are currently not included in the “geographic coverage” measure.

Many Field Assistance

Measures Lack Clarity

IRS

Mission

IRS

Strategic Goal

Wage & Investment

Strategy

Improve Proximity

of Service

Delivery

Locations

Field Assistance

Performance 

Measure

To Be

Decided

Field Assistance

Improvement Projects

Field Assistance

Operational Priority

Meet Taxpayer

Demands for

Timely, Accurate, 

and Efficient

Services
Page 29 GAO-03-143 Tax Filing Performance Measures

management or other stakeholders may determine that TACs are needed

in certain areas of the country to improve geographic coverage when, in

fact, alternate sites and/or kiosks are already serving those areas. IRS

officials said that they have plans to revise the formula to include alternate

sites and kiosks. (The other measures that lack clarity, as described in

table 8 of appendix II, are “return preparation contacts,” “return

preparation units,” “TACs total contacts,” “forms contact,” “tax law

contacts,” “account contacts,” “other contacts,” “tax law accuracy,”

“accounts/notices accuracy,” and “return preparation accuracy.”)

We determined that all of field assistance’s 14 performance measures are

objective because, to the greatest extent possible, they are free of

significant bias or manipulation and indicate specifically what is to be

observed, in which population or conditions, and in what timeframes. Of

the 14 measures, 7 have targets in place to help determine whether overall

goals and objectives were achieved. Of the seven measures without

targets, three were being baselined (i.e., IRS was collecting data for use in

setting first-time targets). The remaining four measures were being

designed at the time of our review. Targets will be set for these measures

upon completion of data collection.

Eight of field assistance’s 14 performance measures are based on a data

collection process that is subject to inconsistencies and human error,

meaning that the same results may not be produced in similar

circumstances. All TAC employees are to use Form 5311 (Field Assistance

Activity Report) to manually report their daily hours and type of assistance

provided. Supervisors are to review the forms for accuracy and forward

them for manual input into the Resources Management Information

System.32 These layers of manual input are subject to error and can hinder

data reliability that could (1) lead managers or other stakeholders to draw

inappropriate conclusions about program performance, (2) not alert them

to the existence of problems, or (3) not help them respond when problems

arise. For example, as we noted in our report on IRS’s 2001 tax filing

season, our calculations showed that the data reported by TACs did not

account for the wait times of about 661,000 taxpayers, or about 13 percent

of taxpayers served.33 IRS expects to minimize this human error by

equipping all of its TACs with an on-line automated tracking and reporting

 32The Resources Management Information System is the primary management information

system that field assistance uses to track workload volume and staff hour expenditures.

33GAO-02-144.

All Field Assistance Measures

Are Objective and Have Targets

That Are Either in Place or

Being Established

Data Collection Process Affects

Reliability of Several Field

Assistance Measures
Page 30 GAO-03-143 Tax Filing Performance Measures

system known as the Queuing Management System (Q-Matic). This system

is expected, among other things, to more efficiently monitor customer

traffic flow and wait times and eliminate staff time spent completing Form

5311.34

IRS has taken steps to solve data reliability problems with field

assistance’s customer satisfaction measure. In a May 2000 report, the

Treasury Inspector General for Tax Administration concluded that IRS had

not established an adequate management process to ensure that the survey

yielded accurate, reliable, and statistically valid results.35 To field

assistance’s credit and with the help of a vendor, it (1) completed major

revisions to the customer satisfaction survey, such as using a different

index scale; (2) included space for written comments, which were to be

provided to managers on a routine basis; and (3) improved controls to

ensure the survey is available to all taxpayers. However, problems arose

regarding the manner in which the vendor was providing site managers

with data containing cumulative responses and, as of June 2002, the

vendor had temporarily stopped providing feedback to site managers and

was in the process of determining a more usable format to relay

information to managers. The improved data collection method is being

implemented and IRS anticipates an increase in the precision with which it

measures field assistance customer satisfaction.

Field assistance’s measures cover its core program activities with limited

overlap. Field assistance identifies its core program activities as face-toface assistance, education, and compliance services, which include such

activities as preparing returns, answering tax law questions, resolving

account and notice inquiries, and supplying forms and publications. For

example, field assistance has an “accounts contact” measure (counts the

number of contacts made) and an “accounts accuracy” measure (measures

the accuracy of the responses) to reflect both the quantity and quality of

its accounts-related assistance.

 34Of about 420 TACs, 123 had Q-Matic as of June 2002. IRS officials stated that installation

and networking of Q-Matic in all offices is scheduled to be complete by September 30, 2005.

In the meantime, IRS plans to pilot an installed and networked Q-Matic system in all the

TACs that are located in one of IRS’s seven management areas during the first quarter of

2003.

35Treasury Inspector General for Tax Administration, Walk-in Customer Satisfaction

Survey Results Should Be Qualified If Used for the GPRA, 2000-10-079 (Washington, D.C.:

May 17, 2000).

Field Assistance Measures

Cover Core Program Activities

with Limited Overlap
Page 31 GAO-03-143 Tax Filing Performance Measures

Field assistance identified some overlap between two measures, “return

preparation contacts” and “return preparation units.” It has decided, for

Strategy and Program Plan purposes, to discontinue the “contacts”

measure (which counts the number of customers assisted) and keep the

“units” measure (which counts the number of returns prepared) because

the “units” measure better reflects the amount of return preparation work

done.36 Field assistance will continue tracking the “contacts” measure

outside of the Strategy and Program Plan in order to determine customer

demand for service at particular sites. We concur with IRS’s plans to track

the “contacts” measure outside of the Strategy and Program Plan because

it is a diagnostic tool that can be used for analysis purposes.

Field assistance continues to develop its suite of performance measures.

As part of that effort, it is beginning to deploy important quality measures,

such as “tax law accuracy.” However, other important measures of

timeliness, efficiency, and cost of service are missing, which impairs

balance.

• Timeliness. Before fiscal year 2001, field assistance had a performance

measure that officially tracked how long customers waited to receive

service from an employee. According to managers, it was discontinued

because employees were serving taxpayers as quickly as possible in

order to meet timeliness goals, which negatively affected service

quality.37 In March 2002, management went further and (1) eliminated

its requirement for TACs not equipped with Q-Matic to submit biweekly

wait-time reports and (2) doubled, from 15 to 30 minutes, the wait-time

interval to be used by TACs with Q-Matic in computing the percentage

of customers served on time. Officials said that they took these steps

because employees continued to feel pressured to hurry assistance

despite the discontinuance of the official timeliness measure. However,

one purpose of balanced measures is to avoid an inappropriate

emphasis on just one aspect of performance. The presence of a quality

measure should provide a disincentive for employees to ignore quality

in favor of timeliness. Similarly, in the absence of a timeliness

performance measure, (1) field assistance may not be balancing its

 36The number of units would generally be larger than the number of contacts. For example,

if a taxpayer received help in preparing his or her return and his or her child’s return, field

assistance would count that service as one return preparation contact and two return

preparation units.

37TACs monitor timeliness, but IRS does not report the measure in the Strategy and

Program Plan.

Field Assistance Is Missing

Some Measures Needed to

Balance Governmentwide

Priorities
Page 32 GAO-03-143 Tax Filing Performance Measures

customers’ needs for timely service with their needs for accurate

information and (2) IRS is not held accountable for timeliness to

stakeholders, such as the Congress.

• Efficiency. Efficiency, or productivity as it is often referred to, shows

how efficiently IRS’s resources are transformed into the production of

field assistance services. Field assistance officials said they would like

to develop an efficiency measure, but no plans are in place. Among

other things, having an efficiency measure would help managers

identify performance strengths and weaknesses.

• Cost of Service. As required by GPRA, agencies should have

performance measures that correlate the level of program activity and

program cost. Without such a measure in field assistance, officials do

not know how much it costs to provide face-to-face service. Field

assistance officials said that they would like to develop a cost of

service measure, but they are not certain how to calculate it.

As shown in table 5, all 11 of submission processing’s performance

measures have many of the attributes of successful performance

measures. However, as summarized in this section, we identified several

opportunities for improvement, especially in the area of reliability.

Table 9 in appendix II has more detailed information about each

submission processing measure, including any weaknesses we identified

and any recommendations for improvement.

Submission Processing

Measures
Page 33 GAO-03-143 Tax Filing Performance Measures

Table 5: Overview of Our Assessment of Submission Processing Measures

Note: A check mark denotes that the measure has the attribute.

a

We were unable to verify the linkages between goals and measures because of insufficient

documentation.

b

Core program activities of submission processing are to efficiently and accurately process returns,

remittances, and refunds and issue notices and letters.

c

Submission processing measures cover various governmentwide priorities, such as efficiency,

timeliness, and accuracy; however, submission processing’s measures did not include a measure for

customer satisfaction or for showing how much it costs to process the average return.

Source: GAO analysis.

No formal documentation exists to show how submission processing’s

11 measures are aligned with IRS’s mission, its agencywide goals, and its

operating division goals. Despite this lack of formal documentation,

submission processing officials said, and we generally concur, that some

linkage does exist. Without complete documentation, however, we could

not verify all the linkages. Submission processing officials stated that staff

and managers are aware of the link between measures and goals because

the submission processing organization has taken action to help ensure

that staff understand the measures and their role in supporting IRS’s

overall mission and strategic and operating goals. For example, according

to submission processing officials, they visited all eight W&I processing

centers in 2001 to talk directly with staff and managers about the

importance of balanced performance measures in ensuring that IRS meets

its goals. Complete documentation of the linkages between goals and

measures could further enhance understanding of those goals and

measures with managers and staff.

Alignment between IRS’s Goals

and Submission Processing

Measures Is Uncertain

Linkagea Clarity

Measurable 

target Objectivity Reliability

Core

program

activitiesb

Limited

overlap

Governmentwide

priorities Balance

Submission

processing

measures

Individual 1040 series returns filed (paper)

Number of individual refunds issued (paper)

Employee satisfaction

Refund timeliness–individual (paper)

Notice error rate

Refund error rate-individual (paper)

Letter error rate

Deposit timeliness (paper)

Deposit error rate

Refund interest paid (per $1 million of refunds)

Submission processing productivity

Submission processing attributes

1040

This attribute 

applies to 

the overall 

suite of 

measures 

rather than 

to the 

measures

individually.c
Page 34 GAO-03-143 Tax Filing Performance Measures

All but one of the submission processing measures have clarity and

provide information to enable executives, other managers, and outside

stakeholders to properly assess performance against goals. The one

exception is the productivity measure.

Managers in different processing centers told us that they did not use the

productivity measure to provide them with performance information or to

help them assess performance because, among other things, the measure

does not provide specific information about their unit’s or center’s

performance or their contribution to overall productivity. This is because

the measure, as designed, is a compilation of different types of work IRS

performs in processing returns, remittances, and refunds and issuing

notices and letters. As a result, unit managers used different productivity

measures specific to their own processes to help them identify how to

increase their area’s productivity. However, according to IRS officials, the

productivity measure is useful and provides adequate information to some

IRS executives.

From our perspective, although the productivity measure may be

meaningful to executives, the fact that field managers use other measures

and profess not to understand the current productivity measure indicates

that the current measure does not provide those managers with useful

information that would alert them to problems and help them respond

when problems arise. In addition, because the measure is calculated by

compiling and weighting different types of processing work per staff year

expended, it may be too confusing to be useful to outside stakeholders,

such as Congress.

All 11 of submission processing’s measures have measurable targets and

most are objective (i.e., reasonably free of significant bias or

manipulation). For example, the “notice error rate” had a target of

8.1 percent for fiscal year 2001. The “deposit timeliness” measure appears

to be objective, for example, because the Integrated Submission and

Remittance Processing System38 automatically calculates data on which

the measure is based. However, the “notice error rate” and “letter error

rate” measures are not objective because the coding required as part of

data collection by individual reviewers is subject to much interpretation

that could systematically bias the results of the measures. In October

 38The Integrated Submission and Remittance Processing System is the system IRS uses to

process tax returns and remittances.

Submission Processing

Measures Have Clarity, with

One Exception

All Submission Processing

Measures Have Targets and

Most Are Objective
Page 35 GAO-03-143 Tax Filing Performance Measures

2002, the Treasury Inspector General for Tax Administration reported,

based on a review at two processing centers, that the “deposit error rate”

measure was not objective, because the associated sampling plan was not

consistently implemented.39 The Treasury Inspector General for Tax

Administration recommended that IRS take steps to ensure consistent

implementation, and IRS reported that steps have been taken.

Five measures are subject to consistency problems that affect the

reliability of the measures. Those measures are “refund timeliness—

individual (paper),” “notice error rate,” “refund error rate,” “letter error

rate,” and “deposit error rate.” Specifically, the five measures are based on

a data collection process, which according to the Director of Submission

Processing, involves about 80 staff who identify, interpret, and analyze

errors at the eight W&I processing centers. The “notice error rate” and

“letter error rate” measures also involve coding that is subject to further

interpretation.

Submission processing managers recognized that staff inconsistently

coded notice and letter errors during the 2001 filing season. Neither IRS

nor we know the extent to which such inconsistencies exist because no

routine studies are done to validate the accuracy of data collection.

Reliability and credibility increase when such studies are done.

Submission processing initiated studies beginning in June 2001 to improve

reliability, but has not established any improvement goals.

Each of submission processing’s measures directly pertains to one of the

core program activities of submission processing’s business operations—

timely, efficiently, and accurately processing returns, remittances, and

refunds and issuing notices and letters—without redundancy or overlap.

For example, the “refund error rate—individual (paper)” measure directly

pertains to one of submission processing’s core program activities,

processing refunds, and does not overlap with any of the other 11

measures.

Unlike the other three program areas we reviewed, submission processing

has two customers—taxpayers, to whom IRS issues refunds and sends

notices, and the Department of the Treasury, for which IRS deposits

 39Treasury Inspector General for Tax Administration, The Internal Revenue Service Needs

to Improve Oversight of Remittance Processing Operations, 2003-40-002 (Washington,

D.C.: Oct. 7, 2002).

Five Submission Processing

Measures Lack Reliability

Submission Processing

Measures Cover Core Program

Activities without Overlap
Page 36 GAO-03-143 Tax Filing Performance Measures

remittances. Therefore, for some measures, such as “refund timeliness,”

IRS views taxpayers as the customer, while for other measures, such as

“deposit timeliness,” IRS views Treasury as the customer. Submission

processing officials believe that this dual-customer perspective provides a

complete view of their operations and the measures cover all aspects of

their operations while still being limited to a manageable number.

Submission processing’s measures cover various governmentwide

priorities, such as efficiency, timeliness, and accuracy. However, at the

time of our review, submission processing measures lacked balance

because they did not include a measure for customer satisfaction or a

measure showing how much it costs to process a return.

Although submission processing officials believe that some existing

measures, such as “notice error rate” and “refund timeliness,” provide

information related to the customer’s experience, they recognize that

directly obtaining customers’ perspectives would be more accurate than

assuming their experience based on such measures. Thus, submission

processing is obtaining customer satisfaction information as part of IRS’s

corporate customer satisfaction survey, which IRS expects will be

available by the 2003 filing season.

Similar to the other three program areas, submission processing does not

have a cost of service measure.40 Among other things, not having a cost of

service measure affects IRS’s ability to adequately compare different types

of processing, such as paper versus electronic. In our view, because IRS

does not take into account the cost to process a particular type of return,

managers cannot fully understand the effectiveness of their unit.

Because the filing season affects so many taxpayers, IRS’s performance is

important. Having successful performance measures that demonstrate

results, are limited to the vital few, cover multiple program priorities, and

provide useful information to decision makers will help IRS management

and stakeholders, such as Congress, make decisions about how to fund

and improve return processing and assistance to taxpayers.

 40Submission processing did have some data related to the average direct labor cost to

process some paper returns in 1999.

Submission Processing

Measures Cover Various

Governmentwide Priorities, but

Are Not Fully Balanced

Conclusions
Page 37 GAO-03-143 Tax Filing Performance Measures

Despite the challenge of developing a set of 53 measures that satisfy our

criteria, IRS has made significant progress. As developed to date, the

measures satisfy many of our nine attributes for successful performance

measures. For example, in all four of the program areas we reviewed, most

measures covered the core activities of each program and had targets in

place. IRS also has several on-going improvement initiatives, such as the

effort to redesign all aspects of its telephone assistance quality measures.

Although the measures satisfied many of the nine attributes, our

evaluation also showed that they do not have all the characteristics of

successful performance measures. The most significant weaknesses

include (1) the inability of some measures to provide clear information to

decision makers about program performance, (2) data collection methods

that hamper objectivity and reliability, and (3) measures to cover

governmentwide priorities that are missing from the Strategy and Program

Plan. Although such weaknesses do not mean that the measures are not

useful, IRS risks basing program and resource allocation decisions on

inadequate or incomplete information and is less accountable until the

weaknesses are addressed.

Correcting these weaknesses is important in order to (1) create a resultsoriented environment that demonstrates and tracks how IRS’s programs

and activities contribute to achieving its mission and strategic goals,

(2) avoid creating an excess of data that could obscure key information

needed to identify problem areas and assess goal achievement, (3) form a

balanced environment that takes the core program activities of the

program into account, and (4) provide managers and other stakeholders

with critical information on which to base their decisions.

We recommend that the Commissioner of Internal Revenue direct the

appropriate officials to do the following:

Take steps to ensure that agencywide goals clearly align with operating

division goals and performance measures for each of the four areas

reviewed. Specifically, (1) clearly document the relationship among

agencywide goals, operating division goals, and performance measures

(the other three program areas may want to consider developing a

template similar to the one field assistance developed, shown in figure

4) and (2) ensure that the relationship among goals and measures is

communicated to staff at all levels of the organization.

Make the name and definition of several field assistance measures (i.e.,

“geographic coverage,” “return preparation contacts,” “ return preparation

Recommendations for

Executive Action
Page 38 GAO-03-143 Tax Filing Performance Measures

units,” “TACs total contacts,” “forms contacts,” “tax law contacts,”

“account contacts,” “other contacts,” “tax law accuracy,”

“accounts/notices accuracy,” and “return preparation accuracy”) more

clear to indicate what is and is not included in the formula.

As discussed in the body of this report and in appendix II, modify the

formulas used to compute various measures to improve clarity. If formulas

cannot be implemented in time for the next issuance of the Strategy and

Program Plan, then modify the name and definition of the following

measures so it is clearer what is or is not included in the measure.

• Remove automated calls from the formula for the “CSR level of

service” measure.

• Revise the “CSR response level” measure to include calls from

taxpayers who tried to reach a CSR but did not, such as those who

(1) hung-up while waiting to speak to a CSR, (2) were provided access

only to automated services and hung up, and (3) received a busy signal.

• Analyze and use new or existing data to determine why calls are

transferred and use the data to revise the “CSR services provided”

measure so that it only reflects transferred calls in which the caller

received help from more than one CSR (i.e., exclude calls in which a

CSR simply transferred the call and did not provide service.)

• Either discontinue use of the “number of IRS digital daily Web site hits”

measure or revise the way “hits” are calculated so that the measure

more accurately reflects usage.

• Revise field assistance’s “geographic coverage” measure by ensuring

that the formula better reflects (1) the various types of field assistance

facilities, including alternate sites and kiosks; (2) the types of services

provided by each facility; and (3) the facility’s operating hours.

• Revise submission processing’s “productivity” measure so it provides

more meaningful information to users.

Refrain from making changes to official targets, such as electronic filing

and assistance did in fiscal year 2001, unless extenuating circumstances

arise. Disclose any extenuating circumstances in the Strategy and Program

Plan and other key documents.

Modify procedures for the toll-free customer satisfaction survey, possibly

by requiring that administrators listen to the entire call, to better ensure

that administrators (1) notify CSRs that their call was selected for the

survey as close to the end of a call as possible and (2) can accurately

answer the questions they are responsible for on the survey.
Page 39 GAO-03-143 Tax Filing Performance Measures

Implement annual effectiveness studies to validate the accuracy of the

data collection methods used for the five telephone measures (“toll-free

tax law quality,” “toll-free accounts quality,” “toll-free tax law correct

response rate,” “toll-free account correct response rate,” and “toll-free

timeliness”) subject to potential consistency problems. The studies could

determine the extent to which variation exists in collecting data and

recognize the associated impact on the affected measures. For those

measures, and for the five submission processing measures that already

have effectiveness studies in place (“refund timeliness-individual (paper),”

“notice error rate,” “refund error rate—individual (paper),” “letter error

rate,” and “deposit error rate”), IRS should establish goals for improving

consistency, as needed.

Ensure that plans to remove overlapping measures in telephone and field

assistance are implemented.

As discussed in the body of this report, include the following missing

measures in the Strategy and Program Plan in order to better cover

governmentwide priorities and achieve balance.

• In the spirit of provisions in the Chief Financial Officers Act of

1990 and Financial Accounting Standards Number 4, develop a cost

of services measure using the best information currently available

for each of the four areas discussed in this report, recognizing data

limitations as prescribed by GPRA. In doing so, adhere to guidance,

such as Office of Management and Budget Circular A-76, and consider

seeking outside counsel to determine best or industry practices.

• Given the importance of automated telephone assistance, develop a

customer satisfaction survey and measure for automated assistance.

• Put the “automated completion rate” measure back in the Strategy and

Program Plan after revising the formula so that calls for recorded tax

law information are not counted as completed when taxpayers hang up

before receiving service.

• Add one or more quality measures to electronic filing and assistance’s

suite of measures in the Strategy and Program Plan. Possible measures

include “processing accuracy,” “refund timeliness, electronically filed,”

and “number of electronic returns rejected.”

• Re-implement field assistance’s timeliness measure.

• Develop a measure that provides information about field assistance’s

efficiency.
Page 40 GAO-03-143 Tax Filing Performance Measures

The Commissioner of Internal Revenue provided written comments on a

draft of this report in a letter dated November 1, 2002, which is reprinted

in appendix III. The Commissioner was pleased to see that many of the

measures had the attributes for successful performance and agreed that

others presented opportunities for further refinement. He stated that the

report was objective and balanced and that our observation of the ongoing nature of the performance measurement process was on point.

Furthermore, he noted that the attributes we developed can be used as a

checklist when performance measures are developed in the future.

Of our 18 recommendations, IRS

• agreed with 10 and cited planned corrective actions that were

responsive to those recommendations;

• cited actions taken or planned in response to 2 that did not fully

address our concerns; and

• disagreed with 6.

The following discussion focuses on the recommendations with which IRS

disagreed or for which we believe additional action is necessary to address

our concerns.

In response to our recommendation about clarifying the name and

definition of several field assistance measures, IRS said that the recently

updated data dictionary addressed our concerns. We reviewed the updated

data dictionary. The modifications are substantial and provide significant

additional information about the measures. However, the definitions

remain unclear. Specifically, the definitions should either define a

taxpayer assistance center or state whether or not alternate sites, such as

kiosks and mobile sites, are included.

IRS did not agree that automated calls should be removed from the

formula for the “CSR level of service” measure. IRS said that including the

count of callers who choose an automated service while waiting for CSR

service is appropriate. IRS’s response does not accurately characterize all

the calls answered by automation that are included in the “CSR level of

service” measure. Rather than choosing an automated service while

waiting for a CSR, some callers complete an automated service after

hearing an announcement that, due to high call volume, only automated

services are available—a choice is not involved. We believe that the “CSR

level of service” measure, because of its name and the way it is calculated,

could be misleading and might misrepresent taxpayers’ access to CSR’s.

For example, increasing the percentage of calls served through

Agency Comments

and Our Evaluation
Page 41 GAO-03-143 Tax Filing Performance Measures

automation because a CSR was not available—meaning that CSR’s were

actually more difficult to reach—would improve the “CSR level of service”

measure, thus giving the impression that access to CSR’s had improved

when it had actually gotten worse. Calls answered through automation,

regardless of the type of assistance (CSR or automation) the caller was

originally seeking, should be reflected in an automated-level-of-service

measure, such as “automated service completion rate.”

IRS did not agree that it should modify the “CSR response level” measure

to include calls in which the caller hung up before receiving service or got

a busy signal. IRS said that altering the measure would deviate from

industry standards and hinder IRS's ability to gauge success in meeting

this “world class service” goal. We support IRS’s efforts to gauge its

progress toward providing world class customer service by telephone.

However, IRS’s use of the same telephone wait-time measure used by

others may actually hinder a meaningful comparison of IRS with industry

leaders. The “CSR response level” measure shows, for the callers who

reached a CSR, the percentage that waited 30 seconds or less. According

to IRS officials, when taxpayers call IRS attempting to reach a CSR, they

are much less likely to reach one than when they call a recognized

telephone service leader (i.e., callers to IRS are more likely to hang up

while waiting to speak to a CSR, hang up after being given access to only

automated service because a CSR is not available, or receive a busy

signal). Therefore, when the “CSR response level” measure (which

excludes these hang-ups and busy signals) is used by IRS, the measure

may represent the experience of a significantly smaller percentage of the

total callers that attempted to reach an a CSR than when the same

measure is used by industry leaders, thus potentially overstating the ease

with which callers reached IRS CSR’s. Data we obtained from IRS suggest

that there were about an equal number of hang-ups and busy signals as

calls answered in this measure in 2001.

In response to our recommendation about implementing annual studies to

validate the accuracy of various data collection methods and establishing

goals for improving consistency, IRS said that it (1) has an ongoing

process to ensure proper administration of the collection methods for the

telephone measures cited in our recommendation, (2) does not agree that

an annual independent review by non-CQRS analysts is merited, and

(3) does not agree that it should incorporate consistency improvement

goals in the Strategy and Program Plan process. As we noted in our report,

telephone assistance’s CQRS has some controls in place to monitor

consistency. However, we believe that reliability and credibility increase

when performance data are checked or tested for significant errors, which
Page 42 GAO-03-143 Tax Filing Performance Measures

IRS currently does not do. We did not recommend that non-CQRS analysts

do these reviews; who does the reviews is for IRS to decide. Also, we

recognized in our report that submission processing has an on-going

process to verify consistency and that it has found problems. Because that

review process has found some problems, we believe that establishing

goals for improving consistency in submission processing is warranted.

Because telephone assistance does not have a review process in place, we

do not know whether improvement goals are needed, but noted that they

could be. We did not recommend that these goals become a part of the

Strategy and Program Plan process. Instead, these goals should become

part of the review process and be made known to staff who are performing

the work.

IRS disagreed with our recommendation that it put the “automated

completion rate” measure back in the Strategy and Program Plan. Instead,

IRS said it would continue to track and monitor that rate as a diagnostic

measure. IRS told us that its decision is based on the fact that data on

automated calls are not good enough to merit the attention the measure

would have at the Strategy and Program Plan level. We recognize that

there are data weaknesses with this measure. That is why our

recommendation calls for IRS to revise the formula before returning the

measure to the Strategy and Program Plan. Because serving more callers

through automation is important to IRS’s strategy for improving taxpayer

service, we believe that IRS needs a measure of the level of service

provided by automation in its Strategy and Program Plan to balance its

measure of the level of service provided by CSRs. Other than counts of the

number of calls served, IRS has no measure of its effectiveness in serving

taxpayers through automation. Without such a measure, IRS risks poorly

serving the increasing number of taxpayers being served through

automation while possibly improving access for a declining number of

callers who need to speak with a CSR.

IRS does not believe that adding one or more quality measures to

electronic filing and assistance’s suite of measures in the Strategy and

Program Plan would enhance the electronic filing program. It noted that it

tracks the quality of electronic filing outside the Strategy and Program

Plan and that quality has been consistently high. We recognize that

electronic filing and assistance tracks quality outside the Strategy and

Program Plan. However, we disagree with IRS's position that adding

quality measures to that plan would not enhance the program. According

to IRS officials, measures in the Strategy and Program Plan are the highest,

most comprehensive level of measures for which they are accountable. In

addition, many of those measures are made available to outside
Page 43 GAO-03-143 Tax Filing Performance Measures

stakeholders. By not elevating these measures of quality to the Strategy

and Program Plan, electronic filing and assistance risks not being held to

any quality standards. Furthermore, not having quality measures hampers

balance among electronic filing and assistance's suite of measures and is

not consistent with IRS's balanced measurement program or the intent of

IRS’s Restructuring and Reform Act of 1998.

IRS disagreed with our recommendation that it re-implement field

assistance’s timeliness measure. IRS said that although timeliness goals

are important in providing service to taxpayers, they are detrimental to

quality service because field assistance employees tend to rush customers

when traffic is high. This position is inconsistent with IRS's balanced

measurement program and the intent of IRS’s Restructuring and Reform

Act of 1998. Although the accuracy of assistance is an important measure

of quality, the timeliness of that assistance is also an important and

balancing aspect of quality. Without this balancing emphasis, staff could

theoretically take excessive time providing quality tax law assistance to a

few taxpayers regardless of the impact on the wait-time for other

taxpayers. We agree that Q-Matic is the best source of this information and

support IRS’s plans to implement it nationwide. IRS also stated that it

could use feedback from its customer satisfaction surveys to obtain

information about the "promptness of service." As we noted in our report,

problems arose in the manner with which the feedback was provided from

the vendor and the vendor had stopped providing feedback to site

managers until the problems could be resolved. Even when those

problems are resolved, a timeliness measure based on actual IRS data

versus taxpayers’ perceptions would be meaningful.

Regarding our recommendation about implementing an efficiency measure

in field assistance, IRS said that it will be testing a system for use as a

"diagnostic tool" to monitor and evaluate the strengths and weaknesses of

various productivity measures. However, IRS’s response was silent as to

whether or when it would establish a field assistance productivity

measure. Maintaining and enhancing organizational productivity is a

fundamental agency management responsibility. The extent to which IRS’s

field assistance organization is meeting this basic responsibility needs to

be visible to IRS, Treasury, and congressional stakeholders in the form of

an organizational performance measure, rather than a "diagnostic tool,"

which is generally visible only to IRS managers.

We are sending copies of this report to the Chairmen and Ranking

Minority Members of the Senate Committee on Finance and the House
Page 44 GAO-03-143 Tax Filing Performance Measures

Committee on Ways and Means and the Ranking Minority Member of this

Subcommittee. We are also sending copies to the Secretary of the

Treasury; the Commissioner of Internal Revenue; the Director, Office of

Management and Budget; and other interested parties. We will make

copies available to others on request. In addition, the report will be

available at no charge on the GAO Web site at http://www.gao.gov.

This report was prepared under the direction of David J. Attianese,

Assistant Director. Other major contributors are acknowledged in

appendix IV. If you have any questions about this report, contact

Mr. Attianese or me on (202) 512-9110.

Sincerely yours,

James R. White

Director, Tax Issues
Appendix I: Expanded Explanation of Our

Attributes and Methodology for Assessing

IRS’s Performance Measures

Page 45 GAO-03-143 Tax Filing Performance Measures

Performance goals and measures that successfully address important and

varied aspects of program performance are key to a results-oriented,

balanced work environment. Measuring performance allows organizations

to track the progress they are making toward their goals and gives

managers critical information on which to base decisions for improving

their programs. Organizations need to have performance measures that

(1) demonstrate results, (2) are limited to the vital few, (3) cover multiple

program priorities, and (4) provide useful information for decision making

in order to track how their programs and activities can contribute to

attaining the organization’s goals and mission. These four characteristics

are important to accurately reveal the strengths and weaknesses of a

program since measures are often the key motivators of performance and

goal achievement.

For use as criteria to determine whether the Internal Revenue Service’s

(IRS) performance measures in four key program areas—telephone

assistance, electronic filing and assistance, field assistance, and

submission processing—demonstrate results, are limited to the vital few,

cover multiple program priorities, and are useful in decision making, we

developed nine attributes of performance goals and measures based on

previously established GAO criteria. In addition, we considered key

legislation, such as the Government Performance and Results Act of

1993 (GPRA) and the IRS Restructuring and Reform Act of 1998, and

performance management literature cited in the bibliography and related

products sections at the end of this report. Our nine attributes may not

cover all the attributes of successful performance measures; however, we

believe these are some of the most important. We shared these attributes

with IRS officials responsible for performance measurement issues, such

as the Acting Director of the Organizational Performance Division; and

several officials in the Wage and Investment (W&I) operating division,

such as the Director of Strategy and Finance; the Chief of Planning and

Analysis; the Director of Customer Account Services; and the Director of

Field Assistance. These officials generally agreed with the relevance of the

attributes and our review approach.

We applied these attributes to the 53 filing season measures in W&I’s fiscal

year 2001-2003 Strategy and Program Plan in a systematic manner, but

some judgment was required. To ensure consistency and reliability in our

application of the attributes, we had one staff person responsible for each

of the four areas. That staff person prepared the initial analysis and at least

two other staff reviewed those detailed results. Several staff reviewed the

results for all four areas. Inherently, the attributes described below are not

weighted equally. Weaknesses identified in a particular attribute do not,

Appendix I: Expanded Explanation of Our

Attributes and Methodology for Assessing

IRS’s Performance Measures
Appendix I: Expanded Explanation of Our

Attributes and Methodology for Assessing

IRS’s Performance Measures

Page 46 GAO-03-143 Tax Filing Performance Measures

in and of themselves, mean that a measure is ineffective or meaningless.

Instead, weaknesses identified should be considered areas for further

refinement.

Detailed information on each attribute, including an explanation,

examples, and the methodology we used to assess that attribute with

respect to the measures covered by our review, follows.

1. Is there a relationship between the performance goals and measures

and an agency’s goals and mission? (Referred to as “linkage”)

Explanation: Performance goals and measures should align with an

agency’s goals and mission. A cascading or hierarchal linkage moving from

top management down to the operational level is important in setting goals

agencywide, and the linkage from the operational level to the agency level

provides managers and staff throughout an agency with a road map that

(1) shows how their day-to-day activities contribute to attaining

agencywide goals and mission and (2) helps define strategies for achieving

strategic and annual performance goals. As agencies develop annual

performance goals as envisioned by GPRA, they can serve as a bridge that

links long-term goals to agencies’ daily operations. For example, an annual

goal that is linked to a program and also to a long-term goal can be used

both to (1) hold agencies and program offices accountable for achieving

those goals and (2) assess the reasonableness and appropriateness of

those goals for the agency as a whole. In addition, annual performance

planning can be used to better define strategies for achieving strategic and

annual performance goals.

Linkages between goals and measures are most effective when they are

clearly communicated to all staff within an agency so that everyone

understands what the organization is trying to achieve and the goals it

seeks to reach. Communicating goals and their associated measures is a

continuous process and supports the basis for everything the agency does

each day. Communication creates a “line of sight” throughout an agency so

that everyone understands what the organization is trying to achieve and

the goals it seeks to reach.

Example: Submission processing’s “notice error rate” measure determines

the percentage of incorrect notices issued to taxpayers by submission

processing employees. The target set for this measure in 2001 was

8.1 percent. This measure could be used to support the “notice redesign”

improvement project as well as the operational priority to “prioritize

Attributes of Successful

Performance Measures
Appendix I: Expanded Explanation of Our

Attributes and Methodology for Assessing

IRS’s Performance Measures

Page 47 GAO-03-143 Tax Filing Performance Measures

notices and monitor and control notice issuance.” It also is used to support

one of W&I’s goals— “to meet taxpayer demands for timely, accurate, and

efficient services.” This W&I strategy aligns with IRS’s strategic goal, “top

quality service to all taxpayers through fair and uniform application of the

law,” which in turn, supports IRS’s mission to “provide America’s

taxpayers top quality service by helping them understand and meet their

tax responsibilities and by applying the tax law with integrity and fairness

to all.”

Methodology: We compared IRS’s measures with its targets, improvement

projects, operational priorities, operating division goals, and agencywide

goals and mission as documented in the Strategy and Program Plan. We

also interviewed operational/unit managers and managers responsible for

the Strategy and Program Plan about linkages and reviewed training

materials.

2. Are the performance measures clearly stated? (Referred to as

“clarity”)

Explanation: A measure has clarity when it is clearly stated and the name

and definition are consistent with the methodology used for calculating

the measure. A measure that is not clearly stated (i.e., contains extraneous

or omits key data elements) or that has a name or definition that is

inconsistent with how it is calculated can confuse users and could cause

managers or other stakeholders to think that performance was better or

worse than it actually was.

Example: Telephone assistance’s “average handle time” measure shows

the average number of seconds Customer Service Representatives (CSRs)

spent assisting callers. Its definition and formula are consistent with the

name of the measure and clearly note that the measure includes talk and

hold times and the time a CSR spends on work related to a call after the

call is terminated.

Methodology: We compared the name of the measure, the written

definition of the measure, and the formula or methodology for computing

the measure. In several instances, we discussed certain components of the

definition and formula with IRS officials to better understand its meaning

and purpose. For example, we discussed components of telephone

assistance’s quality measures with staff in Customer Account Services, and
Appendix I: Expanded Explanation of Our

Attributes and Methodology for Assessing

IRS’s Performance Measures

Page 48 GAO-03-143 Tax Filing Performance Measures

staff in the Centralized Quality Review Site. We also reviewed on-line

information available to field assistance managers from the Queuing

Management System (Q-Matic).1

 We spoke to managers at different levels

within each of the four areas we reviewed and asked them about the

information they received and how they used it. In addition, we used some

of the results of a random telephone survey of managers we conducted in

2001 at 84 of IRS’s 413 Taxpayer Assistance Centers (TAC) to solicit their

views on the services provided at those offices.

3. Do the performance measures have targets, thus allowing for easier

comparison with actual performance? (Referred to as “measurable

target”)

Explanation: Where appropriate, performance goals and measures should

have quantifiable, numerical targets or other measurable values.

Numerical targets or other measurable values facilitate future assessments

of whether overall goals and objectives were achieved because

comparisons can be easily made between projected performance and

actual results. Some goals are self-measuring (i.e., they are expressed

objectively and are quantifiable) and therefore do not require additional

measures to assess progress. When goals are not self-measuring,

performance measures should translate those goals into observable

conditions that determine what data to collect to learn whether progress

was made toward achieving goals. The measures should have a clearly

apparent or commonly accepted relation to the intended performance or

have been shown to be reasonable predictors of desired behaviors or

events. If a goal cannot be expressed in an objective, specific, and

measurable form, GPRA allows the Office of Management and Budget to

authorize agencies to develop alternative forms of measurement.2

 1

Q-Matic is an automated tracking and reporting system that is expected to more efficiently

monitor customer traffic flow and wait times and eliminate staff time completing Form

5311. Of about 420 TACs, 123 had Q-Matic as of June 2002.

2

An alternative form of measurement may be either (1) separate, descriptive statements of

a minimally effective program or (2) a successful program, expressed with sufficient

precision and in such terms that would allow for an accurate, independent determination

to be made of how actual performance compares with the goals stated. An example would

be the polio vaccine and how its value to society is judged by experts through a peer

review.
Appendix I: Expanded Explanation of Our

Attributes and Methodology for Assessing

IRS’s Performance Measures

Page 49 GAO-03-143 Tax Filing Performance Measures

Example: Electronic filing and assistance’s “percent of individual returns

electronically filed” had a numerical target of 31 percent in fiscal year

2001.

Methodology: We determined that a goal or measure had a measurable

target when expected performance could be compared with actual results,

and in general, was not changed during the measurement period. Each of

the measures we reviewed was listed in the Strategy and Program Plan,

which provides projections or targets for the current and two subsequent

fiscal years. We verified that the target was measurable. When the Strategy

and Program Plan did not show a target, we contacted appropriate IRS

officials to determine why.

4. Are the performance goals and measures objective? (Referred to as

“objectivity”)

Explanation: To the greatest extent possible, goals and measures should

be reasonably free of significant bias or manipulation that would distort

the accurate assessment of performance. They should not allow subjective

considerations or judgments to dominate the outcome of the

measurement. To be objective, performance goals and measures should

indicate specifically what is to be observed, in which population or

conditions, and in what timeframe and be free of opinion and judgment.

Objectivity is important because it adds credibility to the performance

goals and measures by ensuring that significant bias or manipulation will

not distort the measure.

Example: The “customer satisfaction” measure for telephone assistance

has the potential for bias and therefore may not be objective. Survey

administrators are instructed to notify the CSR towards the end of the call

that his or her call was selected to participate in the survey. A potential

problem arises because administrators are not required to listen to the

entire call, and it can be difficult to determine when the call is about to

end. Therefore, if a CSR is notified prior to the end of the call that the call

was selected for survey, the CSR could change behavior towards the

taxpayer, thus affecting the results of the survey and the measure.

Methodology: We reviewed information in IRS guidance or procedures,

data collection instruments, reports, and other documents. We held

discussions about objectivity with various staff and officials, such as data

owners and analysts, within each of the four areas we reviewed. Because

our interviews raised questions about the objectivity of some measures for

telephone assistance, we monitored some taxpayer calls and interviewed
Appendix I: Expanded Explanation of Our

Attributes and Methodology for Assessing

IRS’s Performance Measures

Page 50 GAO-03-143 Tax Filing Performance Measures

an official from IRS’s customer satisfaction survey contractor, Pacific

Consulting Group.

5. To what extent do the performance goals and measures provide a

reliable way to assess progress? (Referred to as “reliability”)

Explanation: Reliability refers to whether measures are amenable to

applying standard procedures for collecting data or calculating results so

that they would be likely to produce the same results if applied repeatedly

to the same situation. Errors can occur at various points in the collection,

maintenance, processing, and reporting of data. Significant errors would

affect conclusions about the extent to which performance goals have been

achieved. Likewise, errors could cause the measure to report performance

at either a higher or lower level than is actually being attained. Reliability

is increased when verification and validation procedures, such as checking

performance data for significant errors by formal evaluation or audit,

exist.

Example: Field assistance’s “return preparation contacts” measure tracks

the total number of customers assisted with return preparation by IRS.

This measure may not be reliable because it involves a significant amount

of manual entry on Form 5311 (Field Assistance Activity Report) even at

sites with the Q-Matic system. In addition to the potential for error

associated with manual entry, the instructions for filing Form 5311 require

that service time be recorded in whole hours, which can misconstrue

actual service times and is less exact than the data in Q-Matic, which

records service times in minutes.

Methodology: We looked for weaknesses in IRS’s guidance or procedures,

data collection instruments, reports, and other documents that might

cause errors. We discussed potential weaknesses with various officials,

such as account data analysts, within each of the four areas we reviewed.

Because these efforts revealed the potential for errors in measuring

telephone performance, we monitored employees preparing data

collection instruments for assessing telephone quality and customer

satisfaction in Atlanta. Likewise, we monitored field assistance staff

helping taxpayers and reporting their time using both the automated QMatic system and Form 5311.

6. Do the performance measures sufficiently cover a program’s core

activities? (Referred to as “core program activities”)
Appendix I: Expanded Explanation of Our

Attributes and Methodology for Assessing

IRS’s Performance Measures

Page 51 GAO-03-143 Tax Filing Performance Measures

Explanation: Core program activities are the activities that an entity is

expected to perform to support the intent of the program. Performance

measures should be scoped to evaluate the core program activities.

Limiting the number of performance measures to the core program

activities will help identify performance that contributes to goal

achievement. At the same time, however, there should be enough

performance measures to ensure that managers have the information they

need about performance in all the core program activities. Without such

information, the possibility of achieving program goals is less likely.

Example: The core program activities for submission processing include

(1) processing returns, (2) depositing remittances, (3) issuing refunds, and

(4) sending out notices and letters. Each of submission processing’s

11 measures correspond to one of those core activities. For example, the

“number of individual 1040 series returns filed (paper)” measure

corresponds to processing returns and the “letter error rate” measure

corresponds with sending out notices and letters.

Methodology: We determined the core program activities of each of the

four areas we reviewed based on IRS documentation and discussions with

IRS officials. We reviewed the suite of performance measures for each of

the four areas to determine whether measures existed that covered each

core program activity. We determined whether any measures were missing

or other pieces of information were needed to better manage programs by

using judgment and questioning IRS officials. In addition, we reviewed the

results of a questionnaire that we had used during a review of IRS’s

2001 filing season to ask TAC managers about information needed to

manage their program.

7. Does there appear to be limited overlap among the performance

measures? (Referred to as “limited overlap”)

Explanation: Measures overlap when the results of measures provide

basically the same information. A measure that overlaps with another is

unnecessary and does not benefit program management. Unnecessary or

overlapping measures not only can cost money but also can cloud the

bottom line in a results-oriented environment by making managers or

other stakeholders sift through unnecessary or redundant information.

Some measures, however, may overlap partially and provide stakeholders

some new information. In those cases, management must make a

judgment as to whether having the additional information is worth the cost

and possible confusion it may cause.
Appendix I: Expanded Explanation of Our

Attributes and Methodology for Assessing

IRS’s Performance Measures

Page 52 GAO-03-143 Tax Filing Performance Measures

Example: Telephone assistance’s “toll-free average speed of answer” and

“toll-free CSR response level” measures attempt to show how long a

taxpayer waited before receiving assistance. The difference between the

two measures is that the latter shows the percentage of taxpayers

receiving assistance within 30 seconds while the former shows the average

time taxpayers waited for service. These two measures are likely to be

correlated and thus partially overlap. However, the amount of overlap

between measures is management’s discretion.

Methodology: Within each of the four areas we reviewed, we looked at the

suite of measures and compared the measures’ names and definitions. We

also looked at the correlations between measures’ results. When two

measures seemed similar, we discussed the potential for overlap with IRS

officials.

8. Does there appear to be a balance among the performance goals and

measures, or is there an emphasis on one or two priorities at the

expense of others? (Referred to as “balance”)

Explanation: Balance exists when a suite of measures ensures that an

organization’s various priorities are covered. IRS considers its measures to

be balanced when they address customer satisfaction, employee

satisfaction, and business results (quality and quantity). Performance

measurement efforts that overemphasize one or two priorities at the

expense of others may skew the agency’s performance and keep its

managers from understanding the effectiveness of their programs in

supporting IRS’s overall mission and goals.

Example: Submission processing has an employee satisfaction measure

and several business results measures, such as “deposit timeliness.” As of

October 2002, however, it had not fully implemented a customer

satisfaction measure, which resulted in an unbalanced process that can

overlook something as important as the customer’s perspective.

Methodology: For each of the four areas, we ensured that a measure

existed for each component. If measures did not exist for certain

components, we contacted IRS officials to find out why and to see what

plans IRS has to ensure balance in the future.

9. Does the program or activity have performance goals and measures

that cover governmentwide priorities? (Referred to as

“governmentwide priorities”)
Appendix I: Expanded Explanation of Our

Attributes and Methodology for Assessing

IRS’s Performance Measures

Page 53 GAO-03-143 Tax Filing Performance Measures

Explanation: Agencies should develop a range of related performance

measures to address governmentwide priorities, such as quality,

timeliness, efficiency, cost of service, and outcome. A range is important

because most program activities require managers to balance these

priorities among other demands. When complex program goals are broken

down into a set of component quantifiable measures, it is important to

ensure that the overall measurement of performance does not become

biased because measures that assess some priorities but neglect others

could place the program’s success at risk.

Example: Electronic filing and assistance provides the capability for

taxpayers to transact and communicate electronically with IRS. The

13 measures we reviewed included, for example, the number or percent of

returns filed, the number of hits to or downloads from IRS’s Web site, and

employee and customer satisfaction. The Strategy and Program Plan did

not have any measures on the program’s quality or timeliness. Not having

these measures means that management may not be sufficiently balancing

competing demands.

Methodology: We analyzed the suite of measures in the Strategy and

Program Plan for each of the four areas we reviewed. Based on

discussions with IRS officials and our own judgment, we identified

measures that appeared to be missing. We discussed those identified with

IRS officials.
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 54 GAO-03-143 Tax Filing Performance Measures

The following four tables provide information on the 53 performance

measures we reviewed in the four program areas within the Internal

Revenue Service’s (IRS) Wage and Investment (W&I) operating division

that are critical to a successful filing season. Among other things, the

tables show how each of the 53 measures matched up against the

attributes in appendix I. The attributes not addressed in the tables are

(1) “linkage,” because sufficient documentation did not exist to validate

linkages with any of the measures and (2) “balance,” because that attribute

does not apply to specific measures but, rather, to a program’s entire suite

of measures. When reviewing the suite of measures, we found some

instances where additional measures are warranted; the additional

measures are generally not cited in these tables.

Of the 53 performance measures in our review, 15 are for telephone

assistance.1

 Table 6 has information about each of the 15 telephone

measures.

Table 6: Telephone Assistance Performance Measures

Measure name and definitiona

FY 2001 target

and actual

Weaknesses of measure and

consequences Recommendations

Total automated calls answered

A count of all toll-free calls answered at

telephone assistance centers by an automated

system (e.g., Telephone Routing Interactive

System) and Tele-Tax.b

Target:

85,000,000

calls answered

Actual:

104,228,052

calls answered

Some overlap with automated completion

rate measure. Both attempt to show how

many automated calls were answered, but

the automated completion rate tries to

show the percentage that completed

automated service successfully. Overlap

could cloud the bottom line and obscure

performance results.

See note 1 to the

table.

Customer Service Representative (CSR) calls

answered

The count of all toll-free calls answered at

telephone assistance centers.

Target:

31,500,000

calls answered

Actual:

32,532,503

calls answered

Some overlap with CSR services provided

measure. Both attempt to show how many

calls CSRs answered, but CSR services

provided tries to count calls requiring the

help of more than one CSR as more than

one call. Overlap could cloud the bottom

line and obscure performance results.

See note 1 to the

table.

 1

IRS deleted its “automated completion rate” measure in the 2002 Strategy and Program

Plan and now has only 14 telephone measures. However, IRS still tracks this measure.

Appendix II: The 53 IRS Performance

Measures Reviewed

Telephone Assistance

Performance Measures
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 55 GAO-03-143 Tax Filing Performance Measures

Measure name and definitiona

FY 2001 target

and actual

Weaknesses of measure and

consequences Recommendations

CSR level of service

The relative success rate of taxpayers who

call for toll-free services reaching a CSR.

Target:

55%

Actual:

53.7%

Formula lacks clarity because it includes

some automated calls, which overstates

the number of calls answered by CSRs and

thus the level of service being provided by

CSRs.c

Definition lacks clarity because it does not

disclose inclusion of some automated calls,

which could lead to misinterpreted results

or a failure to take proper action to resolve

performance problems.

Remove automated

calls from the

formula.

Toll-free customer satisfaction

Customer’s perception of service received,

with a rating of “4” being the best.

Target:

3.45 average

score

Actual:

3.45 average

score

Not clear because survey only applies to

calls handled by CSRs. Satisfaction is not

measured for calls handled by automation,

which accounted for 76 percent of all calls

in fiscal year 2001.

Potential bias exists (not objective)

because administrators are not required to

listen to the entire call, (1) CSRs could be

prematurely notified that their call was

selected for the survey, thus changing their

behavior towards the caller and affecting

the results of the survey and (2)

administrators may not be able to correctly

answer certain questions on the survey,

which could impair the accuracy of the

data.

Develop a customer

satisfaction survey

for automated

assistance.

Modify procedures

for the toll-free

customer

satisfaction survey,

possibly by requiring

that administrators

listen to the entire

call, to better ensure

that administrators

(1) notify CSRs that

their call was

selected for the

survey as close to

the end of a call as

possible and (2) can

accurately answer

the questions they

are responsible for

on the survey.
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 56 GAO-03-143 Tax Filing Performance Measures

Measure name and definitiona

FY 2001 target

and actual

Weaknesses of measure and

consequences Recommendations

Toll-free tax law qualityd

Evaluates the correctness of answers given by

CSRs to callers with tax law inquiries as well as

CSRs’ conformance with IRS administrative

procedures, such as whether the CSR gave his

or her identification number to the taxpayer.

Target:

74%

Actual:

75.21%

A reliability weakness exists because

evaluations are based on judgments that

are potentially inconsistent. No routine

studies to determine effectiveness of

procedures to ensure consistency of data

collection. Possible inconsistencies affect

the accuracy of the measure and

conclusions about the extent to which

performance goals have been achieved.

Some overlap with toll-free tax law correct

response rate. Both attempt to show the

percentage of callers receiving accurate

responses to tax law questions, but toll-free

tax law quality includes CSR conformance

with administrative procedures in

computing that percentage. Overlap could

cloud the bottom line and obscure

performance results.

Implement annual

effectiveness

studies to validate

the accuracy of data

collection methods

and establish goals

for improving

consistency, as

needed.

See note 1 to the

table.

Toll-free accounts qualitye

Evaluates the correctness of answers given by

CSRs to callers with account-related inquiries

as well as CSRs’ conformance with IRS

administrative procedures, such as whether a

CSR gave his or her identification number to

the taxpayer.

Target:

67%

Actual:

69.17%

A reliability weakness exists because

evaluations are based on judgments that

are potentially inconsistent. No routine

studies to determine effectiveness of

procedures to ensure consistency of data

collection. Possible inconsistencies affect

the accuracy of the measure and

conclusions about the extent to which

performance goals have been achieved.

Some overlap with toll-free account correct

response rate. Both attempt to show the

percentage of callers receiving accurate

responses to account questions, but tollfree accounts quality includes CSR

conformance with administrative

procedures in computing that percentage.

Overlap could cloud the bottom line and

obscure performance results.

Implement annual

effectiveness

studies to validate

the accuracy of data

collection methods

and establish goals

for improving

consistency, as

needed.

See note 1 to the

table.

Average handle time

The average number of seconds CSRs spent

assisting callers. It includes talk and hold times

and the time a CSR spends on work related to

a call after the call is terminated.

Target:

not available

Actual:

609 seconds

Target to be set upon completion of

baseline data collection.f

None.
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 57 GAO-03-143 Tax Filing Performance Measures

Measure name and definitiona

FY 2001 target

and actual

Weaknesses of measure and

consequences Recommendations

Automated completion rate

The percentage of total callers who completed

a selected automated service.

Target:

not available

Actual:

not available

Formula lacks clarity because it assumes

that all callers seeking recorded tax law

information, including those who hang up

before receiving service, received the

information they needed, which could

produce inaccurate or misleading results.

Not clear because definition does not

disclose the previously mentioned

assumption, which could lead to

misinterpreted results or a failure to take

proper action to resolve performance

problems.

Measure removed from the Strategy and

Program Plan; target not available.

Some overlap with total automated calls

answered. Both attempt to show how many

automated calls were answered, but

automated completion rate tries to show

the percentage that completed an

automated service successfully. Overlap

could cloud the bottom line and obscure

performance results.

Revise the measure

so that calls for

recorded tax law

information are not

counted as

completed when

callers hang up

before receiving

service.

Put this measure

back in the Strategy

and Program Plan

after revising the

formula so that calls

for recorded tax law

information are not

counted as

completed when

taxpayers hang up

before receiving

service.

See note 1 to the

table.

CSR services provided

The count of all calls handled by CSRs.

Target:

not available

Actual:

35,799,122

calls answered

Not clear because definition does not

disclose that IRS counts all calls

transferred from one CSR to another as

receiving an additional service, which could

lead to misinterpreted results or a failure to

take proper action to resolve performance

problems. IRS does not have complete

information on why calls were transferred.

Thus, IRS cannot identify appropriate steps

to reduce any inefficiency associated with

transferred calls.

Target to be set upon completion of

baseline data collectionf

Some overlap with CSR calls answered.

Both attempt to show how many calls

CSRs answered, but CSR services

provided tries to count calls requiring the

help of more than one CSR as more than

one call. Overlap could cloud the bottom

line and obscure performance results.

Analyze and use

new or existing data

to determine why

calls are transferred

and use the data to

revise the measure

so that it only

reflects transferred

calls in which the

caller received help

from more than one

CSR (i.e., exclude

calls in which a CSR

simply transferred

the call and did not

provide service).

See note 1 to the

table.
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 58 GAO-03-143 Tax Filing Performance Measures

Measure name and definitiona

FY 2001 target

and actual

Weaknesses of measure and

consequences Recommendations

Toll-free tax law correct response rateg

Evaluates the correctness of answers given

by CSRs to callers with tax law inquiries.

Target:

81.6%

Actual:

79.53%

A reliability weakness exists because

evaluations are based on judgments that

are potentially inconsistent. No routine

studies to determine effectiveness of

procedures to ensure consistency of data

collection. Possible inconsistencies affect

the accuracy of the measure and

conclusions about the extent to which

performance goals have been achieved.

Some overlap with toll-free tax law quality.

Both attempt to show the percentage of

callers receiving accurate responses to tax

law questions, but toll-free tax law quality

includes CSR conformance to

administrative procedures in computing

that percentage. Overlap could cloud the

bottom line and obscure performance

results.

Implement annual

effectiveness

studies to validate

the accuracy of data

collection methods

and establish goals

for improving

consistency, as

needed.

See note 1 to the

table.

Toll-free account correct response rateh

Evaluates the correctness of answers given

by CSRs to callers with account-related

inquiries.

Target:

90.8%

Actual:

88.72%

A reliability weakness exists because

evaluations are based on judgments that

are potentially inconsistent. No routine

studies to determine effectiveness of

procedures to ensure consistency of data

collection. Possible inconsistencies affect

the accuracy of the measure and

conclusions about the extent to which

performance goals have been achieved.

Some overlap with toll-free accounts

quality. Both attempt to show the

percentage of callers receiving accurate

responses to account questions, but tollfree accounts quality includes CSR

conformance with administrative

procedures in computing that percentage.

Overlap could cloud the bottom line and

obscure performance results.

Implement annual

effectiveness

studies to validate

the accuracy of the

data collection

methods and

establish goals for

improving

consistency, as

needed.

See note 1 to the

table.

Toll-free timelinessi

The successful resolution of all issues resulting

from the caller’s first inquiry (telephone only).

Target:

82%

Actual:

82.8%

A reliability weakness exists because

evaluations are based on judgments that

are potentially inconsistent. No routine

studies to determine effectiveness of

procedures to ensure consistency of data

collection. Possible inconsistencies affect

the accuracy of the measure and

conclusions about the extent to which

performance goals have been achieved.

Implement annual

effectiveness

studies to validate

the accuracy of data

collection methods

and establish goals

for improving

consistency, as

needed.
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 59 GAO-03-143 Tax Filing Performance Measures

Measure name and definitiona

FY 2001 target

and actual

Weaknesses of measure and

consequences Recommendations

Toll-free employee satisfaction

The percentage of survey participants that

answered with a 4 or 5 (two highest scores

possible) to the question “considering

everything, how satisfied are you with your

job?”

Target:

55%

Actual:

46%

None observed. None.

CSR response level

The percentage of callers who started

receiving service from a CSR within a specified

period of time.

Target:

49%

Actual:

40.8%

Not clear because formula does not include

calls that received a busy signal or resulted

in a hang-up before a CSR came on the

line, and the definition does not disclose

that exclusion. Performance may be

overstated and the real customer

experience not reflected.

Some overlap with average speed of

answer. Both attempt to show how long

callers waited before receiving service,

except that CSR response level shows the

number of callers receiving service within

30 seconds. Overlap could cloud the

bottom line and obscure performance

results.

Revise measure to

include calls from

taxpayers who tried

to reach a CSR but

did not, such as

those who (1) hungup while waiting to

speak to a CSR, (2)

were provided

access only to

automated services

and hung up, and

(3) received a busy

signal.

See note 1 to the

table.

Average speed of answer

The average number of seconds callers

waited in queue before receiving service from

a CSR.

Target:

not available

Actual:

295 seconds

Target to be set upon completion of

baseline data collection.f

Some overlap with toll-free CSR response

level. Both attempt to show how long

callers waited before receiving service,

except that CSR response level shows the

number of callers receiving service within

30 seconds. Overlap could cloud the

bottom line and obscure performance

results.

See note 1 to the

table.

Note 1: We identified this measure as having partial overlap with another measure. Telephone

assistance officials generally agreed with our assessment and stated that some of these overlapping

measures will be removed from future Strategy and Program Plans. The following recommendation

applies to several measures as noted in the table: “ensure that plans to remove overlapping

measures are implemented.”

a

The names of some measures have been modified slightly from the official names used by IRS for

ease of reading and consistency purposes. For example, we replaced the word “assistor” with CSR.

Also, the definitions of the measures listed in the table come from various IRS sources, including

interviews.

b

The Telephone Routing Interactive System is an interactive routes callers to CSRS or automated

services and provides interactive services. Tele-Tax is a telephone system that provides automated

services only.

c

About 780,000 automated calls were included in the formula during the 2001 filing season. If they

had not been included, the CSR level of service would have decreased by about 1 percentage point.

The effect could be more significant in the future because IRS plans to increase the number of calls

handled through automation.
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 60 GAO-03-143 Tax Filing Performance Measures

d

IRS plans to discontinue the “toll-free tax law quality” measure in fiscal year 2004.

e

IRS plans to discontinue the “toll-free accounts quality” measure in fiscal year 2004.

f

Although these measures did not have a measurable target in place, IRS is taking reasonable steps

to develop a target.

g

IRS changed the name of the “toll-free tax law correct response rate” measure to “customer accuracy

for tax law inquiries” beginning in October 2002.

h

IRS changed the name of the “toll-free account correct response rate” measure to “customer

accuracy for account inquiries” beginning in October 2002.

i

IRS discontinued the “toll-free timeliness” measure beginning in October 2002, and replaced it with a

new “quality timeliness” measure.

Source: GAO comparison of IRS’s December 13, 2000, July 25, 2001, and October 29, 2001,

Strategy and Program Plans with the attributes in appendix I and an Embedded Quality Discussion

Document (7/23/02), which discusses the changes IRS plans for its telephone assistance quality

measures.
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 61 GAO-03-143 Tax Filing Performance Measures

Of the 53 performance measures in our review, 13 are for electronic filing

and assistance.2

 Table 7 has information about each of the 13 measures.

Table 7: Electronic Filing and Assistance Performance Measures

Measure name and definitiona

FY 2001 target

and actual

Weaknesses of measure and

consequences Recommendations

Number of 1040 series returns

electronically filed (millions)

The number of Forms 1040, 1040A,

and 1040EZ filed electronically.

Target:

40.0

Actual:

40.0

Target changed during filing season from 42.0

to 40.0. Changing the target in this instance

was subjective in nature and resulted in an

objectivity weakness as well.

Some overlap with percent of individual

returns electronically filed. Both measures

show the extent of electronic filing by

individuals—one in absolute numbers, the

other as a percent of total filings. Overlap

could cloud the bottom line and obscure

performance results.

Refrain from making

changes to official targets

unless extenuating

circumstances arise.

Disclose any extenuating

circumstances in the

Strategy and Program

Plan and other key

documents.

See note 1 to the table.

Number of business returns

electronically filed (millions)

The number of Forms 941, 1041, and

1065 filed electronically.

Target:

3.7

Actual:

1.66

None observed. None.

Total number of electronically filed

returns (millions).

The number of Forms 1040, 1040A,

1040EZ, 941, 1041 and 1065 filed

electronically.

Target:

43.7

Actual:

41.7

Target changed during filing season from 45.7

to 43.7. Changing the target in this instance

was subjective in nature and resulted in an

objectivity weakness as well.

Refrain from making

changes to official targets

unless extenuating

circumstances arise.

Disclose any extenuating

circumstances in the

Strategy and Program

Plan and other key

documents.

 2

IRS has since added three measures (“number of information returns filed by magnetic

tape,” “percent of information returns filed by magnetic tape,” and “customer satisfactionbusiness”) that were not part of our review. In addition, electronic filing and assistance is

developing new performance measures and goals because it is in the midst of a major

reorganization. When the reorganization is completed, electronic filing and assistance will

no longer be responsible for all the operational programs for which it was responsible in

2001 and 2002. Electronic filing and assistance will remain responsible for strategic

services, Internet development services, and development services. The IRS organizations

assuming responsibility for electronic filing and assistance’s operational programs will be

responsible for the related performance measures and goals.

Electronic Filing and

Assistance Performance

Measures
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 62 GAO-03-143 Tax Filing Performance Measures

Measure name and definitiona

FY 2001 target

and actual

Weaknesses of measure and

consequences Recommendations

Number of information returns

electronically filed (millions)

The total number of information returns

filed electronically. Includes Forms

1098, 1099, 5498, and W-2G and

Schedules K-1. Excludes Forms W-2

and 1099-SSA/RRB received from the

Social Security Administration.

Target:

334.0

Actual:

322.8

Some overlap with percent of information

returns electronically filed. Both measures

show the extent of electronic filing —one in

absolute numbers, the other as a percent of

total filings. Overlap could cloud the bottom

line and obscure performance results.

See note 1 to table.

Percent of information returns

electronically filed

The percentage of total information

returns filed electronically.

Target:

24.4%

Actual:

not availableb

Some overlap with number of information

returns electronically filed. Both measures

show the extent of electronic filing —one in

absolute numbers, the other as a percent of

total filings. Overlap could cloud the bottom

line and obscure performance results.

See note 1 to table.

Percent of individual returns

electronically filed

The percentage of total 1040 series tax

returns (Forms 1040, 1040A, and

1040EZ) filed electronically.

Target:

31%

Actual:

32%

Some overlap with number of 1040 series

returns electronically filed. Both measures

show the extent of electronic filing by

individuals—one in absolute numbers, the

other as a percent of total filings. Overlap

could cloud the bottom line and obscure

performance results.

See note 1 to table.

Number of payments received

electronically (millions)

All individual and all business tax

payments made through the electronic

federal tax payment system (EFTPS).

Target:

64.4

Actual:

53.8

Some overlap with percent of payments

received electronically. Both measures show

the extent to which payments are received

electronically—one in absolute numbers, the

other as a percent of total receipts. Overlap

could cloud the bottom line and obscure

performance results.

See note 1 to table.

Percent of payments received

electronically

The percentage of all individual and

business tax payments made through

EFTPS.

Target:

30%

Actual:

not availableb

Some overlap with number of payments

received electronically. Both measures show

the extent to which payments are received

electronically—one in absolute numbers, the

other as a percent of total receipts. Overlap

could cloud the bottom line and obscure

performance results.

See note 1 to table.

Number of electronic funds

withdrawals/credit card transactions

(millions)

The total number of credit card and

direct debit payments processed

through EFTPS.

Target:

1.0

Actual:

0.63

Some overlap with number and percent of

payments received electronically. The

payments covered by this measure are

included in the universe of payments covered

by the other two measures. Overlap could

cloud the bottom line and obscure

performance results.

See note 1 to table.

Number of IRS digital daily Web site

hits (billions)

The number of hits to IRS’s Web site.

Target:

2.0

Actual:

2.3

Measure is not clear and lacks reliability

because, for example, initial access counts as

multiple hits and movement throughout the

Web site will count as additional hits.

Either discontinue use of

this measure or revise the

way “hits” are calculated

so that the measure more

accurately reflects usage.
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 63 GAO-03-143 Tax Filing Performance Measures

Measure name and definitiona

FY 2001 target

and actual

Weaknesses of measure and

consequences Recommendations

Number of downloads from “IRS .GOV”

(millions)

The total number of tax forms

downloaded from IRS’s Web site.

Target:

311

Actual:

309

None observed. None.

Customer satisfaction – individual

taxpayers

The percentage of taxpayers who

respond “very satisfied” with individual

E-file products.

Target:

76%

Actual:

83%

None observed. None.

Employee satisfaction – Electronic

filing and assistance

The percentage of survey participants

that answered with a 4 or 5 (two

highest scores possible) to the

question “considering everything, how

satisfied are you with your job?”

Target:

66%

Actual:

38%

None observed. None.

Note: We identified this measure as having partial overlap with another measure. Electronic filing and

assistance officials told us that each of the overlapping measures we identified provides additional

information to managers. Determining whether or not to remove overlapping measures is

management’s discretion.

a

The names of some measures have been modified slightly from the official names used by IRS for

ease of reading and consistency purposes. The definitions of the measures listed in the table come

from various IRS sources, including interviews.

b

Despite setting a target, actual data were not available because electronic filing and assistance did

not begin tracking the measure until 2002.

Source: GAO comparison of IRS’s December 13, 2000, July 25, 2001, and October 29, 2001,

Strategy and Program Plans with the attributes in appendix I.
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 64 GAO-03-143 Tax Filing Performance Measures

Of the 53 performance measures in our review, 14 are for field assistance.

Table 8 has information about each of the 14 field assistance measures.

Table 8: Field Assistance Performance Measures

Measure name and definitiona

FY 2001 target

and actual

Weaknesses of measure and

consequences Recommendations

Customer satisfaction

From surveys established in 1998, an

index was created to represent overall

customer satisfaction with field

assistance services, with a “7” being the

best.b

Target:

6.5 average score

Actual:

6.4 average score

None identified. None.

Return preparation contacts

Total number of customers assisted with

tax return preparation, including

electronic and non-electronic tax return

preparation at taxpayer assistance

centers (TAC).

Target:

979,206

Actual:

1,009,387

Name, definition, and formula of

measure are not clear.

Significant manual data collection

process impedes reliability because of

the potential for errors and

inconsistencies that could affect the

accuracy of the measure and

conclusions about the extent to which

performance goals have been achieved.

Some overlap with return preparation

units measure. Both measures attempt

to show number of services provided,

but the contact measure takes the

number of taxpayers served into

account and the units measure counts

the number of returns prepared for those

taxpayers served. Overlap could cloud

the bottom line and obscure

performance results.

Make the name and/or

definition of the measure

more clear to indicate what

is and is not included in the

formula.

See note 1 to the table.

See note 2 to the table.

Field Assistance

Performance Measures
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 65 GAO-03-143 Tax Filing Performance Measures

Measure name and definitiona

FY 2001 target

and actual

Weaknesses of measure and

consequences Recommendations

Geographic coverage

Percentage of W&I taxpayer population

with distinct characteristics, behaviors,

and needs for face-to-face assistance

within a 45-minute commuting distance

from a TAC.

Target:

70%

Actual:

74%

Name, definition, and formula of

measure are not clear; uncertainties

exist among IRS officials about what is

and is not included in the measure.

The formula does not include all

facilities, which could lead to

misinterpreted results or a failure to

properly identify alternative facility types

to resolve access problems.

Because the formula does not include all

facilities, it is difficult for decision makers

to determine if, when, and where

additional TACs are needed.

Make the name and/or

definition of the measure

more clear to indicate what

is and is not included in the

formula.

Revise the formula to

better reflect (1) the

various types of field

assistance facilities,

including alternate sites

and kiosks; (2) the types of

services provided by each

facility; and (3) the facility’s

operating hours.

Return preparation units

Actual number of tax returns prepared,

in whole or in part, in a TAC or

alternative site. (Multiple returns may

be prepared for a single customer.)

Target:

not available

Actual:

not available

Name, definition, and formula of

measure are not clear.

Target to be set upon completion of data

collection.c

Significant manual data collection

process impedes reliability because of

the potential for errors and

inconsistencies that could affect the

accuracy of the measure and

conclusions about the extent to which

performance goals have been achieved.

Some overlap with return preparation

contacts. Both measures attempt to

show number of services provided, but

the contact measure takes the number

of taxpayers served into account and the

units measure counts the number of

returns prepared for those taxpayer’s

served. Overlap could cloud the bottom

line and obscure performance results.

Make the name and/or

definition of the measure

more clear to indicate what

is and is not included in the

formula.

See note 1 to the table.

See note 2 to the table.

TACs total contacts

Total number of customers assisted,

including number of customers assisted

with tax return preparation, at TACs and

alternate sites and via mobile services.

All face-to-face, telephone, and

correspondence contacts are included.

Target:

9,116,099

Actual:

9,681,330

Name, definition, and formula of

measure are not clear.

Significant manual data collection

process impedes reliability because of

the potential for errors and

inconsistencies that could affect the

accuracy of the measure and

conclusions about the extent to which

performance goals have been achieved.

Make the name and/or

definition of the measure

more clear to indicate what

is and is not included in the

formula.

See note 1 to the table.
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 66 GAO-03-143 Tax Filing Performance Measures

Measure name and definitiona

FY 2001 target

and actual

Weaknesses of measure and

consequences Recommendations

Forms contacts

Total number of customers actually

assisted by employees at TACs,

alternate sites, and via mobile services

by (1) providing forms from stock or (2)

using a CD-ROM.

Target:

2,331,000

Actual:

2,388,039

Name, definition, and formula of

measure are not clear.

Significant manual data collection

process impedes reliability because of

the potential for errors and

inconsistencies that could affect the

accuracy of the measure and

conclusions about the extent to which

performance goals have been achieved.

Make the name and/or

definition of the measure

more clear to indicate what

is and is not included in the

formula.

See note 1 to the table.

Tax law contacts

Total number of customers assisted in

TACs, alternate sites, and via mobile

services with inquiries involving general

tax law questions, non-account related

IRS procedures, preparation or review of

Forms W-7, Individual Taxpayer

Identification Number documentation

verification or rejection, a form request

where probing requiring technical tax

law training takes place, and assisting

customers with audit reconsideration.

Target:

not available

Actual:

1,787,338

Name, definition, and formula of

measure are not clear.

Target to be set upon completion of data

collection.c

Significant manual data collection

process impedes reliability because of

the potential for errors and

inconsistencies that could affect the

accuracy of the measure and

conclusions about the extent to which

performance goals have been achieved.

Make the name and/or

definition of the measure

more clear to indicate what

is and is not included in the

formula.

See note 1 to the table.

Account contacts

Total number of customers assisted in

TACs, alternate sites, and via mobile

services with inquiries involving account

related inquiries including math error

notices, Integrated Data Retrieval

System work, payments not attached to

a tax return, CP2000 inquiries, Individual

Taxpayer Identification Number issues

requiring account research, the issuance

of Form 809 receipts, and account

related procedures.

Target:

not available

Actual:

not available

Name, definition, and formula of

measure are not clear.

Target to be set upon completion of data

collection.c

Significant manual data collection

process impedes reliability because of

the potential for errors and

inconsistencies that could affect the

accuracy of the measure and

conclusions about the extent to which

performance goals have been achieved.

Make the name and/or

definition of the measure

more clear to indicate what

is and is not included in the

formula.

See note 1 to the table.

Other contacts

Total number of customers assisted in

TACs, alternate sites, and via mobile

services with Form 2063, U.S. Departing

Alien Income Tax statement, date

stamping tax returns when the customer

is present, non-receipt or incorrect W-2

inquiries, general information such as

Service Center address and directions

to other agencies.

Target:

3,869,000

Actual:

4,496,566

Name, definition, and formula of

measure are not clear.

Significant manual data collection

process impedes reliability because of

the potential for errors and

inconsistencies that could affect the

accuracy of the measure and

conclusions about the extent to which

performance goals have been achieved.

Make the name and/or

definition of the measure

more clear to indicate what

is and is not included in the

formula.

See note 1 to the table.
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 67 GAO-03-143 Tax Filing Performance Measures

Measure name and definitiona

FY 2001 target

and actual

Weaknesses of measure and

consequences Recommendations

Tax law accuracy

The quality of service provided to TAC

customers. Specifically, the accuracy of

responses concerning issues involving

tax law.

Target:

not available

Actual:

not available

Name, definition, and formula of

measure are not clear.

Target to be set upon completion of data

collection.c

Make the name and/or

definition of the measure

more clear to indicate what

is and is not included in the

formula.

Accounts/notices accuracy

The quality of service provided to TAC

customers. Specifically, the accuracy of

responses and/or IDRS transactions

concerning issues involving account

work and notices.

Target:

not available

Actual:

not available

Name, definition, and formula of

measure are not clear.

Target to be set upon completion of data

collection.c

Make the name and/or

definition of the measure

more clear to indicate what

is and is not included in the

formula.

Return preparation accuracy

The quality of service provided to TAC

customers. Specifically, the accuracy of

tax returns prepared in a TAC.

Target:

not available

Actual:

not available

Name, definition, and formula of

measure are not clear.

Target to be set upon completion of data

collection.c

Make the name and/or

definition of the measure

more clear to indicate what

is and is not included in the

formula.

Employee satisfaction

The percentage of survey participants

that answered with a 4 or 5 (two highest

scores possible) to the question

“considering everything, how satisfied

are you with your job.”

Target:

62%

Actual:

51%

None observed. None.

Alternate contacts

Total number of customers assisted at

kiosks, mobile units, and alternate sites.

It includes all face-to-face (including

return preparation), telephone, and

correspondence contacts.

Target:

not available

Actual:

not available

Target to be set upon completion of data

collection.c

Significant manual data collection

process impedes reliability because of

the potential for errors and

inconsistencies that could affect the

accuracy of the measure and

conclusions about the extent to which

performance goals have been achieved.

See note 1 to the table.

Note 1: IRS expects to minimize this potential for errors and inconsistency by equipping all of its

TACS with an on-line automated tracking and reporting system known as the Queuing Management

System (Q-Matic). This system is expected, among other things, to more efficiently monitor customer

traffic flow and eliminate staff time spent completing Form 5311. Because IRS is in the process of

implementing Q-Matic, we are not making any recommendation.

Note 2: We identified this measure as having partial overlap with another measure. Field assistance

officials agreed with our assessment and stated that they plan to remove the “return preparation

contacts” measure from the Strategy and Program Plan. The following recommendation applies to

two measures, as noted in the table: “ensure that plans to remove overlapping measures are

implemented.”

a

The names of some measures have been modified slightly from the official names used by IRS for

ease of reading and consistency purposes. The definitions of the measures listed in the table come

from various IRS sources, including interviews.

b

Field assistance implemented a new customer satisfaction survey in fiscal year 2002. The index was

changed, and a rating of “5” is now best.
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 68 GAO-03-143 Tax Filing Performance Measures

c

Although these measures did not have a measurable target in place, IRS is taking reasonable steps

to develop a target.

Source: GAO comparison of IRS’s December 13, 2000, July 25, 2001, and October 29, 2001,

Strategy and Program Plans with the attributes in appendix I.

Of the 53 performance measures in our review, 11 are for submission

processing.3

 Table 9 has information about each of the 11 submission

processing performance measures.

Table 9: Submission Processing Performance Measures

Measure name and definitiona

FY 2001 target and

actual

Weaknesses of measure and

consequences Recommendations

Individual 1040 series returns filed (paper)b

The number of Forms 1040, 1040A, and

1040EZ filed at the eight W&I submission

processing centers.

Target:

87,869,000

Actual:

74,972,667

None observed. None.

Number of individual refunds issued (paper)b

The number of individual refunds issued by

the eight W&I submission processing centers

after the initial filing of a return.

Target:

48,000,000

Actual:

45,456,534

None observed. None.

Employee satisfaction

The percentage of survey participants that

answered with a 4 or 5 (two highest scores

possible) to the question “considering

everything, how satisfied are you with your

job.”

Target:

60%

Actual:

54%

None observed. None.

 3

 IRS is developing a measure of customer satisfaction for submission processing.

Submission Processing

Performance Measures
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 69 GAO-03-143 Tax Filing Performance Measures

Measure name and definitiona

FY 2001 target and

actual

Weaknesses of measure and

consequences Recommendations

Refund timeliness – individual (paper)b

The percentage of refunds issued to

taxpayers within 40 days of the date IRS

received the individual income tax return.

Target:

96.1%

Actual:

96.75%

Potential reliability weakness

because data collected manually

and evaluations of data based on

judgment. Possible

inconsistencies affect the

accuracy of the measure and

conclusions about the extent to

which performance goals have

been achieved.

Based on the results of

effectiveness studies,

establish goals to

improve consistency, as

needed.

Notice error rate

The percentage of incorrect submission

processing master file notices issued to

taxpayers (includes systemic errors).c

Target:

8.1%

Actual:

14.84%

Potential reliability weakness

because data collected manually

and evaluations of data based on

judgment. Possible

inconsistencies affect the

objectivity of the measure and

conclusions about the extent to

which performance goals have

been achieved.

Based on the results of

effectiveness studies,

establish goals to

improve consistency, as

needed.

Refund error rate – individual (paper)b

The percentage of refunds that have errors

caused by IRS involving, for example, a

person’s name or refund amount (includes

systemic errors).c

Target:

13.6%

Actual:

9.75%

Potential reliability weakness

because data collected manually

and evaluations of data based on

judgment. Possible

inconsistencies affect the

accuracy of the measure and

conclusions about the extent to

which performance goals have

been achieved.

Based on the results of

effectiveness studies,

establish goals to

improve consistency, as

needed.

Letter error rate

The percentage of letters with errors issued to

taxpayers by submission processing

employees (includes systemic errors).c

Target:

11.9%

Actual:

13.10%

Potential reliability weakness

because data collected manually

and evaluations of data based on

judgment. Possible

inconsistencies affect the

objectivity of the measure and

conclusions about the extent to

which performance goals have

been achieved.

Based on the results of

effectiveness studies,

establish goals to

improve consistency, as

needed.

Deposit timeliness (paper)b

Lost opportunity cost of money received by

IRS but not deposited in the bank by the next

day, per $1 billion of deposits, using a

constant 8% annual interest rate.

Target:

$746,712

Actual:

$878,867

None observed. None.
Appendix II: The 53 IRS Performance

Measures Reviewed

Page 70 GAO-03-143 Tax Filing Performance Measures

Measure name and definitiona

FY 2001 target and

actual

Weaknesses of measure and

consequences Recommendations

Deposit error rate

The percentage of payments misapplied

based on the taxpayer’s intent.

Target:

4.9%

Actual:

not availabled

Objectivity weakness because

sampling plan not consistently

implemented.

Potential reliability weakness

because data collected manually

and evaluations of data based on

judgment. Possible

inconsistencies affect the

accuracy of the measure and

conclusions about the extent to

which performance goals have

been achieved.

See note 1 to the table.

Based on the results of

effectiveness studies,

establish goals to

improve consistency, as

needed.

Refund interest paid (per $1 million of

refunds)

The amount of refund interest paid per $1

million of refunds issued.

Target:

$112

Actual:

$128.63

None observed. None.

Submission processing productivity

The weighted workload or work units

processed per staff year expended.

Target:

28,787

Actual:

28,537

Not clear because (1) definition is

not clearly stated, (2) managers

do not understand their unit’s

contribution to the formula and (3)

unit managers do not use the

measure to assess performance.

Revise the measure so it

provides more

meaningful information to

users.

Note 1: We are not making a recommendation regarding the objectivity weakness for the “deposit

error rate” measure because the Treasury Inspector General for Tax Administration recommended

that IRS take steps to ensure that the sampling plan is being implemented consistently, and IRS

reported that steps have been taken.

a

The names of some measures have been modified slightly from the official names used by IRS for

ease of reading and consistency purposes. The definitions of the measures listed in the table come

from various IRS sources, including interviews.

b

“Paper” means that returns filed electronically (or their resulting refunds) are not included in the

measure.

c

A systemic error is an error caused by a computer programming error as opposed to an IRS

employee.

d

IRS could not provide actual data on this measure due to discrepancies in its data.

Source: GAO comparison of IRS’s December 13, 2000, July 25, 2001, and October 29, 2001,

Strategy and Program Plans with the attributes in appendix I.
Appendix III: Comments from the Internal

Revenue Service

Page 71 GAO-03-143 Tax Filing Performance Measures

Appendix III: Comments from the Internal

Revenue Service

Note: GAO comments

supplementing those in

the report text appear at

the end of this appendix.

See comment 1.
Appendix III: Comments from the Internal

Revenue Service

Page 72 GAO-03-143 Tax Filing Performance Measures

See comment 3.

See comment 2.
Appendix III: Comments from the Internal

Revenue Service

Page 73 GAO-03-143 Tax Filing Performance Measures
Appendix III: Comments from the Internal

Revenue Service

Page 74 GAO-03-143 Tax Filing Performance Measures
Appendix III: Comments from the Internal

Revenue Service

Page 75 GAO-03-143 Tax Filing Performance Measures
Appendix III: Comments from the Internal

Revenue Service

Page 76 GAO-03-143 Tax Filing Performance Measures
Appendix III: Comments from the Internal

Revenue Service

Page 77 GAO-03-143 Tax Filing Performance Measures
Appendix III: Comments from the Internal

Revenue Service

Page 78 GAO-03-143 Tax Filing Performance Measures
Appendix III: Comments from the Internal

Revenue Service

Page 79 GAO-03-143 Tax Filing Performance Measures
Appendix III: Comments from the Internal

Revenue Service

Page 80 GAO-03-143 Tax Filing Performance Measures

1. We recognize that IRS’s performance measures cover entire fiscal

 years. We reviewed 53 of the measures for all of fiscal year 2001,

 and we reported the full year’s results in appendix II.

2. We reviewed the business plans for all four program areas we

 reviewed. Although we did not comment specifically about the

 business performance review process in the report, we noted in the

 background and field assistance sections that the business plans

 communicate part of the relationship among the various goals and

 measures.

3. Figure 4 shows an excerpt of field assistance’s business unit plan.

 As noted in the figure, the template used to communicate the

 relationship between goals and measure is missing some key

 components. Figure 2 is our attempt to show the complete relationship

 among IRS’s various goals and measures—it is based on multiple

 documents.

GAO Comments
Appendix IV: GAO Contacts and Staff

Acknowledgments

Page 81 GAO-03-143 Tax Filing Performance Measures

James White (202) 512-9110

Dave Attianese (202) 512-9110

In addition to those named above, Bob Arcenia, Healther Bothwell, Rudy

Chatlos, Grace Coleman, Evan Gilman, Ron Heisterkamp, Ronald Jones,

John Lesser, Allen Lomax, Theresa Mechem, Libby Mixon, Susan Ragland,

Meg Skiba, Joanna Stamatiades, and Caroline Villanueva made key

contributions to this report.

Appendix IV: GAO Contacts and Staff

Acknowledgments

GAO Contacts

Acknowledgments
Bibliography

Page 82 GAO-03-143 Tax Filing Performance Measures

To determine whether the Internal Revenue Service’s (IRS) performance

goals and measures in four key program areas demonstrate results, are

limited to the vital few, cover multiple program priorities, and provide

useful information in decision making, we developed attributes of

performance goals and measures. These attributes were largely based on

previously established criteria found in prior GAO reports; our review of

key legislation, such as the Government Performance and Results Act of

1993 (GPRA) and the IRS Restructuring and Reform Act of 1998; and other

performance management literature. Sources we referred to for this report

follow.

101st Congress. Chief Financial Officer’s Act of 1990. P.L. 101-576.

Washington, D.C.: January 23, 1990.

103rd Congress. Government Performance and Result Act of 1993. P.L.

103-62. Washington, D.C.: January 5, 1993.

103rd U.S. Senate. The Senate Committee on Government Affairs GPRA

Report. Report 103-58. Washington, D.C.: June 16, 1993.

105th Congress. IRS Restructuring and Reform Act. P.L. 105-206.

Washington, D.C.: July 22, 1998.

Internal Revenue Service. Managing Statistics in a Balanced Measures

System. Handbook 105.4. Washington, D.C.: October 1, 2000.

The National Partnership for Reinventing Government. Balancing

Measures: Best Practices in Performance Management. Washington, D.C.:

August 1, 1999.

Office of Management and Budget, Preparation and Submission of

Budget Estimates. Circular No. A-11, Revised. Transmittal Memorandum

No. 72. Washington, D.C.: July 12, 1999.

Office of Management and Budget. Circular A-76, Revised. Supplemental

Handbook, Performance of Commercial Activities. Washington, D.C.:

March 1996 (Revised 1999).

Office of Management and Budget. Managerial Cost Accounting Concepts

and Standards for the Federal Government. Statement of Federal

Financial Accounting Standards, Number 4. Washington, D.C.: July 31,

1995

Bibliography
Related Products

Page 83 GAO-03-143 Tax Filing Performance Measures

U.S. General Accounting Office. Internal Revenue Service: Assessment of

Budget Request for Fiscal Year 2003 and Interim Results of 2002 Tax

Filing Season. (GAO-02-580T). Washington, D.C.: April 9, 2002.

U.S. General Accounting Office. Tax Administration: Assessment of IRS’s

2001 Tax Filing Season. (GAO-02-144). Washington, D.C.: December 21,

2001.

U.S. General Accounting Office. Human Capital: Practices That

Empowered and Involved Employees (GAO-01-1070). Washington, D.C.:

September 14, 2001.

U.S. General Accounting Office. Managing For Results: Emerging

Benefits From Selected Agencies’ Use of Performance Agreements (GAO01-115). Washington, D.C.: October 30, 2000.

U.S. General Accounting Office. Agency Performance Plans: Examples of

Practices That Can Improve Usefulness to Decisionmakers

(GAO/GGD/AIMD-99-69). Washington, D.C.: February 26,1999.

U.S. General Accounting Office. The Results Act: An Evaluator’s Guide to

Assessing Agency Annual Performance Plans (GAO/GGD-10.1.20).

Washington, D.C.: April 1,1998.

U.S. General Accounting Office. Executive Guide: Effectively

Implementing the Government Performance and Results Act (GAO/GGD96-118). Washington, D.C.: June 1996.

U.S. General Accounting Office. Executive Guide: Improving Mission

Performance Through Strategic Information Management and

Technology (GAO/AIMD-94-115). Washington, D.C.: May 1, 1994.

Related Products

(440087)

The General Accounting Office, the investigative arm of Congress, exists to

support Congress in meeting its constitutional responsibilities and to help

improve the performance and accountability of the federal government for the

American people. GAO examines the use of public funds; evaluates federal

programs and policies; and provides analyses, recommendations, and other

assistance to help Congress make informed oversight, policy, and funding

decisions. GAO’s commitment to good government is reflected in its core values

of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost is

through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and fulltext files of current reports and testimony and an expanding archive of older

products. The Web site features a search engine to help you locate documents

using key words and phrases. You can print these documents in their entirety,

including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and

correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site

daily. The list contains links to the full-text document files. To have GAO e-mail

this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily

E-mail alert for newly released products” under the GAO Reports heading.

The first copy of each printed report is free. Additional copies are $2 each. A

check or money order should be made out to the Superintendent of Documents.

GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a

single address are discounted 25 percent. Orders should be sent to:

U.S. General Accounting Office

441 G Street NW, Room LM

Washington, D.C. 20548

To order by Phone: Voice: (202) 512-6000

TDD: (202) 512-2537

Fax: (202) 512-6061

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800

U.S. General Accounting Office, 441 G Street NW, Room 7149

Washington, D.C. 20548

GAO’s Mission

Obtaining Copies of

GAO Reports and

Testimony

Order by Mail or Phone

To Report Fraud,

Waste, and Abuse in

Federal Programs

Public Affairs