This is the accessible text file for GAO report number GAO-03-143 
entitled 'Tax Administration: IRS Needs to Further Refine Its Tax 
Filing Season Performance Measures' which was released on November 22, 
2002.
This text file was formatted by the U.S. General Accounting Office 
(GAO) to be accessible to users with visual impairments, as part of a 
longer term project to improve GAO products accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.
GAO Highlights:
TAX ADMINISTRATION: IRS Needs to Further Refine Its Tax Filing Season 
Performance Measures
Highlights of GAO- 03-143, a report to the Subcommittee on Oversight, 
Committee on Ways and Means, House of Representatives 
Why GAO Did This Study:
The tax filing season, roughly January 1 through April 15, is when 
most taxpayers file their returns, receive refunds, and call or visit 
IRS offices or the IRS Web site with questions.  To provide better 
information about the quality of filing season services, IRS is 
revamping 
its suite of filing season performance measures.  Because the new 
measures 
are part of a strategy to improve service and because filing season 
service affects so many taxpayers, GAO was asked to assess whether the 
new 
measures have the four characteristics of successful performance 
measures 
graphically depicted below.
What GAO Found:
In assessing 53 performance measures across IRSs four program areas, 
GAO 
found that IRS has made significant efforts to improve its performance 
measurement system. Many of the measures satisfied some of the four key 
characteristics of successful performance measures established in 
earlier 
GAO work. Although improvements are ongoing, GAO identified instances 
where 
measures showed weaknesses including the following: (1) The objectivity 
and 
reliability of some measures could be improved so that they will be 
reasonably 
free from significant bias and produce the same result under similar 
circumstances. For example, survey administrators may notify Telephone 
Assistances customer service representatives (CSR) too soon that their 
call was selected to participate in the customer satisfaction survey, 
which could bias CSR behavior towards taxpayers and adversely affect 
the 
measures objectivity. In addition, the measure Electronic Filing and 
Assistance uses to determine the number of Web site hits was not 
reliable 
because it did not represent the actual number of times the Web site is 
accessed.
(2) The clarity of some performance information was affected when that 
measures definition and formula were not consistent.  For example, the 
definition for CSR response level measure is the percentage of 
callers 
who receive service from a CSR within a specified period of time, but 
the 
measure did not include callers who received a busy signal or hung up. 
(3) 
Some suites of measures did not cover governmentwide priorities such as 
quality, timeliness, and cost of service. For example, Field Assistance 
was missing measures for timeliness and cost of service.
[See PDF for image]
What GAO Recommends:
GAO is making recommendations to the Commissioner of Internal Revenue 
directed at taking actions to better ensure that IRS validates the 
accuracy 
of data collection methods for several measures; modifies the formulas 
used 
to compute various measures; and adds certain measures, such as cost of 
service, 
to its suite of measures.
Of GAOs 18 recommendations, IRS agreed with 12 and discussed actions 
that 
had been taken or would be taken to implement them. For 2 of those 12, 
the 
actions discussed by IRS did not fully address GAOs concerns. IRS did 
not 
agree with the other 6 recommendations.
The full report, including GAOs objectives, scope, methodology, and 
analysis 
is available at www.gao.gov/cgi-bin/getrpt?GAO-03-143. For additional 
information about the report, contact James White, 202-512-9110 or 
WhiteJ@gao.gov.
Report to the Chairman, Subcommittee on Oversight, Committee on Ways 
and Means, House of Representatives:
United States General Accounting Office:
GAO:
November 2002:
Tax Administration:
IRS Needs to Further Refine Its Tax Filing Season Performance Measures:
Tax Filing Performance Measures:
GAO-03-143:
Contents:
Letter:
Results in Brief:
Background:
Scope and Methodology:
Filing Season Performance Measures Have Many of the Attributes of 
Successful Measures, but Further Enhancements Are Possible:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Expanded Explanation of Our Attributes and Methodology for 
Assessing IRSs Performance Measures:
Appendix II: The 53 IRS Performance Measures Reviewed:
Appendix III: Comments from the Internal Revenue Service:
GAO Comments:
Appendix IV: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
Acknowledgments:
Bibliography:
Related Products:
Tables:
Table 1: Key Attributes of Successful Performance Measures:
Table 2: Overview of Our Assessment of Telephone Assistance Measures:
Table 3: Overview of Our Assessment of Electronic Filing and Assistance 
Measures:
Table 4: Overview of Our Assessment of Field Assistance Measures:
Table 5: Overview of Our Assessment of Submission Processing Measures:
Table 6: Telephone Assistance Performance Measures:
Table 7: Electronic Filing and Assistance Performance Measures:
Table 8: Field Assistance Performance Measures:
Table 9: Submission Processing Performance Measures:
Figures:
Figure 1: IRSs Mission and the Link between Its Strategic Goals and 
the Elements of Its Balanced Measurement System:
Figure 2: Linkage from IRS Mission to Operating Unit Measure and 
Target:
Figure 3: Performance Measures Should Have Four Characteristics:
Figure 4: Example of Relationship among Field Assistance Goals and 
Measures:
Abbreviations:
CQRS: Centralized Quality Review Site:
CSR: customer service representative:
GPRA: Government Performance and Results Act of 1993:
IRS: Internal Revenue Service:
Q-Matic: Queuing Management System:
TAC: Taxpayer Assistance Center:
W&I: Wage and Investment:
United States General Accounting Office:
Washington, DC 20548:
November 22, 2002:
The Honorable Amo Houghton
Chairman, Subcommittee on Oversight
Committee on Ways and Means
House of Representatives:
Dear Mr. Chairman:
For most taxpayers, their only contacts with the Internal Revenue 
Service (IRS) are associated with the filing of their individual income 
tax returns. Most taxpayers file their returns between January 1 and 
April 15, which is generally referred to as the filing season.
[Footnote 1] In addition to the filing itself, which can be on 
paper or electronic, these contacts generally involve millions of 
taxpayers seeking help from IRS by calling one of IRSs toll-free 
telephone numbers, visiting one of IRSs field assistance centers, or 
accessing IRSs Web site on the Internet (www.irs.gov). Between January 
1 and July 13, 2002, for example, IRS received about 105 million calls 
for assistance over its toll-free telephone lines.[Footnote 2]
As part of a much larger effort to modernize and become more responsive 
to taxpayers, IRS is revamping how it measures and reports its filing 
season performance. The new filing season performance measures are to 
balance customer satisfaction, employee satisfaction, and business 
results, such as the quality of answers to taxpayer inquiries and the 
timeliness of refund issuance. IRS intends to use the balanced measures 
to make managers and frontline staff more accountable for improving 
filing season performance.
Because so many taxpayers are affected by IRSs performance during the 
filing season and because the revamped measures are part of a strategy 
to improve performance, you asked us to review IRSs new set of filing 
season performance measures. Those measures belong to the four program 
areas critical to a successful filing season: telephone assistance; 
electronic filing and assistance; field assistance; and the processing 
of returns, refunds, and remittances (referred to as submission 
processing). Specifically, our objective was to assess whether the key 
performance measures IRS uses to hold managers accountable in the four 
program areas had the characteristics of a successful performance 
measurement system.
Previous GAO work indicated agencies successful in measuring 
performance had performance measures that demonstrate results, are 
limited to the vital few, cover multiple priorities, and provide useful 
information for decision making.[Footnote 3] To determine whether IRSs 
filing season performance measures satisfy these four general 
characteristics, we assessed the measures using nine specific 
attributes.[Footnote 4] Earlier GAO work cited these specific 
attributes as key to successful performance measures. Table 1 is a 
summary of the nine attributes, including the potentially adverse 
consequences if they are missing. All attributes are not equal and 
failure to have a particular attribute does not necessarily indicate 
that there is a weakness in that area or that the measure is not 
useful; rather, it may indicate an opportunity for further refinement. 
An expanded explanation of the nine attributes is included in appendix 
I.
Table 1: Key Attributes of Successful Performance Measures:
[See PDF for image]
Source: Summary of information in appendix I.
[End of table]
We shared these attributes with various IRS officials, who generally 
agreed with their relevance. As discussed in greater detail in the 
separate scope and methodology section of this report, we took many 
steps to validate and ensure consistency in our application of the 
attributes.
We testified before the Subcommittee on Oversight on some of the 
interim results of our assessment in April 2002.[Footnote 5]
Results in Brief:
In assessing 53 performance measures across four of IRSs key filing 
season program areas, we found that the measures satisfied many of the 
nine attributes of successful performance measures previously listed in 
table 1. As part of its agencywide reorganization, IRS has made 
significant efforts to improve its performance measurement system, 
which is to provide useful information about how well IRS performed in 
achieving its goals. The improvement of this system is an ongoing 
process where, in some cases, IRS is only beginning to collect baseline 
information on which to form targets and develop other measures that 
would provide better information to evaluate performance results. 
Despite IRSs progress, we identified instances in all four program 
areas where the individual measures or suites of measures did not meet 
some of our nine attributes. Some of these instances represent 
opportunities for IRS to further refine its measures.
All of the 15 telephone assistance measures had some of the attributes 
of successful performance measures. Of the more significant problems, 
five measures had either clarity or reliability problems and one had an 
objectivity problem. For example,
* five measures did not provide managers and other stakeholders with 
clear information about the programs performance. For example, the 
definition for customer service representative (CSR) response level 
is the percentage of callers who receive service from a CSR within a 
specified period of time, but the formula did not include callers who 
received a busy signal or hung up; this limitation could lead managers 
and other stakeholders to conclude that IRS is providing significantly 
better service than it is.
All of the 13 electronic filing and assistance performance measures 
fulfilled some of the 9 attributes. The most significant problems 
involved changing targets, objectivity, and missing measures. For 
example,
* electronic filing and assistance changed the targets for two of its 
measures during fiscal year 2001, which could distort the assessment of 
performance because what was to be observed changed. For example, it 
changed the target for the number of 1040 series returns filed 
electronically from 42 million to 40 million because midyear data 
indicated that 42 million 1040 series returns were not going to be 
filed electronically. Because of the subjective considerations 
involved, changing the target in this situation also affected the 
measures objectivity.
All of field assistances 14 performance measures satisfied some of the 
attributes. Many of the more important problems involved clarity and 
reliability. In addition, some measures were missing, which could cause 
an emphasis on some program goals at the expense of a balance among all 
goals. For example,
* the methods used to track workload volume and staff hours expended 
required manual input that is subject to errors and inconsistencies, 
which could affect data accuracy and thus the reliability of 8 of field 
assistances 14 measures.
* Field assistance did not have timeliness, efficiency, or cost of 
service measures.
Many of the 11 submission processing measures had the attributes of 
successful performance measures. Some of the more significant problems 
related to clarity and reliability. For example,
* one measure--productivity--was unclear because it is a compilation 
of different types of work IRS performs in processing returns, 
remittances, and refunds and issuing notices and letters. Managers told 
us that they needed specific information related to their own 
operations and that the measures methodology was difficult to 
understand.
In all four program areas, we were unable, because of documentation 
limitations, to verify the linkages among IRSs goals and measures. 
Among other things, such linkages provide managers and staff with a 
road map that shows how their day-to-day activities contribute to 
attaining agencywide goals.
We are making recommendations to the Commissioner of Internal Revenue 
directed at taking actions to better ensure that IRSs filing season 
measures have the four characteristics of successful performance 
measures. For example, we are recommending that IRS modify the formulas 
used to compute various measures; validate the accuracy of data 
collection methods for several measures; and add certain measures such 
as cost of service, to its suite of measures.
We requested comments on a draft of this report from the Commissioner 
of Internal Revenue. We received written comments, which are reprinted 
in appendix III. In his comments, the Commissioner agreed that there 
were opportunities to refine some performance measures and said that 
our observation about the ongoing nature of the performance measurement 
process was on target. The Commissioner agreed with 12 of our 18 
recommendations and discussed actions that had been taken or would be 
taken to implement them. In 2 of those cases, the actions discussed by 
IRS did not fully address our concerns. The Commissioner disagreed with 
the other 6 recommendations. We discuss the Commissioners comments in 
the Agency Comments and Our Evaluation section of the report.
Background:
In keeping with the Government Performance and Results Act of 
1993 (GPRA),[Footnote 6] IRS revamped its set of filing season 
performance measures as part of a massive, ongoing modernization 
effort. Congress mandated the modernization effort in the IRS 
Restructuring and Reform Act of 1998[Footnote 7] and intended that IRS 
would better balance service to taxpayers with enforcement of the tax 
laws. To implement the modernization mandate, the Commissioner of 
Internal Revenue developed a strategy composed of five interdependent 
components. One of those components is the development of balanced 
performance measures.[Footnote 8]
Balanced measures are to emphasize accountability for achieving 
specific results and to reflect IRSs priorities, which are articulated 
in its mission and its three strategic goals--top quality service to 
all taxpayers through fair and uniform application of the law, top 
quality service to each taxpayer in every interaction, and productivity 
through a quality work environment. IRS has defined three elements of 
balanced measures--(1) customer satisfaction, (2) employee 
satisfaction, 
and (3) business results (quality and quantity measures)--to ensure 
balance 
among its priorities. Figure 1 shows IRSs mission and the link between 
its 
strategic goals and the three elements of IRSs balanced measurement 
system.
Figure 1: IRSs Mission and the Link between Its Strategic Goals and 
the Elements of Its Balanced Measurement System:
[See PDF for image]
Source: GAO depiction of information in IRS Publication 3561 and IRSs 
Progress Report (December 2001).
[End of figure]
IRS intends to use the balanced measures to make managers and frontline 
staff more accountable for improving filing season performance. We 
reviewed the performance measures in the four programs areas that 
interact with taxpayers the most during the filing season--telephone 
assistance, electronic filing and assistance, field assistance, and 
submission processing. Each of these program areas is part of IRSs 
Wage and Investment (W&I) operating division, which generally serves 
taxpayers whose only income is from wages and investments.[Footnote 9] 
Although IRS had measures of performance prior to the reorganization, 
IRS managers have spent much effort to revamp the filing season 
performance measures since that time.
An important aspect of IRSs progress in the challenging task of 
improving its performance measures was the development of a new 
Strategic Planning, Budgeting, and Performance Management process in 
2000. As part of that process, IRS prepares an annual Strategy and 
Program Plan that communicates some of the various levels of IRSs 
goals (e.g., strategic goals, operating division goals) and many 
performance measures.[Footnote 10] Although the Strategy and Program 
Plan does not document all the linkages among the various goals and 
performance measures, figure 2 is an example we developed to 
demonstrate the complete relationship from the agency level mission 
down to the operating units measures and targets.
Figure 2: Linkage from IRS Mission to Operating Unit Measure and 
Target:
[See PDF for image]
Source: GAO Analysis of IRSs Strategy and Program Plan (October 29, 
2001), the W&I Business Performance Review (January 2002), IRSs 
Progress Report (December 2001) and IRS Publication 3561.
[End of figure]
The Strategy and Program Plan is an important document because the 
Commissioner holds IRS managers accountable for the results of the 
performance measures contained within it. In addition, many of the 
measures within the document are presented to outside stakeholders, 
such as Congress and the public, as key indicators of IRSs 
performance. The Strategy and Program Plan is the source of the 53 
measures we reviewed in the four programs.
As we discussed in our June 1996 guide on implementing GPRA,[Footnote 
11] agencies that were successful in measuring performance strived to 
establish performance measures that were based on four general 
characteristics. Those four characteristics are shown in figure 3 as 
applicable to the four filing season programs we reviewed and are 
described in more detail following the figure.
Figure 3: Performance Measures Should Have Four Characteristics:
[See PDF for image]
Source: GAO.
[End of figure]
Demonstrate results. Performance measures should show an organizations 
progress towards achieving an intended level of performance or results. 
Specifically, performance goals establish intended performance, and 
measures can be used to assess progress towards achieving those goals.
Be limited to the vital few. Limiting measures to core program 
activities enables managers and other stakeholders to assess 
accomplishments, make decisions, realign processes, and assign 
accountability without having an excess of data that could obscure 
rather than clarify performance issues.
Cover multiple priorities. Performance measures should cover many 
governmentwide priorities, such as quality, timeliness, cost of 
service, customer satisfaction, employee satisfaction, and outcomes. 
Performance measurement systems need to include incentives for managers 
to strike the difficult balance among competing interests. One or two 
priorities should not be overemphasized at the expense of others. IRSs 
history shows why this balance is important. Because of its emphasis on 
achieving certain numeric targets, such as the amount of dollars 
collected, IRS failed to adequately consider other priorities, such as 
the fair treatment of taxpayers.
Provide Useful Information for Decision Making. Performance measures 
should provide managers and other stakeholders timely, action-oriented 
information in a format that helps them make decisions that improve 
program performance. Measures that do not provide managers with useful 
information will not alert managers and other stakeholders to the 
existence of problems nor help them respond when problems arise.
On the basis of these four characteristics of successful performance 
measures, we used various performance management literature to develop 
a set of nine specific attributes that we used as criteria for 
assessing IRSs filing season performance measures. The nine attributes 
are linkage, clarity, measurable target, objectivity, reliability, core 
program activities, limited overlap, balance, and governmentwide 
priorities. Appendix I describes these attributes in more detail.
Scope and Methodology:
As previously mentioned, we focused our work on four key filing season 
programs--telephone assistance, electronic filing and assistance, 
field assistance, and submission processing--within W&I. IRS officials 
identified the performance measures in the Strategy and Program Plan to 
be the highest, most comprehensive level of measures for which they are 
accountable. After discussions with IRS, we decided to review all
53 measures in the Strategy and Program Plan relating to the four 
filing season programs. We used W&Is draft fiscal year 2001 - 2003 
Strategy and Program Plan (dated July 25, 2001) to conduct our review 
and updated relevant information with the final plan (dated October 29, 
2001). Appendix II describes each measure we reviewed in the four 
program areas and provides other relevant information, such as targets 
and potential weaknesses.
Our review focused on whether IRSs new set of filing season 
performance measures had the characteristics of a successful 
performance measurement system (i.e., demonstrated results, were 
limited to the vital few, covered multiple priorities, and provided 
useful information for decision making). For use as criteria in 
assessing the measures, and as detailed in appendix I, we identified 
nine attributes of performance measures from various sources, such as 
earlier GAO work, Office of Management and Budget Circular No. A-
11,[Footnote 12] GPRA, and IRSs handbook on Managing Statistics in a 
Balanced Measures System.[Footnote 13] We shared our attributes with 
IRS officials from various organizations that have a role in developing 
or monitoring performance measures. Those units included IRSs 
Organizational Performance Division and several W&I units, such as 
Strategy and Finance; Planning and Analysis; Customer Account Services; 
and Communications, Assistance, Research, and Education. Officials in 
these units generally agreed with the relevance of our attributes and 
our assessment approach.
We applied the 9 attributes to the 53 filing season measures in a 
systematic manner, but some judgment was required. To ensure 
consistency and reliability in our application of the attributes, we 
had one staff person responsible for each of the four areas. That staff 
person prepared the initial analysis and at least two other staff 
reviewed those detailed results. Several staff reviewed the results for 
all four areas. We did not do a detailed assessment of IRSs 
methodology for calculating the measures, but looked only at 
methodological issues as necessary to assess whether a particular 
measure met the overall characteristics of a successful performance 
measure.
In applying the attributes, we analyzed numerous pieces of 
documentation, such as IRSs Congressional Budget Justification, Annual 
Performance Plan, and data dictionary,[Footnote 14] and many other 
reports and documents dealing with the four IRS programs, goals, 
performance measures, and improvement initiatives. We interviewed IRS 
officials at various levels within telephone assistance, electronic 
filing and assistance, field assistance, and submission processing to 
understand the measures, their methodology, and their relationship to 
goals, among other things. We also interviewed officials from various 
IRS organizations that are involved in managing, collecting, and/or 
using performance data, such as the Organizational Performance 
Division; Strategy and Finance; Customer Account Services; Statistics 
of Income; and the Centralized Quality Review Site; and a 
representative of an IRS contractor, Pacific Consulting Group, 
responsible for analyzing and reporting the results of telephone 
assistances customer satisfaction survey. Appendix I provides more 
detail on the nine attributes we used, including explanations and 
examples of each attribute and information on our methodology for 
assessing each attribute.
We conducted our review in Atlanta, Ga; Washington, D.C; Cincinnati, 
Ohio; and Memphis, Tenn. from September 2001 to September 2002 in 
accordance with generally accepted government auditing standards.
Filing Season Performance Measures Have Many of the Attributes of 
Successful Measures, but Further Enhancements Are Possible:
The 53 filing season performance measures included in our review have 
many of the attributes of successful performance measures, as detailed 
in appendix I. For example, in all four of the program areas we 
reviewed, most measures covered the core activities of each program and 
had targets in place. In addition, IRS had several on-going initiatives 
aimed at improving its measures, such as telephone assistances efforts 
to revamp all aspects of its quality measures.
At the same time, however, the measures did not satisfy all the 
attributes, indicating the potential for further enhancements. The nine 
attributes we used to assess each measure are not equal and failure to 
have a particular attribute does not necessarily indicate that there is 
a weakness in that area. In some cases, for example, a measure may not 
have a particular attribute because benchmarking data are being 
collected or a measure is being revised. Likewise, a noted weakness, 
such as a measure not having clarity or being reliable, does not mean 
that the measure is not useful. For example, telephone assistances 
CSR level of service measure does not meet our clarity attribute 
because its name and definition indicate that only calls answered by 
CSRs are included, but its formula includes some calls answered by 
automation. This defect currently does not impair the measures 
usefulness because the number of automated calls is fairly 
insignificant. Other weaknesses, however, could lead managers or other 
stakeholders to draw the wrong conclusions, overlook the existence of 
problems, or delay resolving problems. For example, electronic filing 
and assistances number of IRS digital daily Web site hits measure 
was not considered clear or reliable because it systematically 
overstates the number of times the Web site is accessed. In total, 
therefore, the weaknesses identified should be considered areas for 
further refinement. Such refinements are not expected to be costly or 
involve significant additional effort on the part of IRS because in 
many instances our recommendations only include modifications or 
increased rigor to procedures or processes already in place.
The rest of this report discusses the results of our analysis for each 
of the four program areas--telephone assistance, electronic filing and 
assistance, field assistance, and submission processing.
Telephone Assistance Measures:
As shown in table 2, all 15 of IRSs telephone performance measures 
have some of the attributes of successful performance 
measures.[Footnote 15] However, as summarized in this section, the 
measures have several shortcomings. For example, we identified 
opportunities to improve the clarity of five measures and the 
reliability of five other measures. Table 6 in appendix II has more 
detailed information about each telephone measure, including any 
weaknesses we identified and any recommendations for improvement.
Table 2: Overview of Our Assessment of Telephone Assistance Measures:
[See PDF for image]
Note: A check mark denotes that the measure has the attribute.
[A] We were unable to verify the linkages between goals and measures 
because of insufficient documentation.
[B] Core program activities of telephone assistance are to provide 
timely and accurate assistance to taxpayers with inquiries about the 
tax law and their accounts.
[C] IRS also refers to CSRs as assistors.
[D] IRS considers that these measures are balanced because they address 
priorities, such as customer and employee satisfaction and business 
results. However, including measures, such as cost of service, could 
improve the balance of telephone assistances program priorities.
Source: GAO analysis.
[End of table]
No Documentation Shows the Complete Linkage between Agencywide Goals 
and Telephone Measures:
Although telephone assistance management stated that their goals and 
measures generally aligned, we were unable to verify this because no 
documentation shows the complete relationship. For example, some 
documentation may show a link from a measure to an agencywide goal, but 
the operating division level goals were omitted. When we attempted to 
create the linkage ourselves, we found it difficult to determine how 
some measures related to the different agencywide and operating 
division goals. When we asked some IRS officials to describe the 
complete link, they too had a difficult time and were uncertain of some 
connections.
Telephone assistance managers stated that staff received performance 
management training that should help them to understand their role in 
helping the organization achieve its goals. However, having clear and 
complete documentation would provide evidence that linkages exist and 
help prevent misunderstandings. When employees do not understand the 
relationship between goals and measures, they may not understand how 
their work contributes to agencywide efforts and, thus, goals may not 
be achieved.
Most Telephone Measures Have Clarity:
Ten of the 15 measures have clarity (e.g., automated calls answered 
clearly describes the count of all toll-free calls answered at customer 
service sites by automated service). However, five measures contain or 
omit certain data elements that can cause managers or other 
stakeholders to misunderstand the level of performance. For example, 
the CSR response level, is defined as the percentage of callers who 
started receiving service from a CSR within a specified period of time. 
However, this may not reflect the real customer experience at IRS 
because the formula for computing the measure does not include callers 
who tried to reach a CSR but did not, such as callers who (1) hung up 
while waiting to speak to a CSR, (2) were provided access only to 
automated services and hung-up, and (3) received a busy 
signal.[Footnote 16] (The other four measures, as noted in table 6 in 
appendix II, are CSR level of service, automated completion rate, 
CSR service provided, and toll-free customer satisfaction.):
Measures that do not provide clear information about program 
performance may affect the validity of managers and stakeholders 
assessments of IRSs performance, possibly leading to a 
misinterpretation of results or a failure to take proper action to 
resolve performance problems.
Most Telephone Measures Have Targets:
Eleven of the 15 measures have numerical targets that facilitate the 
future assessment of whether overall goals and objectives were 
achieved. Of the four measures with no targets, three were measures for 
which IRS was collecting data for use in developing first-time targets 
and one was a measure (automated completion rate) that IRS was no 
longer tracking in the Strategy and Program Plan. Although we generally 
disagree with the removal of the automated completion rate measure 
from the Strategy and Program Plan, as described in an upcoming 
section, not having targets in these instances is reasonable.
Data Collection Methods for Telephone Assistances Customer 
Satisfaction Measure Are Not Always Objective:
IRS determines customer satisfaction with its toll-free telephone 
assistance through a survey administered to taxpayers who speak with a 
CSR.[Footnote 17] We observed survey collection methods in Atlanta that 
were not always objective; that is, the administrators did not always 
follow prescribed procedures for selecting calls to participate in the 
survey. Not following prescribed procedures produces a systematic bias 
that could compromise the randomness of the sample. Also, IRS 
procedures do not require that administrators listen to the entire 
call. Although administrators are instructed to notify the CSR towards 
the end of a call that the call was selected for the survey, this may 
not occur. If an administrator begins listening to a call after it has 
started, it can be difficult to determine the full nature of the 
taxpayers question and thus whether the conversation is about to end. 
As a result, an administrator could prematurely notify a CSR that the 
call was selected for the survey, which could change the CSRs behavior 
towards the taxpayer and affect the results of the survey and the 
measure. In addition, administrators may not be able to correctly 
answer certain questions on the survey, which could impair any analysis 
of those answers. We discussed these issues with a representative of 
the IRS contractor (Pacific Consulting Group) responsible for analyzing 
and reporting the survey results who said that (1) he was aware of 
these problems and (2) the same problems existed at other locations.
IRS has taken corrective action on one of these weaknesses. Because 
management decided that the procedures for selecting calls to 
participate in the customer satisfaction survey were too difficult to 
follow, it revised them. Sites began using the revised sampling 
procedures in July 2002.
Reliability of Five Telephone Quality Measures Is Suspect:
The reliability of telephone assistances five quality measures (toll-
free tax law quality, toll-free accounts quality, toll-free tax law 
correct response rate, toll-free account correct response rate, and 
toll-free timeliness) is suspect because of potential inconsistencies 
in data collection that arise due to differences among individual 
reviewers judgment and perceptions.[Footnote 18] Although it is not 
certain how much variation among reviewers exists, errors could occur 
throughout data collection and could affect the results of the measures 
and conclusions about the extent to which performance goals have been 
achieved.
Reliability and credibility increase when performance data are checked 
or tested for significant errors. IRS has conducted consistency reviews 
in the past and found problems. It has taken steps to improve 
consistency, the most important of which was the establishment of the 
Centralized Quality Review Site (CQRS).[Footnote 19] Among other 
controls within CQRS that are designed to enhance consistency, 
reviewers are to receive the same training and gather to discuss cases 
where the guidance is not clear. IRS has conducted one review to 
determine the effectiveness of CQRS and its efforts to improve 
consistency since IRSs October 2000 reorganization and continues to 
find some problems.
At the time of our review, IRS was reviewing the five quality measures 
as part of an ongoing improvement initiative. Since that time, it 
redesigned many aspects of the measures, including what is measured, 
how the measures are calculated, how data are collected, and how people 
are held accountable for quality.[Footnote 20] Changes emanating from 
this initiative may further enhance consistency.
Telephone Measures Cover Core Program Activities:
Telephone assistances core program activities are to provide timely 
and accurate assistance to taxpayers with inquiries about the tax law 
and their accounts. IRS has at least one measure that directly 
addresses each of these core activities. For example, toll-free 
accounts quality is a measure that shows the percentage of accurate 
responses to taxpayers account related questions.
Some Overlap Exists between Telephone Assistance Measures:
The amount of overlap that exists between measures is a managerial 
decision. Of the 15 telephone measures we reviewed, 10 have at least 
partial overlap. For example, both the CSR response level and 
average speed of answer measures attempt to show how long a taxpayer 
waited before receiving service, except that the former shows the 
number of taxpayers receiving service within 30 seconds while the 
latter shows the average wait time for all taxpayers. (Table 6 in 
appendix II has information on other overlapping measures.):
IRS officials said that overlapping measures can add value to 
managements decision-making process because each measure provides a 
nuance that can be missed if both measures were not present. For 
example, the CSR calls answered measure shows the number of taxpayer 
calls answered while the CSR services provided measure attempts to 
account for situations in which more than one CSR was involved in 
handling a single call. At the same time, however, overlapping measures 
(1) leave managers to sift through redundant, sometimes costly, 
information to determine goal achievement and (2) could confuse outside 
stakeholders, such as Congress.
Although we are not suggesting that IRS stop tracking or reporting any 
of the overlapping measures, we question whether IRS has limited the 
telephone measures included in the Strategy and Program Plan to the 
vital few. Telephone officials agreed with this assessment and stated 
that some of the overlapping measures will be removed from future 
Strategy and Program Plans.
Telephone Measures Do Not Fully Cover Governmentwide Priorities:
When considering governmentwide priorities, such as quality, 
timeliness, cost of service, and customer and employee satisfaction, 
telephone assistance is missing two measures--(1) cost of service and 
(2) a measure of customer satisfaction for automated services, as 
described below.
* Cost of Service. According to key legislation[Footnote 21] and 
accounting standards,[Footnote 22] agencies should develop and report 
cost information. Besides showing financial accountability in the use 
of taxpayer dollars, the cost information called for can be used for 
various purposes, such as authorizing and modifying programs and 
evaluating program performance. IRS does not report the average cost to 
answer a taxpayers inquiry by telephone. A cost-per-call analysis 
could provide a link between program goals and costs, as required by 
GPRA, and help IRS management and Congress decide about future 
investments in telephone assistance. IRS officials said they would like 
to develop a cost of services measures and are trying to determine what 
information would be meaningful to include or exclude in the 
calculation.
* Customer Satisfaction for Automated Services. Although IRS 
projections show that about 70 percent of its fiscal year 2002 calls 
would be handled by automation, it has no survey mechanism in place to 
determine taxpayers satisfaction with these automated services. IRS 
officials agreed this would be a meaningful measure and want to develop 
one for the future, but no implementation plans have been established.
Also, as previously mentioned, IRS has removed the automated 
completion rate measure from its Strategy and Program Plan. We 
realize, as noted in table 6 in appendix II, that this measure has 
limitations that need to be addressed. However, because such a large 
percentage of calls are handled by automation and because IRS plans to 
serve even more calls with automation in the future, re-inclusion of 
that measure in the Strategy and Program Plan may be warranted if the 
associated problems can be resolved.
Telephone Measures Are Balanced:
Telephone assistance has measures in place for customer satisfaction, 
employee satisfaction, and business results and, therefore, IRS 
considers the measures balanced. However, including other measures, 
such as a cost of service measure, as previously described, could 
further enhance the balance of program priorities.
Electronic Filing and Assistance Measures:
As shown in table 3, all 13 of electronic filing and assistances 
performance measures have some of the attributes of successful 
performance measures. However, as summarized in this section, the 
measures have some shortcomings. For example, several of the measures 
had some overlap and two measures had shortcomings related to the 
changing of targets during the fiscal year. Table 7 in appendix II has 
more detailed information about each electronic filing and assistance 
measure, including any weaknesses we identified and any recommendations 
for improvement.
Table 3: Overview of Our Assessment of Electronic Filing and Assistance 
Measures:
[See PDF for image]
Note: A check mark denotes that the measure has the attribute.
[A] We were unable to verify the linkages between goals and measures 
because of insufficient documentation.
[B] Electronic filing and assistances core program activities are to 
provide individual and business taxpayers with the capability to 
transact and communicate electronically with IRS.
[C] Electronic filing and assistance measures address most 
governmentwide priorities, such as quantity, customer satisfaction, and 
employee satisfaction; however, they do not cover two important 
priorities--quality and cost of service.
Source: GAO analysis.
[End of table]
Overall Alignment of Electronic Filing and Assistances Goals and 
Measures Not Fully Documented:
Electronic filing and assistances 13 performance measures are aligned 
with IRSs overall mission and IRSs strategic goals. However, we were 
unable to validate whether the lower level goals, such as electronic 
filing and assistances operational goals and improvement projects, are 
linked to the agencywide strategic level goals and operating division 
performance measures because there is not complete documentation 
available to show that linkage.
Electronic filing and assistances managers stated that goals and 
measures generally align and that employee briefings were held to 
communicate their goals to the organization. It is essential that all 
staff be familiar with IRSs mission and goals, electronic filing and 
assistances goals and performance measures, and how electronic filing 
and assistance determines whether it is achieving its goals so that 
staff know how their day-to-day activities contribute to the goals and 
IRSs overall mission. When this is lacking, priorities may not be 
clear and staff efforts may not be tied to goal achievement.
Most Electronic Filing and Assistance Measures Have Clarity:
All but one of electronic filing and assistances 13 performance 
measures had clarity. The number of IRS digital daily Web site hits 
measure, which is defined as the number of hits to IRSs Web site, is 
not clear because its formula counts multiple hits every time a user 
accesses the sites home page and counts a hit every time a user moves 
to another page on the Web site. The formula is not consistent with the 
definition because it does not represent the actual number of times the 
Web site is accessed.
In its fiscal year 2003 Annual Performance Plan,[Footnote 23] IRS 
acknowledged limitations with this measure as follows.
..changes in the IRS Web design may cause a decrease in the 
number of hits recorded in both [fiscal years] 2002 and 2003. This 
decrease will be due to improved Web site navigation and search 
functions, which may reduce the amount of random exploration by users 
to find content. The decrease will also be due to better design of the 
Web pages themselves that will reduce the number of graphics and other 
items that are used to create the Web page, all of which are counted as 
hits when a page is accessed.:
In our report on IRSs 2001 tax filing season, we recommended that IRS 
either discontinue the use of hits as a measure of the performance of 
its Web site or revise the way hits are calculated so that the 
measure more accurately reflects usage.[Footnote 24] IRS responded that 
it should continue to count hits as a measure of the Web sites 
performance because hits indicate site traffic and can be used to 
measure system performance and estimate system needs. However, 
officials stated that they could improve their method of counting 
hits once they had implemented a more sophisticated, comprehensive 
Web analytical program. According to electronic filing and assistance 
officials, IRS introduced its redesigned Web site in January 2001 and 
implemented a new analytical program, but hits are still being 
calculated the same way.
Two Electronic Filing and Assistance Measures Had Targets Changed and 
Lack Objectivity:
Electronic filing and assistance changed the targets for two measures-
-number of 1040 series returns filed electronically[Footnote 25] and 
total number of returns electronically filed--during fiscal year 
2001. Changing targets could distort the assessment of performance 
because what was to be observed changed. No major event (such as 
legislation that affected the ability of many taxpayers to file 
electronically) happened that warranted changing the targets in the 
strategic plan. Instead, electronic filing and assistance changed the 
target for the first of those measures from 
42 million returns to 40 million returns because IRSs Research 
Divisions midyear data indicated that 42 million 1040 series returns 
were not going to be filed electronically. Because the number of 1040 
series returns filed electronically is a subset of the total number of 
returns filed electronically, electronic filing and assistance also 
reduced the target for total electronic filings. Because of these 
subjective considerations, changing the targets in this situation also 
affected the objectivity of these measures.
Electronic Filing and Assistance Measures Are Reliable, with One 
Exception:
Of electronic filing and assistances 13 performance measures, we 
considered 12 to be reliable because the data on performance comes from 
sources, such as IRSs masterfile[Footnote 26] and computer program 
runs, that are subject to validity checks. The one measure we did not 
consider reliable was the number of IRS digital daily Web site hits, 
because it does not represent the actual number of times the Web site 
is accessed, as previously described.
Measures Cover Electronic Filing and Assistances Core Program 
Activities:
Electronic filing and assistances core program activities are to 
provide individual and business taxpayers the capability to transact 
and communicate electronically with IRS. Electronic filing and 
assistance focuses on taxpayers ability to file their returns, pay 
their taxes, receive assistance, and obtain information electronically. 
These core activities are all covered by the 13 performance measures.
Overlap Exists among Electronic Filing and Assistance Measures:
Seven of the 13 electronic filing and assistance measures had partial 
overlap. For example, the number of 1040 series returns electronically 
filed and percent of individual returns electronically filed 
measures provide related information on a key program activity. The 
difference is that the former is a count of the number filed 
electronically while the latter is the percentage of total individual 
tax returns filed electronically. (Table 7 in appendix II has 
information on other overlapping electronic filing and assistance 
measures.):
The amount of overlap to tolerate among measures is managements 
judgment. Electronic filing and assistance officials told us that each 
of the overlapping measures we identified provides additional 
information to managers. For example, the number of 1040 series 
returns electronically filed provides managers with information on the 
size of the electronic return workload whereas the percent of 
individual returns electronically filed tells them how they are doing 
in relation to IRSs long-term strategic goal of 80 percent. IRS 
officials also pointed out that both number and percent performance 
measures exist because external customers, such as the press, like to 
use the measures for reporting purposes.
Electronic Filing and Assistances Measures Do Not Cover Some 
Governmentwide Priorities, Thus Hindering Balance:
Although electronic filing and assistances measures address several 
governmentwide priorities, such as quantity, customer satisfaction, and 
employee satisfaction, they do not cover two important priorities--
quality and cost of service. As a result, its performance measurement 
system is not fully balanced.
Electronic filing and assistance classifies four of its performance 
measures as quality measures, but the measures are merely counts of 
certain types of electronic transactions (such as number of payments 
received electronically). On the other hand, it tracks what we 
consider to be quality measures (i.e., processing accuracy[Footnote 
27] and refund timeliness, electronically filed)[Footnote 28] but 
those measures are not in the Strategy and Program Plan. These quality 
measures and others, such as one that tracks the number of electronic 
returns rejected,[Footnote 29] could be important indicators of program 
success or failure. For example, IRS data indicate that many electronic 
tax returns are rejected; a measure that captures the volume of rejects 
could help to focus managements attention on the cause of those 
rejects.
Also, similar to our discussion of a cost of service measure in the 
telephone section, a cost-per-electronically filed return could 
provide a link between program goals and costs, as required by GPRA, 
and help IRS management and Congress decide about future investments in 
electronic filing and assistance.
Field Assistance Measures:
As shown in table 4, all 14 of field assistances performance measures 
have some of the attributes of successful performance measures. 
However, as summarized in this section, the measures have several 
shortcomings, primarily with respect to clarity, reliability, and 
balance. Table 8 in appendix II has more detailed information about 
each field assistance measure, including any weaknesses we identified 
and any recommendations for improvement.
Table 4: Overview of Our Assessment of Field Assistance Measures:
[See PDF for image]
Note: A check mark denotes that the measure has the attribute.
[A] We were unable to verify the linkages between goals and measures 
because of insufficient documentation.
[B] Core program activities of field assistance are to provide face-to-
face assistance, education, and compliance services.
[C] Although field assistance continues to develop its suite of 
performance measures, important measures of timeliness, efficiency or 
productivity, and cost of service are missing and impair balance.
Source: GAO analysis.
[End of table]
Relationship between Goals and Field Assistance Measures Not Complete:
Field assistance recognizes the importance of creating a clear 
relationship between goals and measures and has developed a template 
that shows some of that relationship. Figure 4 is an excerpt of the 
template, with the completed portions, as of October 2002, shown in 
gray.
Figure 4: Example of Relationship among Field Assistance Goals and 
Measures:
[See PDF for image]
Source: GAOs analysis of field assistances business plan template.
[End of figure]
Although the template demonstrates a noteworthy effort to show a clear 
link between goals and measures, it omits the link to IRSs mission, 
IRSs strategic goals, and field assistances improvement projects. 
These links are important because they serve as the bridge between 
long-term strategic goals and short-term daily operational goals, which 
can, among other things, be used for holding IRS and the field 
assistance program accountable for achieving those goals. Also, 
officials told us that the completed template would only cite the type 
of performance measure--employee satisfaction, customer satisfaction, 
or business results--not the specific measure and target. The link to 
the specific measure provides additional information needed to clearly 
communicate the alignment of goals and measures throughout the agency, 
and the target communicates the level of performance the operating 
division hopes to achieve.
Many Field Assistance Measures Lack Clarity:
Many of field assistances measures lack clarity. For example, the 
geographic coverage measure is unclear, even to IRS officials, 
because it is not evident by its name or definition what is or is not 
included in the measures formula. Specifically, officials debated 
whether or not the measure included alternate sites[Footnote 30] and 
kiosks.[Footnote 31] Similarly, the formula only considers the location 
of Taxpayer Assistance Centers (TAC), not their hours of operation or 
services provided. Although we saw no evidence that this lack of 
clarity led to adverse consequences, it could. For example, management 
or other stakeholders may determine that TACs are needed in certain 
areas of the country to improve geographic coverage when, in fact, 
alternate sites and/or kiosks are already serving those areas. IRS 
officials said that they have plans to revise the formula to include 
alternate sites and kiosks. (The other measures that lack clarity, as 
described in table 8 of appendix II, are return preparation contacts, 
return preparation units, TACs total contacts, forms contact, 
tax law contacts, account contacts, other contacts, tax law 
accuracy, accounts/notices accuracy, and return preparation 
accuracy.):
All Field Assistance Measures Are Objective and Have Targets That Are 
Either in Place or Being Established:
We determined that all of field assistances 14 performance measures 
are objective because, to the greatest extent possible, they are free 
of significant bias or manipulation and indicate specifically what is 
to be observed, in which population or conditions, and in what 
timeframes. Of the 14 measures, 7 have targets in place to help 
determine whether overall goals and objectives were achieved. Of the 
seven measures without targets, three were being baselined (i.e., IRS 
was collecting data for use in setting first-time targets). The 
remaining four measures were being designed at the time of our review. 
Targets will be set for these measures upon completion of data 
collection.
Data Collection Process Affects Reliability of Several Field Assistance 
Measures:
Eight of field assistances 14 performance measures are based on a data 
collection process that is subject to inconsistencies and human error, 
meaning that the same results may not be produced in similar 
circumstances. All TAC employees are to use Form 5311 (Field Assistance 
Activity Report) to manually report their daily hours and type of 
assistance provided. Supervisors are to review the forms for accuracy 
and forward them for manual input into the Resources Management 
Information System.[Footnote 32] These layers of manual input are 
subject to error and can hinder data reliability that could (1) lead 
managers or other stakeholders to draw inappropriate conclusions about 
program performance, (2) not alert them to the existence of problems, 
or (3) not help them respond when problems arise. For example, as we 
noted in our report on IRSs 2001 tax filing season, our calculations 
showed that the data reported by TACs did not account for the wait 
times of about 661,000 taxpayers, or about 13 percent of taxpayers 
served.[Footnote 33] IRS expects to minimize this human error by 
equipping all of its TACs with an on-line automated tracking and 
reporting system known as the Queuing Management System (Q-Matic). This 
system is expected, among other things, to more efficiently monitor 
customer traffic flow and wait times and eliminate staff time spent 
completing Form 5311.[Footnote 34]
IRS has taken steps to solve data reliability problems with field 
assistances customer satisfaction measure. In a May 2000 report, the 
Treasury Inspector General for Tax Administration concluded that IRS 
had not established an adequate management process to ensure that the 
survey yielded accurate, reliable, and statistically valid 
results.[Footnote 35] To field assistances credit and with the help of 
a vendor, it (1) completed major revisions to the customer satisfaction 
survey, such as using a different index scale; (2) included space for 
written comments, which were to be provided to managers on a routine 
basis; and (3) improved controls to ensure the survey is available to 
all taxpayers. However, problems arose regarding the manner in which 
the vendor was providing site managers with data containing cumulative 
responses and, as of June 2002, the vendor had temporarily stopped 
providing feedback to site managers and was in the process of 
determining a more usable format to relay information to managers. The 
improved data collection method is being implemented and IRS 
anticipates an increase in the precision with which it measures field 
assistance customer satisfaction.
Field Assistance Measures Cover Core Program Activities with Limited 
Overlap:
Field assistances measures cover its core program activities with 
limited overlap. Field assistance identifies its core program 
activities as face-to-face assistance, education, and compliance 
services, which include such activities as preparing returns, answering 
tax law questions, resolving account and notice inquiries, and 
supplying forms and publications. For example, field assistance has an 
accounts contact measure (counts the number of contacts made) and an 
accounts accuracy measure (measures the accuracy of the responses) to 
reflect both the quantity and quality of its accounts-related 
assistance.
Field assistance identified some overlap between two measures, return 
preparation contacts and return preparation units. It has decided, 
for Strategy and Program Plan purposes, to discontinue the contacts 
measure (which counts the number of customers assisted) and keep the 
units measure (which counts the number of returns prepared) because 
the units measure better reflects the amount of return preparation 
work done.[Footnote 36] Field assistance will continue tracking the 
contacts measure outside of the Strategy and Program Plan in order to 
determine customer demand for service at particular sites. We concur 
with IRSs plans to track the contacts measure outside of the 
Strategy and Program Plan because it is a diagnostic tool that can be 
used for analysis purposes.
Field Assistance Is Missing Some Measures Needed to Balance 
Governmentwide Priorities:
Field assistance continues to develop its suite of performance 
measures. As part of that effort, it is beginning to deploy important 
quality measures, such as tax law accuracy. However, other important 
measures of timeliness, efficiency, and cost of service are missing, 
which impairs balance.
* Timeliness. Before fiscal year 2001, field assistance had a 
performance measure that officially tracked how long customers waited 
to receive service from an employee. According to managers, it was 
discontinued because employees were serving taxpayers as quickly as 
possible in order to meet timeliness goals, which negatively affected 
service quality.[Footnote 37] In March 2002, management went further 
and (1) eliminated its requirement for TACs not equipped with Q-Matic 
to submit biweekly wait-time reports and (2) doubled, from 15 to 30 
minutes, the wait-time interval to be used by TACs with Q-Matic in 
computing the percentage of customers served on time. Officials said 
that they took these steps because employees continued to feel 
pressured to hurry assistance despite the discontinuance of the 
official timeliness measure. However, one purpose of balanced measures 
is to avoid an inappropriate emphasis on just one aspect of 
performance. The presence of a quality measure should provide a 
disincentive for employees to ignore quality in favor of timeliness. 
Similarly, in the absence of a timeliness performance measure, (1) 
field assistance may not be balancing its customers needs for timely 
service with their needs for accurate information and (2) IRS is not 
held accountable for timeliness to stakeholders, such as the Congress.
* Efficiency. Efficiency, or productivity as it is often referred to, 
shows how efficiently IRSs resources are transformed into the 
production of field assistance services. Field assistance officials 
said they would like to develop an efficiency measure, but no plans are 
in place. Among other things, having an efficiency measure would help 
managers identify performance strengths and weaknesses.
* Cost of Service. As required by GPRA, agencies should have 
performance measures that correlate the level of program activity and 
program cost. Without such a measure in field assistance, officials do 
not know how much it costs to provide face-to-face service. Field 
assistance officials said that they would like to develop a cost of 
service measure, but they are not certain how to calculate it.
Submission Processing Measures:
As shown in table 5, all 11 of submission processings performance 
measures have many of the attributes of successful performance 
measures. However, as summarized in this section, we identified several 
opportunities for improvement, especially in the area of reliability. 
Table 9 in appendix II has more detailed information about each 
submission processing measure, including any weaknesses we identified 
and any recommendations for improvement.
Table 5: Overview of Our Assessment of Submission Processing Measures:
[See PDF for image]
Note: A check mark denotes that the measure has the attribute.
[A] We were unable to verify the linkages between goals and measures 
because of insufficient documentation.
[B] Core program activities of submission processing are to efficiently 
and accurately process returns, remittances, and refunds and issue 
notices and letters.
[C] Submission processing measures cover various governmentwide 
priorities, such as efficiency, timeliness, and accuracy; however, 
submission processings measures did not include a measure for customer 
satisfaction or for showing how much it costs to process the average 
return.
Source: GAO analysis.
[End of table]
Alignment between IRSs Goals and Submission Processing Measures Is 
Uncertain:
No formal documentation exists to show how submission processings
11 measures are aligned with IRSs mission, its agencywide goals, and 
its operating division goals. Despite this lack of formal 
documentation, submission processing officials said, and we generally 
concur, that some linkage does exist. Without complete documentation, 
however, we could not verify all the linkages. Submission processing 
officials stated that staff and managers are aware of the link between 
measures and goals because the submission processing organization has 
taken action to help ensure that staff understand the measures and 
their role in supporting IRSs overall mission and strategic and 
operating goals. For example, according to submission processing 
officials, they visited all eight W&I processing centers in 2001 to 
talk directly with staff and managers about the importance of balanced 
performance measures in ensuring that IRS meets its goals. Complete 
documentation of the linkages between goals and measures could further 
enhance understanding of those goals and measures with managers and 
staff.
Submission Processing Measures Have Clarity, with One Exception:
All but one of the submission processing measures have clarity and 
provide information to enable executives, other managers, and outside 
stakeholders to properly assess performance against goals. The one 
exception is the productivity measure.
Managers in different processing centers told us that they did not use 
the productivity measure to provide them with performance information 
or to help them assess performance because, among other things, the 
measure does not provide specific information about their units or 
centers performance or their contribution to overall productivity. 
This is because the measure, as designed, is a compilation of different 
types of work IRS performs in processing returns, remittances, and 
refunds and issuing notices and letters. As a result, unit managers 
used different productivity measures specific to their own processes to 
help them identify how to increase their areas productivity. However, 
according to IRS officials, the productivity measure is useful and 
provides adequate information to some IRS executives.
From our perspective, although the productivity measure may be 
meaningful to executives, the fact that field managers use other 
measures and profess not to understand the current productivity measure 
indicates that the current measure does not provide those managers with 
useful information that would alert them to problems and help them 
respond when problems arise. In addition, because the measure is 
calculated by compiling and weighting different types of processing 
work per staff year expended, it may be too confusing to be useful to 
outside stakeholders, such as Congress.
All Submission Processing Measures Have Targets and Most Are Objective:
All 11 of submission processings measures have measurable targets and 
most are objective (i.e., reasonably free of significant bias or 
manipulation). For example, the notice error rate had a target of
8.1 percent for fiscal year 2001. The deposit timeliness measure 
appears to be objective, for example, because the Integrated Submission 
and Remittance Processing System[Footnote 38] automatically calculates 
data on which the measure is based. However, the notice error rate 
and letter error rate measures are not objective because the coding 
required as part of data collection by individual reviewers is subject 
to much interpretation that could systematically bias the results of 
the measures. In October 
2002, the Treasury Inspector General for Tax Administration reported, 
based on a review at two processing centers, that the deposit error 
rate measure was not objective, because the associated sampling plan 
was not consistently implemented.[Footnote 39] The Treasury Inspector 
General for Tax Administration recommended that IRS take steps to 
ensure consistent implementation, and IRS reported that steps have been 
taken.
Five Submission Processing Measures Lack Reliability:
Five measures are subject to consistency problems that affect the 
reliability of the measures. Those measures are refund timeliness--
individual (paper), notice error rate, refund error rate, letter 
error rate, and deposit error rate. Specifically, the five measures 
are based on a data collection process, which according to the Director 
of Submission Processing, involves about 80 staff who identify, 
interpret, and analyze errors at the eight W&I processing centers. The 
notice error rate and letter error rate measures also involve 
coding that is subject to further interpretation.
Submission processing managers recognized that staff inconsistently 
coded notice and letter errors during the 2001 filing season. Neither 
IRS nor we know the extent to which such inconsistencies exist because 
no routine studies are done to validate the accuracy of data 
collection. Reliability and credibility increase when such studies are 
done. Submission processing initiated studies beginning in June 2001 to 
improve reliability, but has not established any improvement goals.
Submission Processing Measures Cover Core Program Activities without 
Overlap:
Each of submission processings measures directly pertains to one of 
the core program activities of submission processings business 
operations--timely, efficiently, and accurately processing returns, 
remittances, and refunds and issuing notices and letters--without 
redundancy or overlap. For example, the refund error rate--individual 
(paper) measure directly pertains to one of submission processings 
core program activities, processing refunds, and does not overlap with 
any of the other 11 measures.
Unlike the other three program areas we reviewed, submission processing 
has two customers--taxpayers, to whom IRS issues refunds and sends 
notices, and the Department of the Treasury, for which IRS deposits 
remittances. Therefore, for some measures, such as refund timeliness, 
IRS views taxpayers as the customer, while for other measures, such as 
deposit timeliness, IRS views Treasury as the customer. Submission 
processing officials believe that this dual-customer perspective 
provides a complete view of their operations and the measures cover all 
aspects of their operations while still being limited to a manageable 
number.
Submission Processing Measures Cover Various Governmentwide 
Priorities, but Are Not Fully Balanced:
Submission processings measures cover various governmentwide 
priorities, such as efficiency, timeliness, and accuracy. However, at 
the time of our review, submission processing measures lacked balance 
because they did not include a measure for customer satisfaction or a 
measure showing how much it costs to process a return.
Although submission processing officials believe that some existing 
measures, such as notice error rate and refund timeliness, provide 
information related to the customers experience, they recognize that 
directly obtaining customers perspectives would be more accurate than 
assuming their experience based on such measures. Thus, submission 
processing is obtaining customer satisfaction information as part of 
IRSs corporate customer satisfaction survey, which IRS expects will be 
available by the 2003 filing season.
Similar to the other three program areas, submission processing does 
not have a cost of service measure.[Footnote 40] Among other things, 
not having a cost of service measure affects IRSs ability to 
adequately compare different types of processing, such as paper versus 
electronic. In our view, because IRS does not take into account the 
cost to process a particular type of return, managers cannot fully 
understand the effectiveness of their unit.
Conclusions:
Because the filing season affects so many taxpayers, IRSs performance 
is important. Having successful performance measures that demonstrate 
results, are limited to the vital few, cover multiple program 
priorities, and provide useful information to decision makers will help 
IRS management and stakeholders, such as Congress, make decisions about 
how to fund and improve return processing and assistance to taxpayers.
Despite the challenge of developing a set of 53 measures that satisfy 
our criteria, IRS has made significant progress. As developed to date, 
the measures satisfy many of our nine attributes for successful 
performance measures. For example, in all four of the program areas we 
reviewed, most measures covered the core activities of each program and 
had targets in place. IRS also has several on-going improvement 
initiatives, such as the effort to redesign all aspects of its 
telephone assistance quality measures.
Although the measures satisfied many of the nine attributes, our 
evaluation also showed that they do not have all the characteristics of 
successful performance measures. The most significant weaknesses 
include (1) the inability of some measures to provide clear information 
to decision makers about program performance, (2) data collection 
methods that hamper objectivity and reliability, and (3) measures to 
cover governmentwide priorities that are missing from the Strategy and 
Program Plan. Although such weaknesses do not mean that the measures 
are not useful, IRS risks basing program and resource allocation 
decisions on inadequate or incomplete information and is less 
accountable until the weaknesses are addressed.
Correcting these weaknesses is important in order to (1) create a 
results-oriented environment that demonstrates and tracks how IRSs 
programs and activities contribute to achieving its mission and 
strategic goals, (2) avoid creating an excess of data that could 
obscure key information needed to identify problem areas and assess 
goal achievement, (3) form a balanced environment that takes the core 
program activities of the program into account, and (4) provide 
managers 
and other stakeholders with critical information on which to base their 
decisions.
Recommendations for Executive Action:
We recommend that the Commissioner of Internal Revenue direct the 
appropriate officials to do the following:
Take steps to ensure that agencywide goals clearly align with operating 
division goals and performance measures for each of the four areas 
reviewed. Specifically, (1) clearly document the relationship among 
agencywide goals, operating division goals, and performance measures 
(the other three program areas may want to consider developing a 
template similar to the one field assistance developed, shown in figure 
4) and (2) ensure that the relationship among goals and measures is 
communicated to staff at all levels of the organization.
Make the name and definition of several field assistance measures 
(i.e., geographic coverage, return preparation contacts,  return 
preparation units, TACs total contacts, forms contacts, tax law 
contacts, account contacts, other contacts, tax law accuracy, 
accounts/notices accuracy, and return preparation accuracy) more 
clear to indicate what is and is not included in the formula.
As discussed in the body of this report and in appendix II, modify the 
formulas used to compute various measures to improve clarity. If 
formulas cannot be implemented in time for the next issuance of the 
Strategy and Program Plan, then modify the name and definition of the 
following measures so it is clearer what is or is not included in the 
measure.
* Remove automated calls from the formula for the CSR level of 
service measure.
* Revise the CSR response level measure to include calls from 
taxpayers who tried to reach a CSR but did not, such as those who 
(1) hung-up while waiting to speak to a CSR, (2) were provided access 
only to automated services and hung up, and (3) received a busy signal.
* Analyze and use new or existing data to determine why calls are 
transferred and use the data to revise the CSR services provided 
measure so that it only reflects transferred calls in which the caller 
received help from more than one CSR (i.e., exclude calls in which a 
CSR simply transferred the call and did not provide service.):
* Either discontinue use of the number of IRS digital daily Web site 
hits measure or revise the way hits are calculated so that the 
measure more accurately reflects usage.
* Revise field assistances geographic coverage measure by ensuring 
that the formula better reflects (1) the various types of field 
assistance facilities, including alternate sites and kiosks; (2) the 
types of services provided by each facility; and (3) the facilitys 
operating hours.
* Revise submission processings productivity measure so it provides 
more meaningful information to users.
Refrain from making changes to official targets, such as electronic 
filing and assistance did in fiscal year 2001, unless extenuating 
circumstances arise. Disclose any extenuating circumstances in the 
Strategy and Program Plan and other key documents.
Modify procedures for the toll-free customer satisfaction survey, 
possibly by requiring that administrators listen to the entire call, to 
better ensure that administrators (1) notify CSRs that their call was 
selected for the survey as close to the end of a call as possible and 
(2) can accurately answer the questions they are responsible for on the 
survey.
Implement annual effectiveness studies to validate the accuracy of the 
data collection methods used for the five telephone measures (toll-
free tax law quality, toll-free accounts quality, toll-free tax law 
correct response rate, toll-free account correct response rate, and 
toll-free timeliness) subject to potential consistency problems. The 
studies could determine the extent to which variation exists in 
collecting data and recognize the associated impact on the affected 
measures. For those measures, and for the five submission processing 
measures that already have effectiveness studies in place (refund 
timeliness-individual (paper), notice error rate, refund error 
rate--individual (paper), letter error rate, and deposit error 
rate), IRS should establish goals for improving consistency, as 
needed.
Ensure that plans to remove overlapping measures in telephone and field 
assistance are implemented.
As discussed in the body of this report, include the following missing 
measures in the Strategy and Program Plan in order to better cover 
governmentwide priorities and achieve balance.
* In the spirit of provisions in the Chief Financial Officers Act of 
1990 and Financial Accounting Standards Number 4, develop a cost
of services measure using the best information currently available 
for each of the four areas discussed in this report, recognizing data 
limitations as prescribed by GPRA. In doing so, adhere to guidance, 
such as Office of Management and Budget Circular A-76, and consider 
seeking outside counsel to determine best or industry practices.
* Given the importance of automated telephone assistance, develop a 
customer satisfaction survey and measure for automated assistance.
* Put the automated completion rate measure back in the Strategy and 
Program Plan after revising the formula so that calls for recorded tax 
law information are not counted as completed when taxpayers hang up 
before receiving service.
* Add one or more quality measures to electronic filing and 
assistances suite of measures in the Strategy and Program Plan. 
Possible measures include processing accuracy, refund timeliness, 
electronically filed, and number of electronic returns rejected.:
* Re-implement field assistances timeliness measure.
* Develop a measure that provides information about field assistances 
efficiency.
Agency Comments and Our Evaluation:
The Commissioner of Internal Revenue provided written comments on a 
draft of this report in a letter dated November 1, 2002, which is 
reprinted in appendix III. The Commissioner was pleased to see that 
many of the measures had the attributes for successful performance and 
agreed that others presented opportunities for further refinement. He 
stated that the report was objective and balanced and that our 
observation of the on-going nature of the performance measurement 
process was on point. Furthermore, he noted that the attributes we 
developed can be used as a checklist when performance measures are 
developed in the future.
Of our 18 recommendations, IRS:
* agreed with 10 and cited planned corrective actions that were 
responsive to those recommendations;
* cited actions taken or planned in response to 2 that did not fully 
address our concerns; and:
* disagreed with 6.
The following discussion focuses on the recommendations with which IRS 
disagreed or for which we believe additional action is necessary to 
address our concerns.
In response to our recommendation about clarifying the name and 
definition of several field assistance measures, IRS said that the 
recently updated data dictionary addressed our concerns. We reviewed 
the updated data dictionary. The modifications are substantial and 
provide significant additional information about the measures. However, 
the definitions remain unclear. Specifically, the definitions should 
either define a taxpayer assistance center or state whether or not 
alternate sites, such as kiosks and mobile sites, are included.
IRS did not agree that automated calls should be removed from the 
formula for the CSR level of service measure. IRS said that including 
the count of callers who choose an automated service while waiting for 
CSR service is appropriate. IRSs response does not accurately 
characterize all the calls answered by automation that are included in 
the CSR level of service measure. Rather than choosing an automated 
service while waiting for a CSR, some callers complete an automated 
service after hearing an announcement that, due to high call volume, 
only automated services are available--a choice is not involved. We 
believe that the CSR level of service measure, because of its name 
and the way it is calculated, could be misleading and might 
misrepresent taxpayers access to CSRs. For example, increasing the 
percentage of calls served through automation because a CSR was not 
available--meaning that CSRs were actually more difficult to reach--
would improve the CSR level of service measure, thus giving the 
impression that access to CSRs had improved when it had actually 
gotten worse. Calls answered through automation, regardless of the type 
of assistance (CSR or automation) the caller was originally seeking, 
should be reflected in an automated-level-of-service measure, such as 
automated service completion rate.:
IRS did not agree that it should modify the CSR response level 
measure to include calls in which the caller hung up before receiving 
service or got a busy signal. IRS said that altering the measure would 
deviate from industry standards and hinder IRSs ability to gauge 
success in meeting this world class service goal. We support IRSs 
efforts to gauge its progress toward providing world class customer 
service by telephone. However, IRSs use of the same telephone wait-
time measure used by others may actually hinder a meaningful comparison 
of IRS with industry leaders. The CSR response level measure shows, 
for the callers who reached a CSR, the percentage that waited 30 
seconds or less. According to IRS officials, when taxpayers call IRS 
attempting to reach a CSR, they are much less likely to reach one than 
when they call a recognized telephone service leader (i.e., callers to 
IRS are more likely to hang up while waiting to speak to a CSR, hang up 
after being given access to only automated service because a CSR is not 
available, or receive a busy signal). Therefore, when the CSR response 
level measure (which excludes these hang-ups and busy signals) is used 
by IRS, the measure may represent the experience of a significantly 
smaller percentage of the total callers that attempted to reach an a 
CSR than when the same measure is used by industry leaders, thus 
potentially overstating the ease with which callers reached IRS CSRs. 
Data we obtained from IRS suggest that there were about an equal number 
of hang-ups and busy signals as calls answered in this measure in 2001.
In response to our recommendation about implementing annual studies to 
validate the accuracy of various data collection methods and 
establishing goals for improving consistency, IRS said that it (1) has 
an ongoing process to ensure proper administration of the collection 
methods for the telephone measures cited in our recommendation, (2) 
does not agree that an annual independent review by non-CQRS analysts 
is merited, and (3) does not agree that it should incorporate 
consistency 
improvement goals in the Strategy and Program Plan process. As we noted 
in our report, telephone assistances CQRS has some controls in place 
to 
monitor consistency. However, we believe that reliability and 
credibility increase when performance data are checked or tested for 
significant errors, which IRS currently does not do. We did not 
recommend that non-CQRS analysts do these reviews; who does the reviews 
is for IRS to decide. Also, we recognized in our report that submission 
processing has an on-going process to verify consistency and that it 
has found problems. Because that review process has found some 
problems, we believe that establishing goals for improving consistency 
in submission processing is warranted. Because telephone assistance 
does not have a review process in place, we do not know whether 
improvement goals are needed, but noted that they could be. We did not 
recommend that these goals become a part of the Strategy and Program 
Plan process. Instead, these goals should become part of the review 
process and be made known to staff who are performing the work.
IRS disagreed with our recommendation that it put the automated 
completion rate measure back in the Strategy and Program Plan. 
Instead, IRS said it would continue to track and monitor that rate as a 
diagnostic measure. IRS told us that its decision is based on the fact 
that data on automated calls are not good enough to merit the attention 
the measure would have at the Strategy and Program Plan level. We 
recognize that there are data weaknesses with this measure. That is why 
our recommendation calls for IRS to revise the formula before returning 
the measure to the Strategy and Program Plan. Because serving more 
callers through automation is important to IRSs strategy for improving 
taxpayer service, we believe that IRS needs a measure of the level of 
service provided by automation in its Strategy and Program Plan to 
balance its measure of the level of service provided by CSRs. Other 
than counts of the number of calls served, IRS has no measure of its 
effectiveness in serving taxpayers through automation. Without such a 
measure, IRS risks poorly serving the increasing number of taxpayers 
being served through automation while possibly improving access for a 
declining number of callers who need to speak with a CSR.
IRS does not believe that adding one or more quality measures to 
electronic filing and assistances suite of measures in the Strategy 
and Program Plan would enhance the electronic filing program. It noted 
that it tracks the quality of electronic filing outside the Strategy 
and Program Plan and that quality has been consistently high. We 
recognize that electronic filing and assistance tracks quality outside 
the Strategy and Program Plan. However, we disagree with IRSs position 
that adding quality measures to that plan would not enhance the 
program. According to IRS officials, measures in the Strategy and 
Program Plan are the highest, most comprehensive level of measures for 
which they are accountable. In addition, many of those measures are 
made available to outside stakeholders. By not elevating these measures 
of quality to the Strategy and Program Plan, electronic filing and 
assistance risks not being held to any quality standards. Furthermore, 
not having quality measures hampers balance among electronic filing and 
assistances suite of measures and is not consistent with IRSs 
balanced measurement program or the intent of IRSs Restructuring and 
Reform Act of 1998.
IRS disagreed with our recommendation that it re-implement field 
assistances timeliness measure. IRS said that although timeliness 
goals are important in providing service to taxpayers, they are 
detrimental to quality service because field assistance employees tend 
to rush customers when traffic is high. This position is inconsistent 
with IRSs balanced measurement program and the intent of IRSs 
Restructuring and Reform Act of 1998. Although the accuracy of 
assistance is an important measure of quality, the timeliness of that 
assistance is also an important and balancing aspect of quality. 
Without this balancing emphasis, staff could theoretically take 
excessive time providing quality tax law assistance to a few taxpayers 
regardless of the impact on the wait-time for other taxpayers. We agree 
that Q-Matic is the best source of this information and support IRSs 
plans to implement it nationwide. IRS also stated that it could use 
feedback from its customer satisfaction surveys to obtain information 
about the promptness of service. As we noted in our report, problems 
arose in the manner with which the feedback was provided from the 
vendor and the vendor had stopped providing feedback to site managers 
until the problems could be resolved. Even when those problems are 
resolved, a timeliness measure based on actual IRS data versus 
taxpayers perceptions would be meaningful.
Regarding our recommendation about implementing an efficiency measure 
in field assistance, IRS said that it will be testing a system for use 
as a diagnostic tool to monitor and evaluate the strengths and 
weaknesses of various productivity measures. However, IRSs response 
was silent as to whether or when it would establish a field assistance 
productivity measure. Maintaining and enhancing organizational 
productivity is a fundamental agency management responsibility. The 
extent to which IRSs field assistance organization is meeting this 
basic responsibility needs to be visible to IRS, Treasury, and 
congressional stakeholders in the form of an organizational performance 
measure, rather than a diagnostic tool, which is generally visible 
only to IRS managers.
We are sending copies of this report to the Chairmen and Ranking 
Minority Members of the Senate Committee on Finance and the House 
Committee on Ways and Means and the Ranking Minority Member of this 
Subcommittee. We are also sending copies to the Secretary of the 
Treasury; the Commissioner of Internal Revenue; the Director, Office of 
Management and Budget; and other interested parties. We will make 
copies available to others on request. In addition, the report will be 
available at no charge on the GAO Web site at http://www.gao.gov.
This report was prepared under the direction of David J. Attianese, 
Assistant Director. Other major contributors are acknowledged in 
appendix IV. If you have any questions about this report, contact
Mr. Attianese or me on (202) 512-9110.
Sincerely yours,
James R. White
Director, Tax Issues
[Signed by James R. White
[End of section]
Appendix I: Expanded Explanation of Our Attributes and Methodology for 
Assessing IRSs Performance Measures:
Performance goals and measures that successfully address important and 
varied aspects of program performance are key to a results-oriented, 
balanced work environment. Measuring performance allows organizations 
to track the progress they are making toward their goals and gives 
managers critical information on which to base decisions for improving 
their programs. Organizations need to have performance measures that 
(1) demonstrate results, (2) are limited to the vital few, (3) cover 
multiple program priorities, and (4) provide useful information for 
decision making in order to track how their programs and activities can 
contribute to attaining the organizations goals and mission. These 
four characteristics are important to accurately reveal the strengths 
and weaknesses of a program since measures are often the key motivators 
of performance and goal achievement.
For use as criteria to determine whether the Internal Revenue Services 
(IRS) performance measures in four key program areas--telephone 
assistance, electronic filing and assistance, field assistance, and 
submission processing--demonstrate results, are limited to the vital 
few, cover multiple program priorities, and are useful in decision 
making, we developed nine attributes of performance goals and measures 
based on previously established GAO criteria. In addition, we 
considered key legislation, such as the Government Performance and 
Results Act of 1993 (GPRA) and the IRS Restructuring and Reform Act of 
1998, and performance management literature cited in the bibliography 
and 
related products sections at the end of this report. Our nine 
attributes 
may not cover all the attributes of successful performance measures; 
however, we believe these are some of the most important. We shared 
these attributes with IRS officials responsible for performance 
measurement issues, such as the Acting Director of the Organizational 
Performance Division; and several officials in the Wage and Investment 
(W&I) operating division, such as the Director of Strategy and Finance; 
the Chief of Planning and Analysis; the Director of Customer Account 
Services; and the Director of Field Assistance. These officials 
generally agreed with the relevance of the attributes and our review 
approach.
We applied these attributes to the 53 filing season measures in W&Is 
fiscal year 2001-2003 Strategy and Program Plan in a systematic manner, 
but some judgment was required. To ensure consistency and reliability 
in our application of the attributes, we had one staff person 
responsible for each of the four areas. That staff person prepared the 
initial analysis and at least two other staff reviewed those detailed 
results. Several staff reviewed the results for all four areas. 
Inherently, the attributes described below are not weighted equally. 
Weaknesses identified in a particular attribute do not, 
in and of themselves, mean that a measure is ineffective or 
meaningless. Instead, weaknesses identified should be considered areas 
for further refinement.
Detailed information on each attribute, including an explanation, 
examples, and the methodology we used to assess that attribute with 
respect to the measures covered by our review, follows.
Attributes of Successful Performance Measures:
1. Is there a relationship between the performance goals and measures 
and an agencys goals and mission? (Referred to as linkage):
Explanation: Performance goals and measures should align with an 
agencys goals and mission. A cascading or hierarchal linkage moving 
from top management down to the operational level is important in 
setting goals agencywide, and the linkage from the operational level to 
the agency level provides managers and staff throughout an agency with 
a road map that (1) shows how their day-to-day activities contribute to 
attaining agencywide goals and mission and (2) helps define strategies 
for achieving strategic and annual performance goals. As agencies 
develop annual performance goals as envisioned by GPRA, they can serve 
as a bridge that links long-term goals to agencies daily operations. 
For example, an annual goal that is linked to a program and also to a 
long-term goal can be used both to (1) hold agencies and program 
offices accountable for achieving those goals and (2) assess the 
reasonableness and appropriateness of those goals for the agency as a 
whole. In addition, annual performance planning can be used to better 
define strategies for achieving strategic and annual performance goals.
Linkages between goals and measures are most effective when they are 
clearly communicated to all staff within an agency so that everyone 
understands what the organization is trying to achieve and the goals it 
seeks to reach. Communicating goals and their associated measures is a 
continuous process and supports the basis for everything the agency 
does each day. Communication creates a line of sight throughout an 
agency so that everyone understands what the organization is trying to 
achieve and the goals it seeks to reach.
Example: Submission processings notice error rate measure determines 
the percentage of incorrect notices issued to taxpayers by submission 
processing employees. The target set for this measure in 2001 was 
8.1 percent. This measure could be used to support the notice 
redesign improvement project as well as the operational priority to 
prioritize notices and monitor and control notice issuance. It also 
is used to support one of W&Is goals--to meet taxpayer demands for 
timely, accurate, and efficient services. This W&I strategy aligns 
with IRSs strategic goal, top quality service to all taxpayers 
through fair and uniform application of the law, which in turn, 
supports IRSs mission to provide Americas taxpayers top quality 
service by helping them understand and meet their tax responsibilities 
and by applying the tax law with integrity and fairness to all.:
Methodology: We compared IRSs measures with its targets, improvement 
projects, operational priorities, operating division goals, and 
agencywide goals and mission as documented in the Strategy and Program 
Plan. We also interviewed operational/unit managers and managers 
responsible for the Strategy and Program Plan about linkages and 
reviewed training materials.
2. Are the performance measures clearly stated? (Referred to as 
clarity):
Explanation: A measure has clarity when it is clearly stated and the 
name and definition are consistent with the methodology used for 
calculating the measure. A measure that is not clearly stated (i.e., 
contains extraneous or omits key data elements) or that has a name or 
definition that is inconsistent with how it is calculated can confuse 
users and could cause managers or other stakeholders to think that 
performance was better or worse than it actually was.
Example: Telephone assistances average handle time measure shows the 
average number of seconds Customer Service Representatives (CSRs) spent 
assisting callers. Its definition and formula are consistent with the 
name of the measure and clearly note that the measure includes talk and 
hold times and the time a CSR spends on work related to a call after 
the call is terminated.
Methodology: We compared the name of the measure, the written 
definition of the measure, and the formula or methodology for computing 
the measure. In several instances, we discussed certain components of 
the definition and formula with IRS officials to better understand its 
meaning and purpose. For example, we discussed components of telephone 
assistances quality measures with staff in Customer Account Services, 
and staff in the Centralized Quality Review Site. We also reviewed on-
line information available to field assistance managers from the 
Queuing Management System (Q-Matic).[Footnote 41] We spoke to managers 
at different levels within each of the four areas we reviewed and asked 
them about the information they received and how they used it. In 
addition, we used some of the results of a random telephone survey of 
managers we conducted in 2001 at 84 of IRSs 413 Taxpayer Assistance 
Centers (TAC) to solicit their views on the services provided at those 
offices.
3. Do the performance measures have targets, thus allowing for easier 
comparison with actual performance? (Referred to as measurable 
target):
Explanation: Where appropriate, performance goals and measures should 
have quantifiable, numerical targets or other measurable values. 
Numerical targets or other measurable values facilitate future 
assessments of whether overall goals and objectives were achieved 
because comparisons can be easily made between projected performance 
and actual results. Some goals are self-measuring (i.e., they are 
expressed objectively and are quantifiable) and therefore do not 
require additional measures to assess progress. When goals are not 
self-measuring, performance measures should translate those goals into 
observable conditions that determine what data to collect to learn 
whether progress was made toward achieving goals. The measures should 
have a clearly apparent or commonly accepted relation to the intended 
performance or have been shown to be reasonable predictors of desired 
behaviors or events. If a goal cannot be expressed in an objective, 
specific, and measurable form, GPRA allows the Office of Management and 
Budget to authorize agencies to develop alternative forms of 
measurement.[Footnote 42]
Example: Electronic filing and assistances percent of individual 
returns electronically filed had a numerical target of 31 percent in 
fiscal year 2001.
Methodology: We determined that a goal or measure had a measurable 
target when expected performance could be compared with actual results, 
and in general, was not changed during the measurement period. Each of 
the measures we reviewed was listed in the Strategy and Program Plan, 
which provides projections or targets for the current and two 
subsequent fiscal years. We verified that the target was measurable. 
When the Strategy and Program Plan did not show a target, we contacted 
appropriate IRS officials to determine why.
4. Are the performance goals and measures objective? (Referred to as 
objectivity):
Explanation: To the greatest extent possible, goals and measures should 
be reasonably free of significant bias or manipulation that would 
distort the accurate assessment of performance. They should not allow 
subjective considerations or judgments to dominate the outcome of the 
measurement. To be objective, performance goals and measures should 
indicate specifically what is to be observed, in which population or 
conditions, and in what timeframe and be free of opinion and judgment. 
Objectivity is important because it adds credibility to the performance 
goals and measures by ensuring that significant bias or manipulation 
will not distort the measure.
Example: The customer satisfaction measure for telephone assistance 
has the potential for bias and therefore may not be objective. Survey 
administrators are instructed to notify the CSR towards the end of the 
call that his or her call was selected to participate in the survey. A 
potential problem arises because administrators are not required to 
listen to the entire call, and it can be difficult to determine when 
the call is about to end. Therefore, if a CSR is notified prior to the 
end of the call that the call was selected for survey, the CSR could 
change behavior towards the taxpayer, thus affecting the results of the 
survey and the measure.
Methodology: We reviewed information in IRS guidance or procedures, 
data collection instruments, reports, and other documents. We held 
discussions about objectivity with various staff and officials, such as 
data owners and analysts, within each of the four areas we reviewed. 
Because our interviews raised questions about the objectivity of some 
measures for telephone assistance, we monitored some taxpayer calls and 
interviewed an official from IRSs customer satisfaction survey 
contractor, Pacific Consulting Group.
5. To what extent do the performance goals and measures provide a 
reliable way to assess progress? (Referred to as reliability):
Explanation: Reliability refers to whether measures are amenable to 
applying standard procedures for collecting data or calculating results 
so that they would be likely to produce the same results if applied 
repeatedly to the same situation. Errors can occur at various points in 
the collection, maintenance, processing, and reporting of data. 
Significant errors would affect conclusions about the extent to which 
performance goals have been achieved. Likewise, errors could cause the 
measure to report performance at either a higher or lower level than is 
actually being attained. Reliability is increased when verification and 
validation procedures, such as checking performance data for 
significant errors by formal evaluation or audit, exist.
Example: Field assistances return preparation contacts measure 
tracks the total number of customers assisted with return preparation 
by IRS. This measure may not be reliable because it involves a 
significant amount of manual entry on Form 5311 (Field Assistance 
Activity Report) even at sites with the Q-Matic system. In addition to 
the potential for error associated with manual entry, the instructions 
for filing Form 5311 require that service time be recorded in whole 
hours, which can misconstrue actual service times and is less exact 
than the data in Q-Matic, which records service times in minutes.
Methodology: We looked for weaknesses in IRSs guidance or procedures, 
data collection instruments, reports, and other documents that might 
cause errors. We discussed potential weaknesses with various officials, 
such as account data analysts, within each of the four areas we 
reviewed. Because these efforts revealed the potential for errors in 
measuring telephone performance, we monitored employees preparing data 
collection instruments for assessing telephone quality and customer 
satisfaction in Atlanta. Likewise, we monitored field assistance staff 
helping taxpayers and reporting their time using both the automated Q-
Matic system and Form 5311.
6. Do the performance measures sufficiently cover a programs core 
activities? (Referred to as core program activities):
Explanation: Core program activities are the activities that an entity 
is expected to perform to support the intent of the program. 
Performance measures should be scoped to evaluate the core program 
activities. Limiting the number of performance measures to the core 
program activities will help identify performance that contributes to 
goal achievement. At the same time, however, there should be enough 
performance measures to ensure that managers have the information they 
need about performance in all the core program activities. Without such 
information, the possibility of achieving program goals is less likely.
Example: The core program activities for submission processing include 
(1) processing returns, (2) depositing remittances, (3) issuing 
refunds, and (4) sending out notices and letters. Each of submission 
processings 11 measures correspond to one of those core activities.
For example, the number of individual 1040 series returns filed 
(paper) 
measure corresponds to processing returns and the letter error rate 
measure corresponds with sending out notices and letters.
Methodology: We determined the core program activities of each of the 
four areas we reviewed based on IRS documentation and discussions with 
IRS officials. We reviewed the suite of performance measures for each 
of the four areas to determine whether measures existed that covered 
each core program activity. We determined whether any measures were 
missing or other pieces of information were needed to better manage 
programs by using judgment and questioning IRS officials. In addition, 
we reviewed the results of a questionnaire that we had used during a 
review of IRSs 
2001 filing season to ask TAC managers about information needed to 
manage their program.
7. Does there appear to be limited overlap among the performance 
measures? (Referred to as limited overlap):
Explanation: Measures overlap when the results of measures provide 
basically the same information. A measure that overlaps with another is 
unnecessary and does not benefit program management. Unnecessary or 
overlapping measures not only can cost money but also can cloud the 
bottom line in a results-oriented environment by making managers or 
other stakeholders sift through unnecessary or redundant information. 
Some measures, however, may overlap partially and provide stakeholders 
some new information. In those cases, management must make a judgment 
as to whether having the additional information is worth the cost and 
possible confusion it may cause.
Example: Telephone assistances toll-free average speed of answer and 
toll-free CSR response level measures attempt to show how long a 
taxpayer waited before receiving assistance. The difference between the 
two measures is that the latter shows the percentage of taxpayers 
receiving assistance within 30 seconds while the former shows the 
average time taxpayers waited for service. These two measures are 
likely to be correlated and thus partially overlap. However, the amount 
of overlap between measures is managements discretion.
Methodology: Within each of the four areas we reviewed, we looked at 
the suite of measures and compared the measures names and definitions. 
We also looked at the correlations between measures results. When two 
measures seemed similar, we discussed the potential for overlap with 
IRS officials.
8. Does there appear to be a balance among the performance goals and 
measures, or is there an emphasis on one or two priorities at the 
expense of others? (Referred to as balance):
Explanation: Balance exists when a suite of measures ensures that an 
organizations various priorities are covered. IRS considers its 
measures to be balanced when they address customer satisfaction, 
employee satisfaction, and business results (quality and quantity). 
Performance measurement efforts that overemphasize one or two 
priorities at the expense of others may skew the agencys performance 
and keep its managers from understanding the effectiveness of their 
programs in supporting IRSs overall mission and goals.
Example: Submission processing has an employee satisfaction measure and 
several business results measures, such as deposit timeliness. As of 
October 2002, however, it had not fully implemented a customer 
satisfaction measure, which resulted in an unbalanced process that can 
overlook something as important as the customers perspective.
Methodology: For each of the four areas, we ensured that a measure 
existed for each component. If measures did not exist for certain 
components, we contacted IRS officials to find out why and to see what 
plans IRS has to ensure balance in the future.
9. Does the program or activity have performance goals and measures 
that cover governmentwide priorities? (Referred to as governmentwide 
priorities):
Explanation: Agencies should develop a range of related performance 
measures to address governmentwide priorities, such as quality, 
timeliness, efficiency, cost of service, and outcome. A range is 
important because most program activities require managers to balance 
these priorities among other demands. When complex program goals are 
broken down into a set of component quantifiable measures, it is 
important to ensure that the overall measurement of performance does 
not become biased because measures that assess some priorities but 
neglect others could place the programs success at risk.
Example: Electronic filing and assistance provides the capability for 
taxpayers to transact and communicate electronically with IRS. The 
13 measures we reviewed included, for example, the number or percent of 
returns filed, the number of hits to or downloads from IRSs Web site, 
and employee and customer satisfaction. The Strategy and Program Plan 
did not have any measures on the programs quality or timeliness. Not 
having these measures means that management may not be sufficiently 
balancing competing demands.
Methodology: We analyzed the suite of measures in the Strategy and 
Program Plan for each of the four areas we reviewed. Based on 
discussions with IRS officials and our own judgment, we identified 
measures that appeared to be missing. We discussed those identified 
with IRS officials.
[End of section]
Appendix II: The 53 IRS Performance Measures Reviewed:
The following four tables provide information on the 53 performance 
measures we reviewed in the four program areas within the Internal 
Revenue Services (IRS) Wage and Investment (W&I) operating division 
that are critical to a successful filing season. Among other things, 
the tables show how each of the 53 measures matched up against the 
attributes in appendix I. The attributes not addressed in the tables 
are (1) linkage, because sufficient documentation did not exist to 
validate linkages with any of the measures and (2) balance, because 
that attribute does not apply to specific measures but, rather, to a 
programs entire suite of measures. When reviewing the suite of 
measures, we found some instances where additional measures are 
warranted; the additional measures are generally not cited in these 
tables.
Telephone Assistance Performance Measures:
Of the 53 performance measures in our review, 15 are for telephone 
assistance.[Footnote 43] Table 6 has information about each of the 15 
telephone measures.
Table 6: Telephone Assistance Performance Measures:
Measure name and definition[A]: Total automated calls answered; A 
count of all toll-free calls answered at telephone assistance centers 
by an automated system (e.g., Telephone Routing Interactive System) and 
Tele-Tax.[B]; FY 2001 target and actual: Target: 85,000,000 calls 
answered; Actual: 104,228,052 calls answered; Weaknesses of measure 
and consequences: Some overlap with automated completion rate measure. 
Both attempt to show how many automated calls were answered, but the 
automated completion rate tries to show the percentage that completed 
automated service successfully. Overlap could cloud the bottom line and 
obscure performance results; Recommendations: See note 1 to the 
table.
Measure name and definition[A]: Customer Service Representative (CSR) 
calls answered; The count of all toll-free calls answered at 
telephone assistance centers; FY 2001 target and actual: Target: 
31,500,000 calls answered; Actual: 32,532,503 calls answered; 
Weaknesses of measure and consequences: Some overlap with CSR services 
provided measure. Both attempt to show how many calls CSRs answered, 
but CSR services provided tries to count calls requiring the help of 
more than one CSR as more than one call. Overlap could cloud the bottom 
line and obscure performance results; Recommendations: See note 1 to 
the table.
Measure name and definition[A]: CSR level of service; The relative 
success rate of taxpayers who call for toll-free services reaching a 
CSR; FY 2001 target and actual: Target: 55%; Actual: 53.7%; Weaknesses 
of measure and consequences: Formula lacks clarity because it includes 
some automated calls, which overstates the number of calls answered by 
CSRs and thus the level of service being provided by CSRs.[C]; 
Definition 
lacks clarity because it does not disclose inclusion of some automated 
calls, which could lead to misinterpreted results or a failure to take 
proper action to resolve performance problems; Recommendations: Remove 
automated calls 
from the formula.
Measure name and definition[A]: Toll-free customer satisfaction; 
Customers perception of service received, with a rating of 4 being 
the best; FY 2001 target and actual: Target: 3.45 average score; 
Actual: 
3.45 average score; Weaknesses of measure and consequences: Not clear 
because survey only applies to calls handled by CSRs. Satisfaction is 
not measured for calls handled by automation, which accounted for 76 
percent of all calls in fiscal year 2001; Potential bias exists 
(not objective) because administrators are not required to listen to 
the entire call, (1) CSRs could be prematurely notified that their call 
was selected for the survey, thus changing their behavior towards the 
caller 
and affecting the results of the survey and (2) administrators may not 
be 
able to correctly answer certain questions on the survey, which could 
impair the accuracy of the data; Recommendations: Develop a customer 
satisfaction survey for automated assistance; Modify procedures for 
the toll-free customer satisfaction survey, possibly by requiring that 
administrators listen to the entire call, to better ensure that 
administrators (1) notify CSRs that their call was selected for the 
survey as close to the end of a call as possible and (2) can accurately 
answer the questions they are responsible for on the survey.
Measure name and definition[A]: Toll-free tax law quality[D]; 
Evaluates the correctness of answers given by CSRs to callers with tax 
law inquiries as well as CSRs conformance with IRS administrative 
procedures, such as whether the CSR gave his or her identification 
number to the taxpayer; FY 2001 target and actual: Target: 74%; 
Actual: 75.21%; Weaknesses of measure and consequences: A reliability 
weakness exists because evaluations are based on judgments that are 
potentially inconsistent. No routine studies to determine effectiveness 
of procedures to ensure consistency of data collection. Possible 
inconsistencies affect the accuracy of the measure and conclusions 
about the extent to which performance goals have been achieved; Some 
overlap with toll-free tax law correct response rate. Both attempt to 
show the percentage of callers receiving accurate responses to tax law 
questions, but toll-free tax law quality includes CSR conformance with 
administrative procedures in computing that percentage. Overlap could 
cloud the bottom line and obscure performance results; 
Recommendations: Implement annual effectiveness studies to validate the 
accuracy of data collection methods and establish goals for improving 
consistency, as needed; See note 1 to the table.
Measure name and definition[A]: Toll-free accounts quality[E]; 
Evaluates the correctness of answers given by CSRs to callers with 
account-related inquiries as well as CSRs conformance with IRS 
administrative procedures, such as whether a CSR gave his or her 
identification number to the taxpayer; FY 2001 target and actual: 
Target: 67%; Actual: 69.17%; Weaknesses of measure and 
consequences: A reliability weakness exists because evaluations are 
based on judgments that are potentially inconsistent. No routine 
studies to determine effectiveness of procedures to ensure consistency 
of data collection. Possible inconsistencies affect the accuracy of the 
measure and conclusions about the extent to which performance goals 
have been achieved; Some overlap with toll-free account correct 
response rate. Both attempt to show the percentage of callers receiving 
accurate responses to account questions, but toll-free accounts quality 
includes CSR conformance with administrative procedures in computing 
that percentage. Overlap could cloud the bottom line and obscure 
performance results; Recommendations: Implement annual effectiveness 
studies to validate the accuracy of data collection methods and 
establish goals for improving consistency, as needed; See note 1 to 
the table.
Measure name and definition[A]: Average handle time; The average 
number of seconds CSRs spent assisting callers. It includes talk and 
hold times and the time a CSR spends on work related to a call after 
the call is terminated; FY 2001 target and actual: Target: not 
available; Actual: 609 seconds; Weaknesses of measure and 
consequences: Target to be set upon completion of baseline data 
collection.[F]; Recommendations: None.
Measure name and definition[A]: Automated completion rate; The 
percentage of total callers who completed a selected automated 
service; FY 2001 target and actual: Target: not available; Actual: 
not available; Weaknesses of measure and consequences: Formula lacks 
clarity because it assumes that all callers seeking recorded tax law 
information, including those who hang up before receiving service, 
received the information they needed, which could produce inaccurate or 
misleading results; Not clear because definition does not disclose 
the previously mentioned assumption, which could lead to misinterpreted 
results or a failure to take proper action to resolve performance 
problems; Measure removed from the Strategy and Program Plan; target 
not available; Some overlap with total automated calls answered. 
Both attempt to show how many automated calls were answered, but 
automated completion rate tries to show the percentage that completed 
an automated service successfully. Overlap could cloud the bottom line 
and obscure performance results; Recommendations: Revise the measure 
so that calls for recorded tax law information are not counted as 
completed when callers hang up before receiving service; Put this 
measure back in the Strategy and Program Plan after revising the 
formula so that calls for recorded tax law information are not counted 
as completed when taxpayers hang up before receiving service; See 
note 1 to the table.
Measure name and definition[A]: CSR services provided; The count of 
all calls handled by CSRs; FY 2001 target and actual: Target: not 
available; Actual: 35,799,122 calls answered; Weaknesses of measure 
and consequences: Not clear because definition does not disclose that 
IRS counts all calls transferred from one CSR to another as receiving 
an additional service, which could lead to misinterpreted results or a 
failure to take proper action to resolve performance problems. IRS does 
not have complete information on why calls were transferred. Thus, IRS 
cannot identify appropriate steps to reduce any inefficiency associated 
with transferred calls; Target to be set upon completion of baseline 
data collection[F]; Some overlap with CSR calls answered. Both 
attempt to show how many calls CSRs answered, but CSR services provided 
tries to count calls requiring the help of more than one CSR as more 
than one call. Overlap could cloud the bottom line and obscure 
performance results; Recommendations: Analyze and use new or existing 
data to determine why calls are transferred and use the data to revise 
the measure so that it only reflects transferred calls in which the 
caller received help from more than one CSR (i.e., exclude calls in 
which a CSR simply transferred the call and did not provide service); 
See note 1 to the table.
Measure name and definition[A]: Toll-free tax law correct response 
rate[G]; Evaluates the correctness of answers given by CSRs to callers 
with tax law inquiries; FY 2001 target and actual: Target: 81.6%; 
Actual: 
79.53%; Weaknesses of measure and consequences: A reliability weakness 
exists because evaluations are based on judgments that are potentially 
inconsistent. No routine studies to determine effectiveness of 
procedures 
to ensure consistency of data collection. Possible inconsistencies 
affect 
the accuracy of the measure and conclusions about the extent to which 
performance goals have been achieved; Some overlap with toll-free tax 
law quality. Both attempt to show the percentage of callers receiving 
accurate responses to tax law questions, but toll-free tax law quality 
includes CSR conformance to administrative procedures in computing that 
percentage. Overlap could cloud the bottom line and obscure performance 
results; Recommendations: Implement annual effectiveness studies to 
validate the accuracy of data collection methods and establish goals 
for improving consistency, as needed; See note 1 to the table.
Measure name and definition[A]: Toll-free account correct response 
rate[H]; Evaluates the correctness of answers given by CSRs to callers 
with account-related inquiries; FY 2001 target and actual: Target: 
90.8%; 
Actual: 88.72%; Weaknesses of measure and consequences: A reliability 
weakness exists because evaluations are based on judgments that are 
potentially inconsistent. No routine studies to determine effectiveness 
of procedures to ensure consistency of data collection. Possible 
inconsistencies affect the accuracy of the measure and conclusions 
about the extent to which performance goals have been achieved; 
Some overlap with toll-free accounts quality. Both attempt to show the 
percentage of callers receiving accurate responses to account 
questions, but toll-free accounts quality includes CSR conformance with 
administrative procedures in computing that percentage. Overlap could 
cloud the bottom line and obscure performance results; Recommendations: 
Implement annual effectiveness studies to validate the accuracy of the 
data collection methods and establish goals for improving consistency, 
as needed; See note 1 to the table.
Measure name and definition[A]: Toll-free timeliness[I]; The 
successful resolution of all issues resulting from the callers first 
inquiry (telephone only); FY 2001 target and actual: Target: 82%; 
Actual: 82.8%; Weaknesses of measure and consequences: A reliability 
weakness exists because evaluations are based on judgments that are 
potentially inconsistent. No routine studies to determine effectiveness 
of procedures to ensure consistency of data collection. Possible 
inconsistencies affect the accuracy of the measure and conclusions 
about the extent to which performance goals have been achieved; 
Recommendations: Implement annual effectiveness studies to validate the 
accuracy of data collection methods and establish goals for improving 
consistency, as needed.
Measure name and definition[A]: Toll-free employee satisfaction; The 
percentage of survey participants that answered with a 4 or 5 (two 
highest scores possible) to the question considering everything, how 
satisfied are you with your job?; FY 2001 target and actual: Target: 
55%; Actual: 46%; Weaknesses of measure and consequences: None 
observed; Recommendations: None.
Measure name and definition[A]: CSR response level; The percentage of 
callers who started receiving service from a CSR within a specified 
period of time; FY 2001 target and actual: Target: 49%; Actual: 40.8%; 
Weaknesses of measure and consequences: Not clear because formula does 
not include calls that received a busy signal or resulted in a hang-up 
before a CSR came on the line, and the definition does not disclose 
that 
exclusion. Performance may be overstated and the real customer 
experience 
not reflected; Some overlap with average speed of answer. Both attempt 
to show how long callers waited before receiving service, except that 
CSR response level shows the number of callers receiving service within 
30 seconds. Overlap could cloud the bottom line and obscure performance 
results; Recommendations: Revise measure to include calls from 
taxpayers who tried to reach a CSR but did not, such as those who (1) 
hung-up while waiting to speak to a CSR, (2) were provided access only 
to automated services and hung up, and (3) received a busy signal; 
See note 1 to the table.
Measure name and definition[A]: Average speed of answer; The average 
number of seconds callers waited in queue before receiving service from
a CSR; FY 2001 target and actual: Target: not available; Actual: 
295 seconds; Weaknesses of measure and consequences: Target to be set 
upon completion of baseline data collection.[F]; Some overlap with 
toll-free CSR response level. Both attempt to show how long callers 
waited before receiving service, except that CSR response level shows 
the number of callers receiving service within 30 seconds. Overlap 
could cloud the bottom line and obscure performance results; 
Recommendations: See note 1 to the table.
Note 1: We identified this measure as having partial overlap with 
another measure. Telephone assistance officials generally agreed with 
our assessment and stated that some of these overlapping measures will 
be removed from future Strategy and Program Plans. The following 
recommendation applies to several measures as noted in the table: 
ensure that plans to remove overlapping measures are implemented.:
[A] The names of some measures have been modified slightly from the 
official names used by IRS for ease of reading and consistency 
purposes. For example, we replaced the word assistor with CSR. Also, 
the definitions of the measures listed in the table come from various 
IRS sources, including interviews.
[B] The Telephone Routing Interactive System is an interactive routes 
callers to CSRS or automated services and provides interactive 
services. Tele-Tax is a telephone system that provides automated 
services only.
[C] About 780,000 automated calls were included in the formula during 
the 2001 filing season. If they had not been included, the CSR level of 
service would have decreased by about 1 percentage point. The effect 
could be more significant in the future because IRS plans to increase 
the number of calls handled through automation.
[D] IRS plans to discontinue the toll-free tax law quality measure in 
fiscal year 2004.
[E] IRS plans to discontinue the toll-free accounts quality measure 
in fiscal year 2004.
[F] Although these measures did not have a measurable target in place, 
IRS is taking reasonable steps to develop a target.
[G] IRS changed the name of the toll-free tax law correct response 
rate measure to customer accuracy for tax law inquiries beginning in 
October 2002.
[H] IRS changed the name of the toll-free account correct response 
rate measure to customer accuracy for account inquiries beginning in 
October 2002.
[I] IRS discontinued the toll-free timeliness measure beginning in 
October 2002, and replaced it with a new quality timeliness measure.
Source: GAO comparison of IRSs December 13, 2000, July 25, 2001, and 
October 29, 2001, Strategy and Program Plans with the attributes in 
appendix I and an Embedded Quality Discussion Document (7/23/02), which 
discusses the changes IRS plans for its telephone assistance quality 
measures.
[End of table]
Electronic Filing and Assistance Performance Measures:
Of the 53 performance measures in our review, 13 are for electronic 
filing and assistance.[Footnote 44] Table 7 has information about each 
of the 13 measures.
Table 7: Electronic Filing and Assistance Performance Measures:
Measure name and definition[A]: Number of 1040 series returns 
electronically filed (millions); The number of Forms 1040, 1040A, and 
1040EZ filed electronically; FY 2001 target and actual: Target: 40.0; 
; Actual: 40.0; Weaknesses of measure and consequences: Target changed 
during filing season from 42.0 to 40.0. Changing the target in this 
instance was subjective in nature and resulted in an objectivity 
weakness as well; Some overlap with percent of individual returns 
electronically filed. Both measures show the extent of electronic 
filing by individuals--one in absolute numbers, the other as a percent 
of total filings. Overlap could cloud the bottom line and obscure 
performance results; Recommendations: Refrain from making changes to 
official targets unless extenuating circumstances arise; Disclose 
any extenuating circumstances in the Strategy and Program Plan and 
other key documents; See note 1 to the table.
Measure name and definition[A]: Number of business returns 
electronically filed (millions); The number of Forms 941, 1041, and 
1065 filed electronically; FY 2001 target and actual: Target: 3.7; 
Actual: 1.66; Weaknesses of measure and consequences: None observed; 
Recommendations: None.
Measure name and definition[A]: Total number of electronically filed 
returns (millions); The number of Forms 1040, 1040A, 1040EZ, 941, 
1041 and 1065 filed electronically; FY 2001 target and actual: 
Target: 43.7; Actual: 41.7; Weaknesses of measure and consequences: 
Target changed during filing season from 45.7 to 43.7. Changing the 
target in this instance was subjective in nature and resulted in an 
objectivity weakness as well; Recommendations: Refrain from making 
changes to official targets unless extenuating circumstances arise. 
Disclose any extenuating circumstances in the Strategy and Program Plan 
and other key documents.
Measure name and definition[A]: Number of information returns 
electronically filed (millions); The total number of information 
returns filed electronically. Includes Forms 1098, 1099, 5498, and W-2G 
and Schedules K-1. Excludes Forms W-2 and 1099-SSA/RRB received from 
the Social Security Administration; FY 2001 target and actual: 
Target: 334.0; Actual: 322.8; Weaknesses of measure and 
consequences: Some overlap with percent of information returns 
electronically filed. Both measures show the extent of electronic 
filing --one in absolute numbers, the other as a percent of total 
filings. Overlap could cloud the bottom line and obscure performance 
results; Recommendations: See note 1 to table.
Measure name and definition[A]: Percent of information returns 
electronically filed; The percentage of total information returns 
filed electronically; FY 2001 target and actual: Target: 24.4%; 
Actual: not available[B]; Weaknesses of measure and consequences: Some 
overlap with number of information returns electronically filed. Both 
measures show the extent of electronic filing --one in absolute 
numbers, the other as a percent of total filings. Overlap could cloud 
the bottom line and obscure performance results; Recommendations: See 
note 1 to table.
Measure name and definition[A]: Percent of individual returns 
electronically filed; The percentage of total 1040 series tax returns 
(Forms 1040, 1040A, and 1040EZ) filed electronically; FY 2001 target 
and actual: Target: 31%; Actual: 32%; Weaknesses of measure and 
consequences: Some overlap with number of 1040 series returns 
electronically filed. Both measures show the extent of electronic 
filing by individuals--one in absolute numbers, the other as a percent 
of total filings. Overlap could cloud the bottom line and obscure 
performance results; Recommendations: See note 1 to table.
Measure name and definition[A]: Number of payments received 
electronically (millions); All individual and all business tax 
payments made through the electronic federal tax payment system 
(EFTPS); FY 2001 target and actual: Target: 64.4; Actual: 53.8; 
Weaknesses of measure and consequences: Some overlap with percent of 
payments received electronically. Both measures show the extent to 
which payments are received electronically--one in absolute numbers, 
the other as a percent of total receipts. Overlap could cloud the 
bottom line and obscure performance results; Recommendations: See note 
1 to table.
Measure name and definition[A]: Percent of payments received 
electronically; The percentage of all individual and business tax 
payments made through EFTPS; FY 2001 target and actual: Target: 30%; 
Actual: not available[B]; Weaknesses of measure and consequences: 
Some overlap with number of payments received electronically. Both 
measures show the extent to which payments are received electronically-
-one in absolute numbers, the other as a percent of total receipts. 
Overlap could cloud the bottom line and obscure performance results; 
Recommendations: See note 1 to table.
Measure name and definition[A]: Number of electronic funds withdrawals/
credit card transactions (millions); The total number of credit card 
and direct debit payments processed through EFTPS; FY 2001 target and 
actual: Target: 1.0; Actual: 0.63; Weaknesses of measure and 
consequences: Some overlap with number and percent of payments received 
electronically. The payments covered by this measure are included in 
the universe of payments covered by the other two measures. Overlap 
could cloud the bottom line and obscure performance results; 
Recommendations: See note 1 to table.
Measure name and definition[A]: Number of IRS digital daily Web site 
hits (billions); The number of hits to IRSs Web site; FY 2001 
target and actual: Target: 2.0; Actual: 2.3; Weaknesses of measure 
and consequences: Measure is not clear and lacks reliability because, 
for example, initial access counts as multiple hits and movement 
throughout the Web site will count as additional hits; 
Recommendations: Either discontinue use of this measure or revise the 
way hits are calculated so that the measure more accurately reflects 
usage.
Measure name and definition[A]: Number of downloads from IRS .GOV 
(millions); The total number of tax forms downloaded from IRSs Web 
site; FY 2001 target and actual: Target: 311; Actual: 309; 
Weaknesses of measure and consequences: None observed; 
Recommendations: None.
Measure name and definition[A]: Customer satisfaction - individual 
taxpayers; The percentage of taxpayers who respond very satisfied 
with individual E-file products; FY 2001 target and actual: Target: 
76%; Actual: 83%; Weaknesses of measure and consequences: None 
observed; Recommendations: None.
Measure name and definition[A]: Employee satisfaction - Electronic 
filing and assistance; The percentage of survey participants that 
answered with a 4 or 5 (two highest scores possible) to the question 
considering everything, how satisfied are you with your job?; FY 2001 
target and actual: Target: 66%; Actual: 38%; Weaknesses of measure 
and consequences: None observed; Recommendations: None.
Note: We identified this measure as having partial overlap with another 
measure. Electronic filing and assistance officials told us that each 
of the overlapping measures we identified provides additional 
information to managers. Determining whether or not to remove 
overlapping measures is managements discretion.
[A] The names of some measures have been modified slightly from the 
official names used by IRS for ease of reading and consistency 
purposes. The definitions of the measures listed in the table come from 
various IRS sources, including interviews.
[B] Despite setting a target, actual data were not available because 
electronic filing and assistance did not begin tracking the measure 
until 2002.
Source: GAO comparison of IRSs December 13, 2000, July 25, 2001, and 
October 29, 2001, Strategy and Program Plans with the attributes in 
appendix I.
[End of table]
Field Assistance Performance Measures:
Of the 53 performance measures in our review, 14 are for field 
assistance. Table 8 has information about each of the 14 field 
assistance measures.
Table 8: Field Assistance Performance Measures:
Measure name and definition[A]: Customer satisfaction; From surveys 
established in 1998, an index was created to represent overall customer 
satisfaction with field assistance services, with a 7 being the 
best.[B]; FY 2001 target and actual: Target: 6.5 average score; 
Actual: 6.4 average score; Weaknesses of measure and consequences: 
None identified; Recommendations: None.
Measure name and definition[A]: Return preparation contacts; Total 
number of customers assisted with tax return preparation, including 
electronic and non-electronic tax return preparation at taxpayer 
assistance centers (TAC); FY 2001 target and actual: Target: 979,206; 
; Actual: 1,009,387; Weaknesses of measure and consequences: Name, 
definition, and formula of measure are not clear; Significant manual 
data collection process impedes reliability because of the potential 
for errors and inconsistencies that could affect the accuracy of the 
measure and conclusions about the extent to which performance goals 
have been achieved; Some overlap with return preparation units 
measure. Both measures attempt to show number of services provided, but 
the contact measure takes the number of taxpayers served into account 
and the units measure counts the number of returns prepared for those 
taxpayers served. Overlap could cloud the bottom line and obscure 
performance results; Recommendations: Make the name and/or definition 
of the measure more clear to indicate what is and is not included in 
the formula; See note 1 to the table; See note 2 to the table.
Measure name and definition[A]: Geographic coverage; Percentage of 
W&I taxpayer population with distinct characteristics, behaviors, and 
needs for face-to-face assistance within a 45-minute commuting distance 
from a TAC; FY 2001 target and actual: Target: 70%; Actual: 74%; 
Weaknesses of measure and consequences: Name, definition, and formula 
of measure are not clear; uncertainties exist among IRS officials about 
what is and is not included in the measure; The formula does not 
include all facilities, which could lead to misinterpreted results or a 
failure to properly identify alternative facility types to resolve 
access problems; Because the formula does not include all facilities, 
it is difficult for decision makers to determine if, when, and where 
additional TACs are needed; Recommendations: Make the name and/or 
definition of the measure more clear to indicate what is and is not 
included in the formula; Revise the formula to better reflect (1) 
the various types of field assistance facilities, including alternate 
sites and kiosks; (2) the types of services provided by each facility; 
and (3) the facilitys operating hours.
Measure name and definition[A]: Return preparation units; Actual 
number of tax returns prepared, in whole or in part, in a TAC or 
alternative site. (Multiple returns may be prepared for a single 
customer.); FY 2001 target and actual: Target: not available; Actual: 
not available; Weaknesses of measure and consequences: Name, 
definition, 
and formula of measure are not clear; Target to be set upon completion 
of 
data collection.[C]; Significant manual data collection process impedes 
reliability because of the potential for errors and inconsistencies 
that 
could affect the accuracy of the measure and conclusions about the 
extent 
to which performance goals have been achieved; Some overlap with return 
preparation contacts. Both measures attempt to show number of services 
provided, but the contact measure takes the number of taxpayers served 
into account and the units measure counts the number of returns 
prepared for those taxpayers served. Overlap could cloud the bottom 
line and obscure performance results; Recommendations: Make the name 
and/or definition of the measure more clear to indicate what is and is 
not included in the formula; See note 1 to the table; See note 2 
to the table.
Measure name and definition[A]: TACs total contacts; Total number of 
customers assisted, including number of customers assisted with tax 
return preparation, at TACs and alternate sites and via mobile 
services. All face-to-face, telephone, and correspondence contacts are 
included; FY 2001 target and actual: Target: 9,116,099; Actual: 
9,681,330; Weaknesses of measure and consequences: Name, definition, 
and formula of measure are not clear; Significant manual data 
collection process impedes reliability because of the potential for 
errors and inconsistencies that could affect the accuracy of the 
measure and conclusions about the extent to which performance goals 
have been achieved; Recommendations: Make the name and/or definition 
of the measure more clear to indicate what is and is not included in 
the formula; See note 1 to the table.
Measure name and definition[A]: Forms contacts; Total number of 
customers actually assisted by employees at TACs, alternate sites, and 
via mobile services by (1) providing forms from stock or (2) using a 
CD-ROM; FY 2001 target and actual: Target: 2,331,000; Actual: 
2,388,039; Weaknesses of measure and consequences: Name, definition, 
and formula of measure are not clear; Significant manual data 
collection process impedes reliability because of the potential for 
errors and inconsistencies that could affect the accuracy of the 
measure and conclusions about the extent to which performance goals 
have been achieved; Recommendations: Make the name and/or definition 
of the measure more clear to indicate what is and is not included in 
the formula; See note 1 to the table.
Measure name and definition[A]: Tax law contacts; Total number of 
customers assisted in TACs, alternate sites, and via mobile services 
with inquiries involving general tax law questions, non-account related 
IRS procedures, preparation or review of Forms W-7, Individual Taxpayer 
Identification Number documentation verification or rejection, a form 
request where probing requiring technical tax law training takes place, 
and assisting customers with audit reconsideration; FY 2001 target and 
actual: Target: not available; Actual: 1,787,338; Weaknesses of 
measure and consequences: Name, definition, and formula of measure are 
not clear; Target to be set upon completion of data collection.[C]; 
Significant manual data collection process impedes reliability 
because of the potential for errors and inconsistencies that could 
affect the accuracy of the measure and conclusions about the extent to 
which performance goals have been achieved; Recommendations: Make the 
name and/or definition of the measure more clear to indicate what is 
and is not included in the formula; See note 1 to the table.
Measure name and definition[A]: Account contacts; Total number of 
customers assisted in TACs, alternate sites, and via mobile services 
with inquiries involving account related inquiries including math error 
notices, Integrated Data Retrieval System work, payments not attached 
to a tax return, CP2000 inquiries, Individual Taxpayer Identification 
Number issues requiring account research, the issuance of Form 809 
receipts, and account related procedures; FY 2001 target and actual: 
Target: not available; Actual: not available; Weaknesses of measure 
and consequences: Name, definition, and formula of measure are not 
clear; Target to be set upon completion of data collection.[C]; 
Significant manual data collection process impedes reliability because 
of the potential for errors and inconsistencies that could affect the 
accuracy of the measure and conclusions about the extent to which 
performance goals have been achieved; Recommendations: Make the name 
and/or definition of the measure more clear to indicate what is and is 
not included in the formula; See note 1 to the table.
Measure name and definition[A]: Other contacts; Total number of 
customers assisted in TACs, alternate sites, and via mobile services 
with Form 2063, U.S. Departing Alien Income Tax statement, date 
stamping tax returns when the customer is present, non-receipt or 
incorrect W-2 inquiries, general information such as Service Center 
address and directions to other agencies; FY 2001 target and actual: 
Target: 3,869,000; Actual: 4,496,566; Weaknesses of measure and 
consequences: Name, definition, and formula of measure are not clear; 
; Significant manual data collection process impedes reliability 
because of the potential for errors and inconsistencies that could 
affect the accuracy of the measure and conclusions about the extent to 
which performance goals have been achieved; Recommendations: Make the 
name and/or definition of the measure more clear to indicate what is 
and is not included in the formula; See note 1 to the table.
Measure name and definition[A]: Tax law accuracy; The quality of 
service provided to TAC customers. Specifically, the accuracy of 
responses concerning issues involving tax law; FY 2001 target and 
actual: Target: not available; Actual: not available; Weaknesses of 
measure and consequences: Name, definition, and formula of measure are 
not clear; Target to be set upon completion of data collection.[C]; 
Recommendations: Make the name and/or definition of the measure more 
clear to indicate what is and is not included in the formula.
Measure name and definition[A]: Accounts/notices accuracy; The 
quality of service provided to TAC customers. Specifically, the 
accuracy of responses and/or IDRS transactions concerning issues 
involving account work and notices; FY 2001 target and actual: 
Target: not available; Actual: not available; Weaknesses of measure 
and consequences: Name, definition, and formula of measure are not 
clear; Target to be set upon completion of data collection.[C]; 
Recommendations: Make the name and/or definition of the measure more 
clear to indicate what is and is not included in the formula.
Measure name and definition[A]: Return preparation accuracy; The 
quality of service provided to TAC customers. Specifically, the 
accuracy of tax returns prepared in a TAC; FY 2001 target and actual: 
Target: not available; Actual: not available; Weaknesses of measure 
and consequences: Name, definition, and formula of measure are not 
clear; Target to be set upon completion of data collection.[C]; 
Recommendations: Make the name and/or definition of the measure more 
clear to indicate what is and is not included in the formula.
Measure name and definition[A]: Employee satisfaction; The percentage 
of survey participants that answered with a 4 or 5 (two highest scores 
possible) to the question considering everything, how satisfied are 
you with your job.; FY 2001 target and actual: Target: 62%; 
Actual: 51%; Weaknesses of measure and consequences: None observed; 
Recommendations: None.
Measure name and definition[A]: Alternate contacts; Total number of 
customers assisted at kiosks, mobile units, and alternate sites. It 
includes all face-to-face (including return preparation), telephone, 
and correspondence contacts; FY 2001 target and actual: Target: not 
available; Actual: not available; Weaknesses of measure and 
consequences: Target to be set upon completion of data collection.[C]; 
; Significant manual data collection process impedes reliability 
because of the potential for errors and inconsistencies that could 
affect the accuracy of the measure and conclusions about the extent to 
which performance goals have been achieved; Recommendations: See note 
1 to the table.
Note 1: IRS expects to minimize this potential for errors and 
inconsistency by equipping all of its TACS with an on-line automated 
tracking and reporting system known as the Queuing Management System 
(Q-Matic). This system is expected, among other things, to more 
efficiently monitor customer traffic flow and eliminate staff time 
spent completing Form 5311. Because IRS is in the process of 
implementing Q-Matic, we are not making any recommendation.
Note 2: We identified this measure as having partial overlap with 
another measure. Field assistance officials agreed with our assessment 
and stated that they plan to remove the return preparation contacts 
measure from the Strategy and Program Plan. The following 
recommendation applies to two measures, as noted in the table: ensure 
that plans to remove overlapping measures are implemented.:
[A] The names of some measures have been modified slightly from the 
official names used by IRS for ease of reading and consistency 
purposes. The definitions of the measures listed in the table come from 
various IRS sources, including interviews.
[B] Field assistance implemented a new customer satisfaction survey in 
fiscal year 2002. The index was changed, and a rating of 5 is now 
best.
[C] Although these measures did not have a measurable target in place, 
IRS is taking reasonable steps to develop a target.
Source: GAO comparison of IRSs December 13, 2000, July 25, 2001, and 
October 29, 2001, Strategy and Program Plans with the attributes in 
appendix I.
[End of table]
Submission Processing Performance Measures:
Of the 53 performance measures in our review, 11 are for submission 
processing.[Footnote 45] Table 9 has information about each of the 11 
submission processing performance measures.
Table 9: Submission Processing Performance Measures:
Measure name and definition[A]: Individual 1040 series returns filed 
(paper)[B]; The number of Forms 1040, 1040A, and 1040EZ filed at 
the eight W&I submission processing centers; FY 2001 target and 
actual: Target: 87,869,000; Actual: 74,972,667; Weaknesses of 
measure and consequences: None observed; Recommendations: None.
Measure name and definition[A]: Number of individual refunds issued 
(paper)[B]; The number of individual refunds issued by the eight 
W&I submission processing centers after the initial filing of a 
return; FY 2001 target and actual: Target: 48,000,000; Actual: 
45,456,534; Weaknesses of measure and consequences: None observed; 
Recommendations: None.
Measure name and definition[A]: Employee satisfaction; The 
percentage of survey participants that answered with a 4 or 5 (two 
highest scores possible) to the question considering everything, how 
satisfied are you with your job.; FY 2001 target and actual: Target: 
60%; Actual: 54%; Weaknesses of measure and consequences: None 
observed; Recommendations: None.
Measure name and definition[A]: Refund timeliness - individual 
(paper)[B]; The percentage of refunds issued to taxpayers within 40 
days of the date IRS received the individual income tax return; FY 
2001 target and actual: Target: 96.1%; Actual: 96.75%; Weaknesses 
of measure and consequences: Potential reliability weakness because 
data collected manually and evaluations of data based on judgment. 
Possible inconsistencies affect the accuracy of the measure and 
conclusions about the extent to which performance goals have been 
achieved; Recommendations: Based on the results of effectiveness 
studies, establish goals to improve consistency, as needed.
Measure name and definition[A]: Notice error rate; The percentage of 
incorrect submission processing master file notices issued to taxpayers 
(includes systemic errors).[C]; FY 2001 target and actual: Target: 
8.1%; Actual: 14.84%; Weaknesses of measure and consequences: 
Potential reliability weakness because data collected manually and 
evaluations of data based on judgment. Possible inconsistencies affect 
the objectivity of the measure and conclusions about the extent to 
which performance goals have been achieved; Recommendations: Based on 
the results of effectiveness studies, establish goals to improve 
consistency, as needed.
Measure name and definition[A]: Refund error rate - individual 
(paper)[B]; The percentage of refunds that have errors caused by IRS 
involving, for example, a persons name or refund amount (includes 
systemic errors).[C]; FY 2001 target and actual: Target: 13.6%; 
Actual: 9.75%; Weaknesses of measure and consequences: Potential 
reliability weakness because data collected manually and evaluations of 
data based on judgment. Possible inconsistencies affect the accuracy of 
the measure and conclusions about the extent to which performance goals 
have been achieved; Recommendations: Based on the results of 
effectiveness studies, establish goals to improve consistency, as 
needed.
Measure name and definition[A]: Letter error rate; The percentage of 
letters with errors issued to taxpayers by submission processing 
employees (includes systemic errors).[C]; FY 2001 target and actual: 
Target: 11.9%; Actual: 13.10%; Weaknesses of measure and 
consequences: Potential reliability weakness because data collected 
manually and evaluations of data based on judgment. Possible 
inconsistencies affect the objectivity of the measure and conclusions 
about the extent to which performance goals have been achieved; 
Recommendations: Based on the results of effectiveness studies, 
establish goals to improve consistency, as needed.
Measure name and definition[A]: Deposit timeliness (paper)[B]; Lost 
opportunity cost of money received by IRS but not deposited in the bank 
by the next day, per $1 billion of deposits, using a constant 8% annual 
interest rate; FY 2001 target and actual: Target: $746,712; 
Actual: $878,867; Weaknesses of measure and consequences: None 
observed; Recommendations: None.
Measure name and definition[A]: Deposit error rate; The percentage of 
payments misapplied based on the taxpayers intent; FY 2001 target and 
actual: Target: 4.9%; Actual: not available[D]; Weaknesses of 
measure and consequences: Objectivity weakness because sampling plan 
not consistently implemented; Potential reliability weakness because 
data collected manually and evaluations of data based on judgment. 
Possible inconsistencies affect the accuracy of the measure and 
conclusions about the extent to which performance goals have been 
achieved; Recommendations: See note 1 to the table; Based on the 
results of effectiveness studies, establish goals to improve 
consistency, as needed.
Measure name and definition[A]: Refund interest paid (per $1 million of 
refunds); The amount of refund interest paid per $1 million of 
refunds issued; FY 2001 target and actual: Target: $112; Actual: 
$128.63; Weaknesses of measure and consequences: None observed; 
Recommendations: None.
Measure name and definition[A]: Submission processing productivity; 
The weighted workload or work units processed per staff year expended; 
FY 2001 target and actual: Target: 28,787; Actual: 28,537; 
Weaknesses of measure and consequences: Not clear because (1) 
definition is not clearly stated, (2) managers do not understand their 
units contribution to the formula and (3) unit managers do not use the 
measure to assess performance; Recommendations: Revise the measure so 
it provides more meaningful information to users.
Note 1: We are not making a recommendation regarding the objectivity 
weakness for the deposit error rate measure because the Treasury 
Inspector General for Tax Administration recommended that IRS take 
steps to ensure that the sampling plan is being implemented 
consistently, and IRS reported that steps have been taken. :
[A] The names of some measures have been modified slightly from the 
official names used by IRS for ease of reading and consistency 
purposes. The definitions of the measures listed in the table come from 
various IRS sources, including interviews.
[B] Paper means that returns filed electronically (or their resulting 
refunds) are not included in the measure.
[C] A systemic error is an error caused by a computer programming error 
as opposed to an IRS employee.
[End of table]
[D] IRS could not provide actual data on this measure due to 
discrepancies in its data.
Source: GAO comparison of IRSs December 13, 2000, July 25, 2001, and 
October 29, 2001, Strategy and Program Plans with the attributes in 
appendix I.
[End of section]
Appendix III: Comments from the Internal Revenue Service:
Note: GAO comments supplementing those in the report text appear at the 
end of this appendix.
DEPARTMENT OF THE TREASURY
INTERNAL REVENUE SERVICE
WASHINGTON, D.C. 
20224:
November 1, 2002:
Mr. James R. White, Director, Tax Issues, U.S. General Accounting 
Office 
441 G Street, N.W. Washington, D.C. 20548:
Dear Mr. White:
I appreciate your recognition of the substantial progress we have made 
in implementing our balanced measures and our strategic planning 
process. We issued our first Strategy and Program Plan (SPP) and 
performance measures in Fiscal Year (FY) 2000, which I believe was 
great progress in a short period of time. We continue to gain 
experience and focus on the key attributes of performance measures as 
we use them in our day-to-day operations. Your observation that this is 
an ongoing process is exactly on point. The observations of your staff 
will benefit us as we continue to improve our performance measures. I 
believe your report is an insightful review of the measures we 
developed for use in FY 2001. We recently completed our SPP and the 
related performance measures for FYs 2003-2004. We will consider your 
suggestions as we review our current plan and develop plans for our 
next SPP cycle.
I am particularly impressed with the detailed definitions, 
explanations, and examples your staff developed for the nine attributes 
of successful performance measures. I believe the Wage and Investment 
(W&I) Division can use these standards as a helpful checklist when they 
develop future performance measures.
I also was pleased to note your observation that our measures had many 
of the attributes for successful performance. This indicates that we 
appropriately developed and properly targeted key performance measures. 
I also agree the measures that did not satisfy all of the attributes 
will give us opportunities for further refinement rather than 
invalidate their overall value. As you noted, we have several 
initiatives underway to continue improving these measures. Overall, 
your report is objective and balanced.
I want to share some additional points for your consideration:
*Although the filing season is our busiest and most visible period, our 
performance measures are for the entire fiscal year.
*The report is focused on the performance measures and their 
relationship to the SPP without any mention of the importance of the 
Operating Units Business Plans. The Business Plan is a derivative of 
the SPP that is linked to tactical actions, resource allocations, and 
performance milestones that drive the day-today activities and goals 
of the Operating Units. The Business Plan is the primary vehicle for 
accountability through the Business Performance Review Process and 
individual performance appraisals.
*Few of our performance measures are isolated measures. No individual 
measures can adequately reflect the broad range of our responsibilities 
and our mission. We manage our programs by reviewing performance 
measures, diagnostic measures, performance indicators, and numerous 
other data sources to ensure a broad perspective of our service to our 
customers.
I have addressed the recommendations in more detail below:
Recommendations for Executive Action:
We recommend that the Commissioner of Internal Revenue Service direct 
the appropriate officials to do the following:
Recommendation 1:
Take steps to ensure that the agencywide goals clearly align with the 
operating division goals and performance measures for each of the four 
areas reviewed. Specifically, (1) clearly document the relationship 
among agencywide goals, operating division goals, and performance 
measures (the other three program areas may want to consider developing 
a template similar to the one Field Assistance developed, shown in 
figure 4) and (2) ensure that the relationship among goals and measures 
is communicated to staff at all levels of the organization.
Response:
We agree with this recommendation and in the next SPP, we will review 
the performance measures for the four W&I areas to ensure that we align 
and document their relationship to operating division goals and 
agencywide goals. The Operating Units Business Plans communicate the 
relationship of SPP goals and measures throughout the organization. 
Staff at all levels should recognize their role in delivering the 
Business Plan. The four program areas reviewed in the W&I Division 
distributed information on the SPP and Business Plan through annual 
leadership conferences at each site for FYs 2002 and 2003.
Recommendation 2:
Make the name and definition of several field assistance measures 
(i.e., geographic coverage, return preparation contacts, return 
preparation units, TACs total contacts, forms contacts, tax law 
contacts, account contacts, other contacts, tax law accuracy, 
account/notice accuracy, and return preparation accuracy) more 
clear 
to indicate what is and is not included in the formula.
Response:
Field Assistance recently updated the data dictionary for FY 2003. The 
updated dictionary addresses your recommendation on clarity and 
specifically identifies what is or is not included in the formulas. We 
gave a copy of the updated document to your staff. We have updated the 
data dictionary to include the purpose of the performance measurement, 
the data limitations associated with data gathering of the measure, and 
calculation changes from the prior year. It also provides a complete 
description of the methodology used in capturing the data, the critical 
path of how the measure originates and moves through the process, and 
the level of reviews to ensure quality. Field Assistance uses the 
current data dictionary in reporting measures to all levels of the 
organization.
Recommendation 3:
As discussed in the body of this report and in appendix II, modify the 
formulas used to compute various measures to improve clarity. If 
formulas cannot be implemented in time for the next issuance of the 
SPP, then modify the name and definition of the following measures so 
it is clearer what is or is not included in the measure.
Recommendation 3(a):
Remove automated calls from the formula for the CSR level of service 
measure.
Response:
We published the definition of this measure in the SPP, Data 
Dictionary, Measures Matrix, and numerous other sources. We believe 
that including the count of callers who choose an automated service 
while waiting for CSR service is appropriate. The formula accurately 
reflects the percentage of customers that wanted to speak to a CSR and 
subsequently received service. While we are promoting the use of 
automated services as an alternative to CSR service, we expect 
increases to occur before a customer enters the CSR queue. The growth 
in automation service while in queue for CSR service should remain 
small or decrease. We do not believe that this measure merits further 
change.
Recommendation 3(b):
Revise the CSR response level measure to include calls from taxpayers 
who tried to reach a CSR but did not, such as those who (1) hung up 
while waiting to speak to a CSR, (2) were provided access only to 
automated services and hung up, and (3) received a busy signal.
Response:
We do not agree that we should modify this measure. The methodology and 
30 second threshold for this measure is in accordance with the industry 
standard. This measure only applies to services answered and should not 
include abandon calls, automated service disconnects, or busy signals. 
Altering this measure would deviate from the industry standards and 
hinder our ability to gauge success in meeting this world class 
service goal.
Recommendation 3(c):
Analyze and use new or existing data to determine why calls are 
transferred and use the data to revise the CSR services provided 
measure so that it only reflects transferred calls in which the caller 
received help from more than one CSR (i.e., exclude calls in which a 
CSR simply transferred the call and did not provide service.):
Response:
We agree in concept with your recommendation. We are continuing to 
examine previously collected data on transferred calls from FY 2002. We 
are also studying the anticipated impact that our new Toll Free 
Operating Strategy will have on this measure. We specifically designed 
this strategy to simplify the scripts and telephone menus to make the 
customers self-selection process easier and more efficient. After 
assessing the impact of the Toll Free Operating Strategy, we will then 
review the recommendation for possible change in FY 2004.
Recommendation 3(d):
Either discontinue use of the number of IRS digital daily Web site 
hits measure or revise the way hits are calculated so that the 
measure more accurately reflects usage.
Response:
Due to privacy restrictions associated with the use of cookies, we 
cannot track the actual web site use. Instead, for FY 2003, we will 
implement three new diagnostic indicators related to the web site. 
These indicators (page view, unique visitors, and visits) will give us 
additional information to track the system performance and gauge the 
traffic on the web site. We will monitor these indicators for a year 
and decide whether to include them as performance measures in the 2004-
2005 SPP.
We will also continue to measure the number of hits and downloads to 
the web site. However, we will clarify the definition of hits to 
reflect that each file requested by a visitor registers as a hit and 
several hits can occur on each page.
Recommendation 3(e):
Revise Field Assistances geographic coverage measure by ensuring 
that the formula better reflects (1) the various types of field 
assistance facilities, including alternate sites and kiosks; (2) the 
types of services provided by each facility; and (3) the facilitys 
operating hours.
Response:
We agree that we should have revised the geographic coverage 
description to include more than just Taxpayer Assistance Centers 
(TAC).We are working with representatives from the Office of Program 
Evaluation and Risk Analysis to modify the formula to ensure that the 
formula reflects the appropriate elements by June 30, 2003. In 
addition, we will use the model to assist in determining the locations 
for different delivery options.
Recommendation 3(f):
Revise Submission Processings productivity measure so it provides 
more meaningful information to users.
Response:
We recognize that this measure needs improvement. The broad range of 
returns and documents processed and numerous other variables that can 
impact efficiency drives the complexity of the measure. The current 
measure seeks to account for those differences to ensure equity and 
fairness in the measurement process. We have looked at alternative ways 
to measure productivity but have not found a suitable replacement for 
this measure. We will continue our efforts to develop a more meaningful 
productivity measurement.
Recommendation 4:
Refrain from making changes to official targets, such as Electronic 
Filing and Assistance did in FY 2001, unless extenuating circumstances 
arise. Disclose any extenuating circumstances in the SPP and other key 
documents.
Response:
We agree that we should only make changes to official targets under the 
circumstances you describe and that disclosing these changes is 
appropriate. This approach is consistent with our overall practice.
Recommendation 5:
Modify procedures for the toll-free customer satisfaction survey, 
possibly by requiring that the administrators listen to the entire 
call, to better ensure that the administrators (1) notify CSRs that 
their call was selected for the survey as close to the end of the call
as possible and (2) can accurately answer the questions they are 
responsible for on the survey.
Response:
We agree we can improve this process. We will instruct the 
administrators to listen to each call from its beginning to as close to 
the conclusion as practical. Formalizing this practice will also enable 
the administrators to accurately answer the questions on the survey.
Recommendation 6:
Implement annual effectiveness studies to validate the accuracy of the 
data collection methods used for the five telephone measures (toll-
free tax law quality, toll-free accounts quality, toll-free tax law 
correct response rate, toll-free account correct response rate, and 
toll-free timeliness) subject to potential consistency problems. The 
studies could determine the extent to which variation exists in 
collecting data and recognize the associated impact on the affected 
measures. For those measures, and for the five Submission Processing 
measures that already have effectiveness studies in place (refund 
timeliness-individual (paper), notice error rate, refund error 
rateindividual (paper), letter error rate, and deposit error 
rate), IRS should establish goals for improving consistency, as 
needed.
Response:
We have ongoing processes to ensure that we properly administer the 
collection methods for the five telephone measures to minimize 
potential consistency problems. We do not agree that an annual 
independent review by a non-CQRS analyst is merited. Members of the 
Treasury Inspector General for Tax Administration (TIGTA) perform 
indepth oversight activities annually covering these collection 
methods. While we will work to improve consistency, we do not agree 
that we should incorporate a consistency improvement goal in the SPP 
process.
Recommendation 7:
Ensure that plans to remove overlapping measures in Telephone and Field 
Assistance are implemented.
Response:
We will continue our process of reviewing measures identified as 
overlapping and deleted those that truly were redundant.
Recommendation 8:
As discussed in the body of this report, include the following missing 
measures in the SPP in order to better cover governmentwide priorities 
and achieve balance.
Recommendation 8(a):
In the spirit of provisions in the Chief Financial Officers Act of 
1990 and Financial Accounting Standards Number 4, develop a cost of 
services measure using the best information currently available for 
each of the four areas discussed in this report, recognizing data 
limitations as prescribed by GPRA. In doing so, adhere to guidance, 
such as Office of Management and Budget Circular A-76, and consider 
seeking outside counsel to determine best or industry practices.
Response:
Development of cost of services measures for Telephone Assistance, 
Electronic Filing and Assistance, Field Assistance, and Submission 
Processing is dependent on Servicewide deployment of the Integrated 
Financial System (IFS). The first release of IFS, scheduled for October 
2003, will facilitate financial reporting and financial audits. The 
second release of IFS, planned for March 2005, will include Property 
and Performance Management. At this time, the development of cost of 
services measures is directly linked to having a mechanism that 
provides cost information for performance activities. The Service is 
moving towards this goal with successful implementation of the IFS 
system.
Recommendation 8(b):
Given the importance of automated telephone assistance, develop a 
customer satisfaction survey and measure for automated assistance.
Response:
We agree that measuring customer satisfaction with automated services 
is important. Our newer interactive Internet services have satisfaction 
surveys incorporated in the program. We are continuing to upgrade our 
automated services and will be implementing telephone system 
architectural changes as part of the Customer Communications 
Engineering Study. We will review your recommendation to evaluate the 
benefit of programming and implementing a customer satisfaction survey 
system based on outdated delivery systems.
Recommendation 8(c):
Put the automated completion rate measure back in the SPP after 
revising the formula so that calls for recorded tax information are not 
counted as completed when taxpayers hang up before receiving service.
Response:
We continue to track and monitor the automated completion rate as a 
diagnostic measure.We do not plan to modify the formula nor do we 
intend to reinstate it as a measure in the SPP.
Recommendation 8(d):
Add one or more quality measures to Electronic Filing and Assistances 
suite of measures in the SPP.Possible measures include processing 
accuracy, refund timeliness, electronically filed, and number of 
electronic returns rejected.:
Response:
The quality of electronic filing has consistently been high due to the 
pre-submission checks integrated into the system. We do track and 
monitor numerous diagnostic indicators that reflect the quality of 
electronic filing. We use this data to determine if there are error 
trends that need to be addressed. We do not believe incorporating these 
indicators as a performance measure in the SPP would enhance the 
electronic filing program.
Recommendation 8(e):
Re-implement Field Assistances timeliness measure.
Response:
Field Assistance agrees that timeliness goals are important in 
providing service to taxpayers; however, we found that this is 
detrimental to quality service in TACs because the employees tend to 
rush the customers when traffic is high. Realistic expectations provide 
a framework for our workers to provide appropriate service to the 
taxpayer with the goal of taking the requisite time to provide complete 
and accurate assistance. We will continue to use positive and negative 
feedback from customers responding to the promptness of service 
section of the satisfaction survey as a gauge of service. In addition, 
we are still tracking wait-times in locations equipped with the Queuing 
Management System (Q-Matic) System. The Q-Matic System is an on-line 
automated tracking and reporting system. We agree errors occur when 
manual methods of tracking workload volume and staff hours are used. In 
order to minimize reporting errors and better track wait-time, we plan 
to equip all of our TACs with this system.We can have Q-Matic installed 
and networked at all TACs nationwide by the end of FY 2004 with the 
planned funding.
Recommendation 8(j):
Develop a measure that provides information about Field Assistances 
efficiency.
Response:
Field Assistance is implementing a performance monitoring system to 
monitor productivity measures. We will use this system as a diagnostic 
tool to identify organizational performance measures strengths and 
weaknesses and not as an evaluative tool as we are a Section 1204 
organization. We will test the system during FY 2003 to determine the 
validity and usefulness of the data captured. At the end of the fiscal 
year, we will decide whether to continue with the current system, or 
modify it.
Again, I appreciate your observations and recommendations. If you have 
questions or comments, please call Floyd Williams, Director, 
Legislative Affairs, at (202) 622-3720.
Sincerely,
Charles O. Rossotti
Signed by Charles O. Rossotti
1. We recognize that IRSs performance measures cover entire fiscal
 years. We reviewed 53 of the measures for all of fiscal year 2001, 
 and we reported the full years results in appendix II.
2. We reviewed the business plans for all four program areas we 
 reviewed. Although we did not comment specifically about the 
 business performance review process in the report, we noted in the 
 background and field assistance sections that the business plans 
 communicate part of the relationship among the various goals and 
 measures.
3. Figure 4 shows an excerpt of field assistances business unit plan. 
 As noted in the figure, the template used to communicate the 
 relationship between goals and measure is missing some key 
 components. Figure 2 is our attempt to show the complete relationship 
 among IRSs various goals and measures--it is based on multiple 
 documents.
[End of section]
Appendix IV: GAO Contacts and Staff Acknowledgments:
GAO Contacts:
James White (202) 512-9110:
Dave Attianese (202) 512-9110:
Acknowledgments:
In addition to those named above, Bob Arcenia, Healther Bothwell, Rudy 
Chatlos, Grace Coleman, Evan Gilman, Ron Heisterkamp, Ronald Jones, 
John Lesser, Allen Lomax, Theresa Mechem, Libby Mixon, Susan Ragland, 
Meg Skiba, Joanna Stamatiades, and Caroline Villanueva made key 
contributions to this report.
[End of section]
Bibliography:
To determine whether the Internal Revenue Services (IRS) performance 
goals and measures in four key program areas demonstrate results, are 
limited to the vital few, cover multiple program priorities, and 
provide useful information in decision making, we developed attributes 
of performance goals and measures. These attributes were largely based 
on previously established criteria found in prior GAO reports; our 
review of key legislation, such as the Government Performance and 
Results Act of 1993 (GPRA) and the IRS Restructuring and Reform Act of 
1998; and other performance management literature. Sources we referred 
to for this report follow.
101st Congress. Chief Financial Officers Act of 1990. P.L. 101-576. 
Washington, D.C.: January 23, 1990.
103rd Congress. Government Performance and Result Act of 1993. P.L. 
103-62. Washington, D.C.: January 5, 1993.
103rd U.S. Senate. The Senate Committee on Government Affairs GPRA 
Report. Report 103-58. Washington, D.C.: June 16, 1993.
105th Congress. IRS Restructuring and Reform Act. P.L. 105-206. 
Washington, D.C.: July 22, 1998.
Internal Revenue Service. Managing Statistics in a Balanced Measures 
System. Handbook 105.4. Washington, D.C.: October 1, 2000.
The National Partnership for Reinventing Government. Balancing 
Measures: Best Practices in Performance Management. Washington, D.C.: 
August 1, 1999.
Office of Management and Budget, Preparation and Submission of Budget 
Estimates. Circular No. A-11, Revised. Transmittal Memorandum No. 72. 
Washington, D.C.: July 12, 1999.
Office of Management and Budget. Circular A-76, Revised. Supplemental 
Handbook, Performance of Commercial Activities. Washington, D.C.: March 
1996 (Revised 1999).
Office of Management and Budget. Managerial Cost Accounting Concepts 
and Standards for the Federal Government. Statement of Federal 
Financial Accounting Standards, Number 4. Washington, D.C.: July 31, 
1995:
[End of section]
Related Products:
U.S. General Accounting Office. Internal Revenue Service: Assessment of 
Budget Request for Fiscal Year 2003 and Interim Results of 2002 Tax 
Filing Season. (GAO-02-580T). Washington, D.C.: April 9, 2002.
U.S. General Accounting Office. Tax Administration: Assessment of IRSs 
2001 Tax Filing Season. (GAO-02-144). Washington, D.C.: December 21, 
2001.
U.S. General Accounting Office. Human Capital: Practices That Empowered 
and Involved Employees (GAO-01-1070). Washington, D.C.: September 14, 
2001.
U.S. General Accounting Office. Managing For Results: Emerging Benefits 
From Selected Agencies Use of Performance Agreements (GAO-01-115). 
Washington, D.C.: October 30, 2000.
U.S. General Accounting Office. Agency Performance Plans: Examples of 
Practices That Can Improve Usefulness to Decisionmakers (GAO/GGD/AIMD-
99-69). Washington, D.C.: February 26,1999.
U.S. General Accounting Office. The Results Act: An Evaluators Guide 
to Assessing Agency Annual Performance Plans (GAO/GGD-10.1.20). 
Washington, D.C.: April 1,1998.
U.S. General Accounting Office. Executive Guide: Effectively 
Implementing the Government Performance and Results Act (GAO/GGD-96-
118). Washington, D.C.: June 1996.
U.S. General Accounting Office. Executive Guide: Improving Mission 
Performance Through Strategic Information Management and Technology 
(GAO/AIMD-94-115). Washington, D.C.: May 1, 1994.
FOOTNOTES
[1] Although April 15 is generally considered the end of the filing 
season, millions of taxpayers get extensions from IRS that allow them 
to delay filing until as late as October 15.
[2] IRS tracks its performance in providing filing season-related 
telephone service through mid-July instead of April because it receives 
many filing season-related calls after April 15 from taxpayers who are 
inquiring about the status of their refunds or responding to notices 
they received from IRS related to returns they filed. 
[3] Some earlier work includes U.S. General Accounting Office, 
Executive Guide: Effectively Implementing the Government Performance 
and Results Act, GAO/GGD-96-118 (Washington, D.C.: June 1996) and U.S. 
General Accounting Office, The Results Act: An Evaluators Guide to 
Assessing Agency Annual Performance Plans, GAO/GGD-10.1.20 
(Washington, D.C.: Apr. 1998).
[4] The four characteristics are overarching, thus there is not 
necessarily a direct link between any one attribute and any one 
characteristic.
[5] U.S. General Accounting Office, Internal Revenue Service: 
Assessment of Budget Request for Fiscal Year 2003 and Interim Results 
of 2002 Tax Filing Season, GAO-02-580T (Washington, D.C.: Apr. 9, 
2002).
[6] GPRA, P.L. 103-62, was enacted to hold federal agencies accountable 
for achieving program results. IRSs balanced measurement system is 
consistent with the intent of GPRA.
[7] IRSs Restructuring and Reform Act of 1998, P.L. 105-206, was 
enacted on July 22, 1998, and calls for broad reforms in areas such as 
the structure and management of IRS, electronic filing, and taxpayer 
protection and rights.
[8] The other components include revamped business practices, customer-
focused operating divisions, management roles with clear 
responsibility, and new technology.
[9] As part of IRSs reorganization that took effect in October 2000, 
IRS established four operating divisions that serve specific groups of 
taxpayers. The four divisions are (1) Wage and Investment, (2) Small 
Business and Self-Employed, (3) Large and Mid-Size Businesses, and (4) 
Tax Exempt and Government Entities.
[10] The Strategy and Program Plans we used in our analysis had actual 
performance information for part of the current fiscal year and 
planning information for the current and two subsequent fiscal years. 
An IRS manager said the agency plans to stop including actual 
information in Strategy and Program Plans prepared after fiscal year 
2002.
[11] GAO/GGD-96-118.
[12] Office of Management and Budget, Preparation and Submission of 
Budget Estimates, Circular No. A-11, Revised. Transmittal Memorandum 
No. 72 (Washington, D.C.: 
July 12, 1999).
[13] IRS, Managing Statistics in a Balanced Measures System, Handbook 
105.4 (Washington, D.C.: Oct. 1, 2000).
[14] The data dictionary is an IRS document that provides information 
on performance measures, such as the measures name, description, and 
methodology.
[15] IRS deleted its automated completion rate measure in the 2002 
Strategy and Program Plan and now has 14 telephone measures. However, 
IRS still tracks that measure. 
[16] There were about 30 million of these calls during in fiscal year 
2001, which can have a significant impact on the CSR response level 
measure. 
[17] CSRs answer about 24 percent of all incoming calls. 
[18] As of January 2002, there were 53 quality reviewers in the 
Centralized Quality Review Site: 26 for tax law inquiries, 20 for 
account inquiries, and 7 others.
[19] CQRS is responsible for monitoring the accuracy of telephone 
assistance. It produces various reports that show call sites what 
errors CSRs are making so site managers can take action to reduce those 
errors.
[20] IRS significantly modified its five quality measures beginning in 
October 2002 based on the results of its initiative, which was aimed at 
redesigning the way IRS measures quality to better capture the 
taxpayers experience. Specifically, IRS renamed the toll-free correct 
response rate measures for tax law and account inquiries to customer 
accuracy for tax law or account inquiries. Plans call for the tax 
quality measures for tax law and account inquiries to be discontinued, 
but reported in fiscal year 2003 for trending and comparative purposes. 
IRS also eliminated the toll-free timeliness measure and replaced it 
with a new quality timeliness measure. Finally, IRS implemented a new 
measure called professionalism.
[21] The Chief Financial Officers Act, P.L. 101-576,underscores the 
importance of improving financial management in the federal government. 
Among other things, it calls for developing and reporting cost 
information.
[22] Statement of Federal Financial Accounting Standard Number 4, 
Managerial Cost Accounting Concepts and Standards for the Federal 
Government, is aimed at providing reliable and timely information on 
the full cost of federal programs, their activities, and outputs.
[23] The Annual Performance Plan is a key document IRS produces each 
year to comply with the requirements of GPRA. It highlights a limited 
number of IRS performance measures.
[24] U.S. General Accounting Office, Tax Administration: Assessment of 
IRSs 2001 Tax Filing Season, GAO-02-144 (Washington, D.C.: Dec. 21, 
2001).
[25] 1040 series returns are individual income tax returns filed on 
Forms 1040, 1040A, and 1040EZ.
[26] The masterfile is the system where most of IRSs taxpayer data 
resides. 
[27] Processing accuracy refers to the total number of returns that 
do not go to the error resolution system. Transactions that fail 
validity checks during processing are corrected through the error 
resolution system. 
[28] Refund timeliness, electronically filed is the amount of time it 
takes for taxpayers to receive their refunds when filing 
electronically.
[29] Electronic returns can be rejected, for example, if taxpayers fail 
to include required Social Security numbers. IRS requires taxpayers to 
correct such errors before it will accept their electronic returns.
[30] Alternate sites are staffed with field assistance employees and 
offer limited face-to-face services, such as preparing returns and 
distributing forms. Field assistance has about 
50 alternate sites, such as temporary sites in shopping malls and 
libraries. Alternate sites are currently not included in the 
geographic coverage measure. 
[31] Kiosks are automated machines that taxpayers can use to obtain 
certain forms, answers to frequently asked questions, and general IRS 
information in English and Spanish. Kiosks are currently not included 
in the geographic coverage measure.
[32] The Resources Management Information System is the primary 
management information system that field assistance uses to track 
workload volume and staff hour expenditures.
[33] GAO-02-144.
[34] Of about 420 TACs, 123 had Q-Matic as of June 2002. IRS officials 
stated that installation and networking of Q-Matic in all offices is 
scheduled to be complete by September 30, 2005. In the meantime, IRS 
plans to pilot an installed and networked Q-Matic system in all the 
TACs that are located in one of IRSs seven management areas during the 
first quarter of 2003.
[35] Treasury Inspector General for Tax Administration, Walk-in 
Customer Satisfaction Survey Results Should Be Qualified If Used for 
the GPRA, 2000-10-079 (Washington, D.C.: May 17, 2000).
[36] The number of units would generally be larger than the number of 
contacts. For example, if a taxpayer received help in preparing his or 
her return and his or her childs return, field assistance would count 
that service as one return preparation contact and two return 
preparation units.
[37] TACs monitor timeliness, but IRS does not report the measure in 
the Strategy and Program Plan.
[38] The Integrated Submission and Remittance Processing System is the 
system IRS uses to process tax returns and remittances.
[39] Treasury Inspector General for Tax Administration, The Internal 
Revenue Service Needs to Improve Oversight of Remittance Processing 
Operations, 2003-40-002 (Washington, D.C.: Oct. 7, 2002).
[40] Submission processing did have some data related to the average 
direct labor cost to process some paper returns in 1999. 
[41] Q-Matic is an automated tracking and reporting system that is 
expected to more efficiently monitor customer traffic flow and wait 
times and eliminate staff time completing Form 5311. Of about 420 TACs, 
123 had Q-Matic as of June 2002.
[42] An alternative form of measurement may be either (1) separate, 
descriptive statements of a minimally effective program or (2) a 
successful program, expressed with sufficient precision and in such 
terms that would allow for an accurate, independent determination to be 
made of how actual performance compares with the goals stated. An 
example would be the polio vaccine and how its value to society is 
judged by experts through a peer review.
[43] IRS deleted its automated completion rate measure in the 2002 
Strategy and Program Plan and now has only 14 telephone measures. 
However, IRS still tracks this measure.
[44] IRS has since added three measures (number of information returns 
filed by magnetic tape, percent of information returns filed by 
magnetic tape, and customer satisfaction-business) that were not 
part of our review. In addition, electronic filing and assistance is 
developing new performance measures and goals because it is in the 
midst of a major reorganization. When the reorganization is completed, 
electronic filing and assistance will no longer be responsible for all 
the operational programs for which it was responsible in 2001 and 2002. 
Electronic filing and assistance will remain responsible for strategic 
services, Internet development services, and development services. The 
IRS organizations assuming responsibility for electronic filing and 
assistances operational programs will be responsible for the related 
performance measures and goals.
[45] IRS is developing a measure of customer satisfaction for 
submission processing.
GAOs Mission:
The General Accounting Office, the investigative arm of Congress, 
exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAOs commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAOs Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.
Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as Todays Reports, on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select Subscribe to daily E-mail alert for newly 
released products under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone: 	
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548: