IRS Telephone Assistance: Limited Progress and Missed Opportunities to Analyze Performance in the 2001 Filing Season
Released: 2001-12-26T00:00:00

Letter

Background

In the 2001 tax filing season, IRS received more than 70.7 million calls on its three toll-free assistance numbers and answered over 50.5 million calls—assistors answered 22.7 million calls and automated systems answered 27.8 million calls. As in previous years, IRS had three toll-free telephone numbers that taxpayers could call with questions about tax law, taxpayer accounts, and refunds. Located at 26 call sites, IRS has about 10,000 assistors that help taxpayers with a variety of questions ranging from the applicability of tax laws to the status of their accounts. IRS’ call sites are supervised by 10 field directors, each of whom oversees two to three sites.

IRS has four measures to evaluate the extent to which taxpayers are provided with accessible telephone assistance, and four to evaluate the extent to which taxpayers are provided with accurate telephone assistance. (For more information on IRS’ telephone assistance access and accuracy measures see app. I.) IRS’ measures of access are based on actual counts of calls using data collected by IRS’ telephone system. IRS’ measures of the accuracy of assistance, the quality and correct response measures, are estimates based on representative samples of nationwide calls that quality assurance staff monitor and score for accuracy. IRS began collecting data on correct responses in June 2000, so there are no data for the 2000 tax filing season to compare with 2001.

Over the years, IRS has studied its telephone performance and made changes designed to improve it. For example, in 1999, IRS extended its hours of service to 24 hours a day, 7 days a week. By providing around-the- clock service, IRS expected to distribute demand more evenly and thus improve taxpayers’ access to service. With the increased use of call- routing technology in 1999, IRS began to manage its telephone operations centrally at the Joint Operations Center in Atlanta. Routing calls to the first available assistor who had the necessary skills to answer the taxpayer’s question was expected to improve taxpayers’ access to service and lessen the disparity in the level of service across sites. However, the level of service declined in 1999, and the quality of service was mixed in the 2000 tax filing season and below IRS’ long-term goal of providing world-class customer service.

According to IRS, some of the key factors that affected performance in the 2000 tax filing season were the demand for assistance, staffing levels, assistor productivity, assistor skills, and IRS’ guidance for assistors. As we discussed in a previous report, IRS’ analyses did not cover all key management decisions or other key factors that could have affected telephone performance. Additionally, determining how each factor affected performance was made even more difficult because many of the factors are interrelated; changes in one can affect another. The IRS Commissioner has recognized the complex interrelationships within the telephone-operating environment and has stated that years of sustained effort will be required for IRS to achieve its goal of providing world-class telephone service.

Scope and Methodology

To address our objectives, we interviewed IRS officials involved in managing toll-free telephone operations and obtained and analyzed supporting documentation as follows: To assess IRS’ performance in responding to calls on the three main telephone assistance toll-free numbers, we compared the 2001 tax filing season performance for accessibility and accuracy measures with IRS’ performance in the 2000 tax filing season and its 2001 performance targets.

To assess IRS’ efforts to determine the factors that affected performance in the 2001 tax filing season, including actions it took to improve performance, we used as criteria GPRA and IRS’ own guidance on analyzing performance data. We interviewed IRS officials in the Wage and Investment and Small Business and Self-Employed Divisions, and the Joint Operations Center. We also analyzed various documents, including reports on IRS’ efforts to determine the factors that affect telephone performance and the results of actions to improve performance. In addition, we used a questionnaire to obtain information from the 10 field directors about their efforts to identify the factors that affected performance and assess the effectiveness of actions taken to improve performance.

While we did not independently assess the accuracy of IRS’ performance data, we verified that IRS had procedures in place intended to ensure data reliability. We did our work from February 2001 through October 2001 in accordance with generally accepted government auditing standards.

IRS Made Limited Progress Toward Providing World-Class Telephone Service

IRS made limited progress in the 2001 tax filing season toward its long- term goal of providing world-class telephone service. When compared with the 2000 tax filing season, access and accuracy performance improved by 2 percentage points or less in three of the six comparable measures. The quality of responses to account inquiries increased 10 percentage points, and there was a 4 percentage point decline in callers who hung up while waiting to speak with an assistor; however, taxpayers waited 15-percent longer to speak with an assistor. When compared with 2001 performance targets, IRS did not meet any of its accessibility goals. These targets were intended to move IRS toward its goal of providing world-class service. Although it met or exceeded the 2001 quality targets, IRS did not meet the current year’s higher targets for providing taxpayers with correct responses.

Table 1 compares IRS’ actual 2000 performance levels with its 2001 performance levels and targets. (See app. I for more information on the measures.)

According to IRS officials, the access measures are similar to those commonly used by world-class customer service organizations. They are designed to focus efforts on enhancing taxpayers’ experience in getting access to assistance. For example, the “assistor level of service” measureis intended to show IRS’ effectiveness in providing callers with access to an assistor. The “assistor response level” is to measure the percentage of taxpayers that waited 30 seconds or less to speak with an assistor. The “abandon rate” measure is to show the percentage of taxpayers who hang up while waiting to speak with an assistor, while the “average speed of answer” measure is to show the average number of seconds taxpayers wait to speak to an assistor.

IRS’ accuracy measures are designed to gauge the taxpayers’ experience in getting accurate assistance. The “quality” measures are to show, for a representative sample of calls, the percentage for which assistors followed all procedures, such as properly identifying themselves at the beginning of calls, doing appropriate research on taxpayers’ accounts, and providing accurate information to taxpayers. The new “correct response rate” measures are intended to show the percentage of calls for which IRS assistors provided correct responses to inquiries without taking into account procedural errors that would not affect the accuracy of the information given the taxpayer. IRS began collecting these data in June 2000, so there were no data for the 2000 filing season to compare with 2001.

IRS also measures its performance in answering calls through the use of automation. However, we did not consider this measure—“automated service completion rate”—in assessing IRS’ performance because it assumes that callers who get through to TeleTax are served. The TeleTax system does not have data on how many callers hung up before completing an automated service. Although IRS officials recognize that the measure had limitations, according to them, routing refund status calls to TeleTax allowed IRS to answer about 11.3 million more calls made to its three main toll-free assistance numbers as compared with the 2000 tax filing season. We are continuing to assess many of IRS’ new performance measures, including those used to evaluate telephone assistance.

IRS Missed Some Opportunities to Better Understand Telephone Performance

IRS missed some opportunities to better understand the factors that affected performance and to plan evaluations of the actions it took to improve performance. GPRA and IRS guidance outline the benefits of first gathering and then analyzing data to help managers understand the reasons for performance. To this end, IRS collected performance data on access and conducted some analyses. Even so, IRS missed opportunities to do other analyses of the factors affecting performance including actions to improvement it. Contributing to the missed opportunities was a lack of planning for the evaluation of those actions.

IRS Collected Performance Data on Access and Accuracy

IRS has a variety of systems in place to make data on the access and accuracy of telephone assistance available to managers. Two of these systems are its Joint Operations Center in Atlanta and Centralized Quality Review System (CQRS) in Philadelphia. Managers at call sites also collect data on factors affecting access and accuracy performance.

IRS’ Joint Operations Center in Atlanta manages the activities of the 26 call sites, including monitoring access data and routing calls to the next available assistor anywhere in the country. The Center collects data on various accessibility measures and makes those data available daily to IRS managers through an internal Web site. According to IRS officials, IRS improved the collection of performance data in the 2001 tax filing season. For example, IRS implemented the Enterprise Telephone Database to provide a central call information database. The database was designed to provide IRS analysts and management with the most accurate information for analysis and program decision-making by centralizing data collection and producing a standard set of management reports.

IRS’ CQRS staff in Philadelphia are responsible for collecting data on the accuracy of telephone assistance. CQRS provides call-site officials with daily access through its internal Web site on the results of the sample of calls answered by their sites. It also provides weekly and monthly reports on the quality of sites’ responses to taxpayers’ questions about tax law or about their accounts—two of the accuracy performance measures. These data show the call sites what errors assistors are making so site managers can quickly take action to reduce these errors. IRS officials told us that they made better use of the data in the 2001 filing season. They said that Wage and Investment Division and site officials developed strategies to reduce assistor errors based on CQRS reports.

IRS call sites collect data on factors affecting access and accuracy in various ways. For example, supervisors use real-time data and historical reports available at the call sites on how assistors spent their time, including average handle time—the time an assistor spends talking with the taxpayer, keeping the taxpayer on hold, and finishing the call and indicating readiness to receive another call. Also, local staff monitor calls to provide more detailed information on what errors assistors are making and in what units the errors are being made.

IRS Did Some Analyses to Determine the Factors Affecting Performance

IRS, both at the national and call-site levels, conducted some analyses of performance data intended to determine the factors affecting performance. GPRA and IRS guidance stress that analysis is a key part of understanding performance and identifying improvement options. Analysis of performance data is intended to help managers understand changes in performance, determine root causes, and identify improvement options.

We identified several examples of analytical efforts to determine the factors affecting performance at both the national and call-site levels. In one example with regard to access, IRS officials analyzed data provided by the Joint Operations Center to determine the reasons for the lower-than- expected level of service in the first 3 months of fiscal year 2001. They concluded that declining assistors’ productivity, as measured by average handle time, was the major reason for the decline in access. Officials from the operating divisions and the call sites conducted a series of assessments to determine the underlying reasons for the increase in average handle time. The assessments included focus groups with managers and employees to solicit their views on productivity and monitoring of telephone calls to determine how assistors use the time between calls. The assessments identified three major categories of factors that had negatively affected average handle time: management practices, work processes, and computer systems. According to IRS officials, some management practices adversely affected the level of service because managers did not take actions to improve assistors’ use of time between calls, the primary factor that increased average handle time. IRS officials said that they took immediate corrective actions, such as briefing assistors and supervisors and eliminating unnecessary data entry and taxpayer notification requirements. They also organized teams to further evaluate and resolve the more complicated work process and computer systems issues.

According to IRS officials, IRS improved the analysis of Joint Operations Center performance data in the 2001 tax filing season. For example, analysts studied the factors that affected the demand for live assistance regarding refunds, including the impact of increased electronic filing.

Center analysts also began developing quantitative models of the time taxpayers and IRS spend on telephone questions, with the intent to better match IRS’ resources with taxpayer needs.

Regarding accuracy, CQRS staff analyzed assistor errors and made nationwide and individual site suggestions for addressing the causes of the errors. The suggestions included changes to the assistors’ training and guidance. Also, IRS field directors conducted some analyses to determine the factors that affected access and accuracy. For example, one director said analysis staff at one of her sites was doing a study to determine if the site’s extensive use of faxing negatively affected access. She said she believed the site’s average handle time was longer than others owing to the site’s policy to keep the taxpayer on the line until the accounts issue was resolved, even while the taxpayer faxed documents. Another director said he monitored assistors to determine whether the computer-based research tools assistors used to answer taxpayers’ questions met assistors’ needs.

IRS Missed Some Opportunities to Analyze Performance

Although IRS conducted some analyses of performance data, it missed opportunities to do other analyses at the field level that could have provided a better understanding of the factors affecting telephone assistance performance, including the actions it took to improve performance. Identifying the key factors that most affect performance is important, yet difficult because those factors that can affect telephone access and accuracy are often numerous and interrelated. IRS guidance recognizes that there are a variety of approaches to conducting analyses, such as hypothesis testing, which involves forming a tentative conclusion that is tested using the data. We recognize that some analyses can be costly, but as already noted, GPRA and IRS guidance stress that analysis is a key part of identifying improvement options.

Field directors sometimes reached conclusions about the factors affecting access and accuracy without conducting analyses to test their conclusions. Seven of 10 directors said that the relative inexperience of assistors, caused primarily by higher-than-usual attrition, was a key factor affecting performance in the 2001 tax filing season. They said many experienced seasonal assistors had taken permanent positions in other parts of IRS, and the new hires who replaced them tended to take longer on calls and make more errors. The second most common factor they cited was problems with the computer-based research tools that assistors used to answer taxpayers’ questions. Five of 10 directors cited such problems, including difficulties in using the Servicewide Electronic Research Project to search the Internal Revenue Manual, the assistors’ primary guidance for handling calls regarding taxpayers’ accounts. Some directors said the computer systems were cumbersome and difficult to navigate, causing assistors to take longer on calls and make errors, and some said computer systems often failed and thus hampered assistors’ ability to research questions.

Although directors cited high attrition and computer problems as key factors affecting performance, only two directors identified a specific analysis to support their conclusions. These directors said that focused monitoring was done at their sites that confirmed the limitations of a computer system assistors used to answer taxpayers’ questions. When we asked other directors whether they or their staff had conducted analyses to confirm or refute their conclusions about the factors that affected performance, they acknowledged that they had not.

We identified several opportunities to conduct analyses of performance. One way directors could have analyzed the impact of high attrition on access and accuracy would have been to monitor a sample of calls handled by experienced and inexperienced assistors to compare error rates and average handle time. One director acknowledged that her analysis staff could have done more to learn about how accepting additional calls from businesses affected performance, such as comparing the handle time for business taxpayer calls with individual taxpayer calls. In another example, the program manager for the new Accounts Resolution Guide, a computer-based, step-by-step guide on how to resolve an account-related telephone call, agreed that more could have been done to evaluate the guide’s effectiveness. For example, local managers could have observed and compared assistors who used and did not use the guide. According to IRS officials, additional analyses such as these could have been done at relatively low cost.

Analyzing performance data can be important for several reasons. First, there can be disagreement about which actions improve performance. For example, some directors cited the Accounts Resolution Guide as a reason for the significant improvement in their accounts quality rate. However, another director said that the guide actually had a negative effect on accounts quality, saying that because the guide was new to some assistors, the “learning curve” to become proficient in using the guide caused assistors to make errors.

Second, when multiple factors affect performance, knowing the extent to which each factor has an impact can help managers decide where to focus scarce managerial attention. For example, the solutions for addressing high attrition and computer problems are likely to be different. Understanding the relative importance of high attrition and computer problems could help prioritize improvement actions. In addition, in the case of multiple factors, IRS’ use of performance measures to determine the effect of one factor without controlling for other factors can be misleading. In one case, a field director noted the risks of using average handle time as an indicator of the effectiveness of actions taken to improve the productivity of telephone assistors. The director noted that other factors, such as the complexity of calls handled, could also affect average handle time.

Third, the interrelationship among factors makes it difficult to determine which factors most affect performance. For example, as we reported last year, the quality of guidance assistors use can affect not only the accuracy but also the accessibility of telephone assistance. Although step-by-step guidance on how to respond to questions would likely improve accuracy, it could also cause assistors to take more time answering calls, thereby negatively affecting taxpayers’ access to service.

As we previously reported, conducting systematic analyses of program performance is important for determining the factors affecting performance and identifying opportunities for improvement. IRS guidance states that analysis to understand the underlying factors influencing the performance reflected in the balanced measures is necessary to determine how to improve performance and warns that managers should not “jump to conclusions” about the causes of performance problems. As we said in a report on management reform, “an organization cannot improve performance and customer satisfaction if it does not know what it does that causes current levels of performance and customer satisfaction.” Because the factors affecting telephone performance are numerous and are often interrelated, conducting analyses is essential to determining the factors that have the most effect on performance so that corrective actions can be targeted toward those factors.

We recognize that some analysis can be costly. Consequently, the costs need to be balanced against the benefits. Considering that IRS devotes significant resources (about 10,000 assistors) to telephone assistance, the benefits of analysis—identifying ways to more effectively use resources and improve service—could be substantial.

IRS Missed Opportunities to Plan Evaluations of Actions to Improve Access and Accuracy

IRS missed opportunities to plan evaluations to determine the effectiveness of actions it took to improve the access and accuracy of its telephone assistance. IRS guidance presents a seven-step process designed to guide data collection and analysis to identify ways to improve performance. The last step states that managers should establish a plan that tracks the effectiveness of actions taken to improve performance. Without such a plan, IRS may not collect the data needed to judge the action’s effectiveness. Additionally, planning to collect the data before the improvement action is implemented may be less costly than developing the data and evaluating the action later.

IRS field directors cited several different actions they took to address factors that negatively affected access and accuracy in the prior filing season: Assistor skill gaps (the difference between the skills assistors had and the skills needed by IRS). To address skill gaps, field directors most frequently cited training as the action taken, with all 10 directors referring to training as the primary, and most often only, action taken. Although training was designed at the division level, field directors and managers were responsible for implementing it in the field, such as selecting the trainers and determining which assistors need to be trained.

Errors caused by flaws in the guidance assistors used to respond to taxpayers’ account questions. To address the flaws in assistors’ guidance for answering taxpayer calls, 5 of the 10 field directors cited the implementation and use of computer-based tools to improve guidance, including the Accounts Resolution Guide.

Declining assistor productivity. All 10 field directors said that the primary actions taken to address assistor productivity declines were nationwide managers’ training and employee briefings.

Although field directors said all three of these actions were key to improving telephone assistance performance this year, none of the field directors cited specific evaluations or plans for assessing the effectiveness of these actions. Instead, field directors based their assessment of actions on performance trends, not taking into account the multiple factors or the interrelatedness of factors that can affect a performance measure.

One example of a missed opportunity to plan an evaluation of an improvement action on the national level is the lack of a systematic plan to assess the impact the Accounts Resolution Guide had on access and accuracy. The program manager for the guide said that IRS did not develop such a plan because assistors were not required to use the guide and IRS’ remote monitoring system was unable to determine when the guide was used. The program manager agreed that local managers could have done more to evaluate its effectiveness because they could have observed and compared the results of assistors who did or did not use the guide. Having an evaluation plan when the new Accounts Resolution Guide was distributed to the field would have provided local managers with guidance on the type of data to collect.

Another example of a missed opportunity is the lack of evaluation plans in filing season readiness plans. IRS field directors complete a standard plan each year, adding any items unique to their sites, to ensure that the sites have taken all the necessary steps to provide phone assistance in the tax filing season. Such steps include providing appropriate training and having the equipment and guidance assistors need to respond to taxpayer calls. The readiness plans we reviewed, however, did not include steps to ensure that sites collect and analyze data to evaluate the effectiveness of any improvement actions.

IRS officials noted that since some of the improvement actions were national in scope, field directors would not have been individually responsible for evaluating the effectiveness of the actions. We recognize that evaluations of national improvement actions, such as the Accounts Resolution Guide and managers’ training and employee briefings, to address productivity may involve the higher levels in IRS that are responsible for the action. Accordingly, as noted above, we discussed the Guide with its national program manager and were told that no systematic evaluation was done. We also discussed the actions to address productivity with division-level officials. Similar to the field directors, division officials evaluated the actions by monitoring trends in average handle time and comparing average handle time with previous performance.

Conclusions

IRS made limited progress in the 2001 tax filing season toward its long- term goal of providing world-class customer service. To speed progress toward its long-term goal, IRS managers need to identify the causes for performance, plan strategies to improve performance, and evaluate how well those strategies worked. Unfortunately, IRS sometimes missed opportunities to conduct analysis to help managers understand the reasons for performance and to evaluate actions taken to improve performance. The decision on the type of analysis to be done and who will do it should consider the costs and benefits of the analysis and which organizational levels are most responsible for the factor or improvement action being analyzed. We recognize that some analyses can be costly; however, some of the missed opportunities were low-cost and some involved key factors affecting actions taken to improve performance. In addition, there are costs in not using the 10,000 assistors as effectively as possible. Considering IRS’ limited progress, it cannot afford to miss opportunities without determining the most effective use of its resources to improve performance.

Recommendation for Executive Action

We recommend that the Commissioner ensure that managers follow IRS guidance on analyzing the factors that affect performance and evaluating improvement actions. Specifically, we recommend that (1) field directors be required to develop and follow written plans to collect and analyze data to test their conclusions about the key local factors affecting performance and, when appropriate, evaluate local improvement actions, such as actions involving training; (2) field directors include in filing season readiness plans a step to ensure that site managers have plans to evaluate the effectiveness of any local improvement actions; and (3) program managers and other appropriate national officials be required to develop and follow written plans to evaluate the effectiveness of key national improvement actions, such as the Accounts Resolution Guide.

Agency Comments and Our Evaluation

The Commissioner of Internal Revenue provided written comments on a draft of this report in a December 3, 2001, letter, which is reprinted in appendix II. The Commissioner stated that while the report categorized IRS’ progress toward providing world-class telephone assistance as limited, he is confident that IRS is moving in the right direction. He noted that IRS had initiated a number of strategies to improve telephone assistance and agreed with our recommendation. Specifically, he agreed that IRS needs “better testing, documentation, and analytical activities to determine the factors that affect performance and assess the results of our improvement actions.”

In his comments, the Commissioner noted that the report focuses on accessibility to telephone assistors and stated that IRS also assists taxpayers through automated telephone services and other means, such as IRS’ Internet Web site and walk-in Tax Assistance Centers. As noted in the report, we did not assess IRS’ automated telephone services because IRS’ method of measuring its performance in providing automated services had limitations. The measure assumed that all callers that go through one of its automated systems—TeleTax—were served because the TeleTax system does not have data on how many taxpayers hung up before completing an automated service. Other taxpayer services, such as walk-in assistance, are to be addressed in our upcoming report on various aspects of the 2001 tax filing season.

As agreed with your staff, unless you publicly release its contents earlier, we will make no further distribution of this report until 30 days after its issue date. At that time, we will send copies of this report to the Chairmen and Ranking Minority Members of the Senate Committee on Finance and the House Committee on Ways and Means and the Ranking Minority Member of the Subcommittee. We will also send copies to the Secretary of the Treasury; the Commissioner of Internal Revenue; the Director, Office of Management and Budget; and other interested parties. We will make copies available to others on request.

If you have any questions or would like additional information, please call me at (202) 512-9110 or Carl Harris at (404) 679-1900. Key contributors to this report are Ronald W. Jones and Ronald J. Heisterkamp.

Appendix I: Descriptions and Formulas for Measures of IRS Telephone Assistance Accessibility and Accuracy

Appendix II: Comments From the Internal Revenue Service