Tax Administration: IRS Can Improve Its Productivity Measures by Using Alternative Methods
Released: 2005-07-19T00:00:00

Letter

Background

Productivity is defined as the efficiency with which inputs are used to produce outputs. It is measured as the ratio of outputs to inputs. Productivity and cost are inversely related—as productivity increases, average costs decrease. Consequently, information about productivity can inform budget debates as a factor that explains the level or changes in the cost of carrying out different types of activities. Improvements in productivity either allow more of an activity to be carried out at the same cost or the same level of activity to be carried out at a lower cost.

IRS currently relies on output-to-input ratios such as cases closed per FTE to measure productivity and productivity indexes. A productivity change is measured as an index which compares productivity in a given year to productivity in a base year. Measuring productivity trends requires choosing both output and input measures, and the methods for calculating productivity indexes.

In the past we have reported on declining enforcement trends, finding in 2002 that there were large and pervasive declines in six of eight major compliance and collection programs we reviewed. In addition to reporting these declines, we reported on the large and growing gap between collection workload and collection work completed and the resultant increase in the number of cases where IRS has deferred collection action on delinquent accounts. In 2003, we reported on the declining percentage of individual income tax returns that IRS was able to examine or audit each year, with this rate falling from 0.92 percent to 0.57 percent between 1993 and 2002. Since 2000, the audit rate has increased slightly but not returned to previous levels. IRS conducts two types of audits: field exams that involve complex tax issues and usually face-to-face contact with the taxpayer, and, correspondence exams that cover simpler issues and are done through the mail. We also reported on enforcement productivity measured by cases closed per FTE employee, finding that IRS’s telephone and field collection productivity declined by about 25 percent from 1996 through 2001 and productivity in IRS’s three exam programs—individual, corporate, and other audit—declined by 31 to 48 percent.

In January 2004 we reported on the extent to which IRS’s Small Business and Self-Employed (SB/SE) division followed steps consistent with both GAO guidance and the experience of private sector and government organizations when planning its enforcement process improvement projects. We reported on how the use of a framework would increase the likelihood that projects target the right processes for improvement and lead to the most fruitful improvements. In that report, we also reported that more complete productivity data—input and output measures adjusted for the complexity and quality of cases worked—would give SB/SE managers a more informed basis for decisions on how to identify processes that need improvement, improve processes, and assess the success of process improvement efforts. This report elaborates on that recommendation, providing more information about the challenges of obtaining complete productivity data.

Improving productivity by changing processes is a strategy SB/SE is using to address these declining trends. However, the data available to SB/SE managers to assess the productivity of their enforcement activities, identify processes that need improvement, and assess the success of their process improvement efforts are only partially adjusted for complexity and quality of cases worked. This problem of adjusting for quality and complexity is not unique to SB/SE process improvement projects—the data available to process improvement project managers are the same data used throughout SB/SE to measure productivity and otherwise manage enforcement operations.

Measuring Productivity at IRS Is Challenging because Measuring the Output of Services Is Difficult

Because IRS provides services, such as providing information to taxpayers and enforcing the tax laws, that are intangible and complex, measuring output—and therefore productivity—is challenging. Like other providers of intangible and complex services, IRS has a choice of measuring activities or the results of its services. Generally, information about results is preferred, but measuring results is often difficult. In the absence of direct measures of results, activities that are closely related to the results of the service can be used as proxies.

Measuring productivity in services is difficult. Unlike manufacturing, which lends itself to objective measurement because output can be measured in terms of units produced, services, which involve changes in the condition of people receiving the service, often have intangible characteristics. Thus, the output of an assembly line is easier to measure than the output of a teacher, doctor, or lawyer. Services may also be complex bundles of individual services, making it difficult to specify the different elements of the service. For example, financial services provide a range of individual services, such as financial advice, accounts management and processing, and facilitating financial transactions.

IRS provides a service. IRS’s mission, to help taxpayers understand and meet their tax responsibilities and to apply the tax law with integrity and fairness, requires IRS to provide a variety of services ranging from collecting taxes to taxpayer education. IRS, like other service providers, could measure output in terms of its results—its impact on taxpayers—or in terms of activities. The results of IRS’s service are the impacts on the condition or behavior of taxpayers. These taxpayer conditions or behaviors include their compliance with the tax laws, their compliance burden (the time and money cost of complying with tax laws), and their perception of how fairly taxpayers are treated. IRS’s activities are what IRS does to achieve those results. These activities include phone calls answered, notices sent to taxpayers, and exams conducted.

Generally, information about results is preferred, but measuring such results is often difficult. In the case of the public sector, this preference is reflected in GPRA, which requires that federal agencies measure performance, whenever possible, in terms of results or outcomes for people receiving the agencies’ services. However, results such as compliance and fairness have intangible characteristics that are difficult to measure. In addition, results are produced in complicated and interrelated ways. For example, a transaction or activity may affect a number of results: IRS’s exams may affect taxpayers’ compliance, compliance burden, and perceptions of the fairness of the tax system. In addition, a result may be influenced by a number of transactions or activities: A taxpayer’s compliance may be influenced by all IRS exams (through their effect on the probability of an exam) as well as by other IRS activities such as taxpayer assistance services.

IRS’s activities are easier to measure than results but still present challenges. Activities are easier to measure because they are often service transactions such as exams, levies issued, or calls answered that can be easily counted. However, unlike measures of results, more informative measurement of activities requires that they be adjusted for quality and complexity, as we noted in our report on IRS’s enforcement and improvement projects. A productivity measure based on activities such as cases closed per FTE may be misleading if such adjustments are not made. For example, an increase in exam cases closed per FTE would not indicate an increase in true productivity if the increase occurred because FTEs were shifted to less complex cases or the examiner allowed the quality of the case review to decline to close cases more quickly.

Activities-based productivity measures can provide IRS with useful information on the efficiency of IRS operations. Measuring output, and therefore productivity, in terms of activities provides IRS with measures of how efficiently it is using resources to perform specific functions or transactions. However, activities do not constitute—and should not be mistaken for—measures of IRS’s productivity in terms of ultimate results.

While the productivity measures we have examined are based on activities, IRS has efforts under way to measure results such as compliance and compliance burden. Recently, we reported on IRS’s National Research Program (NRP) to measure voluntary compliance and efforts to measure compliance burden. As we mentioned previously, measuring these results is difficult. For some results, such as compliance, measurement is also costly and intrusive because taxpayers must be contacted and questioned in detail. Despite these difficulties, IRS can improve its productivity measurement by continuing its efforts to get measures of results. These efforts would give Congress and the general public a better idea of what is being achieved by the resources invested in IRS.

In the absence of direct measures of results, activities that are closely related to the results of the service are used as proxies. The value of these proxies depends on the extent to which they are correlated with results. By carefully choosing these measures, IRS may gain some information about the effect of its activities on ultimate results. Because activities may affect a number of results and a single result may be affected by a number of activities, a single activity likely will not be a sufficient proxy for the results of the service. Therefore, a variety of activities would likely be necessary as proxies for the results of the service.

Both types of output measures, those that reflect the results of IRS’s service and those that use activities to measure internal efficiency, should be accurate and consistent over time. In addition, both output measures should be reliably linked to inputs. Linking the results of IRS’s service to inputs may be difficult because of outside factors that may also affect measured results. For example, an increase in compliance could result both from IRS actions such as exams and from changes in tax laws. Another challenge is that IRS currently has difficulties linking inputs to activities, as we note in a previous report, where we reported IRS’s lack of a cost accounting system. In particular, IRS only recently implemented a cost accounting system, and has not yet determined the full range of its cost information needs. Table 1 summarizes some of the key differences between activities and results measures. Table 1 also indicates some general criteria that apply to both types of measures.

Because inputs are more easily measured and identifiable than outputs, measuring them is more straightforward. IRS, as a government agency, may be able more often to use labor costs or hours as a single input in its productivity measures because it relies heavily on labor. However, it may be particularly important for IRS to use a multifactor measure that includes capital along with labor during periods of modernization that involve increased or high levels of capital investment. As with outputs, inputs should be measured accurately and consistently over time. Measuring inputs consistently over time may require adjusting for changes in the quality of the labor, which has been done using proxies such as education level or years of experience. Also, as mentioned previously, inputs should be reliably linked to outputs.

However Output Is Measured, IRS Can Improve Its Current Productivity Measures by Using Alternative Methods

The appropriate method for calculating productivity depends on the purpose for which the productivity measure is being calculated. The alternative methods that can be used for calculating productivity range from computing single ratios—exams closed per FTE—to using complex statistical methods to form composite indexes that combine multiple indicators of outputs and inputs. While single ratios may be adequate for certain purposes, the composite indexes based on statistical methods may be more useful because they provide information about the sources of productivity change.

Comparing the ratios of outputs to inputs at different points in time defines a productivity index that measures the percentage increase or decrease in productivity over time. The ratios that form the index may be single, comparing a single output to a single input or composite, where multiple outputs and inputs are compared. The single ratios may be useful for evaluating the efficiency of a single noncomplex activity. Composite indexes can measure the productivity of more complicated activities, controlling for complexity and quality. Composite indexes can also be used to measure productivity of resources across an entire organization, where many different activities are being performed.

One method of producing composite indexes is to use weights to combine such disparate activities as telephone calls answered and exams closed. One common weighting method, used by the Bureau of Labor Statistics (BLS), is a labor weight. Weighting outputs by their share of labor in a baseline period controls for how resources are allocated between different types of outputs. If the productivity of two activities is unchanged but resources are reallocated between the activities, the composite measure of productivity would change unless these weights are employed. For example, if IRS reallocates exam resources so that it does more simple exams and fewer complex exams, the number of total exams might increase. Consequently, a single productivity ratio comparing total exams to inputs would show an increase. Labor weighting deals with this issue. The weights allow any gains from resource reallocation to be distinguished from gains in the productivity of the underlying activities. When types of activities can be distinguished by their quality of complexity, labor weighting can also be used to control for quality and complexity differences when resources are shifted between types of outputs.

More complicated statistical methods can be used for calculating composite indexes that allow for greater flexibility in how weights are chosen to combine different outputs and for a wider range of output measures that include both qualitative and quantitative outputs. Data Envelopment Analysis (DEA), which has been widely used to measure the productivity of private industries and public sector services, is an example of such methods DEA estimates an efficiency score for each producing unit, such as the firms in an industry or the schools in a school district, or for IRS, the separately managed areas and territories composing its business units. DEA estimates the relative efficiency of each producing unit by identifying those units with the best practice—those making the most efficient use of inputs, under current technology, to produce outputs—and measuring how far other units are from this best practice combination of inputs used to produce outputs. DEA estimates provide managers with information on how efficient they are relative to other units and the costs of making individual units more efficient.

These efficiency scores are used to form a composite productivity index called a Malmquist index. An advantage of the Malmquist index is that IRS managers can restrict the weights to adjust for managerial or congressional preferences to investigate the effect on productivity of a shift, for example, from an organization that emphasizes enforcement to one that emphasizes service. IRS can also include many different types of outputs and inputs, control for complexity and quality, and isolate the effects of certain historical changes, such as the IRS Restructuring and Reform Act of 1998 (RRA98).

Another advantage of the Malmquist index is that productivity changes can be separated into their components, such as efficiency and technology changes. In this context, efficiency can be measured holding technology constant, and technology can be measured holding efficiency constant. Holding technology constant, IRS might improve productivity by improving the management of its existing resources. On the other hand, technology changes might improve productivity even if the management of resources has not changed. Thus, the productivity change of a given IRS unit is determined by both changes in its efficiency relative to the current best- practice IRS units and changes in the best practices or technology.

Illustrations of Alternative Methods of Measuring Productivity

Currently available IRS data can be used to produce productivity indexes that control for complexity and quality. The examples that follow focus on productivity indexes that use exams closed as outputs and FTEs as inputs. The data on examinations cover individual returns across IRS and IRS’s LMSB division. For both individuals and LMSB, the complexity and quality of exams can vary over time. For example, the proportion of exams that are correspondence versus field, business versus nonbusiness, and EIC versus non-EIC can vary over time. As already discussed, failing to take account of such variation can give a misleading picture of productivity change.

While these examples do not encompass all the methods, data, and adjustments that may be used, they illustrate the benefits of the additional analysis that IRS can perform using current data. In addition, as we pointed out in our 2004 report, IRS can improve its productivity measurement by investing in better data, taking into account the costs and benefits of doing so. These better data include measures of complexity, improved measures of quality, and additional measures of output.

Figures 1 through 4 illustrate, using currently available data between fiscal years 1997 and 2004, the difference between weighted indexes that make an adjustment for complexity and unweighted indexes that make no adjustments. In the illustrations, a labor-weighted composite index, which can control for complexity, is contrasted with a single unweighted index to show how the simpler method may be misleading. (See app. I for a fuller description of the labor-weighted index.) In each case, complexity is proxied by type of exam. Although the data were limited (for example, the measure of complexity was crude), the illustrations show that making the adjustments that are possible provides a different picture of productivity than would otherwise be available.

The advantage of weighted indexes is that they allow changes in the mix of exams to be separated from changes in the productivity of performing those exams. In the examples that follow, an unweighted measure could be picking up two effects. One effect is the change in the number of exams that an auditor can complete if the complexity or quality of the exam changes. The second effect is the change in the number of exams an auditor can complete if the time an auditor requires to complete an exam changes, holding the quality and complexity of exams constant. By isolating the latter effect, the weighted index more closely measures productivity, or the efficiency with which the auditor is working the exams.

For individual exams, the comparison of productivity indexes shows that the unweighted index understates the decline in productivity. As figure 1 shows, between fiscal years 1997 and 2001, the unweighted productivity index declined by 32 percent while the weighted index declined by 53 percent. The difference is due largely to the increase in EIC exams during the period. Over the period between fiscal years 1997 and 2001, exams were declining. However, the mix of exams was changing, with increases in the number of EIC exams. EIC exams are disproportionately correspondence exams, and IRS can do these exams faster than field exams. IRS shifted to “easier” exams, and that shift caused the unweighted index to give an incomplete picture of productivity. The shift masked the larger productivity decline shown by the weighted index.

Figure 2 provides additional evidence to support the conclusion that the shift to more EIC exams is the reason for the difference in productivity shown in figure 1. Between fiscal years 1997 and 2001, the weighted and unweighted indexes track each other very closely when the EIC exams are removed. Both show a decline in productivity of about 50 percent over this period. The available data were not sufficient to control for other factors that may have influenced exam productivity. For example, RRA98 imposed additional requirements on IRS’s auditors, such as certifications that they had verified that past taxes were due.

Figure 3 compares unweighted and weighted productivity indexes for exams done in LMSB division. As figure 3 shows, between fiscal years 2002 and 2004, the unweighted productivity index increased by 4 percent, while the weighted index increased by 16 percent. This difference appears largely due to the individual exams and small corporate exams done in LMSB. Over the period, total exams were declining but the mix of exams was changing. LMSB was shifting away from less labor-intensive individual returns and small corporation returns to more complex business industry and coordinated industry return exams. This shift caused the unweighted index to give an incomplete picture of productivity. Here, the shift masked the larger productivity increase as shown by the weighted index.

Figure 4 provides additional evidence to support the conclusion that the shift away from individual and small corporate exams is the reason for the difference in productivity shown in figure 3. Between fiscal years 2002 and 2004, when individual and corporate exams are excluded, the two indexes track more closely, with the unweighted index increasing by 15 percent and the weighted index by 17 percent.

There is evidence that adjusting for quality would show that LMSB’s productivity increased more than is apparent in figures 3 and 4 for the years 2002 to 2004. Average quality scores available for selected types of LMSB exams show quality increasing over the 2-year period. Adjusting for this increase in quality, in addition to adjusting for complexity, would show a productivity increase for these types of exams of 28 percent over the period.

While labor-weighted and other more sophisticated productivity indexes can provide a more complete picture of productivity changes, they do not identify the causes of the changes. These productivity indexes would be the starting point for any analysis to determine the causes of productivity changes.

Another example of the advantages of weighted productivity indexes is provided by IRS. As noted earlier, IRS has developed a weighted submission processing productivity measure. The measure adjusts for differences in the complexity of processing various types of tax returns. In an internal analysis, IRS showed how productivity comparisons over time and across the 10 processing centers depended on whether or not the measure was adjusted for complexity. For example, the ranking of the processing centers in terms of productivity changed when the measure was adjusted for the complexity of the returns being processed.

The more sophisticated methods for measuring productivity can provide IRS and Congress with better information about IRS’s performance. By controlling for complexity and quality, IRS managers would have more complete information about the true productivity of activities, such as exams, that can differ in these dimensions. In addition, the weighted measures can be used to measure productivity for the organization, where many different activities are being performed. More complete information about the productivity of IRS resources should be useful to both IRS managers and Congress. More complete productivity measures would provide better information about the effectiveness of IRS resources, IRS’s budget needs, and IRS’s efforts to improve efficiency.

Although there are examples, such as the submission processing productivity measures, of IRS using weighted measures of productivity, IRS officials said they generally use single ratios as measures of productivity. That is consistent with our 2004 report on IRS’s enforcement improvement projects, where we reported on SB/SE’s lack of productivity measures that adjust for complexity and quality.

While there would be start-up costs associated with any new methodology, the long-term costs to IRS for developing more sophisticated measures of productivity may be modest. The examples so far in this section demonstrate the feasibility of developing weighted productivity indexes using existing data. Relying on existing data avoids the cost of having to collect new data. The fact that IRS already has some experience implementing weighted productivity measures could reduce the cost of introducing more such measures.

As we stated previously, IRS could also improve its productivity measurement by getting better data on quality and complexity. These improved data could be integrated with the methods for calculating productivity illustrated in this report to further improve IRS’s productivity measurement. However, as we acknowledged in our prior report, collecting additional data on quality and complexity may require long-term planning and an investment of additional resources. Any such investment, we noted, must take account of the costs and benefits of acquiring the data.

Conclusion

Using more sophisticated methods, such as those summarized in this report, for tracking productivity could produce a much richer picture of how IRS manages its resources. This is important not only because of the size of IRS—it will spend about $11 billion in 2005 and employ about 100,000 FTEs—but also because we are entering an era of tight budgets. A more sophisticated understanding of the level of productivity at IRS and the reasons for productivity change would better position IRS managers to make decisions about how to effectively manage their resources. Such information would also better enable Congress and the public to assess the performance of IRS.

As we illustrate, more can be done to measure IRS’s productivity using current data. However, another advantage of using more sophisticated methods to track productivity is that the methods will highlight the value of better data. Better information about the quality and complexity of IRS’s activities would enable the methods illustrated in this report to provide even richer information about IRS’s overall productivity.

Recommendations for Executive Action

We recommend that the Commissioner of Internal Revenue put in place a plan for introducing wider use of alternative methods of measuring productivity, such as those illustrated in this report, taking account of the costs of implementing the new methods.

Agency Comments and Our Evaluation

The Commissioner of Internal Revenue provided written comments on a draft for this report in a June 23, 2005, letter. The Commissioner agreed with our recommendation to work on introducing wider use of alternative measure of productivity. Although expressing some caution, he has asked his Deputy Commissioner for Services and Enforcement to work with IRS’s Research, Analysis, and Statistics office to assess the possible use of alternative methods of measuring productivity. The Commissioner recognized that a richer understanding of organizational performance is crucial for effective program delivery.

As agreed with your office, unless you publicly release its contents earlier we plan no further distribution of this report until 30 days from the date of this letter. At that time, we will send copies to interested congressional committees, the Secetary of the Treasury, the Commissioner of Internal Revenue, and other interested parties. We will also make copies available to others on request.

If you or your staff have any questions, please contact me at (202) 512-9110. I can also be reached by e-mail at whitej@gao.gov. Key contributors to this assignment were Kevin Daly, Assistant Director, and Jennifer Gravelle.

Methods for Calculating Productivity Indexes

Productivity Indexes

Methods for calculating productivity range from computing single ratios to using statistical methods. In its simplest form, a productivity index is the change in the productivity ratio over time relative to a chosen year. However, this type of productivity index allows for only a single output and a single input. To account for more than one output, the outputs must be combined to produce a productivity index.

One method is to weight the outputs by their share of inputs used in the chosen base year. In a case where only labor input is used, following this method provides a labor-weighted output index, which, when divided by the input index, produces the labor-weighted productivity index. The use of the share of labor used in each output effectively controls for the allocation of labor across the outputs over time. For example, if productivity in producing two outputs remained fixed over time, a single productivity index may show changes in productivity if resources are reallocated to produce more of one of the outputs.

The Bureau of Labor Statistics (BLS) has also used labor-weighted indexes. BLS published, under the Federal Productivity Measurement Program, data on labor productivity in the federal government for more than two decades (1967-94). Due to budgetary constraints, the program is now terminated. BLS’s measures used the “final outputs” of a federal program, which correspond generally to what we have called intermediate outputs in this report, as opposed to the outcomes or results of the program. BLS used labor weights because of their availability and their close link to cost weights. In particular, as with the labor weights in our illustrations, BLS used base year labor weights and updated the weights every 5 years. It relied only on labor and labor compensation, and acknowledges that the indexes did not reflect changes in the quality of labor. BLS measured productivity for a number of federal programs, ranging from social and information services to corrections. However, BLS did not produce productivity measures for IRS.

In addition to weighted productivity indexes, there are a number of composite productivity indexes designed to include all the inputs and outputs involved in production. This group of indexes is called Total Factor Productivity (TFP) indexes. They are called total because they include all the inputs and outputs, as opposed to Partial Factor Productivity indexes, which relate only one input to one output. Many of the main TFP indexes, including Tornqvist, Fisher, Divisia, and Paache, require reliable estimates of input and output prices, data not available for industries in the public sector. Therefore we use the Malmquist index, which does not require that data.

Malmquist indexes are TFP indexes based on changes in the distance from the production frontier, or distance functions. These distance functions are estimated using Data Envelopment Analysis (DEA). Productivity change is represented by the ratio of two different period distance functions. The Malmquist index is the geometric average of these productivity changes (evaluated at the two different periods). This index can be further decomposed into efficiency and technology changes. From the decomposition of the Malmquist index, productivity change can be shown to equal the efficiency change times the technology change. (xt+1,yt+1)/D(x,y)]*[Dt+1(xt+1,yt+1)/ Dt+1(x,y)]}^1/2, where x, xt+1 denote the vector of inputs at time t and t+1, and y, and yt+1 denote the vector of outputs in time t and t+1 and D and Dt+1 are distance functions relative to the technology in time t and t+1. (x,y)]*[Dt+1(xt+1,yt+1)/ Dt+1(x,y)]}^1/2 = [Dt+1(xt+1, yt+1)/ D(x,y)]*{[ D(xt+1,yt+1)/ Dt+1(xt+1,yt+1)]*[D(x,y)/ Dt+1(x,y)]}^1/2=E*T, the efficiency change, E, times the technology change, T. fraction in the first, indicates a movement away from one over time and thus declining productivity. Thus, a productivity change less than one indicates declining productivity and therefore an efficiency change less than one also indicates declining efficiency.

Alternatively, if the efficiency change was one, then the productivity change equals the technology change. Following previous analysis, a productivity change less than one indicates declining productivity. Therefore, a technology change less than one indicates an inward shift of the production frontier. If the technology change is less than one, it must be that the distance function in the first period is less than the distance function in the next period. Thus, the distance in the first period is farther away from one than is the distance in the next period, and the distance from the frontier decreased from the first period to the second period. Since the output and input bundles did not change, the frontier must shift in to produce the decrease in distance.

The Internal Revenue Service (IRS) can follow this method to generate indexes for the areas and territories and then focus on the average for an estimate of overall IRS productivity.

Estimation of Distance Functions

,y)= [max { φ | (x, φy) ∈T}]-1 and φ* = (D(x,y))-1, with φ* >1 and D(x,y)< 1, where φ denotes the value to scale output. therefore, inefficient relative to firms with a scalar value of one. Thus, output distance functions are less than one. IRS can use this method, treating territories and areas as firms. The weights used in the linear program are designed to make each firm look its best; they represent best case scenarios.

While DEA is a nonparametric method, there is also a parametric method available called stochastic frontier analysis. Stochastic frontier analysis (regression) uses a regression model to estimate cost or production efficiency. After running the regression of performance and input data, the frontier is found by decomposing the residuals into a stochastic (statistical noise) part and a systematic portion attributed to some form of inefficiency. Stochastic frontier analysis thus requires specifying the distributional form of the errors and the functional form of the cost (or production) function. Its merits include a specific treatment of noise. While DEA’s use of nonparametric methods eliminates the need to specify functional forms, one drawback is a susceptibility to outliers.