Tax Administration: IRS Can Improve Its Productivity Measures by Using Alternative Methods
Published: 2005-07-11T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In the past, the Internal Revenue Service (IRS) has experienced declines in enforcement productivity as measured by cases closed per Full Time Equivalent. Increasing enforcement productivity through a variety of enforcement improvement projects is one strategy being pursued by IRS. Evaluating the benefits of different projects requires good measures of productivity. In addition, IRS's ability to correctly measure its productivity has important budget implications. GAO was asked to illustrate available methods to better measure productivity at IRS. Specifically, our objectives were to (1) describe challenges that IRS faces when measuring productivity, (2) describe alternative methods that IRS can use to improve its productivity measures, and (3) assess the feasibility of using these alternative methods by illustrating their use with existing IRS data.
section_title: What GAO Found
paragraphs: Measuring IRS's productivity, the efficiency with which inputs are used to produce outputs, is challenging. IRS's output could be measured in terms of impact on taxpayers or the activities it performs. IRS's impacts on taxpayers, such as compliance and perceptions of fairness, are intangible and costly to measure. IRS's activities, such as exams or audits conducted, are easier to count but must be adjusted for complexity and quality. An increase in exams closed per employee would not indicate an increase in productivity if IRS had shifted to less complex exams or if quality declined. IRS can improve its productivity measures by using a variety of methods for calculating productivity that adjust for complexity and quality. These methods range from ratios using a single output and input to methods that combine multiple outputs and inputs into composite indexes. Which method is appropriate depends on the purpose for which the productivity measure is being calculated. For example, a single ratio may be useful for examining the productivity of a single simple activity, while composite indexes can be used to measure the productivity of resources across an entire organization, where many different activities are being performed. Two examples show that existing data, even though they have limitations, can be used to produce a more complete picture of productivity. For individual exams, composite indexes controlling for exam complexity show a larger productivity decline than the single ratio method. On the other hand, for exams performed in the Large and Mid-Size Business (LSMB) division, the single ratio understates the productivity increase shown, after again controlling for complexity. By using alternative methods for measuring productivity, managers would be better able to isolate sources of productivity change and manage resources more effectively. More complete productivity measures would provide better information about IRS effectiveness, budget needs, and efforts to improve efficiency.

section_title: Letter
paragraphs: In the past, we have reported on declines in the Internal Revenue Service’s  (IRS) enforcement programs, including declining exam and collection  efforts.  One factor we have cited as contributing to these declines is  decreased enforcement productivity as measured by cases closed per staff  time.  Increasing enforcement productivity through a variety of  enforcement improvement projects is one strategy being pursued by IRS  that could help reverse the declines.  However, evaluating the benefits of  these different projects requires good measures of productivity.  IRS’s  ability to correctly measure its productivity has important budget  implications.  Productivity declines may indicate that IRS is not using its  resources as efficiently as possible.  Increasing the productivity of existing  resources might lessen, to some extent, the need for budget increases.
Productivity is measured as a ratio of outputs to inputs.  In a January 2004  report on IRS’s enforcement improvement projects, we recommended that  IRS invest in enforcement productivity data that better adjust for  complexity and quality, taking into consideration the costs and benefits of  doing so.  More complete productivity data—data that adjust for  complexity and quality—would give managers a clearer picture of how  effectively resources are being used.  In addition, Congress would have  better information about IRS’s performance and budget needs. To better  understand productivity measurement at IRS, you asked us to illustrate  methods available to better measure it.  Specifically, our objectives were to  (1) describe challenges that IRS faces when measuring productivity, (2) describe alternative methods that IRS can use to improve its  productivity measures, and (3) assess the feasibility of using these  alternative methods by illustrating their use with existing IRS data.
In the context of the productivity literature, output is a general concept  representing what is produced.  However, in the performance measurement  literature, the term “output,” as defined in the Government Performance  and Results Act of 1993 (GPRA) is limited to an activity or effort, while an  outcome is the result of a program activity.  Activities are typically easily  measured, such as transactions completed.  Results such as the difference  an activity makes in the economy or people’s lives are usually less tangible.   In this report, we use the general concept of “output” to define productivity  but then distinguish between outputs that are results and those that are  activities.
To describe the challenges IRS faces when measuring productivity and  alternative methods IRS can use to improve its productivity measures, we  reviewed the literature on the methods used to measure productivity in the  public and private sectors.  We also consulted IRS officials and reviewed  IRS documentation on IRS’s methods for measuring productivity.
To assess the feasibility of using these alternative methods by illustrating  their use with existing IRS data, we used currently available IRS data to  calculate alternative exam, or audit, productivity measures.  These  methods included calculating unweighted productivity indexes and  weighted productivity indexes.  We compared these indexes to show how  implementing different methods can provide IRS with better measures of  productivity and better ways to identify the causes of productivity change.
For this report existing IRS examination data were used to illustrate the  feasibility of using alternative methods of productivity.  The data are from  IRS’s Tax Compliance Report and Automated Inventory Management  System.  In prior reports we recognized that IRS’s existing examination  data have limitations.  For example, direct measures of complexity were  not available.  We use type of exam as a proxy for complexity. We have also  recommended that IRS improve its input data by implementing a cost  accounting system.  While there are reliability issues related to the data, we  are using the available IRS data for illustrative purposes, and we will not be  representing these illustrations as complete measures of IRS productivity.   Therefore, we determined that the information contained in IRS’s Tax  Compliance Report and Automated Inventory Management System  databases were sufficiently reliable for illustrative purposes.
We initiated our review in September 2003 but conducted most of our  review from August 2004 through April 2005 in accordance with generally  accepted government auditing standards.
subsections: section_title: Background
paragraphs: Productivity is defined as the efficiency with which inputs are used to  produce outputs.  It is measured as the ratio of outputs to inputs.   Productivity and cost are inversely related—as productivity increases,  average costs decrease.  Consequently, information about productivity can  inform budget debates as a factor that explains the level or changes in the  cost of carrying out different types of activities.  Improvements in  productivity either allow more of an activity to be carried out at the same  cost or the same level of activity to be carried out at a lower cost.
IRS currently relies on output-to-input ratios such as cases closed per FTE  to measure productivity and productivity indexes.  A productivity change is  measured as an index which compares productivity in a given year to  productivity in a base year.  Measuring productivity trends requires  choosing both output and input measures, and the methods for calculating  productivity indexes.
In the past we have reported on declining enforcement trends, finding in  2002 that there were large and pervasive declines in six of eight major  compliance and collection programs we reviewed. In addition to reporting  these declines, we reported on the large and growing gap between  collection workload and collection work completed and the resultant  increase in the number of cases where IRS has deferred collection action on delinquent accounts.  In 2003, we reported on the declining percentage  of individual income tax returns that IRS was able to examine or audit each  year, with this rate falling from 0.92 percent to 0.57 percent between 1993  and 2002.  Since 2000, the audit rate has increased slightly but not returned  to previous levels.  IRS conducts two types of audits: field exams that  involve complex tax issues and usually face-to-face contact with the  taxpayer, and, correspondence exams that cover simpler issues and are  done through the mail.  We also reported on enforcement productivity  measured by cases closed per FTE employee, finding that IRS’s telephone  and field collection productivity declined by about 25 percent from 1996  through 2001 and productivity in IRS’s three exam programs—individual,  corporate, and other audit—declined by 31 to 48 percent.
In January 2004 we reported on the extent to which IRS’s Small Business  and Self-Employed (SB/SE) division followed steps consistent with both  GAO guidance and the experience of private sector and government  organizations when planning its enforcement process improvement  projects.  We reported on how the use of a framework would increase the  likelihood that projects target the right processes for improvement and  lead to the most fruitful improvements.  In that report, we also reported  that more complete productivity data—input and output measures adjusted  for the complexity and quality of cases worked—would give SB/SE  managers a more informed basis for decisions on how to identify processes  that need improvement, improve processes, and assess the success of  process improvement efforts.  This report elaborates on that  recommendation, providing more information about the challenges of  obtaining complete productivity data.
Improving productivity by changing processes is a strategy SB/SE is using  to address these declining trends.  However, the data available to SB/SE  managers to assess the productivity of their enforcement activities, identify  processes that need improvement, and assess the success of their process  improvement efforts are only partially adjusted for complexity and quality  of cases worked.  This problem of adjusting for quality and complexity is  not unique to SB/SE process improvement projects—the data available to  process improvement project managers are the same data used throughout  SB/SE to measure productivity and otherwise manage enforcement  operations.
section_title: Measuring Productivity at IRS Is Challenging because Measuring the Output of Services Is Difficult
paragraphs: Because IRS provides services, such as providing information to taxpayers   and enforcing the tax laws, that are intangible and complex, measuring  output—and therefore productivity—is challenging.  Like other providers  of intangible and complex services, IRS has a choice of measuring activities  or the results of its services. Generally, information about results is  preferred, but measuring results is often difficult. In the absence of direct  measures of results, activities that are closely related to the results of the  service can be used as proxies.
Measuring productivity in services is difficult.  Unlike manufacturing,  which lends itself to objective measurement because output can be  measured in terms of units produced, services, which involve changes in  the condition of people receiving the service, often have intangible  characteristics.  Thus, the output of an assembly line is easier to measure  than the output of a teacher, doctor, or lawyer.  Services may also be  complex bundles of individual services, making it difficult to specify the  different elements of the service.  For example, financial services provide a  range of individual services, such as financial advice, accounts  management and processing, and facilitating financial transactions.
IRS provides a service.  IRS’s mission, to help taxpayers understand and  meet their tax responsibilities and to apply the tax law with integrity and  fairness, requires IRS to provide a variety of services ranging from  collecting taxes to taxpayer education.   IRS, like other service providers,  could measure output in terms of its results—its impact on taxpayers—or  in terms of activities.   The results of IRS’s service are the impacts on the  condition or behavior of taxpayers.  These taxpayer conditions or  behaviors include their compliance with the tax laws, their compliance  burden (the time and money cost of complying with tax laws), and their  perception of how fairly taxpayers are treated.  IRS’s activities are what IRS  does to achieve those results.  These activities include phone calls  answered, notices sent to taxpayers, and exams conducted.
Generally, information about results is preferred, but measuring such  results is often difficult.  In the case of the public sector, this preference is  reflected in GPRA, which requires that federal agencies measure  performance, whenever possible, in terms of results or outcomes for  people receiving the agencies’ services. However, results such as  compliance and fairness have intangible characteristics that are difficult to  measure.  In addition, results are produced in complicated and interrelated  ways.  For example, a transaction or activity may affect a number of  results: IRS’s exams may affect taxpayers’ compliance, compliance burden,  and perceptions of the fairness of the tax system.  In addition, a result may  be influenced by a number of transactions or activities: A taxpayer’s  compliance may be influenced by all IRS exams (through their effect on the  probability of an exam) as well as by other IRS activities such as taxpayer  assistance services.
IRS’s activities are easier to measure than results but still present  challenges.  Activities are easier to measure because they are often service  transactions such as exams, levies issued, or calls answered that can be  easily counted.  However, unlike measures of results, more informative  measurement of activities requires that they be adjusted for quality and  complexity, as we noted in our report on IRS’s enforcement and  improvement projects.  A productivity measure based on activities such as  cases closed per FTE may be misleading if such adjustments are not made.  For example, an increase in exam cases closed per FTE would not indicate  an increase in true productivity if the increase occurred because FTEs were  shifted to less complex cases or the examiner allowed the quality of the  case review to decline to close cases more quickly.
Activities-based productivity measures can provide IRS with useful  information on the efficiency of IRS operations. Measuring output, and  therefore productivity, in terms of activities provides IRS with measures of  how efficiently it is using resources to perform specific functions or  transactions.   However, activities do not constitute—and should not be  mistaken for—measures of IRS’s productivity in terms of ultimate results.
While the productivity measures we have examined are based on activities,  IRS has efforts under way to measure results such as compliance and  compliance burden.  Recently, we reported on IRS’s National Research  Program (NRP) to measure voluntary compliance and efforts to measure compliance burden.  As we mentioned previously, measuring these results  is difficult.  For some results, such as compliance, measurement is also  costly and intrusive because taxpayers must be contacted and questioned  in detail. Despite these difficulties, IRS can improve its productivity  measurement by continuing its efforts to get measures of results.  These  efforts would give Congress and the general public a better idea of what is  being achieved by the resources invested in IRS.
In the absence of direct measures of results, activities that are closely  related to the results of the service are used as proxies.  The value of these  proxies depends on the extent to which they are correlated with results. By  carefully choosing these measures, IRS may gain some information about  the effect of its activities on ultimate results.  Because activities may affect  a number of results and a single result may be affected by a number of  activities, a single activity likely will not be a sufficient proxy for the results  of the service.   Therefore, a variety of activities would likely be necessary  as proxies for the results of the service.
Both types of output measures, those that reflect the results of IRS’s  service and those that use activities to measure internal efficiency, should  be accurate and consistent over time.  In addition, both output measures  should be reliably linked to inputs.  Linking the results of IRS’s service to  inputs may be difficult because of outside factors that may also affect  measured results.  For example, an increase in compliance could result  both from IRS actions such as exams and from changes in tax laws.   Another challenge is that IRS currently has difficulties linking inputs to  activities, as we note in a previous report, where we reported IRS’s lack of a  cost accounting system.  In particular, IRS only recently implemented a  cost accounting system, and has not yet determined the full range of its  cost information needs. Table 1 summarizes some of the key differences  between activities and results measures.  Table 1 also indicates some  general criteria that apply to both types of measures.
Because inputs are more easily measured and identifiable than outputs,  measuring them is more straightforward.  IRS, as a government agency,  may be able more often to use labor costs or hours as a single input in its  productivity measures because it relies heavily on labor. However,  it may  be particularly important for IRS to use a multifactor measure that includes  capital along with labor during periods of modernization that involve  increased or high levels of capital investment.  As with outputs, inputs  should be measured accurately and consistently over time.   Measuring  inputs consistently over time may require adjusting for changes in the  quality of the labor, which has been done using proxies such as education  level or years of experience.  Also, as mentioned previously, inputs should  be reliably linked to outputs.
section_title: However Output Is Measured, IRS Can Improve Its Current Productivity Measures by Using Alternative Methods
paragraphs: The appropriate method for calculating productivity depends on the  purpose for which the productivity measure is being calculated. The  alternative methods that can be used for calculating productivity range  from computing single ratios—exams closed per FTE—to using complex  statistical methods to form composite indexes that combine multiple  indicators of outputs and inputs.   While single ratios may be adequate for  certain purposes, the composite indexes based on statistical methods may be more useful because they provide information about the sources of  productivity change.
Comparing the ratios of outputs to inputs at different points in time defines  a productivity index that measures the percentage increase or decrease in  productivity over time. The ratios that form the index may be single,  comparing a single output to a single input or composite, where multiple  outputs and inputs are compared.  The single ratios may be useful for  evaluating the efficiency of a single noncomplex activity.  Composite  indexes can measure the productivity of more complicated activities,  controlling for complexity and quality.  Composite indexes can also be used  to measure productivity of resources across an entire organization, where  many different activities are being performed.
One method of producing composite indexes is to use weights to combine  such disparate activities as telephone calls answered and exams closed.  One common weighting method, used by the Bureau of Labor Statistics  (BLS), is a labor weight.  Weighting outputs by their share of labor in a  baseline period controls for how resources are allocated between different  types of outputs.  If the productivity of two activities is unchanged but  resources are reallocated between the activities, the composite measure of  productivity would change unless these weights are employed.  For  example, if IRS reallocates exam resources so that it does more simple  exams and fewer complex exams, the number of total exams might  increase.  Consequently, a single productivity ratio comparing total exams  to inputs would show an increase.  Labor weighting deals with this issue.   The weights allow any gains from resource reallocation to be distinguished  from gains in the productivity of the underlying activities.  When types of  activities can be distinguished by their quality of complexity, labor  weighting can also be used to control for quality and complexity  differences when resources are shifted between types of outputs.
More complicated statistical methods can be used for calculating  composite indexes that allow for greater flexibility in how weights are  chosen to combine different outputs and for a wider range of output  measures that include both qualitative and quantitative outputs.   Data  Envelopment Analysis (DEA), which has been widely used to measure the  productivity of private industries and public sector services, is an example  of such methods   DEA estimates an efficiency score for each producing  unit, such as the firms in an industry or the schools in a school district, or  for IRS, the separately managed areas and territories composing its  business units.  DEA estimates the relative efficiency of each producing  unit by identifying those units with the best practice—those making the  most efficient use of inputs, under current technology, to produce  outputs—and measuring how far other units are from this best practice  combination of inputs used to produce outputs.  DEA estimates provide  managers with information on how efficient they are relative to other units  and the costs of making individual units more efficient.
These efficiency scores are used to form a composite productivity index  called a Malmquist index. An advantage of the Malmquist index is that IRS  managers can restrict the weights to adjust for managerial or congressional  preferences to investigate the effect on productivity of a shift, for example,  from an organization that emphasizes enforcement to one that emphasizes  service. IRS can also include many different types of outputs and inputs,  control for complexity and quality, and isolate the effects of certain  historical changes, such as the IRS Restructuring and Reform Act of 1998  (RRA98).
Another advantage of the Malmquist index is that productivity changes can  be separated into their components, such as efficiency and technology  changes.  In this context, efficiency can be measured holding technology  constant, and technology can be measured holding efficiency constant.   Holding technology constant, IRS might improve productivity by improving  the management of its existing resources.  On the other hand, technology  changes might improve productivity even if the management of resources  has not changed.  Thus, the productivity change of a given IRS unit is  determined by both changes in its efficiency relative to the current best-  practice IRS units and changes in the best practices or technology.
section_title: Illustrations of Alternative Methods of Measuring Productivity
paragraphs: Currently available IRS data can be used to produce productivity indexes  that control for complexity and quality.  The examples that follow focus on  productivity indexes that use exams closed as outputs and FTEs as inputs.   The data on examinations cover individual returns across IRS and IRS’s  LMSB division.  For both individuals and LMSB, the complexity and quality  of exams can vary over time.  For example, the proportion of exams that  are correspondence versus field, business versus nonbusiness, and EIC  versus non-EIC can vary over time.  As already discussed, failing to take  account of such variation can give a misleading picture of productivity  change.
While these examples do not encompass all the methods, data, and  adjustments that may be used, they illustrate the benefits of the additional  analysis that IRS can perform using current data. In addition, as we pointed  out in our 2004 report, IRS can improve its productivity measurement by  investing in better data, taking into account the costs and benefits of doing  so.  These better data include measures of complexity, improved measures  of quality, and additional measures of output.
Figures 1 through 4 illustrate, using currently available data between fiscal  years 1997 and 2004, the difference between weighted indexes that make  an adjustment for complexity and unweighted indexes that make no  adjustments.  In the illustrations, a labor-weighted composite index,  which can control for complexity, is contrasted with a single unweighted  index to show how the simpler method may be misleading. (See app. I for a  fuller description of the labor-weighted index.)  In each case, complexity is  proxied by type of exam. Although the data were limited (for example, the  measure of complexity was crude), the illustrations show that making the  adjustments that are possible provides a different picture of productivity  than would otherwise be available.
The advantage of weighted indexes is that they allow changes in the mix of  exams to be separated from changes in the productivity of performing  those exams.  In the examples that follow, an unweighted measure could be  picking up two effects.  One effect is the change in the number of exams  that an auditor can complete if the complexity or quality of the exam  changes.  The second effect is the change in the number of exams an  auditor can complete if the time an auditor requires to complete an exam  changes, holding the quality and complexity of exams constant. By  isolating the latter effect, the weighted index more closely measures  productivity, or the efficiency with which the auditor is working the exams.
For individual exams, the comparison of productivity indexes shows that  the unweighted index understates the decline in productivity. As figure 1  shows, between fiscal years 1997 and 2001, the unweighted productivity  index declined by 32 percent while the weighted index declined by 53  percent.  The difference is due largely to the increase in EIC exams during  the period.  Over the period between fiscal years 1997 and 2001, exams  were declining.  However, the mix of exams was changing, with increases  in the number of EIC exams.  EIC exams are disproportionately  correspondence exams, and IRS can do these exams faster than field  exams.  IRS shifted to “easier” exams, and that shift caused the unweighted  index to give an incomplete picture of productivity.  The shift masked the  larger productivity decline shown by the weighted index.
Figure 2 provides additional evidence to support the conclusion that the  shift to more EIC exams is the reason for the difference in productivity  shown in figure 1.  Between fiscal years 1997 and 2001, the weighted and  unweighted indexes track each other very closely when the EIC exams are  removed.  Both show a decline in productivity of about 50 percent over this  period.  The available data were not sufficient to control for other factors  that may have influenced exam productivity.  For example, RRA98 imposed  additional requirements on IRS’s auditors, such as certifications that they  had verified that past taxes were due.
Figure 3 compares unweighted and weighted productivity indexes for  exams done in LMSB division.  As figure 3 shows, between fiscal years 2002  and 2004, the unweighted productivity index increased by 4 percent, while  the weighted index increased by 16 percent.  This difference appears  largely due to the individual exams and small corporate exams done in  LMSB.  Over the period, total exams were declining but the mix of exams  was changing.  LMSB was shifting away from less labor-intensive individual  returns and small corporation returns to more complex business industry and coordinated industry return exams.  This shift caused the unweighted  index to give an incomplete picture of productivity.  Here, the shift masked  the larger productivity increase as shown by the weighted index.
Figure 4 provides additional evidence to support the conclusion that the  shift away from individual and small corporate exams is the reason for the  difference in productivity shown in figure 3.  Between fiscal years 2002 and  2004, when individual and corporate exams are excluded, the two indexes  track more closely, with the unweighted index increasing by 15 percent and  the weighted index by 17 percent.
There is evidence that adjusting for quality would show that LMSB’s  productivity increased more than is apparent in figures 3 and 4 for the years  2002 to 2004.  Average quality scores available for selected types of LMSB  exams show quality increasing over the 2-year period.  Adjusting for this  increase in quality, in addition to adjusting for complexity, would show a  productivity increase for these types of exams of 28 percent over the  period.
While labor-weighted and other more sophisticated productivity indexes  can provide a more complete picture of productivity changes, they do not  identify the causes of the changes.  These productivity indexes would be  the starting point for any analysis to determine the causes of productivity  changes.
Another example of the advantages of weighted productivity indexes is  provided by IRS.  As noted earlier, IRS has developed a weighted  submission processing productivity measure.  The measure adjusts for  differences in the complexity of processing various types of tax returns.  In  an internal analysis, IRS showed how productivity comparisons over time  and across the 10 processing centers depended on whether or not the  measure was adjusted for complexity.  For example, the ranking of the  processing centers in terms of productivity changed when the measure was  adjusted for the complexity of the returns being processed.
The more sophisticated methods for measuring productivity can provide  IRS and Congress with better information about IRS’s performance.  By  controlling for complexity and quality, IRS managers would have more  complete information about the true productivity of activities, such as  exams, that can differ in these dimensions.  In addition, the weighted  measures can be used to measure productivity for the organization, where  many different activities are being performed.  More complete information  about the productivity of IRS resources should be useful to both IRS  managers and Congress.  More complete productivity measures would  provide better information about the effectiveness of IRS resources, IRS’s  budget needs, and IRS’s efforts to improve efficiency.
Although there are examples, such as the submission processing  productivity measures, of IRS using weighted measures of productivity, IRS  officials said they generally use single ratios as measures of productivity.   That is consistent with our 2004 report on IRS’s enforcement improvement  projects, where we reported on SB/SE’s lack of productivity measures that  adjust for complexity and quality.
While there would be start-up costs associated with any new methodology,  the long-term costs to IRS for developing more sophisticated measures of  productivity may be modest.  The examples so far in this section  demonstrate the feasibility of developing weighted productivity indexes  using existing data.  Relying on existing data avoids the cost of having to  collect new data.  The fact that IRS already has some experience  implementing weighted productivity measures could reduce the cost of  introducing more such measures.
As we stated previously, IRS could also improve its productivity  measurement by getting better data on quality and complexity. These  improved data could be integrated with the methods for calculating  productivity illustrated in this report to further improve IRS’s productivity  measurement. However, as we acknowledged in our prior report, collecting  additional data on quality and complexity may require long-term planning  and an investment of additional resources. Any such investment, we noted,  must take account of the costs and benefits of acquiring the data.
section_title: Conclusion
paragraphs: Using more sophisticated methods, such as those summarized in this  report, for tracking productivity could produce a much richer picture of  how IRS manages its resources.  This is important not only because of the  size of IRS—it will spend about $11 billion in 2005 and employ about  100,000 FTEs—but also because we are entering an era of tight budgets.  A  more sophisticated understanding of the level of productivity at IRS and  the reasons for productivity change would better position IRS managers to  make decisions about how to effectively manage their resources.  Such  information would also better enable Congress and the public to assess the  performance of IRS.
As we illustrate, more can be done to measure IRS’s productivity using  current data.  However, another advantage of using more sophisticated  methods to track productivity is that the methods will highlight the value of  better data.  Better information about the quality and complexity of IRS’s  activities would enable the methods illustrated in this report to provide  even richer information about IRS’s overall productivity.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Commissioner of Internal Revenue put in place a  plan for introducing wider use of alternative methods of measuring  productivity, such as those illustrated in this report, taking account of the  costs of implementing the new methods.
section_title: Agency Comments and Our Evaluation
paragraphs: The Commissioner of Internal Revenue provided written comments on a  draft for this report in a June 23, 2005, letter.  The Commissioner agreed  with our recommendation to work on introducing wider use of alternative  measure of productivity. Although expressing some caution, he has asked  his Deputy Commissioner for Services and Enforcement to work with IRS’s  Research, Analysis, and Statistics office to assess the possible use of  alternative methods of measuring productivity. The Commissioner  recognized that a richer understanding of organizational performance is  crucial for effective program delivery.
As agreed with your office, unless you publicly release its contents earlier  we plan no further distribution of this report until 30 days from the date of  this letter. At that time, we will send copies to interested congressional  committees, the Secetary of the Treasury, the Commissioner of Internal  Revenue, and other interested parties. We will also make copies available  to others on request.
If you or your staff have any questions, please contact me at (202) 512-9110.   I can also be reached by e-mail at whitej@gao.gov.  Key contributors to this  assignment were Kevin Daly, Assistant Director, and Jennifer Gravelle.
section_title: Methods for Calculating Productivity Indexes
subsections: section_title: Productivity Indexes
paragraphs: Methods for calculating productivity range from computing single ratios to  using statistical methods.  In its simplest form, a productivity index is the  change in the productivity ratio over time relative to a chosen year.   However, this type of productivity index allows for only a single output and  a single input.  To account for more than one output, the outputs must be  combined to produce a productivity index.
One method is to weight the outputs by their share of inputs used in the  chosen base year.  In a case where only labor input is used, following this  method provides a labor-weighted output index, which, when divided by  the input index, produces the labor-weighted productivity index.  The use  of the share of labor used in each output effectively controls for the  allocation of labor across the outputs over time.  For example, if  productivity in producing two outputs remained fixed over time, a single  productivity index may show changes in productivity if resources are  reallocated to produce more of one of the outputs.
The Bureau of Labor Statistics (BLS) has also used labor-weighted indexes.   BLS published, under the Federal Productivity Measurement Program, data  on labor productivity in the federal government for more than two decades  (1967-94).  Due to budgetary constraints, the program is now terminated.   BLS’s measures used the “final outputs” of a federal program, which  correspond generally to what we have called intermediate outputs in this  report, as opposed to the outcomes or results of the program.  BLS used  labor weights because of their availability and their close link to cost  weights.  In particular, as with the labor weights in our illustrations, BLS  used base year labor weights and updated the weights every 5 years. It  relied only on labor and labor compensation, and acknowledges that the  indexes did not reflect changes in the quality of labor.  BLS measured  productivity for a number of federal programs, ranging from social and  information services to corrections.  However, BLS did not produce  productivity measures for IRS.
In addition to weighted productivity indexes, there are a number of  composite productivity indexes designed to include all the inputs and  outputs involved in production.  This group of indexes is called Total  Factor Productivity (TFP) indexes.  They are called total because they  include all the inputs and outputs, as opposed to Partial Factor  Productivity indexes, which relate only one input to one output.  Many of  the main TFP indexes, including Tornqvist, Fisher, Divisia, and Paache,  require reliable estimates of input and output prices, data not available for  industries in the public sector.  Therefore we use the Malmquist index,  which does not require that data.
Malmquist indexes are TFP indexes based on changes in the distance from  the production frontier, or distance functions.  These distance functions are  estimated using Data Envelopment Analysis (DEA). Productivity change is  represented by the ratio of two different period distance functions.  The  Malmquist index is the geometric average of these productivity changes  (evaluated at the two different periods).  This index can be further  decomposed into efficiency and technology changes.  From the  decomposition of the Malmquist index, productivity change can be shown  to equal the efficiency change times the technology change. (xt+1,yt+1)/D(x,y)]*[Dt+1(xt+1,yt+1)/  Dt+1(x,y)]}^1/2, where x, xt+1 denote the vector of inputs at time t and t+1, and y, and yt+1  denote the vector of outputs in time t and t+1 and D and Dt+1 are distance functions relative  to the technology in time t and t+1. (x,y)]*[Dt+1(xt+1,yt+1)/ Dt+1(x,y)]}^1/2 = [Dt+1(xt+1, yt+1)/  D(x,y)]*{[ D(xt+1,yt+1)/ Dt+1(xt+1,yt+1)]*[D(x,y)/ Dt+1(x,y)]}^1/2=E*T, the efficiency change,  E, times the technology change, T. fraction in the first, indicates a movement away from one over time and  thus declining productivity.  Thus, a productivity change less than one  indicates declining productivity and therefore an efficiency change less  than one also indicates declining efficiency.
Alternatively, if the efficiency change was one, then the productivity  change equals the technology change.  Following previous analysis, a  productivity change less than one indicates declining productivity.   Therefore, a technology change less than one indicates an inward shift of  the production frontier.  If the technology change is less than one, it must  be that the distance function in the first period is less than the distance  function in the next period.  Thus, the distance in the first period is farther  away from one than is the distance in the next period, and the distance  from the frontier decreased from the first period to the second period.   Since the output and input bundles did not change, the frontier must shift  in to produce the decrease in distance.
The Internal Revenue Service (IRS) can follow this method to generate  indexes for the areas and territories and then focus on the average for an  estimate of overall IRS productivity.
section_title: Estimation of Distance Functions
paragraphs: ,y)= [max { φ | (x, φy) ∈T}]-1  and φ* = (D(x,y))-1, with φ* >1 and D(x,y)< 1, where  φ denotes the value to scale output. therefore, inefficient relative to firms with a scalar value of one.  Thus,  output distance functions are less than one.  IRS can use this method,  treating territories and areas as firms.  The weights used in the linear  program are designed to make each firm look its best; they represent best  case scenarios.
While DEA is a nonparametric method, there is also a parametric method  available called stochastic frontier analysis.  Stochastic frontier analysis  (regression) uses a regression model to estimate cost or production  efficiency.  After running the regression of performance and input data, the  frontier is found by decomposing the residuals into a stochastic (statistical  noise) part and a systematic portion attributed to some form of  inefficiency.  Stochastic frontier analysis thus requires specifying the  distributional form of the errors and the functional form of the cost (or  production) function.  Its merits include a specific treatment of noise.   While DEA’s use of nonparametric methods eliminates the need to specify  functional forms, one drawback is a susceptibility to outliers.