Tax Administration: IRS Needs to Strengthen Its Approach for Evaluating the SRFMI Data-Sharing Pilot Program
Released: 2008-12-08T00:00:00

Letter

Background

IRS and States Have an Existing Data-Sharing Relationship

Data-sharing programs between IRS and states have been in place for many years. Historically, the flow of information was largely from the federal government to the states. IRS shares tax data with the state revenue agencies based on the states’ needs, and the states use the data for tax-administration purposes. In recent years, however, IRS has taken steps to change the flow of information and pursue additional opportunities to obtain and use states’ taxpayer data for tax-compliance purposes. Some examples of reciprocal data sharing between IRS and state agencies include: Governmental Liaison Data Exchange Program (GLDEP)—IRS and state revenue agencies exchange taxpayer data for tax administration.

Questionable Employment Tax Program (QETP)—IRS shares Form 1099-MISC extract data with the state workforce agencies, and the state workforce agencies investigate whether the people on IRS’s list are employees or independent contractors.

Abusive Tax Avoidance Transactions (ATAT)—IRS and states exchange information on adjustments made during tax shelter audits. According to IRS, this exchange has led to the discovery by IRS and states of tens of millions of dollars in previously unknown fraudulent tax avoidance schemes.

SRFMI Pilot Program Matches State Data with IRS Taxpayer Data to Identify Nonfilers and Underreporters

In calendar year 2006, IRS saw an opportunity to expand data sharing by obtaining more information from states. Accordingly, it created the SRFMI pilot program. SRFMI has a goal to reduce the federal tax gap by improving the tax compliance of individuals and businesses who do not file (nonfilers) or who do not report all of their income (underreporters). Under SRFMI, states match IRS’s return-filing information received through the existing GLDEP data-sharing arrangement against state tax data to identify individuals and businesses who filed a state return but did not file a federal return or reported income to the state but not to IRS. The pilot uses four different types of tax-return data—individual, corporate, sales, and withholding—and takes steps to validate or put state data into a format useable for compliance activities. (App. II, fig. 1 has an overview of data matching under the SRFMI program.)

IRS Uses the Data- Matching Results in Several Different Compliance Activities

The result of matching federal and state tax data is a file with data on potential federal nonfilers and underreporters. IRS then filters the matched data to determine which matching results represent potential noncompliance or “leads” for compliance activities in several different organization units. These include: Automated Substitute For Return (ASFR)—creates a substitute federal tax return where none exists and makes an assessment.

Automated Underreporter—compares filed returns with information- reporting returns to identify unreported income. For the SRFMI pilot, filed federal returns will be compared to filed state returns and not necessarily to information returns.

Field Examination—conducts face to face examinations. Campus Examination—conducts correspondence audits generally involving basic single issues.

Specialty Tax—conducts examinations on excise taxes, employment taxes, and estate and gift taxes.

Collection—collects revenue and secures delinquent returns. Fraud/Bank Secrecy Act—assists other functions in developing civil and criminal fraud investigations.

Criminal Investigation—conducts investigations on criminal violations of the Internal Revenue Code and related financial crimes.

Each compliance activity determines which of the leads will be designated for examination, contacted by correspondence requesting additional information, or referred for further investigation for criminal activity. (App. II, fig. 2 has an overview of the distribution of SRFMI data for further examinations.)

The Pilot Program Has Four Phases with Increasing Numbers of Participating States

Three states participated in phase I, 6 additional states participated in phase II, and 14 agencies from 13 states participated in phase III. (For a table showing the states participating in phases I, II, and III, see app. I.) IRS has enrolled 45 states to deliver SRFMI files in July 2009 and has the following schedule for data exchanges: Phase I started in March 2006. States provided SRFMI extracts for phase II in July 2007. States provided SRFMI extracts for phase III in July 2008. States are to provide extracts for nationwide rollout by July 15, 2009.

Although IRS Has Begun Developing One, IRS Does Not Yet Have a Sound Evaluation Plan for SRFMI

According to IRS officials, the SRFMI pilot includes plans to evaluate program results to make decisions about expanding data sharing with states and using compliance data to evaluate whether SRFMI cases are more or less productive than other cases. IRS’s approach to implementing the pilot is outlined in a series of briefing documents. IRS officials told us that since the program began in 2006, IRS has: worked with states to receive state data in a uniform record layout; validated and perfected federal and state data matches for the nine states in phases I and II; distributed data matches to several business units; conducted compliance activities and tabulated phase I activities for two organization units—ASFR and field examination; and commissioned the Small Business/Self Employed Research (SB/SE Research) unit to provide technical assistance in the areas of data analysis, methodology, and project design; to provide various tabulations of matched data and analyze the usefulness of the matches for identifying nonfilers and underreporters; to develop appropriate sample sizes for each compliance activity; and to conduct studies.

Using phase I data, SB/SE Research compared preliminary results of SRFMI data used by two units in the SB/SE Division. ASFR reported on closed cases, taxpayer returns, defaults, reconsiderations, dollars assessed, and dollars assessed per return. Field examination reported on more data elements than ASFR, including no-change rate, total nonexamined cases, and average hours spent on returns. (App. III shows preliminary compliance measures used by ASFR and field examination activities.) The other compliance units have not reported results because they have not begun to work with the matched data.

IRS officials told us that early results are limited and not statistically sound but said that these results helped them become familiar with how the data should be filtered and with compliance results information. SB/SE Research has provided ongoing support and consultation and has conducted some research studies to help develop an approach for evaluating SRFMI pilot results. For example, SB/SE Research matched Colorado and New Jersey tax-amnesty data to compare the potential use of resources for SRFMI cases with non-SRFMI cases and found that state tax-amnesty files cannot be used to definitively identify federal nonfilers or underreporters and that the data lacked fields needed to determine filing requirements and self-employment status. However, it found that state tax-amnesty data can be a source for potential high-income nonfiler leads and case building, with limitations. The Colorado and New Jersey data were not SRFMI data but were the only data available for a first test of the SRFMI concept. A study of phase I Arkansas, Iowa, and Massachusetts SRFMI ASFR and Field Examination cases found mixed results. SRFMI ASFR cases most clearly were less productive than regular cases, because both the dollars assessed per case and per hour were substantially lower for SRFMI than non-SRFMI cases. Field Examination SRFMI cases, on the other hand, yielded lower dollars per case but higher dollars per hour. However, IRS officials noted that these results were very preliminary and that several refinements are needed in their methodology for comparing cases. IRS SB/SE Research staff also acknowledge more data, including cost data, are needed to conduct a more conclusive cost- benefit analysis.

Well-Developed Evaluation Plans Increase the Likelihood That Evaluations Will Yield Methodologically Sound Results and Support Effective Program and Policy Decisions

A well-developed and documented evaluation plan can help ensure that agency evaluations generate performance information needed to make effective program and policy decisions. Well-developed evaluation plans include, at a minimum, several key features, such as well-defined, clear, and measurable objectives; criteria or standards for determining pilot-program performance; clearly articulated methodology, including sound sampling methods, determination of appropriate sample size for the evaluation design, and a strategy for comparing the pilot results with other efforts; a clear plan that details the type and source of data necessary to evaluate the pilot, methods for data collection, and the timing and frequency of data collection; and a detailed data-analysis plan to track the program’s performance and evaluate the final results of the project.

In addition, to ensure efficient use of its resources, IRS should include a cost-effectiveness analysis to ensure that the program produces sufficient benefits in relation to its costs.

Recognizing the importance of assessing the performance of a new program, IRS requires that an evaluation plan and report be submitted with a plan for IRS reorganizations. IRS requires that the evaluation plan (1) be developed before the proposed reorganization takes place; (2) include how, when, and by whom the impact of the reorganization will be evaluated; (3) specify the measures that will be used and the standards for judging the measures; (4) state how and when the data for each measure will be obtained and the amount of change that will indicate success; (5) measure the costs and benefits, including any savings from the reorganization; (6) establish time lines for the evaluation; and (7) include the standards for deciding whether to expand, make permanent, discontinue, or modify the test organization. The reorganization evaluation report must include a statement of the business need for the reorganization, results for the measures, analysis of the results in relation to the stated business need, conclusions about the impact of the change, and recommendations. The IRS requirements are relevant, because key features of a sound evaluation plan apply to any program for which performance information is needed to make effective program and policy decisions.

IRS Does Not Yet Have a Sound, Documented Evaluation Plan, Which Undermines Its Ability to Support Management Decisions about the Productivity of the SRFMI Pilot

IRS has not yet developed a methodologically sound and documented evaluation plan for the SRFMI program. IRS’s current activities have a number of limitations.

IRS has not defined clear, measurable objectives with specific statements of the accomplishments necessary to successfully meet the objectives. Clear and measurable objectives help ensure that the appropriate evaluation data are collected and that performance can be measured against the objectives. IRS has articulated a goal for SRFMI—to help reduce the federal tax gap by improving tax compliance. IRS outlines its basic approach for the SRFMI pilot in a number of briefing documents and, in its SB/SE FY 2008 and 2009 Plan, alludes to SRFMI in its discussion of strategies to address the tax gap. In its briefing document for the 2008 Federation of Tax Administrators conference, IRS indicates SRFMI has potential to reduce the tax gap and improve voluntary compliance. The slides outline objectives such as: (1) secure compliance results to drive business decisions, (2) modify systems and compliance activities to further automate SRFMI, and (3) reduce the tax gap by making inroads with noncompliant business entities. These objectives are not clear. For instance, “securing compliance results to drive business decisions” does not explain what compliance results would be used or what business decisions the compliance results would drive. Nor are the objectives measurable. For example, “reduce the tax gap by making inroads with noncompliant business entities” does not specify how much of a compliance improvement would indicate success, or how many business entities or what dollar amount of new assessments would indicate that IRS is “making inroads” with noncompliant businesses. IRS officials told us that individual IRS units using the SRFMI data will have their own separate objectives for evaluating all data sources for their compliance activity and that they will measure SRFMI results by the same standards. They added that IRS is incorporating these objectives into an overall SRFMI sampling and evaluation plan.

IRS has no criteria or standards for determining where the pilot program performs adequately to be incorporated into normal IRS compliance processes. IRS officials said the agency plans to use research results to help formulate standards but did not provide a target date when this will occur. IRS officials provided several projects conducted by SB/SE Research using phase I SRFMI data that will serve as the basis for developing standards. For example, they gave us tabulations of results for data matches using federal and Arkansas, Iowa, and Massachusetts state data and a preliminary analysis of the usefulness of Colorado and New Jersey tax amnesty data. Such standards are needed for IRS to determine whether the SRFMI pilot program is effective.

IRS has not completely articulated its methodology to evaluate the pilot program. SRFMI methodology should include developing plans for sound sampling methods, determining appropriate sample sizes, and comparing the pilot results with IRS’s similar ongoing compliance efforts to determine whether SRFMI can identify cases with higher noncompliance potential than similar ongoing cases. IRS officials provided action plans and testing sample plans for individual units when asked about their methodology. The action plans primarily contained project-management- type information such as actions or tasks to be taken by individuals, start date, completion date, and problem areas rather than a comprehensive description of the methodological approach for the overall pilot project. The testing sample plans were questionnaires soliciting information about compliance measures for the individual unit and the resources available for testing SRFMI data. None of the plans they presented outline the methods, timing, or frequency of data collection. Specifying these elements in the plan would help ensure that adequate, accurate, and timely data will be available to complete the evaluation. While IRS officials have begun developing a sampling approach and determining appropriate sample sizes, IRS officials encountered challenges that have delayed progress. First, IRS officials told us that information technology support for identifying cases with higher noncompliance potential than similar ongoing cases did not occur as planned and, as a result, two compliance activities—ASFR and Field Examination—performed manual workload selection on SRFMI cases. Second, IRS is barred by the 3-year statute of limitations from assessing tax liabilities identified by state tax data received in the earlier phases of the pilot program. IRS submitted a legislative proposal that would extend the 3-year limitation for assessments based on information obtained from state and local tax data.

IRS does not have a plan that details the type and source of data necessary to evaluate the pilot program, methods for data collection, or the timing and frequency of data collection. The testing sample plans SB/SE Research has begun developing have information on the type and source of data it will collect for the four of eight compliance activities that have filled out the questionnaire. Agency officials told us that they will ask the rest of the compliance activities to fill out the same questionnaires.

IRS does not have a data-analysis plan that describes how the data will be analyzed to track the program’s performance and evaluate the final results. Instead, IRS has analyzed phase I data in an ad hoc manner, using data from the two compliance activities (i.e., ASFR and Field Examination) that worked the available cases. A detailed data-analysis plan is a key feature of an evaluation plan as it sets out who will do the analysis and when and how data will be analyzed to measure the pilot program’s performance.

IRS does not have a cost-effectiveness-analysis plan to help ensure that the program produces sufficient benefits in relation to its costs. For example, SB/SE Research was asked to conduct an analysis with available phase I matched data using federal tax returns and Arkansas, Iowa, and Massachusetts state tax-amnesty data to determine SRFMI’s cost effectiveness. The results of the SB/SE Research were not conclusive, because the data they used were not a representative sample of ASFR and Field Examination SRFMI cases. Because ASFR and Field Examination did not have cost data, SB/SE Research could not do a cost-effectiveness study. Instead, for ASFR, the study compared dollars assessed per case and dollars assessed per hour worked on SRFMI cases with normal non- SRFMI cases. For Field Examination, it tracked average hours worked per return, no-change rate, cycle time, and pick-up rate. Agency officials told us that IRS has no cost data for the program and that they do not track the costs of the pilot. IRS acknowledges that more data are needed to do a complete cost assessment.

Conclusions

Evaluating the SRFMI pilot can help IRS determine the benefits of using state taxpayer data for compliance purposes. IRS is exploring new territory by obtaining and using state taxpayer data to match against its own to identify taxpayers who do not file federal tax returns or underreport income on their federal tax returns. However, 2½ years after the pilot began, as it is entering phase III and is less than 1 year away from a national rollout with 45 states enrolled to deliver SRFMI files, IRS has yet to develop and document a sound evaluation plan. The need to evaluate the program is underscored because obtaining and using SRFMI data imposes costs not only on IRS but also on the states. Without a sound assessment of pilot program results, IRS may make poor decisions about the program’s future.

Recommendation for Executive Action

We recommend that the Commissioner of Internal Revenue ensure that IRS develops an evaluation plan to accurately and reliably assess the SRFMI pilot program’s results. This plan should address all components of the pilot program and include key evaluation features of a sound plan: well-defined, clear, and measurable objectives; criteria or standards for determining pilot program performance; clearly articulated methodology, including sound sampling methods, determination of appropriate sample size for the evaluation design, and a strategy for comparing the pilot results with other efforts; and a clear plan that details the type and source of data necessary to evaluate the pilot, methods for data collection, and the timing and frequency of data collection; and a detailed data-analysis plan to track the program’s performance and evaluate the final results of the project.

Agency Comments

On behalf of the Commissioner of Internal Revenue, the Deputy Commissioner for Services and Enforcement provided written comments on a draft of this report in a November 3, 2008, letter. The Deputy Commissioner agreed it is important to properly document and assess the SRFMI pilot program as a whole before it is expanded to additional states. The Deputy Commissioner said a summit will be conducted with the business unit owners and their respective research functions to further develop and enhance their existing SRFMI plans to include the key evaluation features we recommend. In addition, IRS agreed to develop an overall evaluation plan to accurately and reliably assess all components of the SRFMI pilot program’s results and include the key evaluation features in our recommendation.

As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after its date. At that time, we will send copies to the Secretary of the Treasury, the Commissioner of Internal Revenue, and other interested parties. This report will also be available at no charge on GAO’s Web site at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9110 or brostekm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix V.

Appendix I: States Participating in Phases I, II, and III

Below is a list of states participating in the first three phases of the pilot program. The Internal Revenue Service (IRS) expects to start a nationwide rollout in July 2009.

Appendix II: Overview of Data-Matching Process and Further Examinations or Investigations under SRFMI

The figure below shows an overview of how the State Reverse File Match Initiative (SRFMI) data flow between the Internal Revenue Service (IRS) and state revenue agencies.

IRS officials told us that when IRS receives SRFMI data on potentially noncompliant business and individual taxpayers, IRS conducts data validation and perfection activities. Next, IRS plans to put the matched data through a filtering process under development to identify cases with higher potential for noncompliance and determine how data should be distributed to the individual units within the Small Business/Self Employed (SB/SE) Division that will use the data for compliance activities. IRS officials told us the SRFMI filter will be continuously refined to take into account law changes, taxpayer schemes, and other nuances that business units expect to identify as they develop a history of using the data in compliance activities. After the filtering process, the SRFMI cases will be distributed to eight different compliance activities where the business units will further screen filtered SRFMI data against their program criteria.

Figure 2 shows the distribution of SRFMI data within IRS as well as the disposition of cases that use SRFMI data.

Appendix III: Preliminary Compliance Measures by ASFR and Field Examination Activities

During 2008, two organization units tracked compliance activities using phase I State Reverse File Match Initiative (SRFMI) pilot data. The Automatic Substitute for Return (ASFR) program is a compliance activity that creates a substitute federal return when none exists. Following are the traditional compliance measures used by ASFR: number of case closures, number of taxpayer returns prepared, number of defaults, number of reconsiderations, dollars assessed, and dollars assessed per return.

Small Business/Self-Employed (SB/SE) Field Examination is a compliance activity during which agents conduct audits in an Internal Revenue Service (IRS) office or at a taxpayer’s place of business. SRFMI cases are examined by either revenue agents or tax compliance officers. IRS agents conducting field examinations used federal-state matched SRFMI data. Field Examination tracked the phase I SRFMI cases it examined using the measures listed below: total closures, related return closures, average hours per return, average dollars per return, average dollars per hour, total dollars assessed, no-change returns, no-change rate, average closed cycle days, and total nonexamined cases.

Appendix IV: Comments from the Internal Revenue Service

Appendix V: GAO Contact and Staff Acknowledgments

Acknowledgments

In addition to the contact named above, Signora J. May, Assistant Director; Amy Bowser; Amy Friedlander; Cindy Gilbert; and Lou V. B. Smith made key contributions to this report.