Tax Administration: IRS Needs to Strengthen Its Approach for Evaluating the SRFMI Data-Sharing Pilot Program
Published: 2008-11-07T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The State Reverse File Match Initiative (SRFMI) is one of the Internal Revenue Service's (IRS) data-sharing strategies to reduce the estimated $345 billion gross federal tax gap. SRFMI matches federal and state taxpayer data to identify noncompliant taxpayers--individuals and businesses who do not file tax returns or do not report all of their income. IRS's document-matching program has proven to be a highly cost-effective way of identifying underreported income, thereby bringing in billions of dollars of tax revenue while boosting voluntary compliance. Based on concerns about whether IRS is fully using data from state and local governments to reduce the tax gap, GAO was asked to assess IRS's plan for evaluating the effectiveness of the SRFMI pilot taxpayer data-sharing program. To address these objectives, GAO consulted social science and evaluation literature, published GAO guidance, and IRS guidance; reviewed and analyzed SRFMI planning and evaluation documents; and interviewed IRS officials about IRS's plans to evaluate SRFMI.
section_title: What GAO Found
paragraphs: IRS is developing a plan for evaluating SRFMI data but does not currently have a documented plan even as the agency enters phase III of the pilot program and is less than 1 year away from rolling out the pilot to 45 states. According to IRS officials, the SRFMI pilot includes plans to evaluate program results to make decisions about expanding data sharing with states and using compliance data to assess whether SRFMI cases are more or less productive than other cases. IRS has little documentation on its evaluation. A well-developed and documented plan can help ensure that the evaluation generates performance information needed to make effective management decisions. A sound evaluation approach should also include criteria or standards for determining pilot-program performance. However, IRS has no criteria or standards for determining where the pilot program performs adequately to be incorporated into normal IRS compliance processes. IRS officials told GAO that they plan to use research results to help formulate standards to determine pilot success but did not provide a target date when this will occur. Moreover, IRS has not completely articulated its methodology to evaluate the pilot program. IRS officials have action plans and testing sample plans for individual units. The action plans primarily contained project-management-type information such as actions or tasks to be taken by individuals, start date, completion date, and comments rather than a comprehensive description of the methodological approach for the overall pilot project. The testing sample plans were questionnaires soliciting information about compliance measures for the individual unit and the resources available for testing SRFMI data. None of the plans outline the methods, timing, or frequency of data collection. While IRS officials have begun formulating plans for developing a sampling approach and determining appropriate sample size, they encountered challenges such as delays in information-technology assistance and time limits for using taxpayer data that have impeded progress in moving forward on its evaluation methodology. The need to evaluate the program is underscored because obtaining and using SRFMI data imposes costs not only on IRS but also on the states. Without a sound assessment of pilot program results, IRS may make poor decisions about the program's future.

section_title: Letter
paragraphs: The State Reverse File Match Initiative (SRFMI) is one of the Internal  Revenue Service’s (IRS) data-sharing strategies to reduce the estimated  $345 billion gross federal tax gap. SRFMI matches federal and state  taxpayer data to identify noncompliant federal taxpayers—individuals and  businesses who do not file tax returns or do not report all of their income.  Data sharing allows agencies to leverage external partnerships with other  agencies and has proven to be a valuable tool to a number of federal  agencies striving to improve decisions about applicants’ eligibility for  federal programs. As we have previously found, federal agencies are  increasingly sharing taxpayer data to help verify applicant-provided  information. IRS’s document-matching program, for example, has proven  to be a highly cost-effective way of identifying underreported income,  thereby bringing in billions of dollars of tax revenue while boosting  voluntary compliance.
In 2007, you expressed concerns about whether IRS is fully using data  from state and local governments to reduce the tax gap. Related to this  interest, you requested that we assess IRS’s plan for evaluating the  effectiveness of the SRFMI pilot taxpayer data-sharing program. To  respond to your request, we (1) consulted social science and evaluation  literature, along with published GAO guidance and IRS guidance, to  identify key features of an evaluation plan; (2) reviewed and analyzed  SRFMI planning and evaluation documents to determine whether they  contained key features of a sound evaluation plan; and (3) interviewed  relevant IRS officials about IRS’s plans to evaluate SRFMI, including what  cost data IRS plans to use to help fully understand how productive SRFMI  cases are.
We conducted this performance audit from July through September 2008,  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objective. We believe that the  evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objective.
subsections: section_title: Background
subsections: section_title: IRS and States Have an Existing Data-Sharing Relationship
paragraphs: Data-sharing programs between IRS and states have been in place for  many years. Historically, the flow of information was largely from the  federal government to the states. IRS shares tax data with the state  revenue agencies based on the states’ needs, and the states use the data  for tax-administration purposes. In recent years, however, IRS has taken  steps to change the flow of information and pursue additional  opportunities to obtain and use states’ taxpayer data for tax-compliance  purposes. Some examples of reciprocal data sharing between IRS and  state agencies include:    Governmental Liaison Data Exchange Program (GLDEP)—IRS and  state revenue agencies exchange taxpayer data for tax administration.
Questionable Employment Tax Program (QETP)—IRS shares Form  1099-MISC extract data with the state workforce agencies, and the  state workforce agencies investigate whether the people on IRS’s list  are employees or independent contractors.
Abusive Tax Avoidance Transactions (ATAT)—IRS and states  exchange information on adjustments made during tax shelter audits.  According to IRS, this exchange has led to the discovery by IRS and  states of tens of millions of dollars in previously unknown fraudulent  tax avoidance schemes.
section_title: SRFMI Pilot Program Matches State Data with IRS Taxpayer Data to Identify Nonfilers and Underreporters
paragraphs: In calendar year 2006, IRS saw an opportunity to expand data sharing by  obtaining more information from states. Accordingly, it created the SRFMI  pilot program. SRFMI has a goal to reduce the federal tax gap by  improving the tax compliance of individuals and businesses who do not  file (nonfilers) or who do not report all of their income (underreporters).  Under SRFMI, states match IRS’s return-filing information received  through the existing GLDEP data-sharing arrangement against state tax  data to identify individuals and businesses who filed a state return but did  not file a federal return or reported income to the state but not to IRS. The  pilot uses four different types of tax-return data—individual, corporate,  sales, and withholding—and takes steps to validate or put state data into a  format useable for compliance activities. (App. II, fig. 1 has an overview of  data matching under the SRFMI program.)
section_title: IRS Uses the Data- Matching Results in Several Different Compliance Activities
paragraphs: The result of matching federal and state tax data is a file with data on  potential federal nonfilers and underreporters. IRS then filters the  matched data to determine which matching results represent potential  noncompliance or “leads” for compliance activities in several different  organization units. These include:    Automated Substitute For Return (ASFR)—creates a substitute federal  tax return where none exists and makes an assessment.
Automated Underreporter—compares filed returns with information- reporting returns to identify unreported income. For the SRFMI pilot,  filed federal returns will be compared to filed state returns and not  necessarily to information returns.
Field Examination—conducts face to face examinations.    Campus Examination—conducts correspondence audits generally  involving basic single issues.
Specialty Tax—conducts examinations on excise taxes, employment  taxes, and estate and gift taxes.
Collection—collects revenue and secures delinquent returns.    Fraud/Bank Secrecy Act—assists other functions in developing civil  and criminal fraud investigations.
Criminal Investigation—conducts investigations on criminal violations  of the Internal Revenue Code and related financial crimes.
Each compliance activity determines which of the leads will be designated  for examination, contacted by correspondence requesting additional  information, or referred for further investigation for criminal activity.  (App. II, fig. 2 has an overview of the distribution of SRFMI data for  further examinations.)
section_title: The Pilot Program Has Four Phases with Increasing Numbers of Participating States
paragraphs: Three states participated in phase I, 6 additional states participated in  phase II, and 14 agencies from 13 states participated in phase III. (For a  table showing the states participating in phases I, II, and III, see app. I.)  IRS has enrolled 45 states to deliver SRFMI files in July 2009 and has the  following schedule for data exchanges:    Phase I started in March 2006.    States provided SRFMI extracts for phase II in July 2007.    States provided SRFMI extracts for phase III in July 2008.    States are to provide extracts for nationwide rollout by July 15, 2009.
section_title: Although IRS Has Begun Developing One, IRS Does Not Yet Have a Sound Evaluation Plan for SRFMI
paragraphs: According to IRS officials, the SRFMI pilot includes plans to evaluate  program results to make decisions about expanding data sharing with  states and using compliance data to evaluate whether SRFMI cases are  more or less productive than other cases. IRS’s approach to implementing  the pilot is outlined in a series of briefing documents. IRS officials told us  that since the program began in 2006, IRS has:    worked with states to receive state data in a uniform record layout;    validated and perfected federal and state data matches for the nine  states in phases I and II;    distributed data matches to several business units;    conducted compliance activities and tabulated phase I activities for  two organization units—ASFR and field examination; and    commissioned the Small Business/Self Employed Research (SB/SE  Research) unit to provide technical assistance in the areas of data  analysis, methodology, and project design; to provide various  tabulations of matched data and analyze the usefulness of the matches  for identifying nonfilers and underreporters; to develop appropriate  sample sizes for each compliance activity; and to conduct studies.
Using phase I data, SB/SE Research compared preliminary results of  SRFMI data used by two units in the SB/SE Division. ASFR reported on  closed cases, taxpayer returns, defaults, reconsiderations, dollars  assessed, and dollars assessed per return. Field examination reported on  more data elements than ASFR, including no-change rate, total  nonexamined cases, and average hours spent on returns. (App. III shows  preliminary compliance measures used by ASFR and field examination  activities.) The other compliance units have not reported results because  they have not begun to work with the matched data.
IRS officials told us that early results are limited and not statistically  sound but said that these results helped them become familiar with how  the data should be filtered and with compliance results information. SB/SE  Research has provided ongoing support and consultation and has  conducted some research studies to help develop an approach for  evaluating SRFMI pilot results. For example, SB/SE Research matched  Colorado and New Jersey tax-amnesty data to compare the potential use  of resources for SRFMI cases with non-SRFMI cases and found that state  tax-amnesty files cannot be used to definitively identify federal nonfilers  or underreporters and that the data lacked fields needed to determine  filing requirements and self-employment status. However, it found that  state tax-amnesty data can be a source for potential high-income nonfiler  leads and case building, with limitations. The Colorado and New Jersey  data were not SRFMI data but were the only data available for a first test  of the SRFMI concept. A study of phase I Arkansas, Iowa, and  Massachusetts SRFMI ASFR and Field Examination cases found mixed  results. SRFMI ASFR cases most clearly were less productive than regular  cases, because both the dollars assessed per case and per hour were  substantially lower for SRFMI than non-SRFMI cases. Field Examination  SRFMI cases, on the other hand, yielded lower dollars per case but higher  dollars per hour. However, IRS officials noted that these results were very  preliminary and that several refinements are needed in their methodology  for comparing cases. IRS SB/SE Research staff also acknowledge more  data, including cost data, are needed to conduct a more conclusive cost- benefit analysis.
subsections: section_title: Well-Developed Evaluation Plans Increase the Likelihood That Evaluations Will Yield Methodologically Sound Results and Support Effective Program and Policy Decisions
paragraphs: A well-developed and documented evaluation plan can help ensure that  agency evaluations generate performance information needed to make  effective program and policy decisions. Well-developed evaluation plans  include, at a minimum, several key features, such as    well-defined, clear, and measurable objectives;    criteria or standards for determining pilot-program performance;    clearly articulated methodology, including sound sampling methods,  determination of appropriate sample size for the evaluation design, and  a strategy for comparing the pilot results with other efforts;    a clear plan that details the type and source of data necessary to  evaluate the pilot, methods for data collection, and the timing and  frequency of data collection; and    a detailed data-analysis plan to track the program’s performance and  evaluate the final results of the project.
In addition, to ensure efficient use of its resources, IRS should include a  cost-effectiveness analysis to ensure that the program produces sufficient  benefits in relation to its costs.
Recognizing the importance of assessing the performance of a new  program, IRS requires that an evaluation plan and report be submitted  with a plan for IRS reorganizations. IRS requires that the evaluation plan  (1) be developed before the proposed reorganization takes place; (2)  include how, when, and by whom the impact of the reorganization will be  evaluated; (3) specify the measures that will be used and the standards for  judging the measures; (4) state how and when the data for each measure  will be obtained and the amount of change that will indicate success; (5)  measure the costs and benefits, including any savings from the  reorganization; (6) establish time lines for the evaluation; and (7) include  the standards for deciding whether to expand, make permanent,  discontinue, or modify the test organization. The reorganization evaluation  report must include a statement of the business need for the  reorganization, results for the measures, analysis of the results in relation  to the stated business need, conclusions about the impact of the change,  and recommendations. The IRS requirements are relevant, because key  features of a sound evaluation plan apply to any program for which  performance information is needed to make effective program and policy  decisions.
section_title: IRS Does Not Yet Have a Sound, Documented Evaluation Plan, Which Undermines Its Ability to Support Management Decisions about the Productivity of the SRFMI Pilot
paragraphs: IRS has not yet developed a methodologically sound and documented  evaluation plan for the SRFMI program. IRS’s current activities have a  number of limitations.
IRS has not defined clear, measurable objectives with specific statements  of the accomplishments necessary to successfully meet the objectives.  Clear and measurable objectives help ensure that the appropriate  evaluation data are collected and that performance can be measured  against the objectives. IRS has articulated a goal for SRFMI—to help  reduce the federal tax gap by improving tax compliance. IRS outlines its  basic approach for the SRFMI pilot in a number of briefing documents  and, in its SB/SE FY 2008 and 2009 Plan, alludes to SRFMI in its  discussion of strategies to address the tax gap. In its briefing document  for the 2008 Federation of Tax Administrators conference, IRS indicates  SRFMI has potential to reduce the tax gap and improve voluntary  compliance. The slides outline objectives such as: (1) secure compliance  results to drive business decisions, (2) modify systems and compliance  activities to further automate SRFMI, and (3) reduce the tax gap by  making inroads with noncompliant business entities. These objectives are  not clear. For instance, “securing compliance results to drive business  decisions” does not explain what compliance results would be used or  what business decisions the compliance results would drive. Nor are the  objectives measurable. For example, “reduce the tax gap by making  inroads with noncompliant business entities” does not specify how much  of a compliance improvement would indicate success, or how many  business entities or what dollar amount of new assessments would  indicate that IRS is “making inroads” with noncompliant businesses. IRS  officials told us that individual IRS units using the SRFMI data will have  their own separate objectives for evaluating all data sources for their  compliance activity and that they will measure SRFMI results by the same  standards. They added that IRS is incorporating these objectives into an  overall SRFMI sampling and evaluation plan.
IRS has no criteria or standards for determining where the pilot program  performs adequately to be incorporated into normal IRS compliance  processes. IRS officials said the agency plans to use research results to  help formulate standards but did not provide a target date when this will  occur. IRS officials provided several projects conducted by SB/SE  Research using phase I SRFMI data that will serve as the basis for  developing standards. For example, they gave us tabulations of results for  data matches using federal and Arkansas, Iowa, and Massachusetts state  data and a preliminary analysis of the usefulness of Colorado and New  Jersey tax amnesty data. Such standards are needed for IRS to determine  whether the SRFMI pilot program is effective.
IRS has not completely articulated its methodology to evaluate the pilot  program. SRFMI methodology should include developing plans for sound  sampling methods, determining appropriate sample sizes, and comparing  the pilot results with IRS’s similar ongoing compliance efforts to  determine whether SRFMI can identify cases with higher noncompliance  potential than similar ongoing cases. IRS officials provided action plans  and testing sample plans for individual units when asked about their  methodology. The action plans primarily contained project-management- type information such as actions or tasks to be taken by individuals, start  date, completion date, and problem areas rather than a comprehensive  description of the methodological approach for the overall pilot project.  The testing sample plans were questionnaires soliciting information about  compliance measures for the individual unit and the resources available  for testing SRFMI data. None of the plans they presented outline the  methods, timing, or frequency of data collection. Specifying these  elements in the plan would help ensure that adequate, accurate, and timely  data will be available to complete the evaluation. While IRS officials have  begun developing a sampling approach and determining appropriate  sample sizes, IRS officials encountered challenges that have delayed  progress. First, IRS officials told us that information technology support  for identifying cases with higher noncompliance potential than similar  ongoing cases did not occur as planned and, as a result, two compliance  activities—ASFR and Field Examination—performed manual workload  selection on SRFMI cases. Second, IRS is barred by the 3-year statute of  limitations from assessing tax liabilities identified by state tax data  received in the earlier phases of the pilot program. IRS submitted a  legislative proposal that would extend the 3-year limitation for  assessments based on information obtained from state and local tax data.
IRS does not have a plan that details the type and source of data necessary  to evaluate the pilot program, methods for data collection, or the timing  and frequency of data collection. The testing sample plans SB/SE Research  has begun developing have information on the type and source of data it  will collect for the four of eight compliance activities that have filled out  the questionnaire. Agency officials told us that they will ask the rest of the  compliance activities to fill out the same questionnaires.
IRS does not have a data-analysis plan that describes how the data will be  analyzed to track the program’s performance and evaluate the final results.  Instead, IRS has analyzed phase I data in an ad hoc manner, using data  from the two compliance activities (i.e., ASFR and Field Examination) that  worked the available cases. A detailed data-analysis plan is a key feature  of an evaluation plan as it sets out who will do the analysis and when and  how data will be analyzed to measure the pilot program’s performance.
IRS does not have a cost-effectiveness-analysis plan to help ensure that the  program produces sufficient benefits in relation to its costs. For example,  SB/SE Research was asked to conduct an analysis with available phase I  matched data using federal tax returns and Arkansas, Iowa, and  Massachusetts state tax-amnesty data to determine SRFMI’s cost  effectiveness. The results of the SB/SE Research were not conclusive,  because the data they used were not a representative sample of ASFR and  Field Examination SRFMI cases. Because ASFR and Field Examination  did not have cost data, SB/SE Research could not do a cost-effectiveness  study. Instead, for ASFR, the study compared dollars assessed per case  and dollars assessed per hour worked on SRFMI cases with normal non- SRFMI cases. For Field Examination, it tracked average hours worked per  return, no-change rate, cycle time, and pick-up rate. Agency officials told  us that IRS has no cost data for the program and that they do not track the  costs of the pilot. IRS acknowledges that more data are needed to do a  complete cost assessment.
section_title: Conclusions
paragraphs: Evaluating the SRFMI pilot can help IRS determine the benefits of using  state taxpayer data for compliance purposes. IRS is exploring new  territory by obtaining and using state taxpayer data to match against its  own to identify taxpayers who do not file federal tax returns or  underreport income on their federal tax returns. However, 2½ years after  the pilot began, as it is entering phase III and is less than 1 year away from  a national rollout with 45 states enrolled to deliver SRFMI files, IRS has yet  to develop and document a sound evaluation plan. The need to evaluate  the program is underscored because obtaining and using SRFMI data  imposes costs not only on IRS but also on the states. Without a sound  assessment of pilot program results, IRS may make poor decisions about  the program’s future.
section_title: Recommendation for Executive Action
paragraphs: We recommend that the Commissioner of Internal Revenue ensure that  IRS develops an evaluation plan to accurately and reliably assess the  SRFMI pilot program’s results. This plan should address all components of  the pilot program and include key evaluation features of a sound plan:    well-defined, clear, and measurable objectives;    criteria or standards for determining pilot program performance;    clearly articulated methodology, including sound sampling methods,  determination of appropriate sample size for the evaluation design, and  a strategy for comparing the pilot results with other efforts; and    a clear plan that details the type and source of data necessary to  evaluate the pilot, methods for data collection, and the timing and  frequency of data collection; and     a detailed data-analysis plan to track the program’s performance and  evaluate the final results of the project.
section_title: Agency Comments
paragraphs: On behalf of the Commissioner of Internal Revenue, the Deputy  Commissioner for Services and Enforcement provided written comments  on a draft of this report in a November 3, 2008, letter. The Deputy  Commissioner agreed it is important to properly document and assess the  SRFMI pilot program as a whole before it is expanded to additional states.  The Deputy Commissioner said a summit will be conducted with the  business unit owners and their respective research functions to further  develop and enhance their existing SRFMI plans to include the key  evaluation features we recommend. In addition, IRS agreed to develop an  overall evaluation plan to accurately and reliably assess all components of  the SRFMI pilot program’s results and include the key evaluation features  in our recommendation.
As agreed with your offices, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days after its  date. At that time, we will send copies to the Secretary of the Treasury, the  Commissioner of Internal Revenue, and other interested parties. This  report will also be available at no charge on GAO’s Web site at  http://www.gao.gov. If you or your staff have any questions about this  report, please contact me at (202) 512-9110 or brostekm@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. Key contributors to this report are  listed in appendix V.
section_title: Appendix I: States Participating in Phases I, II, and III
paragraphs: Below is a list of states participating in the first three phases of the pilot  program. The Internal Revenue Service (IRS) expects to start a nationwide  rollout in July 2009.
section_title: Appendix II: Overview of Data-Matching Process and Further Examinations or Investigations under SRFMI
paragraphs: The figure below shows an overview of how the State Reverse File Match  Initiative (SRFMI) data flow between the Internal Revenue Service (IRS)  and state revenue agencies.
IRS officials told us that when IRS receives SRFMI data on potentially  noncompliant business and individual taxpayers, IRS conducts data  validation and perfection activities. Next, IRS plans to put the matched  data through a filtering process under development to identify cases with  higher potential for noncompliance and determine how data should be  distributed to the individual units within the Small Business/Self Employed  (SB/SE) Division that will use the data for compliance activities. IRS  officials told us the SRFMI filter will be continuously refined to take into  account law changes, taxpayer schemes, and other nuances that business  units expect to identify as they develop a history of using the data in  compliance activities. After the filtering process, the SRFMI cases will be  distributed to eight different compliance activities where the business  units will further screen filtered SRFMI data against their program criteria.
Figure 2 shows the distribution of SRFMI data within IRS as well as the  disposition of cases that use SRFMI data.
section_title: Appendix III: Preliminary Compliance Measures by ASFR and Field Examination Activities
paragraphs: During 2008, two organization units tracked compliance activities using  phase I State Reverse File Match Initiative (SRFMI) pilot data. The  Automatic Substitute for Return (ASFR) program is a compliance activity  that creates a substitute federal return when none exists. Following are  the traditional compliance measures used by ASFR:    number of case closures,    number of taxpayer returns prepared,    number of defaults,    number of reconsiderations,    dollars assessed, and    dollars assessed per return.
Small Business/Self-Employed (SB/SE) Field Examination is a compliance  activity during which agents conduct audits in an Internal Revenue Service  (IRS) office or at a taxpayer’s place of business. SRFMI cases are  examined by either revenue agents or tax compliance officers. IRS agents  conducting field examinations used federal-state matched SRFMI data.  Field Examination tracked the phase I SRFMI cases it examined using the  measures listed below:  total closures,  related return closures,  average hours per return,  average dollars per return,  average dollars per hour,  total dollars assessed,    no-change returns,    no-change rate,  average closed cycle days, and  total nonexamined cases.
section_title: Appendix IV: Comments from the Internal Revenue Service
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: Acknowledgments
paragraphs: In addition to the contact named above, Signora J. May, Assistant Director;  Amy Bowser; Amy Friedlander; Cindy Gilbert; and Lou V. B. Smith made  key contributions to this report.