Tax Debt Collection: IRS Could Improve Future Studies by Establishing Appropriate Guidance
Published: 2010-09-24T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In September 2006, the Internal Revenue Service (IRS) started the private debt collection (PDC) program for using private collection agencies (PCA) to help collect some unpaid tax debts. Aware of concerns that PCAs might cost more than using IRS staff, IRS began studying the collection costs and performance of PCAs and IRS. In March 2009, IRS announced that it would not renew its PCA contracts based on the study and announced plans for increasing collection staffing. As requested, GAO is reporting on whether (1) the study was sound as primary support for IRS's PDC decision and (2) IRS has planned or made changes to its collection approach based on its PCA experience and PDC study. GAO compared IRS's study to federal and other guidance on what should be included in analyses to support program decisions and analyzed IRS's changes given expectations that IRS would consider PCAs' best practices.
section_title: What GAO Found
paragraphs: IRS's comparative study of the PDC program was not soundly designed to support its decision on whether to continue contracting out debt collection. Although the study was not originally intended or designed as primary support for the decision, IRS officials nonetheless used it as such. IRS did not have guidance for program managers on the type of analysis that should be done to support decisions to create, renew, or expand programs. IRS had not retained sufficient documentation on the sample used in the study or documented some analyses that would have been helpful if performed. The study results may be overstated or understated because the study sample was not generalizable to the program as a whole. The study had a narrow objective of comparing results for IRS working the same cases as PCAs had, and as a result, the study design did not consider other factors recommended by Office of Management and Budget and other guidance on conducting program analysis. For example, the study did not analyze alternatives to program scale, such as expanding it or scaling it back. Program analysis guidance states that to the extent possible all costs and benefits should be counted and alternative means of achieving a program's goals should be considered. But the study did not identify important costs and benefits, such as whether taxpayers' compliance costs would be different if IRS or PCAs work debt cases. Nevertheless, neither GAO nor IRS officials know whether the study results and decision on the program would have differed significantly if it had been designed to be primary support for IRS's PDC program. In commenting on a draft of GAO's report, IRS disagreed that the PDC study was not soundly designed. GAO stands by its analysis detailing the study's errors, narrow scope, and lack of adherence to guidance. These design and methodology deficiencies limited the study's usefulness in supporting IRS's decision. IRS has not made or planned changes to its collection approach based on its PCA experience and study. In authorizing the use of PCAs, Congress required IRS to report to Congress its measurement plan to identify any of the PCAs' best practices that IRS could adopt to improve its own collection operations. IRS did not continue to report to Congress as required. In an unpublished draft report, IRS asserted that it had reviewed a number of PCA practices and found no immediate opportunities to change its collection approach. IRS did not provide GAO documentation on the study to support that conclusion. In part because PCA-type cases had previously been considered low priority, IRS officials were surprised by the PDC study results, which indicated that IRS staff might have better results working PCA-type cases than some of the cases IRS normally works. IRS officials said that they initiated a pilot study in 2009 to help them decide whether to use IRS staff to work selected types of PCA cases. As GAO concluded its review, IRS provided conflicting information on the role of the pilot study. On one hand, IRS said a collection selection system to be implemented in January 2011 overtook the need for the study. On the other hand, an IRS official said that the results from PCA-type cases were not used in the development of the new case selection system.
section_title: What GAO Recommends
paragraphs: GAO recommends that IRS (1) establish guidance on analyses to support program decisions, (2) establish a policy requiring documentation of program studies, and (3) ensure that PCA-type case results are considered for IRS's new case selection model. IRS agreed with the first two recommendations and agreed in principle with the third, which GAO revised to reflect updated information that IRS provided.

section_title: Letter
paragraphs: In October 2004, Congress authorized the Internal Revenue Service (IRS)  to create the private debt collection (PDC) program to collect some  portion of the tax debt inventory. The program enabled IRS to collect tax  debts by contracting with private collection agencies (PCA) and paying  them out of a revolving fund of the revenue collected. Some in Congress  and elsewhere in the tax community, including the National Taxpayer  Advocate, expressed concerns that using PCAs might result in higher costs  as compared to increasing IRS resources to collect the debts. Aware of  these concerns, IRS officials said that if Congress authorized the use of  PCAs, IRS would study the comparative performance of PCAs and IRS in  collecting unpaid taxes. The law authorizing the program also required  that IRS create a measurement plan to identify and capture information on  the best practices used by PCAs compared with IRS’s practices to identify  any that IRS could adopt to improve its own collection operations. IRS  began assigning cases to PCAs and began the PDC study in September  2006.
In March 2009, IRS released the PDC study and announced that IRS would  not renew expiring contracts with PCAs. The announcement stated that  IRS had reviewed the PDC program and specifically cited the results of the  study, an independent review of it, and its conclusion that IRS is more cost  effective than PCAs when working similar inventory. The announcement  also noted that IRS anticipated hiring additional collection staff in fiscal  year 2009 and quoted the Commissioner of Internal Revenue as saying, “I  believe this work is best done by IRS employees, and I believe we have  strong support from the Administration and the Congress for increased  IRS enforcement resources going forward.”
You asked us to review the soundness of IRS’s PDC study and address  questions related to changes IRS made since it stopped using PCAs.  Specifically, our objectives were to determine the following:    How sound was IRS’s PDC study as primary support for IRS’s decision to  discontinue contracting out tax debt collection?
What changes has IRS planned or made to its collection approach based  on its PCA experience and the PDC study?
subsections: section_title: Scope and Methodology
paragraphs: To determine the soundness of IRS’s PDC study as primary support for  IRS’s decision to discontinue contracting out tax debt collection, we  reviewed the study report and supporting documents and other data. We  interviewed IRS officials and contractors involved in the study. We also  reviewed the report IRS commissioned to validate the PDC study  conclusion and interviewed contractor staff involved in that effort. We  also interviewed officials from one of the PCA firms IRS contracted with  to obtain their views on the PDC study. We reviewed and summarized  program analysis guidance from various sources in developing our criteria.  Such guidance came from Office of Management and Budget (OMB)  Circular A-94, a previous GAO publication on evaluating federal  programs, an academic research paper on costs and benefits that should  be considered in making decisions on resources for tax enforcement  programs, and accepted quantitative analysis criteria on sampling cases  and projecting results. We compared IRS’s study methodology and report  to the criteria from the various guidance. We also reviewed IRS’s Internal  Revenue Manual (IRM) and interviewed officials to determine if IRS had  guidance on whether and how to conduct and document economic  analyses to support decisions to initiate, renew, or expand programs. We  also reviewed the types of costs and benefits that IRS included in the PDC  study.
To determine what changes IRS has planned or made to its collection  approach based on its PCA experience and the PDC study, we reviewed  program documents and interviewed IRS officials on their processes and  procedures for collecting tax debt. We also compared IRS’s plans for  studying whether to work on PCA-type cases in the future to guidance in  our previous publication on evaluating federal programs and in our  report reviewing IRS’s study of the earned income tax credit.
We conducted this performance audit from November 2009 to September  2010 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Background
paragraphs: In developing the PDC program, IRS officials said that contracting with  PCAs was needed because Congress was unlikely to provide IRS sufficient  staff to attempt collection on the inventory of cases with a lower priority.
Also, officials said that PCAs would pursue cases that IRS staff would not  because IRS had other higher-priority collection cases.
As IRS planned the PDC program, we issued two reports with findings and  recommendations to improve the program.
In May 2004, we reported that IRS’s PDC study approach—comparing PCA  and IRS performance for the same type of simpler cases that would be  sent to PCAs—would provide limited information to judge whether using  PCAs is the best use of resources. In sum, the approach conflicted with  IRS officials’ position that these simpler cases would not be worked by  IRS employees given the higher-priority cases in IRS’s workload. We  recommended instead that IRS compare the use of PCAs to a collection  strategy that officials determine to be the most effective and efficient  overall way of achieving collection goals.  In September 2006, we reported that IRS’s planned PDC study approach  would not meet the intent of our 2004 recommendation. Because the  study would not count the fees paid to PCAs as program costs, it would  not compare the results of using PCAs with the results IRS could get if it  was given the same amount of resources, including the fees to be paid to  the PCAs. We recommended instead that IRS ensure that the study  methodology and the reports on the study include the full costs of the PDC  program, including the fees paid to PCAs and the best use of those federal  funds.
IRS agreed with the recommendations in both reports, and IRS’s actions to  implement them generally are the topic of this report. IRS’s PDC study did  not meet the intent of our 2004 recommendation because it did not  compare using PCAs to what IRS officials determined to be the most  efficient and effective overall strategy. Although IRS met part of our 2006  recommendation in that the PDC study included fees paid to PCAs as  program costs, the PDC study did not fully implement our  recommendation to include the best use of federal funds because, as  discussed further below, the study had methodological errors and a  narrow scope.
IRS’s PDC study report results were released on March 5, 2009, along with  IRS’s announcement that it was ending the program. The study tracked  selected cases assigned to IRS’s Automated Collection System (ACS)  versus PCAs and measured them in terms of cost per dollar collected,  percentage of balance due collected, and percentage of cases in payment  status at the end of the study period. For both PCAs and IRS, the PDC  study counted the dollars collected from full payments or estimated  dollars collected through installment agreements. The study results,  summarized in table 1, showed that IRS performed better in each of the  measures.
On February 3, 2009, legislation was introduced in the House to amend the  Internal Revenue Code of 1986 to repeal the authority of the Secretary of  the Treasury to enter into PDC contracts. The last major action on the  legislation was on February 3, 2009, when it was referred to the House  Committee on Ways and Means.
The Omnibus Appropriations Act of 2009, enacted March 11, 2009,  prevented IRS from using fiscal year 2009 appropriated funds “to enter  into, renew, extend, administer, implement, enforce, or provide oversight  of any qualified tax collection contract (as defined in section 6306 of the  Internal Revenue Code of 1986).” Absent the access to appropriated  funds, IRS funded the administrative costs of the PDC program through its  user fees until IRS was able to end all PDC activities. The Consolidated  Appropriations Act of 2010 placed the same prohibition on IRS’s use of  fiscal year 2010 funds.
section_title: Methodological Errors and Narrow Scope Limit the Study’s Usefulness in Supporting the PDC Decision
paragraphs: According to IRS officials, the PDC study was not originally intended or  designed to be primary support for IRS’s PDC program decision. Even  though other factors, such as potential increases in IRS collection staffing,  were considered, based on our interviews with IRS officials and IRS’s  announcement of the program’s termination, the study results played a  primary role in supporting the decision. IRS officials noted that the  difference in cost per dollar collected using IRS staff and PCAs was so  pronounced that in their view, additional analyses would have been  unlikely to change the decision that was made. Nevertheless, neither we  nor IRS officials know whether the PDC study results and decision on the  program would have differed significantly if the study had been designed  to be primary support for IRS’s PDC program. However, errors in the study  sampling methodology and the study’s narrow scope limit its usefulness in  supporting the PDC decision.
IRS does not have guidance on whether and how to conduct and  document economic analyses to support decisions to initiate, renew, or  expand programs. The IRM, which includes authoritative guidance for IRS  managers, does not include guidance on economic analyses that should  support decisions to initiate, renew, or expand a program.
subsections: section_title: The Study Result May Be Over- or Understated Because the Sample Was Not Generalizable to the PDC Population
paragraphs: Because IRS did not perform certain analyses and documentation is not  available to do those analyses, it is unclear whether the study’s results are  accurate. Guidance for statistical surveys requires that a study’s sampling  procedures be sufficiently described such that policymakers can assess  whether the study’s results can be generalized to the settings and times of  interest. The PDC study documentation mentioned a design that could  represent the full population of PCA-type cases based on sample cases  drawn from various groupings of PCA-type cases. However, IRS did not  generalize the study results back to the full PCA case population, limiting  its analysis to the sample cases. It is common for study results to differ  significantly between results for the sampled cases and results when  generalized back to the full population. IRS did not retain sufficient  documentation on the sample selection and analysis to enable it or others  at this point to calculate estimates for the population, measure the margin  of error, or otherwise ensure the soundness of the sample approach.  Even though IRS documentation referred to a sampling approach, IRS  officials said the study was not intended to provide generalizable results.  The sample cases assigned to the PCAs and IRS for comparative purposes  may or may not have been similar. IRS was concerned about potential  differences between the groups and tested the difference in the average  balance due amount between the cases to be assigned to PCAs and IRS.
The test did not find a difference. However, this was a limited test of only  one variable. IRS could have tested other variables, such as taxpayer filing  history, adjusted gross income, filing status, geographic region, and type of  forms completed by the taxpayer. Differences in any of these variables  could affect taxpayer compliance and payment. Performance differences  between the PCAs and IRS could be due to differences in the composition  of the groups.
section_title: Different Study Objectives and Design Would Have Better Supported the Program Decision-Making Process
paragraphs: Because the PDC study had a narrow objective of comparing the results of  collection efforts by IRS and PCAs for the PCA-type cases, its design did  not consider other factors included in federal and other guidance on  conducting program analyses. An objective more directly focused on  providing information to serve as primary support for IRS’s decision on  the PDC program could have led IRS to conduct additional analyses  relevant to the decision. The findings from those analyses could have  affected the program decision.
section_title: The Study Provides No Statement of the Program’s Rationale and Analysis of Selected Data Related to It
paragraphs: Guidance on conducting program analysis to determine whether to  continue a program states that the rationale for the government program  being examined should be clearly stated in an analysis. Clearly stating the  rationale for the program can help ensure that the analysis includes  relevant factors and outcome measures, particularly if facts related to the  rationale have changed and therefore need to be included in the analysis.  Stating the rationale of the program and conducting related analyses can  help ensure that variables measured in the study answer the questions  decision makers have about the basic reasons the program exists, the  continuing need for it, and the costs and benefits of continuing, revising,  or ending it. For instance, one rationale for starting the program was that  Congress was unlikely to provide funds for staff to work these cases. The  PDC study did not cover whether this had changed and, if so, whether IRS  would use any portion of additional staffing for PCA-type cases. As  discussed later, the number of staff who would normally work PCA-type  cases has increased in recent years.
section_title: Alternatives Were Considered, but Scaling Back the PDC Program Was Not Among Them
paragraphs: To inform a decision on whether to continue a program and ensure the  best use of federal resources, government officials need information on  the range of alternatives for achieving a program’s objectives at the lowest  costs. Guidance states that program analyses should include alternative  means of achieving program objectives by examining different program  scales, methods of provision, and degrees of government involvement.
The PDC study included three alternatives: (1) PCAs attempting to collect  unpaid tax debts for the inventory assigned to them, (2) IRS taking  collection action on the types of cases that were assigned to PCAs, and   (3) IRS taking collection action on another type of inventory normally not  worked. However, beyond not addressing whether to continue the PDC  program, the study did not analyze alternatives for program scale, such as  expanding the PDC program or scaling it back to a segment of cases that  might be more cost effective for PCAs to work than IRS. For example, in  deciding whether to scale back rather than eliminate the PDC program,  IRS could have analyzed the types of cases, if any, where PDC  performance was better than IRS performance. Assuming that this analysis  pointed to favorable types of cases for PCAs, IRS then could have limited  the program to working those cases, unless the benefit of doing this was  so limited that running a separate PDC program with fixed costs would  not be cost beneficial. According to IRS officials, IRS had effectively  scaled back the program during its course because after the initial wave of  cases assigned to PCAs, IRS had difficulties in obtaining a sufficient  volume of cases appropriate for PCAs to work.
In addition, IRS did not compare the PDC program to what it judged to be  the best overall strategy for improving tax collections, as we had  recommended IRS do in our 2004 report, even though the Commissioner  of Internal Revenue agreed with our recommendation. In 2004, we  concluded that IRS should do a study in line with federal guidance, such as  comparing the results of using PCAs to the results from using the same  amount of funds to be paid to PCAs in an unconstrained manner that IRS  determined to be the most effective overall way of achieving its collection  goals. After our report, Congress authorized the PDC program and  included the provision making 25 percent of PDC collections available to  IRS. Conceptually, Congress effectively made available funds for the PDC  program totaling the amounts paid to PCAs plus the up to 25 percent of  PDC collections that IRS could use for enforcement purposes. We  continue to believe that a comparison of results of the PDC program—as  authorized by Congress—to results IRS would achieve if given the same  funds to use in what it judged to be the best possible manner would have  better supported a decision on the PDC program.
section_title: Not All Benefits and Costs Were Included
paragraphs: To inform a decision on whether to continue a program, government  officials need complete and reliable information on all the program’s  benefits and costs. Guidance for doing economic analyses states that to  the extent possible all benefits and costs should be monetized to provide a  standard unit of comparison. If it is not feasible to assign monetary  values, other quantification of costs and benefits should be done. If  quantification is not possible, at a minimum, analyses should include a  comprehensive listing of the different types of benefits and costs to  identify the full range of program effects. Furthermore, analyses should be  explicit about the underlying assumptions used to estimate future benefits  and costs.
The PDC study measured the government’s cost per dollar of direct tax  revenue collected, percentage of balance due collected, and percentage of  cases in payment status at the end of the study period. As cited in IRS’s  announcement of the decision to not renew PCAs’ contracts, the study  showed that when working the same types of cases as PCAs, IRS had  better results than PCAs. Specifically, IRS’s cost was $0.07 to collect a  dollar itself while the government’s cost using PCAs was $0.24 for each  dollar collected. According to IRS officials, IRS’s results on the cases it  worked were comparable to or better than the results of working its  normal cases.
In addition to direct revenues, the study did not estimate or otherwise  discuss indirect revenue (for example, determine whether other taxpayers  are more or less likely to pay their due taxes when IRS works the cases  than when PCAs do). Indirect revenues are difficult to estimate, but the  study could have explored whether any logical reasons exist to indicate  that indirect revenue would have varied based on which party worked the  PCA-type cases.
As discussed further below, the PDC study also did not include potentially  important program costs and benefits for a tax enforcement program, such  as    costs to taxpayers to comply with requirements to pay their delinquent  taxes (are costs to taxpayers lower under IRS or PCAs);    equity (is IRS or are PCAs more effective in collecting taxes from  taxpayers in different income groups, for example); and    economic efficiency (is IRS or are PCAs more effective at working cases  involving taxpayers in different industries, for example, resulting in more  or less distortion of activity across types of businesses).
IRS officials said that these benefits and costs were not addressed in the  study because they are difficult to measure. However, IRS could have  followed OMB guidance to, at a minimum, list and discuss these omitted  costs and benefits in the final report. As a result, the decision to end the  program rests, in part, on the assumption that important differences do  not exist between IRS and PCA handling of these cases related to these  costs and benefits. If IRS assumed that no such differences existed, this  should have been stated in the report with the supporting rationale.
Some important differences might exist. For example, it is not clear that  taxpayer costs would be the same regardless of which party worked on  collecting their tax debt. Taxpayer compliance costs could be lower when  IRS collects taxes because 90 percent of the cases IRS worked were  eligible for systemic actions, such as systemic levies of taxpayer’s assets.  Such actions might incur little cost for the taxpayer beyond reading IRS’s  notification of intent to levy assets, as compared to a PCA case, which  required that a taxpayer first make a payment or answer or return a phone  call in response to a PCA’s letter or call. Policymakers do not know  whether costs to taxpayers differed between IRS and PCA collections  because IRS did not consider this and other circumstances that could  affect taxpayers’ costs.
The study also did not include a type of collected revenue—which IRS  called noncommissionable revenue—that IRS officials tracked in a  measure to assess the PDC program’s performance and, in some cases,  might have gone uncollected otherwise. For example, IRS did not pay  PCAs a commission when debt was collected within a 10-day window after  being assigned to a PCA. In establishing the program, an IRS official told  us that IRS expected that many dollars would be collected within this 10- day window because taxpayers would send in payment after receiving  notification that their debts were being assigned to a PCA. To the extent  this occurred, not taking these revenues into account in the study may  have led to an underestimate of the collections attributable to the PDC  program. Other noncommissionable revenue included collections through  actions IRS systemically took regardless of whether a case was assigned to  a PCA, such as refund offsets. According to IRS officials, such revenues  were not included in the PDC study for either PCAs or IRS collections. IRS  data through fiscal year 2007 show that total collections for the PDC  program were about $32.1 million with $7 million (22 percent) of the total  being noncommissionable; for fiscal year 2008, the total was $37.3 million  with $9.6 million (26 percent) noncommissionable; and for fiscal year  2009, the total was $28.8 million with $11.5 million (40 percent)  noncommissionable.
section_title: For the Types of Government Costs IRS Chose to Include, the Study Was Consistent with Selected OMB Guidance
paragraphs: IRS’s PDC study focused on just a few types of government costs. Those  costs included IRS’s costs as well as the commissions paid to PCAs. Table  2 identifies these types of costs.
IRS used a variable costing methodology rather than a full costing  approach to compare the costs of PCAs and IRS working inventory that  they determined to be similar. According to IRS, the variable costing  approach included expenses that would vary depending on an increase or  decrease in the inventory assigned to either the PCAs or IRS. The study  did not include previous costs, such as costs in setting up the PDC  program. In general, this overall approach to including costs is consistent  with OMB guidance. That guidance says that analyses should be based on  incremental costs and benefits and sunk costs should be ignored.
IRS did not include management oversight or information technology  since such costs would not vary with the volume of cases handled. For  example, IRS did not include the costs of a PDC program oversight unit  that among other things was responsible for monitoring PCA performance.  Although these costs likely would not vary greatly with changes in the  volume of cases handled by IRS or PCAs, if the absolute costs were  significantly greater either for IRS or PCAs, the difference may have  affected the cost comparison, especially over relatively small numbers of  cases. IRS did not document whether these costs varied significantly  based on who handled the cases.
To determine the cost for the PCAs, IRS used historical cost data of the  PDC program for fiscal year 2008 that were generated from IRS’s  Integrated Financial System. IRS apportioned those costs to the number of  cases in its study.
Unlike the PCAs cost data, the IRS costs were not system generated but  were calculated based on estimates and assumptions. For example, IRS  identified the transactions (such as phone calls made or notices sent to  taxpayers) taken on each case in the sample by manually reviewing  individual data posted in the taxpayers’ case files in the Integrated Data  Retrieval System. To then calculate direct labor hours per case, IRS used  information from inventory handling time reports and call handling time  reports. IRS determined an average minutes per transaction, which it  applied to the sampled cases in the study. Because the PCA-type cases  were at least somewhat different than cases ACS normally handled,  applying the average minutes from typical transactions may or may not  have accurately reflected the time actually taken on the PCA-type cases.  Other estimated costs were for telephone calls and postage costs for  mailing a single installment agreement notice to taxpayers.
section_title: Study Assessed Past Performance, Not Likely Future Performance
paragraphs: Another fundamental program analysis principle is to estimate expected  results of the program. Because decisions about the future of a program  depend on the expected future results of it or any alternatives that are  under consideration, future results are to be included on a discounted  basis in either benefit-cost analysis (the value of expected net benefits, i.e.,  benefits minus costs) or cost-effectiveness analysis (to determine which  alternative has the lowest costs for a given amount of benefits or which  has the greatest benefits for a given amount of costs). Past performance  can be relevant in helping to estimate the value of future results.
IRS’s PDC study included no estimates of future costs and revenues for the  alternatives studied. The study presented data and related ratios, including  costs per dollar collected, only for cases worked in the past. Although the  study text states that the cases studied were like those PCAs worked  during the program and would work in the future, it provided no analytical  basis for this assumption.
Documenting any analyses of future costs and revenues was especially  relevant for any decision on the future of the PDC program because IRS  had previously revised the criteria for selecting PDC cases several times in  order to provide the volume of work to PCAs that had been anticipated.  Accordingly, reasonable questions existed about whether IRS would be  able to continue providing a sufficient stream of work to PCAs and how  that might affect the scope of the program and the nature of the cases  PCAs would work. Absent any such documented analysis, decision makers  and those overseeing the agency had a limited basis on which to assess  whether PCAs likely would have been as effective at working the cases  IRS could deliver in the future as they had been in working past cases.
IRS officials said that they considered future results, but only outside the  study, in deciding to end the PDC program. IRS officials said that during  management meetings on deciding whether to continue the PDC program,  they considered past program performance compared to expectations and  how that might affect future performance. For example, they said that they  considered gaps between IRS’s original expectations, what the program  actually realized, and what the program would likely achieve in the future  given actual program experience. Among other things, IRS officials noted  the declining available case inventory levels for the program. They said  that the program’s actual performance compared to expectations was a  reason that IRS ended the program. IRS was unable to provide  documentation of any analyses of future expected costs and results.  Without such analyses, it is unclear to what extent the program did not  meet expectations, whether IRS determined the underlying reason, and  whether the program’s future performance could have been improved in a  manner that could have affected the decision on the program.
section_title: IRS Has Not Changed Its Collection Approach Based on Its PDC Experience and Study, and Whether IRS Will Work Certain PCA- Type Cases Is Unclear
subsections: section_title: Although the Basis Was Not Documented, IRS Concluded That No PCA Practices Should Be Adopted
paragraphs: In authorizing the PDC program in 2004, Congress required IRS to create a  measurement plan to capture information on successful collection  techniques used by the contractors that IRS could adopt. The PCAs’ best  practices were to be compared with IRS’s collection practices. IRS was to  report on this measurement plan and its results in a mandated biennial  report that was to include specific types of information.
In an unpublished draft biennial report, IRS said it reviewed PCA best  practices and concluded that none of them were sufficiently better than  IRS’s practice to merit adoption. IRS officials provided us a draft version  of the biennial report for 2007. IRS neither finalized the report nor released  it. IRS officials pointed to significant transition in the PDC office during  this time, as well as transitions in the Deputy Commissioner’s and the  Commissioner’s offices. This draft report described IRS’s steps to identify  lessons learned from PCAs. It said that IRS did not find any immediate  opportunities to adopt PCA practices, but provided no details beyond this  sentence. The draft report also said IRS would continue to try to identify  lessons learned.
section_title: Conflicting Information on Whether or How IRS Will Work PCA-Type Cases
paragraphs: IRS officials said that IRS had not changed its criteria to start regularly  selecting PCA-type cases to work because the PDC study results were not  sufficient to identify which of the PCA-type cases could be productively  worked. However, IRS officials said that they were surprised by the study  results, which indicated that IRS staff might have better results working  these cases than some of the cases IRS normally works. IRS officials said  that the types of cases sent to PCAs previously had been considered low  priority because of low potential collection return. In establishing the PDC  program, IRS officials indicated that the PCA contractors would be  working inactive collection cases that IRS collection staff would not be  working. For example, in May 2007, after IRS had sent the initial  inventory of cases to PCAs to work, the Acting Commissioner of Internal  Revenue testified that if the money invested in the PDC program was not  used for it, IRS would not use the funds to work cases that would have  been assigned to PCAs. Rather, the funds would be used to work other  cases considered higher priority in the tax debt inventory.
Because of the surprising result, IRS began a pilot study of the types of  cases that had been worked by PCAs. According to IRS officials, the pilot  study’s goals were to (1) provide coverage for a segment of the unpaid tax  debt inventory that would not be worked because of the end of the PCA  contract and (2) provide information on which types of cases IRS could  fruitfully work in the future.
IRS placed approximately 19,000 pilot cases with ACS over a 4-month  period beginning in September 2009. IRS officials said that the agency  randomly drew the cases from the database of cases that IRS had used to  assign cases to the PCA staff. IRS employees cannot identify which of their  assignments are PCA-type cases. As of July 2010, the status of these cases  was as follows:    8,559 cases (45 percent) had been closed, for example, through some form  of payment or other action;    4,954 cases (26 percent) were being worked; and    5,552 cases (29 percent) had yet to be worked.
As of June 2010, IRS officials said that they expected that the work on  these pilot study cases would be finished by December 2010. For these  cases, IRS collected data on its activities (such as liens and levies) and the  results (such as dispositions and dollars collected). Of the 8,559 cases  closed as of July 1, 2010, IRS had received payment in full in 931 cases (11  percent) and had entered into installment agreements in 2,878 cases (34  percent).
IRS officials said they had planned to use the collected data to determine if  changes should be made to the case selection criteria to assign certain  types of PCA-type cases to the IRS active collection inventory. According  to IRS officials, after the pilot cases were worked, IRS’s Office of Program  Evaluation and Risk Analysis (OPERA) was to study the collected data to  see what happened with the cases. IRS expected that the OPERA study  would take a year to complete (i.e., until about December 2011). IRS  officials said that the agency was then to determine whether more PCA- type cases should be routinely worked in IRS and, if so, which ones.
Following guidance on conducting program analyses helps ensure that  relevant information is considered and a sound methodology is used,  including stated study objectives and a clear and appropriate study design  for answering the objectives. As we have reported in the past, these  principles are essential when developing an analysis or evaluation plan.  For example, this plan should not only be written but have sufficient detail  to answer questions such as the following:    Are the test goals and objectives of IRS’s pilot study clearly stated?    Has IRS identified the types of data to be collected?    What specific types of analysis will be performed?
The documentation IRS had available describing the pilot study—a draft  plan prepared in December 2009—addressed summarizing and providing  data on the sampled cases (such as collection action taken, e.g., liens and  levies) and the results (such as the disposition or current status of the  cases and the dollars collected). But as of June 24, 2010, IRS had not  produced an approved plan on how it intended to analyze the pilot study  data even though it had worked about three-fourths of the cases selected  for the pilot. According to IRS officials, progress in developing an analysis  plan for the pilot study had been delayed by efforts to implement  Consolidated Decision Analytics (CDA). Among other things, CDA is to  implement new models to better predict the collection potential of unpaid  tax debt cases to help ensure the best use of IRS’s collection resources.  Officials said that any plan for analyzing the pilot study’s data would need  to ensure that the pilot study results will be useful given IRS’s plans to  more fully implement CDA in January 2011.
Beyond not knowing how the pilot study data will be analyzed, IRS had  not clarified its criteria on how it would use the results of any analyses to  make a decision about assigning more PCA-type cases to be worked in  IRS. Specifically, IRS had no documentation on the criteria that officials  would use in making this decision and what factors, if any, beyond the  analyses of the pilot study data would contribute to those criteria or that  decision. Developing criteria would be important to ensure that the  variables measured in the study would be useful in supporting the decision  on whether to change case selection criteria to regularly pursue such  cases.
As we concluded our review, the status of the pilot study and whether or  how PCA-type cases would be included in active collection case inventory  became less clear. IRS provided conflicting information about determining  whether PCA-type cases have sufficient collection potential to be included  in its collection inventory. On one hand, officials said that CDA models  were built by tracking the characteristics and collection potential of actual  collection cases. On the other hand, PCA-type cases generally have not  been worked by collection staff, which is why IRS began its pilot study of  PCA-type cases to determine their collection potential. In its comments on  our draft report, IRS said that the CDA models that would be implemented  in January 2011 had overtaken the need to complete the pilot study.  Examining the CDA models was outside the scope of our review.  Therefore, we had not reviewed documentation on what types of data  were used in developing the CDA models. However, in response to our  direct question about whether PCA-type case results were used in  developing the CDA models to be implemented in January 2011, an IRS  official said that they had not been used.
section_title: Staff Who Could Work PCA-Type Cases Have Increased, but IRS Has Not Committed to Working the Cases
paragraphs: IRS’s March 2009 announcement ending the PDC program said that IRS  anticipated hiring more collection staff in fiscal year 2009 and referred to  support from the administration and Congress for increased IRS  enforcement resources. According to IRS officials, IRS’s ACS staff could  work on the PCA-type cases. Based on IRS data, ACS staffing levels—as  measured by full-time equivalent (FTE) positions—have increased since  fiscal year 2008. The ACS staffing levels increased by 308 FTEs from  fiscal years 2008 to 2009 (i.e., 3,395 to 3,703) and were to increase by 94  FTEs during fiscal year 2010 (i.e., to 3,797). These staffing levels account  for hiring done to offset attrition as well as additional hiring beyond  attrition.
The PDC program announcement was unclear on whether additional staff  to be hired would be used for PCA-type cases. On the one hand, the  announcement said that “IRS determined the work is best done by IRS  employees,” which appears to refer to PCA-type work. On the other hand,  the announcement also said that “new employees would give the IRS the  flexibility to make assignments based on the areas of greatest need rather  than filtering which cases can be worked using contractor resources.” IRS  officials told us that the announcement did not imply that the collection  staff to be hired in fiscal year 2009 would be used to work PCA-type cases.
section_title: Conclusions
paragraphs: In comparing IRS and PCAs working the same types of cases in the PDC  study, IRS was responsive to concerns of some in Congress, the National  Taxpayer Advocate, and others on the study’s comparison. Although a  study that adhered to federal guidance on analyzing programs would have  better informed IRS’s decision on the fate of the PDC program, it is not  possible to know whether such a study would have had materially  different results or changed IRS’s decision on the program. However it  would have provided more complete information for policymakers to  consider. To the extent possible, and especially for significant program  decisions, IRS should adhere to OMB and other guidance for economic  analyses to better ensure that policymakers have adequate information to  support their decisions. IRS does not have guidance for managers on the  types of analyses that should be done and documented to support program  decisions. Such analyses can yield significant benefits by helping inform  decision making, but they also incur costs. Therefore, careful  consideration of the potential risks and positive impacts of various study  designs is necessary to select an appropriate study design and scope to  answer the relevant questions in a methodologically sufficient manner.
IRS’s PDC study suggested that at least certain PCA-type cases, which IRS  had not been working, may be worth including in ACS’s inventory for  collection action. To the extent that these results are valid and reliable,  IRS may be able to make a relatively low-cost investment in certain PCA- type cases to collect tax debts. However, as we concluded our audit, IRS  provided conflicting information about determining whether PCA-type  cases have sufficient collection potential to be included in its collection  inventory. Given the conflicting information available to us, we believe it is  important that PCA-type case results are considered and incorporated as  appropriate into the CDA models. If IRS determines that completing the  pilot study is the best method to do so, a documented methodology and  criteria for the study’s analysis could help IRS make a better decision on  which PCA-type cases, if any, should be added routinely to active  collection status.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Commissioner of Internal Revenue take the  following three actions:    Establish guidance on the types of analyses that should be done to support  decisions to initiate, renew, or expand programs. The guidance might refer  to OMB Circular A-94 and, if needed, provide any supplementation specific  to IRS.
Establish a policy requiring documentation for the design, analyses, and  conclusions of studies supporting program changes.
Ensure that PCA-type case results are considered and incorporated as  appropriate into the CDA model.
If IRS determines completing the pilot study is the best means to ensure  that PCA-type case results are considered for the CDA models, the  Commissioner should ensure that the pilot study has a documented  methodology and criteria to guide IRS’s analysis and decision.
section_title: Agency Comments and Our Evaluation
paragraphs: The IRS Deputy Commissioner for Services and Enforcement provided  written comments on a draft of this report in a September 8, 2010, letter,  which is reprinted in appendix I. IRS staff also provided technical  comments, which we incorporated into the report as appropriate.
IRS disagreed with our finding that its PDC study was not soundly  designed to support its decision on whether to continue contracting out  debt collection. IRS said the study’s comparison of the cost-effectiveness  of PCAs and IRS working similar cases provided meaningful data that  aided its decision making. IRS cited an independent review of the PDC  study that found the results to be reasonable, even though the study had  limitations and constraints. We continue to believe that the study was not  a soundly designed cost-effectiveness comparison for supporting IRS’s  decision. Our report discusses our reasoning in detail, focusing on the  study’s methodological errors, narrow scope, and lack of adherence to  guidance for doing such studies. For example, IRS did not do the analysis  necessary to generalize the study results to the full PCA case population  even though study results could differ significantly when generalized to  the full population. Our meetings with staff who performed the  independent review and our analyses of their documentation did not  change our finding about IRS’s study.
IRS agreed with our two recommendations dealing with establishing  guidance on analyses to support decisions to initiate, renew, or expand a  program and policies to ensure documentation of such studies. More  specifically, IRS said it would review current guidance and policies and  develop additional guidance where needed.
IRS agreed in principle with our third draft recommendation on ensuring  that a documented methodology and criteria guide IRS’s analysis and  decision on whether to include selected PCA-type cases in its collection  inventory, but said events have overtaken the need to complete the  ongoing study, citing IRS’s plans to implement CDA models in fiscal year  2011. These models are intended to select cases with the best potential for  collection action in one of IRS’s work streams. IRS said that to measure  the impact of the PCA-type cases, as was the plan when the PCA project  was terminated, is no longer necessary.
We had discussed with IRS officials the continued need for the pilot study  when IRS told us in July 2010 that it planned to implement CDA in January  2011. IRS officials, including the Acting Director, Collection Business  Reengineering, said that while CDA selection would focus on collection  potential and not type of case (i.e., PCA-type), the pilot study of  approximately 19,000 PCA-type cases might provide data useful for  improving CDA models. Officials affirmed that they initiated the pilot  study because the PDC study showed that PCA-type cases might have high  collection potential at low cost. Accordingly, our draft report  recommended that IRS document the methodology and criteria for its pilot  study. Information provided in IRS comments on the report and in  response to our subsequent questions suggests that whether and how PCA- type cases may be selected for active collection inventory is uncertain.  Although IRS’s comments on the draft report said that the need for  completing the pilot case study was overtaken by the development of the  CDA models, in separate technical comments IRS officials said they were  continuing to work the pilot cases and provided no indication that they  would stop working them before CDA is implemented in January 2011.  Further, in response to our question about whether PCA-type case results  were used in developing the CDA models, an IRS official said that they had  not been used. In response to IRS’s comments and absent evidence that  CDA will be implemented as planned and that its models will include IRS’s  experience in attempting collection of PCA-type cases, we revised the  third recommendation to better focus on ensuring that PCA-type case  results are considered and incorporated as appropriate into the CDA  models. Further, if IRS determines completing the pilot study is the best  means to ensure that PCA-type case results are considered for the CDA  models, we maintained our recommendation that IRS ensure that the  study has a documented methodology and criteria to guide IRS’s analysis  and decision.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretary of the  Treasury, the Commissioner of Internal Revenue, and other interested  parties. In addition, the report will be available at no charge on the GAO  Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-9110 or brostekm@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributors to this report are listed in appendix II.
section_title: Appendix I: Comments from the Internal Revenue Service
section_title: Appendix II: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Tom Short, Assistant Director;  Ray Bush; George Guttman; Lois Hanshaw; Ronald W. Jones; Veronica  Mayhand; Ed Nannenhorn; Karen O’Conor; and Cynthia Saunders made  key contributions to this report.