Tax Administration: IRS Should Reassess the Level of Resources for Testing Forms and Instructions
Released: 2003-04-11T00:00:00

Letter

Background

Although most organizational components within IRS are involved, the Tax Forms and Publications Division within IRS’s Media and Publications Division of the W&I Division is primarily responsible for creating and improving tax forms, instructions, and other documents. One goal of the Tax Forms and Publications Division is to make tax forms and instructions as clear and understandable as possible. It is divided into three branches— Individual Forms and Publications, Business Forms and Publications, and Tax Exempt/Government Entities and Specialty Forms and Publications. As of late January 2003, 103 persons were assigned to the Tax Forms and Publication Division, including about 15 persons whose primary responsibility was creating, revising, and reviewing individual income tax forms and instructions.

Many tax forms and instructions are revised annually, often with short turnaround times and in response to tax law changes. IRS also periodically reviews tax forms and, when appropriate, schedules them for revision. According to IRS’s estimate, it revised about 450 tax forms and instructions in 2001 that affected individual and business tax returns. In addition to tax law changes, revisions to tax forms and instructions generally reflected procedural changes, legal rulings, and feedback from internal and external stakeholders about the understandability of forms and instructions.

As illustrated in figure 1, the annual tax forms development process generally starts with a review of the current tax forms. IRS’s tax law specialists review the existing forms and instructions to determine what changes, if any, may be needed to reflect tax law changes and other requirements. The tax law specialists consider comments from a variety of sources both within and outside IRS. For example, comments may be obtained from IRS customer service staff in toll-free call centers who answer calls from taxpayers and have firsthand knowledge of particular forms or instructions that were confusing to taxpayers. IRS’s Taxpayer Advocate Service staff may also provide comments useful to the tax law specialists. The Tax Forms Coordinating Committee, comprised of representatives from all of IRS’s key components, the Department of the Treasury, and IRS’s Chief Counsel, reviews draft forms to help ensure that they are not overly burdensome and that they conform to legal and technical requirements. Draft forms are generally posted to IRS’s external website so that external stakeholders and taxpayers may review and comment on them.

The Office of Management and Budget (OMB) is responsible for approving each form once every 3 years. The purpose of OMB’s approval is to assess IRS’s compliance with the Paperwork Reduction Act, which, among other things, requires agencies to assess the extent of burden the information they collect imposes on the public. Under the Paperwork Reduction Act, OMB must approve new forms and major revisions to existing ones. After OMB’s approval, the forms and instructions are sent to IRS’s vendors to be printed. Generally, IRS needs to have approved forms ready for printing by early October to ensure that they can be printed and distributed to the public when the tax filing season starts the following January.

IRS Tested Five Tax Forms and Instructions Over 5 Years

As shown in table 1, IRS used taxpayers or IRS employees to test the clarity of five individual income tax forms and instructions from July 1997 through June 2002. IRS relied primarily on focus groups to do the testing. Contractors using private citizens did three of the tests (i.e., Earned Income instructions, Child Tax Credit instructions, and Schedule D) and IRS did the other two using IRS employees.

The two testing methods that IRS used—focus groups and one-on-one interviews—are among the commonly used methods to obtain data from individuals on whether documents such as forms and instructions are clear and understandable. Focus groups generally consist of a small number of participants—about 8 to 12 persons—-and are usually selected and organized around the focus group topic. Focus groups are a form of group interviewing that relies on interaction within the group to obtain the impressions of a group of people but not necessarily the impressions of each participant. One-on-one interviews, which aim to obtain individual attitudes, beliefs, and feelings, are used to probe individuals about specific difficulties they may have with completing a form or reading instructions. In some instances, these methods may be used in combination depending upon the particular circumstances of the test. As noted in table 1, IRS used both focus groups and one-on-one interviews in testing the Schedule D form.

Testing Written Documents Helps Ensure Clarity and Benefits Taxpayers and IRS

Testing written documents such as tax forms and instructions helps ensure they are clear, thereby benefiting taxpayers and IRS. Researchers from three federal agencies and a private research firm said testing leads to clearer documents as well as more accurate responses. Our guidance on developing and using questionnaires also recommends testing prior to distribution. IRS’s experience indicates that testing likely improves the clarity of tax forms and instructions and may therefore help reduce the number of errors taxpayers make. Although limited data were available on the costs and benefits of testing IRS forms and instructions, recent changes to Earned Income Credit (EIC) and Child Tax Credit forms illustrate the potential for benefits to significantly exceed the costs of testing.

Experiences of Three Federal Agencies, a Private Research Firm, and IRS Support Use of Testing

Researchers from the National Center for Health Statistics (NCHS), the Census Bureau, and the Bureau of Labor Statistics (BLS) said that testing helps ensure that their documents are clear, and therefore users are more likely to understand them and complete them correctly. Consequently, researchers from the agencies said they routinely test written documents, such as forms or surveys, prior to public distribution. A representative of a private research firm also said that testing ensures that documents communicate clearly and that his firm and many others perform such testing for a wide variety of private and public clients.

The researchers from the federal agencies told us that the benefits of testing are difficult to quantify and, in some instances, may not be quantifiable. NCHS officials stated that the experience they have gained over time from testing and revising documents has helped them develop clearer forms from the outset, a benefit that may not be quantifiable. A Census researcher also agreed that testing is beneficial but difficult to quantify. Nevertheless, the Census researcher said that testing documents prior to public use helps ensure that they are clear and understandable, which gives the agency a greater chance of receiving accurate responses and which lessens the need for follow-up interviews. Similarly, a BLS researcher noted that, though difficult to quantify, testing benefits the agency by reducing errors made by respondents and those reductions can result in savings of time, money, and effort for BLS. See appendix II for additional information on the use of testing in these three agencies.

The representative of the private research firm also provided some perspective of its experiences in conducting tests for public and private sector customers. This firm, like many similar companies, provides a variety of testing and data collection services to public and private sector customers. The firm arranges and conducts focus groups, one-on-one interviews, and other tests in order to ensure the clarity of forms, instruction manuals, surveys, and Web sites among other things. Focus groups generally involve 12 participants and cost around $6,700 to $7,600, excluding costs to develop the item to be tested but including incentive pay for participants that could range from $25 to $50 per participant. These costs may be higher, the representative said, if the participants come from special groups. For example, if the participants are medical doctors, incentive pay could be as high as $250 per participant.

The researcher also said that private firms vary in how much they spend to test their documents. He estimated that firms generally spend between $300,000 and $500,000 to ensure that a form or document is clear and will meet the firm’s needs. In some cases, firms spend more than $500,000 to do a series of tests, making revisions between each test, before they arrive at a final version of a form or document. He also added that most firms do not spend at levels that would allow them to test all their forms and documents.

While maintaining that testing is beneficial, researchers also stated that testing is not fail-safe. It can help identify particular parts of a form that are not clear, researchers said, but it cannot ensure that subsequent changes to the form will entirely resolve the clarity issue. In addition, testing may identify problems participants have in completing written documents, but the participants’ problems may be related to other issues, such as poor math skills, rather than confusing or unclear documents.

Testing questionnaires before distribution is also recommended as a quality assurance measure in our guidance on developing and using questionnaires. According to our guidance, testing questionnaires before they are used is one of the best ways to ensure that the document actually communicates what it was intended to communicate and that users will uniformly interpret it. Testing increases the likelihood that respondents will provide the information needed and helps to alleviate inaccurate responses. Our guidance is also consistent with professional literature on survey design. According to professional literature, “reducing measurement error through better question design is one of the least costly ways to improve survey estimates. For any survey, it is important to attend to careful question design and pretesting.”

IRS’s recent limited experience with testing indicates that testing may help ensure the clarity of tax forms and instructions. In 1999, IRS revised the forms and instructions related to the EIC and the Child Tax Credit, tested the revised forms and instructions, and revised them again based on the test results. The following year, when the revised and tested forms and instructions were used by taxpayers, the error rates for the EIC and the Child Tax Credit decreased by 28 and 35 percent, respectively, as shown in table 2.

IRS officials told us they attribute part of the decrease in EIC errors to a new approach officials developed for structuring EIC forms and instructions and part to the improvements in the draft documents that resulted from testing the revised forms and instructions. Before the EIC forms and instructions were revised, the instructions included a definition and example describing a qualifying child that taxpayers had to interpret.

Incorrectly claimed qualifying children have been a major source of EIC errors. IRS revised the instructions so that taxpayers would answer a series of “yes/no” questions to determine if they have a qualifying child instead of relying on their interpretations of the definition or example of a qualifying child. They then tested the old format and the new format. The number of errors decreased substantially when taxpayers used the new format. IRS then made some final changes to clarify the instructions based on test results. In IRS officials’ opinions, the “yes/no” format made it clearer for taxpayers to determine if they had a qualifying child.

Benefits of Testing Forms and Instructions Can Far Exceed IRS’s Costs

The benefits of testing some changes to IRS’s forms and instructions can considerably exceed IRS’s costs to do tests, especially because so many taxpayers can be affected by improvements in clarity that may result from testing. IRS’s contract costs, including travel, for testing changes to the EIC and Child Tax Credit forms and instructions were about $56,000 and these costs may have been offset within IRS alone in the year that the change was implemented. More significantly, if testing changes to forms and instructions for these credits led to a 1-minute reduction, on average, in the time taxpayers needed to understand and complete the forms during the 2000 tax filing season, affected taxpayers would have saved 240,000 hours worth $1.2 million valued at minimum wage.

Taxpayers and IRS Can Realize a Wide Range of Benefits and Costs From Testing

Testing has the potential to yield a wide range of benefits to taxpayers and IRS. Table 3 summarizes some of the potential benefits that could result if testing helps clarify tax forms and instructions. If the form or instruction that has been revised and tested remains unchanged, some potential benefits could recur annually for the life of the form or instruction.

From the taxpayer’s perspective, benefits from testing could include avoiding the burdens associated with (1) interacting with IRS if they make mistakes due to unclear forms and instructions and (2) understanding and complying with unclear forms and instructions. From IRS’s perspective, benefits are generally in the form of opportunities to use its resources better serving other taxpayers and enforcing the tax laws.

Because IRS makes many changes to forms and instructions affecting individual taxpayers every year, ranging from very simple to more complex changes, the benefits can vary according to the type of changes made. Some forms or instructions may change simply to update certain dollar thresholds based on inflation and these changes may be unlikely to be confusing or unclear to taxpayers. However, in other cases changes may introduce new requirements or concepts to taxpayers, such as when new rules are established through legislation or regulation. Changes intended to address situations like these may be more likely to be confusing or unclear to taxpayers, which could result in a burden on taxpayers to understand their obligations in preparing their tax forms and, possibly, to errors that lead to subsequent interactions with IRS to correct their returns.

To the extent that a form or instruction is unclear and the lack of clarity leads to taxpayer errors, the method IRS uses to detect the errors can affect the costs IRS incurs as a consequence. If a taxpayer’s error can be detected by IRS and corrected under its “math error” procedures, which rely extensively on automated processes, the cost to IRS to correct the error is likely to be small. On the other hand, if unclear forms or instructions lead to compliance errors that are detected and addressed through audits conducted through the mail, in IRS offices, or in the taxpayer’s location, the costs to IRS are likely to be higher in part because these processes are more labor intensive.

The burden and costs taxpayers might avoid if testing helps clarify forms and instructions, and thereby helps taxpayers avoid errors, can vary substantially just as IRS’s costs can vary. In general, because taxpayers need only respond if they disagree with an IRS notice stating that it has corrected an error under its math error procedures, the taxpayer’s burden and cost are likely to be lower than if IRS contacts the taxpayer as part of an audit since audits require taxpayer responses and reviews of taxpayers’ books and records.

Testing Benefits Can Far Exceed IRS’s Costs

Illustrations we developed of the potential benefits and costs of testing forms and instructions show that at least in some cases benefits can be substantially greater than the costs to IRS to do tests. The benefits of testing to IRS alone can potentially exceed its testing costs in the first year a change is implemented. But, primarily because a small change in the time required of taxpayers to understand their tax obligations can total to a large aggregate benefit, taking taxpayers’ benefits into account can yield total benefits substantially above IRS’s costs.

IRS officials have not attempted to develop quantitative estimates of the benefits to taxpayers and IRS that may result from testing forms and instructions and the costs IRS incurs to achieve those benefits. IRS officials did believe that because taxpayers made fewer errors when using the revised EIC and the Child Tax Credit forms and instructions as shown in table 2, IRS spent less time and money correcting errors related to them. The officials said they could not quantify the cost savings because IRS does not track error correction costs by type of error.

To provide some perspective on the potential magnitude of benefits and costs that may be realized due to testing changes to forms and instructions, we analyzed the changes IRS made to EIC and Child Tax Credit forms and instructions. Our analyses are illustrations and not actual assessments of benefits and costs that were associated with testing these forms and instructions because complete data were not available on the potential benefits and costs. Further, in constructing our illustrations we sought to be conservative in estimating benefits, in part because we did not have information on the full range of costs IRS incurred to test forms and instructions. Our illustrations focus on (1) a narrow set of benefits to IRS alone due to potential reductions in taxpayer errors, (2) those benefits plus certain benefits to the taxpayers from reduced errors, and (3) potential benefits to taxpayers in reduced time to do their taxes. See appendix I for details on the methodology we used in developing our illustrations.

Our first illustration quantifies a narrow set of benefits to IRS alone from testing EIC and Child Tax Credit forms and instructions—that is, the benefits IRS may have realized due to reduced numbers of errors that are handled under its math error procedures. It is likely that to the extent testing contributed to better taxpayer understanding of these two credits, IRS would have obtained other benefits. For instance, because improperly claimed qualifying children is one of the leading causes of the EIC’s high noncompliance rate, if clarified EIC forms and instructions lead fewer taxpayers to improperly claim the EIC, IRS would likely be able to free some of its EIC-related audit resources for other audits or to audit EIC returns that it might otherwise have had insufficient resources to cover. In fiscal year 2002, IRS used about 1,400 full-time equivalent (FTE) staff years for correspondence audits of EIC issues.

Our analysis in table 4 shows the amount by which IRS’s potential cost savings from not having to correct EIC and Child Tax Credit errors may have exceeded its testing costs given differing assumptions about how much testing alone may have contributed to reduced taxpayer errors. As illustrated, IRS would have saved more in cost avoidance (thereby freeing resources to work elsewhere) in the first year of the change alone than it spent on the contracted testing of the forms and instructions if half of the reduction in errors was due to testing. If only 10 or 25 percent of the reduction was due to testing, then IRS would not have saved more than it spent on the testing contract in the first year. However, some of the benefits of a change in forms or instructions continue to be realized in future years. Again, the illustration does not consider other benefits IRS may have realized.

To provide some perspective on how the potential benefits to taxpayers from testing EIC and Child Tax Credit forms could affect the overall benefits and costs of testing, we next looked at potential reduced burden from credit claimants receiving fewer notices due to reduced errors. First, we assumed that on average all taxpayers receiving an error notice from IRS take 2 or 5 minutes to deal with the notice. Based on those assumptions, we calculated the value to taxpayers of the time saved (using minimum wage levels) from not having to deal with IRS error notices. We used the same assumed reductions in errors due to testing that we made for table 4 and we netted taxpayers’ savings with the savings shown in table 4 for IRS alone.

As table 5 shows, including testing-related benefits to taxpayers from decreased errors suggests that in the first year following testing of EIC and Child Tax Credit forms and instructions, the net benefit to taxpayers and IRS combined could have been positive except for our lowest assumption about the degree to which testing may have reduced taxpayer errors—our 10 percent assumption.

Finally, to illustrate the potential benefits if testing EIC and Child Tax Credit forms and instructions made them clearer and thereby reduced taxpayers’ time needed to understand and complete the credit forms, we calculated the value of time saved by taxpayers (using minimum wage levels) in understanding and completing EIC and Child Tax Credit forms assuming the time saved was 1 minute. Unlike for tables 5 and 6, all taxpayers who used the form or instructions to determine whether they qualified for either credit may have saved time if testing contributed to clearer EIC and Child Tax Credit forms and instructions. However, because we did not know how many taxpayers might have used the forms and instructions for this purpose, in calculating the value of time taxpayers may have saved we used only the number of taxpayers who claimed these credits and did not use paid preparers to prepare their tax returns.

Table 6 shows that if testing the credits’ forms and instructions helped clarify them and that led taxpayers to take 1 minute less to understand and complete the forms, credit claimants would have saved a total of about 240,000 hours worth $1.2 million at minimum wage levels.

To provide another perspective on the potential magnitude of benefits and costs associated with testing changes to forms and instructions, we also looked at IRS’s experience with the rate reduction credit. This one-time credit was enacted in June 2001. When 2001 tax returns were processed during 2002, over 8 million returns had errors related to the credit. IRS did not test the instructions for computing the rate reduction credit that was included on the Form 1040 for tax year 2001. According to IRS officials, they did not test the instructions because the credit was a one- time event, and in their judgment, they had insufficient time to test it. We reported that some of the taxpayers’ errors were probably due to taxpayers not understanding IRS’s instructions on how to compute the credit. We also reported that the demand for telephone assistance related to the credit was significant during the 2002 filing season, and that some of these calls, based on Taxpayer Advocate Service information, were made because taxpayers did not understand how to compute the credit.

Using the same approach to illustrate whether IRS alone may have realized benefits in excess of its testing costs as we did for EIC and Child Tax Credit changes, we developed the illustration shown in table 7. As shown, considering only IRS’s cost and assuming that all errors were corrected by IRS using its math error procedures and assuming IRS would have spent the same amount to test the rate reduction credit instructions as it did for EIC and Child Tax Credit tests, IRS may have been able to save between $233,000 and $666,000. Although this case is somewhat atypical since the rate reduction credit affected essentially all individual taxpayers and the number of errors related to the credit was unusually high, these figures illustrate that the potential for savings to IRS alone from testing instructions at times can substantially exceed its testing costs.

However, just as with EIC and Child Tax Credits, taxpayers would have benefited if testing had been done, led to clearer instructions and consequently led to fewer taxpayer errors for the rate reduction credit. Using the same approach we used for EIC and Child Tax Credits, table 8 shows the potential taxpayers’ savings from dealing with fewer rate reduction credit error notices and the net savings to taxpayers and IRS.

Table 9 shows that if testing had been done and it improved the clarity of the instructions enough to save taxpayers, on average, 30 seconds in understanding whether and how they needed to complete the credit line on their tax returns, the savings would have been larger than savings to IRS and taxpayers from avoided errors alone.

IRS Has Not Addressed Constraints to Increased Testing

Although IRS officials said that making greater use of testing to improve clarity of forms and instructions could be beneficial, officials have not addressed the two constraints—time and resources—that they state limit their ability to do more testing of changes to forms and instructions. Time constraints are not binding for some changes IRS considers to forms and instructions, although IRS cannot realistically test the unknown portion of the changes that are due to laws passed shortly before, or even after, the effective dates for the forms. Also, IRS’s procedures for developing and revising forms (1) do not clearly specify which draft version of forms and instructions should be tested with taxpayers or (2) when in the annual forms development cycle testing should occur. In addition to tight time frames, officials also say that limited resources, such as only one person responsible for coordinating all testing efforts in the Forms and Publications Division, preclude them from increasing tests of forms and instructions. However, IRS has not documented which changes to forms and instructions likely would benefit from testing or demonstrated the benefits that are gained when testing is done. IRS’s planning and budgeting process uses such information in determining the level of resources to be allocated to various units.

Tight Time Frames Do Not Always Exist and IRS’s Procedures Do Not Always Facilitate Testing

IRS officials told us that when new tax laws are enacted during the year that require IRS to create or revise tax forms and instructions in time to distribute them to taxpayers by January 1, the start of the tax-filing season, they lack time to test the forms and instructions before distributing them to taxpayers. However, not all changes to forms and instructions are time constrained and IRS’s procedures lack a clear target for which version of forms and instructions should be tested with taxpayers.

While sufficient data were not available to determine the portion of changes IRS makes to forms and instructions that cannot be tested due to time constraints, not all changes are time constrained. Due to the variability in the time that may be required to test a form or instruction and in the amount of time IRS needs to develop the initial form or instruction to be tested, we cannot say definitively when IRS may or may not have sufficient time to conduct tests. In some cases, IRS likely could have sufficient time to do testing when it identifies a needed change to forms or instructions itself since it largely controls the scheduling of this work. Similarly, when the Congress passes a law that is not effective until a future tax year, or that contains provisions that are not effective until a future tax year, IRS may have sufficient time to conduct tests. For example, the Economic Growth and Tax Reconciliation Act of 2001 was passed on June 7, 2001, with some provisions effective for tax year 2001, but others with later effective dates. The provisions modifying education Individual Retirement Accounts were effective for taxable years beginning after December 31, 2001. This gave IRS approximately 16 months to develop and test any modifications to tax forms and instructions and make final revisions before those forms and instructions needed to go to printing for distribution by January 2003. When a law affects the current tax year, i.e., changes how taxpayers will need to calculate their taxes in the next tax-filing season, IRS is less likely to have sufficient time to test. Even in such a case, however, the new law may be passed early enough to allow testing.

IRS’s current procedures for developing and revising forms and instructions do not clearly specify which draft version of forms and instructions should be tested with taxpayers or when in the annual forms development cycle testing should occur. Officials said that draft forms may be tested with taxpayers either before or after they are posted to IRS’s website for external comments by the public, tax practitioners, software developers, and others. Tax Forms and Publications Division officials said that they consider the particular circumstances surrounding the development of each form and instruction when deciding which version of a draft form or instruction they should test. However, because IRS does not have a clear targeted time for testing, IRS’s ability to plan and conduct tests maybe constrained. If IRS’s procedures defined a point in the annual forms development cycle where a version of a draft form or instruction would be available for testing, IRS would be able to establish processes and deadlines designed to ensure that the opportunity for testing is realized.

To the extent that a draft version of a form or instruction is available for testing early in the process, it would give IRS a fuller range of options for testing. For example, if IRS tested draft versions of forms and instructions before or during the approximately 3-week period that the form is available on its Web site, this would minimize any additional calendar time that testing might otherwise add to IRS’s forms development process.

Figure 2 shows the points in IRS’s annual forms development process where testing can occur. As illustrated, testing may be conducted early in the process and late in the process.

Testing earlier drafts of forms and instructions would also enable officials to select from various testing alternatives depending on how early a draft is available for testing. We did not find a uniform amount of time needed to test a change to a form or instruction. At the low end of the spectrum, an official from NCHS said that it takes about 7 weeks to test that agency’s questionnaires using one-on-one interviews. IRS officials estimated that when IRS employees are used as focus group participants it requires about 8 to 12 weeks to schedule and conduct the tests, analyze the data, and prepare a report summarizing the results. IRS officials estimated that when they contract with a private firm to conduct focus groups using private citizens, 24 to 32 weeks are required to obtain a contract, recruit participants, conduct the tests, analyze the results, and prepare a report. This time frame is based on using regular contracting processes involving developing a statement of work, soliciting bids, and selecting a contractor. Contract options exist that enable agencies to identify a firm or group of firms qualified to undertake work so that an expedited task order procedure can be used to select a firm for when needs arise. According to IRS officials, they recently entered into a multiyear contract with two vendors that will enable them to issue task orders when work is needed.

Data Are Not Collected That Could Be Used by IRS Management to Determine the Proper Allocation of Resources to Support Testing

Although IRS’s Tax Forms and Publications Division officials believe current resources are insufficient to support more testing of forms and instructions, they do not have some of the information needed to determine whether to allocate additional resources. This information is not available at least in part because division guidelines and policies do not require that it be gathered.

Officials said that because they have so few staff available to conduct tests and have a limited budget to contract for testing, they could not increase the number of tests they perform. According to the officials, currently only 1 of 103 persons in the division is trained in testing methods. In addition to other duties, this person coordinates the tests for the division, such as the test of EIC forms and instructions completed in 1999 by a private vendor and the test of the innocent spouse application form completed by IRS in 2002. Officials also told us some staff who are primarily responsible for creating and revising tax documents may occasionally assist in conducting tests, such as the three persons involved in testing the innocent spouse form. Officials also said the total budget for contract support for the division was $150,000 in fiscal year 2002, $185,000 in fiscal year 2001, and $130,000 in fiscal year 2000.

As part of its annual planning and budgeting process, IRS management determines what resources will be needed to accomplish strategies and implement programs. IRS’s planning and budget guidance requires that each operating unit prepare a business plan that, among other things, clearly defines priorities and resource requirements. Requests in the business plan for resources must be substantiated with evidence that allocating additional resources is justified.

However, the division does not systematically identify when testing would be beneficial and does not routinely demonstrate the benefits to taxpayers and IRS that have been gained from such testing. Officials do not identify which of the many changes it makes to forms and instructions each year would most likely benefit from testing. Thus, the officials cannot tell IRS management how many opportunities to improve forms and instructions may be lost due to current resource levels. Further, when tests are performed, officials do not identify, quantitatively or qualitatively, the benefits that taxpayers and IRS may have realized.

One reason that IRS does not have data on forgone testing opportunities is that the division lacks formal, written guidelines and procedures for determining when testing would be beneficial. Currently, testing is an optional step in the process for developing forms. IRS’s Tax Forms and Publications officials said that they decide which forms to test based on informal guidelines and procedures and input from officials in IRS’s four operating division program offices and the Taxpayer Advocate Service. The informal guidelines and procedures call for officials to weigh, among other things, whether a form or instruction (a) affects a large number of taxpayers, (b) has a high error rate based on taxpayers’ prior use of the form, (c) is perceived as complex, and (d) will be used for several filing seasons. Also, according to IRS, the amount of time available to perform tests is factored into testing decisions.

These informal guidelines do not require officials to consider in all cases whether testing would be beneficial and to document the decisions made. Accordingly, even if the informal guidelines are applied, and officials judge that some forms or instructions could benefit from testing but cannot be tested due to scarce resources, those decisions are not made systematically and documented.

Further, although the factors the guidelines suggest taking into account appear to have evolved from officials’ experience and therefore should be useful, they do not consider some pertinent factors that could affect the benefits likely to be realized from testing. For instance, the guidelines suggest taking the number of affected taxpayers into account but not the likely amount of burden they would face due to unclear forms or instructions. They also do not clearly call for officials to consider the costs to test forms and instructions and the benefits that may accrue throughout IRS, such as in telephone service centers. In addition, these informal guidelines and procedures automatically exclude testing forms and instructions that will be used only one time. Also, according to IRS officials, the time frame between the passage of new tax laws and when the newly created or revised forms and instructions must be finalized may preclude some forms and instructions from being tested. Even if one-time- use forms meet other testing criteria, such as affecting a large number of taxpayers who may perceive them as complex, IRS will not consider testing them. As the rate reduction credit situation discussed earlier illustrates, such automatic exclusions may not be appropriate in all situations.

IRS officials do not have information on the results achieved when forms and instructions are tested in part because the division does not have policies that require such evaluations. When IRS obtained information on the reduction in error rates following testing of EIC and Child Tax Credit forms and instructions, the studies did not include collecting other information on the benefits that may have resulted for taxpayers and for IRS. For instance, the studies did not estimate the savings IRS may have realized in its telephone and walk-in service due to increased form clarity.

Capturing fuller information on the results of testing would be consistent with IRS’s strategic planning and budgeting process, which emphasizes assessing the impact of current programs to efficiently allocate resources. Further, by evaluating results of testing decisions, IRS officials would be able to determine if their testing guidelines and procedures lead to good decisions about when testing is most likely to be beneficial. They may also be able to see if the methods they use to test—for example, focus groups formed by IRS employees or one-on-one interviews with individuals—yield the most effective test results.

Conclusions

IRS continually faces the daunting task of developing and revising tax forms and instructions to administer our ever-changing set of federal tax laws. Taxpayers rely on IRS for forms and instructions that are as clear and easy to understand as possible given the complexity of the tax laws and providing clear materials is a key goal of IRS’s Tax Forms and Publications Division.

In attempting to meet this goal, IRS has tested an average of one set of forms and instructions each year over the last 5 years. In contrast, officials from three federal agencies that routinely collect information from the public say that testing documents for clarity before using them is their standard practice. They do so because they believe testing will ensure that their data collection documents are clear and that individuals will understand them and complete them accurately.

Although it is difficult to gauge how much testing alone contributes to the clarity of tax forms and instructions, IRS officials believe testing has contributed to significant declines in taxpayer errors. Illustrations we developed based on IRS’s experience in testing forms and instructions suggest that IRS can completely recover its testing costs in the first year following testing in some circumstances and that when savings to taxpayers from more understandable forms and instructions are considered, total benefits even in the first year following tests can be several times IRS’s testing costs.

Although they recognize that testing is beneficial, officials say time constraints and limited resources preclude more testing. However, IRS’s procedures do not clearly specify when draft versions of forms and instructions should be available for testing. Having a clearly defined point where testing would be performed would facilitate establishing procedures and deadlines to better ensure that testing could be done even within IRS’s annual forms update cycle. Further, IRS officials do not have information that would help IRS management to determine whether to allocate additional resources to support enhanced testing. Because IRS lacks standard written procedures for testing, officials have not documented cases where testing would likely be beneficial and have not demonstrated the benefits that are gained from testing.

Recommendations for Executive Action

Because testing could potentially yield clearer and more understandable tax forms and instructions, thereby producing benefits both to taxpayers and IRS, we recommend that the Acting Commissioner of Internal Revenue take the following actions.

Develop written criteria for determining which changes to tax forms and instructions should be tested with taxpayers before publication.

Develop official written guidance that incorporates those criteria and ensure that the guidance requires staff that develop new or revised forms and instructions to document which changes would merit testing and why.

Clarify procedures by designating when in the annual forms development process that a draft version of forms and instructions should be available for testing with taxpayers.

Ensure that an appropriate range of evaluations are conducted of tests that are performed to better establish the costs and benefits of performing tests and to refine IRS’s approach to testing on the basis of lessons learned.

Use information gained from documenting when changes to forms or instructions likely would be beneficial and from evaluations of tests to reassess an appropriate level of resources to perform testing.

Agency Comments and Our Evaluation

The Acting Commissioner of Internal Revenue provided written comments on a draft of this report in an April 7, 2003, letter, which is reprinted in appendix III. The Acting Commissioner agreed with our recommendations.

We are encouraged that IRS plans to implement all but one of our recommendations in time for the 2004 forms development cycle. Understandably, the remaining recommendation to ensure that an appropriate range of evaluations is conducted of tests would take more time to put into practice. The Acting Commissioner also provided additional comments and observations on our draft report.

The Acting Commissioner commented that the crux of our report is that we do not believe IRS has performed adequate testing on new and revised tax forms and instructions due to a lack of resources. He said that resources for testing forms and instructions have been adequate for the testing IRS wanted to perform. While not questioning whether resources were adequate for the testing IRS performed, we concluded that IRS officials do not have information needed to determine the level of resources that should be allocated to testing forms and instructions. Accordingly, we recommended that IRS systematically identify opportunities to improve forms and instructions through testing and to evaluate the costs and benefits when testing is done.

Although agreeing that testing is beneficial, the Acting Commissioner also said that there are significant staff costs associated with testing that are not included in our cost analysis. We recognize that our analysis excluded staff costs and as stated in our draft report we sought to be conservative in estimating the benefits of testing, in part because we did not have information on the full range of costs IRS incurs when undertaking projects to tests forms and instructions. During the course of our work, we requested estimates of staff costs for testing but IRS was unable to provide them. Nevertheless, at least in the cases we illustrated the potential benefits of testing were so much greater than the costs that including staff costs likely would not have substantially changed the results of our illustrations.

The Acting Commissioner expressed concern about whether IRS could have forms and instructions ready for the filing season if testing was done late in the forms development cycle as shown in our figure 2 depicting IRS’s process. We agree with the Acting Commissioner’s concern; however, our figure shows the various points at which testing can occur in IRS’s current processes based on interviews with IRS officials and the documentation they provided us. As IRS implements our recommendation to clarify when testing should be done, selecting a point as early as possible would help maximize the number of changes that can be tested during the annual forms update cycle.

The Acting Commissioner also said that he disagreed with our conclusion that IRS’s experience with obtaining feedback on its products is limited or recent. He said IRS uses various methods to obtain customer feedback. We agree that IRS uses methods other than testing to obtain feedback on its forms and instructions. However, our report describes the potential benefits and costs of testing as a feedback method. In terms of testing, IRS has only tested five forms and instructions during July 1997 through June 2002; in our view, this is a limited number of tests that were conducted during the recent past.

The Acting Commissioner also disagreed that testing would result in reduced demand for walk-in and toll-free assistance. He said that IRS lacks data to support such a conclusion and, based on its experience, new forms generate requests for assistance and error rates on them tend to be higher. We recognize that there will always be a demand for taxpayer customer assistance. However, we believe that reduced demand for assistance is a potential benefit of testing. We note, for example, that IRS officials seek input from telephone assistors when deciding which forms or instructions need to be clarified, apparently believing that clarifying the forms and instructions may help reduce calls to assistors. Finally, testing is one means for ensuring that even for new forms and instructions requests for assistance and errors made by taxpayers will be minimized.

We are sending copies of this report to the Chairman and Ranking Minority Member of the House Committee on Ways and Means and its Subcommittee on Oversight; the Secretary of the Treasury; the Acting Commissioner of Internal Revenue; the Director of the Office of Management and Budget; and other interested parties. We will make copies available to others on request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

This report was prepared under the direction of Charlie Daniel, Assistant Director. If you have any questions regarding this report, please contact him or me at (202) 512-9110. Key contributors to this report were David Alexander, Christopher Currie, Ronald La Due Lake, Anne Laffoon, Veronica Mayhand, Edward Nannenhorn, and Shellee Soliday.

Appendix I: Objectives, Scope, and Methodology

To determine how often IRS has used taxpayers to test the clarity of new or revised individual income tax forms and instructions, we interviewed officials in its W&I Division’s Tax Forms and Publications Division in Washington, D.C. We sought to obtain (a) an understanding of the process IRS used to develop and revise individual income tax forms and instructions and (b) gather information on the forms and instructions IRS tested using taxpayers for the 5-year period between July 1997 and June 2002. We requested information for a 5-year span to help ensure that the information collected would reflect the amount of testing usually done by IRS. Our work did not include assessing IRS’s processes for developing and revising notices or publications or assessing the clarity of any specific tax forms or instructions.

To obtain insights on the benefits of testing written documents, we interviewed officials from three federal agencies in the metropolitan Washington, D.C., area that perform extensive research and data collection using private citizens. We contacted NCHS, BLS, and the Census Bureau because they have broad experience in conducting tests of the clarity of written documents, such as forms, surveys, and instructions. We also contacted a private research firm that specializes in testing forms and surveys for a variety of public and private sector clients. We also reviewed our own guidance for developing and using questionnaires.

To determine the benefits to taxpayers and IRS of testing tax forms and instructions with taxpayers prior to their use by the public, we took several steps. First, we interviewed IRS officials including Tax Forms and Publications Division officials to obtain information on whether they perceived testing as being beneficial to taxpayers and IRS. We also interviewed IRS’s W&I Research Division officials in Indianapolis to obtain their views on whether testing tax forms and instructions benefits taxpayers and IRS. The research division officials, among other things, collect information on taxpayer errors that the Tax Forms and Publications Division uses when deciding which forms and instructions to test. These officials provided us with data on changes in error rates for EIC and Child Tax Credit forms and instructions before and after IRS revised and tested them.

Second, we developed illustrations of the potential benefits and costs of IRS’s testing EIC and Child Tax Credit forms and instructions by analyzing the data on the changes in error rates obtained from IRS research officials. To construct our illustrations, we used readily available data and made certain assumptions. Data were not available on many of the potential benefits of the changes and on the full range of costs IRS incurred to conduct the tests. We developed similar illustrations of potential benefits to taxpayers and IRS if the rate reduction credit instructions had been tested. In all cases, the illustrations we developed are not actual assessments of the costs and benefits that were associated with testing forms and instructions, or that would have resulted if testing had occurred. In developing the illustrations we sought to be conservative in estimating benefits, in part because we did not have information on the full range of costs IRS incurred to test forms and instructions. Because we had to make various assumptions, our illustrations undoubtedly vary from actual costs and benefits.

To determine the potential cost savings to IRS of testing these forms and instructions, we first estimated IRS’s costs to correct a taxpayer error. Our estimates of IRS’s cost to correct errors were limited to the labor cost associated with correcting errors in IRS’s math error program. IRS provided us the number of FTE staff years for operating the ERS that was used to detect and correct math errors and the number of errors corrected by ERS during fiscal year 2002. We used this information to calculate an average labor cost to correct math errors that we then applied to reductions in EIC and Child Tax Credit errors as well as to error reductions that might have resulted from testing the rate reduction credit. Total error correction costs may be higher because we did not include a number of other costs associated with correcting errors. For example, we excluded printing and postage costs for the notices sent to taxpayers. IRS notices often cover more than one issue associated with a tax return and data were not readily available to determine what portion of the postage cost might be attributable to EIC or Child Tax Credit issues alone. In addition, the estimate does not include costs such as equipment and rent. We did not specifically test the accuracy of the cost information provided; however, our audits of IRS’s annual financial statements have raised concerns regarding IRS’s ability to identify all costs associated with a given program or activity.

Because data were not available on the extent that testing reduced errors related to EIC and Child Tax Credit forms and instructions, we assumed different percentage reduction rates in errors due to testing these forms and instructions. Our illustrations show potential savings based on assumed percentage reduction in errors attributable to testing of 50, 25, and 10 percent. Using our estimate of IRS’s costs to correct an error and the assumed number of errors eliminated by testing, we arrived at the costs that would have been incurred to correct those errors. The difference between costs to correct eliminated errors and the costs to test is the potential cost savings from testing. Cost savings due to reduced errors likely would not mean reductions to IRS’s budget. Further, savings likely would mean that IRS would provide services to other taxpayers or would pursue other compliance or tax collection activities that it would otherwise have been unable to do.

To determine IRS’s costs to test forms and instructions, we used IRS’s actual contract costs for testing EIC and Child Tax Credits in 1999. IRS spent a total of about $56,000 to test both EIC and Child Tax Credit forms and instructions with focus groups. Although the cost of contracted support for testing each form and instruction individually likely would have been somewhat lower than this, we applied the total cost in each case. Because IRS could not provide data, we did not include the costs associated with IRS letting and managing the contract or the cost for Tax Forms and Publications Division staff to work with the contractor in conducting and managing the tests or the cost of any other IRS staff that were involved in these tests.

To determine the potential benefits to taxpayers from testing EIC and Child Tax Credit forms and instructions, we calculated estimated values to taxpayers of the time saved if testing improved form and instruction clarity, thereby reducing taxpayer errors and the burden of dealing with IRS error notices. We also estimated taxpayers’ time saved if testing reduced the time needed to understand and complete tax forms and instructions. We developed similar illustrations for potential benefits to taxpayers if the rate reduction credit had been tested. Our benefit illustrations were based on a series of assumptions. For example, to estimate the value to taxpayers of time saved from not having to deal with an IRS notice, we applied the minimum wage rate that was in effect after IRS revised and tested the forms and instructions to estimates of time taxpayers might save by not having to deal with an IRS notice. For these illustrations we assumed, on average, that taxpayers who received an IRS error notice might spend either 2 or 5 minutes to deal with it. Data were unavailable on how much time taxpayers actually spend dealing with IRS’s notices; however, according to Taxpayer Advocate Service information, IRS’s notices are difficult for taxpayers to understand. Further, taxpayers who decide to contest an IRS notice may take time to call or write letters to IRS or to contact and work with a tax preparer. Data were not readily available to determine what portion of taxpayers who received an EIC or Child Tax Credit error notice contested IRS’s change to their tax returns.

For our illustrations of the estimated value of time saved by taxpayers if testing reduced the time needed to understand and complete EIC and Child Tax Credit forms and instructions, we assumed that taxpayers would save on average 1 minute by using clearer forms and instructions. We excluded from our illustrations those tax returns prepared by preparers because the taxpayers might not have had to read and understand the forms and instructions. Because data were not available on the number of tax returns claiming the Child Tax Credit that were prepared by paid preparers, we reduced the number of returns claiming the credit by the same percentage of EIC returns that were prepared by preparers. The percentage of EIC claimants using paid preparers exceeds the average for all taxpayers. For the rate reduction credit, our illustration is based on a 30-second time savings and the number of taxpayers who filed on paper and did not use preparers. We chose a 30-second potential savings for this illustration because many taxpayers would have had to read only part of the instructions to determine what to do. We aggregated the times for all taxpayers and multiplied the total hours saved by the prevailing minimum wage rate to arrive at estimated benefits to taxpayers.

To determine whether any factors limited IRS’s ability to use individual taxpayers to test forms and instructions and, if so, how these factors can be addressed, we interviewed IRS’s Tax Forms and Publications Division’s officials and analyzed supporting data they provided us. Regarding officials’ view that they lacked sufficient time to do more testing, we reviewed information on the amount of time IRS, NCHS, and the private research firm we contacted took to perform various types of tests. We also reviewed IRS’s process for developing new and revised forms and instructions and determined how many weeks were available between the dates that various laws were enacted or their provisions became effective and IRS’s normal October 1st deadline for printing. Finally, regarding officials’ view that they lacked sufficient resources to do more testing, we obtained information on the resources available within IRS for testing.

We performed our work from May 2002 through March 2003 in accordance with generally accepted government auditing standards.

Appendix II: Use of Testing in NCHS, Census, and BLS

Like IRS, the three federal agencies we contacted create written documents to be completed by the public. The agencies create documents such as forms, surveys, and questionnaires that they use to collect information from the public to fulfill their missions. Unlike IRS, described below, these agencies routinely test their forms, surveys, and questionnaires prior to distribution to the public.

The National Center for Health Statistics, the nation’s principal health statistics agency, compiles statistical information to guide actions and policies relevant to public health and health policy. According to NCHS, obtaining accurate and usable health information is crucial to successfully fulfilling its mission to provide reliable information to the Centers for Disease Control and Prevention and the Department of Health and Human Services. NCHS collects information through various sources including questionnaires that it develops and administers. Researchers at NCHS told us that in support of their research they administer surveys and questionnaires each year in addition to developing questionnaires used by the Centers for Disease Control and Prevention. They test each questionnaire using one-on-one interviews. When documents pertain to a particular rather than a general population, the researchers recruit participants with characteristics similar to those persons who might be completing the forms or questionnaires. For example, researchers recruited asthmatics to test a questionnaire related to asthma. NCHS tests forms in one-on- one settings in which a participant may be asked to work through a form while a moderator observes and then later interviews the participant. This approach allows the researcher to identify specific points at which the forms were confusing or problematic and learn why the participant had difficulty. NCHS prefers to use one-on-one interviews when conducting tests because this method closely resembles the ways in which individuals will be completing the documents since individuals will likely complete the documents by themselves.

The Census Bureau is the principal agency responsible for collecting and providing data about the people and the economy of the United States. An accurate census is important because census results are used to reapportion seats in the House of Representatives, redraw congressional districts and other political boundaries, and address countless other public and private data needs. The Census Bureau collects information through short-form and long-form questionnaires that it develops, tests, and administers. In preparation for the 2000 Census, the Congress budgeted millions of dollars to develop and test questionnaires during the 1990s. The Census Bureau’s policy requires that demographic survey questionnaires be tested. It has used focus groups and one-on-one interviews to test its questionnaires and forms. For example, in fiscal year 1996, the Census Bureau decided to make fundamental changes to the traditional census design such as shortening census questionnaires. In that year, it budgeted funds to test, among other things, respondents’ understanding of race and ethnicity questions. The Census Bureau has also conducted detailed cost-benefit analyses of alternative designs; in 1992 it tested the simplified questionnaire in order to gauge whether the new form would increase response rates and reduce costly follow up with households that did not respond to the census.

The Bureau of Labor Statistics, the principal fact-finding agency for the federal government in the broad field of labor economics and statistics, also depends on clear and understandable written documents to collect accurate information from the public. According to BLS’s policy, testing documents such as forms and surveys prior to use by the general public should be undertaken to help identify factors that may impede users’ ability to understand forms or surveys. Then these factors can be addressed in order to improve the clarity of written documents and increase the accuracy of responses. Testing should be done in the early stages of document development so that any problems with clarity can be identified early. BLS routinely tests its written documents using focus groups and one-on-one interviews, and uses the results of the tests to make improvements to the documents.

Appendix III: Comments from the Internal Revenue Service