Background

Although most organizational components within IRS are involved, the  Tax Forms and Publications Division within IRS’s Media and Publications  Division of the W&I Division is primarily responsible for creating and  improving tax forms, instructions, and other documents.  One goal of the  Tax Forms and Publications Division is to make tax forms and instructions  as clear and understandable as possible. It is divided into three branches— Individual Forms and Publications, Business Forms and Publications, and  Tax Exempt/Government Entities and Specialty Forms and Publications.  As of late January 2003, 103 persons were assigned to the Tax Forms and  Publication Division, including about 15 persons whose primary  responsibility was creating, revising, and reviewing individual income tax  forms and instructions.
Many tax forms and instructions are revised annually, often with short  turnaround times and in response to tax law changes. IRS also periodically  reviews tax forms and, when appropriate, schedules them for revision.  According to IRS’s estimate, it revised about 450 tax forms and  instructions in 2001 that affected individual and business tax returns. In  addition to tax law changes, revisions to tax forms and instructions  generally reflected procedural changes, legal rulings, and feedback from  internal and external stakeholders about the understandability of forms  and instructions.
As illustrated in figure 1, the annual tax forms development process  generally starts with a review of the current tax forms. IRS’s tax law  specialists review the existing forms and instructions to determine what  changes, if any, may be needed to reflect tax law changes and other  requirements. The tax law specialists consider comments from a variety of  sources both within and outside IRS. For example, comments may be  obtained from IRS customer service staff in toll-free call centers who  answer calls from taxpayers and have firsthand knowledge of particular  forms or instructions that were confusing to taxpayers. IRS’s Taxpayer  Advocate Service staff may also provide comments useful to the tax law  specialists. The Tax Forms Coordinating Committee, comprised of  representatives from all of IRS’s key components, the Department of the  Treasury, and IRS’s Chief Counsel, reviews draft forms to help ensure that  they are not overly burdensome and that they conform to legal and  technical requirements. Draft forms are generally posted to IRS’s external  website so that external stakeholders and taxpayers may review and  comment on them.
The Office of Management and Budget (OMB) is responsible for approving  each form once every 3 years. The purpose of OMB’s approval is to assess  IRS’s compliance with the Paperwork Reduction Act, which, among other  things, requires agencies to assess the extent of burden the information  they collect imposes on the public. Under the Paperwork Reduction Act,  OMB must approve new forms and major revisions to existing ones. After  OMB’s approval, the forms and instructions are sent to IRS’s vendors to be  printed. Generally, IRS needs to have approved forms ready for printing by  early October to ensure that they can be printed and distributed to the  public when the tax filing season starts the following January.

	IRS Tested Five Tax Forms and Instructions Over 5 Years

As shown in table 1, IRS used taxpayers or IRS employees to test the  clarity of five individual income tax forms and instructions from July 1997  through June 2002.  IRS relied primarily on focus groups to do the testing.  Contractors using private citizens did three of the tests (i.e., Earned  Income instructions, Child Tax Credit instructions, and Schedule D) and  IRS did the other two using IRS employees.
The two testing methods that IRS used—focus groups and one-on-one  interviews—are among the commonly used methods to obtain data from  individuals on whether documents such as forms and instructions are  clear and understandable. Focus groups generally consist of a small  number of participants—about 8 to 12 persons—-and are usually selected  and organized around the focus group topic. Focus groups are a form of  group interviewing that relies on interaction within the group to obtain the  impressions of a group of people but not necessarily the impressions of  each participant. One-on-one interviews, which aim to obtain individual  attitudes, beliefs, and feelings, are used to probe individuals about specific  difficulties they may have with completing a form or reading instructions.  In some instances, these methods may be used in combination depending  upon the particular circumstances of the test. As noted in table 1, IRS used  both focus groups and one-on-one interviews in testing the Schedule D  form.

	Testing Written Documents Helps Ensure Clarity and Benefits Taxpayers and IRS

Testing written documents such as tax forms and instructions helps  ensure they are clear, thereby benefiting taxpayers and IRS. Researchers  from three federal agencies and a private research firm said testing leads  to clearer documents as well as more accurate responses. Our guidance on  developing and using questionnaires also recommends testing prior to  distribution. IRS’s experience indicates that testing likely improves the  clarity of tax forms and instructions and may therefore help reduce the  number of errors taxpayers make. Although limited data were available on  the costs and benefits of testing IRS forms and instructions, recent  changes to Earned Income Credit (EIC) and Child Tax Credit forms  illustrate the potential for benefits to significantly exceed the costs of  testing.

		Experiences of Three Federal Agencies, a Private Research Firm, and IRS Support Use of Testing

Researchers from the National Center for Health Statistics (NCHS), the  Census Bureau, and the Bureau of Labor Statistics (BLS) said that testing  helps ensure that their documents are clear, and therefore users are more  likely to understand them and complete them correctly. Consequently,  researchers from the agencies said they routinely test written documents,  such as forms or surveys, prior to public distribution. A representative of a  private research firm also said that testing ensures that documents  communicate clearly and that his firm and many others perform such  testing for a wide variety of private and public clients.
The researchers from the federal agencies told us that the benefits of  testing are difficult to quantify and, in some instances, may not be  quantifiable. NCHS officials stated that the experience they have gained  over time from testing and revising documents has helped them develop  clearer forms from the outset, a benefit that may not be quantifiable. A  Census researcher also agreed that testing is beneficial but difficult to  quantify. Nevertheless, the Census researcher said that testing documents  prior to public use helps ensure that they are clear and understandable,  which gives the agency a greater chance of receiving accurate responses  and which lessens the need for follow-up interviews. Similarly, a BLS  researcher noted that, though difficult to quantify, testing benefits the  agency by reducing errors made by respondents and those reductions can  result in savings of time, money, and effort for BLS. See appendix II for  additional information on the use of testing in these three agencies.
The representative of the private research firm also provided some  perspective of its experiences in conducting tests for public and private  sector customers. This firm, like many similar companies, provides a  variety of testing and data collection services to public and private sector  customers. The firm arranges and conducts focus groups, one-on-one  interviews, and other tests in order to ensure the clarity of forms,  instruction manuals, surveys, and Web sites among other things. Focus  groups generally involve 12 participants and cost around $6,700 to $7,600,  excluding costs to develop the item to be tested but including incentive  pay for participants that could range from $25 to $50 per participant.  These costs may be higher, the representative said, if the participants  come from special groups. For example, if the participants are medical  doctors, incentive pay could be as high as $250 per participant.
The researcher also said that private firms vary in how much they spend to  test their documents. He estimated that firms generally spend between  $300,000 and $500,000 to ensure that a form or document is clear and will  meet the firm’s needs. In some cases, firms spend more than $500,000 to  do a series of tests, making revisions between each test, before they arrive  at a final version of a form or document. He also added that most firms do  not spend at levels that would allow them to test all their forms and  documents.
While maintaining that testing is beneficial, researchers also stated that  testing is not fail-safe. It can help identify particular parts of a form that  are not clear, researchers said, but it cannot ensure that subsequent  changes to the form will entirely resolve the clarity issue. In addition,  testing may identify problems participants have in completing written  documents, but the participants’ problems may be related to other issues,  such as poor math skills, rather than confusing or unclear documents.
Testing questionnaires before distribution is also recommended as a  quality assurance measure in our guidance on developing and using  questionnaires. According to our guidance, testing questionnaires before  they are used is one of the best ways to ensure that the document actually  communicates what it was intended to communicate and that users will  uniformly interpret it. Testing increases the likelihood that respondents  will provide the information needed and helps to alleviate inaccurate  responses. Our guidance is also consistent with professional literature on  survey design. According to professional literature, “reducing  measurement error through better question design is one of the least  costly ways to improve survey estimates. For any survey, it is important to  attend to careful question design and pretesting.”
IRS’s recent limited experience with testing indicates that testing may help  ensure the clarity of tax forms and instructions. In 1999, IRS revised the  forms and instructions related to the EIC and the Child Tax Credit, tested  the revised forms and instructions, and revised them again based on the  test results. The following year, when the revised and tested forms and  instructions were used by taxpayers, the error rates for the EIC and the  Child Tax Credit decreased by 28 and 35 percent, respectively, as shown in  table 2.
IRS officials told us they attribute part of the decrease in EIC errors to a  new approach officials developed for structuring EIC forms and  instructions and part to the improvements in the draft documents that  resulted from testing the revised forms and instructions. Before the EIC  forms and instructions were revised, the instructions included a definition  and example describing a qualifying child that taxpayers had to interpret.
Incorrectly claimed qualifying children have been a major source of EIC  errors. IRS revised the instructions so that taxpayers would answer a  series of “yes/no” questions to determine if they have a qualifying child  instead of relying on their interpretations of the definition or example of a  qualifying child. They then tested the old format and the new format. The  number of errors decreased substantially when taxpayers used the new  format. IRS then made some final changes to clarify the instructions based  on test results. In IRS officials’ opinions, the “yes/no” format made it  clearer for taxpayers to determine if they had a qualifying child.

		Benefits of Testing Forms and Instructions Can Far Exceed IRS’s Costs

The benefits of testing some changes to IRS’s forms and instructions can  considerably exceed IRS’s costs to do tests, especially because so many  taxpayers can be affected by improvements in clarity that may result from  testing. IRS’s contract costs, including travel, for testing changes to the  EIC and Child Tax Credit forms and instructions were about $56,000 and  these costs may have been offset within IRS alone in the year that the  change was implemented. More significantly, if testing changes to forms  and instructions for these credits led to a 1-minute reduction, on average,  in the time taxpayers needed to understand and complete the forms during  the 2000 tax filing season, affected taxpayers would have saved 240,000  hours worth $1.2 million valued at minimum wage.

			Taxpayers and IRS Can Realize a Wide Range of Benefits and Costs From Testing

Testing has the potential to yield a wide range of benefits to taxpayers and  IRS. Table 3 summarizes some of the potential benefits that could result if  testing helps clarify tax forms and instructions. If the form or instruction  that has been revised and tested remains unchanged, some potential  benefits could recur annually for the life of the form or instruction.
From the taxpayer’s perspective, benefits from testing could include  avoiding the burdens associated with (1) interacting with IRS if they make  mistakes due to unclear forms and instructions and (2) understanding and  complying with unclear forms and instructions. From IRS’s perspective,  benefits are generally in the form of opportunities to use its resources  better serving other taxpayers and enforcing the tax laws.
Because IRS makes many changes to forms and instructions affecting  individual taxpayers every year, ranging from very simple to more  complex changes, the benefits can vary according to the type of changes  made. Some forms or instructions may change simply to update certain  dollar thresholds based on inflation and these changes may be unlikely to  be confusing or unclear to taxpayers. However, in other cases changes  may introduce new requirements or concepts to taxpayers, such as when  new rules are established through legislation or regulation. Changes  intended to address situations like these may be more likely to be  confusing or unclear to taxpayers, which could result in a burden on  taxpayers to understand their obligations in preparing their tax forms and,  possibly, to errors that lead to subsequent interactions with IRS to correct  their returns.
To the extent that a form or instruction is unclear and the lack of clarity  leads to taxpayer errors, the method IRS uses to detect the errors can  affect the costs IRS incurs as a consequence. If a taxpayer’s error can be  detected by IRS and corrected under its “math error” procedures, which  rely extensively on automated processes, the cost to IRS to correct the  error is likely to be small. On the other hand, if unclear forms or  instructions lead to compliance errors that are detected and addressed  through audits conducted through the mail, in IRS offices, or in the  taxpayer’s location, the costs to IRS are likely to be higher in part because  these processes are more labor intensive.
The burden and costs taxpayers might avoid if testing helps clarify forms  and instructions, and thereby helps taxpayers avoid errors, can vary  substantially just as IRS’s costs can vary. In general, because taxpayers  need only respond if they disagree with an IRS notice stating that it has  corrected an error under its math error procedures, the taxpayer’s burden  and cost are likely to be lower than if IRS contacts the taxpayer as part of  an audit since audits require taxpayer responses and reviews of taxpayers’  books and records.

			Testing Benefits Can Far Exceed IRS’s Costs

Illustrations we developed of the potential benefits and costs of testing  forms and instructions show that at least in some cases benefits can be  substantially greater than the costs to IRS to do tests. The benefits of  testing to IRS alone can potentially exceed its testing costs in the first year  a change is implemented. But, primarily because a small change in the  time required of taxpayers to understand their tax obligations can total to  a large aggregate benefit, taking taxpayers’ benefits into account can yield  total benefits substantially above IRS’s costs.
IRS officials have not attempted to develop quantitative estimates of the  benefits to taxpayers and IRS that may result from testing forms and  instructions and the costs IRS incurs to achieve those benefits. IRS  officials did believe that because taxpayers made fewer errors when using  the revised EIC and the Child Tax Credit forms and instructions as shown  in table 2, IRS spent less time and money correcting errors related to  them. The officials said they could not quantify the cost savings because  IRS does not track error correction costs by type of error.
To provide some perspective on the potential magnitude of benefits and  costs that may be realized due to testing changes to forms and  instructions, we analyzed the changes IRS made to EIC and Child Tax  Credit forms and instructions. Our analyses are illustrations and not actual  assessments of benefits and costs that were associated with testing these  forms and instructions because complete data were not available on the  potential benefits and costs. Further, in constructing our illustrations we  sought to be conservative in estimating benefits, in part because we did  not have information on the full range of costs IRS incurred to test forms  and instructions. Our illustrations focus on (1) a narrow set of benefits to  IRS alone due to potential reductions in taxpayer errors, (2) those benefits  plus certain benefits to the taxpayers from reduced errors, and (3)  potential benefits to taxpayers in reduced time to do their taxes. See  appendix I for details on the methodology we used in developing our  illustrations.
Our first illustration quantifies a narrow set of benefits to IRS alone from  testing EIC and Child Tax Credit forms and instructions—that is, the  benefits IRS may have realized due to reduced numbers of errors that are  handled under its math error procedures. It is likely that to the extent  testing contributed to better taxpayer understanding of these two credits,  IRS would have obtained other benefits. For instance, because improperly  claimed qualifying children is one of the leading causes of the EIC’s high  noncompliance rate, if clarified EIC forms and instructions lead fewer  taxpayers to improperly claim the EIC, IRS would likely be able to free  some of its EIC-related audit resources for other audits or to audit EIC  returns that it might otherwise have had insufficient resources to cover. In  fiscal year 2002, IRS used about 1,400 full-time equivalent (FTE) staff  years for correspondence audits of EIC issues.
Our analysis in table 4 shows the amount by which IRS’s potential cost  savings from not having to correct EIC and Child Tax Credit errors may  have exceeded its testing costs given differing assumptions about how  much testing alone may have contributed to reduced taxpayer errors. As  illustrated, IRS would have saved more in cost avoidance (thereby freeing  resources to work elsewhere) in the first year of the change alone than it  spent on the contracted testing of the forms and instructions if half of the  reduction in errors was due to testing. If only 10 or 25 percent of the  reduction was due to testing, then IRS would not have saved more than it  spent on the testing contract in the first year. However, some of the  benefits of a change in forms or instructions continue to be realized in  future years. Again, the illustration does not consider other benefits IRS  may have realized.
To provide some perspective on how the potential benefits to taxpayers  from testing EIC and Child Tax Credit forms could affect the overall  benefits and costs of testing, we next looked at potential reduced burden  from credit claimants receiving fewer notices due to reduced errors. First,  we assumed that on average all taxpayers receiving an error notice from  IRS take 2 or 5 minutes to deal with the notice. Based on those  assumptions, we calculated the value to taxpayers of the time saved (using  minimum wage levels) from not having to deal with IRS error notices. We  used the same assumed reductions in errors due to testing that we made  for table 4 and we netted taxpayers’ savings with the savings shown in  table 4 for IRS alone.
As table 5 shows, including testing-related benefits to taxpayers from  decreased errors suggests that in the first year following testing of EIC and  Child Tax Credit forms and instructions, the net benefit to taxpayers and  IRS combined could have been positive except for our lowest assumption  about the degree to which testing may have reduced taxpayer errors—our  10 percent assumption.
Finally, to illustrate the potential benefits if testing EIC and Child Tax  Credit forms and instructions made them clearer and thereby reduced  taxpayers’ time needed to understand and complete the credit forms, we  calculated the value of time saved by taxpayers (using minimum wage  levels) in understanding and completing EIC and Child Tax Credit forms  assuming the time saved was 1 minute. Unlike for tables 5 and 6, all  taxpayers who used the form or instructions to determine whether they  qualified for either credit may have saved time if testing contributed to  clearer EIC and Child Tax Credit forms and instructions. However,  because we did not know how many taxpayers might have used the forms  and instructions for this purpose, in calculating the value of time  taxpayers may have saved we used only the number of taxpayers who  claimed these credits and did not use paid preparers to prepare their tax  returns.
Table 6 shows that if testing the credits’ forms and instructions helped  clarify them and that led taxpayers to take 1 minute less to understand and  complete the forms, credit claimants would have saved a total of about  240,000 hours worth $1.2 million at minimum wage levels.
To provide another perspective on the potential magnitude of benefits and  costs associated with testing changes to forms and instructions, we also  looked at IRS’s experience with the rate reduction credit. This one-time  credit was enacted in June 2001. When 2001 tax returns were processed  during 2002, over 8 million returns had errors related to the credit. IRS  did not test the instructions for computing the rate reduction credit that  was included on the Form 1040 for tax year 2001.  According to IRS  officials, they did not test the instructions because the credit was a one- time event, and in their judgment, they had insufficient time to test it. We  reported that some of the taxpayers’ errors were probably due to  taxpayers not understanding IRS’s instructions on how to compute the  credit. We also reported that the demand for telephone assistance related  to the credit was significant during the 2002 filing season, and that some of  these calls, based on Taxpayer Advocate Service information, were made  because taxpayers did not understand how to compute the credit.
Using the same approach to illustrate whether IRS alone may have realized  benefits in excess of its testing costs as we did for EIC and Child Tax  Credit changes, we developed the illustration shown in table 7. As shown,  considering only IRS’s cost and assuming that all errors were corrected by  IRS using its math error procedures and assuming IRS would have spent  the same amount to test the rate reduction credit instructions as it did for  EIC and Child Tax Credit tests, IRS may have been able to save between  $233,000 and $666,000. Although this case is somewhat atypical since the  rate reduction credit affected essentially all individual taxpayers and the  number of errors related to the credit was unusually high, these figures  illustrate that the potential for savings to IRS alone from testing  instructions at times can substantially exceed its testing costs.
However, just as with EIC and Child Tax Credits, taxpayers would have  benefited if testing had been done, led to clearer instructions and  consequently led to fewer taxpayer errors for the rate reduction credit.  Using the same approach we used for EIC and Child Tax Credits, table 8  shows the potential taxpayers’ savings from dealing with fewer rate  reduction credit error notices and the net savings to taxpayers and IRS.
Table 9 shows that if testing had been done and it improved the clarity of  the instructions enough to save taxpayers, on average, 30 seconds in  understanding whether and how they needed to complete the credit line  on their tax returns, the savings would have been larger than savings to  IRS and taxpayers from avoided errors alone.

	IRS Has Not Addressed Constraints to Increased Testing

Although IRS officials said that making greater use of testing to improve  clarity of forms and instructions could be beneficial, officials have not  addressed the two constraints—time and resources—that they state limit  their ability to do more testing of changes to forms and instructions. Time  constraints are not binding for some changes IRS considers to forms and  instructions, although IRS cannot realistically test the unknown portion of  the changes that are due to laws passed shortly before, or even after, the  effective dates for the forms. Also, IRS’s procedures for developing and  revising forms (1) do not clearly specify which draft version of forms and  instructions should be tested with taxpayers or (2) when in the annual  forms development cycle testing should occur. In addition to tight time  frames, officials also say that limited resources, such as only one person  responsible for coordinating all testing efforts in the Forms and  Publications Division, preclude them from increasing tests of forms and  instructions. However, IRS has not documented which changes to forms  and instructions likely would benefit from testing or demonstrated the  benefits that are gained when testing is done. IRS’s planning and budgeting  process uses such information in determining the level of resources to be  allocated to various units.

		Tight Time Frames Do Not Always Exist and IRS’s Procedures Do Not Always Facilitate Testing

IRS officials told us that when new tax laws are enacted during the year  that require IRS to create or revise tax forms and instructions in time to  distribute them to taxpayers by January 1, the start of the tax-filing season,  they lack time to test the forms and instructions before distributing them  to taxpayers. However, not all changes to forms and instructions are time  constrained and IRS’s procedures lack a clear target for which version of  forms and instructions should be tested with taxpayers.
While sufficient data were not available to determine the portion of  changes IRS makes to forms and instructions that cannot be tested due to  time constraints, not all changes are time constrained. Due to the  variability in the time that may be required to test a form or instruction  and in the amount of time IRS needs to develop the initial form or  instruction to be tested, we cannot say definitively when IRS may or may  not have sufficient time to conduct tests. In some cases, IRS likely could  have sufficient time to do testing when it identifies a needed change to  forms or instructions itself since it largely controls the scheduling of this  work. Similarly, when the Congress passes a law that is not effective until  a future tax year, or that contains provisions that are not effective until a  future tax year, IRS may have sufficient time to conduct tests. For  example, the Economic Growth and Tax Reconciliation Act of 2001 was  passed on June 7, 2001, with some provisions effective for tax year 2001,  but others with later effective dates. The provisions modifying education  Individual Retirement Accounts were effective for taxable years beginning  after December 31, 2001. This gave IRS approximately 16 months to  develop and test any modifications to tax forms and instructions and make  final revisions before those forms and instructions needed to go to printing  for distribution by January 2003. When a law affects the current tax year,  i.e., changes how taxpayers will need to calculate their taxes in the next  tax-filing season, IRS is less likely to have sufficient time to test. Even in  such a case, however, the new law may be passed early enough to allow  testing.
IRS’s current procedures for developing and revising forms and  instructions do not clearly specify which draft version of forms and  instructions should be tested with taxpayers or when in the annual forms  development cycle testing should occur. Officials said that draft forms  may be tested with taxpayers either before or after they are posted to  IRS’s website for external comments by the public, tax practitioners,  software developers, and others. Tax Forms and Publications Division  officials said that they consider the particular circumstances surrounding  the development of each form and instruction when deciding which  version of a draft form or instruction they should test. However, because  IRS does not have a clear targeted time for testing, IRS’s ability to plan and  conduct tests maybe constrained. If IRS’s procedures defined a point in  the annual forms development cycle where a version of a draft form or  instruction would be available for testing, IRS would be able to establish  processes and deadlines designed to ensure that the opportunity for  testing is realized.
To the extent that a draft version of a form or instruction is available for  testing early in the process, it would give IRS a fuller range of options for  testing. For example, if IRS tested draft versions of forms and instructions  before or during the approximately 3-week period that the form is  available on its Web site, this would minimize any additional calendar time  that testing might otherwise add to IRS’s forms development process.
Figure 2 shows the points in IRS’s annual forms development process  where testing can occur. As illustrated, testing may be conducted early in  the process and late in the process.
Testing earlier drafts of forms and instructions would also enable officials  to select from various testing alternatives depending on how early a draft  is available for testing. We did not find a uniform amount of time needed  to test a change to a form or instruction. At the low end of the spectrum,  an official from NCHS said that it takes about 7 weeks to test that agency’s  questionnaires using one-on-one interviews. IRS officials estimated that  when IRS employees are used as focus group participants it requires about  8 to 12 weeks to schedule and conduct the tests, analyze the data, and  prepare a report summarizing the results. IRS officials estimated that  when they contract with a private firm to conduct focus groups using  private citizens, 24 to 32 weeks are required to obtain a contract, recruit  participants, conduct the tests, analyze the results, and prepare a report.  This time frame is based on using regular contracting processes involving  developing a statement of work, soliciting bids, and selecting a contractor.  Contract options exist that enable agencies to identify a firm or group of  firms qualified to undertake work so that an expedited task order  procedure can be used to select a firm for when needs arise. According to  IRS officials, they recently entered into a multiyear contract with two  vendors that will enable them to issue task orders when work is needed.

		Data Are Not Collected That Could Be Used by IRS Management to Determine the Proper Allocation of Resources to Support Testing

Although IRS’s Tax Forms and Publications Division officials believe  current resources are insufficient to support more testing of forms and  instructions, they do not have some of the information needed to  determine whether to allocate additional resources. This information is  not available at least in part because division guidelines and policies do  not require that it be gathered.
Officials said that because they have so few staff available to conduct tests  and have a limited budget to contract for testing, they could not increase  the number of tests they perform. According to the officials, currently only  1 of 103 persons in the division is trained in testing methods. In addition to  other duties, this person coordinates the tests for the division, such as the  test of EIC forms and instructions completed in 1999 by a private vendor  and the test of the innocent spouse application form completed by IRS in  2002. Officials also told us some staff who are primarily responsible for  creating and revising tax documents may occasionally assist in conducting  tests, such as the three persons involved in testing the innocent spouse  form. Officials also said the total budget for contract support for the  division was $150,000 in fiscal year 2002, $185,000 in fiscal year 2001, and  $130,000 in fiscal year 2000.
As part of its annual planning and budgeting process, IRS management  determines what resources will be needed to accomplish strategies and  implement programs. IRS’s planning and budget guidance requires that  each operating unit prepare a business plan that, among other things,  clearly defines priorities and resource requirements. Requests in the  business plan for resources must be substantiated with evidence that  allocating additional resources is justified.
However, the division does not systematically identify when testing would  be beneficial and does not routinely demonstrate the benefits to taxpayers  and IRS that have been gained from such testing. Officials do not identify  which of the many changes it makes to forms and instructions each year  would most likely benefit from testing. Thus, the officials cannot tell IRS  management how many opportunities to improve forms and instructions  may be lost due to current resource levels. Further, when tests are  performed, officials do not identify, quantitatively or qualitatively, the  benefits that taxpayers and IRS may have realized.
One reason that IRS does not have data on forgone testing opportunities is  that the division lacks formal, written guidelines and procedures for  determining when testing would be beneficial. Currently, testing is an  optional step in the process for developing forms. IRS’s Tax Forms and  Publications officials said that they decide which forms to test based on  informal guidelines and procedures and input from officials in IRS’s four  operating division program offices and the Taxpayer Advocate Service.  The informal guidelines and procedures call for officials to weigh, among  other things, whether a form or instruction (a) affects a large number of  taxpayers, (b) has a high error rate based on taxpayers’ prior use of the  form, (c) is perceived as complex, and (d) will be used for several filing  seasons. Also, according to IRS, the amount of time available to perform  tests is factored into testing decisions.
These informal guidelines do not require officials to consider in all cases  whether testing would be beneficial and to document the decisions made.  Accordingly, even if the informal guidelines are applied, and officials judge  that some forms or instructions could benefit from testing but cannot be  tested due to scarce resources, those decisions are not made  systematically and documented.
Further, although the factors the guidelines suggest taking into account  appear to have evolved from officials’ experience and therefore should be  useful, they do not consider some pertinent factors that could affect the  benefits likely to be realized from testing. For instance, the guidelines  suggest taking the number of affected taxpayers into account but not the  likely amount of burden they would face due to unclear forms or  instructions. They also do not clearly call for officials to consider the costs  to test forms and instructions and the benefits that may accrue throughout  IRS, such as in telephone service centers. In addition, these informal  guidelines and procedures automatically exclude testing forms and  instructions that will be used only one time. Also, according to IRS  officials, the time frame between the passage of new tax laws and when  the newly created or revised forms and instructions must be finalized may  preclude some forms and instructions from being tested. Even if one-time- use forms meet other testing criteria, such as affecting a large number of  taxpayers who may perceive them as complex, IRS will not consider  testing them. As the rate reduction credit situation discussed earlier  illustrates, such automatic exclusions may not be appropriate in all  situations.
IRS officials do not have information on the results achieved when forms  and instructions are tested in part because the division does not have  policies that require such evaluations. When IRS obtained information on  the reduction in error rates following testing of EIC and Child Tax Credit  forms and instructions, the studies did not include collecting other  information on the benefits that may have resulted for taxpayers and for  IRS. For instance, the studies did not estimate the savings IRS may have  realized in its telephone and walk-in service due to increased form clarity.
Capturing fuller information on the results of testing would be consistent  with IRS’s strategic planning and budgeting process, which emphasizes  assessing the impact of current programs to efficiently allocate resources.  Further, by evaluating results of testing decisions, IRS officials would be  able to determine if their testing guidelines and procedures lead to good  decisions about when testing is most likely to be beneficial. They may also  be able to see if the methods they use to test—for example, focus groups  formed by IRS employees or one-on-one interviews with individuals—yield  the most effective test results.

	Conclusions

IRS continually faces the daunting task of developing and revising tax  forms and instructions to administer our ever-changing set of federal tax  laws. Taxpayers rely on IRS for forms and instructions that are as clear  and easy to understand as possible given the complexity of the tax laws  and providing clear materials is a key goal of IRS’s Tax Forms and  Publications Division.
In attempting to meet this goal, IRS has tested an average of one set of  forms and instructions each year over the last 5 years. In contrast, officials  from three federal agencies that routinely collect information from the  public say that testing documents for clarity before using them is their  standard practice. They do so because they believe testing will ensure that  their data collection documents are clear and that individuals will  understand them and complete them accurately.
Although it is difficult to gauge how much testing alone contributes to the  clarity of tax forms and instructions, IRS officials believe testing has  contributed to significant declines in taxpayer errors. Illustrations we  developed based on IRS’s experience in testing forms and instructions  suggest that IRS can completely recover its testing costs in the first year  following testing in some circumstances and that when savings to  taxpayers from more understandable forms and instructions are  considered, total benefits even in the first year following tests can be  several times IRS’s testing costs.
Although they recognize that testing is beneficial, officials say time  constraints and limited resources preclude more testing. However, IRS’s  procedures do not clearly specify when draft versions of forms and  instructions should be available for testing. Having a clearly defined point  where testing would be performed would facilitate establishing  procedures and deadlines to better ensure that testing could be done even  within IRS’s annual forms update cycle. Further, IRS officials do not have  information that would help IRS management to determine whether to  allocate additional resources to support enhanced testing. Because IRS  lacks standard written procedures for testing, officials have not  documented cases where testing would likely be beneficial and have not  demonstrated the benefits that are gained from testing.

	Recommendations for Executive Action

Because testing could potentially yield clearer and more understandable  tax forms and instructions, thereby producing benefits both to taxpayers  and IRS, we recommend that the Acting Commissioner of Internal  Revenue take the following actions.
Develop written criteria for determining which changes to tax forms  and instructions should be tested with taxpayers before publication.
Develop official written guidance that incorporates those criteria and  ensure that the guidance requires staff that develop new or revised  forms and instructions to document which changes would merit testing  and why.
Clarify procedures by designating when in the annual forms  development process that a draft version of forms and instructions  should be available for testing with taxpayers.
Ensure that an appropriate range of evaluations are conducted of tests  that are performed to better establish the costs and benefits of  performing tests and to refine IRS’s approach to testing on the basis of  lessons learned.
Use information gained from documenting when changes to forms or  instructions likely would be beneficial and from evaluations of tests to  reassess an appropriate level of resources to perform testing.

	Agency Comments and Our Evaluation

The Acting Commissioner of Internal Revenue provided written comments  on a draft of this report in an April 7, 2003, letter, which is reprinted in  appendix III. The Acting Commissioner agreed with our recommendations.
We are encouraged that IRS plans to implement all but one of our  recommendations in time for the 2004 forms development cycle.  Understandably, the remaining recommendation to ensure that an  appropriate range of evaluations is conducted of tests would take more  time to put into practice. The Acting Commissioner also provided  additional comments and observations on our draft report.
The Acting Commissioner commented that the crux of our report is that  we do not believe IRS has performed adequate testing on new and revised  tax forms and instructions due to a lack of resources. He said that  resources for testing forms and instructions have been adequate for the  testing IRS wanted to perform. While not questioning whether resources  were adequate for the testing IRS performed, we concluded that IRS  officials do not have information needed to determine the level of  resources that should be allocated to testing forms and instructions.  Accordingly, we recommended that IRS systematically identify  opportunities to improve forms and instructions through testing and to  evaluate the costs and benefits when testing is done.
Although agreeing that testing is beneficial, the Acting Commissioner also  said that there are significant staff costs associated with testing that are  not included in our cost analysis. We recognize that our analysis excluded  staff costs and as stated in our draft report we sought to be conservative in  estimating the benefits of testing, in part because we did not have  information on the full range of costs IRS incurs when undertaking  projects to tests forms and instructions. During the course of our work, we  requested estimates of staff costs for testing but IRS was unable to provide  them. Nevertheless, at least in the cases we illustrated the potential  benefits of testing were so much greater than the costs that including staff  costs likely would not have substantially changed the results of our  illustrations.
The Acting Commissioner expressed concern about whether IRS could  have forms and instructions ready for the filing season if testing was done  late in the forms development cycle as shown in our figure 2 depicting  IRS’s process. We agree with the Acting Commissioner’s concern;  however, our figure shows the various points at which testing can occur in  IRS’s current processes based on interviews with IRS officials and the  documentation they provided us. As IRS implements our recommendation  to clarify when testing should be done, selecting a point as early as  possible would help maximize the number of changes that can be tested  during the annual forms update cycle.
The Acting Commissioner also said that he disagreed with our conclusion  that IRS’s experience with obtaining feedback on its products is limited or  recent. He said IRS uses various methods to obtain customer feedback. We  agree that IRS uses methods other than testing to obtain feedback on its  forms and instructions. However, our report describes the potential  benefits and costs of testing as a feedback method. In terms of testing, IRS  has only tested five forms and instructions during July 1997 through June  2002; in our view, this is a limited number of tests that were conducted  during the recent past.
The Acting Commissioner also disagreed that testing would result in  reduced demand for walk-in and toll-free assistance. He said that IRS lacks  data to support such a conclusion and, based on its experience, new forms  generate requests for assistance and error rates on them tend to be higher.  We recognize that there will always be a demand for taxpayer customer  assistance. However, we believe that reduced demand for assistance is a  potential benefit of testing. We note, for example, that IRS officials seek  input from telephone assistors when deciding which forms or instructions  need to be clarified, apparently believing that clarifying the forms and  instructions may help reduce calls to assistors. Finally, testing is one  means for ensuring that even for new forms and instructions requests for  assistance and errors made by taxpayers will be minimized.
We are sending copies of this report to the Chairman and Ranking  Minority Member of the House Committee on Ways and Means and its  Subcommittee on Oversight; the Secretary of the Treasury; the Acting  Commissioner of Internal Revenue; the Director of the Office of  Management and Budget; and other interested parties. We will make  copies available to others on request. In addition, the report will be  available at no charge on the GAO Web site at http://www.gao.gov.
This report was prepared under the direction of Charlie Daniel, Assistant  Director. If you have any questions regarding this report, please contact  him or me at (202) 512-9110. Key contributors to this report were David  Alexander, Christopher Currie, Ronald La Due Lake, Anne Laffoon,  Veronica Mayhand, Edward Nannenhorn, and Shellee Soliday.

Appendix I: Objectives, Scope, and Methodology

To determine how often IRS has used taxpayers to test the clarity of new  or revised individual income tax forms and instructions, we interviewed  officials in its W&I Division’s Tax Forms and Publications Division in  Washington, D.C. We sought to obtain (a) an understanding of the process  IRS used to develop and revise individual income tax forms and  instructions and (b) gather information on the forms and instructions IRS  tested using taxpayers for the 5-year period between July 1997 and June  2002. We requested information for a 5-year span to help ensure that the  information collected would reflect the amount of testing usually done by  IRS. Our work did not include assessing IRS’s processes for developing  and revising notices or publications or assessing the clarity of any specific  tax forms or instructions.
To obtain insights on the benefits of testing written documents, we  interviewed officials from three federal agencies in the metropolitan  Washington, D.C., area that perform extensive research and data  collection using private citizens. We contacted NCHS, BLS, and the Census  Bureau because they have broad experience in conducting tests of the  clarity of written documents, such as forms, surveys, and instructions. We  also contacted a private research firm that specializes in testing forms and  surveys for a variety of public and private sector clients. We also reviewed  our own guidance for developing and using questionnaires.
To determine the benefits to taxpayers and IRS of testing tax forms and  instructions with taxpayers prior to their use by the public, we took  several steps. First, we interviewed IRS officials including Tax Forms and  Publications Division officials to obtain information on whether they  perceived testing as being beneficial to taxpayers and IRS. We also  interviewed IRS’s W&I Research Division officials in Indianapolis to obtain  their views on whether testing tax forms and instructions benefits  taxpayers and IRS. The research division officials, among other things,  collect information on taxpayer errors that the Tax Forms and  Publications Division uses when deciding which forms and instructions to  test. These officials provided us with data on changes in error rates for  EIC and Child Tax Credit forms and instructions before and after IRS  revised and tested them.
Second, we developed illustrations of the potential benefits and costs of  IRS’s testing EIC and Child Tax Credit forms and instructions by analyzing  the data on the changes in error rates obtained from IRS research officials.  To construct our illustrations, we used readily available data and made  certain assumptions. Data were not available on many of the potential  benefits of the changes and on the full range of costs IRS incurred to  conduct the tests. We developed similar illustrations of potential benefits  to taxpayers and IRS if the rate reduction credit instructions had been  tested. In all cases, the illustrations we developed are not actual  assessments of the costs and benefits that were associated with testing  forms and instructions, or that would have resulted if testing had  occurred. In developing the illustrations we sought to be conservative in  estimating benefits, in part because we did not have information on the  full range of costs IRS incurred to test forms and instructions. Because we  had to make various assumptions, our illustrations undoubtedly vary from  actual costs and benefits.
To determine the potential cost savings to IRS of testing these forms and  instructions, we first estimated IRS’s costs to correct a taxpayer error. Our  estimates of IRS’s cost to correct errors were limited to the labor cost  associated with correcting errors in IRS’s math error program. IRS  provided us the number of FTE staff years for operating the ERS that was  used to detect and correct math errors and the number of errors corrected  by ERS during fiscal year 2002. We used this information to calculate an  average labor cost to correct math errors that we then applied to  reductions in EIC and Child Tax Credit errors as well as to error  reductions that might have resulted from testing the rate reduction credit.  Total error correction costs may be higher because we did not include a  number of other costs associated with correcting errors. For example, we  excluded printing and postage costs for the notices sent to taxpayers. IRS  notices often cover more than one issue associated with a tax return and  data were not readily available to determine what portion of the postage  cost might be attributable to EIC or Child Tax Credit issues alone. In  addition, the estimate does not include costs such as equipment and rent.  We did not specifically test the accuracy of the cost information provided;  however, our audits of IRS’s annual financial statements have raised  concerns regarding IRS’s ability to identify all costs associated with a  given program or activity.
Because data were not available on the extent that testing reduced errors  related to EIC and Child Tax Credit forms and instructions, we assumed  different percentage reduction rates in errors due to testing these forms  and instructions. Our illustrations show potential savings based on  assumed percentage reduction in errors attributable to testing of 50, 25,  and 10 percent. Using our estimate of IRS’s costs to correct an error and  the assumed number of errors eliminated by testing, we arrived at the  costs that would have been incurred to correct those errors. The  difference between costs to correct eliminated errors and the costs to test  is the potential cost savings from testing. Cost savings due to reduced  errors likely would not mean reductions to IRS’s budget. Further, savings  likely would mean that IRS would provide services to other taxpayers or  would pursue other compliance or tax collection activities that it would  otherwise have been unable to do.
To determine IRS’s costs to test forms and instructions, we used IRS’s  actual contract costs for testing EIC and Child Tax Credits in 1999. IRS  spent a total of about $56,000 to test both EIC and Child Tax Credit forms  and instructions with focus groups. Although the cost of contracted  support for testing each form and instruction individually likely would  have been somewhat lower than this, we applied the total cost in each  case. Because IRS could not provide data, we did not include the costs  associated with IRS letting and managing the contract or the cost for Tax  Forms and Publications Division staff to work with the contractor in  conducting and managing the tests or the cost of any other IRS staff that  were involved in these tests.
To determine the potential benefits to taxpayers from testing EIC and  Child Tax Credit forms and instructions, we calculated estimated values to  taxpayers of the time saved if testing improved form and instruction  clarity, thereby reducing taxpayer errors and the burden of dealing with  IRS error notices. We also estimated taxpayers’ time saved if testing  reduced the time needed to understand and complete tax forms and  instructions. We developed similar illustrations for potential benefits to  taxpayers if the rate reduction credit had been tested. Our benefit  illustrations were based on a series of assumptions. For example, to  estimate the value to taxpayers of time saved from not having to deal with  an IRS notice, we applied the minimum wage rate that was in effect after  IRS revised and tested the forms and instructions to estimates of time  taxpayers might save by not having to deal with an IRS notice. For these  illustrations we assumed, on average, that taxpayers who received an IRS  error notice might spend either 2 or 5 minutes to deal with it. Data were  unavailable on how much time taxpayers actually spend dealing with IRS’s  notices; however, according to Taxpayer Advocate Service information,  IRS’s notices are difficult for taxpayers to understand. Further, taxpayers  who decide to contest an IRS notice may take time to call or write letters  to IRS or to contact and work with a tax preparer. Data were not readily  available to determine what portion of taxpayers who received an EIC or  Child Tax Credit error notice contested IRS’s change to their tax returns.
For our illustrations of the estimated value of time saved by taxpayers if  testing reduced the time needed to understand and complete EIC and  Child Tax Credit forms and instructions, we assumed that taxpayers would  save on average 1 minute by using clearer forms and instructions. We  excluded from our illustrations those tax returns prepared by preparers  because the taxpayers might not have had to read and understand the  forms and instructions. Because data were not available on the number of  tax returns claiming the Child Tax Credit that were prepared by paid  preparers, we reduced the number of returns claiming the credit by the  same percentage of EIC returns that were prepared by preparers. The  percentage of EIC claimants using paid preparers exceeds the average for  all taxpayers. For the rate reduction credit, our illustration is based on a  30-second time savings and the number of taxpayers who filed on paper  and did not use preparers. We chose a 30-second potential savings for this  illustration because many taxpayers would have had to read only part of  the instructions to determine what to do. We aggregated the times for all  taxpayers and multiplied the total hours saved by the prevailing minimum  wage rate to arrive at estimated benefits to taxpayers.
To determine whether any factors limited IRS’s ability to use individual  taxpayers to test forms and instructions and, if so, how these factors can  be addressed, we interviewed IRS’s Tax Forms and Publications Division’s  officials and analyzed supporting data they provided us. Regarding  officials’ view that they lacked sufficient time to do more testing, we  reviewed information on the amount of time IRS, NCHS, and the private  research firm we contacted took to perform various types of tests. We also  reviewed IRS’s process for developing new and revised forms and  instructions and determined how many weeks were available between the  dates that various laws were enacted or their provisions became effective  and IRS’s normal October 1st deadline for printing. Finally, regarding  officials’ view that they lacked sufficient resources to do more testing, we  obtained information on the resources available within IRS for testing.
We performed our work from May 2002 through March 2003 in accordance  with generally accepted government auditing standards.

Appendix II: Use of Testing in NCHS, Census, and BLS

Like IRS, the three federal agencies we contacted create written  documents to be completed by the public. The agencies create documents  such as forms, surveys, and questionnaires that they use to collect  information from the public to fulfill their missions. Unlike IRS, described  below, these agencies routinely test their forms, surveys, and  questionnaires prior to distribution to the public.
The National Center for Health Statistics, the nation’s principal health  statistics agency, compiles statistical information to guide actions and  policies relevant to public health and health policy. According to  NCHS, obtaining accurate and usable health information is crucial to  successfully fulfilling its mission to provide reliable information to the  Centers for Disease Control and Prevention and the Department of  Health and Human Services. NCHS collects information through  various sources including questionnaires that it develops and  administers. Researchers at NCHS told us that in support of their  research they administer surveys and questionnaires each year in  addition to developing questionnaires used by the Centers for Disease  Control and Prevention. They test each questionnaire using one-on-one  interviews. When documents pertain to a particular rather than a  general population, the researchers recruit participants with  characteristics similar to those persons who might be completing the  forms or questionnaires. For example, researchers recruited asthmatics  to test a questionnaire related to asthma. NCHS tests forms in one-on- one settings in which a participant may be asked to work through a  form while a moderator observes and then later interviews the  participant. This approach allows the researcher to identify specific  points at which the forms were confusing or problematic and learn why  the participant had difficulty. NCHS prefers to use one-on-one  interviews when conducting tests because this method closely  resembles the ways in which individuals will be completing the  documents since individuals will likely complete the documents by  themselves.
The Census Bureau is the principal agency responsible for collecting  and providing data about the people and the economy of the United  States. An accurate census is important because census results are  used to reapportion seats in the House of Representatives, redraw  congressional districts and other political boundaries, and address  countless other public and private data needs. The Census Bureau  collects information through short-form and long-form questionnaires  that it develops, tests, and administers. In preparation for the 2000  Census, the Congress budgeted millions of dollars to develop and test  questionnaires during the 1990s. The Census Bureau’s policy requires  that demographic survey questionnaires be tested. It has used focus  groups and one-on-one interviews to test its questionnaires and forms.  For example, in fiscal year 1996, the Census Bureau decided to make  fundamental changes to the traditional census design such as  shortening census questionnaires. In that year, it budgeted funds to  test, among other things, respondents’ understanding of race and  ethnicity questions. The Census Bureau has also conducted detailed  cost-benefit analyses of alternative designs; in 1992 it tested the  simplified questionnaire in order to gauge whether the new form would  increase response rates and reduce costly follow up with households  that did not respond to the census.
The Bureau of Labor Statistics, the principal fact-finding agency for the  federal government in the broad field of labor economics and statistics,  also depends on clear and understandable written documents to collect  accurate information from the public. According to BLS’s policy,  testing documents such as forms and surveys prior to use by the  general public should be undertaken to help identify factors that may  impede users’ ability to understand forms or surveys.  Then these  factors can be addressed in order to improve the clarity of written  documents and increase the accuracy of responses. Testing should be  done in the early stages of document development so that any  problems with clarity can be identified early. BLS routinely tests its  written documents using focus groups and one-on-one interviews, and  uses the results of the tests to make improvements to the documents.

Appendix III: Comments from the Internal Revenue Service

Summary:
Why GAO Did This Study

Taxpayers rated the Internal Revenue Service's (IRS) ability to provide clear and easy-to-use forms and instructions among the lowest of 27 indicators of service in 1993. Due to continuing concerns about unclear forms and instructions, GAO was asked to determine (1) whether and how often IRS tests the clarity of new and revised individual income tax forms and instructions; (2) the benefits, if any, of testing forms and instructions for clarity prior to their use; and (3) whether any factors limit IRS's ability to do more tests and if so, how they can be addressed.

What GAO Found

IRS used taxpayers and its employees to test revisions to five individual income tax forms and instructions from July 1997 through June 2002. According to IRS officials, they revised about 450 tax forms and instructions in 2001, many of which were for individual income tax returns. Testing forms and instructions can help ensure their clarity and thereby benefit taxpayers and IRS by, for instance, reducing taxpayers' time to understand and complete tax forms, reducing calls to IRS for assistance, and reducing taxpayer errors. Due to similar benefits, federal agencies we contacted that routinely collect information from the public test their questionnaires. Quantifying benefits due to testing is difficult, but IRS's experience in revising and testing Earned Income Credit and Child Tax Credit forms and instructions suggests that benefits of testing in some cases can considerably exceed the cost of testing. If taxpayers who did their own tax returns needed 1 less minute to understand these two credits due to testing, their time saved, valued at the minimum wage, would be worth $1.2 million; IRS's contracting cost for the two tests was $56,000. Although IRS officials recognized that testing could be beneficial, they cited tight time frames and constrained resources as limiting their ability to do more tests. While IRS faces time constraints when making some changes to forms and instructions due to the passage of new laws, not all changes are time constrained. IRS does not have procedures specifying which versions of draft forms and instructions should be tested with taxpayers or when in its annual forms development process testing should occur. Resources currently available for testing are limited but the office responsible for testing has not developed data on missed testing opportunities and has limited data on the benefits that have been realized when testing occurred. IRS's planning and budgeting process uses such data to support resource allocation decisions.