Information Quality Act: National Agricultural Statistics Service Implements First Steps, but Documentation of Census of Agriculture Could Be Improved
Released: 2005-09-23T00:00:00

Letter

Background

The IQA directed OMB to issue guidelines to federal agencies covered by the Paperwork Reduction Act designed to ensure the “quality, objectivity, utility, and integrity” of information disseminated to the public. The IQA also directed OMB to include in its guidelines requirements for agencies to (1) develop their own information quality guidelines, (2) establish administrative mechanisms for affected persons to seek correction of information that does not comply with OMB’s guidelines, and (3) annually report to OMB the number and nature of complaints they receive regarding the accuracy of the information they disseminate.

Prior to the IQA, there were several governmentwide actions aimed at improving agency data. For example, Statistical Policy Directive No. 2, first issued in 1952, required statistical agencies to inform users of conceptual or other limitations of the data, including how the data compare with similar statistics. In 1996, the Federal Committee on Statistical Methodology—an OMB-sponsored interagency committee dedicated to improving the quality of federal statistics—established a subcommittee to review the measurement and reporting of data quality in federal data collection programs. The results of the subcommittee’s work were published in a 2001 report that addressed such issues as what information on sources of error federal data collection programs should provide, and how they should provide it. For all federal government information collections, the 1995 amendments to the Paperwork Reduction Act called on federal agencies to manage information resources with the goal of improving “the integrity, quality, and utility of information to all users within and outside the agency.”

OMB’s IQA guidelines were issued in final form in February 2002. They required agencies subject to the IQA to take such steps as issue information quality guidelines designed to ensure the quality, objectivity, utility, and integrity of information disseminated to the public; establish administrative mechanisms for affected persons to seek correction of information they believe is not in compliance with the guidelines; report annually to the Director of OMB on the number and nature of complaints received regarding compliance with the guidelines and how the agencies handled those complaints; and designate an official responsible for ensuring compliance with OMB’s guidelines.

The OMB guidelines defined quality as an encompassing term comprising utility, which is the usefulness of the information to its intended users; integrity, which refers to the security of information and its protection from unauthorized access or revision; and objectivity, which addresses both presentation (i.e., whether the information is being presented in an accurate, clear, complete, and unbiased manner) and substance (i.e., whether the information is accurate, reliable, and unbiased).

In addition, OMB addresses transparency within the definition of objectivity and utility. As recognized in OMB’s guidelines, agencies that disseminate influential scientific, financial, or statistical information must demonstrate a high degree of transparency about data and methods. These measures are in place to facilitate the information’s reproducibility by an outside party or reanalysis of an agency’s results.

The National Research Council of the National Academies considers transparency a key principle for federal statistical agencies, and stated in a recent report that transparency, which it defines as “an openness about the sources and limitations of the data,” is particularly important for instilling credibility and trust among data users and providers.

As an agency within USDA, NASS is required to comply with the IQA. One statistical program administered by NASS is the quinquennial Census of Agriculture. According to NASS, the census provides a detailed picture of U.S. farms and ranches every 5 years and is the only source of uniform, comprehensive agricultural data at the county level. The results are published in 18 reports divided among three categories: Geographic Area Series, Census Quick Stats, and Specialty Products and Special Studies. Users of this information include federal agencies (for program and statistical purposes), farm organizations, businesses, universities, state departments of agriculture, elected representatives, legislative bodies at all levels of government, and academia. The next Census of Agriculture is scheduled for 2007.

Objectives, Scope, and Methodology

Our objectives were to (1) review how NASS met OMB’s guidelines covering the IQA and (2) examine the transparency of the documentation behind the Census of Agriculture’s processes and products, including the recently completed work on the 2002 Census, and the efforts currently underway for the 2007 Census.

To achieve both of these objectives, we reviewed OMB’s and NASS’s information quality guidelines, Census of Agriculture reports, submissions to OMB, and other relevant documents. We also interviewed NASS officials about how NASS conducted the 2002 Census and how it is planning for the 2007 Census. The officials included the NASS Administrator, Associate Administrator, and Deputy Administrator for Programs and Products.

In addition, to evaluate the transparency of Census of Agriculture products, we reviewed eight census reports and the Frequently Asked Questions area of the 2002 Census Web site, to determine the extent to which NASS followed its own procedures for ensuring the transparency of its information products. NASS’s IQA guidelines define transparency as, “a clear description of the methods, data sources, assumptions, outcomes, and related information that allows a data user to understand how an information product was designed and produced.”

NASS’s guidelines state that its survey activities include such activities as sample design, questionnaire design, pre-testing, analysis of sampling, and imputation of missing data. However, the guidelines were not clear as to the specific activities to be documented. Consequently, we reviewed the practices employed by such statistical agencies as the National Academies of Sciences, International Monetary Fund, and U.S. Census Bureau, and developed a set of 20 practices associated with transparent documentation that encompassed the items NASS laid out in its own guidelines. The practices include such actions as defining data items, discussing sample design, and describing how the content of the survey differs from past iterations (see app. II).

We looked for the presence or absence of these practices in 9 out of the 18 census reports and related forms of data that NASS disseminates, and verified the results with a second, independent analysis. In instances where a report did not include a particular documentation practice, we reviewed whether the report instead informed data users where to obtain this information. We chose these 9 reports because they all stem from the original census data collection, represent different product categories, and were available on the census Web site as of February 1, 2005.

To obtain an external perspective of how NASS processes and products address the IQA guidelines, we interviewed six data users from different types of agricultural and research organizations. We selected these data users from lists of registrants for USDA and NASS outreach meetings within the past 5 years. We selected these six data users because they use information from the census on a regular basis. Moreover, these data users attended the most recent NASS outreach meeting, which specifically addressed the 2002 and 2007 Censuses. Some data users had also provided NASS with feedback on the content of the agricultural census. Their views cannot be projected to the larger population of census data users.

We requested comments on a draft of this report from the Secretary of Agriculture. On September 8, 2005, we received the NASS Administrator’s written comments and have reprinted them in appendix I. They are addressed in the Agency Comments and Our Evaluation section of this report.

NASS Met the Procedural and Reporting Requirements of OMB’s IQA Guidelines

NASS fulfilled the various procedural responsibilities and reporting requirements under OMB’s guidelines. For example, NASS released its own IQA guidelines for public comment on March 27, 2002. NASS officials stated they received no substantive comments on them and OMB approved the guidelines with only minimal changes. The officials also noted that no revisions have been made since then. Table 1 shows in greater detail how NASS addressed OMB’s guidelines.

Better Documentation Could Improve the Transparency of Data Products and Correction Procedures

NASS’s IQA guidelines define transparency as, “a clear description of the methods, data sources, assumptions, outcomes, and related information that allows a data user to understand how an information product was designed and produced.” NASS’s guidelines also note that “NASS will make the methods used to produce information as transparent as possible” and that its “internal guidelines call for clear documentation of data and methods used in producing estimates and forecasts. . . .”

To assess the extent to which NASS processes help ensure the transparency of the information it publishes, we examined key publications from the 2002 Census of Agriculture. Census reports vary in terms of scope and intended audience (see table 2). On the one hand, the United States Summary and State Data report contains over 100 data tables, an introduction, and four appendices. On the other hand, County Profile reports summarize each county’s agricultural situation on two pages.

Overall, we assessed eight census reports within three product categories, as well as the Frequently Asked Questions (FAQ) section of the 2002 Census Web site, to determine the extent to which NASS followed its own guidelines for ensuring the transparency of its products. As shown in table 2, the transparency of the data documentation in the reports we reviewed varied between the Geographic Area Series reports—which are the most comprehensive of NASS’s products and addressed 15 of the 20 data documentation practices—and the Specialty Products and Special Studies which, depending on the specific product, addressed no more than 1 of the practices.

All eight reports and the FAQ Web site lacked a discussion of four documentation practices, including the following: 1. Questionnaire testing. NASS produced a separate, internal report that discusses questionnaire testing in detail; however, publicly available census publications do not address this topic. 2. Limitations of the data. NASS does not discuss data limitations in the census reports we reviewed.

3.

Impact of imputations, by item. When a statistical agency receives a report form with missing values, it normally estimates or “imputes” those values based on comparable data sources such as a similar farm operation. Although NASS uses a complex editing and imputation process to estimate missing values, and describes this process in the United States Summary and State Data report appendix, it does not quantify the impact of imputations by item in reports. 4. Whether any of the collected data have been suppressed for data quality reasons. Without information on whether any of the data had been suppressed because the quality was lacking, data users must assume that reports include all data items collected in the census had met agency publication standards.

Although NASS appropriately recognizes the variation in data user needs by publishing several types of specialized reports, none of the reports we reviewed direct data users where to find either a complete set of documentation or additional documentation. For example, given the short length and summary format of the County Profile reports, it is not surprising that they lack documentation. However, in order for users to assess the quality of the data contained in the reports, it is important for NASS to at least provide links on its Web site or to other publications where users can access definitions, response rates, and other relevant information.

NASS Should Document Its Procedures for Handling Correction Requests Not Filed under the IQA

NASS has two methods for handling data correction requests, depending on how they are submitted: a formal approach prescribed by OMB for correction requests filed under IQA, and an informal approach that NASS uses to address correction requests that are not filed under IQA. NASS’s informal correction procedures lack transparency because they are not documented and individual cases are not tracked. As a result, we could not determine the nature of these correction requests or whether or how they were addressed.

Consistent with OMB’s guidelines, NASS detailed its procedures to request corrections under IQA on its Web site, and posted appropriate Federal Register notices. For example, NASS’s Web site explains that to seek a correction under IQA, petitioners must, among other steps: (1) state that their request for correction is being submitted under IQA, (2) clearly identify the information they believe to be in error, and (3) describe which aspects of NASS’s IQA guidelines were not followed or were insufficient.

According to the instructions posted on its Web site, NASS’s IQA procedures are triggered only when petitioners explicitly state they are submitting a correction request under IQA. To date, none have done so. NASS addresses all other correction requests using informal, undocumented procedures that were in place before IQA was enacted. NASS officials explained that such requests are forwarded to the agency official responsible for preparing the report containing the information in question. That official, in turn, determines if the request can be resolved by clarifying the data, or whether a correction is needed. If a data item needs to be corrected, NASS has a set of procedures for documenting errors and issuing errata reports that are detailed in its Policy and Standards Memorandum No. 38. The memorandum describes the circumstances under which errata reports will be printed, and provides a mechanism for NASS staff to describe the nature of the error, its cause, and the action taken to resolve it.

According to the Administrator, Associate Administrator, and other senior NASS officials we interviewed, the requests it has handled from the 2002 Census have so far been resolved to the petitioners’ satisfaction, and none resulted in any corrections to the data from the 2002 Census. However, because NASS does not document its informal procedures for handling inquiries and data correction requests, and lacks a recordkeeping system to log and track them, NASS could not provide us with firm information on the number of inquiries it has handled, the nature of those inquiries, and whether and how they were addressed.

This is not to say that all complaints should follow the same procedures required by the IQA mechanism. For efficiency’s sake, it is important for agencies to respond to complaints in accordance with the magnitude of the problem. However, to provide a more complete picture of the questions NASS receives about its data and how those questions were handled, it will be important for NASS to better document its approach for handling correction requests not filed under IQA, and track their disposition.

NASS Has Taken Steps to Better Document Its Criteria for Assessing Input on Census Content

The 2002 Census of Agriculture was the first in which NASS developed the questionnaire (the 1997 Census of Agriculture was moved from the Census Bureau to NASS after the content had been determined). In doing so, NASS went to great lengths to obtain input from data users on what questions to ask, and evaluated their suggestions using a documented set of criteria. In preparing for the 2007 Census, NASS sought feedback on the questionnaire content from a broader spectrum of data users, in part because NASS solicited suggestions via the Internet. However, unlike the 2002 cycle, the criteria NASS used to assess the feedback were not initially documented, which is contrary to NASS’s IQA guidelines. However, as a result of our review, NASS has developed documented criteria similar to that used during the previous census.

Under the Paperwork Reduction Act, agencies must obtain OMB’s approval prior to collecting information from the public. As part of this process, agencies must certify to OMB that, among other things, the effort is necessary for the proper performance of agency functions, avoids unnecessary duplication, and reduces burden on small entities. Agencies must also provide an estimate of the burden the information collection would place on respondents.

For the 2002 Census, NASS submitted its request for approval—a form called “OMB 83-I”—in August 2001, and OMB approved it in October 2001. NASS estimated that the census would require a cumulative total of more than 1.3 million hours for respondents to complete and would cost them, in terms of their time, in excess of $21 million.

OMB’s approval process also requires agencies to solicit input from external sources. NASS obtained input on the 2002 Agricultural Census content through a Federal Register notice, meetings with data users, and by contacting federal and state agencies that use census statistics to discuss data needs.

Likewise, NASS is obtaining input on the content of the 2007 Census through a variety of channels. According to an agency official, the process began around June 2004, when NASS began releasing publications from the 2002 Census. NASS sent an evaluation form to its state offices requesting feedback on the census, including their suggestions for changing the content. NASS also asked the state offices to identify users from whom it could obtain additional feedback.

NASS solicited further input by reaching out to data users within USDA and other federal agencies, querying organizations included in a list of “typical” data users maintained by NASS’s Marketing and Information Services Office, and holding periodic regional meetings with data users. NASS also has a “hot button” on its Web site where visitors are asked what items, if any, should be added or deleted from the census.

In all, NASS obtained input on the 2007 Census through 10 distinct conduits. Moreover, compared to the process used to develop the content of the 2002 Census, its 2007 efforts were open to a wider spectrum of customers, and involved more direct contact with data users during the planning phase. Indeed, as shown in table 3, NASS’s outreach via the Internet, regional meetings, and queries to data users was over and above the steps it took when developing the 2002 Census. This openness was reflected in the comments of the six data users we interviewed. Five of the six users said NASS’s approach to eliciting input was adequate, while three of the six had requested new content items for the 2007 Census to better meet the needs of their organizations.

The content evaluation process began in December 2004, and NASS is currently testing the questionnaire content. Following any refinements, mail-out of the actual census is scheduled for December 2007.

For both the 2002 and 2007 Census cycles, the solicitation, review, and ultimate determination of the questionnaire content was led by the Census Content Team, a group consisting of experienced NASS statisticians representing different segments of the agency such as livestock, crops, and marketing. The 2002 Content Team used specific, documented criteria to inform its decisions. Specifically, suggestions were assessed according to the following factors, which were also made available to data users: items directly mandated by Congress or items that had strong items proposed by other federal agencies where legislation called for that agency to provide data for Congress; items needed for evaluation of existing federal programs; items which, if omitted, would result in additional respondent burden and cost for a new survey for other agencies or users; items required for classification of farms by historical groupings; items needed for improving coverage in the census; and items that would provide data on current agricultural issues.

However, the criteria the 2007 Team used to assess input on the questionnaire content were not initially documented. According to agency officials we interviewed, NASS largely relied on professional judgment to evaluate the feedback it received, considering such factors as the need to keep the data comparable to past censuses and not increase the length of the questionnaire.

Although a certain amount of professional judgment will invariably be used in making determinations on questionnaire content, the absence of documented assessment criteria is inconsistent with NASS’s guidelines. Indeed, these guidelines note that transparent documentation “allows a data user to understand how an information product was designed and produced.” Moreover, without documented criteria, it is not clear whether members of the Content Team are considering the same set of factors, or even if they are weighing those factors in the same manner.

According to NASS, the shift in approach stemmed from staff turnover and reassignments of members of the 2002 Team and, as a result, the 2007 Team was not aware of the criteria used in 2002. Our review made the 2007 Team aware of the earlier set of criteria, and the Team has since developed similar documentation. NASS noted that all future content teams will use and update these criteria when developing the content of subsequent censuses.

It will be important for NASS to continue with this approach because it is more consistent with its own IQA guidelines, and will also help NASS to do the following: Ensure the utility and relevance of information. A key principle for federal statistical agencies is to provide information relevant to issues of public policy. However, the nation’s information needs are constantly evolving, and it is important for statistical agencies to adapt accordingly. This is particularly true with agriculture, where a variety of factors such as changing technology and agricultural trends can affect what information should be collected. Rigorous content selection criteria could help NASS methodically evaluate the needs of different users, establish priorities, and keep the census synchronized with changing public policy requirements.

Maximize cost-effectiveness and reduce public burden. As with all federal surveys, there are financial and nonfinancial costs to conducting the Census of Agriculture. These costs include the direct expenditures related to planning, implementing, and analyzing the census, as well as disseminating the information. There is also a cost to respondents in terms of the time they take to complete the questionnaire. Additionally, there are opportunity costs in that for every question that is included in the census, another question might need to be excluded so as not to increase the length of the census. Rigorous, consistently applied criteria can help promote cost-effectiveness because they can ensure that only those questions that meet a particular, previously identified need are included in the census. Applying such criteria also help inform decisions on the appropriate role of the federal government in collecting the data, and whether a particular question might be more appropriately addressed by a different survey, government organization, or the by the private sector.

Maintain credibility. Content selection criteria provide a basis for consistent decision making on what to include in the census and what gets left off. This is especially important for maintaining NASS’s credibility given the input it receives from various sources. Without documented criteria, NASS’s actions could be perceived as arbitrary or disproportionately swayed by one particular interest or another; thus, NASS’s decisions would be more defensible.

Further, documented criteria will guard against the loss of institutional memory to the extent there is further turnover in Content Team membership.

Conclusions

NASS satisfied the procedural responsibilities and reporting requirements under OMB’s IQA guidelines. Moreover, to the extent that NASS continues to use the documented criteria it developed to inform future decisions on the content of the Census of Agriculture, it could help establish a closer alignment between the questions included in the census and evolving agricultural policy requirements, resulting in a more cost-effective data collection program.

Building on these efforts, the transparency of census data products could be improved with more robust documentation. NASS’s procedures for addressing correction requests not filed under IQA could be more transparent as well. More than just a paperwork issue, greater transparency will help enhance NASS’s accountability to public data users and increase the credibility of census information.

Recommendations for Executive Action

To help enhance the transparency of the Census of Agriculture’s processes and products, we recommend that the Secretary of Agriculture direct NASS to take the following two steps: 1. Ensure that census products fully address NASS’s own guidelines for data documentation or at least contain links to such information. The list of 20 documentation practices that we developed, while not necessarily exhaustive, represents sound actions used by other statistical agencies and could form a starting point for NASS. 2. Document and post on NASS’s Web site its procedures for handling data correction requests not filed under IQA, and track the disposition of those requests.

Agency Comments and Our Evaluation

The NASS Administrator provided written comments on a draft of this report on September 8, 2005, which are reprinted in appendix I. NASS noted that our “report and recommendations are insightful and will be used to further strengthen the transparency of NASS methods and procedures.”

In particular, NASS concurred with our finding that the methods and procedures in its specialized reports should be better documented and, consistent with our recommendation, stated that these products “will now provide links to this information.” NASS’s efforts, if fully implemented, should make it easier for data users to understand how these products were designed and produced, and NASS should be commended for its actions to continually improve its products and better meet the needs of its customers.

While NASS’s more comprehensive products were better documented, our analysis found that they could also benefit from more robust documentation. Thus, in keeping with our recommendation, it will be important for NASS to ensure that all of its census products—its larger reports and more focused studies--fully address NASS’s own guidelines for data documentation.

In commenting on our recommendation for NASS to document and post on its Web site its procedures for handling data correction requests not filed under IQA, NASS concurred with our view that this information would provide it with a better sense of the questions it receives about its data, but added that “a detailed recordkeeping system to log and track every inquiry” would not be the best use of its resources. Instead, NASS plans to “compile a listing of the more common issues” and make them available on its Web site in the form of frequently asked questions. NASS believes this approach would be useful for future planning, as well as provide answers to questions most likely to arise among other data users.

As noted in our report, our recommendation stemmed from our finding that NASS could not provide us with information on the number of inquiries not filed under IQA, the characteristics of those inquiries, and how they were addressed. Although the details remain to be seen, NASS’s proposed approach could provide this information and, consistent with the intended outcome our recommendation, address the need for greater transparency. NASS’s efforts will be further strengthened if, consistent with our recommendation, it posts on its Web site its procedures for handling correction requests not filed under IQA.

We will send copies of this report to other interested congressional parties, the Secretary of Agriculture, and the NASS Administrator. Copies will be made available to others on request. This report will also be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-6806 or williamso@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Comments from the Department of Agriculture

How Census of Agriculture Reports Address Various Documentation Elements

State and County Reports (Alabama)

State and County Reports (Alabama)

GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, Robert Goldenkoff, Assistant Director; David Bobruff; Jennifer Cook; Richard Donaldson; Andrea Levine; Robert Parker; John Smale; and Michael Volpe made key contributions to this report.