Data Quality: Census Bureau Needs to Accelerate Efforts to Develop and Implement Data Quality Review Standards
Released: 2004-12-17T00:00:00

Letter

Background

The Bureau is best known for counting the nation’s population every 10 years. In the future, the Bureau intends to collect much of the data that have traditionally been collected during the decennial census from the long-form questionnaire with the annual ACS. Beyond the decennial census, the Bureau also conducts numerous other surveys and censuses that measure changing individual and household demographics and the economic condition of the nation. Lawmakers and agency officials at the federal, state, and local levels rely on these data when they make decisions in a wide range of policy areas. Private-sector decision makers also use census data to guide their business plans.

Because of the critical and varied uses of census information, it is important that the Bureau’s published data meet minimum quality standards. In addition, when the data are made public, it is equally important for the Bureau to disclose what has been done to ensure the quality of the data and identify any limitations so that potential consumers can decide whether the data are appropriate for a particular use.

Some degree of error in the census (and in virtually any survey) is inevitable because of limitations in enumeration, processing, and dissemination methods and errors in responses and imputation of data for nonresponses. Given the size and diversity of the U.S. population, the effort to count the entire population and provide detailed demographic characteristics every 10 years is one of the most complex of all government operations. The Bureau devotes significant resources to minimizing error and improving the quality of the decennial census.

Data quality standards and standardized quality control procedures can provide a consistent basis for making data dissemination decisions and informing the public about the quality of the data made available to it. In 2000, Congress passed what is now known as the Information Quality Act. This legislation directed OMB to issue governmentwide guidelines that “provide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies.” The legislation also required each agency to issue its own implementing guidelines that include administrative mechanisms allowing affected persons to correct information maintained and disseminated by the agency.

The OMB guidelines, issued in final form in February 2002, define quality as encompassing utility, objectivity, and integrity. The guidelines require agencies to issue their own implementing guidelines by October 1, 2002. Additionally, they mandate that agencies adopt a standard of quality as a performance goal and act to incorporate data quality criteria into their data dissemination practices. The guidelines also require agencies to develop processes for reviewing the quality of data before they are disseminated.

Although OMB had some general guidance for survey processes prior to the enactment of the Information Quality Act, other than requirements for the evaluation of selected monthly and quarterly economic indicators, there were no governmentwide requirements relating to the quality of data disseminated by the federal agencies. Some statistical agencies within the United States developed their own extensive guidelines and standards that apply to data disseminated to the public. In July 2001, OMB identified the statistical agencies within the Departments of Education and Energy, the National Center for Education Statistics (NCES) and the Energy Information Administration, as good examples of agencies that have developed specific guidelines to implement their broad principles and diverse professional standards.

Statistical agencies in other countries have also developed good examples of comprehensive guidelines for ensuring the quality of data disseminated to the public. Since 1985, Statistics Canada, the central statistical agency of the Canadian government, has published quality guidelines for its statistical activities. Subsequently, it added guidelines on quality assurance processes and management context and developed a policy and standards on informing users about data quality. More recently, the European Union recognized the importance of comprehensive, well-documented guidelines and standards to support its task of developing high-quality, comparable statistics from member countries. All members of the European Statistical System (ESS) have signed a quality declaration and approved 22 recommendations for quality for future work within the system.

Scope and Methodology

To address our first question on the standards that the Bureau had in place to guide its data dissemination decisions, we interviewed census officials, reviewed relevant agency documents, talked to data users, and reviewed various complaints about the quality of 2000 Census data. We built on our prior reports about the quality of data from the 2000 Census on Hispanic subgroups and the homeless and the Bureau’s decision-making processes for its decisions on whether to release those data. We also reviewed other GAO reports addressing aspects of the Bureau’s procedures for assessing the quality of disseminated data. From these reports, we identified examples of several types of problems the Bureau encountered with 2000 Census data, which might have been alleviated if the Bureau had implemented data quality standards and procedures. Our examples of data quality problems are not comprehensive, but illustrative.

To determine whether the Bureau has since developed Bureau-wide data quality standards, and, if so, whether they would likely address for the 2010 Census the data quality problems raised after the 2000 Census, we interviewed census officials responsible for developing agencywide standards, examined documents related to the development of new standards on data quality review, and reviewed the agency’s Internet site for information on data quality review standards available to the public. We also reviewed OMB guidelines on the quality of data disseminated by federal agencies as well as the action taken by the Department of Commerce and the Bureau in response to the guidelines. We attended meetings of the Secretary of Commerce’s Decennial Census Advisory Committee, the National Academy of Science Panel on Research on Future Census Methods, and the Washington Statistical Society’s conference on Quality Assurance in the Government, all of which examined issues related to the quality of the data disseminated by the Bureau. We also discussed information quality standards and guidelines with officials in Eurostat, the statistical directorate of the European Union.

Additionally, we considered how the Bureau’s actions in developing dissemination guidelines could improve the quality of data disseminated after the 2010 Census and for other Bureau data collection programs, such as the ACS that among other things, is intended to replace the long-form census questionnaire. To benchmark the Bureau’s progress in developing data quality review standards with that of other statistical agencies, we also reviewed documents from entities that have developed standards for the quality of data disseminated to the public, including NCES; Statistics Canada, the central statistical agency of Canada; and ESS. However, we did not evaluate the implementation or effectiveness of these guidelines and standards or their specific applicability to the Bureau.

Our work addressed only standards and guidelines on data quality review. Although OMB’s information quality guidelines and the Bureau’s guidelines and performance principals cover all the key steps in data collection, analysis, and dissemination, we did not look at the Bureau’s guidelines or standards for ensuring quality during the planning and data collection stages. Instead, as requested, we looked at Bureau guidance on steps taken after data collection, that is, guidance related to processing data, assessing their quality, and making them available to the public. We looked for documents spelling out standards, guidelines, procedures, and other criteria to guide decisions about identifying and correcting errors, determining if and when to release data, and revising data after release.

Our audit work was conducted in Washington, D.C., and at the Bureau’s headquarters in Suitland, Maryland, from August 2003 through October 2004. Our work was done in accordance with generally accepted government auditing standards.

We requested comments on a draft of this report from the Secretary of Commerce. On September 27, 2004, the Secretary provided written comments on the draft. The comments are reprinted in appendix I.

Professional Judgment Drove Data Dissemination Decisions

The Bureau had no agencywide standards or guidelines in place to guide decisions about disseminating data from the 2000 Census. Instead of agencywide, written guidance, professionals within the different parts of the Bureau primarily used their judgment and program-specific practices to decide when and whether data should be released and what supporting information, if any, should accompany them. This led to instances when (1) data were released with uncorrected and undisclosed quality problems, (2) inconsistent decisions were made on whether to release data sets with similar quality problems, and (3) the reasons for certain data dissemination decisions were inadequately communicated.

The Bureau Lacked Agencywide, Written Standards and Guidelines on the Quality of Census Data Disseminated to the Public

At the time the Bureau was making decisions about disseminating data from the 2000 Census, it did not have written, agencywide guidelines or standards to help inform its decisions on whether the data were of sufficient quality to be released. Although Bureau officials emphasized that the Bureau has a long tradition of high standards and procedures that yield quality data, they acknowledged that these practices were primarily part of the agency’s institutional knowledge. According to one official, key individuals in each program area, relying primarily on professional judgment, determined whether the quality of the data was acceptable for release to the public. The official explained that the program areas develop their own guidance and procedures for ensuring data quality. Sometimes their guidance and procedures were written, but more often they were not. Further, the Bureau had no central inventory or repository of the guidance and practices of the different divisions.

Lack of Data Quality Review Guidelines Led to Inadequate Analysis of Potential Errors and Release of Data without Adequate Disclosure

As noted earlier, decennial census data are used to apportion and redistrict Congress. As release of data for each of these purposes is required by statute, they are known collectively as “public law” data. The Bureau had a number of quality assurance programs and procedures for assessing the accuracy of, and correcting errors in, public law and other data prior to their release. However, the lack of standard procedures and guidelines for dealing with quality problems contributed to lost opportunities to correct errors in the count of the population identified before the data were disseminated.

One such quality assurance program we reviewed was known as Demographic Full Count Review, in which analysts were to identify, investigate, and document suspected data discrepancies or “issues” in order to clear census data files and products for subsequent processing or public release. The Bureau contracted out some of the analysts’ work because it lacked sufficient staff to conduct the Full Count Review on its own. Bureau reviewers were to determine whether and how to correct the data by weighing quality improvements against time and budget constraints. Analysts identified 4,809 possible discrepancies, such as instances when the location, population count, demographic characteristics, or a combination of these for housing units and group living facilities differed from what analysts expected. According to Bureau officials, only 5 of the 4,809 issues were investigated and corrected prior to the release of the public law data. All five involved group living facilities the Bureau calls “group quarters” for which the Bureau had the correct population counts, but placed the living facilities in the wrong places. The Bureau did not investigate most of the remaining issues prior to the release of the data in large part because they were insufficiently documented and the Bureau lacked the time and people to further investigate these issues.

Subsequently, according to Bureau officials, the remaining issues that contained sufficient documentation were investigated as a part of the Count Question Resolution program, which ended in September 2003.

As we noted in our July 2002 report, the fact that public law data were released with over 4,800 unresolved data issues of unknown validity, magnitude, and impact is cause for concern. To the extent these unresolved discrepancies were in fact true errors in the population count or geography, they could have affected the drawing of congressional districts as well as other purposes for which census data are used.

The existence of data quality review guidelines could have helped the Bureau in this situation. For example, we found that the Bureau’s lack of clearly defined requirements for documenting data issues resulted in a significant number of cases with inadequate documentation that the Bureau could not use to resolve the issues. Additionally, the Bureau had no mechanism for setting priorities for resolving these potential data errors. A sufficient set of guidelines could have helped the Bureau to ensure that the documentation of potential errors was adequate for decision making and to maximize the use of scarce resources in addressing the various data issues, giving top priority to investigating discrepancies likely to have the most adverse affect on the data.

The quality of certain data from the census long-form questionnaire have been called into question as well. In its 2004 comprehensive review of the 2000 Census, a panel of the National Research Council of the National Academy of Sciences assessed the quality of the long-form data using various benchmarks, and found that the overall quality of the information was less than that of the short-form questionnaire and had deteriorated since the 1990 Census. For example, at least 32 percent of the respondents failed to provide information on their property taxes, and 30 percent did not respond to all or some of the questions relating to income (compared with 12 and 13 percent, respectively, in 1990). Additionally, the panel noted that the Bureau did not measure and report the impact of some of the steps it took to address problems with missing data and recommended that the Bureau develop such measures and inform users about the need for caution in analyzing and interpreting these data.

Even more significant quality problems plagued the data for residents of group quarters. The panel found these data to be poor in comparison with the data for household residents, and also in comparison with data for group quarters from the 1990 Census. In 2000, missing data rates for some items were over 25 percent—one item was over 50 percent—for all residents of group quarters, and as high as 75 percent for prison inmates. Given the prevalence of missing data from residents of group quarters, the panel questioned whether the Bureau should have published these data at all for some or all types of group quarters.

The Bureau Made Conflicting Dissemination Decisions on Data with Similar Quality Problems

Our earlier work on Hispanic subgroups and the homeless showed that the Bureau’s approach to data quality review led to inconsistent decision making. Faced with similar quality problems in data from the 2000 Census, Bureau officials made different decisions about disseminating data and did not explain the reasons for their decisions.

For example, in an effort to improve the count of Hispanics and simplify the questionnaire, the Bureau redesigned its 2000 Census question on Hispanic origin and dropped a list of examples of Hispanic subgroups included in the 1990 Census. In May 2001, the Bureau released data on Hispanics and Hispanic subgroups as part of its first release summarizing the results of the 2000 Census. The Bureau also published The Hispanic Population, a 2000 Census brief that provided an overview of the size and distribution of the Hispanic population in 2000 and highlighted changes in the population since the 1990 Census. For the first time, the Bureau released data on Hispanic subgroups as a part of its release of the Full Count Review data even though it had not fully tested the impact of questionnaire changes on the subgroup data and provided little discussion of the potential limitations of the data.

Shortly after the Hispanic and Hispanic subgroup data from the 2000 Census were released to the public, questions were raised about the counts for specific Hispanic subgroups. For example, the reported count of Dominican Hispanics was significantly lower than counts reported in other Bureau surveys. Representatives of affected Hispanic subgroups asked for an investigation and explanation of why the Bureau reported data that these subgroups considered to be of questionable quality. We found that a key factor behind the Bureau’s release of apparently less-than-accurate Hispanic subgroup data appeared to be a lack of adequate guidelines governing decisions on quality considerations that should be addressed before making data publicly available.

In contrast, the Bureau, citing quality problems, decided not to separately report certain information on people without conventional housing, including those commonly referred to as “homeless.” Enumerating this segment of the population has been an ongoing challenge for the Bureau. To help locate and count these people in 2000, the Bureau partnered with organizations providing services to the homeless and with local governments, some of which put considerable resources into the effort. When the Bureau decided not to separately report the number of people in transitional and emergency shelters as originally planned because of data quality problems, some of the organizations and local governments, which had expected to use the data for directing services to the homeless, questioned the Bureau’s process for making that decision. Additionally, we found that the decision about when and whether to release data on people in emergency and transitional shelters changed several times. Decisions about the release of data with identified quality problems were not well documented and communicated with some Bureau partners and other stakeholders.

As a result, outside parties interested in both the Hispanic and homeless data from the 2000 Census questioned the quality of the data, the procedures the Bureau used to determine what data to release, and the value of their own participation in helping the Bureau prepare for the 2000 Census. Because the Bureau’s reasons for data release decisions were not obvious, and it had no guidelines or standards that spelled out criteria for decisions, the Bureau left itself open to questions about the objectivity of its decisions and risked loss of public confidence.

In our reports on Hispanic and homeless Census 2000 data, we recommended that the Bureau (1) develop agencywide guidelines for its decisions on the level of quality needed to release data to the public, how to characterize any limitations in the data, and when it is acceptable not to release data and (2) ensure that these guidelines are documented, transparent, clearly defined, and consistently applied. We also recommended that the Bureau ensure that its plans for releasing data are clearly and consistently communicated to the public. The Bureau agreed with each of our recommendations and asked its Methodology and Standards Council to review existing statistical and quality guidelines, bring them together in one place, and develop data quality standards. We discuss the Bureau’s actions later in this report.

The Bureau Has Made Limited Progress in Publicly Issuing New Standards on the Quality of Data Disseminated to the Public since the 2000 Census

Since the first results of the 2000 Census were released, the Bureau has publicly issued a set of information quality guidelines and one new standard on the quality of data disseminated to the public. As required by the Information Quality Act and the OMB guidelines, the Department of Commerce and the Bureau published Information Quality Guidelines, but the guidelines contain only general quality goals and principles and do not provide any specific guidelines or procedures on the implementation of the general principles. Also as required by the Information Quality Act and the OMB guidelines, the Bureau published a standard that described a procedure allowing individuals to seek correction of certain errors in data disseminated by the Bureau. Additionally, the Bureau has begun developing several other standards on the quality of data disseminated to the public, but none have been publicly released in final form.

In March 2003, in response to our recommendations, the Bureau established an interdirectorate working group charged with the broad mandate of developing Bureau-wide standards for quality in data releases. The working group has taken some steps to address the tasks laid out in its charter. However, the Bureau has not provided information on the scope or the time frame for developing these standards.

The Bureau Has Taken Steps to Expand Its Guidance on Data Quality Review

Recognizing the paucity of Bureau-wide written standards on the quality of data disseminated to the public, the Bureau established a Quality Program in 1999 to develop consistent processes for producing quality products across the Bureau. The Bureau’s Associate Director for Methodology and Standards with input from chiefs in a number of divisions compiled an inventory of data quality review documents used in different divisions and developed a Bureau-wide quality framework. The resulting quality framework was adopted to serve as a vehicle through which “the demographic, economic, and decennial areas can share and support common principles, standards, and guidelines.” This framework provides the organization for documents in the intranet portal known as the Quality Management Repository (QMR). Additionally, the Bureau’s description of the quality framework spells out the process for developing, reviewing, and approving quality framework documents. The document describing the quality framework and most of the documents in the QMR are internal documents not available to the public through the agency’s Internet site. However, Bureau officials indicated that they intend to make some of the standards available through the Internet later in calendar year 2004.

The Bureau has publicly issued two data quality review documents and made them available through the Internet. In October 2002, in response to the requirements of the OMB guidelines, the Bureau published a set of information quality guidelines in eight performance areas, including the establishment of review procedures. The Bureau’s guidelines identify broad quality goals and principles, but do not provide specific guidance to ensure consistent decisions. For example, the guideline on predissemination review of data says that “all documents released by the Census Bureau undergo extensive review that encompasses the content, statistical and survey methodology, and policy implications of the document,” and that this review “ensures that the data and text of the document meet Census Bureau standards for quality” or the Bureau reserves the right to withhold the data from the public. However, the guideline does not indicate what the Bureau “standards for quality” are, how the Bureau will know if the data meet the standards, or who within the Bureau is responsible for the review.

The second document issued and made available on the Bureau’s Web site is Census Bureau Standard: Correcting Information That Does Not Comply with Census Bureau Section 515 Information Quality Guidelines in March 2002. This standard was also issued in response to the specific requirements of the Information Quality Act and the OMB guidelines that agencies provide procedures for correcting certain errors identified in data they disseminated and post these guidelines on their Web sites. The standard established procedures that allow individuals to request a correction of information they believe is erroneous and the Bureau to review the evidence and determine whether a correction is warranted.

The Bureau has also approved several additional Bureau-wide data quality review documents for implementation and internal distribution through the QMR on its intranet. On March 18, 2003, the Bureau issued Census Bureau Standard: Minimal Information to Accompany Any Report of Census Bureau Data for a 6-month trial period. The standard identifies 13 specific items that the Bureau should report for every survey or census and specifies who is responsible for ensuring adherence to the standard. An accompanying memorandum from the Associate Director for Methodology and Standards to program associate directors said that implementation issues would be documented during the trial period and appropriate changes made prior to the final release of the standard. Even though the trial period is over, the Bureau has not made such changes or publicly issued the standard in final form. However, the standard is still in effect on a trial basis, according to one Bureau official.

The Bureau also released its Census Bureau Guideline: Quality Profiles on March 9, 2004, through the QMR. The document outlines a standardized quality profile, recommended for all recurring surveys and certain other programs, which is intended to present a consistent set of information on the quality of each program. As a guideline rather than a standard, this guidance is recommended rather than mandatory.

In addition, the Bureau has also initiated work on several proposals for additional standards. For example, a standard for discussion and presentation of errors in data disseminated to the public is under development. This standard is based on a technical paper that was issued in 1974 and revised in 1987. The Bureau said it would be issued in the near future, but has not provided a specific date.

Bureau Working Group Has Begun Developing Additional Standards on Data Quality Review, but None Have Been Issued

In response to our recommendations from reports on both homeless and Hispanic subgroup data from the 2000 Census, the Bureau established an interdirectorate working group on March 3, 2003, with the broad mandate to develop Bureau-wide standards for quality in data releases. However, the working group has not yet issued any draft or final standards or developed a time frame for doing so.

The working group is composed primarily of assistant division chiefs from the program areas—decennial, demographic, and economic. An assistant division chief from the Demographic Statistical Methods Division chairs the group.

According to the working group’s charter, its mission is to “Document current Census Bureau data review procedures, “Benchmark Census Bureau review procedures with that of other “Document Census Bureau situations where review of data indicates data does not meet “quality requirements” and the outcome of those situations, “Propose standards for quality in Census Bureau data products, “Benchmark quality requirements for data release with other agencies, “Develop Census Bureau Standard: Quality in Census Bureau Data Releases.”

Bureau officials told us that the working group has reviewed the published detailed guidelines from NCES and the Canadian national statistical office. Benchmarking discussions have taken place with the Bureau of Labor Statistics and the National Center for Health Statistics. Additionally, the working group met with an official from the New Zealand national statistical office to discuss its standards. The group is also planning meetings with additional federal agencies. These organizations have published detailed guidance on how broad principles on data quality are to be put into practice, notably the organizational responsibilities and internal control mechanisms for applying them.

For example, Statistics Canada, the central statistical agency of the Canadian government, has developed an extensive and detailed set of quality guidelines that covers the quality of data disseminated to the public and the quality control processes that are supposed to be applied to ensure the quality of the data. In March 2000, Statistics Canada published its Policy on Informing Users of Data Quality and Methodology, which specifies the organization’s responsibilities to inform users about the concepts and methodology for collecting, processing, and analyzing its data; the accuracy of these data; and any other features that affect their quality or fitness for use. By detailing mandatory documentation standards, guidelines for additional documentation, and examples of mandatory standardized summary documentation, the policy enhances the likelihood of consistent decision making throughout the organization. Additionally, making this information public ensures that any data user can determine what has been done to ensure the quality of the data and Statistics Canada’s reasons for its decisions about release.

NCES has developed detailed standards designed to implement its broader policies on dissemination of statistical data. An NCES standard includes a section entitled “Establishment of Review Procedures,” which includes a table showing the required reviews for each type of product and an illustration of the key steps in the review and adjudication process. As with the Statistics Canada policy, the NCES standard provides information on the quality assessments and reviews that data must undergo before being released to the public.

According to the Bureau’s Associate Director for Methodology and Standards, the working group is making progress in conducting the work laid out in its charter. She said that the working group has reviewed different practices in divisions across the Bureau and benchmarked these practices against appropriate organizations. It has moved on to the task of identifying quality problems that have resulted from data quality review practices in different parts of the Bureau and assessing what could have been done differently. However, the Bureau did not provide any time frame for the working group’s activities, information on how the Bureau intends to use the benchmarking exercises, or the intended scope and content of the Bureau-wide standard on quality in Bureau data releases.

The working group’s charter indicates that its schedule should reflect an expeditious effort to complete its tasks. The Associate Director for Methodology and Standards, to whom the working group reports, emphasized that setting standards is a long-term process and pointed out that the Bureau has never issued a standard in less than a year. She noted that participation in the working group is added to the other responsibilities of its members and that initially the working group had no dedicated staff. Additionally, she said that the working group does not have a time frame for completing these activities.

Greater Commitment to New Standards for Public Dissemination of Data Could Help Bureau Avoid Problems in Disseminating 2010 Census and Other Data

The standards that the Bureau has under development and activities of the working group are steps in the right direction. However, the Bureau has provided limited indication that developing and implementing standards on the quality of data it disseminates is a priority. It has no official plans for such an initiative, and these issues are not included in the Bureau’s plan for the 2010 Census. Until spring 2004, no additional resources were provided to support the working group, and a year and a half after it began, the group has not developed any new standards or guidelines or indicated when it will be ready to do so. Although Bureau officials said that 2010 Census dissemination decisions would adhere to Bureau dissemination guidelines, the actions the Bureau has taken to date are not enough to ensure that it will avoid in 2010 the types of problems encountered in disseminating data from the 2000 Census. A publicly issued, comprehensive, Bureau-wide data quality framework, with interrelated standards, and specific procedures (as evident in NCES, ESS, and Statistics Canada) could help ensure consistency of decisions about the quality of data from the next decennial census and the conditions under which the data will be disseminated. The benefits the Bureau can achieve by implementing data quality review standards should not be limited to the decennial census. Because the standards could apply to all of the data publicly distributed by the Bureau, the sooner they are developed and implemented across the Bureau, the sooner the Bureau will begin to reap their benefits.

Developing and Implementing Bureau-Wide Data Quality Review Standards Are Not Part of Official Bureau Plans

As noted above, the Bureau has not provided specific plans for further developing Bureau-wide data quality review standards or for implementing the broad data quality principles and guidelines outlined in its response to the OMB guidelines. It has not spelled out what needs to be done, how long it will take, what resources will be required, or how performance will be measured.

The Bureau’s evolving plans for the 2010 Census devote little attention to data quality review issues. As it has for past decennial censuses, the Bureau focuses its plans for the 2010 Census on ensuring the quality of information collected during the data collection phase, rather than on how it will address potential quality problems that might be identified before the data are released. Bureau officials told us that whatever standards are developed will be applied to disseminating data from the 2010 Census. However, they said that the next decennial census is still a number of years away, and disseminating data from the 2010 Census is still farther in the future.

Data Quality Review Standards Could Also Aid Other Data Programs before 2010

The 2010 Census is to differ significantly from is predecessor. The 2010 Census, if implemented as planned, will ask the entire population to provide only basic information on the short form necessary for congressional apportionment. It will no longer collect more extensive information on a longer questionnaire from a sample of the population. Instead, the Bureau has developed the ACS that among other things, is intended to replace the long-form census questionnaire. The detailed data on social and economic conditions that were previously collected as a part of the decennial census will in the future be collected annually in the ACS. In fact, the ACS is a key component of the Bureau’s plan for a reengineered 2010 Census. The ACS data are being collected and released annually for larger geographic areas, and data quality review standards could help improve the quality of these data immediately.

The Bureau has developed several measures of quality for the information included in the ACS and began reporting these measures on its Web site in December 2003. These reported measures are important steps in the right direction for the Bureau, but these program-specific measures have not been adopted as Bureau-wide standards for similar collections. A Bureau official said that these measures meet the requirements for minimum information on data quality of the Bureau’s standard, which is being piloted. The measures developed for the ACS program are being reviewed for possible implementation in other household surveys.

Conclusions

Fully documented, transparent, clearly defined, and consistently applied standards on the quality of data disseminated to the public can help ensure that the Bureau makes consistent decisions about how it addresses data quality problems. Additionally, such standards can help the public understand the Bureau’s reasons for its dissemination decisions, and can help protect the Bureau from allegations that it was inappropriately releasing or suppressing data. Because the cooperation and trust of the public is essential to a successful census, the Bureau must work to avert any loss of public confidence in the quality of data and in the integrity and objectivity of the Bureau.

Taken together, the quality problems that affected certain data from the 2000 Census underscore the importance of comprehensive data quality review guidelines for ensuring the Bureau makes more uniform decisions on data quality review and informs the public of limitations that could affect whether and how the data are employed.

The Bureau still has a long way to go in developing standards for the release of data to the public that will help avoid in the 2010 Census (and the ACS) the types of problems experienced in 2000. Additionally, since the standards would apply to all Bureau data collections, delay in their development and implementation means the Bureau is missing an opportunity for improving the quality of the other data it collects and disseminates. To avoid the problems it had with the dissemination of 2000 Census data the Bureau should place greater emphasis on developing and implementing data quality standards.

Although the Bureau has established a program for addressing standards development, we identified the following causes of concern.

In the absence of more detailed information about the activities and schedule of the working group, it is difficult to assess the Bureau’s progress in developing these standards. Over a year and a half after establishing the working group, the Bureau has publicly issued no new standards and has not publicly released plans that provide information on its schedule and agenda for developing the standards. Also the Bureau has not publicly sought comments on the working group’s initiatives through its advisory committees.

Plans for the 2010 Census do not address procedures for dealing with data quality problems that are identified during the data quality review phase.

The Bureau has not publicly announced any comprehensive plans for developing and implementing written, Bureau-wide quality standards and quality control processes.

A number of statistical agencies in the United States and elsewhere have developed comprehensive data quality review standards and quality control procedures that could serve as models for the Bureau. A Bureau-wide set of quality standards on data disseminated to the public covering both the quality of the data and quality control procedures would apply not only to the decennial census, but also to all other data collected by the Bureau and released to the public. Such standards could help the Bureau avoid some of the problems it experienced in disseminating data from the 2000 Census. Much of the data that were previously collected during the decennial census are now being collected under the ACS. Because these data are collected and released annually, the ACS, or other annual household surveys, could serve as a test for proposed standards.

Recommendations for Executive Action

To ensure that the 2010 Census, the ACS, and other Census data products will provide public data users with more complete, accurate, and useful information, we recommend that the Secretary of Commerce direct the Director of the U.S. Census Bureau to take the following two actions: 1. Accelerate the Bureau’s effort to establish comprehensive data quality standards by developing and making public a detailed plan, including interim milestones, for developing such standards and procedures.

2.

Include the implementation of the data quality review standards in the Bureau’s plans for the 2010 Census, and test new draft guidelines on data quality review using the annual ACS test program and other surveys.

Agency Comments and Our Evaluation

The Secretary of Commerce provided us with written comments on a draft of this report on September 27, 2004, which are reprinted in appendix I. Commerce agreed with one of our two recommendations–namely, to establish data quality review standards as part of its plans for the 2010 Census, and as indicated in the Secretary’s letter, the Bureau is taking steps to implement it. However, Commerce disagreed with our first recommendation that the Bureau accelerate its effort to establish comprehensive data quality standards. Commerce also identified some specific issues and suggested changes to provide additional context and clarification and in some cases technical corrections. We made these changes and corrections to the text as appropriate, but believe our first recommendation still applies.

Commerce took exception to our characterization of the amount of work that the Bureau has completed in developing comprehensive data quality review standards and in developing a specific standard for decisions on data release. However, the activities and documents Commerce cited to demonstrate the Bureau’s progress were mentioned in our draft report. For example, Commerce noted that the Bureau had developed a quality framework for Bureau documents, inventoried quality guidance used in specific program areas, and created an in-house repository of such documents. Commerce also pointed to the quality principles the Bureau developed and included as a part of its Information Quality Guidelines issued in response to OMB requirements. Our draft report credited the Bureau with all of these activities, although not always at the same level of detail as Commerce described in its comments.

Moreover, while these are important steps, most of this work is not available to the public. As we observed in our draft report, the only documents that have been made public on the agency’s Internet site are the documents required by the Information Quality Act and the related OMB guidelines: (1) the Bureau’s Information Quality Guidelines and (2) the standard allowing individuals to seek correction of certain errors in data disseminated by the Bureau.

Indeed, our primary concern is not with how much work has been done but whether the Bureau has well-documented, transparent, clearly defined quality review guidelines and standards, and whether the pace of its efforts is sufficient. As yet, the Bureau has not produced such guidelines nor has it documented plans for completing this work. Bureau officials said they will make existing standards available to the public on the agency’s Internet site by the end of 2004, but have not indicated which standards will be included.

Therefore, we reaffirm our recommendation that the Bureau should accelerate its efforts to establish such data quality review standards by making public a detailed plan, including interim milestones, for developing such standards and procedures. Such a plan can assist the Bureau in prioritizing its work and addressing the resource constraints that will inevitably be present. If, as Commerce maintained, much of the work has already been completed, implementing the recommendation should not be unduly burdensome or time consuming. While we commend the Bureau for agreeing with our recommendation to implement data review guidelines and standards for the 2010 Census and the ACS, we believe it needs to accelerate its efforts to complete, make public, and fully implement these data review standards. The more time that elapses, the greater the risk of releasing data with quality problems.

As agreed with your offices, unless you release its contents earlier, we plan no further distribution of this report until 30 days from its issue date. At that time, we will send copies of this report to the Chairman of the House Committee on Government Reform, the Secretary of Commerce, and the Director of the U.S. Census Bureau. Copies will be made available to others on request. This report will also be available at no charge on GAO’s home page at http://www.gao.gov.

Please contact me on (202) 512-6806 or by e-mail at daltonp@gao.gov if you have any questions. Other key contributors to this report were Robert Goldenkoff, Elizabeth Powell, Robert Parker, Michael Volpe, and Andrea Levine.

Comments from the Department of Commerce

GAO’s Mission

The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to www.gao.gov and select “Subscribe to Updates.”

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Congressional Relations

Public Affairs