Data Quality: Census Bureau Needs to Accelerate Efforts to Develop and Implement Data Quality Review Standards
Published: 2004-11-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Data from the decennial census are used to apportion and redistrict seats in the House of Representatives, distribute billions of dollars of federal funds, and guide the planning and investment decisions of the public and private sectors. Given the importance of these data, it is essential that they meet high quality standards before they are distributed to the public. After questions arose about the quality of certain data from the 2000 Census, the requesters asked GAO to review U.S. Census Bureau (Bureau) standards on the quality of data disseminated to the public.
section_title: What GAO Found
paragraphs: The Bureau did not have detailed agencywide standards for the review of data from the 2000 Census to determine if the data were of sufficient quality for public dissemination. Instead, analysts and managers in different parts of the Bureau primarily used their own judgment and unwritten, program-specific guidance to decide when and whether data should be released and what supporting information should accompany the data. The lack of sufficient data quality review standards led to a variety of problems, including missed opportunities for correcting data before release, inconsistent decisions on disseminating data with similar quality issues, and inadequate communication to users about the reasons for dissemination decisions. As a result, some users of data from the 2000 Census lost confidence in the quality of the data and in the Bureau's review procedures. In the 4 years since the 2000 Census, the Bureau has publicly issued general information quality guidelines, including eight performance principles, and one new standard that allows individuals to request correction of certain errors in data disseminated by the Bureau. Both of these documents resulted from the enactment of the Information Quality Act in 2000 and the subsequent guidelines issued by the Office of Management and Budget in 2002. However, except for the one standard, the Bureau did not provide any specific guidelines or procedures on the implementation of the general guidelines. The Bureau also began work on other standards, including one on minimal information that must be provided with data and another on discussion of errors in data released to the public. Neither has been issued in final form. In response to our earlier recommendations, the Bureau created an interdirectorate working group charged with developing and publicly issuing Bureau-wide standards for quality in data releases. The working group has taken some steps, but the Bureau has not provided information on the scope or the time frame for its efforts to develop these standards. The standards that the Bureau has under development and the activities of the working group are encouraging. However, it will be important for the Bureau to proceed with greater urgency to ensure that fully tested standards are in place for the 2010 Census. Until spring 2004, no additional resources were provided to support the working group, and over a year after it began, it has not issued any new standards or said when it will be ready to do so. A comprehensive, Bureau-wide data quality framework, with interrelated standards, and specific implementing procedures could help ensure consistent decisions about the quality of the data from the next decennial census and conditions under which the data will be disseminated. Moreover, the benefits the Bureau can achieve by developing and effectively implementing comprehensive data quality standards would not be limited to the decennial census. Because they would apply to all data disseminated by the Bureau, it will be important for any new standards to be developed promptly, implemented across the Bureau, and released to the public.

section_title: Letter
paragraphs: As one of the nation’s principal statistical agencies, the U.S. Census Bureau  (Bureau) collects and disseminates data that are used to apportion and  redistrict seats in the House of Representatives, distribute billions of  dollars of federal funds, and guide the planning and investment decisions of  the public and private sectors. Given the importance of Bureau data to our  economy and system of governance, census information, like other federal  statistics, must be of high quality before it is released to the public.  Specifically, the data must be accurate, timely, accessible, relevant, and  objective. Failure to meet this threshold could impair decision making and  erode public confidence in the information and the Bureau’s credibility.
Producing high-quality data is a continuing challenge, in part because the  methods used to collect and process census data are complex and subject  to some degree of error. Consequently, the Bureau must decide if and when  the quality of each set of data is high enough for it to be released and what  caveats, if any, are needed to inform users of any shortcomings that could  affect whether and how the data are used. The development and use of  comprehensive data quality review standards—if they are well  documented, transparent, clearly defined, and consistently applied—help  statistical agencies make such decisions and communicate the results of  the decisions to the public.
After the reliability of certain publicly released data from the 2000 Census  was called into question, concerns were raised about the adequacy of the  Bureau’s data quality review standards. Chief among these concerns was  that the Bureau did not routinely and consistently include an adequate  discussion of limitations to the data it disseminates or provide information  on how it reaches its dissemination decisions.
At your request, we reviewed the Bureau’s data quality review standards.  Specifically, as discussed with your offices, we (1) examined the review  standards that the Bureau had in place to guide decisions to disseminate  2000 Census data, (2) determined if the Bureau has subsequently developed  additional review standards to guide decisions about data quality, and   (3) assessed whether any such standards are likely to address for the 2010  Census the data quality review concerns raised after the release of certain  data from the 2000 Census.
To meet these objectives, we interviewed Bureau officials, reviewed  relevant documents prepared both before and after the enactment of the  Information Quality Act of 2000, and examined the guidelines other  statistical agencies and organizations have developed governing the public  dissemination of data. We did our audit work in Washington, D.C., and at  the Bureau’s headquarters in Suitland, Maryland, from August 2003 through  October 2004, in accordance with generally accepted government auditing  standards.
subsections: section_title: Background
paragraphs: The Bureau is best known for counting the nation’s population every  10 years. In the future, the Bureau intends to collect much of the data that  have traditionally been collected during the decennial census from the  long-form questionnaire with the annual ACS. Beyond the decennial  census, the Bureau also conducts numerous other surveys and censuses  that measure changing individual and household demographics and the  economic condition of the nation. Lawmakers and agency officials at the  federal, state, and local levels rely on these data when they make decisions  in a wide range of policy areas. Private-sector decision makers also use  census data to guide their business plans.
Because of the critical and varied uses of census information, it is  important that the Bureau’s published data meet minimum quality  standards. In addition, when the data are made public, it is equally  important for the Bureau to disclose what has been done to ensure the  quality of the data and identify any limitations so that potential consumers  can decide whether the data are appropriate for a particular use.
Some degree of error in the census (and in virtually any survey) is  inevitable because of limitations in enumeration, processing, and  dissemination methods and errors in responses and imputation of data  for nonresponses. Given the size and diversity of the U.S. population, the  effort to count the entire population and provide detailed demographic  characteristics every 10 years is one of the most complex of all government  operations. The Bureau devotes significant resources to minimizing error  and improving the quality of the decennial census.
Data quality standards and standardized quality control procedures can  provide a consistent basis for making data dissemination decisions and  informing the public about the quality of the data made available to it. In  2000, Congress passed what is now known as the Information Quality Act.  This legislation directed OMB to issue governmentwide guidelines that  “provide policy and procedural guidance to Federal agencies for ensuring  and maximizing the quality, objectivity, utility, and integrity of information  (including statistical information) disseminated by Federal agencies.”  The legislation also required each agency to issue its own implementing  guidelines that include administrative mechanisms allowing affected  persons to correct information maintained and disseminated by the agency.
The OMB guidelines, issued in final form in February 2002, define quality  as encompassing utility, objectivity, and integrity. The guidelines require  agencies to issue their own implementing guidelines by October 1, 2002.  Additionally, they mandate that agencies adopt a standard of quality as a  performance goal and act to incorporate data quality criteria into their data  dissemination practices. The guidelines also require agencies to develop  processes for reviewing the quality of data before they are disseminated.
Although OMB had some general guidance for survey processes prior to the  enactment of the Information Quality Act, other than requirements for the  evaluation of selected monthly and quarterly economic indicators, there  were no governmentwide requirements relating to the quality of data  disseminated by the federal agencies. Some statistical agencies within the  United States developed their own extensive guidelines and standards that  apply to data disseminated to the public. In July 2001, OMB identified the  statistical agencies within the Departments of Education and Energy, the  National Center for Education Statistics (NCES) and the Energy  Information Administration, as good examples of agencies that have  developed specific guidelines to implement their broad principles and  diverse professional standards.
Statistical agencies in other countries have also developed good examples  of comprehensive guidelines for ensuring the quality of data disseminated  to the public. Since 1985, Statistics Canada, the central statistical agency of  the Canadian government, has published quality guidelines for its statistical  activities. Subsequently, it added guidelines on quality assurance processes  and management context and developed a policy and standards on  informing users about data quality. More recently, the European Union  recognized the importance of comprehensive, well-documented guidelines  and standards to support its task of developing high-quality, comparable  statistics from member countries. All members of the European Statistical  System (ESS) have signed a quality declaration and approved 22  recommendations for quality for future work within the system.
section_title: Scope and Methodology
paragraphs: To address our first question on the standards that the Bureau had in place  to guide its data dissemination decisions, we interviewed census officials,  reviewed relevant agency documents, talked to data users, and reviewed  various complaints about the quality of 2000 Census data. We built on our  prior reports about the quality of data from the 2000 Census on Hispanic  subgroups and the homeless and the Bureau’s decision-making  processes for its decisions on whether to release those data. We also  reviewed other GAO reports addressing aspects of the Bureau’s procedures  for assessing the quality of disseminated data. From these reports, we  identified examples of several types of problems the Bureau encountered  with 2000 Census data, which might have been alleviated if the Bureau had  implemented data quality standards and procedures. Our examples of data  quality problems are not comprehensive, but illustrative.
To determine whether the Bureau has since developed Bureau-wide data  quality standards, and, if so, whether they would likely address for the  2010 Census the data quality problems raised after the 2000 Census,  we interviewed census officials responsible for developing agencywide  standards, examined documents related to the development of new  standards on data quality review, and reviewed the agency’s Internet site  for information on data quality review standards available to the public.  We also reviewed OMB guidelines on the quality of data disseminated by  federal agencies as well as the action taken by the Department of  Commerce and the Bureau in response to the guidelines. We attended  meetings of the Secretary of Commerce’s Decennial Census Advisory  Committee, the National Academy of Science Panel on Research on Future  Census Methods, and the Washington Statistical Society’s conference on  Quality Assurance in the Government, all of which examined issues related  to the quality of the data disseminated by the Bureau. We also discussed  information quality standards and guidelines with officials in Eurostat, the  statistical directorate of the European Union.
Additionally, we considered how the Bureau’s actions in developing  dissemination guidelines could improve the quality of data disseminated  after the 2010 Census and for other Bureau data collection programs, such  as the ACS that among other things, is intended to replace the long-form  census questionnaire. To benchmark the Bureau’s progress in developing  data quality review standards with that of other statistical agencies, we also  reviewed documents from entities that have developed standards for the  quality of data disseminated to the public, including NCES; Statistics  Canada, the central statistical agency of Canada; and ESS. However, we did  not evaluate the implementation or effectiveness of these guidelines and  standards or their specific applicability to the Bureau.
Our work addressed only standards and guidelines on data quality review.  Although OMB’s information quality guidelines and the Bureau’s guidelines  and performance principals cover all the key steps in data collection,  analysis, and dissemination, we did not look at the Bureau’s guidelines or  standards for ensuring quality during the planning and data collection  stages. Instead, as requested, we looked at Bureau guidance on steps taken  after data collection, that is, guidance related to processing data, assessing  their quality, and making them available to the public. We looked for  documents spelling out standards, guidelines, procedures, and other  criteria to guide decisions about identifying and correcting errors,  determining if and when to release data, and revising data after release.
Our audit work was conducted in Washington, D.C., and at the Bureau’s  headquarters in Suitland, Maryland, from August 2003 through October  2004. Our work was done in accordance with generally accepted  government auditing standards.
We requested comments on a draft of this report from the Secretary of  Commerce. On September 27, 2004, the Secretary provided written  comments on the draft. The comments are reprinted in appendix I.
section_title: Professional Judgment Drove Data Dissemination Decisions
paragraphs: The Bureau had no agencywide standards or guidelines in place to guide  decisions about disseminating data from the 2000 Census. Instead of  agencywide, written guidance, professionals within the different parts of  the Bureau primarily used their judgment and program-specific practices to  decide when and whether data should be released and what supporting  information, if any, should accompany them. This led to instances when  (1) data were released with uncorrected and undisclosed quality problems,  (2) inconsistent decisions were made on whether to release data sets with  similar quality problems, and (3) the reasons for certain data dissemination  decisions were inadequately communicated.
subsections: section_title: The Bureau Lacked Agencywide, Written Standards and Guidelines on the Quality of Census Data Disseminated to the Public
paragraphs: At the time the Bureau was making decisions about disseminating data  from the 2000 Census, it did not have written, agencywide guidelines or  standards to help inform its decisions on whether the data were of  sufficient quality to be released. Although Bureau officials emphasized that  the Bureau has a long tradition of high standards and procedures that yield  quality data, they acknowledged that these practices were primarily part of  the agency’s institutional knowledge. According to one official, key  individuals in each program area, relying primarily on professional  judgment, determined whether the quality of the data was acceptable for  release to the public. The official explained that the program areas develop  their own guidance and procedures for ensuring data quality. Sometimes  their guidance and procedures were written, but more often they were not.  Further, the Bureau had no central inventory or repository of the guidance  and practices of the different divisions.
section_title: Lack of Data Quality Review Guidelines Led to Inadequate Analysis of Potential Errors and Release of Data without Adequate Disclosure
paragraphs: As noted earlier, decennial census data are used to apportion and redistrict  Congress. As release of data for each of these purposes is required by  statute, they are known collectively as “public law” data. The Bureau had a  number of quality assurance programs and procedures for assessing the  accuracy of, and correcting errors in, public law and other data prior to  their release. However, the lack of standard procedures and guidelines for  dealing with quality problems contributed to lost opportunities to correct  errors in the count of the population identified before the data were  disseminated.
One such quality assurance program we reviewed was known as  Demographic Full Count Review, in which analysts were to identify,  investigate, and document suspected data discrepancies or “issues” in  order to clear census data files and products for subsequent processing or  public release. The Bureau contracted out some of the analysts’ work  because it lacked sufficient staff to conduct the Full Count Review on its  own. Bureau reviewers were to determine whether and how to correct the  data by weighing quality improvements against time and budget  constraints. Analysts identified 4,809 possible discrepancies, such as  instances when the location, population count, demographic  characteristics, or a combination of these for housing units and group  living facilities differed from what analysts expected. According to Bureau  officials, only 5 of the 4,809 issues were investigated and corrected prior to  the release of the public law data. All five involved group living facilities the  Bureau calls “group quarters” for which the Bureau had the correct  population counts, but placed the living facilities in the wrong places. The  Bureau did not investigate most of the remaining issues prior to the release  of the data in large part because they were insufficiently documented and  the Bureau lacked the time and people to further investigate these issues.
Subsequently, according to Bureau officials, the remaining issues that  contained sufficient documentation were investigated as a part of the  Count Question Resolution program, which ended in September 2003.
As we noted in our July 2002 report, the fact that public law data were  released with over 4,800 unresolved data issues of unknown validity,  magnitude, and impact is cause for concern. To the extent these unresolved  discrepancies were in fact true errors in the population count or geography,  they could have affected the drawing of congressional districts as well as  other purposes for which census data are used.
The existence of data quality review guidelines could have helped the  Bureau in this situation. For example, we found that the Bureau’s lack of  clearly defined requirements for documenting data issues resulted in a  significant number of cases with inadequate documentation that the  Bureau could not use to resolve the issues. Additionally, the Bureau had no  mechanism for setting priorities for resolving these potential data errors.  A sufficient set of guidelines could have helped the Bureau to ensure that  the documentation of potential errors was adequate for decision making  and to maximize the use of scarce resources in addressing the various data  issues, giving top priority to investigating discrepancies likely to have the  most adverse affect on the data.
The quality of certain data from the census long-form questionnaire have  been called into question as well. In its 2004 comprehensive review of the  2000 Census, a panel of the National Research Council of the National  Academy of Sciences assessed the quality of the long-form data using  various benchmarks, and found that the overall quality of the information  was less than that of the short-form questionnaire and had deteriorated  since the 1990 Census. For example, at least 32 percent of the  respondents failed to provide information on their property taxes, and  30 percent did not respond to all or some of the questions relating to  income (compared with 12 and 13 percent, respectively, in 1990).  Additionally, the panel noted that the Bureau did not measure and report  the impact of some of the steps it took to address problems with missing  data and recommended that the Bureau develop such measures and inform  users about the need for caution in analyzing and interpreting these data.
Even more significant quality problems plagued the data for residents of  group quarters. The panel found these data to be poor in comparison with  the data for household residents, and also in comparison with data for  group quarters from the 1990 Census. In 2000, missing data rates for some  items were over 25 percent—one item was over 50 percent—for all  residents of group quarters, and as high as 75 percent for prison inmates.  Given the prevalence of missing data from residents of group quarters, the  panel questioned whether the Bureau should have published these data at  all for some or all types of group quarters.
section_title: The Bureau Made Conflicting Dissemination Decisions on Data with Similar Quality Problems
paragraphs: Our earlier work on Hispanic subgroups and the homeless showed that the  Bureau’s approach to data quality review led to inconsistent decision  making. Faced with similar quality problems in data from the 2000 Census,  Bureau officials made different decisions about disseminating data and did  not explain the reasons for their decisions.
For example, in an effort to improve the count of Hispanics and simplify  the questionnaire, the Bureau redesigned its 2000 Census question on  Hispanic origin and dropped a list of examples of Hispanic subgroups  included in the 1990 Census. In May 2001, the Bureau released data on  Hispanics and Hispanic subgroups as part of its first release summarizing  the results of the 2000 Census. The Bureau also published The Hispanic  Population, a 2000 Census brief that provided an overview of the size and  distribution of the Hispanic population in 2000 and highlighted changes in  the population since the 1990 Census. For the first time, the Bureau  released data on Hispanic subgroups as a part of its release of the Full  Count Review data even though it had not fully tested the impact of  questionnaire changes on the subgroup data and provided little discussion  of the potential limitations of the data.
Shortly after the Hispanic and Hispanic subgroup data from the 2000  Census were released to the public, questions were raised about the counts  for specific Hispanic subgroups. For example, the reported count of  Dominican Hispanics was significantly lower than counts reported in other  Bureau surveys. Representatives of affected Hispanic subgroups asked for  an investigation and explanation of why the Bureau reported data that  these subgroups considered to be of questionable quality. We found that a  key factor behind the Bureau’s release of apparently less-than-accurate  Hispanic subgroup data appeared to be a lack of adequate guidelines  governing decisions on quality considerations that should be addressed  before making data publicly available.
In contrast, the Bureau, citing quality problems, decided not to separately  report certain information on people without conventional housing,  including those commonly referred to as “homeless.” Enumerating this  segment of the population has been an ongoing challenge for the Bureau.  To help locate and count these people in 2000, the Bureau partnered with  organizations providing services to the homeless and with local  governments, some of which put considerable resources into the effort.  When the Bureau decided not to separately report the number of people in  transitional and emergency shelters as originally planned because of data  quality problems, some of the organizations and local governments, which  had expected to use the data for directing services to the homeless,  questioned the Bureau’s process for making that decision. Additionally, we  found that the decision about when and whether to release data on people  in emergency and transitional shelters changed several times. Decisions  about the release of data with identified quality problems were not well  documented and communicated with some Bureau partners and other  stakeholders.
As a result, outside parties interested in both the Hispanic and homeless  data from the 2000 Census questioned the quality of the data, the  procedures the Bureau used to determine what data to release, and the  value of their own participation in helping the Bureau prepare for the  2000 Census. Because the Bureau’s reasons for data release decisions were  not obvious, and it had no guidelines or standards that spelled out criteria  for decisions, the Bureau left itself open to questions about the objectivity  of its decisions and risked loss of public confidence.
In our reports on Hispanic and homeless Census 2000 data, we  recommended that the Bureau (1) develop agencywide guidelines for its  decisions on the level of quality needed to release data to the public, how to  characterize any limitations in the data, and when it is acceptable not to  release data and (2) ensure that these guidelines are documented,  transparent, clearly defined, and consistently applied. We also  recommended that the Bureau ensure that its plans for releasing data are clearly and consistently communicated to the public. The Bureau agreed  with each of our recommendations and asked its Methodology and  Standards Council to review existing statistical and quality guidelines,  bring them together in one place, and develop data quality standards. We  discuss the Bureau’s actions later in this report.
section_title: The Bureau Has Made Limited Progress in Publicly Issuing New Standards on the Quality of Data Disseminated to the Public since the 2000 Census
paragraphs: Since the first results of the 2000 Census were released, the Bureau has  publicly issued a set of information quality guidelines and one new  standard on the quality of data disseminated to the public. As required by  the Information Quality Act and the OMB guidelines, the Department of  Commerce and the Bureau published Information Quality Guidelines,  but the guidelines contain only general quality goals and principles and do  not provide any specific guidelines or procedures on the implementation of  the general principles. Also as required by the Information Quality Act and  the OMB guidelines, the Bureau published a standard that described a  procedure allowing individuals to seek correction of certain errors in data  disseminated by the Bureau. Additionally, the Bureau has begun developing  several other standards on the quality of data disseminated to the public,  but none have been publicly released in final form.
In March 2003, in response to our recommendations, the Bureau  established an interdirectorate working group charged with the broad  mandate of developing Bureau-wide standards for quality in data releases.  The working group has taken some steps to address the tasks laid out in its  charter. However, the Bureau has not provided information on the scope or  the time frame for developing these standards.
subsections: section_title: The Bureau Has Taken Steps to Expand Its Guidance on Data Quality Review
paragraphs: Recognizing the paucity of Bureau-wide written standards on the quality of  data disseminated to the public, the Bureau established a Quality Program  in 1999 to develop consistent processes for producing quality products  across the Bureau. The Bureau’s Associate Director for Methodology and  Standards with input from chiefs in a number of divisions compiled an  inventory of data quality review documents used in different divisions  and developed a Bureau-wide quality framework. The resulting quality  framework was adopted to serve as a vehicle through which “the  demographic, economic, and decennial areas can share and support  common principles, standards, and guidelines.” This framework provides  the organization for documents in the intranet portal known as the Quality  Management Repository (QMR). Additionally, the Bureau’s description of  the quality framework spells out the process for developing, reviewing, and  approving quality framework documents. The document describing the  quality framework and most of the documents in the QMR are internal  documents not available to the public through the agency’s Internet site.  However, Bureau officials indicated that they intend to make some of the  standards available through the Internet later in calendar year 2004.
The Bureau has publicly issued two data quality review documents and  made them available through the Internet. In October 2002, in response to  the requirements of the OMB guidelines, the Bureau published a set of  information quality guidelines in eight performance areas, including the  establishment of review procedures. The Bureau’s guidelines identify broad  quality goals and principles, but do not provide specific guidance to ensure  consistent decisions. For example, the guideline on predissemination  review of data says that “all documents released by the Census Bureau  undergo extensive review that encompasses the content, statistical and  survey methodology, and policy implications of the document,” and that  this review “ensures that the data and text of the document meet Census  Bureau standards for quality” or the Bureau reserves the right to withhold  the data from the public. However, the guideline does not indicate what the  Bureau “standards for quality” are, how the Bureau will know if the data  meet the standards, or who within the Bureau is responsible for the review.
The second document issued and made available on the Bureau’s Web site  is Census Bureau Standard: Correcting Information That Does Not  Comply with Census Bureau Section 515 Information Quality  Guidelines in March 2002. This standard was also issued in response to the  specific requirements of the Information Quality Act and the OMB  guidelines that agencies provide procedures for correcting certain errors  identified in data they disseminated and post these guidelines on their Web  sites. The standard established procedures that allow individuals to request  a correction of information they believe is erroneous and the Bureau to  review the evidence and determine whether a correction is warranted.
The Bureau has also approved several additional Bureau-wide data quality  review documents for implementation and internal distribution through the  QMR on its intranet. On March 18, 2003, the Bureau issued Census Bureau  Standard: Minimal Information to Accompany Any Report of Census  Bureau Data for a 6-month trial period. The standard identifies 13 specific  items that the Bureau should report for every survey or census and  specifies who is responsible for ensuring adherence to the standard. An  accompanying memorandum from the Associate Director for Methodology  and Standards to program associate directors said that implementation  issues would be documented during the trial period and appropriate  changes made prior to the final release of the standard. Even though the  trial period is over, the Bureau has not made such changes or publicly  issued the standard in final form. However, the standard is still in effect on  a trial basis, according to one Bureau official.
The Bureau also released its Census Bureau Guideline: Quality Profiles  on March 9, 2004, through the QMR. The document outlines a standardized  quality profile, recommended for all recurring surveys and certain other  programs, which is intended to present a consistent set of information on  the quality of each program. As a guideline rather than a standard, this  guidance is recommended rather than mandatory.
In addition, the Bureau has also initiated work on several proposals for  additional standards. For example, a standard for discussion and  presentation of errors in data disseminated to the public is under  development. This standard is based on a technical paper that was issued  in 1974 and revised in 1987. The Bureau said it would be issued in the near  future, but has not provided a specific date.
section_title: Bureau Working Group Has Begun Developing Additional Standards on Data Quality Review, but None Have Been Issued
paragraphs: In response to our recommendations from reports on both homeless and  Hispanic subgroup data from the 2000 Census, the Bureau established an  interdirectorate working group on March 3, 2003, with the broad mandate  to develop Bureau-wide standards for quality in data releases. However, the  working group has not yet issued any draft or final standards or developed  a time frame for doing so.
The working group is composed primarily of assistant division chiefs from  the program areas—decennial, demographic, and economic. An assistant  division chief from the Demographic Statistical Methods Division chairs  the group.
According to the working group’s charter, its mission is to   “Document current Census Bureau data review procedures,  “Benchmark Census Bureau review procedures with that of other   “Document Census Bureau situations where review of data indicates  data does not meet “quality requirements” and the outcome of those  situations,   “Propose standards for quality in Census Bureau data products,   “Benchmark quality requirements for data release with other agencies,  “Develop Census Bureau Standard: Quality in Census Bureau Data  Releases.”
Bureau officials told us that the working group has reviewed the published  detailed guidelines from NCES and the Canadian national statistical office.  Benchmarking discussions have taken place with the Bureau of Labor  Statistics and the National Center for Health Statistics. Additionally, the  working group met with an official from the New Zealand national  statistical office to discuss its standards. The group is also planning  meetings with additional federal agencies. These organizations have  published detailed guidance on how broad principles on data quality are to  be put into practice, notably the organizational responsibilities and internal  control mechanisms for applying them.
For example, Statistics Canada, the central statistical agency of the  Canadian government, has developed an extensive and detailed set of  quality guidelines that covers the quality of data disseminated to the public  and the quality control processes that are supposed to be applied to ensure  the quality of the data. In March 2000, Statistics Canada published its  Policy on Informing Users of Data Quality and Methodology, which  specifies the organization’s responsibilities to inform users about the  concepts and methodology for collecting, processing, and analyzing its  data; the accuracy of these data; and any other features that affect their  quality or fitness for use. By detailing mandatory documentation standards,  guidelines for additional documentation, and examples of mandatory  standardized summary documentation, the policy enhances the likelihood  of consistent decision making throughout the organization. Additionally,  making this information public ensures that any data user can determine  what has been done to ensure the quality of the data and Statistics Canada’s  reasons for its decisions about release.
NCES has developed detailed standards designed to implement its broader  policies on dissemination of statistical data. An NCES standard includes a  section entitled “Establishment of Review Procedures,” which includes a  table showing the required reviews for each type of product and an  illustration of the key steps in the review and adjudication process. As with  the Statistics Canada policy, the NCES standard provides information on  the quality assessments and reviews that data must undergo before being  released to the public.
According to the Bureau’s Associate Director for Methodology and  Standards, the working group is making progress in conducting the work  laid out in its charter. She said that the working group has reviewed  different practices in divisions across the Bureau and benchmarked these  practices against appropriate organizations. It has moved on to the task of  identifying quality problems that have resulted from data quality review  practices in different parts of the Bureau and assessing what could have  been done differently. However, the Bureau did not provide any time frame  for the working group’s activities, information on how the Bureau intends  to use the benchmarking exercises, or the intended scope and content of  the Bureau-wide standard on quality in Bureau data releases.
The working group’s charter indicates that its schedule should reflect an  expeditious effort to complete its tasks. The Associate Director for  Methodology and Standards, to whom the working group reports,  emphasized that setting standards is a long-term process and pointed  out that the Bureau has never issued a standard in less than a year.  She noted that participation in the working group is added to the other  responsibilities of its members and that initially the working group had no  dedicated staff. Additionally, she said that the working group does not  have a time frame for completing these activities.
section_title: Greater Commitment to New Standards for Public Dissemination of Data Could Help Bureau Avoid Problems in Disseminating 2010 Census and Other Data
paragraphs: The standards that the Bureau has under development and activities of the  working group are steps in the right direction. However, the Bureau has  provided limited indication that developing and implementing standards on  the quality of data it disseminates is a priority. It has no official plans for  such an initiative, and these issues are not included in the Bureau’s plan for  the 2010 Census. Until spring 2004, no additional resources were provided  to support the working group, and a year and a half after it began, the group  has not developed any new standards or guidelines or indicated when it  will be ready to do so. Although Bureau officials said that 2010 Census  dissemination decisions would adhere to Bureau dissemination guidelines,  the actions the Bureau has taken to date are not enough to ensure that it  will avoid in 2010 the types of problems encountered in disseminating data  from the 2000 Census. A publicly issued, comprehensive, Bureau-wide data  quality framework, with interrelated standards, and specific procedures  (as evident in NCES, ESS, and Statistics Canada) could help ensure  consistency of decisions about the quality of data from the next decennial  census and the conditions under which the data will be disseminated.  The benefits the Bureau can achieve by implementing data quality review  standards should not be limited to the decennial census. Because the  standards could apply to all of the data publicly distributed by the Bureau,  the sooner they are developed and implemented across the Bureau, the  sooner the Bureau will begin to reap their benefits.
subsections: section_title: Developing and Implementing Bureau-Wide Data Quality Review Standards Are Not Part of Official Bureau Plans
paragraphs: As noted above, the Bureau has not provided specific plans for further  developing Bureau-wide data quality review standards or for implementing  the broad data quality principles and guidelines outlined in its response to  the OMB guidelines. It has not spelled out what needs to be done, how long  it will take, what resources will be required, or how performance will be  measured.
The Bureau’s evolving plans for the 2010 Census devote little attention to  data quality review issues. As it has for past decennial censuses, the Bureau  focuses its plans for the 2010 Census on ensuring the quality of information  collected during the data collection phase, rather than on how it will  address potential quality problems that might be identified before the data  are released. Bureau officials told us that whatever standards are  developed will be applied to disseminating data from the 2010 Census.  However, they said that the next decennial census is still a number of years  away, and disseminating data from the 2010 Census is still farther in the  future.
section_title: Data Quality Review Standards Could Also Aid Other Data Programs before 2010
paragraphs: The 2010 Census is to differ significantly from is predecessor. The 2010  Census, if implemented as planned, will ask the entire population to  provide only basic information on the short form necessary for  congressional apportionment. It will no longer collect more extensive  information on a longer questionnaire from a sample of the population.  Instead, the Bureau has developed the ACS that among other things, is  intended to replace the long-form census questionnaire. The detailed data  on social and economic conditions that were previously collected as a part  of the decennial census will in the future be collected annually in the ACS.  In fact, the ACS is a key component of the Bureau’s plan for a reengineered  2010 Census. The ACS data are being collected and released annually for  larger geographic areas, and data quality review standards could help  improve the quality of these data immediately.
The Bureau has developed several measures of quality for the information  included in the ACS and began reporting these measures on its Web site in  December 2003. These reported measures are important steps in the right  direction for the Bureau, but these program-specific measures have not  been adopted as Bureau-wide standards for similar collections. A Bureau  official said that these measures meet the requirements for minimum  information on data quality of the Bureau’s standard, which is being  piloted. The measures developed for the ACS program are being reviewed  for possible implementation in other household surveys.
section_title: Conclusions
paragraphs: Fully documented, transparent, clearly defined, and consistently applied  standards on the quality of data disseminated to the public can help ensure  that the Bureau makes consistent decisions about how it addresses data  quality problems. Additionally, such standards can help the public  understand the Bureau’s reasons for its dissemination decisions, and can  help protect the Bureau from allegations that it was inappropriately  releasing or suppressing data. Because the cooperation and trust of the  public is essential to a successful census, the Bureau must work to avert  any loss of public confidence in the quality of data and in the integrity and  objectivity of the Bureau.
Taken together, the quality problems that affected certain data from the  2000 Census underscore the importance of comprehensive data quality  review guidelines for ensuring the Bureau makes more uniform decisions  on data quality review and informs the public of limitations that could  affect whether and how the data are employed.
The Bureau still has a long way to go in developing standards for the  release of data to the public that will help avoid in the 2010 Census (and the  ACS) the types of problems experienced in 2000. Additionally, since the  standards would apply to all Bureau data collections, delay in their  development and implementation means the Bureau is missing an  opportunity for improving the quality of the other data it collects and  disseminates. To avoid the problems it had with the dissemination of  2000 Census data the Bureau should place greater emphasis on developing  and implementing data quality standards.
Although the Bureau has established a program for addressing standards  development, we identified the following causes of concern.
In the absence of more detailed information about the activities and  schedule of the working group, it is difficult to assess the Bureau’s  progress in developing these standards. Over a year and a half after  establishing the working group, the Bureau has publicly issued no new  standards and has not publicly released plans that provide information  on its schedule and agenda for developing the standards. Also the  Bureau has not publicly sought comments on the working group’s  initiatives through its advisory committees.
Plans for the 2010 Census do not address procedures for dealing with  data quality problems that are identified during the data quality review  phase.
The Bureau has not publicly announced any comprehensive plans for  developing and implementing written, Bureau-wide quality standards  and quality control processes.
A number of statistical agencies in the United States and elsewhere have  developed comprehensive data quality review standards and quality control  procedures that could serve as models for the Bureau. A Bureau-wide set of  quality standards on data disseminated to the public covering both the  quality of the data and quality control procedures would apply not only to  the decennial census, but also to all other data collected by the Bureau and  released to the public. Such standards could help the Bureau avoid some of  the problems it experienced in disseminating data from the 2000 Census.  Much of the data that were previously collected during the decennial  census are now being collected under the ACS. Because these data are  collected and released annually, the ACS, or other annual household  surveys, could serve as a test for proposed standards.
section_title: Recommendations for Executive Action
paragraphs: To ensure that the 2010 Census, the ACS, and other Census data products  will provide public data users with more complete, accurate, and useful  information, we recommend that the Secretary of Commerce direct the  Director of the U.S. Census Bureau to take the following two actions:  1. Accelerate the Bureau’s effort to establish comprehensive data quality  standards by developing and making public a detailed plan, including  interim milestones, for developing such standards and procedures.
2.
Include the implementation of the data quality review standards in the  Bureau’s plans for the 2010 Census, and test new draft guidelines on  data quality review using the annual ACS test program and other  surveys.
section_title: Agency Comments and Our Evaluation
paragraphs: The Secretary of Commerce provided us with written comments on a draft  of this report on September 27, 2004, which are reprinted in appendix I.  Commerce agreed with one of our two recommendations–namely, to  establish data quality review standards as part of its plans for the 2010  Census, and as indicated in the Secretary’s letter, the Bureau is taking steps  to implement it. However, Commerce disagreed with our first  recommendation that the Bureau accelerate its effort to establish  comprehensive data quality standards. Commerce also identified some  specific issues and suggested changes to provide additional context and  clarification and in some cases technical corrections. We made these  changes and corrections to the text as appropriate, but believe our first  recommendation still applies.
Commerce took exception to our characterization of the amount of work  that the Bureau has completed in developing comprehensive data quality  review standards and in developing a specific standard for decisions on  data release. However, the activities and documents Commerce cited to  demonstrate the Bureau’s progress were mentioned in our draft report.  For example, Commerce noted that the Bureau had developed a quality  framework for Bureau documents, inventoried quality guidance used in  specific program areas, and created an in-house repository of such  documents. Commerce also pointed to the quality principles the Bureau  developed and included as a part of its Information Quality Guidelines  issued in response to OMB requirements. Our draft report credited the  Bureau with all of these activities, although not always at the same level of  detail as Commerce described in its comments.
Moreover, while these are important steps, most of this work is not  available to the public. As we observed in our draft report, the only  documents that have been made public on the agency’s Internet site are the  documents required by the Information Quality Act and the related OMB  guidelines: (1) the Bureau’s Information Quality Guidelines and (2) the  standard allowing individuals to seek correction of certain errors in data  disseminated by the Bureau.
Indeed, our primary concern is not with how much work has been done but  whether the Bureau has well-documented, transparent, clearly defined  quality review guidelines and standards, and whether the pace of its efforts  is sufficient. As yet, the Bureau has not produced such guidelines nor has it  documented plans for completing this work. Bureau officials said they will  make existing standards available to the public on the agency’s Internet site  by the end of 2004, but have not indicated which standards will be included.
Therefore, we reaffirm our recommendation that the Bureau should  accelerate its efforts to establish such data quality review standards by  making public a detailed plan, including interim milestones, for developing  such standards and procedures. Such a plan can assist the Bureau in  prioritizing its work and addressing the resource constraints that will  inevitably be present. If, as Commerce maintained, much of the work has  already been completed, implementing the recommendation should not be  unduly burdensome or time consuming. While we commend the Bureau for  agreeing with our recommendation to implement data review guidelines  and standards for the 2010 Census and the ACS, we believe it needs to  accelerate its efforts to complete, make public, and fully implement these  data review standards. The more time that elapses, the greater the risk of  releasing data with quality problems.
As agreed with your offices, unless you release its contents earlier, we plan  no further distribution of this report until 30 days from its issue date. At  that time, we will send copies of this report to the Chairman of the House  Committee on Government Reform, the Secretary of Commerce, and the  Director of the U.S. Census Bureau. Copies will be made available to others  on request. This report will also be available at no charge on GAO’s home  page at http://www.gao.gov.
Please contact me on (202) 512-6806 or by e-mail at daltonp@gao.gov if you  have any questions. Other key contributors to this report were Robert  Goldenkoff, Elizabeth Powell, Robert Parker, Michael Volpe, and Andrea  Levine.
section_title: Comments from the Department of Commerce
subsections: section_title: GAO’s Mission
paragraphs: The Government Accountability Office, the audit, evaluation and  investigative arm of Congress, exists to support Congress in meeting its  constitutional responsibilities and to help improve the performance and  accountability of the federal government for the American people. GAO  examines the use of public funds; evaluates federal programs and policies;  and provides analyses, recommendations, and other assistance to help  Congress make informed oversight, policy, and funding decisions. GAO’s  commitment to good government is reflected in its core values of  accountability, integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents at no cost  is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts  newly released reports, testimony, and correspondence on its Web site. To  have GAO e-mail you a list of newly posted products every afternoon, go to  www.gao.gov and select “Subscribe to Updates.”
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Congressional Relations
section_title: Public Affairs