Implementing and Improving Quality Standards 

at the U.S. Census Bureau 

John M. Bushery and Pamela D. McGovern 

U.S. Bureau of the Census, Planning, Research, and Evaluation Division, 

Washington, DC  20233 

john.m.bushery@census.gov; pamela.d.mcgovern@census.gov 

1 Introduction – the Quality Program 

In December of 1998, the Census Bureau prepared a business plan for developing a Quality Program.  The plan relied on 

inter­divisional and inter­directorate teams to contribute to specific efforts that relate to quality.  Essentially, the Quality Program 

builds upon the cross­directorate work where the program areas maintain responsibility for the quality of their products and 

processes (Landman, 2001).  The Methodology and Standards (M&S) Council provides leadership and general direction for the 

Quality Program.  The main goal of the Quality Program is to advance and maintain the Census Bureau's interests in matters 

involving product and process quality.  Key components of the Census Bureau’s Quality Program include the Quality Framework, 

the Methodology and Standards Council, and the Quality Program Staff. 

This paper describes some key components of the Quality Program, but focuses how one element of the program, the Quality 

Program Staff, supports the program regarding the Census Bureau Quality Standards.  Section 2 of this paper discusses the 

standards development process.  As mentioned in Section 1.3 below, a major responsibility of the Quality Program Staff is to 

evaluate the implementation of the Census Bureau’s Quality Standards.  This work includes identifying opportunities for 

improvement and recommending improvements in the standards and their implementation.  Section 3 of this paper discusses the 

implementation and evaluation processes. 

Recently the Quality Program Staff received the assignment of evaluating the Census Bureau’s Quality Standards as an entire 

package, benchmarking them against the standards of other statistical organizations.  In June 2005, the staff delivered a draft report 

to the M&S Council and the Program Associate Directors, detailing their findings and recommending actions to close the gaps 

identified in the Census Bureau’s standards.  Section 4 of this paper describes this benchmarking study. 

1.1 Quality Framework 

The Quality Framework is a key component of the Quality Program and serves as the foundation of quality­related documents 

concerning Census Bureau products and processes.  The framework consists of five kinds of documents: principles, standards, 

guidelines, current practices, and best practices.  These documents reside in the Quality Management Repository (QMR) on the 

Census Bureau’s Intranet.  In addition, principles and standards reside on the Census Bureau’s Internet “Quality Site,” which 

contains information associated with the Office of Management and Budget’s (OMB) 515 Information Quality Guidelines 

(http://www.census.gov/quality/). 

The Quality Framework also currently consists of eight unique process categories: planning, content, design, data collection, data 

processing, dissemination, data products and services, and data quality, analysis, and evaluation.  The Census Bureau plans to 

revise these process categories to address the findings in the benchmarking study discussed in Section 4. 

This report is released to inform interested parties of work in progress and to encourage discussion.  The views expressed on these issues are 

those of the authors and not necessarily those of the U.S. Census Bureau. 
   

       

   

                 

     

1.2 The Methodology and Standards Council 

The Methodology and Standards (M&S) Council serves as the steering committee for the Quality Program.  The M&S Council 

consists of the division chiefs of the methodology and research areas and the Census Bureau’s senior statisticians and 

methodologists.  The Council is tasked with ensuring good methodology and good statistical practices for the Census Bureau’s 

survey, census, and estimation processes. 

1.3 Quality Program Staff 

The Quality Program Staff was formed in early 2004.  The staff consists of three employees with experience in quality control, 

quality assurance, process improvement, quality auditing, and other quality improvement methods.  They provide full­time support 

to the Quality Program and its initiatives.  The Quality Program Staff: 

C Supports Census Bureau programs in complying with the OMB 515 Information Quality Guidelines; 

C Supports the development, implementation, and evaluation of standards and guidelines; 

C Assists program area efforts to improve the quality of Census Bureau products and processes; 

C Provides tools to assess, measure, and improve quality; and 

C Provides consulting services and guidance to the program areas, with respect to quality management, quality 

assurance, quality control, and quality improvement initiatives. 

The second item, the responsibility to support the development, implementation, and evaluation of standards and guidelines, is 

of particular relevance to this discussion and will be the focus of this paper. 

2 The Standards Development Process 

The Quality Program Staff serves on the teams that develop standards and provides an interface between the team and the M&S 

Council.  Even when not assigned to a team, they help the team to keep the development process moving forward. 

To date, the Census Bureau has developed nine quality standards.  

•  Correcting Information that Does Not Comply with Census Bureau 515 Information Quality Guidelines 

•  Definitions for Survey and Census Metadata 

•  Describing the Sources and Accuracy of Data Tabulations 

•  Disclosure Review 

•  Dissemination of Census and Survey Data Products 

•  Minimal Information to Accompany Any Report of Survey or Census Data 

•  Pretesting Questionnaires and Related Materials for Surveys and Censuses 

•  Response Rate Definitions 

•  Review and Approval of Census Bureau Documents and Presentations 

In addition to these standards, a standard on data release is currently under development. 

Originally, an ad hoc process existed to determine the need for a standard.  Often the Associate Director for Methodology and 

Standards identified a need and, with the M&S Council, chartered a team to develop the standard.  The teams work on the 

development of the standards in addition to their normal work assignments. 

Recently in a benchmarking study, the Quality Program Staff identified areas where the Census Bureau should develop quality 

standards (see Section 4).  These areas are based on the OMB Standards for Statistical Surveys (OMB, 2005).  The Census Bureau 

plans to focus first on developing standards related to data dissemination (i.e., data release and data review standards) since this 

has been identified as a critical area. 

In developing standards, the teams follow this general process: 

1)  The need for a standard is identified. 

2)  The M&S Council charters a team with expertise on the standard topic. 

3)  The team develops and drafts the standard. 

4)  The M&S Council and the affected directorates review the draft and the team revises, as appropriate. 

5)  The M&S Council and Associate Directors approve and sign the final version. 
                 

   

       

  

     

     

     

               

 

  

6)  The Quality Program Staff publishes the new standard. 

The Quality Program Staff has drafted a charter template which outlines the development process and documents the procedure 

for developing new quality standards.  This charter documents the roles and responsibilities (of the M&S Council, the standards 

team, and the Quality Program Staff), deliverables, and a milestone schedule for developing a quality standard.  To facilitate 

communication between the teams and the M&S Council, the charter also specifies that the M&S Council must receive monthly 

status reports on the progress of the standard and whether it is on schedule.  By creating a documented procedure for the standards 

development process, the Quality Program Staff believes the process will be more efficient and timely. 

3 Implementation and Evaluation of Standards 

3.1 Implementation of Standards 

To raise staff awareness of the standards, the Associate Director for Methodology and Standards and the chairs of the standard 

development teams met with staff from the program directorates to discuss the standards and their implementation.  In a related 

effort, the Quality Program Staff has also presented training to staff attending the quality course of the project management training 

series.  The Census Bureau continues to explore methods to ensure that employees are knowledgeable about the quality standards 

and their roles in implementing them. 

Each of the program directorates is responsible for implementing and enforcing the standards.  Within a directorate, the subject 

matter areas generally are responsible for following the quality standards when creating data products, and the statistical 

methodology areas generally are responsible for assuring compliance with the standards.  For the most part, except for the report 

review and clearance procedure, no documented procedures exist for implementing the standards or assuring compliance.  Instead, 

the Census Bureau relies on long­held, but undocumented, practices that are transmitted by word of mouth and on­the­job training. 

The M&S Council, supported by the Quality Program Staff, is encouraging the program directorates to develop and document 

procedures to implement the quality standards, including follow­up quality assurance activities to verify compliance with the 

standards. 

The Census Bureau’s Economic Directorate, in partnership with the Quality Program Staff, has begun to develop a quality audit 

program.  This audit program is intended to provide managers of the Economic Directorate recommendations for improving the 

quality of their products and processes.  One of the main objectives of this quality audit is to assess the program area’s compliance 

with the OMB Standards for Statistical Surveys.  These standards are the benchmark against which the Quality Program Staff 

compared the Census Bureau’s Quality Standards and are broader in scope than the Census Bureau’s Quality Standards.  Although 

not a formal quality assurance process, this audit will identify weaknesses in the survey programs, regarding their compliance with 

the OMB standards.  The first quality audit, of the Manufacturers’ Shipments Inventories, and Orders Survey (the M3 program) 

was completed in the summer of 2005. 

3.2 Evaluation of Standards 

The M&S Council is responsible for the “initiation of periodic evaluations, reviews, and updates to the standards, as necessary.” 

The Quality Program Staff performs these evaluations at the direction of the M&S Council.  The evaluations have the goal of 

identifying what problems occur in implementing the standards and recommending ways to address those problems.  Although 

the Quality Program Staff evaluates the implementation of the quality standards, their job is not to assure that specific products 

or processes comply with the standards.  As discussed in the preceding section, the program areas are responsible for performing 

this quality assurance function.  However, the Quality Program Staff does provide guidance to the program areas with their quality 

assurance activities. 

To date the Quality Program Staff has evaluated one standard, Minimal Information to Accompany Any Report of Survey or 

Census Data.  In late 2004, the Quality Program Staff evaluated the implementation of the Census Bureau Standard: Minimal 

Information to Accompany Any Report of Survey or Census Data.  This standard sets forth requirements for what information must 

be provided when releasing a report of survey or census data.  The M&S Council chose this as the first standard to be evaluated 

because it addresses the dissemination of data.  The purpose of the evaluation was not to serve as a quality assurance function, 

identifying noncompliance instances, but to evaluate how well the standard was working and to identify any difficulties the 

program areas had in implementing the standard. 
                    

                 

                 

     

  

       

3.2.1 Evaluation Methodology 

The Quality Program Staff followed this procedure to evaluate the implementation of the standard and plans to continue using 

this procedure in future evaluations. 

1)  Review the standard, itemize its requirements, and create a checklist based on the requirements.  

2)  Create a questionnaire to interview the stakeholders involved in generating and reviewing the data products. 

3)  Select data products to review against the checklist. 

4)  Review the data products against the checklist. 

5)  Interview managers, authors, and reviewers of the selected data products to understand the process used to 

generate the data products. 

6)  Review the completed checklists and interviews for patterns and draft a report, including recommendations. 

7)  Send the draft to all interviewed stakeholders for their input and concurrence.  Incorporate comments or indicate 

issues where the stakeholders did not concur, and deliver the final report to the M&S Council. 

Because this was the first evaluation of a standard that the Quality Program Staff conducted, they were uncertain how the program 

areas would react to the evaluation.  The Quality Program Staff made sure to explain the purpose of the evaluation to the 

interviewees, to minimize apprehension that the evaluation was “checking up” on them.  In general, interviewees seemed to be 

receptive of the evaluation and welcomed the chance to voice their concerns regarding the standard. 

3.2.2 Findings and Recommendations 

The evaluation revealed two global findings (1 and 2) related to the implementation of the quality standards and two specific 

findings (3 and 4) which were directly related to the scope and content of the Minimal Information Standard (Bushery and 

McGovern, 2005).  Recommendations made by the Quality Program Staff are listed below after each finding. 

1)  Generally, the statistical methodology areas were aware of the standard, but awareness of the standard needs 

improvement in the subject matter areas.  Assign someone in each affected division to serve  as a liaison 

regarding the quality standards and communicate with division staff to improve awareness. 

2)  Responsibility for assuring the compliance with the standard is not always clearly assigned and documented 

procedures do not exist for implementing or enforcing the standard.  Each directorate should assign 

responsibility for assuring compliance.  They should also develop written procedures to guide staff in 

generating, reviewing and publishing data products, with responsibilities for all activities and intermediate 

products clearly defined. 

3)  The scope of the standard needs to be expanded.  Currently the standard covers survey and census data 

products, but does not address data products generated from administrative records, or products generated using 

demographic projections or complex modeling methods.  A team of experts and stakeholders on the issues 

associated with administrative records, demographic projections, and modeling should identify the appropriate 

minimal information requirements, so they can be included in the standard. 

4)  The language and terminology of the standard is not always clear and easily understandable.  Revise the 

standard to provide more complete definitions of the unclear terms, provide more specification of vague 

requirements, and provide examples to aid understanding. 

3.2.3 Implementing the Recommendations 

The M&S Council is responsible for providing “guidance to program areas in the development and usage of the standards.”  To 

address findings 1 and 2 in the evaluation of the “Minimal Information” standard, the Quality Program Staff, with the approval 

of the M&S Council, formed a cross­directorate team in June 2005.  This team will define the responsibilities of all organizational 

units, and prepare a general procedure to guide employees in developing, reviewing, and publishing data products in a way that 
   

       

       

            

   

                

   

     

                     

          

ensures compliance with the quality standards.  They also have the task of reviewing four Census Bureau standards that address 

disseminating data products, and identifying the responsibilities of the various organizational units, with respect to compliance 

with these standards. 

C Describing the Sources and Accuracy of Tabulations and Estimates 

C Dissemination of Census and Survey Data Products 

C Minimal Information to Accompany Any Report of Survey or Census Data 

C Review and Approval of Census Bureau Documents and Presentations 

Until more progress has been made to address the issues identified in the benchmarking study (discussed below), no actions will 

be taken to implement the recommendations (findings 3 and 4) to revise the standard. 

4 Benchmarking the Census Bureau Quality Standards 

Recently the Census Bureau has begun to place more emphasis on developing quality standards for its data products.  The OMB 

Section 515 Information Quality Guidelines (OMB, 2002) are one reason for this increased emphasis.  Another reason is the 

realization that such standards can promote consistent decisions in issues regarding data quality. 

For example, in 2001, because of poor data quality, the Census Bureau changed its plan to release separate estimates on people 

counted at “emergency and transitional shelters.”  Low response to a key question and respondent error in the answers obtained 

caused the data quality problems (Government Accountability Office (a), 2003).  Also in 2001, the Census Bureau released 

Hispanic subgroup estimates with counts substantially lower than expected for some subgroups.  The Government Accountability 

Office (GAO) concluded that the question design resulted in respondent confusion and misleading Hispanic subgroup data (GAO 

(b), 2003). 

Although both the “shelter” estimates and the Hispanic subgroup estimates suffered from similar problems of nonsampling error, 

the Census Bureau made different decisions about whether to publish the data.  In both situations, the Census Bureau’s decisions 

resulted in a dissatisfied data user community.  The GAO stated that a key factor in the inconsistent decisions appeared to be “a 

lack of adequate guidelines governing decisions on quality considerations that should be addressed before making data publicly 

available” (GAO (b), 2003).  They also indicated that “the lack of data quality guidelines resulted in similar difficulties when the 

Bureau initially decided not to release data on the homeless and others without conventional housing” (GAO (b), 2003). 

In March 2003, in response to these two GAO reports, the Census Bureau chartered a team to develop “data release standards.” 

In November 2004, the GAO published another report, titled “Census Bureau Needs to Accelerate Efforts to Develop and 

Implement Data Quality Review Standards.”  In that report, the GAO cited quality standards, such as those of Statistics Canada, 

the National Center for Education Statistics (NCES), and Eurostat, as a means to help ensure consistent decisions about data 

quality (GAO, 2004). 

In response to this third GAO report, the M&S Council instructed the Quality Program Staff to review the complete set of the 

National Center for Education Statistics’ (NCES) Statistical Standards and Statistics Canada Quality Guidelines, benchmark the 

complete set of Census Bureau Quality Standards against them, report and make recommendations.  The Quality Program Staff 

included the draft OMB Standards for Statistical Surveys (OMB, 2005) in this benchmarking exercise, since the Census Bureau 

must comply with these standards. 

4.1 Methodology 

To conduct the benchmarking analysis of the Census Bureau’s Quality Standards against the OMB standards and those of our 

counterparts, the Quality Program Staff reviewed and summarized each set of standards.  The OMB and the NCES standards also 

included recommended guidelines.  The Quality Program Staff did not include these guidelines in the summaries, focusing only 

on comparing the required standards among the different organizations. 

Using the summary documents, the Quality Program Staff compared the Census Bureau’s Quality Standards with the standards 

of each organization separately.  In some cases, a one­to­one correspondence existed between the Census Bureau standards and 

those of the OMB and our counterparts.  In other cases, one­to­many or many­to­one matches existed.  The Quality Program 

Staff’s analysis identified where the Census Bureau standards corresponded completely, partially, or not at all with the standards 

of the OMB, the NCES and Statistics Canada. 
     

         

  

   

    

       

4.2 Benchmark Findings and Recommendations 

The benchmarking study helped identify deficiencies in the Census Bureau’s Quality Standards and provided insight into how our 

counterparts have organized and developed their quality standards.  The study generated two major findings. 

C The OMB, the NCES, and Statistics Canada all use a formal definition of data quality as a foundation for their 

standards.  A formal definition of data quality provides a focus for developing quality standards.  The Census 

Bureau’s  program areas could use this definition to guide decisions affecting the quality of their processes and 

products.  

The Census Bureau plans to develop and adopt a formal definition of data quality. 

C Gaps exist in the Census Bureau’s Quality Standards, relative to the draft OMB Standards for Statistical 

Surveys. 

The Census Bureau plans to develop comprehensive data quality standards within the OMB framework.  In 

developing these quality standards, the Census Bureau will consider the elements of the quality standards and 

guidelines used by our counterpart statistical organizations. 

5 Conclusions and Next Steps 

With the formation of the Quality Program Staff in 2004 to assist the M&S Council in promoting and supporting the Quality 

Program, the Census Bureau has been able to focus its quality­related efforts more effectively, including the development, 

implementation, and evaluation of its quality standards.  The Quality Program Staff supports the M&S Council and teams 

chartered to draft quality standards.  The results of the benchmarking study have given the M&S Council a focus for directing the 

future development of quality standards at the Census Bureau.  The Quality Program Staff uses their expertise in quality assurance, 

quality control, and quality auditing to assist the program areas in understanding, implementing, and verifying compliance with 

the quality standards. 

As of mid­August 2005, the benchmarking report is in the hands of the M&S Council and the Program Associate Directors.  The 

M&S Council has developed an action plan for closing the gaps in the Census Bureau standards.  The plan is currently undergoing 

the Commerce Department’s clearance process for transmittal to the GAO. 

The M&S Council will charter teams to develop the standards needed to close the gaps between the Census Bureau’s Quality 

Standards and the OMB standards.  Because of their importance, the standards on data dissemination will receive priority.  An 

important part of an effective system of quality standards is developing procedures to implement the standards.  The program areas 

need to develop and document these procedures to institutionalize the standards and ensure compliance.  The team mentioned in 

Section 3.2.3 of the paper likely will serve as a pilot for developing the procedures to implement the quality standards. 

While much work remains, the Census Bureau has made a solid start in developing and instituting comprehensive quality standards 

that will promote its interests in product and process quality. 
               

     

       

6 References 

Bushery, John M. and McGovern, P., “Evaluation of Census Bureau Standard: Minimal Information to Accompany Any Report 

of Survey or Census Data,”internal Census Bureau report, January 25, 2005. 

Government Accountability Office (a), “Methods for Collecting and Reporting Data on the Homeless and Others without 

Conventional Housing Need Refinement,”GAO­03­227, January 2003. 

Government Accountability Office (b), “Methods for Collecting and Reporting Hispanic Subgroup Data Need 

Refinement,”GAO­03­228, January 2003. 

Government Accountability Office, “Census Bureau Needs to Accelerate Efforts to Develop and Implement Data Quality Review 

Standards,”GAO­05­86, November 2004. 

Landman, Cheryl, Donnalley, G., and Clark, C., “Quality Program at the U.S. Census Bureau,” Proceedings of the International 

Conference on Quality in Official Statistics, Stockholm, 2001 (Q2001). 

National Center for Education Statistics, Statistical Standards, U.S. Department of Education, Institute of Education Sciences, 

NCES 2003­601 (http://nces.ed.gov/statprog/2002/stdtoc.asp). 

Office of Management and Budget, “Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of 

Information Disseminated by Federal Agencies,” Federal Register, Vol. 67, No. 36, February 22, 2002. 

Office of Management and Budget Standards for Statistical Surveys, Federal Register notice: 

http://www.whitehouse.gov/omb/inforeg/proposed_standards_for_statistical_surveys.pdf, July 14, 2005. 

Statistics Canada, Statistics Canada Quality Guidelines, Fourth Edition, October 2004 

(http://www.statcan.ca/english/freepub/12­539­XIE/12­539­XIE03001.pdf).