2000 Census: Coverage Measurement Programs' Results, Costs, and Lessons Learned
Released: 2003-01-29T00:00:00

Letter

Background

The Bureau puts forth tremendous effort to conduct a complete and accurate count of the nation’s population. However, some degree of error in the form of persons missed or counted more than once is inevitable because of limitations in census-taking methods. Because census results are used, among other purposes, to apportion Congress, redraw congressional districts, and allocate federal aid to state and local governments, the size and demographic composition of these coverage errors have become increasingly sensitive since the Bureau was first able to generate detailed data on them during the 1980 Census. However, the Bureau has never used the results of its coverage measurements to correct estimated coverage errors.

The Bureau first attempted to measure the accuracy of the census in the 1940s when it compared the census numbers to birth and death certificates and other administrative data using a procedure called demographic analysis. Modern coverage measurement began with the 1980 Census when the Bureau compared census figures to the results of an independent sample survey of the population. Using statistical methods, the Bureau generated detailed measures of the differences among undercounts of particular ethnic, racial, and other groups. In the months that followed, many lawsuits were filed, most contending that the results of the 1980 coverage measurement should have been used to adjust the census. However, the Bureau designed the evaluation to measure errors, not to correct the census results, and the Director of the Census Bureau decided against adopting the adjusted numbers, as they were deemed flawed due to missing and inaccurate data.

The quality of the coverage measurement data improved for the 1990 Census, and the Bureau recommended statistically adjusting the results. However, the Secretary of Commerce determined that the evidence to support an adjustment was inconclusive and decided not to adjust the 1990 Census. The adjustment decision was complicated by the fact that the 1990 Census figures had already been released when the coverage measurement results became available in the spring of 1991. The Secretary of Commerce was concerned that two sets of numbers—the actual census results and the adjusted figures—could create confusion and might allow political considerations to play a part in choosing between sets of numbers when the outcome of the choices, such as congressional apportionment, could be known in advance of a decision.

Scope and Methodology

To determine the objectives of 2000 Census I.C.M./A.C.E. programs and their results, we reviewed Bureau and other documents that included Federal Register notices; Census Operational Plans; reports to Congress; internal memorandums; research and feasibility studies; and reports of the Executive Steering Committee for Accuracy and Coverage Policy (ESCAP) I and II, which assessed the results of the A.C.E. program and recommended how they should be used.

To determine costs for consultants and technical studies for 2000 Census I.C.M./A.C.E. programs, we focused on object class code 25 from the financial management reports to obtain contract data. With Bureau assistance, we identified I.C.M./A.C.E. project accounts and analyzed amounts by fiscal year using the financial management reports generated by the Department of Commerce’s Administrative Management System (CAMS). We reviewed and analyzed obligated and expended data for all coverage measurement programs that existed during the 2000 Census for fiscal years 1991 to 2003. We did not audit financial data provided by the Bureau.

To determine ways to track future costs, we reviewed current Bureau financial management reports and considered established standards of accounting, auditing, and internal controls.

In addition, we met with key Bureau officials to discuss the results of our analysis and obtain their observations and perspectives. The limitations we encountered in the scope of our work on this assignment are as follows.

We were unable to determine the complete contractual and technical studies costs of the I.C.M./A.C.E. programs because the Bureau considered any I.C.M./A.C.E.-related costs from fiscal years 1991 through 1995 as part of its general research and development programs and thus did not separately track these costs. Although some costs were tracked in fiscal year 1996, the Bureau still considered these costs as research and development and did not include these costs as I.C.M./A.C.E. program costs.

We were unable to identify I.C.M./A.C.E. portions of costs from projects that covered the entire census, such as the 2000 Census Evaluation program.

We did not evaluate the propriety of contracts for I.C.M./A.C.E. programs.

Our work was performed in Washington, D.C., and at U.S. Census Bureau headquarters in Suitland, Maryland, from June 2002 through October 2002 in accordance with generally accepted government auditing standards. On January 7, 2003, the Secretary of Commerce provided written comments on a draft of this report. We address these comments in the “Agency Comments and Our Evaluation” section, and have reprinted them in appendix I.

Coverage Measurement Programs for the 2000 Census Did Not Meet Bureau Objectives

In planning the 2000 Census, the Bureau developed a new coverage measurement program, I.C.M., that was designed to address the major shortcomings of the 1990 coverage measurement program. However, as shown in table 1, much like similar programs in earlier censuses, the Bureau did not use I.C.M. and its successor program, A.C.E., to adjust the census because of legal challenges, technical obstacles, and the inability to resolve uncertainties in the data in time to meet the deadlines for releasing the data.

The Bureau Canceled I.C.M. in Response to a Supreme Court Ruling

In designing I.C.M., the Bureau’s goal was to produce a single, consolidated count or “one-number” census and thus avoid the controversy of having two sets of census results as occurred during the 1990 Census. Thus, as shown in table 1, the objectives of I.C.M. were to (1) measure census coverage, (2) generate, using statistical sampling and estimation methods, the detailed data required for apportionment, congressional redistricting, and federal program purposes, and (3) produce a one-number census.

The Bureau’s plans for I.C.M. emerged in response to the unsatisfactory results of the 1990 Census. Although the 1990 headcount was, at that time, the most costly in U.S. history, it produced data that were less accurate than those from the 1980 Census. The disappointing outcome was due in large part to the Bureau’s efforts to count housing units that did not mail back their census questionnaires. The operation, known as nonresponse follow-up, where enumerators visited and collected information from each nonresponding housing unit, proved to be costly and error-prone when a higher-than-expected workload and a shortage of enumerators caused the operation to fall behind schedule. The final stages of nonresponse follow- up were particularly problematic. Indeed, while enumerators finished 90 percent of the follow-up workload within 8 weeks (2 weeks behind schedule), it took another 6 weeks to resolve the remaining 10 percent. Moreover, in trying to complete the last portion of nonresponse follow-up cases, the Bureau accepted less complete responses and information from nonhousehold members such as neighbors, which may have reduced the quality of the data.

In the years following the 1990 Census, Congress, the Bureau, several organizations, and GAO, concluded that fundamental design changes were needed to reduce census costs and improve the quality of the data. In response, the Bureau reengineered a number of operations for the 2000 Census.

For example, to save time and reduce its nonresponse follow-up workload, the Bureau planned to enumerate a sample of the last remaining portion of nonresponse follow-up cases instead of visiting every nonresponding household as it had done in previous censuses. To adjust for enumeration errors, the Bureau developed I.C.M., which was intended to reconcile the original census figures with data obtained from a separate, independent count of a sample of 750,000 housing units using a statistical process called Dual System Estimation. The Bureau believed that this approach offered the best combination of reduced costs, improved accuracy expected at various geographic levels, and operational feasibility.

However, concerned about the legality of the Bureau’s planned use of sampling and estimation, members of Congress challenged the Bureau’s use of I.C.M. in court. In January 1999, the Supreme Court ruled that the Census Act prohibited the use of statistical sampling to generate population data for reapportioning the House of Representatives.

A.C.E. Did Not Meet Bureau Objectives

Following the Supreme Court ruling, the Bureau planned to produce apportionment numbers using traditional census-taking methods, and provide statistically adjusted numbers for nonapportionment uses of the data such as congressional redistricting and allocating federal funds. The Bureau initiated the A.C.E. program, which was designed to take a national sample of approximately 300,000 housing units to evaluate coverage errors among different population groups and statistically correct for them. Thus, as shown in table 1, the Bureau’s objectives for A.C.E. were to (1) measure how many people were missed in the census and how many were erroneously included and (2) produce the detailed data required in time for redistricting and federal program purposes.

However, while the Bureau generally conducted A.C.E. in accordance with its plans, the Bureau later determined that the A.C.E. results did not provide a reliable measure of census accuracy and could not be used to adjust the nonapportionment census data.

The first decision against A.C.E. occurred in March 2001, when the Acting Director of the Census Bureau recommended to the Secretary of Commerce that the unadjusted census data be used for redistricting purposes. He cited as a primary reason an apparent inconsistency between the population growth over the prior decade, as implied by A.C.E. results, and demographic analysis, which estimated the population using birth, death, and other administrative records. The inconsistency raised the possibility of an unidentified error in either the A.C.E. or census numbers. He reported that the inconsistency could not be resolved prior to April 1, 2001, the legally mandated deadline for releasing redistricting data.

The second decision against A.C.E. came in October 2001 when, based on a large body of additional research, ESCAP decided against adjusting census data for allocating federal aid and other purposes, because A.C.E. failed to identify a significant number of people erroneously included in the census, and other remaining uncertainties. According to Bureau officials, it might be possible to use adjusted data to produce intercensal population estimates for federal programs that require this information; however, the Bureau would need to revise the A.C.E. results before any use of the data could be considered.

The Bureau’s Experience in Implementing Coverage Measurement Programs Highlights Important Lessons Learned

Although I.C.M. and A.C.E. did not meet their formal objectives, they did produce a body of important lessons learned. As the Bureau’s current approach for the 2010 Census includes coverage measurement to assess the accuracy of the census (but not necessarily to adjust the numbers themselves), it will be important for the Bureau to consider these lessons as its planning efforts continue. The lessons include (1) developing a coverage measurement methodology that is both technically and operationally feasible, (2) determining the level of geography at which coverage measurement is intended, (3) keeping stakeholders, particularly Congress, informed of the Bureau’s plans, and (4) adequately testing the eventual coverage measurement program. 1. A.C.E. demonstrated operational, but not technical feasibility.

According to Bureau officials, an important result of the A.C.E. program was that it demonstrated, from an operational perspective only, the feasibility of conducting a large independent field check on the quality of the census. The Bureau canvassed the entire A.C.E. sample area to develop an address list, collected census response data for persons living in the sample areas on census day, and conducted an operation to try and match A.C.E. respondents to census respondents, all independent of the regular census operations and within required time frames.

Our separate reviews of two of these operations—interviewing respondents and matching A.C.E. and census data—while raising questions about the impact on final A.C.E. results due to apparently small operational deviations, also concluded that the Bureau implemented those two operations largely as planned.

Nevertheless, while the Bureau demonstrated that it could execute A.C.E. field operations using available resources within required time frames, as the Bureau has noted, feasibility also consists of a technical component—that is, whether the A.C.E. methodology would improve the accuracy of the census. Although the Bureau clearly stated in its justification for A.C.E. that the effort would make the census more accurate, as noted earlier, because of unresolved data discrepancies, its experience in 2000 proved otherwise. Moreover, according to the Bureau, because the A.C.E. was designed to correct a census with a net coverage error similar to that observed in previous censuses, the Bureau commented that applying the methodology to the historically low levels of net error observed in the 2000 Census represented a unique and unanticipated challenge for A.C.E. Thus, it will be important for the Bureau to refine its coverage measurement methodology to ensure that it is technically feasible. 2. The level of geography at which the Bureau can successfully measure coverage is unclear. Since the October 2001 decision to not rely on adjusted census data for nonapportionment and nonredistricting purposes, Bureau officials have told us that they now doubt whether census data can reliably be improved down to the level of geography for which A.C.E. was intended to improve the accuracy—the census tract level (neighborhoods that typically contain around 1,700 housing units and 4,000 people). The Bureau’s current position differs from that taken in 2000, when it reported to Congress that it expected accuracy at the tract level to be improved, on average, by A.C.E. statistically adjusting numbers at an even lower level of geography—the census block level. Uncertainty in the level of geography at which accuracy is to be measured or improved can affect the overall design of coverage measurement, as well as its technical feasibility. Therefore, it will be important for the Bureau to determine the level of geography at which it intends to measure accuracy as it decides the role and design of future coverage measurement programs. 3. Keeping stakeholders informed is essential. Throughout the 1990s, Congress and other stakeholders, including GAO, expressed concerns about the Bureau’s planned use of sampling and statistical estimation procedures to adjust the census. A key cause of this skepticism was the Bureau’s failure to provide sufficiently detailed data on the effects that I.C.M. would have at different levels of geographic detail. Information was also lacking on the various design alternatives being considered, their likely implications, and the basis for certain decisions. As a result, it was difficult for Congress and other stakeholders to support the Bureau’s coverage measurement initiatives.

For example, on September 24, 1996, the House Committee on Government Reform and Oversight issued a report that criticized the Bureau’s initiatives for sampling and statistical estimation. Among other things, the Committee found that the Bureau had not clarified issues of accuracy, particularly for small geographic areas, raised by the sampling initiative. Congress’s perspective on the process was later reflected in its enactment of legislation in 1997 that included provisions requiring the Department of Commerce to provide Congress with comprehensive information on its planned use of statistical estimation within 30 days.4. Adequate testing of coverage measurement methodologies is critical.

Although the Bureau conducted a dress rehearsal for the census in three locations across the country that was intended to demonstrate the overall design of the census, the 1998 operation did not reveal the problems that the Bureau encountered in dealing with the discrepancies between the 2000 A.C.E. results and its benchmarks. According to Bureau officials, this was partly because the sites were not representative of the nation at large. Additionally, as a result of a compromise between Congress and the administration to simultaneously prepare for a nonsampling census, the I.C.M. was tested at only two of the three dress rehearsal sites—an urban area and an Indian reservation—but was not tested in a rural location as was originally planned. An earlier test in 1995 was also not comprehensive in that it did not test a sampling operation designed to help determine whether nonresponse follow-up of the magnitude projected by the Bureau’s current plan could be completed in time for the I.C.M to be done on schedule.

I.C.M./A.C.E. Contractor Costs Are Not Complete

From fiscal year 1996 through fiscal year 2001, the Bureau obligated about $207 million for I.C.M./A.C.E. activities. As shown in table 2, of that $207 million, we identified about $22.3 million (11 percent) in obligated amounts for contracts involving more than 170 vendors. These contracts were primarily for technical advisory and assistant services, computer systems support, and training. ($2) (10) (130) (130)

Although the Bureau tracked some costs of contracts for the I.C.M./A.C.E. programs, we found that the $22.3 million did not represent the complete contractor costs of the programs because of the following three factors.

First, the Bureau only tracked the contractor costs associated with conducting the I.C.M./A.C.E. programs, which covers the period from fiscal year 1997 through 2003. Although life cycle costs for the 2000 Census cover a 13-year period from fiscal years 1991 through 2003, senior Bureau officials said that the I.C.M./A.C.E. program was not viable for implementation until fiscal year 1997. Therefore, the Bureau considered contractor costs from earlier years as part of its general research and development programs, and the Bureau did not assign unique project codes to identify I.C.M./A.C.E. programs and related costs in its financial management system.

Second, although $182,000 of fiscal year 1996 obligated contractor costs were identifiable in the Bureau’s financial management system as an I.C.M. special test, the Bureau did not consider these costs as part of the I.C.M./A.C.E. programs. Instead, these costs were considered general research and development. However, because the Bureau separately identified these costs as I.C.M. program contractor costs, we have included the $182,000 as part of the I.C.M./A.C.E. program contractor costs in this report.

Finally, we were unable to identify the I.C.M./A.C.E. portions of costs that were part of other programs. For example, in late fiscal year 2000 and after, the Bureau did not separate A.C.E. evaluations from its other 2000 Census evaluations in its financial management systems. Bureau officials stated that the contracts for evaluations included overall 2000 Census and A.C.E. evaluations, and did not have a separate code identifying A.C.E. costs.

Tracking Future Coverage Measurement Costs

During the 2000 Census, the Bureau, its auditors, and GAO, found extensive weaknesses in the Bureau’s financial management system, the components of which include hardware, software, and associated personnel. The weaknesses included difficulties in providing reliable and timely financial information to manage current government operations and preparing financial statements and other reports. Together, they affected the completeness, accuracy, and timeliness of data needed for informed management decisions and effective oversight. In light of these weaknesses, the Bureau’s ability to track future costs of coverage measurement activities will largely depend on three factors.

First, a sound financial management system is critical. As discussed in our December 2001 report, the Bureau’s core financial management system, CAMS, had persistent internal control weaknesses in fiscal year 2000. In its latest financial report, the Bureau indicated that these weaknesses have continued through fiscal year 2001. The Bureau expects to issue its fiscal year 2002 financial report shortly.

Second, it would be important to set up project codes to capture coverage measurement activities as early in the planning process as possible. The Bureau did not set up a specific project code to identify I.C.M. program costs until 1996 because, according to Bureau officials, the I.C.M. program was not viable until 1997 and all costs up to this point were considered general research.

Finally, it would be important for Bureau personnel to correctly charge the project codes established for the coverage measurement program activities. During the 2000 Census, for example, while the Bureau established a project code and a budget for the remote Alaska enumeration, the project costs were erroneously charged to and commingled with a project code for enumerating special populations. As a result, the actual costs for remote Alaska enumeration were reported by the Bureau’s financial management system as zero and are unknown, while enumerating special population costs are overstated.

Conclusions

The Bureau’s 2000 Census coverage measurement programs did not achieve their primary objectives of measuring the accuracy of the census and adjusting the results because of legal challenges, technical hurdles, and questionable data. However, beyond these formal objectives, there emerged several important lessons learned that Bureau managers should consider because current plans for the 2010 Census include coverage measurement. At the same time, it will also be important for the Bureau to be capable of fully tracking the money it spends on coverage measurement and other census activities so that Congress and other stakeholders can hold the Bureau accountable for achieving intended results.

Although the Bureau has never used the results of its coverage measurement programs to adjust census numbers, we believe that an evaluation of the accuracy and completeness of the census is critical given the many uses of census data, the importance of identifying the magnitude and characteristics of any under- and overcounts, and the cost of the census overall. Less clear is whether the results of the coverage measurement should be used to adjust the census. Any Bureau decisions on this matter should involve close consultation with Congress and other stakeholders, and be based on detailed data and a convincing demonstration of the feasibility of the Bureau's proposed approach. Whatever the decision, it is imperative that it be made soon so that the Bureau can design appropriate procedures and concentrate on the business of counting the nation’s population. The longer the 2010 planning process proceeds without a firm decision on the role of coverage measurement, the greater the risk of wasted resources and disappointing results.

Recommendations for Executive Action

To help ensure that any future coverage measurement efforts achieve their intended objectives and costs can be properly tracked, we recommend that the Secretary of Commerce direct the Bureau to in conjunction with Congress and other stakeholders, come to a decision soon on whether and how coverage measurement will be used in the 2010 Census; consider incorporating lessons learned from its coverage measurement experience during the 2000 Census, such as (1) demonstrating both the operational and technical feasibility of its coverage measurement methods, (2) determining the level of geography at which coverage can be reliably measured, (3) keeping Congress and other stakeholders informed of its plans, and (4) adequately testing coverage measurement prior to full implementation; and ensure that the Bureau’s financial management systems can capture and report program activities early in the decennial process and ensure that project costs are monitored for accuracy and completeness.

Agency Comments and Our Evaluation

The Secretary of Commerce forwarded written comments from the Census Bureau on a draft of this report, which are reprinted in appendix I. The Bureau agreed with our recommendations highlighting the steps that should be followed in the development of a coverage measurement methodology for the 2010 Census and acknowledged their importance. However, the Bureau maintained that it followed most of these steps for the 2000 Census including (1) keeping stakeholders, particularly Congress, informed of the Bureau’s plans, (2) determining the level of geography at which coverage measurement is intended, and (3) adequately testing coverage measurement methodologies. The Bureau also maintained that throughout the 1990s, it had an open and transparent process for implementing the coverage measurement program, including the levels of geography to which its results would be applied.

We disagree. As we stated in our report, the Bureau’s failure to provide important information was a key cause of congressional skepticism over the Bureau’s coverage measurement plans. In fact, Congress was so concerned about the lack of comprehensive information on the Bureau’s proposed approach that in July 1997, it passed a law that included provisions requiring the Department of Commerce to provide detailed data on the Bureau’s planned use of statistical estimation within 30 days. We revised the report to include this, and provide other examples to further support our position that the Bureau’s I.C.M. and A.C.E. planning and development processes were less than fully open and transparent.

The Bureau also commented that each major component of the I.C.M./A.C.E. program underwent “rigorous” testing in the middle of the decade as well as during the dress rehearsal for the 2000 Census held in 1998. We believe this overstates what actually occurred. As we noted in the report, the dress rehearsal failed to detect the problems that A.C.E. encountered during the 2000 Census because the sites were not representative of the nation. Additionally, because of an agreement between Congress and the administration to simultaneously prepare for a census that did not include sampling, the I.C.M. was only tested at two of the three dress rehearsal sites—an urban area and an Indian reservation— but was not tested in a rural location as was originally planned. We made this and other revisions to strengthen our point.

Because the A.C.E. was designed to correct a census with a net coverage error similar to that observed in previous censuses, the Bureau commented that applying the methodology to the historically low levels of net error observed in the 2000 Census represented a unique and unexpected challenge for A.C.E. We revised the report to reflect this additional context.

The Bureau took exception to the way we presented our conclusions concerning its ability to properly classify certain costs associated with the development of the Bureau’s coverage measurement programs. The Bureau noted that it decided not to separately track coverage measurement development costs in 1994, because there was no internal or external request for a separate cost accounting of the program.

Our report does not make interpretive conclusions or qualitative judgments about which coverage measurement program costs the Bureau decided to track. Instead, the report (1) points out that we could not identify all of the contractor costs associated with the I.C.M./A.C.E. programs because of the three factors described in the report, and (2) underscores the importance of a sound financial management system for tracking, planning, and development costs for the 2010 Census.

We are sending copies of this report to other interested congressional committees, the Secretary of Commerce, and the Director of the U.S. Census Bureau. Copies will be made available to others upon request. This report will also be available at no charge on GAO’s home page at http://www.gao.gov.

Please contact Patricia A. Dalton on (202) 512-6806 or by E-mail at daltonp@gao.gov if you have any questions. Other key contributors to this report were Robert Goldenkoff, Roger Stoltz, Carolyn Samuels, Cindy Brown-Barnes, Ty Mitchell, and Linda Brigham.

Related GAO Products

2000 Census: Complete Costs of Coverage Evaluation Programs Are Not Available. GAO-03-41. Washington, D.C.: October 31, 2002. 2000 Census: Lessons Learned for Planning a More Cost-Effective 2010 Census. GAO-03-40. Washington, D.C.: October 31, 2002. 2000 Census: Refinements to Full Count Review Program Could Improve Future Data Quality. GAO-02-562. Washington, D.C.: July 3, 2002. 2000 Census: Coverage Evaluation Matching Implemented as Planned, but Census Bureau Should Evaluate Lessons Learned. GAO-02-297. Washington, D.C.: March 14, 2002. 2000 Census: Best Practices and Lessons Learned for More Cost-Effective Nonresponse Follow-up. GAO-02-196. Washington, D.C.: February 11, 2002. 2000 Census: Coverage Evaluation Interviewing Overcame Challenges, but Further Research Needed. GAO-02-26. Washington, D.C.: December 31, 2001. 2000 Census: Analysis of Fiscal Year 2000 Budget and Internal Control Weaknesses at the U.S. Census Bureau. GAO-02-30. Washington, D.C.: December 28, 2001. 2000 Census: Significant Increase in Cost Per Housing Unit Compared to 1990 Census. GAO-02-31. Washington, D.C.: December 11, 2001. 2000 Census: Better Productivity Data Needed for Future Planning and Budgeting. GAO-02-4. Washington, D.C.: October 4, 2001. 2000 Census: Review of Partnership Program Highlights Best Practices for Future Operations. GAO-01-579. Washington, D.C.: August 20, 2001.

Decennial Censuses: Historical Data on Enumerator Productivity Are Limited. GAO-01-208R. Washington, D.C.: January 5, 2001. 2000 Census: Information on Short- and Long-Form Response Rates. GAO/GGD-00-127R. Washington, D.C.: June 7, 2000.

GAO’s Mission

The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily E-mail alert for newly released products” under the GAO Reports heading.

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Public Affairs