2010 Census: Counting Americans Overseas as Part of the Decennial Census Would Not Be Cost-Effective
Released: 2004-09-14T00:00:00

Letter

Background

According to the Bureau, no accurate estimate exists of the total number of Americans living abroad. The Constitution and federal law give the Bureau discretion to decide whether to count American citizens living abroad. In prior censuses, the Bureau has generally included “federally affiliated” groups—members of the military and federal employees and their dependents—but has excluded private citizens residing abroad from all but the 1960 and 1970 Censuses. The 2000 Census, using administrative records, found 576,367 federally affiliated Americans residing overseas, including 226,363 military personnel, 30,576 civilian employees, and 319,428 dependents of both groups.

In response to congressional direction and the concerns of various private organizations, the Census Bureau launched a research and evaluation program to assess the practicality of counting both private and federally affiliated U.S. citizens residing abroad. The key part of this effort, the enumeration, took place from February 2004 through July 2, 2004, in France, Kuwait, and Mexico. To promote the overseas census test the Bureau relied on third parties—American organizations and businesses in the three countries—to communicate to their members and/or customers that an overseas enumeration of Americans was taking place and to make available to U.S. citizens either the paper questionnaire or Web site address.

Currently, the Bureau is processing and analyzing the overseas questionnaire data and plans to complete an evaluation of the test results in early 2005. The Bureau estimates that it will have spent approximately $7.8 million in fiscal years 2003 through 2005 to plan, conduct, and evaluate the 2004 test. The Bureau has requested additional funds for fiscal year 2005 to plan for further testing scheduled for 2006. The Bureau also plans to include overseas testing in the 2008 dress rehearsal if it were to receive the necessary funding.

In May 2004 we reported on the design of the overseas enumeration test and concluded that because of various methodological limitations, the test results will only partially answer the Bureau’s key research objectives concerning feasibility (as measured by such indicators as participation and number of valid returns), data quality, and cost. Further, we noted that the Bureau should not decide on its own whether or not to enumerate Americans overseas, and in fact would need congressional guidance on how to proceed. As shown in figure 1, the key decisions facing Congress in this regard include, in addition to the threshold question of whether American residing overseas should be counted, how the data should be used and whether to enumerate this population group as part of the decennial census.

We also recommended that if further testing were to occur, that the Bureau resolve the shortcomings of the design of the 2004 test and better address the objectives of an overseas enumeration.

Scope and Methodology

As agreed with your offices, our objectives for this report were to assess (1) whether the Bureau implemented the test consistent with its design, and (2) the initial lessons learned from the test results and their implications for future overseas enumerations. To assess the first objective, we interviewed Bureau officials and compared the Bureau's test plans with what was actually done at the three test sites. We visited Paris, France, and Guadalajara, Mexico, to obtain the views of 12 private, civic, and other organizations on the implementation of the overseas census test, and/or to confirm at 36 organizations the availability of census material. In addition, to a more limited extent, we interviewed officials from third party organizations in Kuwait via the telephone or e-mail. We judgmentally selected these organizations because they had agreed to display census promotional materials and, in some cases, had also agreed to do one or more of the following activities: make available paper copies of the census questionnaire, publish information in a newsletter, post a link to a Web site, send outreach e-mail to members, and/or create speaking opportunities to discuss the census. The results of these visits are not necessarily representative of the larger universe of third-party organizations.

To assess the implications of the test results on future overseas enumerations and the 2010 census, we obtained from Bureau officials preliminary results of the overseas census by test site and response mode as well as cost data. We also interviewed officials from the Bureau and third-party organizations to determine what lessons were learned from the test and the implications on future overseas enumeration efforts.

2004 Overseas Test Was Generally Implemented as Designed

The Bureau’s design for the 2004 overseas enumeration test was generally implemented as planned and completed on schedule. The Bureau’s design had four key components: the mode of response, the questionnaire designed specifically for Americans living overseas, three test sites, and an outreach and promotion program designed to communicate and educate Americans abroad that a test census was being conducted. Table 1 describes each of these components in greater detail.

However, while the test was generally implemented as designed, our earlier report pointed out several methodological limitations with the design, such as not being able to calculate response rates because the universe of Americans is unknown or not being able to measure the quality of data because of the impracticality of developing an address list. As we discuss later in this report, it is these methodological limitations that impede the Bureau’s ability to implement a successful overseas enumeration.

Overseas Census Test Results Were Disappointing and Costly

Although the 2004 overseas enumeration test ended in early July 2004 and the Bureau has just begun evaluating the results, the response levels were poor, and very expensive to obtain on a per unit basis. The response level to the overseas enumeration suggests that the current approach to counting overseas Americans—a voluntary survey that relies heavily on marketing to get people to participate—by itself cannot secure a successful head count. Further, obtaining the additional resources needed to produce substantially better results may not be feasible, and still not yield data that are comparable in quality to the stateside enumeration.

Response Levels Were Disappointing and Costly to Obtain

The 5,390 responses the Bureau received for this test were far below what the Bureau planned for when printing up materials and census forms. While the Bureau ordered 520,000 paper forms for three test sites, only 1,783 census forms were completed and returned. Of these, 35 were Spanish language forms that were made available in Mexico. The remaining 3,607 responses were completed via the Internet. Table 2 below shows the number of census questionnaires that the Bureau printed for each country and the number of responses they actually received in both the paper format and via the Internet.

Because of the low response levels, in early April 2004, the Bureau reversed its decision to not use paid advertising and in May 2004 initiated a paid advertising campaign in France and Mexico. This included print and Internet ads in France and print and radio ads in Mexico. See figure 2 for examples of the ads used in the paid advertising campaign.

A Bureau official told us the ad campaign for the 2004 overseas test cost about $206,000. This official said there were surplus funds available in the project budget to use for this purpose due to lower than expected processing and postage costs for the overseas test. While the Bureau saw some increase in the number of responses after the paid advertising campaign began, this official said the increase was slight.

Return on Investment for Overseas Enumeration is Low

Not only were response levels low, they were extremely expensive to obtain on a unit basis—roughly $1,450 for each returned questionnaire, based on the $7.8 million the Bureau spent preparing for, implementing, and evaluating the 2004 overseas test. In contrast, the unit cost of the 2000 Census was about $56 per household. Although the two surveys are not directly comparable because the 2000 Census costs covered operations not used in the overseas test, the 2000 Census was still the most expensive census in our nation’s history.

Securing a Higher Return Rate Would Be an Enormous Challenge

The main reason for the high unit cost is the low return rate. However, significantly boosting participation levels may not be feasible. The Bureau’s experience in the 2000 Census highlights the level of effort that was needed to raise public awareness about the census and get people to complete their forms. For the 2000 decennial, the Bureau spent $374 million on a comprehensive marketing, communications, and partnership effort. The campaign consisted of a five-part strategy conducted in three waves beginning in the fall of 1999 and continuing past Census Day (April 1, 2000). The effort helped secure a 72-percent return rate. Specific elements included television, radio, and other mass media advertising; promotions and special events; and a census-in-schools program. Thus, over a period of several months, the American public was on the receiving end of a steady drumbeat of advertising aimed at publicizing the census and motivating them to respond.

The Bureau also filled 594 full-time partnership specialist positions. These individuals were responsible for mobilizing support for the census on a grassroots basis by working with governmental entities, private companies, and religious and social service groups, and other organizations.

Replicating this level of effort on a worldwide basis would be impractical, and still would not produce a complete count. Indeed, even after the Bureau’s aggressive marketing effort in 2000, it still had to follow-up with about 42 million households that did not return their census forms. Moreover, because there are no reliable figures on the number of Americans residing overseas, the Bureau would not have a good measure of the number of people that did not participate, or the overall quality of the data.

Ensuring a Smooth Overseas Enumeration Could Stretch the Bureau’s Resources

The Bureau’s experience in conducting the 2004 overseas test underscored the difficulties of administering a complex operation from thousands of miles away. Not surprisingly, as with any operation this complex, various challenges and unforeseen problems arose. While the Bureau was able to resolve them, its ability to do so should there be a full overseas enumeration as part of the 2010 Census, would be highly questionable as far more resources would be required. This was particularly evident in at least two areas: grappling with country-specific issues and overseeing the contractor responsible for raising public awareness of the census at the three test sites.

Country-specific Issues Created Implementation Problems

The Bureau encountered a variety of implementation problems at each of the test sites. In some cases the problems were known in advance, in others, glitches developed at the last minute. Although such difficulties are to be expected given the magnitude of the Bureau’s task, a key lesson learned from the test is that there would be no economy of scale in ramping up to a full enumeration of Americans abroad. In fact, just the opposite would be true. Because of the inevitability of country-specific problems, rather than conducting a single overseas count based on a standard set of rules and procedures (as is the case with the stateside census), the Bureau might end up administering what amounts to dozens of separate censuses—one for each of the countries it enumerates—each with its own set of procedures adapted to each country’s unique requirements. The time and resources required to do this would likely be overwhelming and detract from the Bureau’s stateside efforts.

For example, during the overseas test, the Bureau found that French privacy laws restrict the collection of personal data such as race and ethnic information. However, these data are collected as part of the decennial census because they are key to implementing a number of civil rights laws such as the Voting Rights Act.

Addressing France’s privacy laws took a considerable amount of negotiation between the two countries, and was ultimately resolved after a formal agreement was developed. The Bureau issued and posted on its Web site an advisory that informed Americans living in France that it was not mandatory to respond to the questionnaire, the only recipient of the collected data is the Census Bureau, the data will be kept for one year, and the respondent has a right to access and correct the data collected. The Bureau was able to collect race and ethnic data—generally a prohibited practice without the respondents’ permission—only after it received special approval from a French government agency. Initially, however, it looked as if the Bureau might have to redesign the census form if it wanted to use it in France.

In Kuwait, delivery of the census materials was delayed by several weeks at the beginning of the test because they were accidentally addressed to the wrong contractor. Ultimately, the U.S. Embassy stepped-in to accept the boxes so that the enumeration could proceed. In Mexico, there was some initial confusion on the part of Mexican postal workers as to whether they could accept the postage-paid envelopes that the Bureau had provided to return the paper questionnaires for processing in the United States.

Because of the small number of countries involved in the test, the Bureau was able to address the various problems it encountered. Still, the Bureau’s experience indicates that it will be exceedingly difficult to identify and resolve in advance all the various laws, rules, societal factors, and a host of other potential glitches that could affect a full overseas enumeration.

On-site Supervision of Contractor Was Problematic

As noted previously, the Bureau hired a public relations firm to develop a communications strategy to inform and motivate respondents living in the test countries to complete the census. The firm’s responsibilities included identifying private companies, religious institutions, service organizations, and other entities that have contact with Americans abroad and could thus help publicize the census test. Specific activities the organizations could perform included displaying promotional materials and paper versions of the census questionnaire, publishing information in a newsletter, and posting information on their Web sites. Although the public relations firm appeared to go to great lengths to enlist the participation of these various entities—soliciting the support of hundreds of organizations in the three countries—the test revealed the difficulties of adequately overseeing a contractor operating in multiple sites overseas.

For example, the public relations firm’s tracking system indicated that around 440 entities had agreed to perform one or more types of promotional activities. However, our on-site inspections of several of these organizations in Paris, France, and Guadalajara, Mexico, that had agreed to display the census materials and/or distribute the questionnaires, uncovered several glitches. Of the 36 organizations we visited that were supposed to be displaying promotional literature, we found the information was only available at 15. In those cases, as shown in figure 3, the materials were generally displayed in prominent locations, typically on a table with posters on a nearby wall.

Five of these 15 organizations were also distributing the census questionnaire, but the forms were not readily accessible.

However, at 21 sites we visited, we found various discrepancies between what the public relations firm indicated had occurred, and what actually took place. For example, while the firm’s tracking system indicated that questionnaires would be available at a restaurant and an English-language bookstore in Guadalajara, none were available. In fact, the owner of the bookstore told us that no one from the Census Bureau or the public relations firm had contacted her about displaying materials for the overseas test.

At the University of Guadalajara, although the tracking system indicated that an official had been contacted about, and agreed to help support the census test, that official told us no one had contacted him. As a result, when boxes of census materials were delivered to his school without any explanatory information, he did not know what to do with them, and had to telephone the U.S. Consulate in Guadalajara to figure out what they were for.

Likewise, in Paris, we went to several locations where the tracking system indicated that census information would be available. None was. In fact, at some of these sites, not only was there no information about the census, but there was no indication that the organization we were looking for was at the address we had from the database.

The results of the overseas test point to the difficulties of overseeing the contractor’s performance. As census materials were made available at scores of locations across the three test countries, it would have been impractical for the Bureau to inspect each site. The difficulty of supervising contractors—and any field operation for that matter—would only be magnified in a global enumeration.

The Design of the Overseas Census Lacks the Building Blocks of a Successful Census

The Bureau’s experience in counting the nation’s population for the 2000 and earlier censuses sheds light on some of the specific operations and other elements that together form the building blocks of a successful head count (see fig. 4).

While performing these activities does not necessarily guarantee a cost- effective headcount, not performing them makes a quality count far less promising and puts the entire enterprise at risk. The current approach to counting overseas Americans lacks these building blocks, as most are infeasible to perform on an overseas population. Each is discussed in greater detail below.

Mandatory participation: Under federal law, all persons residing in the United States regardless of citizenship status are required to respond to the decennial census. By contrast, the overseas enumeration test was conducted as a voluntary survey where participation was optional. The Bureau has found that response rates to mandatory surveys are higher than the response rates to voluntary surveys. This in turn yields more complete data and helps hold down costs.

Early agreement on design: Both Congress and the Bureau need to agree on the fundamental design of an overseas census. Concurrence on the design helps ensure adequate planning, testing and funding levels. Conversely, the lack of an agreed-upon design raises the risk that basic design elements might change in the years ahead, while the opportunities to test those changes and integrate them with other operations will diminish. Under the Bureau’s current plans, after the 2006 test, the Bureau would have just one more opportunity to test its prototype for an overseas enumeration—a dress rehearsal in 2008. Any design changes after 2008 would not be tested in a real-world environment.

The design of the census is driven in large part by the purposes for which the data will be used. Currently, no decisions have been made on whether the overseas data will be used for purposes of congressional apportionment, redistricting, allocating federal funds, or other applications. Some applications, such as apportionment, would require precise population counts and a very rigorous design that parallels the stateside count. Other applications, however, could get by with less precision and thus, a less stringent approach.

As we noted previously, Congress will need to decide whether or not to count overseas Americans, and how the results should be used. The basis for these determinations needs to be sound research on the cost, quality of data, and logistical feasibility of the range of options for counting this population group. Possibilities include counting Americans via a separate survey, administrative records such as passport and voter registration forms, and/or records maintained by other countries such as published census records and work permits.

The Bureau’s initial research has shown that each of these options has coverage, accuracy, and accessibility issues, and some might introduce systemic biases into the data. Far more extensive research would be needed to determine the feasibility of these or other potential approaches. Once Congress knows the tradeoffs of these various alternatives, it will be better positioned to provide the Bureau with the guidance it needs to go beyond research and conduct field tests of specific approaches. The Bureau can conduct the research, or it can contract it out. Indeed, the National Research Council of the National Academy of Sciences has conducted a number of studies on the decennial census, including a review of the 2000 Census and an examination of reengineering the 2010 Census.

A complete and accurate address list: The cornerstone of a successful census is a quality address list. For the stateside census, the Bureau goes to great lengths to develop what is essentially an inventory of all known living quarters in the United States, including sending census workers to canvass every street in the nation to verify addresses. The Bureau uses this information to deliver questionnaires, follow up with nonrespondents, determine vacancies, and identify households the Bureau may have missed or counted more than once. Because it would be impractical to develop an accurate parallel address list for overseas Americans, these operations would be impossible and the quality of the data would suffer as a result.

Ability to detect invalid returns: Ensuring the integrity of the census data requires the Bureau to have a mechanism to screen out invalid responses. Stateside, the Bureau does this by associating an identification number on the questionnaire to a specific address in the Bureau’s address list, as well as by field verification. However, the Bureau’s current approach to counting overseas Americans is unable to determine whether or not a respondent does in fact reside abroad. So long as a respondent provides certain pieces of information on the census questionnaire, it will be eligible for further processing. The Bureau is unable to confirm the point of origin for questionnaires completed on the Internet, and postmarks on a paper questionnaire only tell the location from which a form was mailed, not the place of residence of the respondent. The Bureau has acknowledged that ensuring such validity might be all but impossible for any reasonable level of effort and funding.

Ability to follow up with non-respondents: Because participation in the decennial census is mandatory, the Bureau sends enumerators to those households that do not return their questionnaires. In cases where household members cannot be contacted or refuse to answer all or part of a census questionnaire, enumerators are to obtain data from neighbors, a building manager, or other nonhousehold member presumed to know about its residents. The Bureau also employs statistical techniques to impute data when it lacks complete information on a household. Thus, by the end of each decennial census, the Bureau has a fairly exhaustive count of everyone in the nation. As noted above, because the Bureau lacks an address list of overseas Americans, it is unable to follow up with nonrespondents or impute information on missing households. As a result, the Bureau will never be able to obtain a complete count of overseas Americans.

Cost model for estimating needed resources: The Bureau uses a cost model and other baseline data to help it estimate the resources it needs to conduct the stateside census. Key assumptions such as response levels and workload are developed based on the Bureau’s experience in counting people decade after decade. However, the Bureau has only a handful of data points with which to gauge the resources necessary for an overseas census, and the tests it plans on conducting will only be of limited value in modeling the costs of conducting a worldwide enumeration in 2010.

The lack of baseline data could cause the Bureau to over- or underestimate the staffing, budget and other requirements of an overseas count. For example, this was evident during the 2004 overseas test when the Bureau estimated it would need around 100,000 Spanish- language questionnaires for the Mexican test site. As only 35 Spanish- language questionnaires were returned, it is now clear that the Bureau could have gotten by with printing far fewer questionnaires for Mexico. However, the dilemma for the Bureau is that its experience in the 2004 overseas test cannot be used to project the number of Spanish-language questionnaires it would need for Mexico or other Spanish-speaking countries in 2010. Similar problems would apply to efforts to enumerate other countries.

Targeted and aggressive marketing campaign: The key to raising public awareness of the census is an intensive outreach and promotion campaign. As noted previously, the Bureau’s marketing efforts for the 2000 Census were far-reaching, and consisted of more than 250 ads in 17 languages that were part of an effort to reach every household, including those in historically undercounted populations. Replicating this level of effort on a global scale would be both difficult and expensive, and the Bureau has no plans to do so.

Field infrastructure to execute census and deal with problems: The Bureau had a vast network of 12 regional offices and 511 local census offices to implement various operations for the 2000 Census. This decentralized structure enabled the Bureau to carry out a number of activities to help ensure a more complete and accurate count, as well as deal with problems when they arose. Moreover, local census offices are an important source of intelligence on the various enumeration obstacles the Bureau faces on the ground. For example, during the 2000 Census, the Bureau called on them to identify hard-to-count population groups and other challenges, and to develop action plans to address them. The absence of a field infrastructure for an overseas census means that the Bureau would have to rely heavily on contractors to conduct the enumeration, and manage the entire enterprise from its headquarters in Suitland, Maryland.

Ability to measure coverage and accuracy: Since 1980, the Bureau has measured the quality of the decennial census using statistical methods to estimate the magnitude of any errors. The Bureau reports these estimates by specific ethnic, racial, and other groups. For methodological reasons, similar estimates cannot be generated for an overseas census. As a result, the quality of the overseas count, and thus whether the numbers should be used for specific purposes, could not be accurately determined.

Conclusions

The 2004 test of the feasibility of an overseas enumeration was an extremely valuable exercise in that it highlighted the numerous obstacles to a cost-effective count of Americans abroad as an integral part of the decennial census. Although more comprehensive results will not be available until the Bureau completes its evaluation of the test early next year, a key lesson learned is already clear: The current approach to counting this population group—a voluntary survey that largely relies on marketing to ensure a complete count—would be costly and yield poor results. The tools and resources the Bureau has on hand to enumerate overseas Americans are insufficient for overcoming the inherent obstacles to a complete count, and it is unlikely that any refinements to this basic design would produce substantially better results, and certainly not on a par suitable for purposes of congressional apportionment.

What’s more, the Bureau already faces the difficult task of carrying out a cost-effective stateside enumeration in 2010. Securing a successful count of Americans in Vienna, Virginia, is challenging enough; a complete count of Americans in Vienna, Austria—and in scores of other countries around the globe—would only add to the difficulties facing the Bureau as it looks toward the next national head count. As a result, we believe that any further tests or planning activities related to counting Americans overseas as part of the decennial census would be an imprudent use of the Bureau’s limited resources.

That said, to the extent that Congress desires better data on the number and characteristics of Americans abroad for various policy-making and other nonapportionment purposes that require less precision, such information does not necessarily need to be collected as part of the decennial census, and could, in fact, be acquired through a separate survey or other means. To help inform congressional decision making on this issue, including decisions on whether Americans should be counted and how the data should be used, it will be important for Congress to have the results of the Bureau’s evaluation of the 2004 overseas census test. Equally important would be information on the cost, quality of data, and logistical feasibility of counting Americans abroad using alternatives to the decennial census. Once Congress knows the tradeoffs of these various alternatives, it would be better positioned to provide the Bureau with the direction it needs so that the Bureau could then develop and test an approach that meets congressional requirements at reasonable resource levels.

Matters for Congressional Consideration

Given the obstacles to a cost-effective count of overseas Americans as part of the decennial census and, more specifically, obtaining data that is of sufficient quality to be used for congressional apportionment, Congress may wish to consider eliminating funding for any additional research, planning, and development activities related to counting this population as part of the decennial headcount, including funding for tests planned in 2006 and 2008. However, funding for the evaluation of the 2004 test should continue as planned to help inform congressional decision making.

Should Congress still desire better data on the number of overseas Americans, in lieu of the method tested in 2004, Congress might wish to consider authorizing and funding research on the feasibility of counting Americans abroad using alternatives to the decennial census.

Recommendations for Executive Action

To facilitate congressional decision making, we recommend that the Secretary of Commerce ensure that the Bureau completes its evaluation of the 2004 overseas census test as planned. Further, to the extent that additional research is authorized and funded, the Bureau, in consultation with Congress, should explore the feasibility of counting overseas Americans using alternatives to the decennial census. Potential options include conducting a separate survey, examining how the design and archiving of various government agency administrative records might need to be refined to facilitate a more accurate count of overseas Americans, and exchanging data with other countries’ statistical agencies and censuses, subject to applicable confidentiality and other provisions.

Consideration should also be given to whether the Bureau should conduct this research on its own or whether it should be contracted out to the National Academy of Sciences.

Agency Comments and Our Evaluation

The Secretary of Commerce forwarded written comments from the U.S. Census Bureau on a draft of this report on August 5, 2004, which are reprinted in the appendix. The Bureau agreed with our conclusions and recommendations. Furthermore, the Bureau noted, “should Congress request and fund” further research on counting overseas Americans, it would be equipped to do that research itself.

As agreed with your offices, unless you release its contents earlier, we plan no further distribution of this report until 30 days from its issue date. At that time we will send copies to other interested congressional committees, the Secretary of Commerce, and the Director of the U.S. Census Bureau. Copies will be made available to others upon request. This report will also be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me on (202) 512-6806 or by e-mail at daltonp@gao.gov or Robert Goldenkoff, Assistant Director, at (202) 512-2757 or goldenkoffr@gao.gov. Key contributors to this report were Ellen Grady, Lisa Pearson, and Timothy Wexler.

Comments from the Department of Commerce

GAO’s Mission

The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to www.gao.gov and select “Subscribe to Updates.”

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Congressional Relations

Public Affairs