2010 Census: Overseas Enumeration Test Raises Need for Clear Policy Direction
Published: 2004-05-21T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In the 1990 and 2000 Censuses, U.S. military and federal civilian employees overseas were included in the numbers used for apportioning Congress. Currently, the U.S. Census Bureau (Bureau) is assessing the practicality of counting all Americans abroad by holding a test census in France, Kuwait, and Mexico. GAO was asked to (1) assess the soundness of the test design, and (2) examine what past court decisions have held about Americans' rights and obligations abroad.
section_title: What GAO Found
paragraphs: Although the overseas enumeration test was designed to help determine the practicality of counting all Americans abroad, because of various methodological limitations, the test results will only partially answer the Bureau's key questions concerning feasibility, data quality, and cost. For example, one research questions asks, "How good is the quality of the data?" However, the Bureau will only measure item nonresponse, which indicates whether a person completed a particular question. As a measure of quality, it is far from complete. Similarly, although a key research objective was to determine the cost of counting Americans overseas, the Bureau's data will not inform the cost of conducting future tests or an overseas enumeration in 2010. Overall, the Bureau overstated the test's ability to answer its key research objectives. Overseas Americans have various rights and obligations to federal programs and activities. For example, Americans abroad are generally taxed on their worldwide income and can vote in federal elections, but are generally not entitled to Medicare benefits. There is nothing in the Constitution, federal law, or court decisions that would either require the Bureau to count overseas Americans, or not count this population group. As a result, Congress would need to enact legislation if it wanted to require the Bureau to include overseas Americans in the 2010 Census. Counting Americans abroad as part of the census would add new risks to an enterprise that already faces an array of challenges. Therefore, it will be important for Congress to decide whether overseas Americans should be counted as part of the census or counted as part of a separate survey or whether there are so many obstacles to a successful count regardless of the approach that the Bureau should shelve any plans for further research and testing. To the extent a second test is required, the Bureau will need to take steps to develop a more rigorous design.

section_title: Letter
paragraphs: One of the thornier issues facing the U.S. Census Bureau (Bureau) as it  prepares for the 2010 Census is whether to count American citizens  residing abroad, and if so, how to use the results.  Under federal law, the  Bureau has discretion over whether to count this population group.  Thus,  in prior censuses, the Bureau has generally included “federally affiliated”  groups—members of the military and federal employees and their  dependents—but has excluded private citizens residing abroad from all but  the 1960 and 1970 Censuses.  Moreover, when these overseas groups were  included in the census, their population totals were generally not included  with the counts used for apportioning Congress.
According to the Bureau, no accurate estimate exists of the total number of  Americans living abroad.  Further, while it is unclear how counting these  individuals would affect congressional apportionment and other purposes  for which census data are used, in some instances it could be significant.   For example, in January 2001, Utah sued the U.S. Department of Commerce  claiming that it lost a congressional seat because the 2000 Census excluded  the state’s 11,000 Mormon missionaries and other private citizens living  abroad.  Although the suit was unsuccessful, according to a report by the  Congressional Research Service, Utah would have gained a congressional  seat had an additional 855 people been counted in the census.
In response to congressional direction and the concerns of various  stakeholder groups representing overseas Americans, the Census Bureau  launched a research and evaluation program to assess the practicality of  counting both private and federally affiliated U.S. citizens residing abroad,  as well as their dependents.  The first step of this effort, a test enumeration  of Americans residing in France, Kuwait, and Mexico, is currently  underway. A sound feasibility test is essential because the Bureau has  already identified a number of operational, conceptual, and policy issues  that would make an accurate count of overseas Americans even more  difficult than the stateside enumeration, an endeavor that is a daunting  challenge in its own right.  As a second step, the results of the 2004 test will  be used to provide information and recommendations for another overseas  enumeration test planned for 2006.
To facilitate Congress’s oversight of the test and provide information for  future decision making on this issue, you asked us to review the overseas  test design.  As agreed with your offices, we (1) assessed the soundness of  the Bureau’s test design and its suitability for addressing the Bureau’s  specific research questions, and (2) examined what past court decisions  have held about Americans’ rights and obligations abroad that could help  inform whether and how they should be included in the census.
To meet these objectives, we interviewed knowledgeable Bureau officials;  reviewed pertinent documents such as test plans; and examined relevant  statutes, court decisions, and legal analyses.  We also systematically rated  the soundness of the Bureau’s test design using a checklist of over 30  design elements that, based on our review of program evaluation literature,  are important for a sound study plan.  We conducted our work from June  2003 through February 2004, in accordance with generally accepted  government auditing standards.
This report is the second in our ongoing series on the planning and  development of the 2010 Census.  In a study issued earlier this year, we described the various challenges facing the 2010 Census, and the need for  the Bureau to address cost and design issues.
subsections: section_title: Background
paragraphs: The Census Bureau has discretion under the Constitution and federal  statutes to decide whether to count Americans residing overseas.  The  federal decennial census is conducted pursuant to the requirement  imposed by Article I, Section 2, Clause 3 of the Constitution, and Section 2  of the Fourteenth Amendment, that Congress enumerate “the whole  number of persons in each State” as the basis for apportionment of seats in  the United States House of Representatives.  Under the Constitution, the  census is to be conducted every 10 years “in such Manner as   shall by Law direct.”  Congress has exercised its authority under the  Constitution by passing the Census Act, which assigns to the Secretary of  Commerce the responsibility of “tak a decennial census of population  as of the first day of April” of each census year.  The Secretary does so with  the assistance of the Census Bureau and its Director.
The statutes governing the earliest censuses provided that enumerators  should record all persons reported to them within their respective districts  as having a usual place of abode there or as usually residing within that  district, even though such persons might be “occasionally absent at the time of enumeration.”  Statutes governing later censuses, including the  current provisions in Title 13 of the United States Code, contain no similar  provision, or any provision specifically governing the enumeration of  inhabitants of the United States who are outside of its borders on the  enumeration date.
To determine who should be included in the census, the Bureau applies its  “usual residence rule,” which has been defined as the place where a person  lives and sleeps most of the time.  People who are temporarily absent from  that place are still counted as residing there.  One’s usual residence is not  necessarily the same as one’s voting residence or legal residence.   Noncitizens living in the United States are counted in the census,  regardless of their immigration status.
Historically, the census has focused primarily on the domestic population  and typically has not included any procedures designed to enumerate  Americans residing outside of the United States. The first attempts to count  Americans residing overseas were in the 1830 and 1840 censuses, which  included procedures for counting the “crews of naval vessels at sea.”  The  naval personnel included in those censuses, however, were not allocated to  any individual state, and thus were not included in the apportionment  population.
As shown in table 1, various overseas population groups were included in  the census at different times.  For example, while federally affiliated  personnel were typically included in the enumerations that took place from  1900 through 2000, only the 1970, 1990, and 2000 censuses used the  numbers for purposes of apportioning Congress.  At the same time, private  citizens living abroad were included only in the 1960 and 1970 censuses,  but not for purposes of apportionment.
In response to congressional direction and the concerns of various  business, political, and other groups that represent overseas Americans,  the Census Bureau embarked on a research and evaluation program aimed  at determining the feasibility, quality, and cost of counting both federally  affiliated and private citizens living abroad.  The test enumeration began  February 2004 and is to run through July 2004 at three sites:  France,  Kuwait, and Mexico.
The Bureau selected these countries based on several criteria including  their geographic diversity, the fact that large numbers of U.S. citizens reside  there, and because of the existence of administrative records that can be  used to compare to the test census counts for evaluation purposes.  The  Bureau estimated the implementation costs for the 2004 test at  approximately $2.5 million in fiscal year 2004.  Further, the Bureau  estimates that by the end of fiscal year 2004, it will have spent an additional  $3.5 million for planning and preparation during fiscal years 2003 and 2004.
Americans can participate in the test census by completing a short-form  paper questionnaire that is available at embassies, consulates, and other  organizations that serve overseas Americans, or by completing the form on  the Internet.  The Bureau hired a public relations firm to develop a  communications strategy to inform and motivate respondents living in the  selected countries to answer the census.  Responses from the paper and  the Internet returns will be captured in order to analyze, among other  things, the demographic characteristics of respondents and patterns of  item nonresponse.  The Bureau plans to conduct a 2006 overseas test if  Congress appropriates requested funds in fiscal years 2005, 2006, and 2007.   If Congress then indicates its desire that the Census Bureau conduct a  general overseas enumeration in 2010, the Bureau will seek a  supplementary appropriation in calendar year 2007 for that purpose and to  conduct a 2008 overseas dress rehearsal beginning in 2007.
A sound test is essential in order for the Bureau, Congress, and other  stakeholders to resolve the numerous logistical, conceptual, policy, and  other questions that surround the counting of overseas Americans.  They  include:  Who should be counted?  U.S. citizens only?  Foreign-born spouses?
Children born overseas?  Dual citizens?  American citizens who have no  intention of ever returning to the United States?  Naturalized citizens?
How should overseas Americans be assigned to individual states?  For  certain purposes, such as apportioning Congress, the Bureau would  need to assign overseas Americans to a particular state.  Should one’s  state be determined by the state claimed for income tax purposes?   Where one is registered to vote?  Last state of residence before going  overseas?  These and other options all have limitations that would need  to be addressed.
How should the population data be used?  To apportion Congress?  To  redistrict Congress?  To allocate federal funds?  To provide a count of  overseas Americans only for general informational purposes?  The  answers to these questions have significant implications for the level of  precision needed for the data and ultimately, the enumeration  methodology.
How can the Bureau verify U.S. citizenship?  Administrative records  such as passports and Social Security data have limitations.  For  example, Americans can reside in Mexico and Canada without a  passport and many Americans overseas do not have Social Security  numbers, especially dependents.
How can the Bureau ensure a complete count without a master address  list? The foundation of the stateside decennial census is a master  address list.  Because the list is essentially the universe of all known  living quarters in the United States, the Bureau uses it to deliver  questionnaires, follow up with nonrespondents, determine vacancies,  and determine individuals the Bureau may have missed or counted more  than once.  The Bureau lacks a complete and accurate address list of  overseas Americans.  Consequently, these operations would be  impossible and the quality of the data would suffer as a result.
Can administrative records be used to help locate and count overseas  Americans?  Administrative records such as passport and visa files,  voter registration forms, as well as records held by private companies  and organizations have the potential to help the Bureau enumerate  Americans abroad.  However, the accuracy of these records, the  Bureau’s ability to access them, confidentiality issues, and the  possibility of duplication all remain open questions.
Do certain countries have requirements that could restrict the Bureau’s  ability to conduct a count?  According to the Bureau, in planning the  overseas test, the Bureau was informed that French privacy laws  prohibit asking about race and ethnicity, two questions that are included  on the U.S. census questionnaire.  Although the Bureau worked with  French officials to address this problem, the extent to which the Bureau  will encounter restrictions in other countries, or whether other  countries will cooperate with the Bureau at all, is unknown.
section_title: Scope and Methodology
paragraphs: As agreed with your offices, our objectives for this report were to   (1) assess the soundness of the Bureau’s test design and its suitability for  addressing the Bureau’s specific research questions, and (2) examine what  past court decisions have held about Americans’ rights and obligations  abroad that could help inform whether and how they should be included in  the census.
To assess the soundness of the Bureau’s 2004 overseas enumeration test  design, we interviewed knowledgeable Bureau officials and reviewed  existing documents that described the Bureau’s test objectives, research  questions, and test plans.  We then systematically rated the Bureau’s  approach using a checklist of over 30 design elements that, based on our  review of program evaluation literature, are relevant to a sound study plan.   For example, we reviewed the Bureau’s approach to determine, among  other things,  (1) how clearly the Bureau presented the research objectives,  (2) whether the research questions matched the research objectives,   (3) whether potential biases were recognized and addressed, and (4) the  appropriateness of the data collection strategy for reaching the intended  sample population.  See appendix II for a complete list of the 30 design  elements.
We supplemented our ratings on the suitability of the test by gathering  additional information through telephone and in-person interviews with  representatives of several stakeholder organizations that represent various  groups of Americans residing abroad.  The organizations included  Democrats Abroad, Republicans Abroad, Association of Americans  Resident Overseas, and the American Business Council of the Gulf  Countries.  In addition, we interviewed representatives of the Mexican  American Legal Defense and Educational Fund, National Coalition for an  Accurate Count of Asians and Pacific Islanders, and California Rural Legal  Assistance, Inc., to get their perspectives on the Bureau’s plans for  counting American citizens living in Mexico, particularly migrant and  seasonal farm workers, a group that the Bureau had trouble counting  during the 2000 Census. These three organizations, while not actively  involved in the planning of the overseas enumeration test, are members of  the Secretary of Commerce’s Decennial Census Advisory Committee, a  panel that advises the Bureau on various census-related issues.
To examine what past court decisions have held about Americans’ rights  and obligations living abroad, including their right to be counted in the  census, we reviewed a judgmental selection of five federal laws and/or  programs that cover large numbers of Americans stateside, in order to  determine how those laws and programs treat U.S. citizens should they live  overseas.  We examined federal election law, federal income tax law, and  federal laws relating to Social Security benefits, Supplemental Social  Security Income, and Medicare.  For each of these laws and programs, we  reviewed relevant statutes, court decisions, and legal analyses.
We requested comments on a draft of this report from the Secretary of  Commerce, which were sent to us April 13, 2004 (see app. I).  We address  them in the Agency Comments and Our Evaluation section of this report.
section_title: Design Limitations Could Undermine the Usefulness of the Overseas Test
paragraphs: According to the Bureau, its objectives for the 2004 overseas test are “to  determine the feasibility, quality, and cost of collecting data from U.S.  citizens living overseas.”  To meet those objectives, the Bureau developed  eight research questions designed to gather data on such salient factors as  participation levels, data quality, and the relative response from the two  enumeration modes (Internet and paper questionnaire; see appendix III for  a sample of the paper questionnaire).  To assess the overseas test the  Bureau is planning to complete a series of evaluations due in early 2005.   The test objectives and related research questions are appropriate as  written, but, as shown in table 2, because of various methodological  limitations, the data that will result from the test will not fully answer key  questions concerning feasibility, data quality, and cost.  In short, the Bureau  overstated the research test’s ability to answer its key research objectives  and, as a result, congressional decision making on this issue will be that  much more difficult.
subsections: section_title: Research Question #1: Participation Data Could be Unreliable
paragraphs: For the decennial census, the Bureau uses participation data as a key  indicator of public cooperation with the census.  The Bureau measures  participation levels by what it calls the “return rate,” which it calculates as  a percentage of all forms in the mailback universe (excluding vacant and  nonexistent housing units) from which it receives a questionnaire.
Stateside, the Bureau is able to perform this calculation because, as noted  above, it has a master address list of all known housing units in the United  States, an inventory that takes the Bureau several years and considerable  resources to compile.
However, no such address list exists for overseas Americans.   Consequently, participation rates for overseas Americans cannot be  calculated and the Bureau will only be able to tally the number of  responses it receives (both overall for each site and within several  demographic categories) and compare the results to counts obtained from  administrative records.  The sources for the records include a combination  of tax, Medicare, and State Department data, as well as foreign census data  if available.  For a variety of reasons—some of which the Bureau has  already acknowledged—the information generated from this exercise may  not be relevant.
First, the administrative records were developed for different purposes,  and as a result, are not well suited as a base for comparing against overseas  counts; thus, their relevance is uncertain.  As the Bureau has already  reported, each of the records it plans to use to compare to the census  counts has coverage limitations.  Further, as each of these records is  associated with particular demographic groups, they could introduce  systemic biases (we discuss potential problems with administrative  records in greater detail below, under Research Question #5).
Second, the census counts could be problematic because it is unclear who  should participate in the overseas census, which in turn could confuse  potential respondents.  For the stateside enumeration, to determine where  an individual should be counted, the Bureau uses the concept of “usual  residence,” which it defines as “the place where a person lives and sleeps  most of the time.”  The Bureau has developed guidelines, which it prints on  the stateside census questionnaire, to help people figure out who should  and should not be included.  However, the Bureau has not developed  similar guidance for the overseas test.  According to the Bureau, this was  intentional; because this was an initial feasibility test, the Bureau did not  want to restrict response, but rather to encourage the widest possible  participation.
Further, the guidance the Bureau has developed, which is available on its  Web site and promotional literature, is vague and could confuse potential  respondents.  For example, the guidelines inform potential respondents  that, “All U.S. citizens living in France, Mexico and Kuwait, regardless of  shared citizenship, can and should participate in the test.  U.S. citizens on  vacation or on short business trips should not.”  Unclear is what constitutes  a short or long business trip.  Is it 3 weeks or 3 months?  Does it matter  whether one stays in a hotel or an apartment?  Also, should naturalized U.S.  citizens, some of whom may not return to the United States, participate?   What about children born in the United States to noncitizens, but who only  lived in the United States a short time?  Should students spending a  semester abroad but who maintain a permanent residence stateside be  included?  Without clear residence rules and appropriate guidance  indicating who should be counted, it is quite possible that some people  might inappropriately opt in or out of the census, which would reduce the  quality of the data.  Bureau officials have told us that they are working to  develop residence rules that it will apply if there is a second overseas  enumeration test in 2006.
Participation data might also be problematic because the Bureau’s  enumeration methods strategy might not be as effective with certain  groups compared to others.  To the extent this occurs, it could introduce a  systemic undercount.  This is particularly true for dual nationals who, for  cultural reasons, may not think of themselves as American citizens.
For example, according to representatives of two advocacy groups we  contacted, Mexican dual nationals include migrant farm workers, a group  that often consists of poor, less-educated people living in rural areas.  They  noted that this population group has low literacy levels and thus might not  understand the questionnaire, and is not likely to have Internet access.  At  the same time, they are not likely to pick up a copy of the questionnaire at  an embassy.  Further, the barriers that make it difficult to count migrant  farm workers in the United States, such as a distrust of government and the  fact that they may speak indigenous languages, also make it difficult to  count this group in Mexico.
section_title: Research Question #2: Information on Data Quality Will be Limited
paragraphs: The Bureau plans on measuring the quality of the data collected in the  overseas test by tabulating item nonresponse, which refers to whether a  respondent completed a particular question.  The Bureau is to calculate  this information by enumeration mode, test site, and various demographic  categories.  The Bureau also plans to compare this measure of nonresponse  for key variables to those obtained in an earlier, stateside test held in 2003  by tabulating the rate respondents did not complete a particular question.
According to the Bureau, patterns of item nonresponse are critical for  improving question design, training, and procedures.  However, as the  Bureau acknowledges in its study plan for evaluating the quality of the  overseas enumeration data, item nonresponse by itself does not address  the quality of the data.  Thus, at the end of the test, the Bureau will have, at  best, only limited information on the quality of the overseas data.
By comparison, the Bureau’s guidelines for measuring data quality in other  surveys they conduct use measures such as coverage, unit response rates,  imputation rates, and data collection errors.  Because the Bureau lacks  information on the universe of overseas Americans it will be unable to  calculate these indicators.  Therefore, it is misleading for the Bureau to  state in its research objectives that it will determine the quality of the  overseas data, when in fact it will deliver something far more limited.   Comprehensive measures of data quality are critical because they could  help Congress decide whether the data are sufficiently reliable to use for  specific purposes.  If the numbers were to be used to obtain a simple count  of Americans abroad, absolute precision is not as critical.  However, for  other uses of the data, particularly congressional apportionment and  redistricting, the quality of the data would need to be far higher.
section_title: Research Question #3: The Bureau Will be Unable to Geocode Certain U.S. Addresses
paragraphs: Counting people in their correct locations is essential for congressional  apportionment, redistricting, and certain other uses of census data.  With  respect to Americans abroad, if the data are to be used for apportionment,  the Bureau would need to assign respondents to a specific state.  For  purposes of redistricting and allocating federal funds, the Bureau would  need to assign overseas respondents to specific neighborhoods and street  addresses in the United States—a far more challenging task.  Geocoding is  the  process of linking an address in the Bureau’s Master Address File  (MAF) to a geographic location in the Bureau’s geographic database,  known as the Topologically Integrated Geographic Encoding and  Referencing (TIGER) mapping system.  To obtain this information, the  overseas enumeration form asks respondents to indicate their U.S. state of  last residence, and their last street address within that state.
Although the Bureau is assessing the level of geography to which it can  geocode the overseas population, an important limitation is that the Bureau  will not be able to make this assessment for people who live in certain  noncity-style U.S. addresses; that is, U.S. addresses without a housing  number and/or street name.  Specifically, people whose U.S. addresses  consist of a post office box will be excluded from the study.  For the  stateside enumeration, Bureau employees canvass the country, identify  noncity-style addresses, and mark the locations of those residences on a  map.  During the 2000 Census, around 20 percent of U.S. households had  noncity-style addresses.
Indeed, the overseas enumeration questionnaire instructs respondents not  to provide a post office box number for their last stateside address.   However, if a respondent’s address includes a post office box or rural route  number, it is unclear how they are supposed to complete this question.   Also unclear is how migrant farm workers, who may not have had an  address in the United States, would complete this question.
To the extent they leave the question blank, the Bureau would be unable to  distinguish between those people who did so because they have a noncity- style address, or left it blank for privacy or other reasons.  This could affect  the accuracy of the Bureau’s assessment.  Moreover, the construction of the  question could introduce a systemic bias because those states with large  rural areas are more likely to have noncity-style addresses.
The Bureau’s evaluation plan recognizes that respondents in the 2004  overseas enumeration test may provide noncity-style addresses which  cannot be geocoded by the TIGER system. The Bureau intends to provide  data on how many city-style and noncity-style addresses could be  geocoded.
section_title: Research Question #4: Measuring the Marketing Program’s Effectiveness Will be Difficult
paragraphs: Although the Bureau has taken a number of steps to publicize the overseas  enumeration, evaluating the effectiveness of that effort will present a  challenge. The Bureau awarded a $1.2 million contract to a public relations  firm to develop a promotion strategy for the overseas enumeration test.  As  part of that effort, the public relations firm identified a number of  stakeholder organizations that represent U.S. citizens living overseas in  each of the three test countries.  The organizations included advocacy  groups, universities, church groups, and corporations.
The Bureau anticipates that these stakeholders will help get the word out  via e-mail, newsletters, and other media that a test census of Americans  overseas is underway.  In addition, the Bureau is to provide copies of the  overseas questionnaire to stakeholders so that they can distribute them to  their members and constituents.  As noted earlier, questionnaires will also  be available on the Internet, as well as at public places that Americans may  visit, such as embassies and consulates.  The Bureau has produced posters  and pamphlets to promote the test (see fig. 1).  The Bureau also plans to  have articles about the census test placed in newspapers and magazines  and stories run on local television and radio.  Although paid advertising  was not part of its original plans, the Bureau later decided to run a limited  amount of paid advertising in Mexico and France.
Fill out a U.S. Census questionnaire, available It’s easy, important — and confidential.
The Bureau will attempt to gauge the effectiveness of the marketing  program by measuring participation (as measured by the number of  responses) and public awareness.  According to the Bureau, respondents  who submit a questionnaire via the Internet will be asked to complete a  short survey eliciting information on how they learned about the census  test and what motivated them to participate.  No similar survey is planned  for people who pick up their surveys at an embassy or other distribution  site.  Thus, the Bureau will not have a parallel set of data from a group of  respondents that might be demographically or behaviorally different from  Internet respondents.
The Bureau does, however, expect to conduct focus group interviews and  debriefings to obtain feedback from mail respondents and stakeholder  organizations.  Focus group interviews targeting nonrespondents are  planned as well.
Yet, as the Bureau acknowledges, participation, or the final count of U.S.  citizens living in the selected countries, will only be an indicator of the  number of people that heard about the test, completed the questionnaire,  and submitted it to the Census Bureau.  It will not be able to measure the  Bureau’s success in getting Americans to respond because the universe of  Americans overseas is unknown.
Public awareness will also be difficult to measure because it includes an  unknown number of people who were aware that a test census was being  conducted but chose not to respond.  Nevertheless, as noted above, the  Bureau intends to interview both respondents and nonrespondents in an  effort to determine their awareness and motivation for responding or not  responding to the census test.  To the extent the Bureau conducts these  interviews, it will be important for it to include hard-to-count groups, such  as dual nationals and migrant farm workers, that may have been outside  the reach of the Bureau’s marketing campaign.
section_title: Research Question #5: Use of Administrative Records Needs to be Thoroughly Tested
paragraphs: The Bureau plans on using administrative records such as Medicare and  passport data to provide comparison information to assess   (1) participation, (2) an invalid return detection system, and (3) the  records’ potential use for building an address list.  Specifically, the Bureau  plans to compare the number of people counted at each site to federal tax,  Medicare, U.S. Department of State, and foreign census records.  While it is  important for the Bureau to assess the utility of administrative records, it is  unclear from the Bureau’s study plans how it will make this determination  given what Bureau officials have said is a large disparity between  administrative record counts on the number of Americans living overseas  at the three test sites.
Further, because these administrative records were developed for different  purposes and as a result are not well suited as a base for comparing against  overseas counts, their relevance is uncertain.  For example, not all  American citizens who live abroad file tax returns; dependents are not  always listed on tax returns; and dependents that are included in the tax  form may not be U.S. citizens.  After living abroad 30 days, Americans are  no longer eligible for Medicare benefits; therefore, Medicare records may  not be the most useful and only apply to U.S. citizens over 65 years old.   U.S. State Department records are nonofficial and according to the Bureau  inaccurate because the Department of State does not officially track either  the number or location of U.S. citizens living in other countries.  Finally, the  type of administrative records kept by each country is unknown and earlier  Bureau research found that census data from foreign countries do not  contain the detailed information required for apportionment, redistricting,  or other census uses.
section_title: Research Question #6: Effectiveness of Enumeration Methods Will be Difficult to Assess
paragraphs: To determine the relative response from the two enumeration modes and  their relative effectiveness, the Bureau will look at such data as the timing  of returns by mode and site, and whether one mode provided more valid  returns than the other.  Respondents can either fill out a paper  questionnaire or complete the form on the Internet.  The Bureau will  deploy an invalid return detection system to determine whether a form is  valid and responses will be tallied by mode and by country.
However, the Bureau recognizes that its analysis will contain several  limitations.  Key among them is that it will not be able to determine  participation rates because the universe of Americans overseas is  unknown.  As a result, the analysis is limited to a count of the total number  of forms returned at each site.   In light of this and other limitations, it is  unclear what conclusions the Bureau will be able to draw about the  effectiveness of the two response modes.
Indeed, just because one enumeration mode results in a larger number of  returns from a particular demographic groupolder Americans, for  examplemay not necessarily have anything to do with the mode itself.  As  a result, it would be inappropriate to say that one enumeration method was  more effective than the other in counting senior citizens.  This is because  there are other factors that can influence the mode such as advertising or  accessibility to the Internet or the paper questionnaire.  Overall, while it  will be important to collect information on the returns by method of  enumeration, this information should not be construed as a measure of the  effectiveness of that mode.
section_title: Research Question #7: System to Detect Invalid Returns Has Shortcomings
paragraphs: Part of the processing of overseas returns involves validating that the  respondents are within the scope of the enumeration; that is, that they in  fact reside in one of the three overseas test areas.  Thus, the Bureau needs a  better method to detect invalid returns.
To determine whether responses are within scope, the overseas  enumeration questionnaire asks respondents to provide their Social  Security and U.S. passport numbers.  Although we agree with the  importance of determining whether respondents reside overseas, the  Bureau’s analysis will not assess this.  Rather, the Bureau defines a valid  return as one where at least one person in the household checked the U.S.  citizen box on the questionnaire, provided a valid Social Security or U.S.  passport number, and has been subject to and passes an algorithm that  analyzes data from the questionnaire.  All other returns are invalid.  Thus,  what the Bureau is really measuring is whether a questionnaire is eligible  for further processing, and not whether the respondent lives abroad.  Put  another way, anyone who completes a questionnaire with valid data,  including Social Security numbers, would be considered a valid return,  regardless of whether the individual lived in the test areas.
This is not an unlikely scenario given the way the Bureau set up its Internet  site.  Indeed, anyone—even if they live outside of the three test sites-can  be included in the overseas count, so long as they provide the required  information.  The reason they can slip through the invalid return detection  system is because the Bureau is unable to confirm the point of origin for  questionnaires completed on the Internet.  The Bureau is aware of this gap  in the invalid return detection system, but has been unable to resolve this  condition.
Another potential problem is the Bureau’s sole reliance on Social Security  numbers to validate returns for the 2004 test.  The Bureau had also wanted  to use passport numbers to validate returns.  Although the Bureau has been  negotiating with the Department of State for access to the passport  database, the Bureau does not expect this to occur in time for use in the  2004 test.  Bureau officials said they were aware that there was a strong  possibility that they would not be able to gain access to the passport file  because the Bureau had not worked with Department of State data and that  a memorandum of understanding would first need to be in place before the  data would be released.  Nevertheless, the Bureau believes that the impact  of this would be low if it could be shown that using Social Security  numbers alone to validate returns was sufficient.
A third potential problem is that some people may be reluctant to provide  their Social Security and passport numbers for privacy reasons.  Based on  the Bureau’s research, requesting this information could reduce  participation levels.
section_title: Research Question #8: Overseas Enumeration Will Not be Integrated With the Stateside Enumeration
paragraphs: The Bureau, based on its earlier research, has already identified a list of  barriers to integrating overseas with stateside data.  They include different  questionnaire content for the overseas form, detecting and eliminating  duplication within and between overseas and stateside enumerations,  timely geocoding of addresses, and limited resources.
The Bureau plans to document the lessons learned from the 2004 overseas  test and how they might apply to a more integrated test in 2006.  In  particular, the Bureau is to focus on the issues encountered or associated  with collecting, capturing, and processing overseas data.  While it will be  important for the Bureau to thoroughly document these issues and their  implications for integrating the two data sets, the Bureau does not intend to  actually integrate any data from the overseas test with data being collected  from a parallel stateside test it’s conducting at three Georgia counties and  in Queens, New York.  Without an actual integration, the Bureau may miss  problems that will not likely be detected until a next test in 2006.
section_title: Overseas Test Design Has Other Shortcomings
paragraphs: In addition to the limitations noted above, the overseas enumeration test  has other limitations that will affect the Bureau’s ability to answer its key  research objectives.
subsections: section_title: Cost Data Will Not Identify the Cost of Conducting Future Tests or an Overseas Enumeration in 2010
paragraphs: Although one of the Bureau’s objectives for the overseas enumeration test  is to determine the cost of collecting data from overseas citizens, the  Bureau’s test design lacked a specific research question aimed at obtaining  this information.  More importantly, the cost information that the Bureau  will collect will be of limited value because it will not be used to estimate  the costs of future tests, nor model the costs of conducting a broader  overseas enumeration in 2010.
The Bureau developed a cost model for the 2000 Census that provided the  agency with an automated means to estimate staffing and budget  requirements.  The Bureau used the cost model to support the budget  process, as well as to answer questions from Congress, the Office of  Management and Budget (OMB), and our office.  The cost model could also  estimate the budgetary impact of certain assumptions and alternative  census-taking scenarios.
The Bureau maintains that it would need more data than those that are now  available to develop a cost model for an overseas enumeration in 2010.   Nevertheless, while only at the beginning stages, the Bureau has some data  points to begin developing a cost model for overseas enumeration or, at a  minimum, for identifying major cost components.  For example, Bureau  officials told us that their agency will track and report the marketing,  printing, postage, data capture, and processing costs for conducting the  overseas enumeration at the three test sites.  All of the costs provided by  the Bureau are direct costs and could be specifically and uniquely  attributed to a cost model for the overseas enumeration.  Indirect costs,  however—those that are not easily attributable to the overseas  enumeration such as executive management or technical labor—would still  need to be determined and captured.
Cost will be an important factor to consider when making a decision on  whether to enumerate Americans overseas.  The cost of the 2010 Census,  now estimated at more than $11.3 billion in current dollars, is the most  expensive enumeration in the nation’s history.  Consequently, it will be  critical for the Bureau to have a mechanism for accurately and quickly  estimating overseas enumeration costs so that Congress, other oversight  groups, and the Bureau itself can have reliable information on which to  base or advise decisions.
section_title: Testing Questionnaire Would Help Ensure Clarity and Accurate Responses
paragraphs: Testing a questionnaire with a sample of intended respondents before it is  distributed is a standard approach that survey organizations employ to  ensure questions are clear and understandable, and that respondents will  be able to provide accurate information.  In short, testing is an important  quality assurance procedure that increases the likelihood that respondents  will provide the information needed and help reduce the likelihood of  inaccurate responses.
The Bureau is aware of the benefits of testing questionnaires.  In  preparation for the 2000 Census, Congress budgeted millions of dollars for  the Bureau to develop and test questionnaires during the 1990s, which it  did using focus groups and one-on-one interviews.  Moreover, the Bureau’s  policy requires that demographic survey questionnaires be tested.
However, the Bureau did not test the overseas enumeration questionnaire.   Instead, the Bureau gave stakeholders the opportunity to review and  comment on the questionnaire before it went to OMB for final approval.  Stakeholder feedback was generally positive.  One change that was made in  response to stakeholder feedback was that “United States” was added to  “passport” in the question that asked for a respondent’s passport number.  While sharing the questionnaire with stakeholders is important, it should  not be seen as a replacement of questionnaire testing.
According to Bureau officials, the Bureau developed the overseas  questionnaire by modifying the Census 2000 short form to collect data  needed for the overseas count.  New questions asked respondents for their  employment status, Social Security number, and passport number.  The  Bureau believed that the new questions did not require testing.  However,  as stated earlier, collecting Social Security numbers and passport numbers  could be problematic.  While some stakeholders believed this to be  acceptable, other stakeholders believe it could reduce participation,  especially in Mexico where dual nationals reside.  In addition, the  questionnaire requests data on everyone in the household even if a person  is not a U.S. citizen.  These questions could be seen as too intrusive and  potentially could stop someone from completing the form, thus resulting in  an undercount.  Since there was no testing of the questionnaire, the Bureau  cannot be certain of the impact of these questions.
section_title: Data Analysis Will Not Provide Congress with Requested Information
paragraphs: In a 2001 report to Congress, “Issues of Counting Americans Overseas in  Future Censuses,” the Bureau indicated that it would provide Congress  with data on the number of people in the military, federal, private business,  nonprofit, and other categories.  However, the Bureau is only collecting  data on the number of military and federal workers; people working in  other sectors will be grouped in an “other” category.  The reason that the  Bureau is unable to provide the additional breakdown is because the length  of the questionnaire did not allow for additional check boxes. Therefore,  the Bureau collapsed people working in the private sector and nonprofit  organizations and others into one category.
section_title: The Bureau Has Identified a Number of New Challenges to Counting Overseas Americans
paragraphs: The soundness of the test design notwithstanding, the Bureau has already  identified several country-specific challenges to counting American  citizens at each of the test sites.  Together, they suggest that an  enumeration of Americans on a more global scale in 2010 would introduce  a number of unforeseen obstacles that the Bureau would need to address  on a country-by-country basis.  For example, shortly before the test was to  begin in France, the French government contacted the Census Bureau  indicating that French law prohibited the collection of race and Hispanic  origin data. Furthermore, they were also opposed to the U.S. government  asking for information on persons who were not American citizens.  The  Bureau worked with the French government and it was agreed that an  advisory would be posted on the Internet site explaining that under French  law it was not mandatory to respond to the questionnaire.
Problems have also surfaced in Mexico and Kuwait.  Stakeholders and  Bureau officials have told us that the mail system in Mexico is not always  reliable.  The concern is that the questionnaires may not make it to the  Census Bureau, or arrive too late to be processed.  In Kuwait, security  concerns have prevented the Census Bureau from posting the location of  sites where Americans can pick up the questionnaire.  While the impact of  these problems is difficult to quantify, it may prevent some Americans from  completing the questionnaire and being counted.
section_title: The Rights and Obligations of Americans Overseas Vary
paragraphs: Americans residing abroad do not have the same rights and obligations  under federal programs and activities as Americans living in the United  States.  In order to determine the rights and obligations of Americans  residing abroad, one must examine the specific statutes governing each  program.  For this study, we examined whether overseas Americans can  vote in federal elections; are subject to federal income tax; and can receive  Social Security, Supplemental Security Income, and Medicare benefits (see  table 3).  Stateside, these programs cover millions of Americans; whether  and how they extend to Americans living overseas could help inform the  treatment of U.S. expatriates in the decennial census, to the extent there  are any patterns.
subsections: section_title: Voting Rights
paragraphs: American citizens who reside outside of the United States have the right to  vote in federal elections under the 1986 Uniformed and Overseas Citizens  Absentee Voting Act (UOCAVA).   Under this law, U.S. citizens residing on  foreign soil can vote in federal elections as absentee voters of their last  state of residence, even if they have no intention of ever returning to the  United States. (American citizens residing in U.S. territories, however,  cannot so vote. The territories include Puerto Rico, the Northern Mariana  Islands, the U.S. Virgin Islands, Guam, and American Samoa.)
UOCAVA repealed legislation enacted in 1955 that was designed to prevent  members of the Armed Forces and their families from being denied their  voting rights when absent from their home or in a far-off place.  The goal  was to make it easier for military personnel to cast votes in any federal  primary, general, or special election through absentee balloting  procedures.  While the 1955 law was amended several times to, among  other purposes, specifically include other Americans living overseas, in  1986 Congress acknowledged that there was a legitimate need for further  legislation.   UOCAVA’s main purpose was to facilitate absentee ballot  voting, while also providing “for a write-in absentee ballot that may be used  in Federal general elections by overseas voters who, through no fault of  their own, fail to receive a regular  absentee ballot in sufficient time  to vote and return the ballot prior to the voting deadline in their State.”
section_title: Federal Income Tax
paragraphs: U.S. citizens are taxed on their worldwide income, subject to certain  exclusions specified in the Internal Revenue Code.  In 1913, the enactment  of the Sixteenth Amendment to the U.S. Constitution gave Congress “the  power to lay and collect taxes on income, from whatever source derived,  without apportionment among the several States, and without regard to  any census or enumeration.”  After ratification of this constitutional  provision, Congress imposed a tax on the net income of every U.S. citizen,  wherever they lived.
That the Constitution vests Congress with the power to tax Americans  living overseas on their income earned outside of the United States was  reinforced by the Supreme Court in 1924 when it ruled:  “overnment, by  its very nature, benefits the citizen and his property wherever found, and  therefore has the power to make the benefit complete.  Or, to express it  another way, the basis of the power to tax was not and cannot be made  dependent upon the situs of property in all cases, it being in or out of the  United States, nor was not and cannot be made dependent upon the  domicile of the citizen, that being in or out of the United States, but upon his relation as citizen to  the United States and the relation of the latter to him as citizen.”
There are statutory exclusions, however.  Generally, a U.S. citizen or  resident may exclude a portion (ascending each year to $80,000 during or  after 2002) of his earned income if he is a resident of a foreign country for a  full calendar year or is physically present in a foreign country for 330 days  of any 12 consecutive months and otherwise meets certain requirements.   This foreign earnings exclusion principally aims to encourage U.S. citizens  and residents to work abroad without worrying about how it might  increase their tax liability.  Indeed, in view of the nation’s continuing trade  deficits, Congress deemed it important to allow Americans working  overseas to contribute to the effort to keep American business  competitive.
section_title: Social Security Payments
paragraphs: The Social Security Act provides individuals over the age of 62 who have  worked for a minimum number of years with monthly benefit payments.   Social Security checks generally follow recipients wherever they go around  the world, subject to only a few exceptions.  In fact, Social Security  payments continue no matter how long a beneficiary stays outside of the  United Stateseven if the individual retires overseas.   Thus, if you are a U.S. citizen residing abroad you generally continue to receive your monthly  benefits.
section_title: Supplemental Security Income
paragraphs: Supplemental Security Income (SSI) is a “need-based” program designed to  help individuals who are over the age of 65, blind, or disabled.  SSI  benefits cease once a recipient remains outside the United States for a  period of at least 30 days.  If the recipient returns to this country within 30  days, SSI benefits are to continue as before.
section_title: Medicare Coverage
paragraphs: Since Medicare became effective almost 35 years ago, the program has  excluded coverage for health care services received outside of the United  States, even if those services are medically necessary.  There is a limited  exception for services occurring near U.S. borders.
The constitutionality of foreign exclusion was raised in a 1986 court  decision.  In that case, plaintiffs argued that by leaving Medicare  beneficiaries without health care coverage once they leave the United  States, the foreign exclusion deters these same individuals from traveling  overseas and consequently infringes on their constitutional right to travel  abroad.  The district court, however, found the Medicare foreign exclusion  to pass constitutional muster.  The court reasoned that the difficulties of  administering medical services abroad and the concern that Medicare  funds be spent within the United States were not particularly compelling,  but were rationally based.  It thus concluded that the foreign exclusion  satisfied the rational basis test and summarily dismissed the plaintiff’s  claim.
section_title: Legislation Would be Needed To Require the Census Bureau to Enumerate Americans Abroad
paragraphs: The Bureau has the discretion to decide whether to count American  citizens abroad.  Indeed, there is nothing in the Constitution, the Census  Act, or court decisions that would require the Bureau to count Americans  living overseas, or to not count such individuals.  Consequently, if Congress  wanted to require the Census Bureau to include this population in the 2010  Census, legislation would be needed.
The issue of whether, and if so how, to count Americans living overseas  was raised in federal court after both the 1990 and 2000 censuses.   Massachusetts challenged the results of the 1990 Census claiming that it  lost a seat in the House of Representatives as a result of the Secretary of  Commerce’s decision to enumerate and apportion federal employees  stationed abroad.  Conversely, Utah challenged the results of the 2000  Census maintaining that it lost a congressional seat because no overseas  Americans other than federally affiliated groups were included in the  apportionment numbers.  In both cases, the courts determined that the  Census Bureau has discretion under the Constitution and the Census Act to  decide whether to count Americans residing overseas.
section_title: Conclusions
paragraphs: The design of the Bureau’s overseas enumeration test falls short in many  respects, and the data that the Bureau will collect as a result of this  exercise may only partially answer key questions relating to feasibility,  cost, and data quality.  The Bureau overstated the test’s ability to answer its  key research objectives and, as a result, congressional decision making on  this issue will be that much more difficult.
The full results of the overseas enumeration test will not be available until  early 2005, when the Bureau expects to issue the last of a series of  evaluations.  However, its experience thus far has made it clear that  counting Americans abroad as an integral part of the 2010 Census would be  an extraordinary challenge, one that would introduce new resource  demands, risks, and uncertainties to an endeavor that is already costly,  complex, and controversial.
That said, to the extent that better data on the number and characteristics  of Americans abroad might be useful for various policymaking and other  nonapportionment purposes, such information does not necessarily need  to come from the decennial census.  This data could, in fact, be acquired  through a separate survey or some other type of data collection effort,  although it would still be a difficult undertaking.
The Bureau is unlikely to decide—and in fact should not decide—on its  own, whether or not to enumerate Americans abroad, and will need  congressional guidance on how to proceed.  Therefore, to give the Bureau  as much planning time as possible, it will be important for Congress to  soon decide whether the Bureau should be required to count this  population group as part of the 2010 Census or as part of a separate data  collection effort or whether there are so many obstacles to a successful  overseas count regardless of the approach that the Bureau should shelve  any plans for further research and testing.
Should Congress desire an overseas count—as part of the decennial census  or a separate effortit should consider providing the Bureau with input on  how it expects to use the data on overseas Americans.  The purposes of the  data drive the design of the enumeration; therefore, once the Bureau has a  clear idea on how the data will be used, it would be better positioned to  plan a test that will more accurately assess the feasibility of the  procedures, methodology, and resource requirements needed to  accomplish the type of count that Congress desires.  Moreover, if a second  test of enumerating Americans abroad is needed in 2006, it will be  important for the Bureau to address the shortcomings of the design of the  2004 overseas test.  Conducting a second test without this information and  a sound design to fulfill it would not be cost-effective.
section_title: Matters for Congressional Consideration
paragraphs: In order to give the Bureau as much planning time as possible, Congress  may wish to consider coming to an early decision on whether the Bureau  should be required to enumerate overseas Americans, and if so, whether  they should be counted as part of the decennial census or by some other,  separate data collection effort.  Should Congress desire an overseas  count—be it part of the decennial census or a separate data collection  effortit should consider telling the Bureau how the data would be used  (e.g., for purposes of apportionment, redistricting, allocating federal funds,  or a tally of the U.S. overseas population).  This information would enable  the Bureau to more thoroughly evaluate procedures and resources needed  to meet Congress’s specific requirements, and ultimately provide Congress  with better information with which to gauge the feasibility of such an  approach.
section_title: Recommendations for Executive Action
paragraphs: To the extent that the Bureau proceeds with plans to conduct a second  feasibility test in 2006, we recommend that the Secretary of Commerce  direct the Bureau to take appropriate steps to improve the soundness of the  test design and better address the objectives of an overseas enumeration.   Specific steps should include the following 12 actions:  Be more transparent with Congress and other stakeholders on what  variables the Bureau is able to quantitatively measure, as well as what  research questions the Bureau can and cannot answer.
Develop and pretest clear residence rules and appropriate guidelines on  who should be included in the count to prevent confusing prospective  respondents.
Ensure that its outreach and promotion strategy, data collection  methods, and other aspects of the design are free from cultural and  other biases that could introduce systemic errors.
Explore the possibility of developing more comprehensive measures of  data quality.
Test the Bureau’s ability to geocode certain noncity-style addresses such  as those with post office box numbers.
Research how best to market the overseas census to hard-to-count  groups and other, less visible, segments of a country’s overseas  American population.
Develop procedures to determine whether a return is within the scope  of the enumeration—i.e., that it is truly from an overseas source.
Actually integrate overseas data with stateside data to more thoroughly  test this activity.
Develop a cost model to provide the Bureau and Congress with better  estimates of the costs of conducting an overseas census under different  methodological and other scenarios.
Thoroughly pretest the overseas questionnaire with various groups of  potential respondents to ensure the questions are clear, appropriate, and  can be accurately answered by the unique population groups that are  found overseas.
Add more response options to the questionnaire item concerning  respondents’ primary activity.  Specific information on whether an  individual is retired, a student, or working for a private company, etc.,  could provide the Bureau with valuable data on the characteristics of  overseas Americans that could be important for some of the purposes  for which the data might be used, and just as important, could provide  the Bureau with invaluable marketing data that the Bureau could use to  develop a more effective outreach and promotion campaign.
Work with Congress and other stakeholders to develop a broad  consensus on what would be acceptable levels of accuracy,  completeness, participation, and other key measures of performance.
section_title: Agency Comments and Our Evaluation
paragraphs: The Secretary of Commerce forwarded written comments from the U.S.  Census Bureau on a draft of this report on April 13, 2004, which are  reprinted in appendix I.  The Bureau generally agreed with our key findings,  conclusions, and recommendations, and suggested some additional  context, technical corrections, and clarifications, which we have  incorporated.
The Bureau disagreed with our recommendation that it be more  transparent with Congress and other stakeholders on what variables the  Bureau is able to quantitatively measure, as well as what research  questions the Bureau can and cannot answer.  The Bureau believes that it  has always been transparent about its plans and the likely limitations of  this first test.
Although the Bureau’s test plan describes the limitations associated with  answering its various research questions, nowhere does it disclose that its  fundamental research objective to “determine the feasibility, quality, and  cost of collecting data from U.S. citizens living overseas” will only be  partially answered.  Indeed, none of the documentation we reviewed,  including the test project plan or briefing slides provided to congressional  staff in April 2003, indicated either explicitly or implicitly that this test was,  as the Bureau maintains, “only the most basic assessment of feasibility,” or  that the Bureau would not be able to fully answer its key questions  regarding feasibility, data quality, and cost.  For example, as noted in our  report, even though one of the principal objectives of the test was to  determine the cost of collecting data from U.S. citizens living overseas, the  Bureau’s test design lacked a research question for this objective.
The Bureau agreed, however, that as it completes its evaluations and  documents its findings from the test, it will be “critical” to highlight the  various qualitative and quantitative limitations that could affect  congressional deliberations on this subject.
As agreed with your offices, unless you release its contents earlier, we plan  no further distribution of this report until 30 days from its issue date.  At  that time we will send copies to other interested congressional committees,  the Secretary of Commerce, and the Director of the U.S. Census Bureau.   Copies will be made available to others upon request.  This report will also  be available at no charge on GAO’s Web site at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (202) 512-6806 or by e-mail at daltonp@gao.gov or Robert  Goldenkoff, Assistant Director, at (202) 512-2757 or goldenkoffr@gao.gov.   Key contributors to this report were Lisa Pearson, Charlesetta Bailey, Betty  Clark, Ellen Grady, Ronald La Due Lake, Andrea Levine, and Daniel  Messler.
section_title: Comments from the Department of Commerce
section_title: Criteria for Evaluating Census Research Design for Overseas Enumeration Test
paragraphs: 1. How clearly are the objectives of the research design presented? a. Are the test/research objectives and/or questions specified clearly in  the design?  b. Are concepts defined where necessary?  2. How sound is the research design?  a. Do the research questions match the research objectives?  b. Is the rationale for the determination of the size and type of sample  explained?  c. Are potential biases recognized and addressed (e.g., cultural bias,  question item bias, or sample bias)?  d. Does the sample selection reflect the full range and full cycle of  entities or processes that should be considered?   3. How appropriate is the data collection strategy?  a. Is the mode of data collection stated clearly? b. Is the timing and frequency of data collection considered?  c. Is the data collection method appropriate for reaching the intended  sample population?  d. Is the data collection instrument appropriate for the sample  population?  e. Is a plan for administering and monitoring the data collection  discussed in the design?  f. How well does the design address factors that may interfere with  data collection?  g. How well are methods for assuring adequate response rates  addressed?  4. How thoroughly did Census test the survey instrument(s)? a. Were any new survey items cognitively tested? b. Were field tests conducted to identify the best question wording and  determine whether changes in questions are likely to achieve the  change objectives? c. Were research studies used to address possible changes in the  questionnaire? 5. How involved were relevant stakeholders in the research planning? a. Were relevant stakeholders for the research objectives identified? b. Were stakeholders involved in the planning or review of the methods  of data collection? c. Were appropriate stakeholders participants in the review and testing  of the questionnaire?  6. How sound is the design’s plan for reaching the target sample? a. Are the goals of the outreach strategy feasible? b. Are the methods of the outreach strategy viable?  7. How sound is the plan for implementing test site activities? a. Does the design consider possible mistakes and their consequences  (including their seriousness)? b. Does the design assure that sufficient evidence will be gathered to  answer the research questions? c. Does the design consider the level of difficulty in obtaining the data?  8. How good is the relationship between the research design and the time  and resources allocated? a. Does the execution of the design appear feasible within the stated  time frame? b. Do the proposed resources for the execution of the design appear  feasible? c. Are the roles and responsibilities of key parties specified?  9. How well does the design develop a data analysis plan? a. Is the method of enumeration clearly presented? b. Are the proposed analytical techniques for analysis presented? c. Does the design provide a basis for comparing the results of the  research? d. Does the design discuss the possible limitations of the findings/test  results?
section_title: 2004 Overseas Test Census Questionnaire
subsections: section_title: GAO’s Mission
paragraphs: The General Accounting Office, the audit, evaluation and investigative arm of  Congress, exists to support Congress in meeting its constitutional responsibilities  and to help improve the performance and accountability of the federal government  for the American people. GAO examines the use of public funds; evaluates federal  programs and policies; and provides analyses, recommendations, and other  assistance to help Congress make informed oversight, policy, and funding  decisions. GAO’s commitment to good government is reflected in its core values of  accountability, integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents at no cost is  through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older  products. The Web site features a search engine to help you locate documents  using key words and phrases. You can print these documents in their entirety,  including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and  correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site  daily. The list contains links to the full-text document files. To have GAO e-mail this  list to you every afternoon, go to www.gao.gov and select “Subscribe to   e-mail alerts” under the “Order GAO Products” heading.
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Public Affairs