Comparison of the 2003 American Community Survey Voluntary 

 versus Mandatory Estimates 

By Karen E King, Michael Starsinic, and Alfredo Navarro

Decennial Statistical Studies Division 

US Census Bureau 

Presented by Karen E. King 

Prepared For the 2012 Federal Committee on 

 Statistical Methodology (FCSM) Research Conference 

January 10-12, 2012

This paper is released to inform interested parties of research and to encourage discussion. Any views expressed on 

statistical, methodological, or technical issues are those of the author and not necessarily those of the U.S. Census 

Bureau. 

 
Comparison of the 2003 American Community Survey Voluntary 

 versus Mandatory Estimates 

I. Introduction 

The American Community Survey (ACS) is a rolling monthly survey, which collects essentially the same detailed 

demographic, housing, and socio-economic data as were collected on the 2000 Decennial Census Long Form

questionnaire. Since its introduction, the ACS has used a mandatory collection method except for a couple of 

months in 2003 when as voluntary collection method was tested. 

The US Census Bureau was requested by Congress in 2002 to conduct research to determine if the ACS could be 

implemented as a voluntary collection survey. In 2003, working closely with staff of the Technology, Information 

Policy, Intergovernmental Relations and Census Subcommittee and the House Government Reform Committee, a 

test was designed and developed to assess the effect of a switch from a mandatory to a voluntary ACS on 

feasibility – by looking at costs and workloads 

public reaction – by analyzing response rates by mode 

quality – in terms of sampling error and levels of unit and item response 

While the test was not a randomized experiment, the Census Bureau did conclude that: 1

The estimated annual cost of implementing the ACS would increase by at least 38 percent if the survey was 

voluntary and the survey maintained the current reliability levels. (See Report # 3, Tables 13 and 14, pages 16-

17.) 

A dramatic decrease occurred in mail response when the survey was voluntary. The mail cooperation rate fell 

by over 20 percentage points, and the final response rate after all three modes of data collection was about 4 

percentage points lower. The reliability of estimates was adversely impacted by the reduction in the total 

number of completed interviews and a shift of a large number of interviews to the personal follow-up data 

collection mode. 

Perhaps of greatest concern, the use of voluntary collection methods had a negative impact on traditionally low 

response areas that will compromise our ability to produce reliable data for these areas and for small population 

groups such as Blacks, Hispanics, Asians, and American Indians and Alaska Natives. 

The original study does not assess if estimates produced from a voluntary ACS would differ from estimates from a 

mandatory ACS. Our motivation to do further analysis of the test was the decision made in 2010 by the government 

of Canada to cancel their mandatory 2011 Long Form of the Canadian Census and replace it with a voluntary 

survey. Many experts expressed concern about the potential for such change to result in biased estimates.2 The US 

Census Bureau had the survey data for the 2003 voluntary test available for further examination. It was thought 

additional analysis might help to answer the following research questions: 

1. Do annualized ACS estimates based on data collected using a voluntary collection method differ from those 

based on data collected using the mandatory collection method? 

2. Is the answer different if the ACS estimates are based on initial selection weights versus modified final 

weights? 

II. Background 

II.A. ACS Data Collection Methods 

 

1

Two detailed reports on the design, analysis, and findings from this research can be found at 

http://www.census.gov/acs/www/library/by_series/implementing_the_acs/ as Report #3 and Report #11. 

2

http://bulletin.imstat.org/archive-vol40-2011/#Issue01 page 6 -7.
The data for a single month of ACS sample cases are collected over a three-month period. The design of the ACS 

relies on three modes of data collection – mail, telephone follow-up, and personal visit follow-up. Significant cost 

variations exist among the three modes; personal visit follow-up is by far the most expensive mode of data 

collection. 

The Census Bureau first attempts collecting ACS data using mail-out and mail-back techniques. To maximize mail 

response, several mailings are used, including an advance letter, initial questionnaire package, reminder card, and a 

targeted second mailing to nonrespondents. Call centers provide telephone assistance to help households complete 

the forms they receive in the mail. Returned mail forms are data captured by keying and reviewed for completeness.3

Telephone interviewers resolve incomplete forms in an edit follow-up operation. 

ACS interviewers follow up on nonrespondents to the mail-out and mail-back collection through separate telephone 

and personal visit follow-up operations. For example, data collection for the March panel starts when we mail the 

survey questionnaires in late February. In April, interviewers conduct a telephone follow-up operation to collect data 

for nonresponding addresses for which a telephone number is available. At the end of April, a sample of about 1-in3 of the addresses that do not respond by mail or telephone is selected for a personal visit follow-up operation in 

May. A 2-in-3 sample of addresses that could not be mailed (due to incomplete address information) is added to the 

personal visit follow-up workload. For more details about the ACS data collection methods refer to the ACS Design 

and Methodology Report.4

II.B. 2003 ACS Sample Design 

The ACS sampled approximately 829,000 housing unit addresses in 2003 from the Master Address File (MAF). 

This was during the ACS's demonstration phase, when a nationwide representative sample was collected from 1,240

counties. The ACS used two distinct sampling methods in the 2003 sample, one for 36 "comparison" counties 

designed to mimic the then-current design for the full-sample ACS, and the other for the remainder of the country 

(non-comparison counties). 

For the non-comparison counties, the first stage of sampling involved dividing the United States into primary 

sampling units (PSUs) —most of which comprised a metropolitan area, a large county, or a group of smaller 

counties. PSUs were contained within state boundaries. The PSUs were then grouped into strata on the basis of 

independent information, that is, information obtained from the decennial census or other sources. The strata were 

constructed so that they were as homogeneous as possible with respect to social and economic characteristics that 

were considered important by ACS data users. PSUs with a sufficiently large population were placed in their own 

stratum and selected with certainty. The remaining PSUs were grouped into strata and a pair of PSUs was selected 

from each non-certainty stratum. The probability of selection for each PSU in the stratum was proportional to its 

estimated 1996 population. In the second stage of sampling, a sample of housing units within the sample PSUs was 

drawn. Housing units were identified as the ultimate sampling units (USUs). The USUs sampled in the second 

stage consist of housing units which were systematically drawn from sorted lists of addresses of housing units from 

the MAF. In 2003, the selected PSUs contained 1,204 non-comparison counties. The housing unit sampling rate was 

based on a targeted annual national sample size of 829,000 housing units. The final sampling interval for most states 

in 2003 was determined to be roughly 189. 

The 36 comparison counties were: Pima County, AZ; Jefferson County, AR; San Francisco County, CA; Tulare 

County, CA; Broward County, FL; Upson County, GA; Lake County, IL; Miami County, IN; Black Hawk County, 

IA; De Soto Parish, LA; Calvert County, MD; Hampden County, MA; Madison County, MI; Iron, Reynolds, and 

Washington Counties, MO; Flathead and Lake Counties, MT; Douglas County, NE; Otero County, NM; Bronx 

Borough, NY; Rockland County, NY; Franklin County, OH; Multnomah County, OR; Fulton County, PA; 

Schuylkill County, PA; Sevier County, TN; Fort Bend and Harris Counties, TX; Starr and Zapata Counties, TX; 

Petersburg City, VA; Yakima County, WA; Ohio County, WV; and Oneida and Vilas Counties, WI.5 

 

3

 In the 2003 ACS, data was captured by keying from paper. Currently data is captured by keying from image. 

4

 The ACS Design and Methodology Report http://www.census.gov/acs/www/methodology/methodology_main/. 

5

 For more details, see the 2003 Accuracy Document at 

http://www.census.gov/acs/www/data_documentation/documentation_main/ 
The sampling rate within each comparison county was determined at the Census block level, depending on the size 

of the governmental units and the tract that the block was contained within. 

2003 ACS Sampling Rates

Type of Area Fort Bend and 

Harris, TX

All Other 

Counties

Blocks in smallest governmental units 

(fewer than 200 HUs) 

10% 10%

Blocks in smaller governmental units 

(200 than 800 HUs) 

3% 7.5% 

Blocks in small governmental units 

(between 800 and 1200 HUs) 

1.5% 3.75% 

Blocks in Large Tracts (more than 2000 

HUs) 

0.735% 1.837%

All other blocks (including ungeocoded 

records) 

1% 2.5% 

Source: 2003 Accuracy of the American Community Survey 

 

For 2003, all comparison test counties except Fort Bend and Harris had overall sampling rates of about 2.5 percent. 

The overall sampling rate in Fort Bend and Harris Counties was about 1 percent. 

II.C. Design of the 2003 Voluntary Test 

The 2003 voluntary test was conducted using the March and April sample panels, which contained about 140,000 

sampled addresses. The sample for the ACS voluntary test was designed to study four experimental mail

treatments–two mandatory and two voluntary. One voluntary collection method used a standard survey approach to 

explain the voluntary nature of the survey, the approach that the Census Bureau uses for its current surveys. A 

second voluntary collection method explained more directly that the survey was voluntary. Since the primary focus 

of this test was to evaluate the effect of voluntary collection methods on the ACS, we evenly distributed 75 percent 

of the combined 2003 March/April sample to the two voluntary mail collection methods and the remaining 25 

percent between the two mandatory mail collection methods.

Unlike the mail portion, the test used only voluntary collection methods in the telephone and personal visit followup operations starting in April. We concluded that assigning both voluntary and mandatory collection methods to a 

sample of cases or a sample of interviewers introduced potential implementation risks to the study that the use of 

one method could avoid during the test period. 

The test’s sample design divided the universe into two strata, high response areas (HRA) and low response areas 

(LRA). We created these strata using tract-level long form mail return rates from Census 2000. Based on data from 

the 2001 ACS, people in the LRA stratum were younger, more likely to be Hispanic and non-White, and had more 

―other relative‖ and ―non-relative‖ household members than people in the HRA stratum. The LRA stratum also had 

fewer people with college educations, more renters, more households who speak a language other than English at 

home, and more households with lower incomes compared with the HRA stratum. Within strata, experimental 

methods were designated in a systematic manner to ensure that we assigned 75 percent of the sample evenly to the 

two voluntary collection methods and 25 percent of the sample to the two mandatory collection methods.

For this analysis, the two voluntary mail collection methods in March and April are combined as the voluntary 

panel, and the two mandatory mail methods are omitted. This resulted in a loss of about 25 percent of the March 

and April 2003 samples. 
III. Methodology 

For this analysis, we use a set of weighted estimates produced from the data provided by respondents from the 

voluntary panels in March and April 2003 and another set of weighted estimates produced from the data provided by 

respondents from the mandatory panels in January, February, and May-December 2003. The mandatory estimates 

include sample cases from the November and December 2003 sample panels that responded in January and 

February, 2004. The weights for these estimates depend on the particular comparison being done. It starts with the 

2003 ACS initial selection weights (the ACS basic weight adjusted for Computer Assisted Personal Interviewing 

(CAPI) sub-sampling). To produce weighted collection method estimates that would approximate the estimates from 

a full year's worth of ACS data, the initial selection weights are annualized in the following manner: 

The initial weights of sample cases receiving the voluntary collection method are multiplied by 8.0, which is 

derived from (12/2 x 4/3) because only two months of the twelve were designated voluntary, and only 3⁄4 of the 

sample in those months were collected using this method. 

The initial weights of sample cases receiving the mandatory collection method are multiplied by 1.2 because ten 

months of the twelve were designated mandatory. 

The first comparison of collection based estimates was done using the annualized initial weights described above. 

These estimates reflect the characteristics of the respondent population and housing included in a mandatory and 

voluntary ACS prior to any adjustments for nonresponse and coverage bias. 

From the initial 2003 research, we determined that there was a difference in the overall survey response rates 

between the mandatory and voluntary collection method panels. As a result, the collection method count estimates 

are different for many characteristics because no noninterview adjustments are applied to the weights used for this 

set of tabulations. So, this analysis focuses on weighted percent and ratio estimates. 

A set of annualized final weights is also derived. These weights are adjusted to correct for nonresponse and 

noncoverage bias. Specifically, the annualized initial selection weights of voluntary and mandatory panel cases are 

put through a modified version of the 2003 weighting process separately by collection method. The following 

modifications to the 2003 methodology are warranted due to the small sample size of the voluntary panel. 

1. The sample panel month is used instead of interview month in the weighting process. This is the best way to 

keep the collection method groups separate and reduce contamination during the weighting process. 

2. The weighting process is controlled at the state level instead of the 2003 weighting areas due to the small 

number of expected person interviews in the voluntary panels. 

3. A modification is made to the nonresponse adjustment to use state instead of county/tract to define the 

weighting cells 

4. Both the mandatory and the voluntary collection method estimates are controlled to July 1, 2003 Population 

Estimates Program estimates. 

This set of comparisons is similar to the comparisons described above. This time the collection method estimates 

are calculated using annualized final weights. It looks for differences between collection method estimates, but this 

time for the total population rather than for the respondent population. 

Weighted estimates by collection method are tabulated for over 400 key ACS characteristics – specifically the 

estimates that are included in the ACS data profiles. These included totals of persons, households, and housing units, 

as well as ratios and percents. Additional percent estimates are calculated for many of the count estimates. More 

details about the characteristics examined are provided in the next section. 

For each characteristic, the voluntary collection method estimate is compared to the mandatory estimate and the 

difference between the collection method estimates is examined in two ways. First, we assess the differences based 

on size alone. A characteristic is considered to have a ―large‖ difference if either the absolute difference between 

collection method estimates is greater than 1 percent or the relative difference is greater than 10 percent. Changes of 

this size could have practical implications if the ACS switches from a mandatory collection survey to a voluntary 
collection survey. It may also indicate symptomatic problems with a voluntary collection method. Secondly, we 

focus on characteristics with statistically significant differences regardless of size. 

 All significant differences are determined at the 90 percent confidence level.6 Characteristics with differences that 

are both ―large‖ and statistically significant are of particular interest. 

The majority of the work is done at the national level. Due to the small number of interviews in the voluntary 

panels, only a few sets of sub-national estimates are produced. These collection method estimates are for the eight 

2010 Census Integrated Communication segmentation groups and for the four largest states (California, Florida, 

New York, and Texas). 7

Some additional information on the 2010 Census Integrated Communication eight segmentation groups is warranted 

here. All census tracts are assigned to segmentation groups based on social, economic, housing, and demographic 

characteristics and historical information about census participation. The appendix provides some summary 

information about the characteristics of these eight segmentation groups. In this analysis, we refer to the 

segmentation groups as: 

1. Average - Homeowners 

2. Average - Renters 

3. Economically Disadvantaged - Homeowners 

4. Economically Disadvantaged - Renters 

5. Ethnic Enclave - Homeowners 

6. Ethnic Enclave - Renters 

7. Single Unattached Mobiles 

8. Advantaged - Homeowners 

Group 1 (Average – Homeowners) and group 8 (Advantaged - Homeowners) combined contain about 60 percent of 

the total U.S. population. On the other hand, the two Economically Disadvantaged and the two Ethnic Enclave 

groups (groups 3 through 6) combined represent only about 15 percent of the total U.S. population. The remaining 

two groups, Average - Renter and Single Unattached Mobiles, make up the balance. 

IV. Key Estimates Considered 

As mentioned in the methodology section, estimates by collection method for over 400 key ACS characteristics are 

tabulated using annualized weights. These items come from the standard ACS data profiles and include social,

economic, housing, and demographic characteristics. Given that our comparison is aggregating and comparing data 

from different months of the year, we want to remove from consideration any of the 400 characteristics that might 

be influenced by seasonal differences within the year or any year to year trends. To help with this, estimates from 

the 2002 ACS and the 2004 ACS are formed using the same monthly breakdowns, March and April panels versus 

the rest of the year, and the appropriate annualized initial weights. Although there was only one collection method 

used in the 2002 ACS and the 2004 ACS, we refer to these month-based estimates as collection method estimates 

for consistency. By characteristics, these collection method estimates are compared for the 2002 ACS and 2004 

ACS. If the resulting differences meet criteria, they are flagged as large and/or statistically significant. The next 

two sections describe how we use this information. 

IV.A. Number of Large Differences 

In Table 1 we see the percent of characteristics with differences considered by our criteria as large for the 2002 

ACS, the 2003 ACS, and the 2004 ACS respectively. At the national level, about 4.4 percent of the characteristics 

from the 2002 ACS and the 2004 ACS have absolute differences greater than 1 percent, whereas the 2003 ACS rate 

 

6

 No adjustments were made for multiple comparisons.

7

 For the 2010 Census eight tailored communication approaches were developed to encourage participation in the 

Census. A behavioral segmentation model was built based on indicators related to census mail-back behavior, 

media usage and attitudinal information. The model segments the U.S. population into relatively homogeneous 

groups that exhibited different response rates to the Census 2000. 2010 Census Integrated Communication 

Campaign Plan, US Census Bureau, August 2008, pages 33 – 43.
is twice that at 8.5 percent.8

 The same is true for the percent of characteristics with relative differences greater than 

10 percent when the 2003 ACS is compared to the other two years at the national level. 

Table 1. The Percent of ACS Profile Estimates with Absolute Difference > 1% or with Relative Difference>10% by 

Year and by Nation, State and Segmentation Group * 

Absolute Difference > 1.0% Relative Difference > 10.0% 

2002 2003 2004 2002 2003 2004

Nation 4.4% 8.5% 4.4% 6.2% 12.9% 5.9% 

Segmentation Groups: 

Average - Homeowner 6.9% 10.3% 6.7% 12.6% 16.6% 11.9% 

Average - Renter 13.6% 19.5% 9.7% 16.8% 26.0% 18.1% 

Economically 

Disadvantaged - 

Homeowner 26.3% 26.0% 19.5% 34.3% 34.2% 29.3% 

Economically 

Disadvantaged - Renter 36.9% 37.7% 36.0% 36.4% 47.0% 40.5% 

Ethnic Enclave - 

Homeowner 32.1% 33.6% 31.4% 41.5% 33.9% 41.3% 

Ethnic Enclave - Renter 36.3% 47.5% 36.4% 38.3% 52.1% 42.4% 

Single Unattached Mobiles 20.3% 22.9% 17.4% 28.5% 34.0% 25.0% 

Advantaged - Homeowner 7.9% 12.8% 6.9% 15.8% 24.3% 14.1% 

States: 

California 14.9% 18.6% 15.4% 20.1% 27.2% 21.8% 

Florida 17.1% 28.5% 20.5% 27.1% 35.0% 33.3% 

New York 19.9% 23.2% 21.6% 23.0% 30.5% 26.6% 

Texas 19.0% 29.4% 18.4% 28.1% 33.7% 25.0% 

Source: U.S. Census Bureau: 2002, 2003,and 2004 American Community Survey Special Tabulation 

http://www.census.gov/acs/www/data_documentation/data_main/ 

* The base of the percentages is the roughly 400 key characteristics. 

At the segmentation group and state levels, the 2003 ACS also have higher numbers of large differences, but the 

results do not stand out so much. These results give us our first indication that there may be something more going 

on for 2003 than just seasonal effects or random noise. 

IV.B. Number and Patterns of Significant Differences

For each characteristic, the number of statistically significant differences between the collection method estimates 

across the 2002 ACS, the 2003 ACS, and the 2004 ACS are counted.9 Of the roughly four hundred ACS 

characteristics, about 35 percent show statistically significant differences between collection method estimates in the 

2003 ACS. For the 2002 ACS and the 2004 ACS, the comparable numbers of monthly differences are 22 percent 

and 13 percent, respectively. This result gives us our second indication that there may be something more going on 

for 2003 than just seasonal effects or random noise.

The pattern of these differences across the years is also of interest. That is, if a characteristic is significantly 

different in all three years then the pattern is Yes, Yes, and Yes. If the opposite is true then the pattern is No, No, 

and No. Characteristics with consistent patterns like those described above are of no interest to us and are removed 

from further consideration. Most characteristics with patterns showing statistically significant differences between 

estimates for two out of the three years are also removed. The exceptions to this rule are described below. In total, 

about 300 of the 400 key characteristics at the national level are removed and ignored in the annualized estimate 

 

8

 Whether or not a difference is statistically significant was ignored at this point in the analysis. 

9

 The size of the difference was ignored in this examination. 
comparisons described above. The remaining 112 key characteristics seem to indicate a possible voluntary 

collection method effect. 10

 

The 112 key characteristics with suspected voluntary collection method effects fall into five difference patterns. 

Table 2 gives a description of the five patterns of interest and the number of key characteristics falling into each

pattern group. The majority of key characteristics fall into Pattern Y1 where the 2003 ACS difference is statistically 

significant while the differences for the other two years are not. Another interesting pattern is Pattern Y2 where the 

2002 ACS and the 2004 ACS have a statistically significant difference, but the 2003 ACS does not. A small set of 

characteristics have difference patterns where there is a detectable shift in the direction of the significant difference. 

For example in Pattern Y3, the 2002 mandatory month-based estimate is significantly larger than the voluntary 

estimate, but the 2003 mandatory collection method estimate is significantly smaller than the voluntary estimate. 

Table 2. Number of Key Characteristics Showing a Possible Voluntary Collection Method Effect by Difference 

Pattern at the National Level 

Pattern Difference Pattern Description 

Significantly Different? Number of Key 

Characteristics 

2002 2003 2004

Y1 Only 2003 is different No Yes No 92

Y2 Only 2003 is not different Yes No Yes 6 

Y3 2002 & 2003 are in different directions Yes Yes No 6 

Y4 2003 & 2004 are in different directions No Yes Yes 7 

Y5 2003 is in a different direction from 2002 & 2004 Yes Yes Yes 1 

Source: U.S. Census Bureau: 2002, 2003,and 2004 American Community Survey Special Tabulation 

http://www.census.gov/acs/www/data_documentation/data_main/ 

The number of statistically significant differences is smaller at the segmentation group level. This is not surprising 

considering many of the groups are extremely small, i.e., containing 2 to 3 percent of the population. Only three out 

of the eight segmentation groups show statistically significant differences for 20 percent or more of the 

approximately 400 key characteristics examined. A similar three year comparison is done for this level, but there 

are no consistent significant difference patterns. Some of the more populated segmentation groups show results 

consistent with those at the national level so the same set of key characteristics is examined further at this level. See 

section V Results. 

V. Results 

Recall the research questions are: Do annualized estimates based on data collected using a voluntary collection 

method differ from those based on data collected using the current mandatory ACS? Is the answer different if 

estimates are based on initial selection weights versus modified final weights? 

V.A. Comparison Using Estimates Based on Initial Weights 

V.A.1. National Level 

Of the over 400 key estimates studied, 112 have statistically significant differences that are not believed to be due to 

seasonal effects. About 15 percent of the 112 have an absolute difference greater than 1 percent and roughly 25 

percent have a relative difference greater than 10 percent. Of those characteristics with results considered large 

(those meeting either of those criteria), several important statistically significant differences are noted. For a few 

important characteristic percentages, Table 3 gives the 2003 ACS mandatory and voluntary collection method 

estimates, indicates whether the difference between them is significant at the 90 percent confidence level, and 

provides the absolute difference (Abs Diff) and the relative difference (Rel Diff) between the two collection method 

estimates. These characteristics are discussed in more detail below. 

 

10 See Appendix 3 of U.S. Census Bureau (2011 b) for more detailed information about the patterns of significant 

differences across ACS 2002, 2003, and 2004. 
Table 3: Comparisons of the 2003 ACS Mandatory and Voluntary Percentage Estimates Using Initial Weights at the 

National Level 

Topic Characteristic 

Mand 

Mand Vol Sig Diff Abs Diff Rel Diff 

Residence 1 year ago Different house in US 14.5 12.9 Yes 1.6 11.2 

Educational attainment High school graduate or

equivalent 29.8 31.2 Yes 1.4 4.7 

Employment status Unemployed in the Labor 

Force 4.9 4.7 Yes 0.2 4.1 

Income 

HH income $200,000 or 

more 2.5 2.0 Yes 0.5 16.9 

Family income $200,000 

or more 3.2 2.6 Yes 0.6 17.1 

Poverty Families with related 

children < 5 years only 15.9 17.5 Yes 1.6 10.0 

Housing occupancy Vacant Units 10.8 12.4 Yes 1.6 14.8 

Race Alone or In 

Combination Some Other Race 5.2 4.4 Yes 0.8 15.3 

Source: U.S. Census Bureau: 2003 American Community Survey Special Tabulation 

http://www.census.gov/acs/www/data_documentation/data_main/ 

Residence 1 year ago: Table 3 shows results for the percent of the population reporting living in a different 

house in the U.S. last year. However, all seven categories of residence 1 year ago show statistically significant 

differences with a shift from recent movers towards non-movers for the voluntary collection method. Most 

have a relative difference of 7 percent or higher. This suggests that the voluntary ACS estimate included a 

smaller proportion of movers. 

Educational attainment: The percent of the population that are a high school graduate or equivalent is shown in 

Table 3, but differences for the four categories representing those with some college or less are all statistically 

significantly of various magnitudes with the voluntary collection method estimates being higher. Therefore the 

voluntary ACS estimate includes a smaller proportion of respondents with higher levels of education. 

Employment status: The percent of the labor force that is unemployed is one of several categories in this topic 

that is significantly different. For this category, the voluntary collection method estimate includes a smaller 

proportion of the unemployed than the mandatory collection method estimate. 

Income: For both the household income and family income categories of $200,000 or more, the voluntary 

collection method estimate is significantly lower than the mandatory collection method estimate. All those with 

income between $75,000 and $199,000 show significantly higher voluntary collection method estimates. This 

suggests the voluntary ACS estimates includes a smaller proportion of respondents with high income. 

Poverty: Three out of the nineteen family and person poverty rate categories have statistically significant 

differences between the voluntary and mandatory collection method estimates. All three categories show a 

voluntary collection method estimate that is higher than their respective mandatory collection method estimates. 

Of the nineteen categories in this topic, the category families with related children under 5 years only (presented 

in Table 3) have both the largest absolute difference and the largest relative difference. This suggests the 

voluntary ACS estimate includes a higher proportion of lower-income families. 

Housing occupancy: The percent of vacant housing units has a voluntary collection method estimate that is

significantly larger than the mandatory collection method estimate. This is not surprising because there is more 

nonresponse among occupied housing units in the voluntary panels and the initial weights used to form the 

collection method estimates do not include an adjustment for nonresponse bias. It is expected that the 

difference would disappear once modified final weights are used to form the collection method estimates. 
Race alone or in combination: Four out of the six categories are significantly different with a significant shift in 

the distribution away from Asian and Hawaiian and Other Pacific Islander towards Black and White. For some 

other race, the voluntary collection method estimate is significantly smaller than the mandatory collection 

method estimate and is considered to be a large difference. 

V.A.2. Segmentation Group Level 

The same set of 112 key characteristics is examined closely at this level. Table 4 demonstrates how the size of the 

differences between the collection method estimates varies by segmentation group for the characteristic percent of 

the population, ―residing in a different housing unit 1 year ago‖. There is one thing consistent across groups; the 

voluntary collection method estimate for this characteristic is never significantly larger than the mandatory 

collection method estimate. This result is similar to what is seen at the national level. 

Table 4: Comparison of the 2003 ACS Voluntary and Mandatory Percentage Estimates of Residing in a Different 

Residence 1 Year Ago Using Initial Weights By Segmentation Groups 

Specific 

Characteristic Segmentation Group Mand Vol Sig Diff Abs Diff 

Rel 

Diff 

Percent of 

Population Residing 

in a Different house 

in US 1 Year Ago 

Average - Homeowner 13.5 12.3 Yes 1.2 8.9 

Average - Renter 19.8 16.9 Yes 2.9 14.6 

Economically Disadvantaged - 

Homeowner 17.5 17.4 No 0.1 0.6 

Economically Disadvantaged - 

Renter 18.5 15.7 Yes 2.8 15.1 

Ethnic Enclave - Homeowner 12.8 9.7 Yes 2.9 22.7 

Ethnic Enclave - Renter 13.8 12.5 No 1.3 9.4 

Single Unattached Mobiles 25.6 22.7 Yes 2.9 11.3 

Advantaged - Homeowner 10.4 9.1 Yes 1.3 12.5 

U.S. Census Bureau: 2003 American Community Survey Special Tabulation 

http://www.census.gov/acs/www/data_documentation/data_main/ 

Other characteristics showing similar results to those seen at the national level for the majority of segmentation 

groups include percent high school graduate or equivalent, percent of households and families with income of 

$200,000 or more and the percent in the some other race category. 

Characteristics showing different results for the segmentation group level include the percent of unemployed in the 

labor force and the multiple categories of poverty. For the characteristic percent of unemployed in the labor force, 

six out of eight segmentation groups do not show a statistically significant collection method difference, which 

includes the four groups characterized as having a higher than average unemployment rate. For the nineteen 

categories of poverty, very few segmentation groups show a statistically significant difference between collection 

method estimates except for Ethnic Enclave – Renters. For this group, twelve of the nineteen poverty categories 

show a statistically significant difference with the voluntary collection method estimates being significantly higher 

than the mandatory collection method estimates. 

V.A.3. State Level 

Limited analysis is done for the four largest states: California, Texas, New York and Florida. Again the focus is on 

the 112 key characteristics mentioned above in the national and segmentation comparison. For most characteristics, 

there are few consistent patterns of statistically significant differences between collection method estimates observed 

among these states. Household income of $200,000 or more is the one characteristic studied with a statistically 

significant difference for all four states. Three out of four states have statistically significant differences between 

collection method estimates for key characteristics such as residency 1 year ago and race alone or in combination. 

Only one state out of four shows a statistically significant difference estimates for either high school graduate (or 

equivalent), or unemployed in the labor force, or families with related children under 5 only. 
V.B. Comparison Using Estimates Based on Modified Final Weights 

V.B.1. National Level 

Only 74 of the 112 key characteristics of interest continued to show a voluntary collection method effect after 

modified final weights are applied that adjust for nonresponse and noncoverage bias. The percent of characteristics 

with large differences between collection method estimates is about the same compared with what is shown earlier 

in Table 1 for the 2003 ACS. Table 5 gives percentage estimates by collection method for the same topics and 

characteristics seen in Table 3. In general there is little change in the relationship between the collection method 

estimates for most of the characteristics presented here when compared to those using only the initial selection 

weights. 

Table 5: Comparison of the 2003 ACS Mandatory and Voluntary Percentage Estimates Using Modified Final 

Weights at the National Level 

Topic Characteristics Mand Vol Sig Diff Abs Diff Rel Diff 

Residence 1 year ago Different house in US 

14.9 13.4 Yes 1.5 10.1 

Educational 

Attainment 

High School Graduate or

equivalency 29.7 30.9 Yes 1.2 4.0 

Employment Status Unemployed in the labor 

force 5.0 4.8 Yes 0.2 3.8 

Income 

HH income $200,000 or 

more 2.4 2.1 Yes 0.3 15.0 

Family income $200,000 or 

more 3.1 2.6 Yes 0.5 15.0 

Poverty Families with related kids < 5 

years only 16.4 17.5 No 1.1 6.4 

Housing occupancy Vacant Units 10.2 11.3 Yes 1.1 10.6 

Race alone or in 

combination Some Other Race 5.4 4.5 Yes 0.9 16.4 

Source: U.S. Census Bureau: 2003 American Community Survey Special Tabulation 

http://www.census.gov/acs/www/data_documentation/data_main/ 

There are a few characteristics in Table 5 that are impacted by the change of weights. For example, while several of 

the poverty estimates are significantly different across collection methods using the initial weights, no poverty 

estimates are significantly different when calculated with the modified final weights. For employment status, eleven 

out of the twelve characteristics have statistically significant differences after the final weighting compared with 

only five out of twelve based on the initial weights. 

One interesting result seen in Table 5 is the voluntary collection method estimate for the percent of vacant housing 

units is still significantly larger than the mandatory collection method estimate. The modified weighting process 

does not remove the statistically significant difference between the collection method estimates. The result seems 

beyond sampling variability, but we suspect that this could be an artifact of the modified weighting methodology. 

V.B.2. Segmentation Group Level 

Using final weights, the percent of the roughly 400 key characteristics showing large differences and the percent of 

the 400 showing a statistically significant difference between mandatory and voluntary collection estimates is

comparable with what is seen with the initial weights. The Ethnic Enclave – Renters segmentation group have one 

of the highest percent of key characteristics with large differences – 48 percent with an absolute difference greater 

than 1 percentage point, and 52 percent with a percent difference of 10 percent or more. These are comparable with 

what is seen with the initial weights. Table 6 gives the percentage estimates by collection method for the same 
characteristic in Table 4. In general there is little noticeable change in the relationship between the collection 

method estimates for most of the segmentation groups compared to those using the initial selection weights. 

Table 6: Comparison of the 2003 Voluntary and Mandatory Percentage Estimates of Residence 1 Year Ago Using 

Modified Final Weights by Segmentation Groups 

Specific 

Characteristic Segmentation Groups Mand Vol Sig Diff Abs Diff 

Rel 

Diff 

Percent of the 

Population 

Residing in a 

different 

 house in US 

 1 year ago 

Average - Homeowner 13.9 12.6 Yes 1.3 9.6 

Average - Renter 20.3 17.4 Yes 2.9 14.5 

Economically Disadvantaged - 

Homeowner 17.8 18.1 No 0.3 1.6 

Economically Disadvantaged - Renter 18.4 16.3 No 2.1 11.7 

Ethnic Enclave - Homeowner 13.0 9.9 Yes 3.1 23.7 

Ethnic Enclave - Renter 14.1 12.9 No 1.2 8.5 

Single Unattached Mobiles 26.1 23.0 Yes 3.1 12.1 

Advantaged - Homeowner 10.7 9.5 Yes 1.2 10.9 

Source: U.S. Census Bureau: 2003 American Community Survey Special Tabulation 

http://www.census.gov/acs/www/data_documentation/data_main/ 

Other characteristics showing similar results to those seen using initial weights include percent unemployed, the 

many categories of poverty, and the percent in the some other race category for the race alone and in combination 

topic group. 

V.B.3. State Level 

Again a limited analysis is done for the four largest states: California, Texas, New York and Florida. The patterns 

observed using the initial selection weights and discussed above are seen again using modified final weights. 

In general there is little change in results when collection method estimates are calculated using modified final 

weights instead of initial selection weights. 

VI. Final Observations 

Only a small number of key characteristics repeatedly show a statistically significant collection method effect of 

a voluntary survey at the national, state, and segmentation group levels. 

For those key characteristics that show a statistically significant collection method effect using the initial 

selection weights, the modified weighting adjustments do not, in general, bring the voluntary and mandatory 

estimates closer together. The characteristics showing statistically significant collection method effects before 

adjustments for nonresponse and noncoverage also show statistically significant effects after these adjustments. 

There is no clear evidence of a collection method effect by race and other demographics at the national and state 

level. The few characteristics highlighted in the results section are those that appear to show the largest effect. 

These noted effects do not appear to be correlated in any way with the traditionally hard to interview 

populations such as ethnic or economically disadvantaged groups. Any potential effect of a voluntary survey 

appears to be more likely for the White or Asian populations. 

Results that we observe for educational attainment and income may be related. The voluntary collection 

method estimates for high school graduate or less are higher than the similar mandatory collection method 

estimates. The voluntary collection method estimates of household or family income of 200,000 or more are 

lower than the comparable mandatory collection method estimate. This may suggest that going to a voluntary 

collection method obtains a smaller proportion of respondents with higher levels of education and with high 

income. 
In summary for most analyses, the conclusions drawn from using estimates produced from a voluntary ACS 

would not be substantively different from those drawn from using estimates produced from a mandatory ACS. 

The more pronounced effects of a shift from a mandatory to a voluntary collection method might be the change 

in costs and reliability seen in the original analysis. 

References 

Institute of Mathematical Statistics, IMS Bulletin Volume 40, Issue 1: January/February 2011, ―The Canadian LongForm Census: Lessons for Statistics‖, page 6 -7. http://bulletin.imstat.org/archive-vol40-2011/#Issue01

U.S. Census Bureau (2003). Meeting 21st Century Demographic Data Needs—Implementing the American 

Community Survey. Report 3: Testing the Use of Voluntary Methods. 

http://www.census.gov/acs/www/library/by_series/implementing_the_acs/. 

U.S. Census Bureau (2004). Meeting 21st Century Demographic Data Needs—Implementing the American 

Community Survey. Report 11: Testing the Use of Voluntary Methods – Additional Results. 

http://www.census.gov/acs/www/library/by_series/implementing_the_acs/. 

 

 

U.S. Census Bureau, August (2008). 2010 Census Integrated Communication Campaign Plan, pages 33 – 43.

U.S. Census Bureau (2010a). Design and Methodology of the American Community Survey. 

http://www.census.gov/acs/www/methodology/methodology_main/

U.S. Census Bureau (2010b). 2003 Accuracy of the American Community Survey. 

http://www.census.gov/acs/www/data_documentation/documentation_main/

U.S. Census Bureau (2011a). 2009 American Community Survey Sampling Memorandum Series #ACS09-S-14, 

American Community Survey and Puerto Rico Community Survey: Unweighted Housing Unit Sample Disposition 

Counts. 

 

U.S. Census Bureau (2011b). 2011 American Community Survey Research and Evaluation Report Memorandum 

Series #ACS11-RER-08, Comparison of American Community Survey Voluntary versus Mandatory Estimates. 

http://www.census.gov/acs/www/Downloads/library/2011/2011_Navarro_01.pdf

Appendix: 2010 Census Integrated Communication Campaign Segmentation Group Summary 
Appendix 

2010 Census Integrated Communication Campaign Segmentation Group Summary 

Average – homeowners 

 35 percent of all occupied housing units in the U.S. 

 Second highest Census mail response 

 Large percent rural 

 Skews homeowners 

 Skews older 

Average – renters 

 15 percent of all occupied housing units in the U.S.

 Average Census mail response 

 Skews renter, densely populated 

 Urban 

 Skews younger 

Economically Disadvantaged - homeowners 

 6 percent of all occupied housing units in the U.S. 

 Largely urban, higher percent poverty, public 

assistance, unemployment, less than high school 

education 

 Skews older , homeowner 

 36 percent with children under 18 

Economically Disadvantaged - renters 

 3 percent of all occupied housing units in the U.S. 

 Lowest census mail response 

 Skews renters in urban multi-units 

 Highest poverty, public assistance, unemployment 

 1/3 speak language other than English at home 

 35 percent with children under 18 

Ethnic Enclave - homeowners 

 3 percent of all occupied housing units in the U.S. 

 Above average crowding, poverty, public 

assistance, unemployment, low education 

Ethnic Enclave – homeowners (con’t) 

 Less urban and densely populated, skews 

homeowner, stable and married households 

 50 percent with children under 18 

 43 percent foreign-born, 58 percent speak Spanish 

at home 

Ethnic Enclave - renters 

 2 percent of all occupied housing units in the U.S. 

 62 percent foreign-born, 54 percent speak Spanish, 

20 percent speak another language other than 

English at home 

 Higher poverty, unemployment, public assistance 

 Skewed renters in urban, crowded multi-units – 

most densely populated 

 44 percent with children under 18 

Single, Unattached, Mobiles 

 7 percent of all occupied housing units in the U.S. 

 Higher education 

 Highly mobile single renters in urban multi-units, 

densely populated 

 Racial and ethnic diversity 

 Skews younger and single 

Advantaged homeowners 

 28 percent of all occupied housing units in the U.S.

 Highest Census mail response 

 Stable, married homeowners 

 Least densely populated 

 Higher education 

 39 percent with children under 18 

Source: U.S. Census Bureau. (2008). 2010 Census Integrated Communications Campaign Plan