Measuring Intent to Participate and Participation in the 2010 Census and Their Correlates and Trends: Comparisons of RDD Telephone and Non-probability Sample Internet Survey Data

This study explored whether probability sample telephone survey data and data from nonprobability sample Internet surveys yielded similar results regarding intent to complete the 2010 Census form and actual completion of the form, the correlates of these variables, and changes in these variables and their correlates over time. Using data collected between January and April, 2010, the telephone samples were more demographically representative of the nation’s population than were the Internet samples after post-stratification. Furthermore, the distributions of opinions and behaviors were often significantly and substantially different across the two data streams, as were relations between the variables and changes over time in the variables. Thus, research conclusions would often be different depending on which data stream was used. Because the telephone data collection methodology rests on well-established theory of probability sampling and produced the most demographically representative samples, the substantive results yielded by these data may also be more accurate than the substantive results generated with the non-probability sample Internet data. COMPARING RDD & NON-PROBABILITY SAMPLES 3 In 2009 and 2010, the U.S. Census Bureau commissioned the collection of two parallel data streams to monitor public reactions to the 2010 Census. In both a random digit dial (RDD) telephone survey conducted by the Gallup Organization and a series of nonprobability Internet surveys administered by E-Rewards, respondents reported their intent to complete the Census form, whether they had completed it, a variety of purported predictors of intention and behavior, and demographic characteristics. These data were collected to track changes over time and to identify opinions that might enhance or reduce Census form completion. This paper explores whether the results generated by the telephone and Internet data streams are equivalent. If the two data streams support identical conclusions about distributions of, changes over time in, and relations between the variables measured, then future Census Bureau efforts can choose to employ just one of these methods, perhaps the one that generates the most cases at the least cost per case. But if the data streams yielded different results, the Bureau must decide whether in the future, it makes sense to collect both or just one, and this decision can be facilitated by knowledge about which data stream was most accurate in describing the nation’s population. This paper outlines the results of our empirical analyses of these issues. Specifically, we report answers to five questions: 1. Did the two data streams differ in their degree of demographic representativeness of the nation’s adult population? 2. Did the two data streams produce similar distributions of opinions and behaviors? 3. Were the relations between variables within the two data streams similar? 4. Were predictors of intent to complete the Census form and completion of the Census form similar across the two data streams? 5. Did measurements of opinions and behaviors in the two data streams yield similar patterns of change over time? To answer each of these questions, the results of an analysis using the probability sample telephone survey data were compared with an identical analysis using the non-probability COMPARING RDD & NON-PROBABILITY SAMPLES 4 sample Internet survey data. We compared both data streams with known population benchmarks to determine which data collection method yielded the most representative samples. We begin below by describing the methods of data collection that were employed to yield each data stream. Then, we describe the measures administered and the analytic strategies employed. Finally, we report our findings and describe their implications.