Hard to Get: Understanding Why People Take Our Surveys

Mina Muller, Randall K. Thomas, & Seth Messinger 

Ipsos Public Affairs

2021 FCSM Research & Policy Conference

November 2nd – November 4th, 2021

© 2021 Ipsos

‹#›

1

Current Study

Surveys that seek to reflect the views, attitudes, beliefs and opinions of a wide spectrum of respondents achieve a high degree of representativeness with the broader population, however that population is segmented. 

Greater representativeness results from including a wide variety of respondents and ensures that bias is minimized. Survey researchers face several challenges in maintaining a representative participant population. 

© 2021 Ipsos

‹#›

2

Current Study

The potential difficulty of recruiting and retaining participants from racial and ethnic groups that tend to be underrepresented is recognized by researchers. 

For instance, according to the National Science Foundation1 in health-related research the African Americans, Hispanic people, Indigenous People and Pacific Islanders tend to be underrepresented2. 

This pattern is repeated in market research and social research as well as in political polling. 

1https://www.nsf.gov/statistics/showpub.cfm?TopID=2&SubID=27

2 https://extramural-diversity.nih.gov/diversity-matters/underrepresented-groups

© 2021 Ipsos

‹#›

Current Study

There are a few areas in particular that can be examined to determine any differential impact on engagement and survey participation.  Some predictors of future participation include:

Reasons for participating

Survey topics

Negative prior experiences with surveys

Overall survey enjoyment

© 2021 Ipsos

‹#›

4

Current Study

Political polls may seem topical and highly relevant to every citizen in a specific population to the commissioners and designers but may be dismissed by respondents in environments of low trust or exhaustion.

Individuals who no longer ascribe legitimacy to the political process or to the choices presented in any given election may become disengaged from political issues.  Similar processes may occur for health-related topics or to marketing research. 

© 2021 Ipsos

‹#›

Similarly political polls 

5

Current Study

Our goal in this study is to examine if the survey experience of participants from different race-ethnicity groups differ in ways that could help explain their differential rates of participation and engagement in the survey research process.

We present two studies we’ve conducted where we asked about:

Online survey experience (number of surveys taken, number of panels joined)

Reason for participating in surveys

Interest in survey topics

Negative experiences with surveys

Overall survey enjoyment

© 2021 Ipsos

‹#›

Method

© 2021 Ipsos

‹#›

7

We conducted the web-based study with two different sample sources - opt-in non-probability sample from 17 different providers (ARF FoQ2) and a separate study using the KnowledgePanel, the largest U.S. probability-based sample.

Method - Sample
|                     | Study 1                | Study 2               |
|---------------------|------------------------|-----------------------|
| Sample Type         | Opt-in, non-probabilty | KP, probability-based |
| Sample N            | 71,385                 | 2,523                 |
| White, Non-Hispanic | 48,404                 | 1842                  |
| Black, Non-Hispanic | 8,257                  | 123                   |
| Hispanic            | 10,036                 | 504                   |

Data were weighted to CPS benchmarks for age, sex, education, ethnicity and income.

© 2021 Ipsos

‹#›

8

Results

© 2021 Ipsos

‹#›

| Sample 1 - Opt-inMean number of surveys                                                      | White, non-Hispanic | Black, non-Hispanic | Hispanic |
|----------------------------------------------------------------------------------------------|---------------------|---------------------|----------|
| About how many ONLINE surveys have you completed within the past 30 days?                    | 24.0                | 20.5                | 17.0     |
| How many ONLINE surveys in any form would you estimate that you complete in a TYPICAL month? | 27.6                | 23.4                | 19.9     |
| How many different research panels are you currently a member of?                            | 5.1                 | 4.9                 | 4.1      |
|                                                                                              |                     |                     |          |
| Sample 2 - Probability KP                                                                    | White, non-Hispanic | Black, non-Hispanic | Hispanic |
| About how many ONLINE surveys have you completed within the past 30 days?                    | 3.4                 | 2.3                 | 2.2      |
| How many ONLINE surveys in any form would you estimate that you complete in a TYPICAL month? | 4.8                 | 3.5                 | 3.4      |
| How many different research panels are you currently a member of?                            | 1.4                 | 1.2                 | 1.2      |

Results – Online Activity by Sample Type & Ethnicity

As many have noted, opt-in sample participants take a lot more surveys than a probability-based sample.

In spite of the differences in sample types, we see that Black and Hispanic participants tend to be less active in online survey research for both sample types.

© 2021 Ipsos

‹#›

. 

10

Results – Survey Motivation – Opt-in
| How motivating are the following in getting you to participate in a survey? (4 category - Not motivating - Extremely motivating) | White, non-Hispanic | Black, non-Hispanic | Hispanic |
|----------------------------------------------------------------------------------------------------------------------------------|---------------------|---------------------|----------|
| The survey has direct rewards (cash or points)                                                                                   | 3.3                 | 3.4                 | 3.3      |
| The survey is about an interesting topic                                                                                         | 2.8                 | 2.9                 | 2.9      |
| The survey allows me to express my opinion                                                                                       | 2.7                 | 2.8                 | 2.8      |
| The survey allows me to think more about a topic                                                                                 | 2.4                 | 2.6                 | 2.6      |
| The survey makes me feel part of a community                                                                                     | 2.1                 | 2.4                 | 2.4      |
| The survey has a sweepstakes                                                                                                     | 2.1                 | 2.4                 | 2.3      |
| The survey allows me to find out how other people think or what they do                                                          | 2.0                 | 2.4                 | 2.3      |

Opt-in Sample

While Black and Hispanic participants have somewhat higher ratings than White participants, the order of the items are often in parallel. 

© 2021 Ipsos

‹#›

11

Results – Survey Motivation - KP
| How important are the following in getting you to take an online survey?    (4 category - Not important - Extremely important) | White, non-Hispanic | Black, non-Hispanic | Hispanic |
|--------------------------------------------------------------------------------------------------------------------------------|---------------------|---------------------|----------|
| Allows me to express my opinions                                                                                               | 2.7                 | 3.0                 | 3.1      |
| Offers cash or reward points                                                                                                   | 2.9                 | 3.0                 | 3.0      |
| Could improve products/services I might use                                                                                    | 2.7                 | 2.9                 | 3.0      |
| Is being done by a company/organization I trust                                                                                | 2.8                 | 2.8                 | 2.9      |
| Helps others in their research                                                                                                 | 2.7                 | 2.7                 | 3.0      |
| Is about an interesting topic                                                                                                  | 2.6                 | 2.8                 | 2.9      |
| Makes me think about things I would not normally think about                                                                   | 2.2                 | 2.7                 | 2.7      |
| Allows me to think more about a topic                                                                                          | 2.3                 | 2.5                 | 2.8      |
| Is entertaining to take                                                                                                        | 2.2                 | 2.6                 | 2.5      |
| Makes me feel part of a community                                                                                              | 2.0                 | 2.4                 | 2.8      |
| Allows me to find out how other people think or what they do                                                                   | 1.9                 | 2.3                 | 2.6      |
| Has a sweepstakes (entered into a drawing)                                                                                     | 1.7                 | 2.1                 | 2.1      |

KP Probability Sample

Again, Black and Hispanic participants have somewhat higher ratings than White participants, but also the order of the items are often in parallel.  The opportunity to express opinions and obtain case or rewards were both in the top 3 for participants from both sample types. 

© 2021 Ipsos

‹#›

. 

12

Results – Topic Interest – Opt-in

Opt-in Sample

Generally, a similar pattern of topic interest from highest to lowest across of race-ethnicity groups
| How much do you like doing surveys on the following topics? (3 category scale - Do not like - Like a lot) | White, non-Hispanic | Black, non-Hispanic | Hispanic |
|-----------------------------------------------------------------------------------------------------------|---------------------|---------------------|----------|
| Surveys that ask about a variety of different topics                                                      | 2.5                 | 2.5                 | 2.5      |
| Entertainment (movies, TV, games, etc.)                                                                   | 2.3                 | 2.5                 | 2.5      |
| Household goods                                                                                           | 2.4                 | 2.5                 | 2.4      |
| Technology-oriented                                                                                       | 2.2                 | 2.3                 | 2.3      |
| Advertising evaluations                                                                                   | 2.2                 | 2.3                 | 2.3      |
| Health-related issues                                                                                     | 2.2                 | 2.2                 | 2.2      |
| Political issues                                                                                          | 2.1                 | 2.0                 | 2.0      |
| Automotive, vehicle-related                                                                               | 2.0                 | 2.0                 | 2.1      |

© 2021 Ipsos

‹#›

13

Results – Topic Interest - KP
| How much would you like to do an online survey on the following topics? (3 category scale - Do not like - Like a lot) | White, non-Hispanic | Black, non-Hispanic | Hispanic |
|-----------------------------------------------------------------------------------------------------------------------|---------------------|---------------------|----------|
| Health-related issues                                                                                                 | 2.1                 | 2.4                 | 2.4      |
| One that asks about many different topics                                                                             | 2.2                 | 2.3                 | 2.3      |
| Household goods                                                                                                       | 2.2                 | 2.4                 | 2.2      |
| Entertainment (movies, TV, games, etc.)'                                                                              | 2.0                 | 2.4                 | 2.2      |
| Technology                                                                                                            | 1.9                 | 2.0                 | 2.2      |
| Advertising                                                                                                           | 1.8                 | 1.9                 | 2.0      |
| Financial services and products                                                                                       | 1.7                 | 1.8                 | 1.9      |
| Automotive, vehicle-related'                                                                                          | 1.8                 | 1.7                 | 1.8      |
| Political issues                                                                                                      | 2.0                 | 1.6                 | 1.7      |

KP Probability Sample

Slight evidence that Black and Hispanic participants have somewhat higher liking for topics than White participants, but also the order of the items are often in parallel across race-ethnicity groups.  

An interesting difference in ratings of political issues, though, with Black and Hispanic both indicating lower interest than White participants.

© 2021 Ipsos

‹#›

14

Results – Survey Laments – Opt-in
| How often did you experience the following in online surveys in the PAST YEAR? (4 category - Never to Very often) | White, non-Hispanic | Black, non-Hispanic | Hispanic |
|-------------------------------------------------------------------------------------------------------------------|---------------------|---------------------|----------|
| The survey took much longer to complete than expected.                                                            | 3.2                 | 3.3                 | 3.2      |
| The rewards were insufficient for the effort involved.                                                            | 3.2                 | 3.1                 | 3.2      |
| The survey presented items that were not relevant to me.                                                          | 3.0                 | 2.9                 | 3.0      |
| The questions did not have good responses for me.                                                                 | 2.8                 | 2.7                 | 2.8      |
| I felt I wasted my time answering questions.                                                                      | 2.7                 | 2.6                 | 2.6      |
| The questions were not clearly worded.                                                                            | 2.4                 | 2.4                 | 2.5      |
| The questions were complex and difficult to answer.                                                               | 2.3                 | 2.3                 | 2.4      |
| I chose answers that required the fewest responses.                                                               | 2.1                 | 2.4                 | 2.4      |
| I had problems with a video presentation in the survey.                                                           | 1.9                 | 2.0                 | 2.1      |
| I chose answers that would let me finish faster.                                                                  | 1.9                 | 2.1                 | 2.1      |

Opt-in Sample

Not much difference in the frequency of negative experiences between race-ethnicity groups. Order of laments about the same from highest to lowest in all groups.

© 2021 Ipsos

‹#›

15

Results – Survey Laments - KP
| How often did you experience the following in online surveys in the PAST YEAR? (4 category - Never to Very often) | White, non-Hispanic | Black, non-Hispanic | Hispanic |
|-------------------------------------------------------------------------------------------------------------------|---------------------|---------------------|----------|
| Not enough money/points for the effort involved.                                                                  | 2.2                 | 2.6                 | 2.3      |
| The survey took much longer to complete than expected.                                                            | 2.3                 | 2.6                 | 2.1      |
| The survey included questions that were not relevant to me.                                                       | 2.4                 | 2.2                 | 2.1      |
| The questions did not have good responses for me.                                                                 | 2.1                 | 2.1                 | 2.0      |
| The topic/questions were boring.                                                                                  | 2.1                 | 2.1                 | 1.9      |
| The questions were not clearly worded.                                                                            | 1.8                 | 1.8                 | 1.8      |
| I felt I wasted my time answering questions.                                                                      | 1.9                 | 1.7                 | 1.6      |
| I chose the fewest answers possible.                                                                              | 1.5                 | 1.7                 | 1.8      |
| The questions were complex and difficult to answer.                                                               | 1.7                 | 1.6                 | 1.7      |
| I found it hard to navigate through the survey.                                                                   | 1.5                 | 1.5                 | 1.5      |
| I chose answers that would let me finish faster.                                                                  | 1.4                 | 1.4                 | 1.6      |
| I had problems with a video presentation in the survey.                                                           | 1.4                 | 1.4                 | 1.4      |
| The survey did not look right on my screen.                                                                       | 1.3                 | 1.3                 | 1.3      |

KP Probability Sample

The top 3 laments for opt-in sample were the top 3 laments for KP sample, although the frequency of occurrence were much lower for KP.  One interesting finding is that Black participants experienced a slightly higher frequency of a negative experience with length and reward for effort.

© 2021 Ipsos

‹#›

16

Results – Overall 
| Excluding political polls, what has been your general experience when participating in online surveys? (5 category from Never enjoyable to Always enjoyable) | White, non-Hispanic | Black, non-Hispanic | Hispanic |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|---------------------|----------|
|                                                                                                                                                              |                     |                     |          |
| Opt-in non-probability sample                                                                                                                                | 3.7                 | 3.8                 | 3.8      |
|                                                                                                                                                              |                     |                     |          |
| KP probability sample                                                                                                                                        | 3.0                 | 3.1                 | 2.9      |

Finally, we asked a question about the respondents’ enjoyment of taking online surveys. While the opt-in participants who had much higher online survey activity enjoyed surveys more, there were no significant difference between race-ethnicity groups.

© 2021 Ipsos

‹#›

17

Discussion

© 2021 Ipsos

‹#›

Summary

Overall, we found that opt-in non-probability samples participated in online surveys at a much higher rate than the KP probability sample.  However,  we found that Black and Hispanic participants were participating in both sample types at lower rates than White participants.  It is this lower rate of engagement and participation that may reduce the impact of the voices of people of color.

Of those who do participate, there are very few differences between the race-ethnicity groups in reasons for participation (reasons), topics of interest, or negative experiences (laments) with surveys. 

Finally, we did not find significant differences between the race-ethnicity groups in the enjoyment they experienced in taking surveys.

19

© 2021 Ipsos

‹#›

19

Conclusions

In spite of the great similarities in survey experience of participants from the different race-ethnicity groups, we should note that in our experience in recruitment, people of color tend to be harder to recruit, harder to motivate to take a survey, and higher rates of attrition for both probability and non-probability samples.  In order to better represent people of color, we believe we need to redouble our efforts to better understand the entire survey and panel engagement experience for people of color in order to ensure that these key groups bring their voices to the table.

Further research exploring how these variables predict survey participation using regression models.

© 2021 Ipsos

‹#›

20

Mina Muller

Director at Ipsos Public Affairs 

Mina.Muller@Ipsos.com

Thank you!

© 2021 Ipsos

‹#›

21

# image2.jpg

# image3.png

# image1.png