Recruiting a probability sample of 18 year olds for a longitudinal study on interpersonal violence

David Cantor and Reanne Townsend

Westat

November 3, 2021

1

Background

Young adults are a population of interest for many different topics, including victimization, smoking, drug use and other risky behaviors.

They are also difficult to survey because they are highly mobile, are hard to contact and generally not highly motivated to participate in surveys. 

To study many outcomes of interest, a large sample is required.

For this study: Research questions include risk of interpersonal violence for those in post secondary school and those that are not in school.  

Start with a cohort of 18 year olds and follow them for 4-6 years.

2

October 23, 2021

2

National Study of Young Adults

Longitudinal pilot study sponsored by the National Institute of Justice

Collaborative effort between Westat, New York University, and Dr. Bonnie Fisher (University of Cincinnati) 

Push-to-web design to recruit young adults into the study

Longitudinal studies of young adults on other topics have found promising results with push-to-web designs (e.g. Cantrell et al 2018). 

Even with push-to-web design, finding and contacting 18 year olds can be very expensive (2% of population)

October 23, 2021

3

3

Research Questions

Can a general population of 18 year olds be successfully recruited for a panel study using a push to web design?

Overall response and retention rates

What are the social and demographic profiles?

What level of incentives work to maximize response rate?

Does gamefication work to recruit this population?

4

Sample design

Started with an address based sample (ABS) of all households in the US.

National sample frame was supplemented by a list of high school seniors purchased from ASL marketing

ASL list included date of birth and gender

Excludes non-students (e.g. dropouts)

The ASL list was matched to a general population ABS sample and used to stratify the sample.

October 23, 2021

5

5

Sample design

October 23, 2021

6
|                   | General population sample | Percent of general population sample | Final sample of addresses | Percent sampled |
|-------------------|---------------------------|--------------------------------------|---------------------------|-----------------|
| Matched to list   | 15,510                    | 1.5%                                 | 6,549                     | 42.2%           |
| Unmatched to list | 1,005,877                 | 98.5%                                | 7,500                     | 0.7%            |
| Total sample      | 1,021,387                 | 100.0%                               | 14,049                    | 1.4%            |

Heavy oversample of matched group.

Oversample was done as part of Pilot to maximize returns for later analysis

6

Recruitment process overview

Invitation mailed to sampled households in the Spring 2019

Sampled person goes to study website and enters access code

Person enters date of birth to confirm age, and completes the recruitment survey

Nonresponders are given option to complete survey by mail on the fourth contact

After submitting the web survey, respondents land on a personalized study homepage, with study info and a dashboard tracking study activities.

Starting in Fall 2019, qualified respondents are invited to subsequent data collections (wave 1, micro-assessments, wave 2)

October 23, 2021

7

7

Experimemts
|                                         | Recruitment Survey |            | Wave 1 | Wave 2 | Micro-Assessments | Gamefication |
|-----------------------------------------|--------------------|------------|--------|--------|-------------------|--------------|
|                                         | Survey             | Early Bird |        |        |                   |              |
| $20 vs. $30                             | x                  |            |        |        |                   |              |
| $0 vs. $10                              |                    | x          |        |        |                   |              |
| $40 vs. $60                             |                    |            | x      | x      |                   |              |
| (No Micro) vs. (Biweekly) vs. (Monthly) |                    |            |        |        | x                 |              |
| (No Game) vs. (Game) vs (Game with $)   |                    |            |        |        |                   | x            |

8

Study website

October 23, 2021

9

9

Recruitment results

October 23, 2021

10
| Group             | Initial sample | Recruitment responses | Yield rate |
|-------------------|----------------|-----------------------|------------|
| Matched           | 6,549          | 1,759                 | 26.9%      |
| Unmatched         | 7,500          | 163                   | 2.2%       |
| Total ABS records | 14,049         | 1,922                 | 13.7%      |

Highest yield rate for matched sample

Unmatched rate is close to percentage of 18 year olds in the population

10

Estimated percent of 18 year olds responding to recruitment, wave 1 and wave 2

Approximated a response rate by using the base-weighted totals for 18 year olds and divided it by the number of 18 year olds in the country.

11
|                    | Base Weighted Number of 18 year olds | Response ratio |
|--------------------|--------------------------------------|----------------|
| Recruitment Survey | 2,336,198                            | 55%            |
| Wave 1             | 1,667,594                            | 39%            |
| Wave 2             | 1,472,485                            | 34%            |

Estimated number of 18 Year represented and response ratio

+ Response ratio = Base weighted total divided by total number of 18 year olds in the US.

Socio-Demographic representation of recruitment and Wave 1 respondents (base-weighted)

12

Percent

Retention of respondents by wave by School attendance in last 3 months at time of recruitment (base-weighted)

13

Percent of Wave 1

Retention of respondents by wave by whether ever been victim of stalking at time of recruitment (base-weighted)

14

Wave 2 retention rates by micro-assessment experimental condition by whether or not base weights are applied

15

Results of incentive experiments

None of the incentive experiments found a difference between incentive conditions

Some suggestion that Wave 1 experiment result in higher rate, but not significant.

16
|             | Yield Rates |
|-------------|-------------|
| Recruitment |             |
| $20         | 12.6        |
| $30         | 13.4        |
| Early Bird  |             |
| $0          | 13.3        |
| $10         | 12.7        |
| Wave 1      |             |
| $40         | 61.7        |
| $60         | 67.4        |
| Wave 2      |             |
| $40         | 90.4        |
| $60         | 91.4        |

16

Results of experiment with gamefication 

17

Percent Responding

Avg # Completed

Summary

Overall recruitment was successful.  

Approximation of response rate was respectable (55% and 39%).

There was good representation of key populations, with exceptions

Retention rate between waves was good

Young adults respond well to incentives.  Lower amounts work well.

Basic points and badges related to gamefication strategies did not significantly increase participation

Micro-assessments had very high response rate.  They did not affect the response rate at wave 2.

18

# image2.png

# image3.png

# image7.png

# image8.png

# Microsoft_Excel_Worksheet.xlsx

# image1.png

# image6.png