Centers for Disease Control and Prevention

National Center for Chronic Disease Prevention and Health Promotion

Comparison of Paper-and-Pencil Versus 

Tablet Administration of the 

2021 National Youth Risk Behavior Survey 

(YRBS)

October 25, 2023

Jingjing Li PhD, MD, MPH

Division of Adolescent and School Health
Disclaimer: The findings and conclusions in the presentation are those of the authors and do not necessarily represent 

the official position of the Centers for Disease Control and Prevention.

2
Table of Contents

• BACKGROUND

• METHODS

• RESULTS

• LIMITATIONS

• CONCLUSIONS

3
Background

• Overview of Youth Risk Surveillance System (YRBSS)

• Largest public health behavior surveillance system in the 

U.S. for high school students.

• Monitors health behaviors and experiences via the Youth 

Risk Behavior Survey (YRBS).

• Items include: 

• Student demographics

• Youth health behaviors (e.g., substance use, physical activity)

• Student experiences (e.g., school connectedness, community 

violence exposure)

4
Background

• YRBSS Structure and Data Collection

• Comprises: national, state, school district, territorial, and 

tribal surveys.

• Traditionally administered using paper-and-pencil 

instrument (PAPI) since 1991.

• CDC's Public Health Data Modernization Initiative:

• Aim: Modernize surveillance infrastructure.

• Goal: Achieve near-real-time data collection. 

• Faster and efficient data collection.

• Timely data reporting for decision-making.

• Reduced burden on student respondents.

5
Background

• Transitioning to Electronic YRBS Surveys

• Electronic survey administration found feasible and advantageous over 

PAPI.

• Some state and local YRBSs started electronic data collection in 2019

• Study by Bryan et al (2019):

• Compared: Electronic (7 sites) vs. PAPI (38 sites).

• Findings:

• Electronic data more complete.

• Health behavior prevalence estimates consistent across modes.

• Limitations:

• Lacking random assignment to survey mode.

• Demographic differences observed between electronic and PAPI survey 

respondents

6

Bryan LN, Smith‐Grant J, Brener N, et al. Electronic versus paper and pencil survey administration mode comparison: 2019 youth 

risk behavior survey. Journal of School Health 2022.
Objectives for the 2021 YRBS tablet study 

• Compare electronic tablet administration vs. PAPI in the 

same schools: 

• Prevalence estimates for youth behaviors and experiences.

• Data missingness

• Understand the influence of tablet administration on 

students' survey responses.

7
Methods- 2021 National YRBS

Sampling Methodology

• Three-stage, cluster sample design for national representation

• U.S. students grades 9-12 across 50 states & D.C.

• Stage 1: Primary sampling units (PSUs) consisting of large-sized 

counties or groups of smaller, adjacent ones

• Stage 2: In PSU, 3 large (≥28 students/grade) and 1 small (<28 

students/grade) were selected with probability proportional to 

enrollment size

• Stage 3: Classes were selected randomly

8
Methods - 2021 YRBS Tablet Study 

Participation & sampling: 57 high schools

• All 57 schools agreeing to participate both the PAPI survey and the 

tablet survey

• Staff generated 2 or 3 unique random numbers per grade, the first 

number(s) were assigned to classes for the PAPI survey, and the 

remaining number was assigned to the class for the tablet component

• 4684 students (PAPI) & 3645 (tablets)

9
Methods

Data Collection Procedures

• Data Protection:

• Voluntary, anonymous participation

• Local parental permission sought before survey

• CDC's Institutional Review Board-approved protocol

• PAPI Survey:

• Computer-scannable booklets

• Response checks by data collectors

• Scanning through iNSIGHT 70-5400140 optical scanner

• Tablet Survey:

• Pre-programmed tablets with random access codes

• Survey saved locally and synced to a central MS SQL Server database post-collection

10
2021 YRBS 

TABLET STUDY 

11
Sample Questions • Student uses their finger to 

select their response and 

then the “>” button to 

advance

• User experience enhanced 

by including images of 

specific products

• Progress bar shows 

completion level

12
Methods

2021 YRBS Questionnaire:

• Total Items: 99

• Demographic characteristics: 9 items.

• Health topics: 89 items (injuries, mental health, tobacco, alcohol, sexual 

behaviors, diet, physical activity, COVID-19, etc.).

• One item assessed level of English fluency.

• Survey Mode:

• PAPI: Standard text-only.

• Tablet: Text with color images for 13 items.

• No skip patterns; every student saw every question in the same order.

13
Methods

Demographic comparison:

• PAPI vs. Tablet using chi-square tests.

Effect of mode on the prevalence of 89 health behavior & experience items.

• Logistic regression 

• Controlled for sex, race/ethnicity, and grade.

Missingness comparison:

• Compared item-level missingness between modes using Z-tests.

Tool & significance:

• All analyses were conducted using SAS 9.4.

• Due to numerous comparisons, p value <.01 was used for statistically 

significant.

14
RESULTS

¡ Demographics

¡ No differences between PAPI and tablet 

modes

15

Characteristics

Study Sample 

(%)

Unwt N=8329

PAPI (%)

Unwt 

n=4684

Tablet (%)

Unwt 

n=3645

Chi-square Test

P value

Sex

Female 49.4 50.0 48.6 0.214

Male 50.6 50.0 51.4

Race/ethnicity

White # 49.8 49.5 50.3 0.073

Black # 18.4 18.6 18.2

Hispanic/Latino 18.0 18.8 17.0

Asian # 6.0 5.9 6.2

Other # 7.7 7.1 8.4

Grade

9 24.6 24.3 25.0 0.551

10 25.7 26.3 24.9

11 25.3 25.0 25.5

12 24.5 24.4 24.5

Sexual identity

Heterosexual 75.4 75.1 75.7 0.798

Lesbian, gay, or 

bisexual

15.4 15.6 15.2

Other/questioning 9.2 9.4 9.1

Demographic characteristics of students in the study sample, overall and by mode, 2021 YRBS tablet study
¡ Missingness:

¡ Less missingness in 

tablet mode for sex and 

race/ethnicity

16

Characteristics

Overall 

Missingness 

(n, col %)

PAPI 

Missingness 

(n,%)

Tablet

Missingness 

(n,%)

Z score 

Test 

P value

Sex 100 (1.2) 76 (1.6) 24 (0.7) <0.001

Race/ethnicity 169 (2.0) 118 (2.5) 51 (1.4) <0.001

Grade 74 (0.9) 49 (1.1) 25 (0.7) 0.082

Sexual identity 435 (5.2) 250 (5.3) 185 (5.1) 0.596

Demographic missingness in the study sample, overall and by mode, 

2021 YRBS tablet study

p level at 0.01, two-tailed. P value is based on Z test of two proportions. 
Results

Prevalence (proportions) of outcome variables 

• Overall:

• 6 out of 89 variables (6.7%) showed significant prevalence 

(proportion) differences between modes.

• Lower Proportions in Tablet vs. PAPI Mode:

1. Tried to quit using all tobacco products 

2. Did not eat breakfast

3. Spent 3+ hours/day on screen time 

• Higher Proportions in Tablet vs. PAPI Mode:

1. Experienced sexual violence in the past year

2. Used synthetic marijuana

3. Offered/sold/given illegal drugs at school 

17
SUMMARY OF DIFFERENCES IN PREVALENCE 

(PROPORTIONS), BY ADMINISTRATION MODE AND 

TOPIC, 2021 YRBS TABLET STUDY

18

Topic Total number of 

items

Number of significantly different 

proportions between modes

PAPI > tablet Tablet > PAPI

Unintentional injuries and 

violence 17 1 0 1

Mental health and suiciderelated behaviors 5 0 – –

Tobacco use 10 1 1 0

Alcohol and other drug use 19 2 0 2

Sexual behaviors 7 0 – –

Dietary behaviors 11 1 1 0

Physical activity 6 1 1 0

COVID-19 and other healthrelated topics 14 0 – –

Total 89 6 3 3
Results

Missingness differences by mode

• Overall Results:

• 33.7% (30 of 89 items) showed different missingness between 

modes.

• Higher Missingness in PAPI Mode:

• 10 items:

• 5 items: unintentional injuries and violence.

• 3 items: mental health and suicide behaviors.

• 2 items: dietary behaviors.

• Higher Missingness in Tablet Mode:

• 20 items:

• 2 items: unintentional injuries and violence.

• 5 items: tobacco product use.

• 5 items: alcohol and drug use.

• 7 items: sexual behaviors.

• 1 item: other health topics.

19
SUMMARY OF DIFFERENCES IN MISSINGNESS, BY 

ADMINISTRATION MODE AND TOPIC, 2021 YRBS 

TABLET STUDY

20

Topic Total number of 

variables

No. of variables with significant 

differences in missingness: PAPI vs. 

tablet

No. of variables with 

greater missingness in 

PAPI

No. of variables with 

greater missingness in 

tablet

Unintentional injuries and 

violence 17 7 5 2

Mental health and suicide-related 

behaviors 5 3 3 0

Tobacco use 10 5 0 5

Alcohol and other drug use 19 5 0 5

Sexual behaviors 7 7 0 7

Dietary behaviors 11 2 2 0

Physical activity 6 0 – –

COVID-19 and other health-related 

topics 14 1 0 1

Total 89 30 10 20
Missingness pattern by demographics 

• For these 30 variables that were significantly different in 

missingness: 

• Further Chi-square tests were conducted to analyze 

differences in missingness by demographic characteristics.

• Supplement results revealed that only two variables 

showed differences.

21
22

Chi-square Test

Number of missing observations in PAPI vs. Tablet

By Sex By Grade By Race/ethnicity By Sexual Identity

P value γ

Did not always wear a seat belt 0.8031 0.3100 0.4626 0.3598

Drove after drinking alcohol 0.8256 0.0106 0.6574 0.5574

Texted or e-mailed while driving 0.5550 0.1108 0.5851 0.6821

Ever saw someone get attacked, beaten, or shot in neighborhood 0.6344 0.3674 0.5805 0.1527

Ever forced to have sexual intercourse 0.5704 0.2157 0.0928 0.2551

Were bullied on school property 0.1534 0.8546 0.6955 0.6282

Were electronically bullied 0.2766 0.8885 0.4905 0.3084

Felt sad or hopeless 0.3145 0.5058 0.9228 0.8868

Seriously considered attempting suicide 0.0778 0.8342 0.3713 0.8799

Made a suicide plan 0.0097 0.6067 0.8846 0.8272

Ever tried cigarette smoking& 0.4419 0.2581 0.3751 0.6251

First tried cigarette smoking before age 13 years& 0.1313 0.1560 0.1807 0.5304

Currently smoked cigarettes& 0.5450 0.9267 0.8001 0.5041

Smoked more than 10 cigarettes per day& 0.1623 0.4134 0.0422 0.9733

All tobacco product cessation 0.2037 0.5186 0.0918 0.2947

Had their first drink of alcohol before age 13 years 0.8034 0.8752 0.3108 0.5267

Currently drank alcohol 0.1334 0.3946 0.1414 0.6255

Largest number of drinks 0.1888 0.9960 0.1863 0.3850

Source of alcohol 0.3584 0.3077 0.0877 0.0941

Ever used marijuana 0.9317 0.0669 0.1043 0.7484

Ever had sexual intercourse 0.9882 0.9984 0.0498 0.6272

Had first sexual intercourse before age 13 years 0.7081 0.9504 0.0613 0.8485

Had more than 4 sex partners during lifetime 0.4721 0.8333 0.0596 0.9972

Currently sexually active 0.9634 0.8652 0.0598 0.7915

Alcohol/drugs and sex# 0.9412 0.8116 0.1174 0.8807

Condom use# 0.5941 0.6843 0.0625 0.7931

Birth control pill use&# 0.3492 0.0827 0.5495 0.2194

Did not drink milk 0.9135 0.0032 0.8749 0.9533

Did not drink plain water 0.5491 0.0657 0.8662 0.0375

Had a sunburn 0.4550 0.0846 0.3589 0.2542

Supplement results
Limitations

• Classrooms, not individual students, randomly assigned to survey modes.

• Differences may exist in unobserved demographics (e.g., socioeconomic 

status).

• Results are not nationally representative.

23
Conclusions

24

 Transition from PAPI to electronic surveys aligns with CDC's 

Data Modernization Initiative.

 Electronic administration (via tablets) maintains data quality 

across student behaviors.

 Findings support electronic surveys in U.S. schools

 Continued investment and improvements in public health 

monitoring are needed.
Centers for Disease Control and Prevention

National Center for Chronic Disease Prevention and Health Promotion

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

THANK YOU

Jingjing Li

ppy8@cdc.gov

Division of Adolescent and School Health
QUESTIONS

26