You didn’t answer our survey, but what about this text? 

Converting hard-to-reach respondents through text 

messaging

Maura Spiegelman, National Center for Education Statistics

Allison Zotti, U.S. Census Bureau

2023 FCSM Research and Policy Conference

October 25, 2023

This presentation is intended to promote ideas. The views expressed are part of ongoing research and do not necessarily reflect the position of the 

National Center for Education Statistics, the U.S. Department of Education, or the U.S. Census Bureau.
National Center for 

Education Statistics

Background

2

• “Hard-to-reach” individuals may be difficult to identify, sample, contact, or persuade

• For a longitudinal survey, can we persuade participants to respond by:

– Changing the contact mode?

– Offering a new response mode?

– Reducing survey length?

• Can we replace mailed questionnaires with text-to-web links OR an abbreviated text 

message survey?
National Center for 

Education Statistics

Teacher Follow-up Survey (TFS) to the

National Teacher and Principal Survey (NTPS)

3

• TFS is a longitudinal follow-up (1 year later) with public and private 

school K-12 teachers who completed the baseline NTPS survey during the 

previous school year

• Approximately 20 minutes to complete the TFS 

(and 40 minutes to complete the NTPS)

• NTPS collects personal and work contact information during base year

– Respondents asked to provide home mailing address, e-mail address, and 

cellphone number in the baseline survey, as well as affirmative consent to receive 

follow-up communication by text message
National Center for 

Education Statistics

4

1/13/22-4/18/22 6 e-mailed web survey invitations, 3 mailed web survey invitations, 1 mailed paper survey. 

Experiment starts later in April

Time period Control Text-to-web Text questions

4/26/22-4/28/22 2nd mailed paper survey Welcome text with link to 

survey

Welcome text with SMS survey 

questions

5/2/22 Reminder e-mail

5/9/22- 6/3/22 CATI collection

5/17/22-5/19/22 3rd mail paper survey Texted link to web survey Texted SMS survey questions

5/23/22 Reminder e-mail

5/31/22 End of experimental treatments; beginning of “last ditch” strategies

6/1/22 Welcome text with link to 

survey

6/6/22 Texted SMS survey questions

6/9/22 Reminder e-mail

Data collection strategy
National Center for 

Education Statistics

Methods

5

Teachers who provided cell phone 

number and consent assigned to three 

groups during the primary data collection 

period (about n=1,000 per group): • Control: no text reminders • Text-to-web: received text message with 

web survey link in place of later mail 

reminders

• Text questions: received text message 

survey questions in place of later mail 

reminders

In the final stage of data collection, Control 

and Text-to-web nonrespondents received 

“last ditch” text message communication
National Center for 

Education Statistics

6

Welcome 

message

Do you still 

teach any 

regularly-

scheduled 

classes in 

any of 

grades K-

12?

Are you 

teaching at 

the same 

school as 

the 2020-21 

school 

year?

END

END

Are you 

working in 

K-12 

education, 

but not 

currently 

teaching 

any 

classes?

END

Are you 

retired?

END

Are you still 

teaching but 

currently on 

leave?

Do you plan 

to return to 

teaching this 

school 

year? Do you plan 

to return to 

teaching 

next year?

Will you be 

teaching at 

the same 

school as 

you were in 

the 2020-21 

school 

year?

END

END

END

Are you still 

teaching at 

the same 

school as 

the 2020-21 

school 

year?

END

END

END

Y

N

Y

Y

Y

Y

YY

Y

N

N

N

NN

N

N

YN

Text questions
National Center for 

Education Statistics

7

For hard-to-reach individuals, can we replace a mailed 

paper questionnaire with text messages?
National Center for 

Education Statistics

8

Data collection strategy

1/13/22-4/18/22 6 e-mailed web survey invitations, 3 mailed web survey invitations, 1 mailed paper survey. 

Experiment starts later in April

Time period Control Text-to-web Text questions

4/26/22-4/28/22 2nd mailed paper survey Welcome text with link to 

survey

Welcome text with SMS survey 

questions

5/2/22 Reminder e-mail

5/9/22- 6/3/22 CATI collection

5/17/22-5/19/22 3rd mail paper survey Texted link to web survey Texted SMS survey questions

5/23/22 Reminder e-mail

5/31/22 End of experimental treatments; beginning of “last ditch” strategies

6/1/22 Welcome text with link to 

survey

6/6/22 Texted SMS survey questions

6/9/22 Reminder e-mail
National Center for 

Education Statistics

9

Replacing mailed questionnaires with text-to-web contacts did not affect 

response rates, nor did replacing mailed questionnaires with a separate text 

message survey.

Control Text-to-web Text questions

Response rate: Full survey (s.e.) 32.1 (4.21) 30.2 (3.74) 30.7 (4.51)

Among nonrespondents as of 3/31/22, response rates as of 5/31/22 * Significant difference from Control group with α = .05 ~ Significant difference from Text-to-web group with α = .05 

Interactive text messages increased engagement, but did not convert 

individuals to respondents (nor did they divert them from the full-length 

survey).

Control Text-to-web Text questions

Response rate: Full survey OR text questions (s.e.) 32.1 (4.21) 30.2 (3.74) 52.6 *~ (4.63)
National Center for 

Education Statistics

10

Data collection strategy

1/13/22-4/18/22 6 e-mailed web survey invitations, 3 mailed web survey invitations, 1 mailed paper survey. 

Experiment starts later in April

Time period Control Text-to-web Text questions

4/26/22-4/28/22 2nd mailed paper survey Welcome text with link to 

survey

Welcome text with SMS survey 

questions

5/2/22 Reminder e-mail

5/9/22- 6/3/22 CATI collection

5/17/22-5/19/22 3rd mail paper survey Texted link to web survey Texted SMS survey questions

5/23/22 Reminder e-mail

5/31/22 End of experimental treatments; beginning of “last ditch” strategies

6/1/22 Welcome text with link to 

survey

6/6/22 Texted SMS survey questions

6/9/22 Reminder e-mail
National Center for 

Education Statistics

11

Our conclusion was similar when looking at the start of data collection 

through the end of the experimental treatments:

There was no difference in survey response rates at the end of the 

experimental treatment, even when we replaced a survey invitation with a 

contact that didn’t include an invitation. We had higher interaction rates 

(i.e., responded to full survey OR abbreviated texted survey) when texted 

questions replaced a paper questionnaire or texted web link. 

Response rates as of 5/31/22 * Significant difference from Control group with α = .05 ~ Significant difference from Text-to-web group with α = .05 

Control Text-to-web Text questions

Response rate: Full survey (s.e.) 84.6 (2.05) 82.6 (2.16) 83.5 (2.73)

Response rate: Full survey OR text questions (s.e.) 84.6 (2.05) 82.6 (2.16) 88.6 *~ (2.30)
National Center for 

Education Statistics

12

For the hardest-to-reach individuals, can we engage 

them with text messages?
National Center for 

Education Statistics

13

Data collection strategy

1/13/22-4/18/22 6 e-mailed web survey invitations, 3 mailed web survey invitations, 1 mailed paper survey. 

Experiment starts later in April

Time period Control Text-to-web Text questions

4/26/22-4/28/22 2nd mailed paper survey Welcome text with link to 

survey

Welcome text with SMS survey 

questions

5/2/22 Reminder e-mail

5/9/22- 6/3/22 CATI collection

5/17/22-5/19/22 3rd mail paper survey Texted link to web survey Texted SMS survey questions

5/23/22 Reminder e-mail

5/31/22 End of experimental treatments; beginning of “last ditch” strategies

6/1/22 Welcome text with link to 

survey

6/6/22 Texted SMS survey questions

6/9/22 Reminder e-mail
National Center for 

Education Statistics

14

For the hardest-to-reach individuals (after 14 contacts + CATI), texting 

questions that, when answered, don’t qualify as a completed survey, still 

increased survey response rates

Control [got text link 

& text questions] 

Text-to-web [got 

text questions] 

Text questions [got no 

more texts] 

Response rate: Full survey (s.e.) 24.8 (5.01) 14.2 * (5.12) 4.7 *~ (2.01)

Among nonrespondents as of 5/31/22, response rates as of 7/12/22 * Significant difference from Control group with α = .05 ~ Significant difference from Text-to-web group with α = .05 

The interaction rate (survey + questions) was highest when text questions were 

sent early 

Control [got text link 

& text questions] 

Text-to-web [got 

text questions] 

Text questions [got no 

more texts]

Response rate: Full survey 

OR text questions (s.e.)

29.3 (5.12) 24.4 (5.29) 45.6*~ (6.26)
National Center for 

Education Statistics

15

Data collection strategy

1/13/22-4/18/22 6 e-mailed web survey invitations, 3 mailed web survey invitations, 1 mailed paper survey. 

Experiment starts later in April

Time period Control Text-to-web Text questions

4/26/22-4/28/22 2nd mailed paper survey Welcome text with link to 

survey

Welcome text with SMS survey 

questions

5/2/22 Reminder e-mail

5/9/22- 6/3/22 CATI collection

5/17/22-5/19/22 3rd mail paper survey Texted link to web survey Texted SMS survey questions

5/23/22 Reminder e-mail

5/31/22 End of experimental treatments; beginning of “last ditch” strategies

6/1/22 Welcome text with link to 

survey

6/6/22 Texted SMS survey questions

6/9/22 Reminder e-mail
National Center for 

Education Statistics

16

Looking at the FULL data collection:

Response rates were highest for the group that had the most contacts.

However, we had higher interaction rates (i.e., responded to full survey or 

abbreviated texted survey) when texted questions replaced texted web links 

(even with fewer total contacts). 

Response rates as of 7/12/22 * Significant difference from Control group with α = .05 ~ Significant difference from Text-to-web group with α = .05 

Control Text-to-web Text questions 

Response rate: Full survey (s.e.) 87.2 (1.68) 83.6* (1.82) 84.0* (2.22)

Response rate: Full survey 

OR text questions (s.e.)

87.9 (1.71) 85.2 (1.78) 89.1~ (1.76)
National Center for 

Education Statistics

17

Discussion
National Center for 

Education Statistics 18

IN SUMMARY

• For hard-to-persuade individuals, we found that replacing mailed questionnaires with 

texted links OR abbreviated text questions did not help or hurt survey response rates

• Texting questions increased engagement when sent before the final stages of data 

collection

FUTURE RESEARCH

• Who answers texted questions but not a longer web survey?

– Could a follow-up or integrated text-to-web link convert them to respondents?

– Would they complete a series of text question surveys instead of a single web 

survey? 

• Are mobile web completes preferable to paper questionnaires? To web surveys 

completed on a computer?
National Center for 

Education Statistics 19

maura.spiegelman@ed.gov

allison.zotti@census.gov

https://nces.ed.gov/surveys/ntps/