Text Messaging the Survey of Income and Program Participants (SIPP) Field Interviewers to Increase Prioritization Protocol Cooperation

Kevin Tolliver (presenter), Assistant Survey Director for SIPP

 Any views expressed on (statistical, methodological, technical, or operational) issues are those of the author(s) and not necessarily those of the U.S. Census Bureau. 

1

About the SIPP

53,000 Households sampled annually

Conducted primarily by field interviewers

Longitudinal panel survey consisting of both new (wave 1) and returning(wave 2+) sample

Utilizes case prioritization methodologies since 2016

Chief uses of the SIPP:

Allows for examination of the interaction between tax, transfer, and other government and private policies

Government policy formulators depend heavily upon SIPP for information on the distribution of income and the success of government assistance programs.

2

Motivation for Texting Field Interviewers

SIPP only texts field interviewers. At this time, there is no plan to text people in sample. 

The Survey of Income and Program Participation (SIPP) uses case prioritization methodologies as part of their data collection protocols

Every case has either a high, medium, or low priority associated with it

There are specific instructions that are provided to the field interviewers that explain how high, medium, and low priority cases should be worked

Field interviewers work their own schedules, meaning that it is difficult to enforce or manage how interviewers work their cases

Furthermore, priority status change throughout data collection and if field interviewers do not transmit regularly the priority status does not update

3

Logistics

Policy: No Title 5 or Title 13 Information can be sent via text

Platform: GovDelivery - FedRamp compliant platform

Sending SMS Text messages through GovDelivery limits our content to 162 characters

Content: Did not use a focus group to form content. However, did use input from Field staff and Center for Behavioral Sciences.

4

SMS Text Messages used in 2019, 2020, &2021

Reminder: SIPP: Your case priorities have been updated. Please transmit and work accordingly. Contact your supervisor with any concerns.

Positive Feedback: SIPP: You worked your cases according to priorities last week.  Thanks for your hard work! Contact your supervisor with any concerns.

Constructive Feedback: SIPP: To ensure data quality, please work your cases according to priorities. Contact your supervisor with any concerns.

5

Additional SMS Text Messages used in 2021

Varying Thanks: SIPP: Thank you for varying your days and times of your contact attempts.

Varying Reminder: SIPP Tip: Varying your days and times can help increase your chances of getting a successful contact.

Not Working: SIPP: Our records indicate many of your cases are not receiving regular attempts. Please make well-planned attempts on your cases. Thank you for working SIPP!

6

Experiments Conducted

2019 Text Message Experiment:

Sequentially randomized content (Feedback vs. Reminder vs. No Text) and timing (morning vs. afternoon) of text messages 13 times over a 19- week data collection

2020 Text Message Experiments:

1st Month: Reminder vs. No Text

2nd Month: Reminder vs. Positive Feedback vs. Constructive Feedback vs. No Text

3rd Month: 0 vs. 1 vs. 2 vs. 3 text messages 

4th Month: Random message vs. Tailored approach

2021 Text Message Experiment:

Strategic Plan vs. Random

Effectiveness of three new contents

7

Evaluation:  Scaled of Evenness of Finding Attempts (SEFA) Assesses Contact Effort

SEFA is our measure of contact efforts assessed during a period of time t.

Borrowed concept  of diversity indices from ecology and economics

Coombs and Walsh (2014) adapted this idea for survey research methods when analyzing contact attempts

8

Scaled Evenness of Finding Attempts (SEFA)

Computation

Windows

9
| Weekend 8am-12pm | Weekday 8am-12pm |
|------------------|------------------|
| Weekend 12pm-3pm | Weekday 12pm-3pm |
| Weekend 3pm-6pm  | Weekday 3pm-6pm  |
| Weekend 6pm-11pm | Weekday 6pm-11pm |

Scaled Evenness of Finding Attempts (SEFA)

Scaled EFA

“Corrected” Scaled Evenness of Finding Attempts

In order to not penalize any interviewer who successfully reached their person in sample, any case that was reached by interviewer (completed interview, partial completed interview, or refusal) is given an EFA value that is the Max EFA value

10

Change in SEFA for Priority P

 is the change in the mean scaled evenness-of finding-attempts score for a case c during a trial. The SEFA assesses if contact attempts are made at a variety of times.

The summary of  for different priorities are

11

Analyses

Generalized Estimating Equations (GEE) allows us to model the effect of the text sent on their behavior () when the behavior of one week is correlated with the behavior of the prior week

using PROC GENMOD in SAS allows us to estimate this relationship

Positive relationships among high priority cases suggest the text improved activity on high priority cases

Negative relationships among low priority cases suggest the text improved activity on low priority cases

12

2019 Results

Feedback significantly increased SEFA on high priority cases by 9%     (p-value of 0.04)

Reminder insignificantly increased SEFA on high priority cases by 4.8%      (p-value of 0.26)

Texting at a time between 9-12pm local time as opposed to 2-7pm local time decreased 3.4% which was insignificant     (p-value of 0.40)

Feedback significantly decreased SEFA on low priority cases by 20%    (p-value of 0.06)

13

2020 Results

Positive Feedback significantly increased SEFA on high priority cases by 57% (p-value < 0.01) while constructive feedback was insignificant

Reminders significantly increased SEFA on high priority cases by 7.7%(p-value of 0.04) 

None of the messages significantly decreased SEFA on low priority cases

14

2020 Results Cont’d

First Month (Reminder Test): 

Reminders significantly increased SEFA on high priority cases by 15.7% (p-value of 0.02)

Second Month (Positive Feedback vs. Constructive Feedback vs. Reminder Test): 

Positive feedback significantly increased SEFA on high priority by 54.2% (p-value < 0.01) 

Reminders marginally increased SEFA on high priority by 16.1% (p-value of 0.09)

Third Month (Frequency of Texts):

Texting every week significantly increased SEFA on high priority

Texting only one time did worse than not texting at all (p-value < 0.01)

Texting every week with constructive feedback had harmful effects

Fourth Month (Tailoring Texts):

Using the first three months to choose a text message did not improve FR behavior on high priorities than no continued randomizations

15

2021 Strategic Plan

16

2021 Results

Positive Feedback and Varying Thanks had the most positive impact on high priority cases

Constructive Feedback and Varying Days still had positive impacts, but not as much as their more positive counterparts

Not updating (Not working) had positive effects among interviewers who did not work the week before

Reminders had the most positive impact on low priority cases, while none of the others had any significant impact on low priority

The strategic plan as laid out had negative effects on SEFA

17

Thank you

Kevin.p.tolliver@census.gov

18

# image2.png

# image3.png

# image4.png

# image5.png

# image11.png

# image6.png

# image1.png