Lessons from Tulsa, OK: Sending text messages to encourage residents to connect with support…
By: What Works Cities
Tags: Cities, Tulsa, Data, Local Government, Behavioral Science

How do we know if this text message campaign worked and how cities can run their own similar evaluation and project

Tulsa designed and launched a randomized control trial to evaluate the impact of this text alert on people who were considered at risk of eviction because they were experiencing financial distress. The at-risk population was identified by an in-house predictive model based on water billing payment data: the model selects people who are considered more vulnerable to eviction based on whether they have a history of missing utility payments or have had their utilities shut off.

Note: Your city does not need to have a predictive model to replicate this trial! If you have access to a database that contains information indicating the financial status of residents and their phone numbers you can use that information to target recipients. Another option could be to use census level data or a resource like the Opportunity Atlas to identify individuals with higher needs. Or, you can roll out the intervention to a broader group of residents, as long as you can get contact information for all the recipients.

Once the city had identified the recipient group and assembled its phone contact details from the water billing information, the team used Twilio, a cloud communications platform, to send a text message to a random selection of half the people in the sample, while the other half (which served as a control group) was only exposed to the 211 material they would have normally encountered, such as emails or print advertisements.

One of the best parts? The trial was inexpensive to run. Sending messages for this trial cost only $0.0075 per message, for a total expense of about $40 to reach over 5,100 residents.

The city was able to measure the impact of the text message by comparing the number of calls, uptake in support, and service connections from the group that received the message to the group that did not (i.e., the control group).

Tulsa found significant differences between the two groups: text recipients were 17.6 percent more likely to call the helpline and 18.5 percent more likely to be connected to financial services. The success of this intervention showed that text messages can be an effective and cheap method of outreach to at-risk residents. They can help lower the barriers for residents to connect with available financial services — a key step in increasing the uptake of these services for eligible residents. Over the five-month trial, 144 households that received the text message called 211 and got connected to housing services. When compared to the expected rate for the uptake of these services in this period, the text message resulted in an additional 11 households receiving assistance and an additional $16,500 of housing assistance provided.

The Evaluation Protocol

The remaining section of this guide will walk you through a step-by-step on how to adapt this intervention and evaluate its impact in your community.

An evaluation project has four key steps:

Scoping Your Evaluation: Identify your goals, target population, and approach

Identify your goals, target population, and approach Designing Your Evaluation: Develop your evaluation method, outcome measures, and randomization approach

Develop your evaluation method, outcome measures, and randomization approach Implementing Your Evaluation: Deliver your solution, ensure everything goes to plan, and collect data

Deliver your solution, ensure everything goes to plan, and collect data Analyzing Your Results: Conduct data analysis based on your collected data

To design an effective evaluation, Tulsa developed an “Evaluation Protocol” to guide their evaluation design and implementation. While sending a text message may seem straightforward and might not feel like it needs a formal protocol, evaluations need to be implemented in a careful and precise way to generate results that you (and others) can learn from. A protocol can help you to document the decisions and intentions around your intervention. It’s a forward-looking plan that will help you down the road — and also make it easy for people to replicate your work in the future (like we are doing now). We’ve made it easy for you to implement a similar project in your city by giving you Tulsa’s evaluation protocol to follow along.

Download the evaluation replication worksheet to see Tulsa’s evaluation protocol and develop your own approach!