COVID-19 Health Messaging to Underserved Communities
Phase: NA
Conditions: Coronavirus
Sponsor: National Bureau of Economic Research, Inc.

Pre-Analysis Plan
COVID-19 Health Messaging to Underserved Communities
This Draft: May 14, 2020
1.
Introduction
In the US, recent statistics show that African American and Latinx communities bear a disproportionate
burden from Covid-19. Reaching vulnerable and underserved populations is therefore crucial to com-
bating the disease. However, most public messaging campaigns are not targeted toward underserved
communities and don’t address fears of social stigma, mistrust in the healthcare system, or concerns
about immigration status.
This study aims to tackle these issues by delivering video messages, recorded by a racially- and ethnically-
diverse set of doctors from Massachusetts General Hospital and the Lynn Community Health center, to
a large sample of black and Latinx individuals across the US.
The study has two purposes
1. Identify the combination of videos that are the most effective among those we have, for broad
scale up in a second phase.
2. Test some fundamental hypotheses that would apply in a broader sample.
We are cross-randomizing both the message content and the race/ethnicity of the doctor delivering the
message. All messages contain information on the symptoms of Covid-19, social distancing, hygiene
and mask wearing. Experimental variants include: a) explicit acknowledgment of inequalities pertain-
ing to i) treatment by the healthcare system, ii) impacts to immigration status from seeking care or iii)
economic opportunity; b) variation in whether the social distancing and hygiene information is deliv-
ered by Dr. Birx of the CDC or by the diverse set of doctors; and c) giving debiasing information about
the (relative lack of) stigma associated with wearing a mask in public.1 Subjects see the video messages
recorded by a) doctors of concordant race or ethnicity (African-American doctors for African-American
subjects or Latinx doctors for Latinx subjects) or b) doctors of discordant race or ethnicity (white, non-
hispanic or Asian doctors).
We aim to understand which messaging strategies are most successful at communicating information
about Covid-19 to undeserved communities. The goal is to understand how to best structure messaging
to make them more impactful. We are measuring “impact” as a) trust in information coming from the
healthcare system; b) attention and retention of key facts about Covid-19 presented in the videos; c) in-
terest in following the advice in the videos pertaining to social distancing, hygiene and mask-wearing;
1See Appendix Section A for transcripts of the videos.
1
NCT04371419

and d) health-seeking behavior.
We hypothesize that overall videos delivered by concordant messengers will be more effective. More-
over, acknowledging the “elephant in the room” –i.e., explicit acknowledgment of inequalities – will
make the videos more impactful, especially for discordant doctor messengers. We also hypothesize
that videos delivered by our partner doctors (even of discordant race or ethnicity) will be more impact-
ful than one delivered by a representative of the CDC. Finally, for individuals who believe that mask
wearers are likely to be viewed as sick or up to no good by larger than what we are informing them,
we predict that presenting information about beliefs will increase demand for mask wearing.
2.
Treatments, and Experimental Protocols
2.1
Treatments
Each subject watches three videos. Each video features a different doctor. All partner doctors viewed
in the videos by a given subject are of the same racial or ethnic background. 2
1. Video 1:
• Introduction
• Discussion of symptoms
• Acknowledgement of “elephant in the room” issues, where applicable.
2. Video 2:
• Information about social distancing and hygiene
3. Video 3:
• Information about masks
We are varying four aspects of the videos across treatments:
1. Racial or ethnic concordance with partner doctors in the videos (note that we side-step the issue
of language: for Spanish speakers, they can chose English or Spanish, and there are non ethnically
Hispanic doctors speaking in Spanish).3
2. Video 1 content: Elephant in the room
• No elephant in the room content
• Elephant in the room 1: Trust or Fear
• Elephant in the room 2: Economic Inequality
2However, some individuals may see two black doctors and Dr. Birx from the CDC, a white doctor and representative
of the US government.
3However we do not have recordings in other languages.
2

3. Identity of the doctor delivering Video 2
• Pre-recorded video of Dr. Birx of the CDC in front of the White House
• Partner doctor
4. Video 3 content: mask debiasing
• Basic mask information, no perceptions information
• Basic mask information, perceptions information
– For the latinx community, the video discusses perceptions that the mask-wearer is sick.
– For the black community, the video discusses perceptions that the mask-wearer is “up to
no good.”
– In both cases the message reports responses from a prior survey of perceptions that mask
wearers in photos of the same racial/ethnic group are “protecting the community.”
All of the video scripts can be found in Appendix Section A. Figure 1 shows how these different treat-
ments are incorporated in the randomization design.
Figure 1: Treatment Design: There are 8 content-variants for each video, shown here. Cell 1c/d includes
the baseline information package. For each combination of message content, the partner doctors in the
video may either be of a concordant (c) or discordant (d) racial/ethnic background.
All treatments 
receive basic 
information + 
Doctor-
Delivered CDC 
Message
Birx
-Delivered CDC 
Message
Elephant 1: 
Fears (L) or 
Trust (A)
Elephant 2: 
Economic 
Inequality
No Mask 
Debiasing
1c/d
2c/d
3c/d
4c/d
Mask 
Debiasing
5c/d
6c/d
7c/d
8c/d
2.2
Recruitment and Sampling
We are using Lucid to recruit a sample and to compensate subjects for their participation. Our total
target sample size is 15,000 subjects, 10,000 African American respondents, and 5,000 Hispanic respon-
dents. We require all participants to be age 18 or older. We plan to draw from two samples (within each
racial/ethnic group).
1. 5% will be drawn from individuals with a college education or beyond
2. 95% will be drawn from individuals with a high school education or below
3

2.3
Experimental Protocols and Treatments
Our experiment has the following structure:
1. Recruitment and Baseline survey
(a) Recruit target sample via Lucid
(b) For Hispanic respondents, elicit a preference for English or Spanish. Those with a preference
for Spanish will be given Spanish language survey questions and will see treatment videos
recorded in Spanish as well.4
(c) Collect demographic information and healthcare experience. Several demographics vari-
ables are being collected directly by Lucid.
2. Randomize Subjects to treatment
• Treatment scripts are tailored to each racial/ethnic group
• Within racial/ethnic group, treatment is stratiﬁed on age, gender and geographic region.
• Randomization is at the individual level according to Figure 1.
• There will also be a control group who will receive COVID-19 information at the very end of
the survey.
3. Video delivery for all individuals not in the pure control group.
(a) After each of the three videos, respondents evaluate the video’s usefulness, and trustwor-
thiness, along with the respondents intention to follow the advice in the video and share
information from the video.
(b) Prior to the third video (masks), individuals will answer questions about second order beliefs
about mask wearers.
4. Endline survey
5. Videos shown to the pure control group.
6. Debrief script
7. Follow-up survey
(a) We are tentatively planning to follow up with participants several days post-treatment
(b) The chief purpose would be to measure health-seeking and health-preserving behaviors.5
8. Once we learn which messaging strategy is most effective, we will scale up that treatment more
broadly (distributing it through social media etc.)
4The one exception is the CDC video with Dr. Birx. This is always shown in English.
5We do not commit on analyzing this data or even fully implementing a follow up because it is quite difﬁcult to success-
fully follow up with online survey participants, so it will depend on our success in ﬁnding people in our pilot.
4

3.
General Hypotheses to be Tested
This section lists the general hypotheses we want to test. For those, we are planning to run broadly
pooled treatments (and speciﬁc interactions speciﬁed below).
Main hypotheses:
1. What is the overall effect of any combination of videos, compared to the control group?
2. Does racial concordance between the doctor messenger and recipient give the message higher
impact across the board?
3. Does addressing the “elephant in the room” lead to greater video impact?
4. Are the doctors in our videos more effective at communicating the same message with under-
served populations than Dr. Birx, a representative of the CDC and the US federal government’s
response? (To test this hypothesis, we will focus on the outcomes associated with the social dis-
tancing and hygiene video.)
5. Does the Dr Birx video lead to a "shut down" and refusal to pay attention to the rest of the message:
for this we can compare the attitude to masks (which are not mentioned in the Dr Birx video, and
are discussed in a video segment shown after that with Dr. Birx).
6. Does providing information on perceptions about mask wearers change an individual’s willing-
ness to wear a mask? We proxy for willingness to wear a mask in two key ways: a) demand for
information about mask-making, b) incentivized allocations of money to support masks for black
or latinx communities vs. other charitable projects targeted at the same communities.
(a) Focus groups and baseline data suggest that on average, members of the black community
worry that they will be viewed as “up to no good” when wearing a mask by a larger percent
of Americans than actually hold that belief. For Latinx communities, the fear is more that
mask-wearers will be perceived to be sick, generating a different kind of stigma.
(b) Speciﬁcally, we predict that the effects of the "social norm debiasing" treatment should vary
with the respondent’s priors about how mask-wearers are viewed. Individuals learning that
their beliefs were too pessimistic should increase their willingness to wear a mask, while
those learning that their beliefs were too optimistic (this should be rare) might respond the
opposite way.
Secondary hypotheses involving speciﬁc interactions between treatments:
1. Racial concordance might also give more credibility to other information presented in the videos
such as info delivered by Dr. Birx of the CDC or information about mask perceptions.
2. Does addressing the “elephant in the room” increase impact of the videos more for discordant
doctor messengers?
3. Does acknowledging the “elephant in the room” give greater credibility to other information in
the video, such as the CDC video or the mask debiasing information? (Interaction effects with
CDC treatment and with mask debiasing treatment.)
5

In addition to testing these hypotheses, we will also explore what is the most effective combination of
messages and messengers overall (after “smart statistical pooling” of variants that prove similar), and
what is its effect, accounting for any small sample bias induced by picking a winner.
4.
Data Collection and Outcomes
We will run our experiment beginning on May 13, 2020. The data will be collected from respondent
surveys. There is a short baseline module before the treatment group views the videos. Before the
mask video (video 3), we elicit beliefs about perceptions of mask wearers. Then following video 3,
there is an endline outcome module that all participants complete. (The control group only sees the
videos after the endline has been completed.) We may decide at a later date to do a follow up survey
with a subsample of the original participants. (Clearly, for this exercise, there will no longer be a valid
control group. All comparisons will be done across treatments.) All of the survey responses will be
downloaded is a .csv ﬁle for cleaning and analysis in STATA.
4.1
Baseline Survey Variables
The baseline survey includes demographic characteristics, information about health status, and access
to healthcare:
• Education
• Household Size
• Employment Status
• Essential Worker Status
• Insurance Status
• Pre-Existing Health Conditions
• Healthcare Usage Patterns
• Health Information Sources
• Food Security
See Section 5, below for a summary of our key Endline survey variables.
4.2
Data quality checks
We have several questions in the survey that capture respondent attention. We will check that low
quality survey responses are not affected by the treatments. If they are not, we will exclude low quality
survey responses. We also plan to exclude those that take very little time on the survey overall and/or
on the videos.
6

5.
Empirical Analysis
5.1
Balance Checks
We will conduct a series of balance tests across treatment arms to ensure that there are no chance differ-
ences between subjects in the various arms. We will regress characteristics measured pre-treatment on
indicators for the arms and test their individual and joint signiﬁcance. Balance tests will be conducted
using all of the variables measured in the baseline survey.
5.2
Key Outcomes
We intend to measure the treatment effects on the following families of outcomes. Outcomes marked
(T) pertain to trust in the medical system or in health information. Outcomes marked (SD-H) pertain
to the social distancing and hygiene content from the CDC video. Outcomes marked (M) pertain to
mask-wearing.
1. Attention and Assessments (T):
(a) Ratings and trust, willingness to share videos as described above,
(b) Time spent watching the videos
2. Knowledge of information presented in the video (retention)
(a) Basic information about symptoms
(b) Information covered in the CDC video (SD-H)
(c) Information covered in the mask video (M)
3. Measures related to trust in the health-care system and health-seeking behavior (T).
4. Views on Mask-wearing (M)
5. Interest in more information about masks and allocations to organizations supporting mask-
wearing (M)
6. Interest in more information pertaining to social distancing and hygiene (SD-H)
We will construct indices for each family, where applicable. We will also adjust our inferences for
multiple outcome testing.
5.3
Regression Analysis
We will do two types of analysis: identifying effectives videos in our sample, and general tests of the
hypotheses described above.
• Best treatment
We are interested in selecting the best combination of types of video for scale up. In order to identify
the most effective combination of videos, we will adapt a disciplined pooling methods of Banerjee et
al (2020) for selecting the most effective combination of treatments (to a settings were we have large
7

sample within each of 16 treatment cells), and Andrews et al (2020) to correct for small sample bias in
estimating the impact the most effective method.6
For this speciﬁc question of whether our messages of a given type (concordonant race, correcting priors,
etc), since random assignment is done by individuals, we do not adjust standard errors. This is the
correct analysis for scaling up these videos per-se.
• Hypothesis tests
In order to test the hypotheses listed above, we run the following contrasts (Hispanic and Black will be
run separately for all the contrasts:
Main hypotheses tests:
1. Is there any impact of the videos (treatment vs control)
2. Concordant vs. discordant messengers: pooled across all
3. Birx with "no elephant" and the two different versions of "elephant in the room", no Birx each
compared to the "no elephant, no Birx" condition
4. Birx versus everything else.
• For social distancing and hygiene (largest effect expected here for the Birx treatment)
• For other outcomes
5. Mask-debiasing treatment on mask outcomes, heterogenous based on respondent prior beliefs
(lower or higher than 8 out of 10 people think mask wearer is helping the community).
Secondary hypotheses tests
1. Birx treatment effects will be interacted with concordant doctor
2. Birx treatment effect will be interacted with social debiasing treatment (which may be less effec-
tive after presenting Birx)
3. Elephant treatment will be interacted with concordant doctor
4. Mask treatment effect will be interacted with concordant doctor
5. Mask treatment effect will be interacted with elephant treatment
6. Birx treatment effect will be interacted with elephant treatment
7. Gender of provider will be interacted with concordant doctor and/or respondent gender
Everywhere, we will control for the strata, race/ethnicity, and where applicable, choice of English ver-
sus Spanish. Robustness of results to any possible chance unbalance will be checked by running the
Belloni et al. Lasso procedure to select control variables.
6Banerjee, Abhijit, Arun Chandrasekhar, Esther Duﬂo, Matthew O. Jackson, John Floretta, Harini Kannan, Francine
Loza, Anirudh Sankar, and Anna Schrimpf (2020) “Selecting the most effective nudge: evidence from a large scale experi-
ment on immunization,” Working Paper. )
8

For this more general analysis of whether a given type of video is more effective in general, one may
want to consider a hypothetical where the videos we use were drawn from a larger pool of possible
videos (i.e. we recruited N doctors but could have recruited 2N doctors from the same pool). In this
broader exercise, one also wants to adjust standard errors to account for the possible doctor-video
random effect. We will develop and implement the correct methods to do so.
5.4
Heterogeneous Effects
We are interested in understanding differences by baseline characteristics. We believe that there may
be heterogeneous treatment effects based many dimensions. The three most important heterogeneous
treatment effects are the following:
1. Heterogeneity in the effectiveness of the mask treatment by prior beliefs (i.e. whether people
would update up or down from their priors based on the message seen).
2. Heterogeneity by gender, especially for understanding impacts on masks. (Our background re-
search suggests that African American men feel mask stigma most strongly.)
3. Heterogeneity by language choice within the Latinx group (in particular for CDC message which
is not translated)
We are also interested in exploring heterogeneity on the following categories of traits/characteristics.
• Vulnerability to Covid-19 (age, essential worker status, pre-existing health conditions)
• Level of baseline knowledge (education, prior interaction with the health-care system, informa-
tion sources)
• Location effects (covid-19 policies, covid-19 infection rates, and socio-economic conditions)
Given the many ways to cut the data we will follow the methodology of Chernozhukov et al (2018) for
this latter set of potential heterogeneous treatment effects.7
6.
Robustness
6.1
Threats to Interpretation
We would like to assume that differences across videos come from either differences in the racial/ethnic
identity of the doctors in the video or from differences in the content of the messages, rather than from
other chance differences across videos. We are considering doing a separate round of surveys where
respondents watch portions of the videos common to all treatments and rate rate them on characteristics
such as production quality, voice quality, voice appeal, and attractiveness. We would do this exercise
following completion of the main experiment.
7Chernozhukov, Victor, Mert Demirer, Esther Duﬂo, and Ivan Fernandez-Val (2019). Generic machine learning inference
on heterogenous treatment effects in randomized experiments. No. w24678. National Bureau of Economic Research.
9

6.2
Attrition
All baseline and endline outcomes will be obtained in the same online survey instrument. Attrition
will be deﬁned as those who have viewed the video but then drop out of the survey before completing
it or have to be removed due to low qality video. We will check that attrition is not differential across
study arms.
7.
Funding and Human Subjects Review
Funding is provided by the National Science Foundation RAPID-2029880 for Covid-19 research, and
RAI Italian Broadcasting corporation (via an unrestricted gift to J-PAL that we attributed to this project).
The IRB at MIT is serving as the primary institution of record and has entered into a reliance agreement
with Harvard, Massachusetts General Hospital, and Yale. We have also received IRB approval from
Stanford.
10

Appendix
A.
Video Scripts
A.1
Videos for African American Viewers
Full script is: Video 1A+[Video 1B or Video 1C or Video 1D] + Video 1E + Video 2 + Video 3A+ [Nothing
or Video 3B] + Video 3C
Video 1A:
[start with mask, remove it] Hello, I?m Dr [YOUR LAST NAME HERE] from [YOUR INSTITUTIONAL
AFFILIATION HERE], and I?d like to tell you a little about Coronavirus or COVID-19. COVID-19 is
a new virus that can infect the respiratory tract and lungs. Although many people who get sick from
COVID will get better, some people who get it become very ill and some even die.
Video 1B:
Although there?s no cure, there are ways medical professionals have found to protect you and your
community from COVID. I hope that this message can give you information that will help you protect
you or someone you love from COVID infection.
Video 1C:
Even though everyone is affected, communities of color have been especially hit hard by the COVID
19 epidemic. Although there is no cure there are ways medical professionals have found to protect you
and your community from COVID. Now, I know the medical system has not always earned your trust
and still remains unequal in how it treats individuals today. But I hope that this message can give you
information that will help protect you or someone you love from COVID infection.
Video 1D:
Even though everyone is affected, communities of color have been especially hard hit by the COVID
19 epidemic, due to a long history of inequality. Many of the jobs done by minorities are essential and
cannot be done remotely, which increases the risk of getting the virus. Additionally, when one lives in
tight spaces, it is also much more difﬁcult to keep a safe distance. But I hope that this message can give
you information that will help you protect you or someone you love from COVID infection.
Video 1E:
First, I would like to tell you about the symptoms of COVID-19. The most common symptoms of
COVID-19 are cough, fever, and trouble breathing. Another odd symptom some people have is loss of
taste or smell. A large number of people who have COVID-19 actually don?t show any symptoms at
all. Unfortunately, people can still spread the disease to others even with no symptoms. The next video
will provide you with more information on how you can protect yourself and others.
11

Video 2:
[start with mask, remove it]
Hello, I?m Dr [YOUR LAST NAME HERE] from [YOUR INSTITUTIONAL AFFILIATION HERE], and
I?d like to tell you a little about how to protect you and others from transmitting COVID-19. The
absolute most important thing is for people to remember is to be six feet away from every other human
being. What does that mean? Whether you’re outside, whether you?re shopping, whether you have
to go to the pharmacy or grocery store to get the absolutely essential supplies, make sure that you are
staying six feet away from the next person. Ensure that you know exactly where your hands are, what
you touched and make sure you?re washing them or using hand sanitizer on a regular basis. When
you come home make sure you disinfect everything that has touched something else.
Video 3A:
[start with mask, remove it]
Hello, I am doctor [YOUR LAST NAME HERE] from [YOUR INSTITUTIONAL AFFILIATION HERE],
and I will tell you a bit more about masks. Wearing a mask is a key way to prevent the spread of
COVID-19. You are not just protecting yourself but also your grandma and your community, just in
case you have COVID-19 but don?t know it.
Video 3B:
Not very long ago, people might have been afraid if they saw someone, especially a person of color,
walking around town with a mask on. They may have thought the person was up to no good. But,
in a MIT survey done in mid-April, 8 out of 10 people who saw a photo of an African-American man
wearing a mask said they thought they were protecting the community. Still, some people may act
uncomfortable around you when you are wearing a mask.
Video 3C:
Even if wearing a mask may sometimes put you in a difﬁcult situation, it is important to protect you
and the community from COVID 19 disease. As medical professionals, I am committed to delivering
the best care I can to every patient. My goal is to make sure that you and everyone you love survives
this COVID-19 pandemic. Thank you for listening to these messages.
A.2
Videos for Hispanic Viewers
Full script is: Video 1A+[Video 1B or Video 1C or Video 1D] + Video 1E + Video 2 + Video 3A+ [Nothing
or Video 3B] + Video 3C
Video 1A:
[start with mask, remove it]
Hello, I?m Dr [YOUR LAST NAME HERE] from [YOUR INSTITUTIONAL AFFILIATION HERE], and
I?d like to tell you a little about Novel Coronavirus or COVID-19. COVID-19 is a new virus that can
12

infect the respiratory tract and lungs. Although many people who get sick from COVID will get better,
some people who get it become very ill and some even die.
Video 1B:
Although there?s no cure, there are ways medical professionals have found to protect you and your
community from COVID-19. I hope that this message can give you information that will help you
protect you or someone you love from COVID infection.
Video 1C:
Even though everyone is affected, LatinX communities have been especially hard hit by the COVID 19
epidemic. Although there?s no cure, there are ways medical professionals have found to protect you
and your community from COVID-19. Now, I know, for some families, it can be very difﬁcult to get
advice from doctors. There?s a lot of fear about immigration status for some members of the Hispanic
community, and of what might happen for those people if they get in contact with hospitals or doctors.
I hope that this message can give you information that will help you protect you or someone you love
from COVID infection.
Video 1D:
Even though everyone is affected, LatinX communities have been especially hit hard by the COVID
19 epidemic. Many of the jobs done by minorities are essential and cannot be done remotely, which
increases the risk of getting the virus. Additionally, when one lives in tight spaces, it is also much more
difﬁcult to keep a safe distance. I hope that this message can give you information that will help you
protect you or someone you love from COVID infection.
Video 1E:
First I would like to tell you about the symptoms of COVID-19. The most common symptoms of
COVID-19 are cough, fever, and trouble breathing. Another odd symptom some people have is loss
of taste or smell. A large number of people who have COVID-19 actually don?t show any symptoms at
all. Unfortunately, people can still spread the disease to others even with no symptoms. The next video
will provide you with more information on how you can protect yourself and others.
Video 2:
[start with mask, remove it]
Hello, I?m Dr [YOUR LAST NAME HERE] from [YOUR INSTITUTIONAL AFFILIATION HERE], and
I?d like to tell you a little about how to protect you and others from transmitting COVID-19. The
absolute most important thing is for people to remember is to be six feet away from every other human
being. What does that mean? Whether you’re outside, whether you?re shopping, whether you have
to go to the pharmacy or grocery store to get the absolutely essential supplies, make sure that you are
staying six feet away from the next person. Ensure that you know exactly where your hands are, what
you touched and make sure you?re washing them or using hand sanitizer on a regular basis. When
you come home make sure you disinfect everything that has touched something else.
13

Video 3A:
[start with mask, remove it]
Hello, I am doctor [YOUR LAST NAME HERE] from [YOUR INSTITUTIONAL AFFILIATION HERE]„
and I will tell you a bit more about masks. Wearing a mask is a key way to prevent the spread of COVID-
19. You are not just protecting yourself but also your grandma and your community, just in case you
have COVID-19 but don?t know it.
Video 3B:
Not very long ago, people might have been afraid if they saw someone walking around town with a
mask on was sick or perhaps was up to no good. But, in a MIT survey done in Mid-April, 8 out of
10 people who saw a photo of a person wearing a mask said they thought they were protecting the
community. Still, some people may act uncomfortable around you when you are wearing a mask.
Video 3C:
Even if wearing a mask may sometimes put you in a difﬁcult situation, it is important to protect you
and the community from COVID-19 disease. As medical professionals, we are committed to delivering
the best care we can to every patient and my goal is to make sure that you and everyone you love
survives this COVID-19 pandemic. Thank you for listening to these messages.
14