Persuasion in Medicine: Experimental Evidence on Sender and Signal Effects
Phase: NA
Conditions: Flu, Human, Influenza, Human, Covid19
Sponsor: Harvard University

Pre-Analysis Plan
Persuasion in Medicine:
Experimental Evidence on Sender and Signal Effects
This Draft: October 12, 2020
1.
Introduction
The aim of this study is to identify which sender/signal combinations are most persuasive in encourag-
ing low socioeconomic males living in the United States to take up seasonal ﬂu vaccination. We plan to
recruit subjects and randomly assign them to persuasion treatments in the form of video infomercials
that vary dimensions of the sender of the medical recommendation (race and gender concordance, "ex-
pert," and empathy treatments). Speciﬁcally, we will cross-randomize race with the "expert" treatment
as the actor in the video will portray either a doctor or a layperson. In addition, we may randomize
gender of the actor for subjects assigned to a race-concordant "expert" sender, given sufﬁcient funds.
The script used in the experiment will be varied between one that acknowledges past injustices and
one that does not.1 Our outcomes of interest are posterior beliefs about seasonal ﬂu vaccination, WTP
for a free ﬂu shot coupon and redemption of the coupon.
We are interested in testing the following hypotheses: (1.a) whether low-income male subjects (both
pooled and separately by race) are more likely to be persuaded by race-concordant doctors; (1.b)
whether the subjects assigned to a race-concordant doctor are more likely to update/change their be-
liefs/behavior by gender-discordant senders; and, (2) whether the racial concordance effect is similar
for Black and white male respondents. In addition, given the pipeline issues associated with increasing
the supply of Black male physicians in the short-run, a policy relevant question is whether commu-
nity health workers (CHW) or other ancillary staff could function in place of medical doctors when
recommending or informing about preventive health services. This could be a particularly effective
strategy for distributing routine preventive medical care.2 Theoretically, doctors may have more exper-
tise when it comes to medical issues and may be viewed as more trustworthy, but laypeople might be
socially more proximate to the men we are trying to reach. Thus, we will test, within Black respondents,
whether a Black male doctor has the same effect as a Black male layperson (Hypothesis (3)). Finally, for
similar (pipeline) reasons, we are interested in whether the current stock of medical doctors (which has
relatively few African-American and Hispanic doctors) could be trained in such a way to bridge any
racial discordance effect (if one is shown to exist in our context). Thus, among Black respondents, we
1See Section 3.3 for transcripts of the scripts.
2The CHW model has been used extensively in developing world settings and for prenatal care in the U.S. See Srinivas
et al (2019).
1

will randomize them to white doctors that give a standard script vs. those that provide a more empa-
thetic signal3, and we will test, within Black respondents, whether the white doctor empathy treatment
has the same effect as the white doctor standard treatment (Hypothesis (4)).
The basic design requires collection of baseline and endline surveys combined with administrative
data from pharmacies about coupon redemption. The outcomes of interest are posterior beliefs about
the safety of seasonal ﬂu vaccination, WTP for a free ﬂu shot coupon, and redemption of the coupon.
Our ﬁeld experiment has the following structure:
1. Recruitment and Baseline Survey
(a) Recruit via Qualtrics during the ﬂu season.4
(b) Collect demographic information and healthcare experience.
(c) Elicit prior likelihood/intent for getting vaccinated and probability of getting sick with the
ﬂu.
(d) Elicit priors from subjects about ﬂu shot safety.
(e) Assign video treatment.
(f) Elicit willingness-to-pay (WTP) for a ﬂu shot coupon with a multiple price list.
(g) Elicit posterior likelihood/intent for getting vaccinated.
(h) Elicit posteriors about ﬂu shot safety .
(i) Elicit social distance between the sender and receiver.
(j) Elicit ratings on the sender in the video.
(k) Elicit attention to signal in the video.
(l) Distribute ﬂu shot coupons by email.
2. Endline Survey (2 weeks after baseline)5
(a) Elicit posteriors about ﬂu shot safety.
(b) Elicit self-report on whether received ﬂu shot.
(c) Elicit recall of information from baseline video.
(d) Debrief subjects.
3. Between Baseline and Endline
(a) Collect information on ﬂu shot coupon redemption from corporate partner.
3Although we could test both the layman and empathy with white respondents as well, it’s not clear how white respon-
dents will interpret the signal. Prior piloting showed that white respondents assumed that Black respondents dressed as
laymen were less educated, which might make them more socially proximate than white men dressed as laymen. In ad-
dition, by focusing our hypotheses on policy-relevant parameters, we increase our power by reducing the number of arms
from 16 to 6.
4In a pilot, we also experimented recruiting via online ads on Facebook and Google, but discontinued this path of
recruitment due to budget concerns and insufﬁcient targeting capabilities.
5We sent out the endline survey 3 months after baseline in a pilot, but have shifted the release of the endline earlier to
reduce attrition.
2

2.
Hypotheses Tested
In this study, we seek to answer the following research questions:
1. (a) Does racial concordance between an expert (i.e. doctor) sender and receiver affect ﬂu shot
beliefs and/or behavior, and (b) does a subject update beliefs more or change behavior with a
gender-discordant expert in a racial concordance setting?6
2. Is the effect of race concordance between expert sender and receiver stronger for Black vs. white
men?
3. Does a layperson sender work as well as an expert ﬁgure (i.e. medical doctor) in shifting the
beliefs and/or behavior of Black respondents?
4. If negative effects of a race-discordant expert sender to receiver exist, can these be mitigated by a
signal that acknowledges past injustices?
3.
Research Strategy
3.1
Recruitment and Sampling
We will recruit for this study using Qualtrics. Our target population includes adult US males with a
high school degree or less, who have not yet received the ﬂu vaccine in the current ﬂu season. We aim
to recruit 70% non-Hispanic Blacks and 30% non-Hispanic Whites.
The target population is deﬁned by the following criteria:
Demographic criteria For inclusion in our study are the following:
• Age: between 30 and 50 years old.
• Education: High school education or less.
• Gender: Male.
• Race: non-Hispanic Black or non-Hispanic white.
• Has not received the ﬂu shot in the current ﬂu season yet.
• Has sound on his device turned on.
Data quality checks Among the subjects who complete the baseline survey, we will further exclude
subjects from the analysis if they satisfy the following indicator for low-quality survey responses:
• Fast survey-taking: Within race and treatment group, among the 5% fastest in terms of total time
spent on survey.
6We will test Hypothesis 1(b) funds permitting.
3

• Nonsense answers in the open text sections that suggest low effort.7
• Inconsistent responses to questions. 8
• Repeat survey takers.9
Flu shot coupon assignment Non-zero price draw: Among the subjects who complete the baseline
survey, we will further exclude subjects from the analysis if they are offered the chance to trade-in their
ﬂu shot coupon for a cash reward in the WTP elicitation module.10
3.2
Statistical Power
In the power calculation below, we state minimum sample sizes needed in order to detect effect sizes of
0.2 in terms of standard deviation units of 10 main outcome variables11. Details on how we computed
statistical power can be found in this document.
For hypothesis 1(a), we need 385 subjects for each treatment (i.e. racial concordance) and control (i.e.
racial discordance) to test an overall racial concordance effect across both Black and white respondents
assigned to an expert ﬁgure, detecting a difference in means (in standard deviation units) of 0.2. To de-
tect this effect size within each racial group, we need 415 subjects for treatment (i.e. racial concordance)
and control (i.e. racial discordance) group within Black respondents, and 370 subjects for treatment (i.e.
racial concordance) and control (i.e. racial discordance) group within white respondents.
For hypothesis 1(b), we need 430 subjects for treatment (i.e. concordance) and control (i.e. gender dis-
cordance) group within Black respondents assigned to a race-concordant expert delivering a standard
message. We need 450 subjects for treatment (i.e. gender concordance) and control (i.e. gender dis-
cordance) group within white respondents assigned to a race-concordant expert delivering a standard
message.
For hypothesis 2, we need to double the sample size needed to test an overall racial concordance effect
across both races assigned to an expert ﬁgure in order to test an interaction effect (i.e testing whether
the effect of concordance varies by race of respondent).
7We refer to nonsense answers in the open text sections as answers that do not have decipherable English. We plan to
have Mturkers code this.
8Examples of inconsistent responses are (1) a respondent responds that he earns a high income but is unemployed or on
disability; (2) a respondent who replies inconsistently on WTP questions (i.e. he is willing to accept $1 for trading in a ﬂu
shot coupon but not $10).
9We will identify repeat survey takers by checking for duplicate email or IP addresses in survey data.
10The probability of receiving the coupon trade-in opportunity is
1
1000.
11The 10 main outcome variables are: (1) difference between prior and posterior beliefs in ﬂu shot side effects, (2) dif-
ference between posterior intent and prior intent for ﬂu shot, (3) WTP price, (4) whether R wanted the link to look up
pharmacy to redeem the coupon, (5) R’s likelihood of recommending a ﬂu shot to his friends/family, (6) R’s likelihood of
recommending the ﬂu shot video to his friends/family, (7) R’s subjective trustworthiness on the person in the video, (8) R’s
subjective rating on the qualiﬁcation of the person in the video, (9) whether R would want to ask the person in the video
about other health issues, and (10) whether person like the one in the video would be in R’s contact.
4

For hypothesis 3, we need 395 subjects for each treatment (i.e. Black respondents assigned to a race-
concordant expert delivering a standard message) and control group (i.e. Black respondents assigned to
a race-concordant layperson delivering a standard message) to detect a 0.2 effect size in units of standard
deviations.
For hypothesis 4, we need 430 for each treatment (i.e. empathetic treatment arm) and control (i.e. stan-
dard message arm) within Black respondents assigned to a race-discordant expert in order to detect a
0.2 effect size in units of standard deviations.
In conclusion, we need a sample size of 2,620 (1,720 Black and 900 white respondents) to test our main
hypotheses described in section 5.4 with 80% power and at a signiﬁcance level of 0.05.
3.3
Treatment
We will randomize subjects across four different treatments, with respect to variation in the sender and
message. Each treatment will involve an infomercial video in which a male actor gives information
about the safety and efﬁcacy of the ﬂu shot. We will experimentally vary the: 1) race of the sender
(either African American or Caucasian), 2) the sender "expert" (either doctor or layperson), and 3) the
message (either M1 or M2). Funds permitting, we may also 4) vary the gender of the sender (either
male or female) for the subjects assigned to a race-concordant sender.
We produced videos with a total of 5 African American and 5 Caucasian actors. All actors wear the exact
same clothes, and they record the video four times representing the experimental variation discussed
above. All actors were recruited from the same casting agency and have separately been rated on at-
tractiveness and other characteristics via Mturk. We also collected data from imdb.com and mandy.com
about age, education, height, and weight. In the doctor role, the actors wear a button-down blue shirt,
striped tie, a lab coat and stethoscope. In the layperson role, they wear a white T-shirt. The baseline
script video is about 40 seconds long and is reproduced below.12
Message 1 (M1) script reads:
The Centers for Disease Control and Prevention, or CDC, recommends everyone 6 months and older get the ﬂu
shot. The shot protects you from getting sick by cutting your chance of catching the ﬂu in half. It’s also very safe:
less than 1 in 100 vaccinated people experiences a side effect such as fever or chills. The ﬂu shot does not contain
an active ﬂu virus, so you cannot get the ﬂu virus from the shot. I get the ﬂu shot every year to protect myself,
my family, and my community. I recommend you look into getting vaccinated as soon as possible.
Message 2 (M2) script is identical to the above, except that we added three more sentences acknowl-
edging historical injustices committed by the medical establishment. They are highlighted in bold-face
below:
12See appendix materials to this pre-analysis plan for two examples of the videos we recorded.
5

The Centers for Disease Control and Prevention, or CDC, recommends everyone 6 months and older get the ﬂu
shot. I know some people are nervous to follow medical advice about vaccines. In the past, there may
have been times when the medical community broke your trust. But I hope that sharing some informa-
tion with you can help you understand how important the ﬂu shot is. The shot protects you from getting
sick by cutting your chance of catching the ﬂu in half. It’s also very safe: less than 1 in 100 vaccinated people
experiences a side effect such as fever or chills. The ﬂu shot does not contain an active ﬂu virus, so you cannot
get the ﬂu virus from the shot. I get the ﬂu shot every year to protect myself, my family, and my community. I
recommend you look into getting vaccinated as soon as possible.13
We include as a screening question the capability to turn the sound on so individuals can hear the
video.
3.4
Assignment to Treatment
Subjects will be randomly assigned to one of eight main treatment groups that differ in the four dimen-
sions described in Section 3.3, with respect to variation in the sender and message. There will be an
equal draw for Black and white senders, an equal draw for male and female senders within each race
concordance group, an equal draw of M1 and M2, and an equal draw towards a expert vs. layperson.
Within each treatment group except the gender treatment group, subjects will be randomly assigned
to one of the 5 recorded actors of the assigned race. For the gender treatment group, subjects will be
randomly assigned to one of the 10 recorded female or male race-concordant actors; we will assign
subjects to each actor with equal proportion.
All subjects will receive a free ﬂu shot coupon, with all individuals informed during WTP elicitation
that they may have the opportunity to trade-in the coupon for a cash reward. Participants will be pro-
vided the ﬂu shot coupon without the opportunity to trade it in for cash with probability 999
1000. This is
so that we can measure coupon redemption in the majority of the sample.
We checked whether asking priors changes behaviors via a “doubling down effect” or engaging System
II vs. System I in decision-making. We did not see a strong effect and scaled up the asking of priors to
all respondents (See Appendix Section D for pilot lessons).
Please see Figure 1 for the overview of the study design.
13For the layperson video , we replaced the word “cannot” with “can’t” for the phrase “cannot get the ﬂu virus from the
shot.”
6

Figure 1: Study Design
Recruitment
Black
White
Racial
Concordance
Racial
Discordance
Racial
Concordance
Racial
Discordance
Doctor
Lay
Doctor
Doctor
Doctor
Standard
Standard
Empathetic
Standard
Standard
Standard
Gender
Discordance
Gender
Concordance
Gender
Discordance
Gender
Concordance
Receiver Variation
Sender Variation 1
Sender Variation 2
Signal Variation
Sender Variation 3
7

3.5
WTP elicitation
Redemption rates of the ﬂu shot coupon (Redemption) as well as willingness to pay (WTP) for the
coupon are the main outcomes of interest in this study. We will use the following method to elicit ﬂu
shot coupon valuations in an incentivized manner.
Immediately after the video treatment stage, we will offer the ﬂu shot coupon to all participants. We
will then inform participants that they may be offered the chance to trade in their ﬂu shot coupon for
cash. Participants will then be shown a multiple price list and asked, for each amount, whether they
would prefer to keep the ﬂu shot coupon or receive a cash reward of the given amount instead.14 The
price list includes prices $1, $2, $5, and $10. Prices will be presented as a take-it-or-leave-it offers.
Following WTP elicitation, the computer will randomly choose at random whether the participant will
be offered the chance to trade in the ﬂu shot coupon for cash. If the participant is selected for a chance
to trade in the coupon, the computer will proceed to randomly draw one price offer for that participant.
If the subject accepted at the price drawn, he will receive the cash reward of the given amount in lieu
of the ﬂu shot coupon. If the subject rejected the price, he will not receive the coupon and keep the ﬂu
shot coupon instead. We will skew towards providing subjects with the ﬂu shot coupon without the
opportunity to trade it in so that we can observe redemption decisions for most subjects.
Since our target population is expected to have lower literacy and numeracy then other experimental
samples, we have striven to keep our questions about WTP straightforward and provide an explana-
tion in words of the consequences of each decision.15
3.6
Coupon Redemption
Our initial pilot revealed a very low take-up of the ﬂu shot coupon. We have endeavored to improve
take-up using two strategies. The ﬁrst is to inform respondents that they can enter a lottery for $100 if
they redeem the coupon. The second is to make entry into the lottery contingent on redemption in a
narrow window of time after coupon receipt so as to militate against procrastination.
This strategy is helpful under the following circumstance: Although our intervention shifts beliefs in
the right direction (i.e. towards increased net beneﬁts of the ﬂu shot), beliefs about net beneﬁts are still
below the internal “threshold" for taking the action to redeem the coupon – see theories of behavior
change in Prochaska and Wayne (1997).16 That is, even after our intervention subjects are potentially
14In a pilot, we paid participants $10 for completing the baseline survey and elicited WTP by asking how much partici-
pants would be willing to deduct from their overall survey earnings to receive the ﬂu shot coupon. For our main study, we
are using Qualtrics to handle participant payment and deducting from earnings is not logistically feasible, particularly as
participants will be paid less than $10 and we do not have mechanisms in place to charge participants for the coupon.
15We also wanted to keep the survey length manageable and hence we do not have training/understanding questions in
the survey.
16According to the transtheoretical model in Prochaska and Wayne (1997), behavioral change is a “process involving
progress through a series of stages", which is composed of precontemplation, contemplation, preparation, action, mainte-
nance, and termination.
8

still in the contemplation phase. Our supposition is that the lottery will help clear the threshold for
some, while preserving differences between the treatment groups that are due to treatment.17
3.7
Attrition from the Sample
The vast majority of our outcome variables will be collected in the baseline survey, and thus will not
be subject to attrition. We expect few participants to drop out after watching the video but prior to
completing the survey as they would forfeit payment. Secondary outcome variables collected in the
endline survey may be impacted by attrition. These variables are "Flu shot - self report / spouse /
child", all secondary outcomes in the "beliefs about the ﬂu and ﬂu shot" section, and all attention/recall
questions elicited at endline.
4.
Fieldwork
4.1
Instruments
We will collect data from two sources - online surveys and administrative data on ﬂu coupon redemp-
tion from the pharmacies. The surveys will be hosted by the survey platform Qualtrics. Individual-level
data on ﬂu shot coupon redemption will be sent by TotalWellness, the company that will provide the
ﬂu shot coupons to us. The coupons are redeemable at all major pharmacies, such as CVS, Walgreens,
Kroger, and Wal-Mart, in the US.18
4.2
Incentives
For the main study, Qualtrics will be responsible for administering payment for completing the baseline
survey.19 Baseline survey payment will vary depending on the panel to which the participant belongs,
but will average about $2.5. Participants will receive a $5 gift card for completing the endline survey.
Among baseline survey participants who redeem the ﬂu shot coupon within 5 days of coupon receipt,
we will draw a lottery winner and pay the winner a $100 gift card.
4.3
Data Collection
The ﬁeld experiment will take place during the ﬂu seasons of 2019-20 and 2020-21. Our piloting process
commenced in October 2019 and ended in March 2020.20 Our main study will launch in October 2020,
17The one potential drawback is that ﬁnancial incentive/lottery overpowers the lighter touch treatment that we give via
the videos. Because we are not interested in whether incentives change behavior, we are not randomizing the lottery.
18See list of participating pharmacies here: https://www.totalwellnesshealth.com/pharmacy-2/.
19Participants received a $10 gift card for completing the baseline survey in our pilot, and a payment varying between
$5 and $10 for completing the endline survey. We are planning to have Qualtrics administer the payment for budgetary and
logistical reasons.
20We ran a total of two pilots. The ﬁrst, which ran from October to December 2019, recruited using Qualtrics and was
unsuccessful due to imprecise targeting of low-education individuals and too many (16) treatment arms. The second, which
ran from December 2019 to March 2020, recruited using Qualtrics and online ads on Facebook and Google. We accrued 853
completed responses through Qualtrics. Our recruitment via online ads was unsuccessful, with an extremely low number
of completed responses and high costs. Learning from the experiences of the pilots, we plan on scaling-up the Qualtrics
9

with data collection projected to be completed by the end of the year.
Data collection will entail three components: baseline survey responses, endline survey responses, and
data on ﬂu shot coupon redemption. We will link the data across the three components using a unique
numeric study identiﬁer (ID) for each subject. In practice, we will assign the ID in the baseline sur-
vey, where we also collect subjects’ email addresses. After a participant who is eligible for the coupon
completes the baseline survey, he will receive the coupon immediately by email. The coupon has the
subject’s study ID printed on it. When a coupon is redeemed at a pharmacy, the pharmacist will record
the ID and automatically pass it on electronically to TotalWellness for billing. TotalWellness in turn will
collect the IDs of redeemed ﬂu shot coupons and pass the list on to the research team for billing.
In addition to the above, we will collect data from a separate sample of low-income men regarding
feedback on the actors including their attractiveness, tone and trustworthiness.
4.4
Data Processing
Survey responses collected on Qualtrics will be downloaded as a .csv ﬁle, and then cleaned and ana-
lyzed in Stata. Data on ﬂu shot coupon redemption will be merged with the unique study ID of each
subject. Any PII will be dropped from the dataset, since it is not needed for the analysis. It is anticipated
that do ﬁles that create and analyze the data will be made available to other research teams. The clean
data (without identiﬁers) will also be made available. Only non-PII information on the actors will be
released.
5.
Empirical Analysis
5.1
Outcome Variable Construction
5.1.1
Flu shot take-up
Primary outcomes:
• Flu shot coupon redemption: Indicator for whether a ﬂu shot coupon was redeemed. Coded 1 if
yes, and 0 if no.
Secondary outcomes:
• Flu shot likelihood - self: Collected before and after display of the video at baseline. Measured
by answer to the survey question “How likely are you to get a ﬂu shot before February 2020? - scaled
from 0 (Not at all likely) to 10 (Extremely likely).
• Flu shot likelihood - self: Collected at endline. Measured by answer to the survey question “How
likely are you to get a ﬂu shot next ﬂu season? - scaled from 0 (Not at all likely) to 10 (Extremely likely).
• Likelihood of recommending the ﬂu shot to friends or family - self: Collected after display of
the video at baseline. Measured by answer to the survey question “How likely are you to recommend
the ﬂu shot to a family member or friend? - scaled from 0 (Not at all likely) to 10 (Extremely likely).
survey from the second pilot with an additional 2,500 responses, for a total sample of about 3,350 individuals.
10

• Flu shot - self report: Collected at endline. Indicator based on answer to the survey question Did
you get the ﬂu shot since you completed our ﬁrst survey (i.e. since October 2019)? - Yes/No.
• Flu shot - spouse: Collected at endline. Based on answer to the survey question Did your spouse
get a ﬂu shot this season? - Yes/No/Don’t have a spouse/Don’t know. - Coded 1 if "Yes", 0.5 if "Don’t
know", 0 if "No", and missing if "Don’t have a spouse".
• Flu shot - child: Collected at endline. Based on answer to the survey question Did your children
get a ﬂu shot this season? - Yes/No/Don’t have children/Don’t know. - Coded 1 if "Yes", 0.5 if "Don’t
know", 0 if "No", and missing if "Don’t have children".
• Flu shot - household: Collected at endline. Based on answer to the survey question Did anyone
in your household get a ﬂu shot this season? - Yes/No/Live alone/Don’t know. - Coded 1 if "Yes", 0.5 if
"Don’t know", 0 if "No", and missing if "Live alone".
Index of ﬂu shot take-up:
• Includes all primary and secondary outcomes listed above that are measured at baseline.21
5.1.2
Flu Shot Valuation (WTP)
• Flu shot coupon WTP: Collected after display of the video at baseline. Measured using the WTP
elicitation as detailed in Section 3.5. Coded as the price on the multiple price list immediately
below the minimum price at which a subject opted to receive the corresponding cash reward
rather than the ﬂu shot coupon. If a subject opted to trade-in the coupon for cash at all prices,
WTP will be coded as 0. Because our funding is currently limited, we may not have the power to
detect all differences but we are actively pursuing sources.
5.1.3
Beliefs about Flu Shot Safety
Primary outcomes:22
• Flu shot safety: Collected before and after display of the video at baseline. Measured by answer
to survey question "Take 100 adult men, selected from your community at random. Let’s say all of the
100 adult men receive a ﬂu shot at the start of the ﬂu season. How many of them do you believe get the ﬂu
from the ﬂu shot?". Coded as 100 minus the subject’s answer. 23
• Flu shot safety: Collected before and after display of the video at baseline. Measured by answer
to survey question "Take 100 adult men, selected from your community at random. Let’s say all of the
100 adult men receive a ﬂu shot at the start of the ﬂu season. How many of them do you believe get the ﬂu
from the ﬂu shot?". Coded as 100 minus the subject’s answer.
21The index is based on baseline responses only to reduce any bias due to selective attrition between baseline and endline.
22Note we also considered incentivizing the belief elicitation, whereby people would make predictions and their guesses
would be rewarded if correct. However we found it challenging to ﬁnd an accepted arbitrator of the “truth”.
23We have developed a novel interactive graphic for this and the question above that use a frequentist interpretation of
probability and involves highlighting gray individual stick men with blue shading. This form of probability has been found
to be easier to understand than others – see work by Gerd Gigerenzer.
11

• Self-Conﬁdence on Flu Shot Safety: Collected before and after display of the video at baseline.
Measured by answer to survey question that asks subject to distribute 10 balls over 10 different
bins. Each bin represents a range of people, out of 100, who get sick from the ﬂu shot. Coded as
total number of balls that are put into the "0-9" bin. 24
Index of beliefs - primary:
• Includes all variables from Section 5.1.3.
Secondary outcomes:25
• Flu shot safety: Collected at endline. Answer to the survey question phrased exactly as the
question described in "Flu shot safety".
Index of beliefs - secondary:
• Includes all variables from the "Secondary outcomes" section of Section 5.1.3.
5.1.4
Social Distance Between Sender and Receiver
All of the outcomes listed will be collected at baseline.
• Measure of Social Distance 1: Collected after display of the video at baseline. Measured by
answer to the survey question “If a person like the one in the video was located near you, would you
want to ask him about other health issues?" - coded as 1 if yes, and 0 if no.
• Measure of Social Distance 2: Collected after display of the video at baseline. Measured by
answer to the survey question “Would a person like the one in the video be a contact in your phone or a
friend on social media?" - coded as 1 if yes, and 0 if no.
5.1.5
Sender and Signal Rating
All of the outcomes listed will be collected at baseline.
• Likelihood of recommending the video to friends or family - self: Collected after display of the
video at baseline. Measured by answer to the survey question “How likely are you to recommend
this video to your friends or family? - scaled from 0 (Not at all likely) to 10 (Extremely likely).
• Trustworthiness: Answer to survey question "How much do you agree or disagree with the following
statement?: I trust the person in the video to give me medical advice. - Disagree strongly/Disagree/Neither
agree nor disagree/Agree/Agree strongly". Coded as -2, -1, 0, 1, 2.
• Qualiﬁcation: Answer to survey question "How much do you agree or disagree with the following
statement?: The person in the video is qualiﬁed to give me medical advice. - Disagree strongly/Disagree/Neither
agree nor disagree/Agree/Agree strongly". Coded as -2, -1, 0, 1, 2.
24We have developed a new graphic interface for this question based on work by Delavande and Rohwedder (2008). See
a picture of the graphic in the Appendix.
25All of the outcomes listed will be collected at endline
12

• Applicability: Answer to survey question "How much do you agree or disagree with the following
statement?: The information provided in the video applies to people like me. - Disagree strongly/Disagree/Neither
agree nor disagree/Agree/Agree strongly". Coded as -2, -1, 0, 1, 2.
For the outcomes listed, a subject will be assigned into one of the ten actors’ picture. All of the outcomes
listed will be collected at endline.
• Subjective Rating - Age: “How old do you think this person is?”
• Subjective Rating - Education: “What is the highest degree or level of schooling that you think the
person completed? - 1 (Less than a high school diploma), 2 (High school diploma or equivalent), 3 (Some
college but no degree), 4 (Associate’s degree), 5 (Bachelor’s degree), and 6 (Graduate degree)”
• Subjective Rating - Trustworthiness: “How trustworthy is this person? - scaled from 1 (Not at all
Trustworthy) to 10 (Extremely Trustworthy)”
• Subjective Rating - Attractiveness26: “Please pick the three most attractive individuals from the list
below.”
Index of messenger rating:
• Includes all variables collected at baseline from the Section 5.1.5.
5.1.6
Attention / Recall
Primary outcomes:27
• Frequency recall: Measured by answer to the survey question What did the person in the video
say about who should get the ﬂu shot?
- Everyone 6 months and older/Everyone 5 years and
older/Everyone 18 years and older/I don’t know. Coded 1 if selected "Everyone 6 months and
older", and 0 otherwise.
• Flu shot ingredient recall: Measured by answer to the survey question What did the person in the
video say about what the ﬂu shot contains? - Contains active ﬂu virus/Contains no active ﬂu virus/I
don’t know. Coded 1 if selected "Contains no active ﬂu virus", and 0 otherwise.
• Background color recall: This question is incentivized: Subjects will earn an extra $ if they answer
correctly. Measured by answer to the survey question What color was background displayed in the
video? - Gray/White/Blue/I don’t know. Coded 1 if selected "Blue", and 0 otherwise.
Index of attention/recall - primary:
• Includes all variables from the Section 5.1.6.
Secondary outcomes:28
26A subject will be given the pictures of 10 actors.
27All of the outcomes listed will be collected at baseline.
28All of the outcomes listed will be collected at endline.
13

• Gender recall: Measured by answer to the survey question Was the person in the video a man or a
woman? - Man/Woman/I don’t know. Coded 1 if "Man", and 0 otherwise.
• Race recall: Measured by answer to the survey question What was the race of the person in the video?
- Black/White/Other/I don’t know. Coded 1 if selected correct answer, and 0 otherwise.
• Expert recall: Measured by answer to the survey question What was the person in the video wear-
ing? - A doctor’s coat/Casual clothes/I don’t know. Coded 1 if selected correct answer, and 0
otherwise.
Index of attention/recall - secondary:
• Includes all variables from the "Secondary outcomes" section of Section 5.1.6.
5.1.7
Beliefs about Coronavirus Infection
All of the outcomes listed will be collected at endline.
• Likelihood of getting the coronavirus infection - self: How likely do you think you are to get the
coronavirus infection? - scaled from 0 (Not at all likely) to 10 (Extremely likely).
• Likelihood of getting the coronavirus infection - self: If there was a coronavirus vaccine available to
you, how likely would you be to get the vaccine? - scaled from 0 (Not at all likely) to 10 (Extremely likely).
5.2
Index Construction
We plan to construct indices of all outcomes related to beliefs in the ﬂu vaccine, WTP, likelihood of
getting a ﬂu shot, and coupon redemption according to the procedure outlined in Anderson et al. (2008)
and Kling Liebman and Katz (2007). We will also create additional indices of other outcomes related to
beliefs about the actor in the video and attention to the video. XXYY decide on groupings
5.3
Balance Checks
We will conduct a series of balance tests across treatment arms to ensure that there are no chance differ-
ences between subjects in the various arms. We will regress characteristics on indicators for the arms
and test their individual and joint signiﬁcance. Balance tests will be conducted using the following
subject background covariates:29
• Age
• Income
• Married
29We will also have a balance test for the actors across race. Caucasian respondents are included reducing the chance
that a racial concordance effect for African-American subjects, for example, is being driven by the African-American actors
being more attractive, higher quality etc than the Caucasian actors. This is because the same set of African-American
actors/videos would not have the same predicted effect on Caucasian subjects. It’s also possible that if there are differential
perceptions across expert race by respondent race, this is mediating any racial concordance effect we ﬁnd.
14

• Living in Urban
• High School Completion
• Insured
• Subjective Health Status
• Prior Subjective Belief on Cost of Flu Shot
• Ever Vaccinated in the Past 5 Years
• Has PCP
• Prior Belief in Community Flu shot Take-up
• Prior Belief on Flu Effectiveness and Safety
5.4
Treatment Effects
We intend to measure the treatment effects in the following primary outcomes:
5.4.1
Beliefs
• y1: difference between prior and posterior beliefs about ﬂu shot safety, such that:
y1i = µ0
1i −µ1
1i
where µ0
1i is subject i’s prior belief on the perceived safety of the ﬂu vaccine, while µ1
1i is subject i’s
update belief on the perceived safety. If the information from the video shifts the subject’s priors
to the truth, y1i is positive.
• y2: difference between prior and posterior intention for a ﬂu shot, such that:
y2i = µ1
2i −µ0
2i
where µ0
2i is subject i’s prior intention for getting a ﬂu shot, while µ1
2i is subject i’s posterior inten-
tion for getting a ﬂu shot. If the subject becomes more willing to get a ﬂu shot after watching the
video, y2i is positive.
5.4.2
Actions
• y3: highest WTP price for a ﬂu shot coupon within a set of prices, $1, $2, $5, $10. If a subject rejects
all the prices, it is recorded as $0.
• y4: pharmacy look-up, which is equal to 1 if the subject wanted a link to look up pharmacy where
he can redeem his coupon, or 0 otherwise.
• y5: coupon redemption, which is equal to 1 if redeemed, or 0 otherwise.
15

5.4.3
Endorse
• y6: likelihood to recommend the video to friends or family.
• y7: likelihood to recommend a ﬂu shot to friends or family after watching the video.
5.4.4
Ratings
• y8: video rating about trustworthiness of the actor in providing medical advice (scaled from 1 to
5).
• y9: video rating about qualiﬁcation of the actor in providing medical advice. (scaled from 1 to 5)
5.4.5
Social Proximity
• y10: willingness to ask the actor about other health issues, which is equal to 1 if willing, or 0
otherwise.
• y11: likelihood of having the one in the video in the subject’s contact, which is equal to 1 if willing,
or 0 otherwise.
Using these variables, we construct six indices, for each individual i, such that:
• Y′
i,Belie f s consisted of y1 and y2
• Y′
i,Actions consisted of y3, y4 and y5
• Y′
i,Endorse consisted of y6 and y7
• Y′
i,Ratings consisted of y8 and y9
• Y′
i,Soc.Prox. consisted of y10 and y11
• Y′
i,All consisted of y1...y11
We intend to test the following hypotheses with this outcome variable:
H1(a): Y′ is higher for subjects randomized to a race-concordant expert than those randomized to a
race-discordant expert.
Y′
i,j = α + β11c1|a=1,e=0
i
+ X′
iΩ+ ϵi
where j is a type of indices among Beliefs, Actions, Endorse, Ratings, Soc. Prox. or All (as described
abov), and i is an individual subject. The following indicators describe subjects’ treatment status:
• c1 indicates whether a respondent is assigned to a race-concordant sender; 1 if so, or 0 otherwise.
• a indicates whether a respondent is assigned to an expert sender; 1 if so, or 0 otherwise.
16

• e indicates whether a respondent is assigned to receive an empathetic message; 1 if so, or 0 other-
wise.
1c1|a=1,e=0
i
is an indicator for random assignment to a race-concordant setting for the respondents as-
signed to an expert sender delivering no empathetic message.
X is a vector of attributes of the subjects and of the actors they are assigned to. We will obtain per-
ceptions of the actors from a separate sample of men that are similar to the target population.30 Ideally
these subject and actor characteristics would be balanced across arms but it’s possible there is imbal-
ance by chance. Additional important controls in our context may be the timing in which recruitment
occurs relative to the ﬂu season and the local intensity of the ﬂu virus nationally or at a more local
geographic level (i.e date and place of recruitment). We will use double-selection LASSO to choose a
control set.31
The coefﬁcients of interest are the β. If race concordant pairs update more than race-discordant among
those assigned to an expert sender delivering no empathetic message, then β1 should be positive (neg-
ative) and signiﬁcant for the outcomes of effectiveness (harm).32
H1(b): Y′ is higher for subjects randomized to a gender-discordant expert than those randomized to a
gender-concordant expert within race concordance treatment group.
Y′
i,j = α + β21c2|a=1,e=0
i,r=c
+ X′
iΩ+ ϵi
where c2 indicates whether a respondent is assigned to a gender-concordant sender; 1 if so, or 0 other-
wise. 1c2|a=1,e=0
i,r=c
is an indicator for random assignment to a gender-concordant expert actor among the
respondents assigned to a racial concordant sender delivering a standard message, M1. If β2 is nega-
tive and signiﬁcant, this suggests that a subject updates more with gender-discordant expert in a racial
concordant setting.
H2: Racial concordance is more important for Blacks than for whites.
Y′
i,j = α + β31c|a=1,e=0
i
+ β41r=b
i
+ β5[1c|a=1,e=0
i
· 1r=b
i
] + X′
i,rΩ+ ϵi
where 1c|a=1,e=0
i
· 1r=b
i
is an interaction term between a racial concordance effect and subject’s race. This
term tests the effect of racial concordance(or discordance) across Black and white respondents.
30A subset of these questions are also asked in the baseline survey to the experimental sample. These variables include
perceived age, qualiﬁcations, education, attractiveness, and trustworthiness. We will also use a voice analytic software to
characterize the actors’ voices.
31http://www.nber.org/programs/dev/slides/18Duﬂo.pdf.
32This follows Hjort et al. (2019) who examine posterior beliefs based on the signal (re: early childhood intervention
effect size) and the prior.
17

H3: The racial concordance effect will be greater for a Black respondent assigned to a Black expert
than for a Black respondent assigned to a Black layperson sender.
Y′
i,j,r = α + β61a|c=1,e=0
i,r=b
+ X′
i,r=bΩ+ ϵi,r=b
where 1a|c=1,e=0
i,r=b
is an indicator for random assignment to an expert actor among the Black respondents
assigned to a racial concordant sender delivering a standard message, M1. If β4 is positive and signif-
icant, this suggests that an expert is speciﬁcally important for Black respondents in terms of updating
beliefs in a racial concordant setting.
H4: For Black respondents, Y′ is higher for subjects randomized to a race-discordant expert delivering
an empathetic message, M2, than those randomized to a race-discordant expert delivering a standard
message.
Y′
i,j,r = α + β71e|a=1,c=0
i,r=b
+ X′
i,r=bΩ+ ϵi,r=b
where 1e|a=1,c=0
i,r=b
is an indicator for random assignment to receive a empathetic message among the
Black respondents assigned to a race-discordant expert sender. If β6 is positive and signiﬁcant, this
suggests that, in a race-discordant setting, validating perceived mistreatment in the past by an expert
ﬁgure makes Black respondents update their beliefs more.
5.5
Heterogeneous Effects
Important secondary analyses will include investigating variation in the treatment response through
interaction effects and sample splits. In particular, we are interested in whether the:
• persuasion treatment effects (TE) (racial and gender concordance, expert and/or empathy) are
different for Black and white respondents
• persuasion treatment effects interact with each other
• TE vary by level of attention33
• TE vary by priors (beliefs about safety of shot)
• TE vary by perceived cost of ﬂu shot
• TE vary by sociodemographic characteristics or medical history of the respondent (e.g. insurance
status, marital status, has PCP, education)
• TE vary by “social distance” between the sender and receiver.34
• TE vary by local conditions, such as ﬂu severity, ﬂu shot availability, perceived and actual number
of individuals vaccinated in community
• TE vary by distance to nearest pharmacy
• TE vary by double selection LASSO for ﬁnding heterogeneous treatment effects
33Attention may also be an outcome, if so we would be estimating complementary effects.
34Social distance proxied by age, education, race, income.
18

5.6
Standard Error Adjustments
Although the level of randomization is the individual (respondent) there could be correlation across
individuals randomized to the same actor-script-expert combination. Since we have 40 such treatment
clusters we will plan to cluster at this level. We will also use randomization inference to calculate
p-values (i.e. RI-test command developed by Simon Heß).35
5.7
Attrition
Attrition will be deﬁned as those who have viewed the video but then drop out of the survey afterwards
and do not complete it within four days of initiation.36 We will check that attrition is not differential
across study arms. We will send respondents links to complete the survey from the point that they
dropped out.37 The target for the sample is 3,350 completed surveys.
6.
COVID-19 Addendum
In light of the relevance of vaccination take-up in addressing the COVID-19 pandemic, particularly
among groups with historical mistrust of medical institutions, the main study will include an addi-
tional module designed to examine the effect of a concordant source on demand for information on
a potential COVID-19 vaccine. This module will be presented to participants in the baseline survey
following the inﬂuenza vaccination modules.
Subjects will be invited to receive the results of an upcoming expert review of a COVID-19 vaccine. The
institution conducting the expert review will be randomized to either be the Food and Drug Administa-
tion (FDA) or the National Medical Association (NMA), the largest national organization representing
African American physicians in the United States. The two messages presented to participants are re-
produced below.
Message 1C (M1C) reads:
The National Medical Association (NMA), which represents Black physicians and health professionals in the US,
will perform an independent review of the safety and efﬁcacy of any COVID-19 vaccine approved by the FDA.
Once a vaccine has been developed and the NMA has completed its review, would you like to receive an email
notiﬁcation with the results of the review?
Message 2C (M2C) reads:
The Food and Drug Administration (FDA) has formed an advisory committee of experts, who will perform an
independent review of the safety and efﬁcacy of any COVID-19 vaccine approved by the FDA. Once a vaccine
35See https://github.com/simonheb
36Qualtrics allows up to four days to ﬁnish the survey after starting it.
37This is a modiﬁcation from the originally posted plan on 10/31/19 and takes into account that some subjects had
technical difﬁculty returning to the survey after clicking on the map.
19

has been developed and the advisory committee has completed its review, would you like to receive an email noti-
ﬁcation with the results of the review?
The outcomes of interest will be demand for information on COVID-19 vaccine safety and efﬁcacy, as
well as interest in learning more about personally participating in a COVID-19 vaccine trial. The former
will be measured using the share of respondents demanding information on COVID-19 vaccine safety
and efﬁcacy from the assigned source. The latter will be measured using the share of respondents
indicating their interest in learning about opportunities provided through the National Institutes of
Health (NIH) to participate in COVID-19 vaccine trials. Heterogeneity by race will be examined.
7.
Research Team
The principal investigators of this study are Marcella Alsan and Sarah Eichmeyer. They will contribute
equally to all stages of the project - that is the design and execution of the ﬁeld experiment, as well as
the analysis of the collected data. Min Jeong (Joyce) Kim and Nikhil Shankar will support the project
as research assistants, and will assist in all aspects of project implementation.
8.
Funding and Ethics
Funding is provided by the J-PAL Health Care Delivery Initiative from MIT as well as Harvard Kennedy
School. The IRB at Harvard is serving as the primary institution of record and has entered into a reliance
agreement with Stanford and MIT.
9.
References
[1] Allcott, Hunt, Luca Braghieri, Sarah Eichmeyer, and Matthew Gentzkow. 2019. “The Welfare Ef-
fects of Social Media.” NBER Working Paper, 25514.
[2] Alsan, Marcella, Owen Garrick, and Grant Graziani. 2019. “Does Diversity Matter for Health?
Experimental Evidence from Oakland.” American Economic Review 109(12): 4071-4111.
[3] Anderson, Michael L. 2008. "Multiple inference and gender differences in the effects of early inter-
vention: A reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects." Journal
of the American statistical Association 103(484): 1481-1495.
[4] Centers for Disease Control. 2017. Behavioral Risk Factor Surveillance System. Accessed Septem-
ber, 2019. https://www.cdc.gov/brfss/annual_data/annual_2017.html.
[5] Duﬂo, Esther. 2018. Machinistas meet randomistas: useful ML tools for empirical researchers. Ac-
cessed September, 2019. https://www.nber.org/programs/dev/slides/18Duflo.pdf.
[6] Heß, Simon. 2017. "Randomization inference with Stata: A guide and software." Stata Journal
17(3):630-651
20

[7] Hjort, Jonas, Diana Moreira, Gautam Rao, and Juan Francisco Santini. 2019. “How Research Affects
Policy: Experimental Evidence from 2,150 Brazilian Municipalities." NBER Working Paper, 25941.
[8] Kling, Jeffrey R., Jeffrey B. Liebman, and Lawrence F. Katz. 2007. "Experimental analysis of neigh-
borhood effects." Econometrica 75(1): 83-119.
[9] Srinivas, Sindhu, Celeste Durnwald, Laura Line, Melissa Patti, Melissa Bucher, Shayna Cunning-
ham, and Valerie Riis. 2019. "“Safe Start”: A Community Health Worker Program That Improves
Perinatal Outcomes in High Risk Women." American Journal of Obstetrics and Gynecology 20(1): 83-
119.
[10] Prochaska, James O., and Wayne F. Velicer. 1997. "The transtheoretical model of health behavior
change" American journal of health promotion 12(1): 38-48.
21

Appendix
A.
Baseline Survey Link
https://harvard.az1.qualtrics.com/jfe/form/SV_ellmX4HMb5h3NVH
B.
Infomercial Video Link
• Version Message 1: https://youtu.be/CxxWBT0ew-U
• Version Message 2: https://youtu.be/TlruIaBOk3o
C.
Distribution of Beliefs
D.
Information on Initial Pilots
Prior to launching the main survey, we ran two pilots in order to calibrate our questions and experi-
mental design. Our ﬁrst pilot included approximately 1000 male subjects recruited using the Qualtrics
panel. Unfortunately, our targeting of low-education was not precise - 500 of the subjects had some col-
lege education. We only asked half the sample priors, and piloted 16 treatment arms, making it difﬁcult
to detect effects. However, among less educated, African-American subjects, there was some indication
that the aforementioned interventions might prove effective (see power calculation above for further
details).
During the ﬁrst pilot, we had difﬁculty ﬁlling in cells for older African-American men using a Qualtrics
panel. We modiﬁed recruitment for part of our second pilot by adding online recruitment via Google/
Facebook. Unfortunately, targeting was more imprecise using online advertisements, and our cost per
22

completed response was extremely high. In a second Qualtrics pilot that we ran concurrently with the
online ads, we found that tightening our targeting criteria proved successful in ﬁelding a sample rep-
resentative of our population of interest, motivating our scaling-up of our second Qualtrics pilot.
We also noted in our ﬁrst pilot that despite shifting posteriors in the direction of more favorable views
of the ﬂu vaccine, ﬂu shot coupon redemption was virtually zero. This ﬁnding indicates that among
the Qualtrics survey sample population, even our most "persuasive" intervention did not shift beliefs
enough so as to pass individuals’ net beneﬁt thresholds and trigger coupon redemption. Thus, we
added in our second pilot a deadline to the coupon (currently within 5 days of receipt) for entry into a
lottery among those who redeem in that time frame.
Third, in the ﬁrst pilot, we randomly assigned half the sample to a survey that elicited prior beliefs
about the ﬂu and the ﬂu shot and the other half to a survey that did not elicit priors. The rationale was
that while priors are key heterogeneity dimensions of interest and while they increase the power of
our analysis, we were worried that they would trigger a "doubling-down" effect or lead individuals to
engage System II more often, thereby muting treatment effects in an undesirable (because not externally
valid) way. However, our initial pilot revealed no such effects. Therefore, starting in the second pilot,
we plan on asking these questions to everyone.
23