arXiv:2012.11055v1 [cs.CY] 21 Dec 2020 

Social Media COVID-19 Misinformation Interventions Viewed Positively, 

But Have Limited Impact 

Christine Geeng1, Tiona Francisco1, Jevin West2, and Franziska Roesner1 

1Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA

2Information School, University of Washington, Seattle, WA, USA

Abstract 

Amidst COVID-19 misinformation spreading, social media platforms like Facebook and Twitter rolled out design interventions, including banners linking to authoritative resources and more specifc “false information” 

labels. In late March 2020, shortly after these interventions began to appear, we conducted an exploratory 

mixed-methods survey (N = 311) to learn: what are social media users’ attitudes towards these interventions, 

and to what extent do they self-report effectiveness? We 

found that most participants indicated a positive attitude towards interventions, particularly post-specifc labels for misinformation. Still, the majority of participants discovered or corrected misinformation through 

other means, most commonly web searches, suggesting 

room for platforms to do more to stem the spread of 

COVID-19 misinformation. 

1 Introduction 

In late March 2020, social media platforms had recently 

increased implementation of misinformation interventions (such as banners or labels) in response to the proliferation of COVID-19 health misinformation. Twitter and Facebook both added generic banners directing 

users to COVID-19 information, as well as added misinformation warnings to specifc posts. To better understand user responses to these changes, we conducted 

a mixed-methods online survey, recruiting through Prolifc and our personal networks, to gauge attitudes (of 

participants who had seen them) towards these interventions on Facebook, Instagram, and Twitter. Our survey 

was exploratory and no hypotheses were tested. We also 

collected accounts of how participants had learned that 

COVID-19 misinformation they had seen was false. Our 

research questions were: 

1. What are people’s attitudes towards social media 

platform interventions for COVID-19 misinformation, including generic banners linking to authoritative sources and specifc false information labels? 

2. How did people discover that COVID-19 misinformation was actually false? Specifcally, what was 

the role of social media platform interventions in 

this discovery, compared to other methods? 

Our results show that participants rated the helpfulness 

of Facebook’s “False Information” label — which appears on specifc posts — signifcantly higher than Facebook’s generic COVID-19 information banner, suggesting that post-specifc interventions may be more effective. Some participants reacted negatively to the interventions, e.g., expressing a distrust of the platform. Despite the general acceptance of the interventions, we fnd 

76.7% of participants instead discovered information to 

be false through web searches or trusted health sites. 

Our results suggest that social media platform interventions are not yet doing the heaviest lifting when it 

comes to correcting misinformation, but people are receptive to these attempts. Our exploratory study raises 

open research questions, and our results suggest there is 

room for platforms to augment or support existing user 

strategies, as well as increase post-specifc fact-check labeling. 

2 Related Work 

The rise of misinformation on social media has prompted 

sites like Facebook and Twitter to design platform affordances addressing misinformation, such as showing links 

to trusted public health sites for vaccine-related search 

terms [20]. Facebook has experimented with various interventions, ranging from showing “disputed” fags or 

“false information” labels on posts to more subtly showing fact-checking “related articles” [3]. While having 

post-specifc “disputed” labels might raise concerns over 

triggering the backfre effect [14], i.e., entrench existing false beliefs, recent replication [21] and review work 
suggests that “backfre effects are not a robust empirical 

phenomenon” [19]. 

Findings about the effectiveness of interventions have 

been varied. Bode et al. found Facebook’s “related articles” to reduce health misperceptions [1]. Pennycook 

et al. found that attaching warnings to fake news headlines could lead to incorrect belief that non-labeled headlines are not false [16]. The latter study only displayed 

headlines to participants; we note that other work has 

shown that people use multiple heuristics on and off social media to determine information credibility [5, 6, 13]. 

Other approaches include pre-emptive debunking, which 

has been shown to be effective at preventing anti-vaccine 

conspiracy beliefs [8]. 

Since the COVID-19 pandemic began, Facebook, 

Twitter, and others have implemented more factchecking affordances [17, 18], given the various health 

misinformation that has arisen [9]. To investigate the 

effectiveness of these interventions in this specifc, currently highly-relevant context, we qualitatively and quantitatively surveyed sentiments of users who have seen 

these interventions on their own feeds, as well as their 

other experiences with COVID-19 misinformation. At 

the highest level, our results suggest that while most respondents are receptive to social media platforms’ attempts to curb COVID-19 misinformation, there remains 

room for improvement and future research to inform both 

platform designs and related policy discussions. 

3 Methodology 

To answer our research questions, we conducted an 

anonymous online survey (approximately 10 minutes 

long) from March 20-26, 2020 to elicit quantitative 

and qualitative responses. Our study was reviewed and 

deemed exempt by the University of Washington Human Subjects Review Board (IRB). We did not collect identifying information about participants. For any 

quotes used in this paper, the quoted participant explicitly provided their consent (in the survey) to have their 

anonymized quotes used in publications. 

To recruit participants, we used both Prolifc, a paid 

crowdsourcing service, and our personal networks via 

social media. Prolifc participants were paid $13.86/hr, 

with an average 7 minute survey completion time. We 

also sought volunteers via our personal networks on 

Facebook and Twitter. Participants were screened out if 

they were not at least 18 years old or had not used Facebook, Twitter, or Instagram since March 1st (around the 

time when the COVID-19 misinformation interventions 

started to roll out). 

We recruited 111 participants through our personal 

networks and 202 through Prolifc, and we removed 2 

disingenuous responses (based on our review of answers 

Age Range Count 

18-24 130 

25-34 107 

35-44 36 

45-54 13 

55-64 13 

65-74 3 

Table 1: Participant age ranges. 

Figure 1: Facebook’s design interventions. On top is 

a generic banner shown when someone searches for 

COVID-19 or related terms, and on the bottom is a label for posts labeled as false by Facebook’s fact checking 

partners. 

to free-response questions), for a total of 311 completed 

surveys. In this paper, we discuss and analyze the results 

of both populations combined. 

Demographic questions were optional. The majority 

of our participants were 18-24 years old (refer to Table 1). 37.94% of our participants live in the United 

States, 10.29% in Portugal, 9.97% in the United Kingdom, 7.40% in Canada, and 7.40% in Poland. The rest 

of our participants live in a variety of other countries. Of 

the 118 participants living in the United States, 48 identify as Democrat, 11 as Republican, and 34 as Independent; the rest did not answer the question. 

Our survey asked if participants had seen Facebook, 

Twitter, or Instagram COVID-19 or misinformation interventions (circa March 2020) before. If so, we asked 

both an open-ended question about their thoughts, as 

well as a question about how helpful they considered the 

intervention, on a 5-point scale from “Not at all helpful” (1) to “Extremely helpful” (5). For interventions that 

labeled specifc misinformation, we asked how that had 

2 
Figure 2: Twitter’s design interventions. On top is 

a generic banner shown when someone searches for 

COVID-19 or related terms, and on the bottom is a label 

for posts known to Twitter to contain manipulated media. 

changed the participant’s view of the post, if at all. The 

screenshots we showed participants of the interventions 

we asked about are shown in Figures 1-3. 

We also asked for anecdotes of when participants had 

seen or believed COVID-19 misinformation, where they 

had seen it, how they discovered its falsity, and what they 

did upon realizing this. Finally, we asked participants to 

select from a list of known COVID-19 misinformation 

which they had seen. 

To analyze open-ended responses about perceptions 

of interventions, three coders independently inductively 

coded a subset of these answers before discussing and 

agreeing on a codebook of 17 codes. Following McDonald et al.’s guidelines on when to seek coding agreement [11], we double coded a subset (46.34% of total 

responses) to check for agreement and then had a single coder code the rest of the responses. For the doublecoded subset, we calculated Cohen’s κ for inter-coder 

reliability, given that we had two coders and nominal 

data [12]. We had a κ of “substantial” (0.61–0.80) to 

“almost perfect agreement” (0.81–1.00) for 87.5% of categories (see Appendix). We discussed code usage discrepancies between coders until we reached a consensus 

Figure 3: Instagram’s design intervention: a generic banner shown when someone searches for COVID-19 or related terms. 

in the fnal coding. 

To analyze our helpfulness scale data, we compared 

helpfulness ratings between interventions from the same 

site by comparing between participants who had seen 

both. Since our scale data is ordinal, we used a Wilcoxon 

signed rank test and only report on tests with signifcance. 

4 Results 

Our results reveal a variety of reactions to misinformation labels and different modes of discovering misinformation. 

Social media interventions are used, but are outweighed by other strategies for debunking misinformation. When asked in a multiple-response question 

how they learned something they saw was false (whether 

or not they initially believed it), participants told us most 

frequently that they conducted a web search (39.6% of 

240 who answered this question), sought out trusted 

sources (37.1%), saw a correction in a social media 

comment (19.2%), or heard a correction from someone 

directly (12.1%). Only 4.2% learned something was 

false because the social media platform had labeled it 

as such. The majority (71.7%) of respondents indicated 

they “knew it wasn’t true”, though we cannot verify 

whether respondents’ baseline knowledge was correct. 

Post-specifc social media interventions are viewed as 

more helpful, and seem to be more effective, than 

generic interventions for our participants. We fnd 

that participants tended to rate (on a 1-to-5 scale) postspecifc interventions as more helpful. For example, 

comparing the 30 participants who had seen both Facebook interventions, these participants found the postspecifc “False Information” label signifcantly more 

helpful (median rating of 4 “very helpful”) than the 

generic banner (median rating of 2 “slightly helpful”). 

(Wilcoxon signed-rank test, V = 4, Z = -4.13, p = 0.018, 

r = 0.75). 

Considering effectiveness, only 13.3% of 105 partic3 
Reactions Count Choice % Count 

Positive 150 Nothing 54.62% 130 

Nothing/Neutral 39 

Unnecessary because I’m already informed 27 

I ignored it 19 

Other 16 

Annoying 12 

Fear/worry about the future 11 

It’s common to see these banners now 10 

Cautious/suspicious of the banner 8 

I don’t trust the company, so I do not trust 

the banner 7 

Done too late 6 

It looked offcial 6 

Thought it was a ad 4 

Don’t know enough to comment on 

execution 4 

Angry 3 

Could be abused to censor 2 

Table 2: Counts of qualitatively coded reactions to 

“What did this intervention make you think or feel?” for 

all social media interventions we showed participants, 

out of 246 responses. 

ipants who saw the Facebook banner said that they had 

ever clicked on it. Meanwhile, 32.3% of the 65 participants who saw the Facebook “False Information” label 

said they no longer believe the content of the post due to 

the label. 50.8% self-reported (albeit in retrospect) that 

they had already not believed the false-labeled post, and 

only 6.2% said that they continued to believe the post, or 

believed it more, given the label. 

The median helpfulness rating of the generic Twitter banner was 3 (“somewhat helpful”), with a reported 

clickthrough rate of 32.8% by 58 participants who saw 

the banner. The difference in helpfulness rating between 

the Facebook and Twitter banners (medians of 2 and 3 

respectively), for the 26 participants who saw both, was 

not statistically signifcant (Wilcoxon signed-rank test, V

= 4.5, Z = -2.11, p = 0.06, r = 0.41). 

Most participants had positive or neutral responses to 

social media platform misinformation interventions. 

We collected qualitative free-response data about participants’ attitudes and highlight key themes here. We fnd 

participants’ opinions about platform interventions range 

from positive (“I thought it was good that Facebook was 

trying to do something to inform people better”) to neutral (“I didn’t think much of it. I follow the news so I 

didn’t click on this one because I already know the basic 

details”) to negative (“I don’t like it. I don’t need Facebook to tell me this, and I don’t trust their automated 

Corrected the person publicly 18.49% 44 

Corrected the person privately 16.81% 40 

Other 11.34% 27 

I don’t know/remember 3.36% 8 

Table 3: Multiple-choice responses to “What did you do 

when you realized COVID-19 information someone else 

shared was false?” N = 238. 

Choice % Count 

Shared the correction 27.18% 28 

Other 12.62% 13 

I don’t know/remember 8.74% 9 

Unshared it if you had shared it 1.94% 2 

Nothing 57.28% 59 

Table 4: Multiple-choice responses to “What did you do 

when you realized COVID-19 information that you believed was false?” N = 103. 

way of detecting it”1) to — rarely — hostile (“I was irritated because it is another in a long list of ‘tools’ to 

‘protect’ users. In my opinion, this label assumes people are morons and unable to discern what’s true, false 

and/or misleading”). 

Table 2 shows how often themes we coded appeared in 

responses. Our analysis focused on negative reactions, as 

these provide more actionable information. A sentiment 

expressed by both positively and negatively-reacting participants was that they found the interventions unnecessary, because they were already suffciently informed 

about COVID-19. 

When participants came across misinformation and 

realized it was false, 54.62% did nothing, but 35.3% 

made a correction. Our results suggest that COVID19 misinformation was rampant on social media and 

the web in late March, 2020: 79.5% of participants reported having seen others share COVID-19 related misinformation, and 33.9% reported believing something 

false themselves. Table 3 and Table 4 show participants’ self-reported reactions when they realized they or 

their contacts had shared COVID-19 misinformation. In 

both cases, a slight majority of participants did nothing, 

though a signifcant fraction also publicly or privately 

shared a correction. 

Public corrections sometimes occurred in group chats. 

One participant stated, “On the same group where the 

message was shared, with my friends, we discussed the 

fact that it was false after it appeared on the news.” Oth1We note that as of at least June 2020, on Facebook, misinformation 

is labeled as false only after a human fact-checker reviews it [4]. 

4 
ers added comments with corrections to posts or “liked” 

an existing correction. Private corrections involved inperson conversations, email, or direct messages. Some 

“Other” responses included reporting the post or fltering unwanted content from one’s social media feed. 

Though we did not collect data on reasons for taking 

no action, we note that these reasons might include not 

wanting to engage in a debate, not being able to fnd the 

original post again, not considering the issue personally 

relevant enough, or not having re-shared the false information themselves after believing it. 

5 Discussion 

From our results, we make some suggestions towards improving misinformation labeling efforts. 

Social media platforms should increase specifc misinformation labeling efforts. In the context of COVID19, our participants had generally positive responses to 

the interventions, and found specifc misinformation labels to be more helpful than generic banners pointing to 

authoritative sources. We also found that these specifc 

labels worked for many participants: out of the 65 people who saw the Facebook label, 21 people heeded the 

label, while only 2 people continued to believe the post 

and 2 people believed it more. Our results thus suggest 

that — at least among our study population — the labels generally produce the intended effect rather than a 

“backfre effect” [14, 10]; this supports other work that 

fnd no robust evidence for this phenomenon [19, 21]. 

However, only 4.2% of respondents who said they have 

seen misinformation stated they learned it was not true 

through social media labeling, suggesting that they often 

see misinformation on social media that is not labeled 

by the platform. This fnding suggests a strong motivation for social media platforms to signifcantly increase 

the amount and frequency of misinformation that they 

explicitly label as false. 

Authoritative banners should be designed to not look 

like ads, and warning fatigue should be considered. 

While the banners were the result of collaborations between social media sites and the World Health Organization and other national public health agencies [17, 18], 

11 responses (out of 246 responses) mentioned the banners looked like ads or that they did not trust the social 

media company enough to trust the banner. The sheer 

frequency with which participants see these or similar 

banners across different sites may also lead to warning 

fatigue [2]. Indeed, 27 responses noted ignoring the banner because they have already seen so much other information about COVID-19. Future research should further 

study these effects and how to avoid them. 

Open research questions remain around intervention 

design and effectiveness, side effects, and interventions beyond COVID-19. As discussions around platform responsibility and potential liability in the face of 

misinformation intensify, policymakers will need substantive evidence to lean on to inform these discussions. 

Our results suggest that different interventions have different impacts, so multiple and continued studies are 

needed. We call on future research to help answer these 

open questions, considering different types of users, different types of content, and different types of intervention designs. 

For example, our study does not attempt to differentiate different types of people, who may react differently to 

interventions. A Pew research study showed that Americans engage with online information in varying ways, 

ranging from eager or curious to distrustful of information sources [7]. Future research should study whether 

interventions like the ones we study here are most effective for certain types of information consumers — for example, people who trust the fact-checking sources used 

by social media platforms, and people who are not already convinced of the relevant misinformation but are 

attempting to become informed. Different intervention 

designs may be effective for different information consumers. 

Future work should also explore if there are other potential side effects of the interventions, beyond debunking misinformation directly. For example, while the 

generic banners may not change behaviors in the moment, perhaps they have a more subtle, sustained impact on how people evaluate information in their feeds. 

This impact might be positive (reducing trust in misinformation) but may also be negative, e.g., increasing trust 

in misinformation that doesn’t have a fact-checking label [16]. 

We found that 35.3% of participants corrected others sharing misinformation, and 27.18% of participants 

shared a correction when they themselves had believed 

misinformation. These numbers are far below 100%, but 

non-trivial. Future research and design should explore 

how much room there is to increase (self-)correcting behavior from users, experimenting with ways to make 

sharing corrections easier. 

Finally, COVID-19 is a unique situation, and future research should study how people use and react to 

platform-based interventions on other topics (e.g., political misinformation, climate change). People may consider certain platform-based interventions more appropriate during a global pandemic, but may prefer that social media platforms take a less active role in labeling 

content in other circumstances. The normalization of 

current platform practices may also shift user perspectives for the future. 

5 
6 Limitations 

Our exploratory study is based on a convenience sample of participants and our results may not be generalizable to broader, more representative populations. Most 

of our participants live in the United States; individuals 

living in other countries may have seen other misinformation more relevant to their geographic location that 

we did not ask about, or different versions of the platform interventions than the screenshots we showed. For 

participants sampled from our personal networks, they 

may have skewed towards academics and people with 

an interest in computer security and privacy. With any 

self-report methodology, responses are susceptible to recall bias, infuence from wording, and erroneous statements [15]. We did not compare differences in data between our two sampling populations as it is unclear what 

variations and similarities there are between these two 

groups. We make no strong quantitative claims about 

our qualitative results. Finally, the COVID-19 situation and platform interventions themselves are changing 

rapidly; our results represent one snapshot in time (late 

March 2020). Nevertheless, this study sheds light on participants’ reactions to platform interventions in a hotlydebated and quickly evolving space, and raises new research questions and directions for future work. 

7 Conclusion 

To better understand people’s responses to social media platform interventions for COVID-19 misinformation, we conducted an exploratory mixed-methods online 

survey in late March 2020 to gauge attitudes (of participants who had seen them) towards these interventions 

on Facebook, Instagram, and Twitter, as well as to collect accounts of how participants have learned COVID19 misinformation was false. Our results suggest that 

post-specifc interventions may be more effective. and 

that social media platform interventions are not yet doing 

the heaviest lifting when it comes to correcting misinformation, but people are receptive to these attempts. 

8 Acknowledgements 

We thank Tadayoshi Kohno, Lucy Simko, and Miranda 

Wei for their feedback on our survey instrument, and Yim 

Register for feedback on an earlier version of this paper. 

This paper was supported in part by the National Science 

Foundation under Award CNS-1651230 and the John S. 

and James L. Knight Foundation. 

References 

[1] BODE, L., AND VRAGA, E. K. In related news, that was wrong: 

The correction of misinformation through related stories functionality in social media. Journal of Communication 65, 4 (2015), 

619–638. 

[2] BRAVO-LILLO, C., CRANOR, L., KOMANDURI, S., 

SCHECHTER, S., AND SLEEPER, M. Harder to ignore? 

revisiting pop-up fatigue and approaches to prevent it. In 

10th Symposium On Usable Privacy and Security (SOUPS

2014) (Menlo Park, CA, July 2014), USENIX Association, 

pp. 105–111. 

[3] FACEBOOK. Replacing Disputed Flags With Related Articles, Dec. 2017. https://newsroom.fb.com/news/2017/

12/news-feed-fyi-updates-in-our-fight-againstmisinformation/. 

[4] FACEBOOK. Facebook’s approach to fact-checking: How 

it works, Aug. 2020. https://www.facebook.com/

journalismproject/programs/third-party-factchecking/how-it-works. 

[5] FLINTHAM, M., KARNER, C., BACHOUR, K., CRESWICK, H., 

GUPTA, N., AND MORAN, S. Falling for fake news: Investigating the consumption of news via social media. In Proceedings of

the 2018 CHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2018), CHI ’18, ACM, pp. 376:1– 

376:10. 

[6] GEENG, C., YEE, S., AND ROESNER, F. Fake news on facebook and twitter: Investigating how people (don’t) investigate. In 

Proceedings of the 2020 CHI Conference on Human Factors in

Computing Systems (New York, NY, USA, 2020), CHI ’20, Association for Computing Machinery, p. 1–14. 

[7] HORRIGAN, J. B. How people approach facts and information, 2017. https://www.pewresearch.org/

internet/2017/09/11/how-people-approach-factsand-information/. 

[8] JOLLEY, D., AND DOUGLAS, K. M. Prevention is better than 

cure: Addressing anti-vaccine conspiracy theories. Journal of

Applied Social Psychology 47, 8 (2017), 459–469. 

[9] KASPRAK, A. No, holding your breath is not a ’simple selfcheck’ for coronavirus, Aug. 2018. https://www.snopes.

com/fact-check/taiwan-experts-self-check/. 

[10] LEWANDOWSKY, S., ECKER, U. K. H., SEIFERT, C. M., 

SCHWARZ, N., AND COOK, J. Misinformation and its correction: Continued infuence and successful debiasing. Psychological Science in the Public Interest 13, 3 (2012), 106–131. PMID: 

26173286. 

[11] MCDONALD, N., SCHOENEBECK, S., AND FORTE, A. Reliability and inter-rater reliability in qualitative research: Norms and 

guidelines for cscw and hci practice. Proceedings of the ACM on

Human-Computer Interaction 3, CSCW (2019), 1–23. 

[12] MCHUGH, M. L. Interrater reliability: the kappa statistic. Biochemia Medica 22, 3 (Oct. 2012). 

[13] METZGER, M. J., AND FLANAGIN, A. J. Credibility and trust of 

information in online environments: The use of cognitive heuristics. Journal of Pragmatics 59 (2013), 210 – 220. Biases and 

constraints in communication: Argumentation, persuasion and 

manipulation. 

[14] NYHAN, B., AND REIFLER, J. When corrections fail: The persistence of political misperceptions. Political Behavior 32 (June 

2010), 303–330. 

[15] PAULHUS, D. L., AND VAZIRE, S. The self-report method. 

Handbook of research methods in personality psychology 1

(2007), 224–239. 

6 
[16] PENNYCOOK, G., BEAR, A., COLLINS, E. T., AND RAND, 

D. G. The implied truth effect: Attaching warnings to a subset 

of fake news headlines increases perceived accuracy of headlines 

without warnings. Management Science 66, 11 (2020), 4944– 

4957. 

[17] POLICY, T. P. Stepping up our work to protect the public conversation around covid-19, 2020. https://blog.twitter.com/

en_us/topics/company/2020/stepping-up-our-workto-protect-the-public-conversation-around-covid19.html. 

[18] ROSEN, G. An update on our work to keep people informed and 

limit misinformation about covid-19, 2020. https://about.

fb.com/news/2020/04/covid-19-misinfo-update/. 

[19] SWIRE-THOMPSON, B., DEGUTIS, J., AND LAZER, D. Searching for the backfre effect: Measurement and design considerations. Journal of Applied Research in Memory and Cognition” 9, 

3 (2020), 286 – 299. 

[20] TWITTER. Helping you fnd reliable public health information 

on Twitter, May 2019. https://blog.twitter.com/en_us/

topics/company/2019/helping-you-find-reliablepublic-health-information-on-twitter.html. 

[21] WOOD, T., AND PORTER, E. The elusive backfre effect: Mass 

attitudes’ steadfast factual adherence. Political Behavior 41 (01 

2018). 

Appendix 

A Survey Instrument 

1. When did you last use each social media site? 

After March 1st Before March 1st Never 

Facebook ◦ ◦ ◦ Twitter ◦ ◦ ◦ Instagram ◦ ◦ ◦

2. Have you seen this banner on Facebook before? 

◦ Yes ◦ No ◦ I don’t know 

3. You said you’ve seen this banner on Facebook before. What did you feel or think about it? 

4. How helpful was this banner? ◦ Extremely helpful ◦

Very helpful ◦ Somewhat helpful ◦ Slightly helpful 

◦ Not at all helpful 

5. Did you click on the banner? ◦ Yes ◦ No ◦ I don’t 

know/remember 

6. Have you seen this ”False Information” label on 

Facebook before? 

◦ Yes ◦ No ◦ I don’t know 

7. You said you’ve seen this ”False Information” label on Facebook before. What did you think or feel 

about it? 

8. How helpful was this label? ◦ Extremely helpful ◦

Very helpful ◦ Somewhat helpful ◦ Slightly helpful 

◦ Not at all helpful 

9. Think of a recent time when you saw this label. Did 

the label change your view of the post it was referring to? ◦ Yes, I no longer believed the post ◦ Yes, I 

believed the post more ◦ No, I already didn’t believe 

the post ◦ No, I still believe the post ◦ Other 

10. Have you seen this banner on Twitter before? 

◦ Yes ◦ No ◦ I don’t know 

11. You said you’ve seen this banner on Twitter before. 

What did you think or feel about it? 

12. How helpful was this banner? ◦ Extremely helpful ◦

Very helpful ◦ Somewhat helpful ◦ Slightly helpful 

◦ Not at all helpful 

13. Did you click on this banner? ◦ Yes ◦ No ◦ I don’t 

know/remember 

7 
14. Have you seen this ”Manipulated media” label on 

Twitter before? 

◦ Yes ◦ No ◦ I don’t know 

15. You said you’ve seen this ”Manipulated media” label on Twitter before. What did you think or feel 

about it? 

16. How helpful was this label? ◦ Extremely helpful ◦

Very helpful ◦ Somewhat helpful ◦ Slightly helpful 

◦ Not at all helpful 

17. Think of a recent time when you saw this label. Did 

the label change your view of the post it was referring to? ◦ Yes, I no longer believed the post ◦ Yes, I 

believed the post more ◦ No, I already didn’t believe 

the post ◦ No, I still believe the post ◦ Other 

18. Have you seen this banner on Instagram before? 

◦ Yes ◦ No ◦ I don’t know 

19. You mentioned that you’ve seen this banner on Instagram before. What did you think or feel about 

it? 

20. How helpful was this banner? ◦ Extremely helpful ◦

Very helpful ◦ Somewhat helpful ◦ Slightly helpful 

◦ Not at all helpful 

21. Did you click on this banner? ◦ Yes ◦ No ◦ I don’t 

know/remember 

22. To the best of your knowledge, have you believed 

any misinformation about COVID-19? ◦ Yes ◦ No 

For the following questions, think of one instance 

where you had believed misinformation about 

COVID-19. 

23. Where did you see/hear this misinformation? ◦ Social media site (e.g. Facebook, TikTok, etc.) ◦ News 

site ◦ Messaging app/group chat ◦ Word of mouth ◦

I don’t know/remember ◦ Other 

24. How did you discover this information was false? 

(Select all that apply.)  I knew it wasn’t true  The 

website/platform itself labeled it as misinformation 

 I conducted a web search (e.g. checking for multiple sources that agree or a fact-checking site)  I 

looked at a trusted health site (such as WHO.int or 

CDC.gov)  I saw a social media comment where 

someone debunked it  Someone told me directly it 

was false (in-person, personal chat, etc.)  I don’t 

know/remember  Other 

25. What was the misinformation? 

26. What did you do when you realized this information 

was false?  Nothing  Unshared it if you had 

shared it (please elaborate):  Shared the correction 

(please elaborate):  Other (please elaborate):  I 

don’t know/remember 

27. To the best of your knowledge, have you noticed 

others sharing misinformation about COVID-19? ◦

Yes ◦ No 

28. For the following questions, think of one instance 

where you had noticed others sharing misinformation about COVID-19. 

29. Where did you see/hear this misinformation? ◦ Social media site (e.g. Facebook, TikTok, etc.) ◦ News 

site ◦ Messaging app/group chat ◦ Word of mouth ◦

I don’t know/remember ◦ Other 

30. How did you discover this information was false? 

(Select all that apply.)  I knew it wasn’t true  The 

website/platform itself labeled it as misinformation 

 I conducted a web search (e.g. checking for multiple sources that agree or a fact-checking site)  I 

looked at a trusted health site (such as WHO.int or 

CDC.gov)  I saw a social media comment where 

someone debunked it  Someone told me directly it 

was false (in-person, personal chat, etc.)  I don’t 

know/remember  Other 

31. What was the misinformation? 

8 
32. What did you do when you realized this information was false?  Nothing  Corrected the person 

privately (please elaborate):  Corrected the person 

publicly (please elaborate):  Other (please elaborate):  I don’t know/remember 

33. Please check below all of the COVID-19 rumors 

that you have heard (whether or not you thought 

they might be true). 

(Please be aware that these are all false rumors. For 

up-to-date information on the virus, please go to 

WHO.int or CDC.gov.)  The novel coronavirus 

sickness is caused by 5G  There’s a plot to “exterminate” people infected with the new coronavirus 

 Scientists have proven that humans got the novel 

coronavirus from eating bats  Scientists predicted 

the virus will kill 65 million people  China built 

a biological weapon that was leaked from a lab in 

Wuhan  Chinese spies smuggled the virus out of 

Canada  A coronavirus vaccine already exists 

There were 100,000 confrmed cases in January 

A teen on TikTok is the frst case in Canada  There 

will be a mass quarantine and martial law in a certain state (e.g. Washington)  Other 

34. Can we use anonymized quotes from your freeresponse answers in future research publications? ◦

Yes ◦ No 

35. In which country do you currently reside? ◦ United 

States of America ... Zimbabwe 

36. In which state do you currently reside? ◦ Alabama 

... I do not reside in the United States 

37. What is your political affliation? ◦ Democrat ◦ Republican ◦ Independent ◦ Other ◦ Not applicable 

38. What gender(s) do you identify as?  Male  Female  Non-binary  Prefer to self-describe: 

39. What is your age? ◦ 18-24 years old ◦ 25-34 years 

old ◦ 35-44 years old ◦ 45-54 years old ◦ 55-64 

years old ◦ 65-74 years old ◦ 75 years or older 

40. Anything you want to tell us? (Not required.) 

9 
B Inter-Rater Reliability 

Table 5 shows the inter-rater reliability percentages for our qualitative codes. 

Code Percent Agreement Cohen’s Kappa 

Nothing/Neutral 96.52% 83.73% 

Positive 100% undefned* 

Fear/worry about the future (generic) 99.13% 90.46% 

It’s common to see these banners now 100% 100.00% 

Thought it was a ad 98.26% 65.77% 

Unnecessary because I’m already informed 98.26% 86.58% 

Annoying 99.13% 88.44% 

Done too late 97.39% 65.33% 

Cautious/suspicious of the banner 95.65% 42.44% 

I don’t trust the company, so I do not trust the banner 99.13% 79.57% 

It looked offcial 99.13% 66.28% 

Angry 100% 100.00% 

Don’t know enough to comment on execution 100% 100.00% 

Could be abused to censor 100% 100.00% 

I ignored it 98.26% 49.12% 

Other 89.57% 8.73% 

Table 5: Inter-rater reliability percentages. 

*http://dfreelon.org/2008/10/24/recal-error-log-entry-1-invariant-values/

10