Narrative Style and the Spread of Health Misinformation on Twitter
Authors: Achyutarama R Ganti
Eslam Ali Hassan Hussein
Steven R. Wilson
Zexin Ma
Xinyan Zhao
Venue: EMNLP 2023 Findings (2023)
Keywords: narrative communication
misinformation
computational social sciences
natural language processing
linguistic analysis
classification

Abstract: Using a narrative style is an effective way to communicate health information both on and off social media. Given the amount of misinformation being spread online and its potential negative effects, it is crucial to investigate the interplay between narrative communication style and misinformative health content on user engagement on social media platforms. To explore this in the context of Twitter, we start with previously annotated health misinformation tweets (n ≈15,000) and annotate a subset of the data (n=3,000) for the presence of narrative style. We then use these manually assigned labels to train text classifiers, experimenting with supervised fine-tuning and in-context learning for automatic narrative detection. We use our best model to label remaining portion of the dataset, then statistically analyze the relationship between narrative style, misinformation, and user-level features on engagement, finding that narrative use is connected to increased tweet engagement and can, in some cases, lead to increased engagement with misinformation. Finally, we analyze the general categories of language used in narratives and health misinformation in our dataset.

Narrative Style and the Spread of Health Misinformation on Twitter
Achyutarama R. Ganti", Eslam Hussein Steven R. Wilson', Zexin Ma', Xinyan Zhao 'Oakland University, VVirginia Tech, 3 University of Connecticut, University of North Carolina ganti@oakland edu, ehussein@vt. edu, stevenwillon@@okkladd.. edu, zexin. www.mmmm..meaaeoooooooooo ezhao@unc.
Abstract
narrative tweet is, "We got our first dose of the vaccine! Can't 't tell you how excited and thankful we feel, and can't wait to get the next dose".
Using narrative style is an effective way to communicate health information both on and off social media. Given amount of misin- formation being spread online and its poten- tial negative effects, is crucial to investigate the interplay between narrative communication style and misinformative health content on user engagement on social media platforms. To ex- plore this in the context of Twitter, we start with previously annotated health misinforma- tion tweets (n 15, 000) and annotate a sub- set of the data (n 3, 000) for the presence of narrative style. We then use these manually assigned labels to train text classifiers, exper- imenting with supervised fine-tuning and in- context learning for automatic narrative detec- tion. We use our best model label remaining portion of the dataset, then statistically analyze the relationship between narrative style, misin- formation, and user-level features engage- ment, finding that narrative use is connected to increased tweet engagement and can, some cases, lead to increased engagement with mis- information. Finally, we analyze the general categories of language used in narratives and health misinformation in our dataset.
Considering the power of narrative information in influencing individuals' health-related beliefs, attitudes, and behaviors (Ma and Yang, 2022; Mur- phy et al., 2013), important to understand and address narrative misinformation on social media. In particular, research has found that online health misinformation is often created using narratives (Peng al., 2022). Further, emerging research in- dicates that individuals are less likely to discern verify misinformation when it presented in nar- rative (vs. non-narrative) format, possibly because narrative misinformation perceived as more relat- able and engaging (Zhao and Tsang, 2023). There- fore, is crucial to classify misinformation as nar- rative and non-narrative and examine how this af- fects user engagement social media.
Misinformation is defined as false or misleading information that is contrary to the consensus of the scientific community based on the best avail- able evidence at the time (Vraga and Bode, 2020). The prevalence and diffusion of incorrect informa- tion and fabricated narratives on social media have been a growing concern (Southwell al., 2022). According to one study, 28% of the most viewed coronavirus videos included non-factur informa- tion, totaling 62 million views as of March 21, 2020 (Li et al., 2020). US-based respondents of another survey estimate that about 33% of social media con- tent that they view contains some misinformation (Nielsen et al., 2020).
1 Introduction
A narrative is a fundamental form of human com- munication. Although colloquially used inter- changeably with "story"" the formal concept of of
narrative is defined more Proally-cossisee both story (connected events and characters) and narrative discourse (how the story is told) (Abbott, 2008). In other words, a narrative represents the presentation of a sequence events experienced by a character or characters (Bilandzic and Bus- selle, 2012a; Dahlstrom, 2021). This study uses this definition to label and classify tweets into nar- rative and non-narrative based on whether or not they are written in narrative style. An example
How individuals assess and act on misinforma- tion can be affected by many factors, such as mes- sage features, source credibility, and individual mo- tivations and emotions (Marsh and Yang, 2018). Existing research has shown that individuals are particularly susceptible to narrative misinformation that uses negative emotions such as anger or fear, in- correct personal stories, and made-up up testimonials
Equal contribution, listed in alphahetical order.

---

(Zhao and Tsang, 2023; Lee and Shin, 2022). Yet, most studies were conducted in a laboratory setting, limiting their ecological validity and generalizabil- ity. Further, prior studies have not quantitatively examined how social media misinformation in the narrative (vs. non-narrative form affects user en- gagement at scale. Therefore, in this study, we use a large set of health tweets to investigate the pres- ence of narrative misinformation and its impact on user engagement, and explore the features of both narratives and misinformation on Twitter.
pins on pinterest used more narrative information while pro-vaccine used more statistical information, and that the latter type had higher levels of user en- gagement. Additionally, Betsch et al. (2011) stud- ied the influence of narratives on the perception of vaccination risks, and highlighted the effectiveness of narrative communication in inftuencing human decision making about vaccine risk.
Issues with the spread of misinformation on plat- forms like Twitter are also widely researched. For example, Sharevski et al. (2022) studied COVID- 19 misinformation on Twitter and the change in user perception of COVID-19 vaccine content as a result of soft moderation of misinformation content on the platform. The spread of misinformation on social media is not just limited the common pub- lic: study done by Shahi al. (2021) reveals that verified Twitter handles waaleerrrraaiittton/eeeeeetiiiie are also responsible for creating and spreading mis- information. The authors also observed that fake claims propagate faster than semi-fake claims, and tweets containing misinformation often attempt to discredit other information social media.
The contributions' of this work are (1) narra- tivity annotations for two existing misinformation datasets, which can be used to study the interaction between narrative and misinformation Twitter; (2) experiments with several types of models for narrative detection, and the use of the best model label all of the available data; and (3) analy- ses the relationship between narrative style and user engagement in the presence of misinformation, as well as variations in the categories of language used in both narrative and non-narrative tweets. We find that tweets containing narratives typically had higher engagement, while the presence of mis- information was associated with less engagement. Especially in the context of vaccine-relatee content, using narratives was related to increased engage- ment with misinformation, even while controlling for number of followers. Additionally, through our analyses, we find supporting evidence for existing studies on the linguistic features of narratives even in the social media context.
While most studies of narratives have relied on manual annotations, Dirkson et al. (2019) and Ver- berne et al. (2019) used human-assigned labels to train text classification models to detect narrativity in social media posts in the health domain. Both studies focused on bag-of-words models with n- gram features, while more recent research found that deep learning models were more successful than classical machine learning models at detecting narratives in Facebook posts (Ganti al., 2022). These studies suggest that automatic narrative clas- sification could enable the labeling of even larger social media datasets.
2 Related Work
Prior work established the importance of narratives both on social media and in the context of sharing health-related information. On Twitter, specifically, Dayter (2015) analyzed multi-tweet stories to study the importance of storytelling for identity construc- tion. In the health domain, Overcash (2003) em- phasized the importance of narrative research in developing healthcare services for patients, while Andy et al. (2021) studied how users on online health communities show support for one another. Prior research suggests that narratives are more effective at communicating health risks than non- narratives (Janssen et al., 2013; Ma, 2021) and promoting health behavior change (Kreuter et al., 2010). Guidry et al. (2015) found that anti-vaccine
Data Collection and Annotation
We leveraged existing misinformation datasets and provided new annotations for the presence of narra- tives to create a single dataset that contained both sets of labels for each text: misinformation (or not) and narrative (or not).
3.1 Data Collection
We selected two datasets: ANTiVax (Hayawi et al., 2022), and CMU-MisCov19 (Memon al., 2020) based on three criteria: the quality of the annota- tion, relevance to the topie investigated, and the amount of data. Each dataset was annotated for medical misinformation by subject matter experts.
Cooee and other resources related to this work available at: https:/ //ti /gg com/ou-nete ooooooonllll wwwooounnnnnnneee http:/aarrissiniofr.... sinfoEMLLP23

---

Table shows the number of tweets collected by the authors of each dataset and the number of hy- drated tweets that were available at the time of our data collection (September 2022), which is the set tweets used this work. Most of the unavailable tweets fall under the misinformation category.
pha 0.71. If the two annotators agreed. the agreed upon label was used, otherwise the third annotator stepped in to break the tie and the majority label was used in our final dataset.
4 Narrative Detection
The ANTiVax dataset was collected to train machine learning algorithms to classify and de- tect COVID-19 vaccine misinformation. Authors used the Twitter API to collect tweets related to the COVID-19 vaccines using keywords (e.g.. vac- cine, Pfizer, Moderna). Tweets were annotated into misinformation (misinformed) tweets and general vaccine-related tweets (informed), and annotations were verified by medical experts.
Given the annotated tweets, we then set out to train classification models that could be used to label the entirety of the misinformation datasets that we collected.
4.1 Classification Methodology
The annotated data were shuffled and split 80- 10- 10 for training, validation, and testing. We explore three categories of models: classical machine learn- ing models with bag-of-words features, fine-tuned transformer encoder models, and auto-regressive generative models with in-context learning. All models are trained three times with different ran- dom seeds, and the average results are presented.
The CMU-MisCov19 dataset was collected based on tweets of two groups: informed and misin- formed users who wrote tweets related COVID- 19. Authors used the Twitter API to collect tweets related to COVID-19 using keywords (c.g., coro- navirus, covid). Tweets were annotated into five false information categories and five true informa- tion categories, which we merge into one misin- formed and one informed category, respectively,
The oot-ooff-oro-bbaae machine learning mod- els we used were scikit-learn's 's (Pedregosa et al., 201 1) implementations of Naive Bayes, Support Vector Machine classification, and Logistic Regres- sion. We conducted hyperparameter tuning as out- lined in Appendix B.
3.2 Annotation Process
Our annotation guidelines were developed itera- tively through discussion and revision. During each our four pilot annotation phases, three of the au- thors annotated between 20-100 randomly sampled tweets for narrative style. After each round of an- notation, the annotators met together along with experts in narrative communication to discuss the annotated tweets, and updated the annotation guide- lines from the previous round. After four rounds, all authors reached a consensus on the final annota- tion guidelines (appendix A).
For the transformer-based deep-learning models, considered BERT (Devlin et al., 2019). DistilBERT (Sanh et al., 2019), RoBERTa (Liu et al., 2019). CardifNLP's TWitteRooBRRTa al., 2020) and DeBERTa (He et al., 2021) models available on HuggingFace Wolf et al. (2019) using bert-base-uncased, roberta-base, and
(Barbieri
disti sestibbert-base-uncseedd cardi tarr checkpoints, each
mi microsoft//eeerta--i33 with their default tokenizers and the output of the [CLS] input token as the input to trainable classification layer. (training hyperparameters listed in Appendix B).
After finalizing the annotation guidelines, we sampled 3,000 tweets (20.6% of the 14,561 hy- drated tweets in our dataset) stratified with respect to dataset and misinformation stance. Each tweet initially independently annotated by two of the three annotators who developed the guidelines, with agreement measured using Krippendorff's al-
For the generative models, we used the and GPT-3 (Brown et al., 2020) text-davinci -003 models from the OpenAl API Both models take in an instruction or prompt as the input and respond with completion to match our context question. The goal of this experi- ment was investigate if auto-regressive models such as GPT-3 and GPT-3.5-Turie are capable of detecting the presence of narratives tweets with little or no training data, which, very successful,
VNTiVa
CMU-MisCov19
Combinee
Misin!
Info
Misint
Total
Info
MMisint]
Total
Info
Misint] Toral
Collected
2146
7174
Hyyratedd
8059
3939
11998
1772
791
2563
9831
4730 14561
5.4%
%6 unavail
13.%
51.5% 20.4%
17.4%
44.45
28.2%
14.3%
34.1% 21.9%
Table I: Number of available tweets in both datasets after rehydration.
2nttps://openal: cotpp/oducco///uuuctcccc

---

GPT33 (Davinel)
GPT-J.S-Turbo
Delinttion: (1) Acconting Kiester et (2007) a nemetive is derined ax representation of connected events and characters that hos an denertable structuro
Prec
Rec
Prec
Rec
iounded in space and time, and comsins implice explect menrages about the taple gi assressed. (?)
Pero-4hot
0,484
0.9338
0.326
0.302
01402
0.241
+del
0.939
0.080
0.040
(2013)
+instr
0.328
0.801
0.206
0.240
0.400
0.171
eventta) experienced by specific characterk) in switing
+dof-instr
0.864
0.466
(1) Acconting Datrntioo (2021), nomiitin detooed we mensage the destribee exporianos scedits characters acioss serios rolated events over delineel aino period intumiitate character oo colltityy and lemporality At core, nomativv experience somsitinng.
Tew-shot
0.699
0.835
0.601
0.384
0.460
0.799
edef
0.697
0.879
0.578
0.675
0.598
0.780
+imst
0.698
0.941
0.555
0.642
0.554
0.763
+def-instr
0.712
0.952
0.570
0.636
0,546
0.762
follows: (noemally parson) exper a sories related events. You may assure that the presence of muliiplo evenss impl temporalit and de not spec tica by check for temporality during annotation.
Based defirdions.
speciiic
Table 2: Narrative class FI, Precision and Recall, eraged across three runs, for GPT-3 models trained on detecting narratives using the validation set only. Bold indicates the top score for each metric per model.
Tweet: My good frand was wacdr uaad Joday as one the first peocle eliiblle for Bocing trose around gedingg inmunized is bringing so much hope, and I'm excited
opportunity soon.
Tweet an encouriinng ni move, a mejor sitto shoring platform has profitited videos abourt canapinecy thacrtes incommetly linking CDDVEE 15 symptons to 5G ratworks. Thos not the dime not plece talk about the effects of EM warvss on people and animala. though in importent topic surte this narsativ?? No
Model
FT
Proe
Recall
Logistic Reg
0.704
0.800
0.629
GPT:3
0,718
0.977
0.568
Bayes
0.734
0.708
0.761
SVM
0.775
0.680
0.900
Tweet: wonk ls shop loday, but Bill Gatos already know that duo bucking sweet narrativ??
BPRT
0.8%6
0.873
0.900
RoBReT
0.837
0.885
0.838
DistiBERT
0.893
0,900
0.886
DeBERTa
0.910
0.907
0.914
RoBERTs
0.924
0.918
0.931
Figure I: Prompt components for GPT Models. From top to bottom, the blocks display the definition (blue), instructions (orange), few-shot examples (green), and the target tweet to be classified (yellow).
Table 3: Narrative class F1, Precision, and Recall scores, averaged across three runs for the text classification models. The best score for each metric is in bold. GPT- 3 indicates the result of the best GPT-3 model based various prompting schemes.
could reduce or avoid the entire manual annotation process that was required to obtain our initial set of narrative labels.
The experimental setup for the reto-regee models consists of two main settings: zero-shot and few-shot. In the zero-shot setting, the model is shown the target tweet and asked "Is this tweet a narrative?", while in the few-shot setting, sev- eral example tweets with their correct labels are prepended to the prompt in a question-answw for- mat, with the labeled examples being evenly dis- tributed across the two datasets and narrativity la- bels, i.e. n narratives and n non-narratives from each dataset are used, leading to 4n total examples. We tune the value of n using the validation set and set n - 5 for our experiments. Within each setting (zero- or few-shot), we also experimented with the inclusion of the definitions of narratives and addi- tional guidelines that the annotators developed and used during the annotation process. A summary of the components that can be included or removed from the prompt is outlined in Figure 1.
experiments is reported in Appendix C. The GPT- based text-davinci-003 model outperforms ppt-3.5-turbee in nearly all of the zero-shot and few-shot experiments. The best overall GPT model, based on fext-davinci-000 used few-shot learn- ing and included the narrative definitions and label- ing guidelines in the prompt, and we evaluate this model on the test set (GPT-3 in Table 3).
The results for all models on the test set are presented in Table 3. Our findings align with those of Ganti et al. (2022), who found that transformer- models consistently outperform bag-of- models for narrative detection, with the fine-tuned base RoBERTa model performing best overall, outperforming even the version of RoBERTa pretrained specifically on Twitter data. The GPT-3 text text-davinci-e03 model achieved high precision but lacked in recall, resulting in an Fl-score similar to the bag-of-words models, though it requires less than 1% of the amount of training data (in the form of examples for in-context learning). In the end, we use our top perform- ing finettuned RoBERTa model to label our entire dataset, which we analyze in the next section.
4.2 Experimental Results
Given that we explore eight configurations for the GPT models, we first use the validation set to select the best performing setup (Table 2). The Fl scores for both the GPT models across all the few-shot

---

Dilasets
ANxiVax
CMU-MisCov19
Combined
Info or Misinto
Info
Misinf
Tonal
Info Misint
Total
Info
Misinf
Total
Facerte
Narr
5077
492
5569 274
78
352
5351
570
5921
Narratvve
Noa
2982
3447
6429
1498
713
2211
4480
4160
86-00
Total 8059
3939
11998
1772
791
2563
9831
4730
14561
Table Number of tweets in both datasets after apply- ing the fine-tuned RoBERTa model on the unlabeled portion of the dataset.
5 Narrative and Misinformation
(a) favorite count
In this section, we investigate the effect of narrativ- ity and misinformation on user engagement metrics and variations in linguistic features. First, alyze the distribution of each category of tweets the full set of ~ 14.5k tweets labeled by our annotators and the best RoBERTa model from the previous section. Second, we discuss how the di- mensions of narrativity and misinformation affect user engagement (i.e., counts of retweets and likes). Finally, we investigate how linguistic variations significant between narratives and narraarraivvss and between informed and misinformed tweets.
Relmnd
3.0
BB6 NIT
Wore
YO vor
(b) retweet count
Figure 2: Box plots of (a) favorite count and (b) retweet count each of the datasets under each misinformation and narrativity category. Note: outliers are removed.
5.1 Data Analysis
After applying the best model the unlabeled portion of the dataset, we have the narrativity dis- tribution in table 4. Where 46.4%, 13.7%, and 40.7% are narrative tweets of the ANTiVax, CMU- MisCov19, and combined datasets, respectively. Narrativity. Although the ratio between in- formed and misinformed tweets in the ANTi Vax, CMU-MisCov19 datasets are 2.07 and 2.24, respec- tively, we notice that narrativity used more in in- formed than misinformed tweets, where informed narratives are 10.32, 3.51, and 9.39 times more common than misinformed narratives the AN- Vax, CMU-MisCov19, and combined datasets, respectively. On the other hand, the non-narrative style is used more in misinformed tweets in AN- Tixax and more in informed tweets in the CMU- MisCov19 19 and combined datasets. We see that narrativity more commonly used communi- cation style among informed users to share their vaccination and CO experiences than mis- informed users. We also observe that narrativity is less prevalent in the CMU-MisCov 19 than the Vax dataset, with narrative to non-narrative ratios of 0.27 and 1.14, respectively.
have ratios of 1.70 and 0.18 in each dataset, respec- tively, while the ratio of narrative to non-narrative misinformed tweets is 0.14 and 0.11 in each datasct, respectively.
We believe the discrepancies in percentages be- tween both datasets can attributed to the nature of each dataset, where ANTi Vax was collected to study COVID-19's vaccine misinformation specifi- cally, and CMU-MisCov19 was collected to study misinformation related to COVID-19 in general. Hence, VaTiVax may contain more vaccination experiences that were shared by users in narrative style compared to CMU-MisCov19, Also, each dataset was annotated by different annotators and annotation methods.
5.2 Engagement with Narrative Tweets in the Presence of Misinformation
In this section, we investigate how narrativity and misinformation affect users' engagement (i.e., likes/favorites and retweets) with tweets. Our al- ternate hypothesis is that narrativity and misinfor- mation are related to an increase or decrease in the count of likes (favorites) and retweets of tweet. We control for the number of followers of tweet's author since this might also affect the engagement with their tweets.
Misinformation. Informed tweets are more prevalent in the ANTiVax and CMU-MisCov 19 datasets by a ratio of 2.05 and 2.24, respectively, where narrative to nor-narrative informed tweets
Figure 2 depicts box plots of counts of favorites

---

Engagenent
Favorite
Retweet
the of coefficients for one model listed in each column. We normalized each of the numbers of followers, retweets, and favorites so that they are in the range [O-1]. Similar trends can generally be observed in the cases of favorites and retweets. Tweets that use misinformation had a lower en- gagement, and tweets with narrative had higher engagement, but tweets that used narratives in the context of misinformation (narrative x misinfo) had higher engagement only in the ANTiVax dataset. This suggests that false narratives about receiv- ing vaccinations got more engagement than nar- ratives containing misinformation in a general set of COVID-19 tweets, which actually got less en- gagement. When considering follower count and its interaction with these variables, raw follower count does have a positive relationship with the number of likes or retweets that a tweet gets, and this effect actually amplified if the tweet con- tains misinformation (misinfo x follower count) and even more so if the tweet uses narrative style (narrative x misinfo x follower count).
CMU-
CMU-
Datase
ANTTVa
MisCov19
Comb.
ANTIVa
19sCov19
Comb.
intercept
3.03
3.40
3.2
1.27
2.1
1.65
misinfo
-0.47
0.64
-0.12
0.35
0.36
0.21
narrative
0.28
.0.4
misinlo
0.04
-3.5
-0.99
0.20
-288
-0.25
narrative
Tollower cound
3.58
5.15
5.26
4.53
5.50
5.88
marrativ
23.92
-0.93
21.73
-0.52
1466
follower count
misinfo
follower count
4,88
17.76
11.65
3.11
17.16
9.63
narrative x
misinfo
1500.48
1962.69
1546.66
1632.43
2070.20
1679.24
follower count
Table 5: Coefficients of variables resulting from the Poisson regression GLMs with log link function. All coefficients are significant with p << 0.01. Positive (negative) values indicate positive (negative) associa- tion with the outcome variable.
(likes) and retweets after removing outliers. Mis- informed tweets receive significantly fewer likes than informed tweets, regardless of their narrativ- ity, across both datasets. Narrative tweets typically received more favorites than their non-narrative counterparts within a given dataset and the absence or presence of misinformation. For the ANTi- Vax dataset, informed narrative tweets have the least number of retweets (zero) compared to the other three categories of tweets in that dataset. For the CMU-MisCov19 19 dataset, informed non- narrative tweets have a higher number of retweets than the other groups, where both misinformed nar- rative and nar-narrative tweets typically received no retweets.
5.3 The Language of Narratives and Misinformation
To study the linguistic variations between narrative and narraavaeive tweets and between informed and misinformed tweets, we analyzed tweets from each pair of groups using the 2022 version of the Linguistic Inquiry and Word Count (LIWC) pro- gram (Boyd et al., 2022). LIWC text analysis tool that examines different lexical categories con- sidered psychologically meaningful. It calculates the percentage of words from various lexical cate- gories that are present in given text. We ran the LIWC analysis on all tweets from the two datasets.
Next, we built statistical models of the relation- ships between narrativity, misinformation, follower count, and user engagement. Given that distribu- tions of both likes and retweets follow a Poisson distribution with many tweets having value of zero likes or retweets, and a heavy right skew for the other values, we utilize generalized linear mod- els (GLM) (Nelder and Wedderburn, 1972) with a Poisson distribution (Adelson, 1966; Thompson, 2001) (as our dependent variables are counts) and the canonical log link function (Manning and Mul- lahy, 2001). We fit six GLM model on the vari- ables: for each dataset (and for both combined), we fit one model with likes as the outcome variable and another with retweets the outcome. The independent variables are the annotations for mis- information and narrativity, the count of followers, and the interactions between those three main vari- ables.
We use the nonparametrie independent Mann- Whitney U test (Mann and Whitney, 1947) to es- tablish statistical significance by determining the difference in means. The mean LIWC score for each category served as the test statistic. Since perform extensive set of statistical tests, we may find significant differences stemming from type errors (i.e., false positives). Hence, we apply the Bonferroni correction method (Dunn, 1961) to avoid such errors by setting a = Ont5/nt, where nt is the number of tests performed. Then we as- sume the significance of the difference between means if p-value < a (i.e., accept H, hypothesis).
We follow three approaches to examine the con- tent of tweets that share narratives: based on as-
Table 5 shows the results of the GLMs, with

---

Datasct Info or Misinfa
ANTiVax
CMU-MisCorly
Combined
Dataset Nartaliiity
ANTIVIA
CMU-M
Combined Non Nart
Info Misinfo
All
Info Misinfo
All Info Misinfo
All
Non Narr
All Non Narr
All
All
LIWC
LIWC
R
calegury
caregory
WC
1.5 1.38
1.34
1.3
1.36
1.27
1.12
1.23
1.22
1.22
1,00
1.14
Analytic
1.19
1.1
1.11
1.17
1.13
Analytic
0.82
0.83
0.85
0.85
0.84
0.83
0.83
Authenric
0.63 0.66
0.52
0.81
0.79
0.69
0.66
0.55
Authenti
1.88
1.48
1.54
1.83
tonc pane_pos____
0.35
0.38
0.44
tone pos
2.74
2.9
2.44
1:2
2.59
tone neg
0.99 1.53 1.39
1.46
1.3
neg
0.52
0:55
0.54
0.79
0.57
smo_pos
0..44 0.15
0.47
0.16 0.14
ema pas
5.71
2.0
7.71
1.71 6.22
2.06
7.63
emno Reg
0.65 1.11
0.74
1.1
pog
0.49
0.83
0.6
0.71
0.72
0.55
0.82
0.63
bealth
0.89 0.74 0.86
1.17
0.75
0.9
health
1.03
0.86
1,13
0.86
1,11
death
17.2 29.4 18.8
4,88
16.25 7.0
deuh
1.71
0.35
1,79
1.53
0.5
1.67
0.43
Conversation
1.03 1.02 1.14
1.01
1.02 1.12
function
1.16
1:23
1.18
1.1
1.12
swear
1.14
0.38
pronoun
1.38
1.25
1.5
1662
1.53
1.61
1.44
1:29
1.52
function
0.9 0.91 0.88
0.93
0.91
0.92 0.88
article
0.73
0.73
0.79
0.77
pronoun
0.81 0.74 0.67
0.85
0.76
0.71
prop
0.75
1.2
0.89
0.8
L.18
0.91
family
0.32
0.41
0.33
auxverb
0.85
0.87
0.85
friend
0.22
0.38
conj
1.13
Degato
0.47
0.45
0.44
0.46
Table 7: Summary of statistical testing for significance of LIWC categories related to misinformation using Mann-Whitne Whhntey test. R = Hmmisin fin the ra- tio between the means of LIWC category for misin- formed and informed tweets, respectively. denotes non-significant results, i.e. p-value a. Blue denotes R 2 (i.e., the category is used more in misinformed tweets than informed ones) and Red denotes R < 1.
family
4.42
3.4
4.76
5.0
5.13
4.67
3.43
5.06
friend
L.67
3.5
2.5
4.0
4.5
4.0
1.83
4.5
2.5
Cognition
0.59
0.88
0.6
0.89
0.63
ppron
L.5
1.73
1.97
195
2.04
2.0
1.72
1.77
2,03
Time
1,26
1.1
1.34
1.37
1,34
1,09
1.09
1.15
cause
0.76
0.49
Table 6: Summary of statistical testing for significance of LIWC categories related to narratives using Mann- Whitney U test. R = Pnarr/ / Pron. the ratio between the means of a LIWC category for narrative and non- narrative tweets, respectively. denotes non-significant results, i.e., p-value 2 a. Blue denotes 12 2 I (i.e., the category is used more in narratives than non- narratives) and Red denotes I <1 1. WC = word count.
Pennebaker (2011); Boyd et al. (2020) and the defi- nition of a narrative (Dahlstrom, 2021), where such categories are often used to represent characters. On the other hand, cause words show insignifi- cant differences in most tests significantly fewer cause words in the ANTiVax dataset. We believe the discrepancy in causality results from the high amount of non-narrative information that explicitely uses causality- language ("5G causes COVID-19!"), 19!"), skewing the ratio of casual language away from the narrative class.
pects of our definition of narrative used in annota- tion, based on the analysis of Boyd et al. (2020) about how narrative is structurally and linguisti- cally built, and we follow an approach that exam- ines LIWC linguistic dimensions, slightly similar to the approach proposed by Memon and Carley (2020) which examines the linguistic dimensions between Twitter communities that share narratives about COVID-19.
5.3.2 Narrative Arc
In LIWC'22 (Boyd et al., 2022), Narrative Arc analysis was introduced to understand and mea- sure the text's narrativity (Boyd et al., 2020). The Are of narrative is defined as three stages: I. Staging, where storyteller uses more articles and prepositions introduce their narrative. 2. Plot Progression, where staging declines and uses more words pronouns, auxiliary verbs, conjunctions, and negations - which signal events and who is in- volved in those events and how events progress. 3. Cognitive Tension, where more Insight, Causation, Discrepancy, Tentative and Certitude words are used to describe psychological tension and conflict, where characters strife to achieve possible goals. While we found tweets to be too short to use the full narrative are feature of LIWC'22, we examine the scores for several of the important categorics found to be related to narratives.
5.3.1 Narrative Definition
Dahlstrom (2021) defines a narrative a triad of character, causality, and temporality, where character shares a set of related (i.e. causality) events and experiences over a period of time (i.e. temporality). Based on that definition, we uti- lize LIWC categories that correspond to charac- ters (pronour), causality (cause), and temporality (Time). In table 6, Narrative tweets show signifi- cantly higher usage of pronouns and Time words than non- narrative tweets, which aligns with the definitions. We also note that family (e.g., par- ent, baby, son) and friend (c.g., dude, boyfriend) categories, indicative of characters, are used sig- nificantly more when in expressing narratives than non non-narratives for both datasets. This aligns with

---

Staging. From table 6, we can see that the use of articles and prepositions significantly differs be- tween narrative and non-narrative tweets for the whole dataset. Where narratives use fewer words of those categories than non-narrative.. For CMU- MisCov 19, all tests show no significant differences between narrative and non-narratives. For the AN- Vax dataset, only the misinformed tweets show significant differences in the use of articles and more use prepositions.
lone_neg, where the higher the score, the more pos- itive or negative the tone is, respectively. Emotion measures how much a text has positive or nega- tive emotions (e.g., anxiety, anger, sadness) that correspond to the LIWC categories emo_pos and emo_ _eo__neg respectively. According to table 6, narra- tive tweets significantly share more positive tones and emotions than non-narrative tweets in ANTi- Vax, while it's insignificant in almost all cases for CMU-MisCov19, 19. While negative tones and emotions are more commonly shared among non- narrative tweets. Table 7 shows that in ANTiVax dataset, misinformation was not as clearly related to tone, though in cases where significant results were found, positive tone and emotion were used less in tweets that contained misinformation. While for CMU-MisCov 19, it is all insignificant.
Plot Progression. Only pronouns show higher significant usage in narrative tweets. Contrary to pronouns, the other categories show insignificant differences in most tests or significance when fewer words are used in narrative than in non-narrative tweets, except for conjunctions.
Cognitive Tension. In LIWC'22, Cognition is a super category that sums the indices of all in- sight, causation, discrepancy, tentative, and cer- titude subcategories. From table 6, cogmition is significantly lower in narrative than non-narrative tweets for the ANTiVax dataset, while it's insignif- icant for CMU-MisCov 19.
Health and Death. The health LIWC category contains 715 words related to health, illness, well- ness, and mental health (c.g., medic, patient, hos- pital, gym, ... etc). The death category has 109 words (e.g., dead, kill, etc). From table 6, find that health words overall significantly used more in narrative than non- narrative tweets, while death words are overall used more in non- narrative tweets, and that might be attributed to having many tweets where news agencies sharing COVID-19-related deaths in a non-narrative formal style. Investigating how health and death are shared among informed and misinformed tweets, we find in table 7 that the CMU-MisCovI9 19 dataset mainly shows insignificant differences in means. While for the ANTiVax dataset, misinformed tweets share fewer health words and many more death words than informed tweets, which shows that COVID-19 vaccine-related misinformation is often focused on death and negative side effects.
We believe that the short length of tweets forced by the platform hinders Twitter users from using more words that would fall under the LIWC cate- gories proposed by Boyd et al. (2020) which were intended to measure narrative ares in longer sto- ries. This also justifies the discrepancies between our results and those of Boyd et al. (2020), where they concluded those categories after analyzing large set of corpora collected from long novels, film transcripts, short stories, etc., where the average word count ranges from a few hundred to tens of thousands of words.
5.3.3 Linguistic Dimensions of Narratives and Misinformation
Analytical and Authentic. The LIWC categories Analytic and Authentic are two summary variables that measure 1) logical and formal thinking and 2) perceived honesty and genuineness, respectively. We observe from table 6, that narratives have signif- icantly lower analytic and higher authentic scores than non-narratives in both datasets. This also aligns with Boyd et al. (2022); Memon and Car- ley (2020). Additionally, we notice from table 7 that misinformed tweets are less honest and more analytical in both datasets than informed tweets. Tones and Emotions. Tone measures how positive or negative a sentiment a piece of text has. LIWC, calculate two categories tone_pos and
6 Thematic Analysis
In order to understand the prevalent themes within the discourse surrounding COVID-19 vaccinations on social media, we conducted a manual thematic analysis on a selected sample of 400 tweets, equally balanced across each of the following categories: narrative information, non-narrative information, narrative misinformation, and non-narrative mis- information (100 tweets from each category). We first examined the sampled tweets to uncover the recurring themes, and then labeled each tweet ac- cording to the predominant theme present. Table 8 presents the most common themes within each sub-

---

Narrative
Sonn-arrative
1. don't warry about what's in the vaccine
30%
2. gel vaceinated
13%
1. got vaccinated and feeling positive
39%
3. sharing news
9%
2. feeling/anicipating vaccine side effects
155
4. quedloning/dehwnking couspiracy thenries
7%
Informative
friends/family member received the shot/feeling positive
5. frustration lowards those not following guidelines
6%
cager receive vaccine
10%
6. sharing local information
6%
5. someone got covid
3%
7. vaccine efficacy
6%
vaccine scheduling
3%
8. mislabeled should be narrative
4%
friends/famill member vaccine scheduled/feeling positive
3%
9. public health tips joke
3%
8. expressing gratitude towards healthcare workers
3%
9. other
13%
11. criticizing government
3%
12. other
10%
1. suspecting vaccine as gene therapy
19%
Mishformatti
conspiracy theory
2. vaccine hesitancy and concerns
19%
2. vaccine related deaths and side effects
28%
3. covid as a bioweapon
18%
3. vaccine hesituncy
20%
4. govt/large corp
14%
covid home remedy
6%
5. covid depopulation agenda
13%
5. other
8%
6. covid home remedy
10%
7. other
7%
Table 8: Results from the thematic analysis of COVID-19 Vaccine Narratives and Non-narratives and their proportions of the total dataset.
set, along with their respective percentages within the subset. Appendix E presents detailed thematic analysis results with paraphrased examples.
non-narratives presented similar ideas the mis- informative narratives, but these were presented in the form of "facts" about the dangers vaccines or supporting conspiracy theories, rather than stories.
We find that a majority of the informative nar- ratives described people's experiences with getting the vaccine, expressing excitement that loved was vaccinated. This focus vaccination- related stories can be partially attributed the larger size of the ANTiVax, which is specifically focused on vaccine-related information and misin- formation, relative to the CMU-MisCov19 dataset, which is more generally about COVID-19. How- ever, even across both datasets, we find that many users' stories centered around either vaccine expe- riences or experiences with COVID-19 itself, as these were some the main ways that everyday users were personally affected by the pandemic. Mis- informative narratives typically centered around storylines that provided evidence for conspiracy theories or harmful side effects of the vaccine, and were less often written as first-person narratives. When users did talk about themselves in the misin- formative narrative category, the stories described trying home remedies or hypothetical stories justi- fying why they would not get vaceinated.
Conclusion
In this paper, we study the relationship between narrative communication and health misinforma- tion on social media, specifically in the context of Twitter. We manually labeled a subset of tweets from misinformation datasets to add labels for nar- rative, and trained models to extend these labels to the entire datasets. Then, using statistical model- ing, find that narrativity is connected higher user engagement and misinformation to less en- gagement, but narrativity may help increase en- gagement of misinformation in some contexts such as for users with many followers. This finding may have implications for better promoting accu- rate health information by presenting and ampli- fying narrative messages. In line with studies of other domains, narratives showed more authentic- ity, had more mentions of people, time and many more expressions positive emotion, while mis- information was less authentic used more analytic language, and had dramatically more references to death. Many narratives focused on experiences of vaccines, illness, or home remedies, while non- narratives focused more news updates, public health information, and conspiracy theories. These findings shed light on how both the style and con- tent of messages can relate to how users engage with then, suggesting that both are important when seeking to combat misinformation.
Non-narrative informative (i.e., those not la- beled as misinformation or narrative) tweets con- tained a wide range of topics, from sharing news and public health tips to memes in the form "If X, don't worry about what's in the vaccine", where X is something commonly done yet potentially un- healthy, such as eating fast food. These tweets are meant to downplay concerns others may be ex- pressing about getting vaccinated. Misinformative

---

8 Limitations
Ethics Statement
As we used English-languagge data from Twitter dur- ing the COVID-19 pandemic, the language present in the dataset will be biased to this particular pop- ulation and may limit the ability to extrapolate conclusions about narrativity and misinformation to other contexts. For example, given the univer- sality of the pandemic, everyday users may have already been more invested in searching for and understanding health related information than they would have for other health issues. In the future, re- search should focus on other languages, platforms, and health contexts.
In this study, only pre-released Twitter datasets were used and collected via the official Twitter API using their IDs. Tweets that had been removed by their authors or by Twitter were not used in the study. All example tweets presented in the paper are paraphrased in order to preserve the anonymity of their original authors and in accordance, and data will only be shared list of tweet IDs with our own additional annotations. The annotators used in the study were authors and all are employed in research-related positions which compensate them for their work. While it is possible that the findings in this paper could be leveraged in order to help spread misinformation through the effective use of narratives, we do not provide any causal connec- tions between narrativity and engagement nor do we aim to provide specific recommendations about how increase tweet engagement, and this work may also help to provide new lenses through which to study misinformation and its spread in order to better combat it.
Further, the tweets were rehydrated based on their IDs, and some portion of the tweets were no longer available, disproportionately affecting the misinformatioo--rlateee tweets (and potentially some of the eiggeee-ngagemmntt misinformation tweets, since those may have been more likely to be debunked and subsequently deleted). In a less transparent way, Twitter's 's own content moderation policy also likely affected how much misinforma- tion was shown users, decreasing engagement, though we are unsure to what extent. Lastly, with recent changes to the Twitter (recently re-branded as "X")) it unclear how available "witter-based datasets will be in the future 3.
References
H. Porter Abbott. 2008. The Cambridge Introduction to Narrative, 2 edition. Cambridge Introductions to Literature. Cambridge University Press.
This paper uses the LIWC (Language Inquiry with Word Count) tool, which, though very popular, only uses surface-level analysis in the form of word count analysis and fails to provide an in-depth inter- pretation of the data. Despite participating in three rounds of training sessions, the annotators still en- countered several disagreements among each other which were resolved through discussion. With ad- ditional training, it may be possible to achieve an even higher inter-coder agreement and more reli- able annotations.
RM Adelson. 1966. Compound poisson distribu- tions. Journal of the Operational Research Society. 17(1):73-75
Anietie Andy, Brian Chu, Ramie Fathy, Barrington Ben- nett, Daniel Stokes, and Sharath Chandra Guntuku. 2021. Understanding social support expressed in a COVID-19 online forum. In Proceedings of the 12th International Workshop on Health Text Mining and Information Analysis, pages 19-27, online. Associa- tion for Computational Linguistics.
Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa-Anke, and Leonardo Neves. 2020. TweetE- Val:Unified Benchmark and Comparative Evaluation for Tweet Classification. In Proceedings of Findings of EMNLP.
Additionally, the narrative labels used for the full dataset are based on the predictions of a deep- learning model with imperfect predictive power,
and if errors are made in a systematic with respect to the relationship between narrative and misinfor- mation, some of the later analyses could be slightly skewed by this. We illustrate the extent of some possible differences in Appendix D, showing that the overall conclusions are unlikely to change.
Cornelia Betsch, Corina Ulshöfer, Frank Renkewitz, and Tilmann Betsch. 2011. The influence of narrative V. statistical information on perceiving vaccination risks. Medical decision making : an international journal of the Society for Medical Decision Making. 31:742-53.
H. Bilandzic and R. Busselle, 2012a. Narrative persua- sion. pages 219.
'Those seeking to access the dataset for replication pur- poses are encouraged to reach out to the authors of the paper for discussion of how data may or may not be shared based on the evolution of the current situation and best practices.
Helena Bilandzic and R. Busselle. 2012b. Narrative persuasion, pages 200-219.

---

Ryan L Boyd, Ashwini Ashokkumar, Sarah Seraj, and James W Pennebaker. 2022. The development and psychometric properties of liwc-22. Austin, TX: Uni- versity of Texas at Austin.
Kadhim Hayawi, Sakib Shahriar, Mohamed Adel Ser- hani, Ikbal Taleb, and Sujith Samuel Mathew. 2022. Anti-vax: novel twitter dataset for covid-19 vaccine misinformation detection. Public health, 203:23-30.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre- training with pradient-disentangee embedding shar- ing.
Ryan L. Boyd, Kate G. Blackburn, and James W. Pen- nebaker. 2020. The narrative are: Revealing core narrative structures through text analysis. Science Advances, 6(32):eaba2196
Eva Janssen, Liesheth van Osch, Hein de Vries, and Lilian Lechner. 2013. The influence of narrative risk communication on feelings of cancer risk. British Journal of Health Psychology, 18(2):407441.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan. Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Matcusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are seo-shot leamers. CoRR, Abs/2005.14165
Matthew Kreuter, Melanie Green, Joseph Cappella, Michael Slater, Meg Wise, Douglas Storey, Eddie Clark, Daniel O'Keefe, Deborah Erwin, Kathleen Holmes, Leslie Hinyard, Thomas Houston, and Sabra Woolley. 2007. Narrative communication in can- cer prevention and control: A framework to guide research and application. Annals of behavioral medicine: : a publication of the Society of Behavioral Medicine, 33:221-35.
Michael F. Dahlstrom. 2021. The narrative truth about scientific misinformation. Proceedings of the Na- tional Academy of Sciences, (18(15)x1111408811
Matthew W Kreuter, Kathleen Holmes, Kassandra Al- caraz, Bindu Kalesan, Suchitra Rath, Melissa Richert, Amy McQueen, Nikki Caito, Lou Robinson, and Ed- die M Clark. 2010, Comparing narrative and infor- mational videos to increase mammography in low- income african american women. Patient education and counseling, 81:S6-S14.
Daria Dayter. 2015. Small stories and extended narra- tives on twitter. Discourse, Context & Media, 10.
Jacob Devlin, Ming-Wey Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistiex: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jiyoung Lee and Soo Yun Shin. 2022. Something that they never said: Multimodal disinformation and source vividness in understanding the power of ai- enabled deepfake news. Media Psychology, pages 1-16.
Heidi Oi-Yee Li, Adrian Bailey, David Huynh, and James Chan. 2020. Youtube as a source of infor- mation 19: a pandemic of misinformation? BMJ Global Health, 5(5).
AR Dirkson, Suzan Verberne, Wessel Kraaij, AM Jorge, R Campos, A Jatowt, and S Bhatia. 2019, Narrative detection in online patient communities. In Proceed- ings of Sare2Stor-Secooee
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 190. 1692.
Workshop on Narrative Extraction From Texts co-located with 41th European Conference on Information Retrieval (ECIR 2019), pages 21-28. CEUR-WS.
Zexin Ma. 2021. The role of narrative pictorial warning labels in communicating alcohol-ree reeated cancer risks. Health Communication, pages 1-9.
Olive Jean Dunn. 1961. Multiple comparisons among means. Journal of the American statistical associa- tion, 56(2,3)))3) 64.
Zexin Ma and Guolan Yang. 2022. Show me photo of the character: Exploring the interaction between text and visuals in narrative persuasion. Journal of Health Communication, 27(2):125-133) PMID: 35422202.
Achyutarama Ganti, Steven Wilson, Zexin Ma. Xinyan Zhao, and Rong Ma. 2022. Narrative detection and feature analysis in online health communities. In Proceedings of the 4th Workshop of Narrative Under- standing (WNU2022), pages 57-65, Seattle, United States. Association for Computational Linguistics.
H. B. Mann and D. R. Whitney. 1947. On Test of Whether one of Two Random Variables is Stochas- tically Larger than the Other. The Annals of Mathe- matical Statistics, 18(1):50 -60
Jeanine P.D. Guidry, Kellie Carlyle, Marcus Mess- ner, and Yan Jin. 2015. On pins and needles: How vaccines are portrayed on pinterest. Vaccine, -3533)
Willard G Manning and John Mullahy, 2001. Estimat- ing log models: to transform or not to transform? Journal of health economics, 20(4):461-44

---

Elizabeth J. Marsh and Brenda W. Yang. 2018. One believing things that are not true: A cognitive science perspective misinformation. Misinformation and Mass Audiences.
Filipo Sharevski. Raniem Alsaadi, Peter Jachim, and Emma Pieroni. 2022. Misinformation warnings: Twitter's soft moderation effects covid-19 vaccine belicf echoes. Computers & Security, 114:102577.
Shahan Ali Memon and Kathleen M Carley. 2020. Characterizing covid-19 misinformation communi- ties using novel twitter dataset. arXiv preprint
Brian G. Southwell, J. Scott Babwah Brennen, Ryan Paquin, Vanessa Boudewyns, and Jing Zeng. 2022. Defining and measuring scientific misinformation. The ANNALS of the American Academy of Political and Social Science, 700(1 (111)))8-111
arXiv:2008.00991.
Shahan Ali Memon, Aman Tyagi. David R. Mortensen, and Kathleen M. Carley. 2020. Characterizing so- ciolinguistic variation in the competing vaccina- tion communities. In Social, Cultural, and Behav- ioral Modeling: 13th International Conference, SBP- BRiMS 2020, Washington, DC, USA, October 18-21, 2020, Proceedings, page 118-129, Berlin, Heidel- berg. Springer- Verlag.
William J Thompson. 2001. Poisson distributions. Com- puling in Science & Engineering, 3(3):78-82.
Suzan Verberne, Anika Batenburg. Remco Sanders, Mies van Eenbergen, Enny Das, Mattijs Lambooij. et al. 2019. Analyzing empowerment processes among cancer patients in online community: A text mining approach. JMIR cancer, 5(1)
Sheila T. Murphy, Lauren B. Frank, Joyee S. Chatter- jee, and Lourdes tive Versus Nonnarrative: The Role of Identification, Transportation, and Emotion in Reducing Health Dis- parities. Journal of Communication, 63(1): 16-137.
2013. Narra-
Emily K. Vraga and Leticia Bode. 2020. Defining mis- information and understanding its bounded nature: Using expertise and evidence for describing misinfor- mation. Political Communication, 37(1):136 136-144.
John Ashworth Nelder and Robert WM Wedderburn. 1972. Generalized linear models. Journal of the Royal Statistical Society: Series A (General), 13703)437--384
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame. Quentin Lhoest, and Alexander M. Rush. 2019. Hug- gingface's 's transformers: Statc-of-the-an natural lan- guage processing.
Rasmus Nielsen, Richard Fletcher, Nic Newman, J Bren- nen, and Philip Howard. 2020. Navigating the 'info- demic': How people in six countries access and rate news and information about coronavirus. Reuters Institute for the Study of Journalism.
Xinyan Zhao and Stephanie Tsang. 2023. How people process different types of misinformation on social media: A taxonomy based on falsity level and evi- dence type. SSRN, pages 1-39.
Janine Overcash. 2003. Narrative research: re- view of methodology and relevance to clinical prac- tice. Critical Reviews in Oncology/Hematology. 1882)1199-184
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Duch- esnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 282802-2830.
A Annotation Guidelines
Below are the guidelines developed during the an- notation process and agreed upon by the annota- tors. Note that these guidelines were specifically developed for annotating tweets and should not be directly used annotating other types of text as narrative before considering and adapting to other contexts and media. All examples provided are related to vaccines and/or COVID-19.
Wei Peng. Sue Lim, and Jingbo Meng. 2022. Persuasive strategies in online health misinformation: a system- atic review. Information, Communication & Society, 0(0):1-18.
James W. Peunebaker. 2011. The secret life pronouns. New Scientist, 211(2828):42-45
A.1 Narrative definitions
According to (Kreuter et al., 2007), a narrative is defined as a representation of connected events and characters that has an identifiable structure, is bounded in space and time, and contains implicit or explicit messages about the topic being addressed. (Bilandzic and Busselle, 2012b) say that a narra- tive refers to a presentation of an event(s) experi- enced by a specific character(s) in a setting.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiy preprint arX:::
Gautam Kishore Shahi, Anne Dirkson, and Tim A. Ma- jehrzak. 2021. An exploratory study of covid-19 misinformation on twitter. Online Social Networks and Media, 22: 100104.

---

And according to (Dahlstrom, 2021), narrative is defined a message that describes the experi- ence of specific characters across series of related events over a defined time period- a triumvirate character, causality, and temporality. At its core, narrative is the telling of somcone's experience about something.
happy now and the vaccine is in the past, so there temporality and causality, Only saying "I got the vaccine" is not a narrative for our study because it does not reflect a series of related events.
A.2.3 Additional cases consider
Hypothetical situations using an if-then format, such as "ifI I had been to Dr. Dennis as a child, then would be naturally immune to covid" are not narratives. It could be rewritten as "I went to Dr. Dennis as child, and now have natural immunity to covid" which conveys the same information, but in more of narrative form. The narrative is not only about what information is being presented, but the style of communication being used (how it is presented). Quotes like "CDC director said 'there are 1000 new covid cases' " is not a narrative it is just mentioning that someone said a fact but there is no story. But if the CDC director says "I have been sick for three days and the medicine didn't 't have any effect me... "his can be considered narrative. Similarly, narratives within jokes/non-seriou texts are still considered narratives. F.g., "Bill Gates is going to come to my house to get the tracker out of my arm from the covid vaccine." " This is a narrative. Advice is not a narrative: "You should get your covid vaccine".. News headlines can be narratives, but they are not always narratives. For example, "Vaccine Mass Sterilization Depopulation Agenda Revealed on Amazon 'Utopia' Show" is not narra- tive. general, try not to overthink it and perform mental gymnastics to determine how a tweet might possibly meet the criteria. Ask yourself: does this tweet tell a story about someone's experience of something? If so, is a narrative.
A.2 Rules
Based on the definitions provided, the rules for la- beling the tweets are as follows: the tweet must contain (1) at least one specific character (normally is person) who experiences (2) a series of re- lated events. You may assume that the presence of multiple events implies temporality and do not need to specifically check for temporality during annotation.
A.2.1 Characters
Character/characters need to refer specific indi- viduals. Characters can be the author of the text (Ist person), but can also be someone else who mentioned in the text (2nd or 3rd person).
Examples: "Scientist", "CDC", any mentions like "this journalist", "14 people" (where this refers to specific set of individuals that you could hypo- thetically identify) etc., be considered charac- ters. "People", "humanity", "the world", etc. don't count as characters. "You' or 'your' (and they) can also be used in a generic way that does not refer to specific person and should therefore only be considered a character if it directly refers to some- one specific such as "Joe Biden, you should con- sider giving more free covid tests". "We" and "us" can be character because it includes the author. Mentions of a country or country's government in general ("The U.S.") are usually too vague to be considered a character, but identifiable individuals ("The United States Supreme Court") within the government could be characters.
A.3 Examples of difficult cases
During pilot rounds of annotation, the following cases were difficult to come to an agreement about. After further discussion with communication re- search experts, the following decisions and ratio- nales were made and provided as guidance for the final annotation process. Note that these were some of the most annotators) edge cases, and are not representative of a majority of the dataset.
A.2.2 Events
Emotions, thoughts, or other non-observable ac- tions can be considered an event. Emojis and punc- tuation can indicate emotions. The characters in- volved don't necessarily need to take any actions, but should be involved in or experiencing the events somehow. Events can be fictional, false, occur- ring in the future. They don't need be actual things that have definitely happened.
(according the
New Jersey has launched website debunk rumors and hoaxes associated with the spread of the coronavirus, following a false text mes- sage of impending national lockdown that cir- culated widely across the U.S.
Examples: "I am so happy that I got the covid vaccine" can be a narrative because the person is

---

Label: Not narrative
suffering from COVID19. You are misleading the public with dangerous misinformation
Reason: New Jersey Lgovernment] is not a character because it is too vague.
Label: Narrative
2. CIA knew about #coronavirus #Bio Weapons before Chinese health authorities #USNews #COVID19 was released by #DonaldTrump
This tweet author "speaks" directly to Rudy Giuliani. The author is one of the "many of us" who has taken the drug.
Reason:
Label: Narrative
B Hyperparameter Tuning and Preprocessing
Reason: There two events that nected: CIA knowing about bioweapons be- fore Chinese health authorities, and Donald Trump releasing COVID-19.
For SVM classification, used grid search over both kernel type (linear, polynomial, RBF) and degree (1, 2, and 3), with a polynomial kernel and giving the best results on the validation set. For logistic regression, we considered [1, 2, 3} and C = 2 led to the best results.
3. Me thinks so too! The gov can't say they have been poisoning us for years imagine the uproar- it affects us all eating synthetic foods for decades. Why does supermarket meat taste like Rubber? coronvirus smoke- screen.. .ubut then there is COVID-19 is that the bioweapon?
degree
For the bag-of-words models, also experi- mented with various pre-processing techniques, in- cluding lowercasing, lemmatization, removal of URLs, stop words, and hashtags. We used a com- bination of these techniques along with hyper- parameter tuning that gave us the best performance on the validation set and used that model to predict the labels for the test set. The best results for all Bag-of-wordd models were achieved when remov- ing stopwords, URLs, and hashtag symbols (*#) while keeping the text of the hashtag itself.
Label: Narrative
Reason: This tells someone's (the tweet thor's) experience of something: the govern- ment poisoning food and supermarket meat tasting like rubber.
4. Got my first vaccine dose today!
Label: Narrative
For the transformer-based deep learning models, we initialize each from the model checkpoint and fine-tune
The at the end conveys an emotion that the author is experiencing after getting the vaccine.
Reason:
training data for 5 epochs with batch size of 16, weight decay of 0.01, and learning rate of 2c-5.
5. #vaccinated can flly on Hamericanairlines af- ter the second COVID-19 vaccine?
C Few-Shot Experiments
Label: Narrative
This section presents detailed results on the effect of the number of in-context examples that are pro- vided to the GPT models for few-shot learning. In each case, we balance the number of narrative and non- non-navratvve examples from each of the two souree datasets, meaning that our number of training ex- amples, A is always multiple of to maintain this balance. From results on our development set, we find that the overall best results occur when k - 20, and therefore we select this value for the rest of experiments in the paper. This means that have narratives from each dataset and 5 narraaivvaives from each of the two datasets. The results for the GPT-3 Davinci model are presented in Table 9, and the results for the GPT-3.5-Tum model are presented Table 10.
Reason: The author expresses that they got their second COVID-19 vaccine (#vaccinated) and is now wondering about the possibility of air travel.
6. I have had my money Oxford- AstraZeneca from day 1. So excited to hear this news!
Label: Narrative
Reason: The author has been hoping Oxford- AstraZeneca would be successful and is now expressing excitement about the fact that they were.
7. @ @RddyGiulia Many of us have taken this drug for Malaria prevention but never while

---

GPT-3 (Davinci)
Dataset
ANTiVa
CMU-MisCoTTy
Combined
k-16
k-400
Info or
tew-shoe
01579
0.001
0.612
0.646
01.999
01,777
Misinfo
Info Misinfo All
Info Misinfo All
Info Misinfo All
0.643
0.700
LIWC
+inst
0.560
0.629
0.630
0.666
0.698
0.642
eategory
R
0.601
0.689
0.685
WC
1.26
1.12
Analytic
0.83
0.84
0.84
0.84
Table Narrative class Fl scores averaged across three runs for GPT-3 models across various fho-shot settings, trained on detecting narratives using the validation set only, Bold indicates the top Fl score per model.
Authentic
1.85
185
tone_poc pos
2.29
2.5
2.17
2.29
nep
00.54
0.57
0.54
0.57
emoupox pos
3.62
5.17
3.87
5.16
0:49
0.6
0.54
0.53
0.61
health
deuth
0.44
0.5
function
1.08
pronoun
1.27
1.41
1.99
1,8
1.34
1.44
article
0.78
0.75
0.84
0.8
prep
0.78
0.89
0.82
1.2
0.91
Model
k=4
k=12
k=16
l==20
auxverb
0.84
0.85
few shot
0.539
0.548
0.542
0.565
0.584
0.568
conj
+der
0.566
0.31
0.564
negule
0.48
0.47
0.45
0.49
tinst 0.571
0.591
0.597
0.616
0.642
0.575
family
3.04
4.67
3.71
4.16
3.2
4,57
3.9
+def+inns
0.596
0.611
0.648
0.644
0.636
0.590
friend
5.0
4.0
Cognition 0.65
0.64
0.64
0.65
Table 10: Narrative class F1 scores averaged across three runs for GPT-3.5-Turbo models across various few-shot settings, trained on detecting narratives using the validation set only. Bold indicates the top F1 score per model.
ppron
1.59
1.74
197
2.04
2,07
1.72
1.76
2.02
Time
1.54
1,46
1.12
cause
0.49
Table 11: Summary of statistical testing for significance of LIWC categories of the 3,000 manually annotated tweets related to narratives using Mann- W Whitney U test. R = p the ratio between the means of LIWC category for narrative and non-narrative tweets, respectively. -eenotes non-significant results, i.e., p- value 2 a. Blue denotes R 2 (i... the category is used more in narratives than non-narratives) and Red denotes R < 1. WC word count.
D LIWC analysis of manual-annotations versus machine-annotations
In this appendix, we computed the LIWC scores similar to both tables 6 and 7 for only the 3,000 tweets that were manually annotated. See tables 11 and 12. We found that there is no meaningful difference between the LIWC scores of the 3000 tweets annotated by the authors and all the 14561 tweets annotated by the best-performing model (RoBERTa). The means of differences of LIWC scores between numan-annotain annotated data are 0.225 and 0.282 with standard deviation of 0.601 and 0.927 for all values in ta- bles 6 and 11, and tables 7 and 12, respectively. These small differences reflect the strength of the perf-perg RoBERTa model that achieved an Fl-score of 0.924, and we did identify any val- ues that contradict the claims made in the paper. However, we did notice that fewer claims could be reliably made given the smaller size of the data when using only 3,000 versus 14,561 texts. The number of LIWC scores that were not statistically significant in human-annotated data only, but were statistically significant in the full hudan+model- annotated data, is 40 scores and 28 scores for tables 6 and 7 respectively. This can be attributed to the increased size of the dataset, making a larger num- ber of the results statistically significant while still controlling for multiple comparisons.
Dataser
ANTIVix
CML-MMaC WWT))
Comhined Narr
Narrativity
Non Narr
All
Non Nair
All
Non
All
LIWC earggory
WC
1,46 1.35
1.31
1.29 1.33
1.24
Analytic
1.22
1.14
1.16
1.24
1.14
Authentic
0.64
0.49
0.71 0.55
0.54
jone_pos
0.39
tone, neg
1.52
1.37
1.44
1.29
emo_pos
0.09
0.12
0.34 0.09
0.13
emo neg
0.67
0.74
health
0.86
21.0
18.6
60 7.73
data and model-
Conversation
1.19
1.15
swear
function
0.91
0.9
promoun
0.77
0.74
0.68
0.79 0.74
0.69
family
0.39
0.38
friend
Table 12: Summary of statistical testing for significance of LIWC categories of the 3,000 mamually annotated tweets related to misinformation using Mann- WhinnWW U test. R Bmmiein fo/ Men the ratio between the means of a LIWC category for misinformed and in- formed tweets, respectively. denotes signniiffi results, i.e., p-value the category is used more in misinformed tweets than informed ones) and Red denotes R < I.
Blue denotes R > 1 (i.c.,
E Thematic Analysis with Examples
In this section of the appendix, we present break- down of our thematic analysis results COVID-19 tweets. Here each subset of the analysis is pre-

---

Theme
Example
Quick reminder, this isn't It any normal vaccine.
suspecting vaccine as gene therapy
19 THIS GENE THERAPY. which will make a human forever slave to the big tech
They be calling them idiots for not wanting a
vaccine hesitancy and concerns
19 rushed untested vaccine, when it's the vaccine that causes infertility
Wake up!! This is not virus, it's carefully launched bioweapon.
covid is bioweapon
Fauci knows about the man-made virus long ago, it's all cover up by the govt to keep the public
govt/large corp
14
under their shoes.
There are 10 million unplanned pregnancy terminations since the beginning of this virus. Don't be stupid, this is depopulation
covid is depopulation agenda
attempt by Rill Gates and the govts control population growth.
10 Take mint leaves, honey, garlic and onion to prevent covid.
covid home remedy
you don't 't need the !!c!ine!!!!!!!!
other
7
Sign this petition people! covid is hoax, #stopdepopulation #novax
Table 13: Thematic Analysis of Non-Narrative Misinformation COVID-19: Predominant Themes, Proportions. and Representative paraphrased Examples
Theme
% Example
you ve ever heen that KFC on the street, don't worry about what's 's in the vaccine
don't I worry about what's in the vaccine get vaccinated
30
13
Will be getting vaccine next week. __
The government will now hegin vaccinating kids below the age of 10, starting Wednesday
sharing news
SG towers don't 't cause covid and there is scientific
questioniig//dehunkingg conspiracy theories
evidence support that claim. Stop spreading misinformation
and please follow ede guidelines and save lives
So frustrating see selfish people not follow rules. The pandemic isn't over yet. Please stay home and follow the rules.
frustration towards those not following guidelines
vaccine information: vaccinations available for ages 14+ this Sunday 3/09 at ABC Pharmacy in Carlton. If interested please call their number at .9...99..99999.. 999.9999 #stay vaccinated
sharing local information
pleass
The vaccine protects you and your loved ones from this deadly virus. Please believe in science and get vaccinated and save lives.
vaccine efficacy
mislabeled should be narrative
Hurrayyy!! I am getting vaceinated in days! Let's go guys!
Stay indoors, wear masks and always sanitize your
public health tips
hands when you touch public areas.
my American friend says covid is planned agenda. but what about Bill Gates acquiring large areas of farm lands?LOL
joke
Only 6% of the adults have received both doses. This is bud governance its best. You should learn from South Korea My sister argues with me about this vaccine everyday!
criticizing government
other
Table 14: Thematic Analysis of Non-Narrative Information on COVID-19; Predominant Themes, Proportions,
and Representative paraphrased Examples
sented individually with paraphrased examples, in- cluded preserve the anonymity of the authors while still capturing the main types of messages that existed in each theme. Discussion of the themes was originally presented in Section 6.
volve around various conspiracy theories and mis- information regarding COVID-19 and the vaccines. They touch upon unfounded claims such as the virus being a bioweapon, and concerns about the vaccine's ' safety and potential side effects.
Table 13 shows the predominant themes occur- ring in the non-narrative misinformation subset. These are the tweets that were originally labeled as misinformation from the source datasets, and la- beled as narratives through our methodology. The themes presented in this table predominantly re-
Table 14 describes themes from the non- narrative information category, and predominantly captures the various facets of public sentiment and discourse regarding COVID-19 but non- narrative style. The themes range from humorous takes and encouragement for vaccination to the

---

Theme
Example
Fauci knew that the virus has been cooking in the since 2005 but he covered it up and now Bill Gates inserting chips in our body through vaccine track us
conspiracy theory
38
A boy in Netherlands found dead 30 hours after taking
vaccine related deaths and side effects
28
his second covid shot. He was fit until a day before he
took that damn vaccine!!!!
I will not take that vaccine until i get pregnant and give birth to baby. I don't I want to lose my fertility
vaccine hesitancy
20
manufactured virus and supposed vaccine
I have been cating garlic, one table spoon of honey, some onions before go to bed and let me tell you, have not had covid since! This totally works! and is the best solution for the virus.
covid home remedy
6
unfriended good friend of mine yesterday because she's totally into this 5G conspiracy theory shit.
other
I don't have any regrets
Table 15: Thematic Analysis of Narrative Misinformation on COVID-19: Predominant Themes, Proportions, and Representative paraphrased Examples
Theme
% Example
I got my second shot yesterday. I can finally go visit my mom and dad this weekend and i foel so grateful! <3
got vaccinated and feeling positive
39
Got the jab two days ago and i am really feeling effects.
feeling/anticiaating vaccine side effects
15
My arm is still sore and i feel the chills too. Not going to work tomorrow
friends/family member received the shot/
My mom and dad got their first shot today and feel so relived.
11
feeling positive
Let's defeat this pandemic together. Peace
I could finally hook my appointment for the vaccine.
cager to receive vaceine
10
am excited and can't wait to get the shot and meet my friends
Robert did not want get vaccinated and guess what,
somcone got covid
3
he caught the virus and he's sick!
3 I have been trying to book vaccine appointment for weeks
vaccine scheduling
and still haven't 't made it, i feel so disappointed. Urghhhh!!!!!
friends/family member vaccine scheduled/ feeling positive
3 My 78yo grandmother could finally book vaccine appointment for tomorrow. feel so happy and grateful!!
got my first shot yesterday and thank all the frontline workers
expressing gratitude towards healthcare workers
3 who have done phenomenally well during this pandemic. You saved countless lives. Feeling grateful!!
13 (includes themes like "criticizing govt's handling of covid", "a "ajoke", "setting example for others by getting vaccinated" etc.)
other
Table 16: Thematic Analysis of Narrative Information on COVID- 19: Predominant Themes, Proportions, and Representative paraphrased Examples
active debunking of conspiracy theories, and so on.
These themes capture sentiments ranging from grat- itude and optimism after receiving the vaccine to challenges faced during the vaccine scheduling and concerns about side effects.
Table 15 highlight various forms of misinforma- tion and misconceptions surrounding COVID-19 and its vaccines which were presented using nar- rative style. They capture sentiments from deeply entrenched conspiracy theories and fears related to vaccine side effects. The nature of these themes present the challenge of battling misinformation in the era of the pandemic.
Finally, in the themes presented in Table 16 re- volve around personal experiences, emotions, and perspectives related to the COVID-19 vaccination process, which were presented in narrative style.


Reviews:
[{"review_id": "KoZECf8FdT", "reviewer": "Reviewer_eVvg", "summary": "", "questions": "Questions are mentioned above.\n\nThere is an assumption that the dataset would be available for free use later.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read and engage with our work! We appreciated your insights and suggestions. Below we have done our best to address all of your questions and concerns.\n\nQ1a) Concerns regarding the quality of the automatically annotated samples.\nA) We manually annotated a large subset (n=3000) of the original dataset (n~=15000); trained large language models on the data across three runs, and chose the best-performing model to annotate the rest of the dataset automatically. Our F1 scores in Table 3 measure the quality of the automatically annotated samples for our held-out test set which was specifically sampled so that it would be representative of the entire (previously) unlabeled portion of the dataset. Given the reported metrics on the test set held fairly consistently across multiple runs, we can expect that the performance on the rest of the held-out data will be similarly strong. \n\nQ1b) How much control do we have to maintain the quality of the automatically generated samples?\nA) If you are referring the tweets themselves, we would like to clarify that we did not generate any of the samples. They are all real tweets that come from existing datasets already labeled by experts for the presence of misinformation. The generative models we employed in one of our experiments only output a binary narrativity class label (Yes or No) using the prompting approach described in section 4 of the paper.\n\nQ2) How do we know that the number of instances that have been manually annotated is enough for the computational model to learn reliably?\nA) While it's generally true that the more quality data we have, the better our models learn from them, we believe that the number of data points (n=3000) we chose to annotate was sufficient. The F1 score, our best-performing RoBERTa model (F1=0.924) indicates high levels of precision and recall in the binary classification task, which would be difficult to achieve without sufficient training data for the task.\nOther recent works on narrative analysis such as the ones listed below, successfuly trained narrative detection models with < 1/3 of the data we used for training.\n\nNarrative detection in online patient communities. Anne Dirkson, Suzan Verberne, Wessel Kraaij [European Conference on Information Retrieval 2019]\n\nNarrative Detection and Feature Analysis in Online Health communities. Achyutarama Ganti, Steven Wilson, Zexin Ma, Xinyan Zhao, Rong Ma [Workshop on Narrative Understanding 2022]", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A new dataset that might help modelling user narratives online especially related to misinformation.\nThe studies that the paper presents are interesting and useful to understand the importance of user narratives.", "reasons_to_reject": "One of the key concerns is the quality of the automatically annotated samples. How much control we have to maintain the quality of the automatically generated samples is something that needs to be mentioned in the paper. While the authors have used different machine learning models, these models have their own pros and cons.\nBesides that, how do we know that the number of instances that have been manually annotated are enough for the computational model to learn reliably?", "questions_to_authors": "Questions are mentioned above.\n\nThere is an assumption that the dataset would be available for free use later.", "paper_topic_and_main_contributions": "In this paper, the authors annotate an existing misinformation tweet dataset for the presence of narratives. The authors selected two existing medical datasets. They then followed the annotation process through an iterative process involving discussion and revision. They first annotated some samples, and then the rest of the samples were automatically annotated using machine learning techniques.\n\nThe conclusions that the authors made are:\n1. narratives involve higher user engagement than misinformation\n2. narratives may help increase engagement of misinformation\n\nThe key contribution of the work is the dataset and the follow-up conclusions that the paper makes.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WBYXSNHZjo", "reviewer": "Reviewer_uo6d", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read and engage with our work! We appreciated your insights and suggestions. Below we have done our best to address all of your questions and concerns.\n\n1) If the annotators did 3k annotation, It is not clear how the rest of the tweets are labeled by the model. Does it mean around 11.5k tweets were used as test data?\nThanks for the question -- to clarify: the full dataset consists of 14561 tweets. We manually annotated a subset of 3000 tweets from the actual dataset using the process described in section 3.2, and then used a (80-10-10) split of that data to train our models as described in section 4. The best-performing model found in section 4 was then used to label the remaining 11.5k tweets for narrativity to use in the analyses in section 5. \nWe would like to clarify that those 11.5k tweets are not considered part of the test set because our test set is data that has been labeled by human annotators in order to evaluate the performance of the model. Our test set is a representative sample of the full 14.5k that was selected in order to establish the generalization performance of our detection models, but these 11.5k were previous unlabeled data that we would like to analyze. Thanks for pointing this out; we will clarify this in the paper at the beginning of section 5 so it is completely clear how all of the portions of the data were labeled. Note that based on a question from R3, we also re-ran all of our experiments in section 5 using only the manually labeled 3k tweets to confirm that the results are in fact consistent, only lacking in the statistical power that we can achieve with the full set of 14.5k tweets.\n\n2) Regarding examples of data annotation: the Annotation procedure is described in Section 3.2, with detailed guidelines in Appendix A for your reference, however we chose not show the exact examples of tweets based on prior scholarly works such as the work by Fiesle and Proferes: “Participant” perceptions of Twitter research ethics.\" Social Media+ Society 4.1 (2018): 2056305118763366. which suggests that users prefer not to have their exact messages presented in research papers or being used as examples. Our commitment to not directly showcasing examples stems from our ethical stance to safeguard the privacy of the authors of the tweets. However, based on your helpful suggestion, we conducted a thematic analysis, which summarizes the main themes that were present in the tweets so that you can understand the types of tweets that were annotated as narrative or non narrative, along with some paraphrased examples. The results will be added to the paper if accepted and we have included them at the end of this rebuttal (see item 5) for your reference.\n\n3) We also noticed that we were given a low score for reproducibility (2) which contains the comment \"Would be hard pressed to reproduce the results. The contribution depends on data that are simply not available outside the author's institution or consortium; not enough details are provided.\" We wanted to clarify that we will release our annotations and all data can be rehydrated using Twitter API; all of the annotation procedures and guidelines will be shared, pre-trained models will be shared, and other tools such as LIWC are available and will produce consistent results for anyone who uses them on our data. While we agree that the annotation process itself might have some variation depending on the annotators (as with most annotation procedures), we believe that the rest of our pipeline will be reproducible given the content of the paper and the aforementioned resources that we plan to release.\n\n4) Thank you for pointing out these useful references which can be added to our paper.\n\n5) Based on reviewers' comments, in order to demonstrate the types of tweets that exist in our dataset, we conducted a manual thematic analysis of 400 tweets. 100 tweets were sampled from our full dataset from each of the following 4 categories: narrative information, non-narrative information, narrative misinformation, and non-narrative misinformation. Following a standard thematic analysis procedure, we first read through the sampled tweets, and then determine the main themes that existed. We coded each tweet and present the aggregate results below, including each theme, the percentage of the data subset that contained that theme, and one (paraphrased) example tweet to illustrate the theme. We will include these results in a formatted table in the final version of the paper, along with a discussion, if accepted. \n\nnon-narrative information\n\n- **don't worry about what's in the vaccine (30%)** Example: \"If you've ever been to that KFC on the street, don't worry about what's in the vaccine.\"\n- **get vaccinated (13%)** Example: \"Will be getting vaccine next week. _/\\_\"\n- **other (10%)** \"My sister argues with me about this vaccine everyday!\"\n- **sharing news (9%)** Example: \"The government will now begin vaccinating kids below the age of 10, starting Wednesday\"\n- **questioning/debunking conspiracy theories (7%)** Example: \"5G towers don't cause covid and there is no scientific evidence to support that claim. Stop spreading misinformation and please follow cdc guidelines and save lives\"\n- **frustration towards those not following guidelines (6%)** Example: \"So frustrating to see selfish people not follow rules. The pandemic isn't over yet. Please stay home and follow the rules.\"\n- **sharing local information (6%)** Example: \"vaccine information: vaccinations available for ages 14+ this Sunday 3/09 at ABC Pharmacy in Carlton. If interested please call their number at 999-999-9999 #stay vaccinated\"\n- **vaccine efficacy (6%)** Example: \"The vaccine protects you and your loved ones from this deadly virus. Please believe in science and get vaccinated and save lives.\"\n- **mislabeled - should be narrative (4%)**\n- **public health tips (3%)** Example: \"Stay indoors, wear masks and always sanitize your hands when you touch public areas.\"\n- **joke (3%)** Example: \"my American friend says covid is a planned agenda, but what about Bill Gates acquiring large areas of farm lands?LOL\"\n- **criticizing government (3%)** Example: \"Only 6% of the adults have received both doses. This is bad governance at its best. You should learn from South Korea\"\n\n\nnarrative information\n- **got vaccinated and feeling positive (39%)** Example: \"I got my second shot yesterday. I can finally go visit my mom and dad this weekend and i feel so grateful!!! <3\"\n- **feeling/anticipating vaccine side effects (15%)** Example: \"Got the jab two days ago and i am really feeling the effects. My arm is still sore and i feel the chills too. Not going to work tomorrow\"\n- **friends/family member received the shot/feeling positive (11%)** Example: \"My mom and dad got their first shot today and i feel so relived. Let's defeat this pandemic together. Peace\"\n- **eager to receive vaccine (10%)** Example: \"I could finally book my appointment for the vaccine. I am excited and can't wait to get the shot and meet my friends\"\n- **someone got covid (3%)** Example: \"Robert did not want to get vaccinated and guess what, he caught the virus and he's sick!!!!\"\n- **vaccine scheduling (3%)** Example: \"I have been trying to book vaccine appointment for weeks and still haven't made it, i feel so disappointed. Urghhhh!!!!!\"\n- **friends/family member vaccine scheduled/feeling positive (3%)** Example: \"My 78yo grandmother could finally book vaccine appointment for tomorrow. I feel so happy and grateful!! \"\n- **expressing gratitude towards healthcare workers (3%)** Example: \"i got my first shot yesterday and I thank all the frontline workers who have done phenomenally well during this pandemic. You saved countless lives. Feeling grateful!!\"\n- **other (13%)**  include themes like \"critisizing govt's handling of covid\", \"a joke\", \"setting example for others by getting vaccinated\" etc\n\nnon-narrative misinformation\n\n- **suspecting vaccine as gene therapy (19%)** Example: \"Quick reminder, this isn't any normal vaccine. THIS IS GENE THERAPY, which will make a human forever slave to the big tech\"\n- **vaccine hesitancy and concerns (19%)** Example: \"They be calling them idiots for not wanting a rushed untested vaccine, when it's the vaccine that causes infertility\"\n- **covid is a bioweapon (18%)** Example: \"Wake up!! This is not a virus, it's a carefully launched bioweapon.\"\n- **govt/large corp (14%)** Example: \"Fauci knows about the man-made virus long ago, it's all a cover up by the govt to keep the public under their shoes.\"\n- **covid is a depopulation agenda (13%)** Example: \"There are 10 million unplanned pregnancy terminations since the beginning of this virus. Don't be stupid, this is a depopulation attempt by Bill Gates and the govts to control population growth.\"\n- **covid home remedy (10%)** Example: \"Take mint leaves, honey, garlic and onion to prevent covid. you don't need the vaccine!!!!!!!!!\"\n- **other (7%)** Example: \"Sign this petition people! covid is hoax. #stopdepopulation #novax \"\n\nnarrative mis-information\n- **conspiracy theory (38%)** Example: \"Fauci knew that the virus has been cooking in the lab since 2005 but he covered it up and now Bill Gates is inserting chips in our body through vaccine to track us\"\n- **vaccine related deaths and side effects (28%)** Example: \"A boy in Netherlands found dead 30 hours after taking his second covid shot. He was fit until a day before he took that damn vaccine!!!!\"\n- **vaccine hesitancy (20%)** Example: \"I will not take that vaccine until i get pregnant and give birth to a baby. I don't want to lose my fertility to a manufactured virus and supposed vaccine -_-\"\n- **other (8%)** Example: \"I unfriended a good friend of mine yesterday because she's totally into this  5G conspiracy theory shit. I don't have any regrets\"\n- **covid home remedy (6%)** Example: \"I have been eating garlic, one table spoon of honey, some onions before i go to bed and let me tell you, i have not had covid since! This totally works! and is the best solution for the virus.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The analysis part of the paper where the authors used statistical modeling to find out the relationship among narrativity, misinformation, follower count, and user engagement. ", "reasons_to_reject": "1. Line 327, 14.5k tweets were labeled by the annotators and the best RoBERTa model. If the annotators did 3k annotation,  It is not clear how the rest of the tweets are labeled by the model. Does it mean around 11.5k tweets were used as test data?\n\n2. As data annotation was mentioned as one of the contributions, the authors could show some examples of annotated data and annotation procedures.\n\n3. Authors could include a thematic analysis of the data. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper explores the relationship between narrative communication and health misinformation on social media, focusing on Twitter. The authors manually labeled a subset of tweets from misinformation datasets to identify narratives and then used the best model (their case RoBERTa) to extend these labels to the entire datasets. They found that narrativity can increase engagement with misinformation, especially for users with many followers. ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Authors should explore the stance, morality, and reason detection task on COVID-19 by Pacheco et al 2022 (A holistic framework for analyzing the covid-19 vaccine debate). \n\nAlso for COVID-19 moral foundation and theme, authors could look into work by Islam and Goldwasser 2022 (Understanding COVID-19 Vaccine Campaign on Facebook using Minimal Supervision)."}}, {"review_id": "MgtMmeNwBz", "reviewer": "Reviewer_93eC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read and engage with our work! We appreciated your insights and suggestions. Below we have done our best to address all of your questions and concerns.\n\n1) Regarding the concern about systematic errors: as you have suggested, we recomputed the LIWC scores in both tables 6 and 7 for only the 3000 tweets that were manually annotated. We will report these results in the appendix so that readers may compare the full set of values directly. We found that there is no meaningful difference between the LIWC scores of the 3000 tweets annotated by the authors and all the 14561 tweets annotated by the best-performing model (RoBERTa). The means of differences of LIWC scores between human-annotated data and model-annotated data are 0.225 and 0.282 with standard deviation of 0.601 and  0.927 for all values in tables 6 and 7, respectively. These small differences reflect the strength of the best-performing RoBERTa model that achieved an F1-score of 0.924, and we did not identify any values that contradict the claims made in the paper. However, we did notice that fewer claims could be reliably made given the smaller size of the data when using only 3000 vs 14561 texts. The number of LIWC scores that were not statistically significant in human-annotated data only, but were statistically significant in the full human+model-annotated data, is 40 scores and 28 scores for tables 6 and 7 respectively. This can be attributed to the increased size of the dataset, making a larger number of the results statistically significant while still controlling for multiple comparisons.\n\n2) Regarding the small coefficients in the GLMs, many thanks for pointing this out. After normalizing the follower count so that it is in the range 0-1, better matching the other variables which are boolean, we found it retained the same trend reported in the paper for the follower count and interaction terms that use the follower count. However, we agree with your suggestion and find these normalized numbers are more easily comparable and interpretable, so we will update with those numbers in the final version of the paper if accepted.\n\n3) We will also be eager to expand on the discussion section given an extra page if the paper is accepted. We believe there are useful insights regarding the potential for narratives to amplify the spread of information on Twitter and will add a specific subsection dedicated to \"recommendations\" for future work in this area.\n\n4) Regarding your request for examples: we chose not show the exact examples of tweets based on prior scholarly works such as the work by Fiesle and Proferes: “Participant” perceptions of Twitter research ethics.\" Social Media+ Society 4.1 (2018): 2056305118763366. which suggests that users prefer not to have their exact messages presented in research papers or being used as examples. Our commitment to not directly showcasing examples stems from our ethical stance to safeguard the privacy of the authors of the tweets. However, based on your helpful suggestion, we conducted a thematic analysis of the data in each category (narrative info, narrative misinfo, non-narrative info, non-narrative misinfo), which summarizes the main themes that were present in the tweets so that you can understand the general content of them, along with some paraphrased examples. These results will be added to the paper if accepted and we have included them at the end of this rebuttal (see item 6) for your reference.\n\n5) Regarding \"The narrative arc analysis in section 5.3.2 doesn't seem to add any value\" we mainly included this in order to compare to the tool proposed by Boyd et al. which was used to analyze narratives in prior work, since it is one of the only existing tools against which we could compare. Given that we did not want to blindly test for differences in every possible LIWC category, we chose categories that had been shown to be important to narratives by some previous work, and this one was set of such categories that we decided to explore. While some of the results were not statistically significant, others, such as differences in the use of Cognition words and personal pronouns did differ significantly between narratives and non narratives in ways that help to confirm previous findings about narrative texts.\n\n6) Based on reviewers' comments, in order to demonstrate the types of tweets that exist in our dataset, we conducted a manual thematic analysis of 400 tweets. 100 tweets were sampled from our full dataset from each of the following 4 categories: narrative information, non-narrative information, narrative misinformation, and non-narrative misinformation. Following a standard thematic analysis procedure, we first read through the sampled tweets, and then determine the main themes that existed. We coded each tweet and present the aggregate results below, including each theme, the percentage of the data subset that contained that theme, and one (paraphrased) example tweet to illustrate the theme. We will include these results in a formatted table in the final version of the paper, along with a discussion, if accepted. \n\nnon-narrative information\n\n- **don't worry about what's in the vaccine (30%)** Example: \"If you've ever been to that KFC on the street, don't worry about what's in the vaccine.\"\n- **get vaccinated (13%)** Example: \"Will be getting vaccine next week. _/\\_\"\n- **other (10%)** \"My sister argues with me about this vaccine everyday!\"\n- **sharing news (9%)** Example: \"The government will now begin vaccinating kids below the age of 10, starting Wednesday\"\n- **questioning/debunking conspiracy theories (7%)** Example: \"5G towers don't cause covid and there is no scientific evidence to support that claim. Stop spreading misinformation and please follow cdc guidelines and save lives\"\n- **frustration towards those not following guidelines (6%)** Example: \"So frustrating to see selfish people not follow rules. The pandemic isn't over yet. Please stay home and follow the rules.\"\n- **sharing local information (6%)** Example: \"vaccine information: vaccinations available for ages 14+ this Sunday 3/09 at ABC Pharmacy in Carlton. If interested please call their number at 999-999-9999 #stay vaccinated\"\n- **vaccine efficacy (6%)** Example: \"The vaccine protects you and your loved ones from this deadly virus. Please believe in science and get vaccinated and save lives.\"\n- **mislabeled - should be narrative (4%)**\n- **public health tips (3%)** Example: \"Stay indoors, wear masks and always sanitize your hands when you touch public areas.\"\n- **joke (3%)** Example: \"my American friend says covid is a planned agenda, but what about Bill Gates acquiring large areas of farm lands?LOL\"\n- **criticizing government (3%)** Example: \"Only 6% of the adults have received both doses. This is bad governance at its best. You should learn from South Korea\"\n\n\nnarrative information\n- **got vaccinated and feeling positive (39%)** Example: \"I got my second shot yesterday. I can finally go visit my mom and dad this weekend and i feel so grateful!!! <3\"\n- **feeling/anticipating vaccine side effects (15%)** Example: \"Got the jab two days ago and i am really feeling the effects. My arm is still sore and i feel the chills too. Not going to work tomorrow\"\n- **friends/family member received the shot/feeling positive (11%)** Example: \"My mom and dad got their first shot today and i feel so relived. Let's defeat this pandemic together. Peace\"\n- **eager to receive vaccine (10%)** Example: \"I could finally book my appointment for the vaccine. I am excited and can't wait to get the shot and meet my friends\"\n- **someone got covid (3%)** Example: \"Robert did not want to get vaccinated and guess what, he caught the virus and he's sick!!!!\"\n- **vaccine scheduling (3%)** Example: \"I have been trying to book vaccine appointment for weeks and still haven't made it, i feel so disappointed. Urghhhh!!!!!\"\n- **friends/family member vaccine scheduled/feeling positive (3%)** Example: \"My 78yo grandmother could finally book vaccine appointment for tomorrow. I feel so happy and grateful!! \"\n- **expressing gratitude towards healthcare workers (3%)** Example: \"i got my first shot yesterday and I thank all the frontline workers who have done phenomenally well during this pandemic. You saved countless lives. Feeling grateful!!\"\n- **other (13%)**  include themes like \"critisizing govt's handling of covid\", \"a joke\", \"setting example for others by getting vaccinated\" etc\n\nnon-narrative misinformation\n\n- **suspecting vaccine as gene therapy (19%)** Example: \"Quick reminder, this isn't any normal vaccine. THIS IS GENE THERAPY, which will make a human forever slave to the big tech\"\n- **vaccine hesitancy and concerns (19%)** Example: \"They be calling them idiots for not wanting a rushed untested vaccine, when it's the vaccine that causes infertility\"\n- **covid is a bioweapon (18%)** Example: \"Wake up!! This is not a virus, it's a carefully launched bioweapon.\"\n- **govt/large corp (14%)** Example: \"Fauci knows about the man-made virus long ago, it's all a cover up by the govt to keep the public under their shoes.\"\n- **covid is a depopulation agenda (13%)** Example: \"There are 10 million unplanned pregnancy terminations since the beginning of this virus. Don't be stupid, this is a depopulation attempt by Bill Gates and the govts to control population growth.\"\n- **covid home remedy (10%)** Example: \"Take mint leaves, honey, garlic and onion to prevent covid. you don't need the vaccine!!!!!!!!!\"\n- **other (7%)** Example: \"Sign this petition people! covid is hoax. #stopdepopulation #novax \"\n\nnarrative mis-information\n- **conspiracy theory (38%)** Example: \"Fauci knew that the virus has been cooking in the lab since 2005 but he covered it up and now Bill Gates is inserting chips in our body through vaccine to track us\"\n- **vaccine related deaths and side effects (28%)** Example: \"A boy in Netherlands found dead 30 hours after taking his second covid shot. He was fit until a day before he took that damn vaccine!!!!\"\n- **vaccine hesitancy (20%)** Example: \"I will not take that vaccine until i get pregnant and give birth to a baby. I don't want to lose my fertility to a manufactured virus and supposed vaccine -_-\"\n- **other (8%)** Example: \"I unfriended a good friend of mine yesterday because she's totally into this  5G conspiracy theory shit. I don't have any regrets\"\n- **covid home remedy (6%)** Example: \"I have been eating garlic, one table spoon of honey, some onions before i go to bed and let me tell you, i have not had covid since! This totally works! and is the best solution for the virus.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides a good motivation that narrative is an issue that needs to be addressed with misinformation and takes first steps toward a computational and quantitative analysis.", "reasons_to_reject": "Since the RoBERTa classifier chosen to annotate the reset of the dataset surely has some systematic errors, it would be more convincing to present results with respect to engagement (section 5) also just for the subset of manually annotated tweets to verify that there is no significant difference between those and the tweets that were annotated for narrative with the classifier.\n\nOtherwise, this paper would benefit most from a more thorough discussion and interpretation of all the results, which as it currently stand are difficult to tie together and so lack coherent findings that could benefit future work.\n\nThe results from the GLM seem inconclusive on the relationship between misinformation and narrative. As noted, there is a positive relationship for misinfo x narrative for ANTiVax but not CMU-MisCov19. The paper says that this indicates that false narratives about getting the vaccine get more engagement than other misinformation narratives (lines 433-435). Examples of those other types of misinformation narratives would be informative. I assume false narratives about getting the vaccine would be narratives about people getting the vaccine and then terrible things happening to them? Examples here would be valuable as well.\n\nEffect sizes or something else may help readers interpret the strength of these relationships, which is currently lacking. There is a positive relationship for both datasets for narrative x misinfo x follower count, but with very small coefficient values. The range of coefficient values varies dramatically overall (the -3.5 coefficient for misinfo x narrative for CMU-MisCov19 has a particularly large absolute value, which may be worth talking about).\n\nA more in-depth discussion of the results is needed to properly interpret and contextualize all of the results given. For example, it seems counterintuitive and interesting that there is more narrativity with informed tweets (if I'm reading section 5.1 right). What might be evidence from the tweets to explain this? Are people just sharing stories of getting the vaccine? In general, more interpretation of main themes coming from results in each subsection of section 5 would be helpful.\n\nThe LIWC analysis in section 5.3.3 seems to yield unsurprising results when it comes to what terms narratives would use more. More informative would be differences between narrative info and narrative misinfo, but the only difference detected is a focus on death terms. Unsurprising results are okay in general, and maybe the focus on death can contribute to what is not known about COVID misinformation (it is perhaps interesting that the informed tweets are not talking about avoiding death through taking the vaccine, for example), but more interpretation and discussion would be needed. The narrative arc analysis in section 5.3.2 doesn't seem to add any value, since as noted, tweets just seem too short for this tool to work properly and it seems like most of the results are statistically insignificant.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a system for automatic narrative detection with an application of characterizing the use of narrative in health misinformation on Twitter. The authors manually annotated an existing misinformation tweet dataset for the presence of narrative. They then built text classifiers to predict narrative style, and found that a fine-tuned RoBERTa model performed the best. Using that classifier to label the rest of the dataset, they explore different properties of the relationship between narrative style and misinformation. They find that narrative style is related to higher engagement and that for one of their datasets, narrative is associated with higher engagement with misinformation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* What does the Hydrated % row refer to in Table 1?\n* Figure 1 text is too small to easily read, other figures like Fig 2 and tables too\n* Line 338 , instead of .\n* Line 417: model -> models\n* Line 632: authentic used -> authentic, used"}}]

Author rebuttal:

Meta-review: This paper examines the interplay between narrative communication style and misinformative health content on user engagement on Twitter. Three reviewers, who reviewed the submission, highlighted the strengths of the paper. R1 appreciated its valuable dataset, R2 appreciated the analyses, and R3 appreciated the paper’s outcome in motivating this line of research. However, all the reviewers raised significant concerns about the paper, some of which were addressed during the rebuttal phase by the authors—and the reviewers accordingly updated their ratings. Particularly the reviewers commended the authors on the additional thematic analysis.
Decision: Accept-Findings