COVID-19 Vaccine Misinformation in Middle Income Countries
Authors: Jongin Kim
Byeo Rhee Bak
Aditya Agrawal
Jiaxi Wu
Veronika J. Wirtz
Traci Hong
Derry Wijaya
Venue: EMNLP 2023 Main (2023)
Keywords: COVID-19
vaccine
misinformation
NLP applications
domain-specific pre-training
text augmentation
distributed lag model

Abstract: This paper introduces a multilingual dataset of COVID-19 vaccine misinformation, consisting of annotated tweets from three middle-income countries: Brazil, Indonesia, and Nigeria. The expertly curated dataset includes annotations for 5,952 tweets, assessing their relevance to COVID-19 vaccines, presence of misinformation, and the themes of the misinformation. To address challenges posed by domain specificity, the low-resource setting, and data imbalance, we adopt two approaches for developing COVID-19 vaccine misinformation detection models: domain-specific pre-training and text augmentation using a large language model. Our best misinformation detection models demonstrate improvements ranging from 2.7 to 15.9 percentage points in macro F1-score compared to the baseline models. Additionally, we apply our misinformation detection models in a large-scale study of 19 million unlabeled tweets from the three countries between 2020 and 2022, showcasing the practical application of our dataset and models for detecting and analyzing vaccine misinformation in multiple countries and languages. Our analysis indicates that percentage changes in the number of new COVID-19 cases are positively associated with COVID-19 vaccine misinformation rates in a staggered manner for Brazil and Indonesia, and there are significant positive associations between the misinformation rates across the three countries.

COVID-19 Vaccine Misinformation in Middle Income Countries
Jongin Kim', Byeo Rhee Bak?, Aditya Agrawal', Jiaxi Wu", Veronika J. Wirtz Traci Hong Derry Wijaya
DDept. of Computer Science, Boston University 'Dept. of Economics, Boston University Coolleee of Communication, Boston University "School of Public Health, Boston University CCntter for Emerging Infectious Diseases Policy & Research, Boston University
MMonsh University Indonesia
(jongin, brbak, adityaai, jiaxiw, wwirtz, tjhong, wijaya}ebu. edu
Abstract
et al., 2018). Given that middle income countries (MIC) make up 75% of the world's population and 62% of the world's poor (World Bank, 2022), study- ing vaccine misinformation MIC important to preparing for future pandemics especially be- cause global uptake to vaccines is needed to end pandemics (Asundi et al., 2021). Importantly, the impact of misinformation vaccine hesitancy tends beyond COVID-19 in that there has been a precipitous decline in childhood vaccination rates in MIC (Guglielmi. 2022). The residual effects of vaccine misinformation on vaccine hesitancy presents a global health problem for containing infectious diseases.
This paper introduces multilingual dataset of COVID-19 vaccine misinformation, consisting annotated tweets from three middle-income countries: Brazil, Indonesia, and Nigeria. The expertly curated dataset includes annotations for 5,952 tweets, assessing their relevance to COVID-19 19 vaccines, presence of misinforma- tion, and the themes of the misinformation. To address challenges posed by domain speci- ficity, the low-resource setting, and data im- balance, we adopt two approaches for devel- oping COVID-19 vaccine misinformation de- tection models: domain-specitn pre-traie traininig text augmentation using a large language model. Our best misinformation detection mod- els demonstrate improvements ranging from 2.7 percentage points macro Fl-score compared to the baseline models. Additionally, we apply our misinformation detection models in large-scale study of 19 million unlabeled tweets from the three countries between 2020 and 2022, showcasing the practical application of our dataset and models for detecting and analyzing vaccine misinformation in multiple countries and languages. Our analysis indicates that percentage changes in the number of new COVID-19 cases are positively associated with COVID-19 vaccine misinformation rates in staggered manner for Brazil and Indonesia, and there are significant positive associations be- tween the misinformation rates across the three countries.
In this study, we focus on three middle-income countries: Brazil, Indonesia, and Nigeria, which have the majority of their population using social media (Schumacher and Kent, 2020) and where vaccine confidence has eroded in recent years (De Figueiredo et al., 2020). In addition, each of these three countries are population hubs from three distinct regions in the world. While the ma- jority of the population in these countries use so- cial media (Guglielmi, 2022), social media usage is still projected to grow in these MIC countries whereas usage has plateaued in high-income coun- tries (Poushter al., 2018). This presents unique opportunity to study the impact of social media on vaccine misinformation in MIC.
The development of an automated system that can detect vaccine misinformation will allow for epidemic surveillance in three heavily populated countries in distinct parts of the world. This will allow government agencies, health agencies, and researchers to monitor misinformation and coordi- nate intervention strategies deter emerging infec- tious diseases. In addition, the automatic system for misinformation detection allows for identification of shared misinformation themes across three dis- tinct countrics/regions, which would enable health organizations to tailor health messages.
1 Introduction
Affluent countries have enjoyed ample supply of COVID-19 vaccines but distribution to low- and middle-income countries were slow. The void cre- ated by slow vaccine roll out may create space for misinformation to further flourish. Misinforma- tion on social media already documented to con- tribute to vaccine hesitancy in developed countries (Puri et al., 2020), but comparatively few studies have focused beyond high income settings (Hagg

---

In this work, we make two main contributions. Firstly, we have curated a new multilingual dataset of COVID-19 vaccine misinformation from MIC, where social media misinformation is understud- ied. This 'ataset! is the first of its kind in several aspects: it is geolocated, in multiple languages, carefully curated, covers wide time range from 2020 to 2022, and includes not only domain-expert annotations of misinformation, but also the iden- tified themes based on a previous study of glob- ally cireulating COVID-19 19 vaceine information on online platforms (Islam et al., 2021). Secondly, utilizing this dataset, we have trained models that effectively detect misinformation and its associated themes, outperforming competitive baselines, in- cluding GPT-3 models. We apply our best models in large scale study to analyze the trend of mis- information in the three MIC between 2020 and 2022. Our analysis yields interesting findings re- garding the relationship between COVID-19 cases, vaccine misinformation rates, and variations across the three countries.
et al., 2013). Media plays prominent role in spreading misinformation. For instance, longi- tudinal study on mother's s attitudes towards MMR vaccines reveals a link between media-published misinformation, their perception of vaccine safety, and vaccination rates among children in the United Kingdom (Smith et al., 2007). Similarly, negative press in local media in the UK was associated with decline in MMR vaccinations (Mason and Don- nelly, 2000). The impact of misinformation on vaccination coverage is not limited to Europe; it is observed globally. A well-documented case is polio in Central and West Africa, where in the late 1990s and early 2000s misinformation about the po- lio vaccine spread within Muslim communities in Northern Nigeria, Benin, Burkina Faso, Cameroon, Central African Republic, and other neighboring countries (Jegede, 2007). leading to leaders refus- ing vaccination and subsequent outbreaks of the disease (WHO, 2006).
Dataset
2 Related Works
We collect all publicly available Twitter posts in Brazil, Indonesia, and Nigeria that contain vaccine- related terms from January 2020 to December 2022 (N = (89800,.31). We use the hydration process (Arafat et al., 2021) where we use Brandwatch API to scrape IDs of tweets that contained predefined vaccine-related terms, then apply Twitter API to retrieve tweet contents.
While there have been studies vaccine misin- formation, no prior study has examined misinfor- mation across countries or studied shared misin- formation themes across MIC. Studies on vaccine misinformation utilizing Twitter data tend to be country-specific, predominantly focused on the United States (Pierri et al., 2022), restricted to the English language only (Feathhesttone et al., 2020), lack specified geolocations (Argyris et al., 2022). Moreover, these studies conducted during lim- ited periods in 2021 (Pierri et al., 2022: Argyris et al., 2022), or in 2020 (Featherstone et al., 2020). Similarly, most previous works introducing new datasets related to COVID-19 misinformation pri- marily focus on English content from specific target locations (Hayawi et al., 2022; Muric et al., 2021; DeVerna et al.. 2021; Weinzierl and Harabagiu, 2022b, 2021, 2022a; Hong al., 2023), while Mubarak et al. (2022) released an Arabic tweet dataset covering countries in the Arab region.
Brandwatch enables us to scrape tweets specific to geolocations associated with the three focal coun- tries. There are language exclusions. In addition. we include "shot" as a search term because it is of- ten used to describe vaccine, but we set limits reduce data noise, particularly from references to parties and alcohol that are synonymous with the term "shot". Our complete list of search query is listed in Appendix A. In addition, we include country- terms including "vacina," "inje- cao," and "Zé Gotinha" for Brazil; and "vaksin" and "suntik" " for Indonesia per the Oxford Languages Word for Vax (Oxford Languages, 2021).
For data annotation, we employ Quantitative Content Analysis (QCA) in communication re- search (Krippendorff, 2018) where representa- tive sample of the data is drawn and on which two or more trained coders (i.e., annotators) apply a codebook protocol that contains all the variables for annotation and their definitions. Similar to (Liu et al., 2019; Guo et al., 2021), prior to coding the
Studies on vaccine misinformation are impor- tant as misinformation exposure contributes to in- creased vaccine hesitancy and reduced behavioral intention to get vaccinated (Lee et al., 2022; Dubé
'The dataset and annotation codebook, which contains the operational definitions and examples of annotation variables including the misinformation themes, are available at https: //github. cwm/ zzol iman/covid d-vaccine woeciie--misifaa misinfo--Ii

---

Q
yes (%)
no (%)
uncertain (%)
total
evance (0.98); (Q2) non-COvID vaccine (0.97);
Q1
3, 3,666 (61.6)
2,286 (38.4)
5,952
(Q3) misinformation (0.72). For misinformation
Q2
655 (17.9)
3 (82.1)
3,666
Q3
1,119 (37.2)
1,596 (53.0)
296 (9.8)
3,011
themes: (Q4a) vaccine development, availability,
Q4a
186 (16.6)
933 (83.4)
1,119
or access (0.88); (Q4b) safety, efficacy or accep- tance (0.88); (Q4c) infertility (0.89); (Q4d) po- litical/cconomic motives (0.87); (Q4c) mandatory vaccine and ethics (0.88); (Q4f) vaccine reagents (0.87); (Q4g) vaccine morbidity or mortality (0.88); and (Q4h) vaccine alternatives (0.89). Upon deter- mination that human coding reliable, the trained coders independently code the remaining posts. Fi- nally, tweets annotated with "uncertain" for misin- formation are reviewed by our expert public health panel who make the final determination.
Q4b
539 (48.2)
580 (51.8)
I.119
Q4c
26 (2.3)
1,093 (97.7)
1,119
Q4d
412 (36.8)
707 (63.2)
1,119
Q4e
106 (9.5)
1,013 (90.5)
1,119
Q41
93 (8.3)
1,026 (91.7)
1,119
Q4g
272 (24.3)
847 (75.7)
1,119
Q4h
29 (2.6)
1,090 (97.4)
1,119
Table I: The statistics of annotated tweets for each ques- tion. Q1: relevance to vaccine; Q2: mentions specific non-COVID vaccine; Q3: presence of misinformation; and misinformation themes: Q4a: vaccine development, availability, or access; Q4b; safety, efficacy, or accep- tance; Q4c: infertility; Q4d: political/economic mo- tives; Q4e: mandatory vaccine and ethics; Q4f: vaccine reagents: Q4g: vaccine morbidity or mortality; and Q4h: vaccine alternatives.
To balance the data for misinformation, we iden- tify more tweets that contain misinformation, sulting in the final annotated dataset of N=5,952. Table shows the statistics of the annotated dataset.
entire sample independently, coders trained on the codebook and their agreement on how to apply the codes is measured with inter-coder-reliabiltty (ICR), where high values imply that the coders con- sistently annotate the data and signal the high valid- ity of the annotations. Once they reach an accept- able ICR, coders code the rest of the sample inde- pendently. We use GWET's ICR measures (Gwet, 2008) where agreements are considered substan- tial (between 0.61-0.8) or near-perfect (between 0.81-0.99) (Landis and Koch, 1977).
COVID-19 Vaccine Misinformation Models
In this section, we describe how we develop our COVID-19 vaccine misinformation models and present the experiment results.
4.1 Classification Models
To conduct study on the large-scale tweets we collected using our vaccine-related query terms ($3), we first train vaccine relevance model to distinguish tweets that are truly relevant to vac- cines from the dataset. This model is trained using the annotations for QI (Table 1). Next, we train a COVID-19 19 vaccine relevance model to identify among vaccine-relevant tweets, tweets that are spe- cific to COVID-19 vaccines. This model is trained using the annotations for Q2 Then, we train a model to detect tweets containing COVID-19 19 vac- cine misinformation. Training for this model was based on annotations for Q3. Tweets that were annotated to contain misinformation (Q3) were fur- ther annotated for whether they are related to a specific theme (Q4a to Q4h). We develop 8 binary classification models to identify themes of misin- formation using the corresponding annotations.
To apply QCA, we randomly sample 5,500 tweets, stratified by the distribution of tweets col- lected in each of the three focal countries. Two communication graduate students are trained as coders to examine the tweets for: (Q1) rele- vance to vaccine (yes/no), (Q2) mentions of spe- cific non-COVID vaccine (such as the MMR vac- cine) (yes/no), (Q3) presence of misinformation (yes/no/uncertain). Content deemed to be misin- formation is labeled further for its themes based on a pre-identified list of COVID-19 vaccine mis- information themes (Islam et al., 2021), which in- cluded misinformation pertinent (Q4a) vaccine development, availability or access; (Q4b) safety, efficacy, or acceptance; (Q4c) infertility; (Q4d) po- litical/cconomic motives; (Q4e) mandatory vaccine and ethics; (Q4f) vaccine reagents; (Q4g) vaccine morbidity or mortality: (Q4h) vaccine alternatives.
4.2 Tweets Preprocessing
The labeled dataset was preprocessed before being used train the models. All mentions of username
To assess inter-coder reliability, 550 posts are randomly selected from the data (N=5,500) and both trained coders independently code the selected posts. The mean reliability for the variables in our study is .8, with the following scores: (Q1) rel-
'hhe assumption made was that tweets that do not mention a specific non-COVID vaccine are COVID-19 vaccine-related tweets. This assumption takes into account that the tweets were collected during the COVID-19 pandemic.

---

were replaced with "@ user"" and all URLs were replaced with "http".
in Figure 1, we prompt GPT-3 with set of 10 ex- ample tweets belonging to the same category and append a new example line for GPT-3 to generate tweet that is aligned with the target category and the examples provided. For each generation, the example tweets are randomly selected from posi- tive examples in the training set. We refer to the model that is based on XLM-R+ fine-tuned the annotated and augmented data as XL.M-R+AUGG.
4.3 Pre-trained Model and Domain-specific Pre-training
In our experiment, we use XLM- LLLM-RaERRTaa (Con- neau et al., 2020, XLM-R), which is Transformer- based multilingual language model pre-trained on large multilingual corpus, as our base encoder for all our classification models. This model chosen since our dataset contains tweets written from vari- ous languages including English, Portuguese, and Indonesian. English is the official and most widely spoken language in Nigeria and is also widely used in Brazil and Indonesia. Portuguese and Indone- sian are the official and national language of Brazil and Indonesia, respectively.
Input Prompt
The following tweets containing misinformation about vaccines belong to the same category Infertility:
The only reason why I'm not getting vacoinated so can conate my sperm for mill Example 2: Vaccine whistleblower says new GSK vaccine causes infertility in 97% of women http
Example 1:
Given that our experiment focuses on domain- specific data, specifically texts from tweets that are related to COVID-19 vaccine, we recognize the importance of using pre-trained model optimized for this type of data. Therefore, we further pre-trail) train the XLM-R on the entirety of tweets collect using our vaccine-related query terms and use it as another base encoder for our experiments. We refer classification models that are fine-tuned on this encoder as XLM-R+. By pre-training the model on the sppaci- dataset, we aim to improve its performance and ability to accurately classify COVID-19 vaccine-related tweets. This approach has been shown to be effective in improving the performance of natural language processing models domain-specifee texts (Gururangan al., 2020).
Example 10: How many pregnant women have to lose their babies before the government acknowledes that
vaccines should NEVER be given during pregnancy http Example 11:
Model Output
My friend's cousin got COVID-19 vaccine and now she can't get pregnant. Vaccines dangerous and should be avoided. #vaccinesarepoison
Figure I: Data augmentation process using GPT-3 with example input and generated output. Given a category and 10 example tweets, the prompt is constructed fol- lowing the template above. The prompt is then fed into GPT-3 obtain generated tweet augment our dataset.
4.4 Text Augmentation
4.5 Evaluation Setting
To alleviate the issue of imbalanced dataset when training models for classifying misinformation (Q3) and themes (Q4a to Q4h), we leverage GPT- 3 (Brown et al., 2020), large language model, augment the training data for each model. In particular, given the limited availability of labeled positive examples, i.e., tweets containing misinfor- mation and tweets associated with specific themes of misinformation (Table 1), we employ an augmen- tation technique to amplify the positive examples. Drawing inspiration from the work of Sahu et al. (2022), we employ straightforward prompt-based based approach to generate additional data. As illustrated
In order to train and evaluate our model, we employ 5-fold cross validation. For hyperparameter search, we experiment on the combinations of batch size {8, 16, 32), learning rate 1e-5, 2e-5, 3e-5, 4e-5, 5e-5), and training epochs ranging from 1 to 10. The final models used in large-scale study are trained the entire annotated data with the best hyperparameters determined during the cross vali- dation process. When applying text augmentation (described in $4.4), we combine the augmented data from each fold with the annotated data for training the final models.
We employ the macro F-1 score (averaged over the 5-folds) as the performance metric to evaluate our models as the classification accuracy is not a reliable measure for evaluating model performance
'The model is initialized with weights xlm-roberta- from HuggingFace
"We use GPT-3 fext-davinci-000 with temperature of 01. to generate tokens.

---

when the distribution of class labels in the dataset is highly skewed such as in our dataset (Table 1).
pairs to construct an input prompt. To ensure that both positive and negative examples are evenly in- cluded the input prompt, three (or two) positive examples and two (or three) negative examples are randomly chosen from the training set. The exam- ples and details of the input prompts are described in Appendix A.2. Table 3 leftmost columns show the results for GPT-3 zero-shot (GPT-3 ZS) and few-shot (GPT-3 FS) classification performances. Our fine-tuned models SELM-R+AAUGE outper- form GPT-3 model performance on the dataset.
4.6 Experiment Results
Table 2 shows the performances for vaccine rele-
vance (QI) and COVTD-19 19 vaccine relevance (Q2) models. Both models achieved high macro F-1 scores (i.e., a score of 97.8 and 92.9 for QI and Q2 respectively, when fine-tuning XLM-R+). Due to the high performance of the baseline XLM-R model and the relatively larger size of the dataset dataset
compared to the misinformation-relle (i.e., Q3 and Q4a to Q4h), the performance gain from domain-speciftt pre-training (i.e., XLM-R+) is smaller compared to the misinforraaioon-rllteed results presented in Table 3.
Total w/ CI
Brazil
Indonesia
Nigeria
Q1
97.85 (# 0.44)
97.26
97.00
98.34
Q2
92.89 (= 0.94)
96.36
92.41
92.99
Q3
81.25 (+ 1.88)
72.25
76.99
$1.05
XLMMR
XIM-R+
Gain
Q4a
80.60 (# 2.85)
79.67
68.49
76.27
Q1
97.0
97.8
0.8
Q4b
80.16 (# 1.19)
77.54
75.44
82.19
Q2
91.6
92.9
1.3
Q4c
87.66 (#: 7.73)
96.34
77.77
89.87
Q4d
79.15 (+: 3,15)
80.80
74.95
81.84
Table 2: Macro F-1 Score for vaccine relevance (QI) and COVID-19 vaccine relevance (Q2) models. Best performance for each question is in bold.
Q4e
85.04 (4 3.05)
68.89
81.71
89.85
Q4f
83.84 (+: 2.47)
69.87
77.85
83.93
Q4g
81,83 (# 3.95)
78.23
76.54
85.66
As shown in Table 3, the models combining domain-specifie pre-training and text augmenta- tion strategy (i.c., XLM-R+ AUG) significantly outperform the baseline models (XLM-R), with the performance gain ranging from 2.7 to 15.9 percent points. In particular, we observe that the performance gain from text augmentation more pronounced when the original annotated data is extremely imbalanced (i.e.. Q4c and Q4h). This highlights the effectiveness of text augmentation using large language models to address challenges posed by imbalanced datasets.
Q4h
75.36 (1: 15.15)
78.79
81.38
77.90
Table 4: Macro F-1 Score of the best-performing model on the total dataset (with 95% Confidence Interval) and on each country-specific dataset. For QI and Q2, the best pest-performing model is XLM-R+, and for the rest is XLM-R+ --GG.A For each Question, the country with the highest performance is in bold.
As we conduct an extensive analysis of tweets collected from three middle- countries ($5), we also present per-country evaluations, focusing on the best-performing model for each question: XLMMR+ for Q1 and Q2, and XLM-R+_AUG for Q3 and Q4a to Q4h. As shown in Table 4, perfor- mance varies across countries. Notably, most mod-
25
3S
XLM-R XLM-R+
RtMAR
Q3
+1.6
Q4a
77.9
+2.
76.5
43.6
83.8
57.7
439
els perform best on tweets from Nigeria, followed
45.6
61.1
76.5
78.0
79.2
+27
by Brazil and Indonesia. Further analysis suggests that this performance closely linked to the lan- guage distribution of tweets within each country. Specifically, for Nigeria, 99.37% of the tweets were in English, followed by Brazil at 77.16% and In- donesia at 45.59%. The enhanced performance, correlated with higher English tweet ratio, can be attributed the significant proportion of English in 1) the corpus used for XLM-R pre-training, 2) the tweets employed during the further pre-training of XLM-R+, and 3) the annotated tweets utilized in fine-tuning the base encoders.
Q4c
55.5
782
81.1
$4.0
85.01
+4.9
q4t
479
51.1
79.7
82.3
8388
142
Q9:
43.1
67.8
77.7
50.6
81.8
+4.3
Q4h
49.3
48.4
59.4
63.7
75,4
+159
Table 3: Macro F-1 Score for GPT-3 Zero-Shot (GPT-3
7S) and Few-Shot (GPT-3 FS), XLMMRR XIM-R+, and XLM-R+ for Q3 and Q4a to Q4h. Best perfor- mance for each question is in bold.
We also evaluate GPT-3's zero-shot and few-shot classification performance on our dataset, averaged over the same 5-folds ($4.5). In the few-shot set- ting, we use (=5) examples of tweets and label

---

Boaii
Indeocia
Nigeria
2021
2022
County
2031
Cresasy Roal
2020
2021
Country Toed
oC Eveeh Colloced
722,06
1441077
1.93.011
5.344644
L514,56
4,221,655
1.58,214
7,493,06
2827.268
L925,223
1,772,000
5,25.201
18,901221
(100)
(188)
(000)
(000)
(100)
1280)
(192)
(093)
(198)
(100)
1200)
(100)
6C Eveets Addvus
211,81
853.139
2138095
3.93.114
634,90
3.103359
193,019
4571827
51386:
055.48
143.00
1.52,
9.51,445
(896)
(901)
(16.7)
(907)
(51.3)
of Towess Belinorr in
SW
$71.373
2
2580-33
1699201
646,271
3.61.355
T76
131/85
7100800
115,80
1 162.016
COVID 19 Vusine
(95.5)
(98.0)
(96.4)
(55.0)
195.9)
(96.6
(96.))
(9.3)
(56.7)
(98.5
68.00
(90.)
(95.7)
COVID Musiie Misiflommzicn
(25.0)
136.3
(20.)
(268)
Table 5: Summary of prediction results the total collected tweets.
5 Analysis
with diseases such HIV and Hepatitis B posing risks to the public.
For the final models, the best-perforning model for each question, discovered during the cross- validation process (Table and Table 3), is trained on the entire labeled data. Final models for Q3 and Q4a to Q4h are trained on the combination of labeled data and augmented data from all folds (as described in $4.4 and $4.5).
5.2 Percentage changes in new COVID-19 cases and COVID-19 vaccine misinformation
This section aims to examine the impact of daily percentage changes in new COVID-19 cases on the daily rate of COVID-19 vaccine misinforma- tion in Brazil, Indonesia, and Nigeria. Our anal- ysis entails a causal interpretation of the findings, contingent upon the fulfillment of standard assump- tions inherent to distributed lag model, including the exogeneity of (contemporaneous and lagged) daily rates of misinformation. The absence of the assumptions implies that the outcomes should be construed merely as indicative of an association.
Subsequently, these final models are applied step-by-step from Q1 to Q4 to 19 million col- lected tweets. First, we predict whether tweet is relevant to vaccines using our vaccine relevance classification model (Q1) and filter out tweets that are not relevant to vaccines. Then, we filtered out tweets that are predicted to mention non-COVID vaccines (Q2). Next, we use our COVID-19 vac- cine misinformation classification model (Q3) to predict whether tweets contain misinformation. Through these sequential predictions, we extract COVID-19 vaceine misinformation from the col- lected tweets, and perform analysis on them.
The model is as follows:
Y/ +o -12+ 1 115 (1)
The above model was separately ran for each country. Subscript t represents date, and ye is COVID-19 vaccine misinformation rate, defined #tweets with vaccine misinformation in date 6. H, is de- #tweets with any vaccine information
5.1 Statistics of prediction results
Table 5 shows the summary of prediction results us- ing our final models. In Nigeria, among the tweets collected, only 22.0% of tweets were predicted as relevant to vaccines. Upon further examination, in October 2020 in particular, we find that most of the tweets that contain the term "shots" are related to the EndSARS protest, which was protest against the illegal violence by the Special Anti-Robbery Squad (SARS) that has spread throughout Nige- ria. This shows that relying solely on the search query terms to identify vaccine-related tweets has limitations of including posts that are not actually relevant to the topic; highlighting the need for ma- chine learning-basse based classification system.
as fined as 100 #new
#new cases in date I-i and uy is an error term. In addition to considering the immediate number of newly infected individuals, the histor- ical number of cases may also contribute to the propagation of misinformation. Thus, we have in- corporated lagged percentage changes up to the 15th day (or 14th lag) for each country. The num- ber of new cases are drawn from WHO COVID-19 data (OWID).
date
We exclude the early time periods which the total number of COVID-19 cases in each country had not yet surpassed 1,000. This exclusion is nec- essary due to the lack of a well-defined percentage change in new cases during these periods with nu- merous instances of zero new cases. Furthermore, for this analysis, we truncate the data after the first quarter of 2022, as the increasing trends in misin- formation rates suggest potential structural changes that undermine the stationarity assumption of the distributed lag model. However, it is worth noting
Also, compared to the other two countries, Nige- ria has relatively lower proportion of COVID-19 vaccine-related tweets: 90.6% of vaccine-related tweets. Tweets related to non-COVID vaccines mention vaccines for HIV, Hepatitis B, HPV, Mon- keypox, etc. This shows that Nigeria faces signifi- cant public health challenges beyond COVID-19,

---

Crock
Pegara
Maccra MArTi Changio Clean
KiaLIn Maaira
gour Caack
Figure 2: Weekly Time Series of Misinformation Rate and % Change in News Cases
Dynamio
Donariic Effcct Number of Cond-19
Oynamio Effect 1% Incresso humnarr Covid Canse
hammer New Codd- 19 Canes
(a)
Figure 3: Dynamic Effect of 1 1% increase in the Number of New COVID-19 Cases
that the inclusion of these later periods does not significantly alter the qualitative results, and the Augmented Dickey-Fuller test confirmed that all six time series depicted in Figure 2 are stationary at 0.01 significance level. Heteroskedasticity and autocorrelation-coniiitee (HAC) standard errors are used for the inference.
in the number of new cases today increases mis- information rate by 0.0029 during the first 6 days. This is sizable compared to the mean and the stan- dard deviation of misinformation rate of Indonesia. which are .1996 and .0936, respectively, where 32% increase in new cases amounting almost standard deviation increase in the misinformation rate. The coefficient increases again from the 8th lag and subsides after 1th lag. but we cannot reject that the coefficient for 8th to 14th lag are jointly 0 (p=2255)).
Figure 3a illustrates the dynamic effect in Brazil with 95% confidence interval for each lag. The % changes in new cases take more than week to affect misinformation rate in Brazil, and the ef- feet subsides. For the early lags up 7th lag, cannot reject that the regression coefficients are jointly 0 (p=0.3896). However, the coefficients for 8th to 14th lags are jointly different from (p-=0.0248). Specifically, the 10th lag has cocffi- cient of approximately ,0001, and it interprets as I percent increase in the number of new cases to- day increases misinformation rate by .0001 after 10 days. The coefficient is statistically significant but not effectively large in magnitude, considering that the mean and the standard deviation of misin- formation rate Brazil during the sample period are .2655 and 1101, respectively.
Figure 3c shows some coefficients with statisti- cal significance, but these are small in magnitude and their signs are alternating. This inconsistent finding is likely attributed the Nigerian govern- ment banning Twitter from 2021-2022 which re- stricted Nigerians from accessing Twitter. The gov- ernment ban was due to political and economic factors and was lifted when Twitter agreed to legal and financial terms (BBC News, 2022).
IDN
NGA
NGA
BRA
0.189
0251 (0:017)
(0.045)
IDN
0
0.200*
(0.04)
(0.083)
Figure 3b indicates that the % changes in new cases contributes to misinformation in Indonesia at rate faster than Brazil. The magnitude of the dynamic effect increases until the 2nd lag and sub- sides lose statistical significance from the 6th lag. The effect is much larger in magnitude, even when we only consider the carly lags with statistical sig- nificance. The model estimates that I % increase
NGA
0.288
0, 0.164"
(0.08)
(0.067)
Constant
0.214
0.194
0.14 (0.014)
0.159 (0.010)
0L174 00.02)
0
(0016)
(0013)
(0.015)
Oiservation
703
074
703
04
014
174
Not:: BRA, IND, and NGA indeate meinformation rate of Braeil Indunesia, Nigeia, nopertivdl Column varséles are dkepmdenn varishles, and mon varsiliss ane independen: varialles, Each columa repeesents a differest regression. HAC standard erors porentheses. 1.10 0,05 (.0).
Table 6: Coefficients from OLS Regressions Using COVID-19 Vaccine Misinformation Ratios

---

COVID VoccinG Nonnbmador Trands Councy SCMATTS
Hight
10000
20000
00:00
12
30W6
Mechie developeets
COAN 39 IMHO.
Figure 4: Trends of COVID-19 Vaccine Misinformation Themes by Country (Counts)
tripprtiont
and rooicaityy
VACOINE reageris
manderio derttry vnociie
and eCSTAIT: masions
and neceptance
Vacc no development.
Figure 5: Trends of COVID-19 Vaccine Misinformation Themes by Country (Proportion)
5.3 Association of COVID-19 vaccine misinformation between countries
(Muhammed T and Mathew, 2022). also worth noting that a qualitative study of COVID-19 mis- information in Indonesia has observed that misin- formation were prevalent during the early times of the pandemic where there was lot of uncertainty about the discase and later during the peak of the Delta-variant crisis that brought about lot of cases and deaths (Sodikin, 2022). Understanding this phenomenon can potentially inform effective models for mitigating misinformation.
We hypothesize a positive association for vaccine misinformation rates across the three countries. We employed simple linear regression model with HAC standard errors. The same data truncation is applied in 5.2, yielding different number of observations across specifications in Table 6. The truncation the observations that are second quar- ter of 2022 or later should decrease the coefficients due to the co-rising trends in the period illustrated in Figure 2. Nonetheless, misinformation rates are positively associated at least at the .05 significance level in the truncated period.
Figure 5 illustrates the monthly proportion of each misinformation theme. In all three countries, the proportion of the vaccine development, avail- ability, and access theme rises and declines over time. This can be attributed to the prevalence of unverified information regarding vaccine develop- ment, trials, trial participants, and procurement prior to the introduction of vaccines (Jan March 2021). Additionally, the safety, efficacy, and accep- tance theme was the most dominant across all three countries. This may reflect concerns surrounding the rapid development of COVID-19 vaccines and their emergency use authorization.
5.4 Qualitative Analysis of Tweets
Figure 4 shows the number of tweets containing COVID-19 vaccine misinformation basis, along with the monthly COVID-19 con- firmed cases. The peak months for the number of tweets containing misinformation vary across coun- tries and often coincide with different periods of rapid increase in confirmed COVID-19 cases, sug- gesting that the prevalence misinformation may be correlated with the number of cases. This ob- servation aligns with previous studies which have observed how misinformation thrives where people have little control over their environmental threats, which include the global pandemic (Nyilasy, 2019); how misinformation tends be more prevalent in the time of crisis when there is a lack in- formation needed to make emergency decisions
monthly
In addition, we analyze tweets belonging to dif- ferent themes and examine variations across coun- tries. The analysis revealed that even within the same theme. there are cultural and political differ- ences among countries that contribute to variations in tweets. This highlights the need for customized measures tailored local cireumstances effec- tively address vaccine misinformation.
Tweets falling under political and economic mo- tives often mentioned profit-driven pharmaceutical

---

companies developing vaccines despite the exis- lence of potential COVID-19 treatments. Some tweets also suggested intentional virus spread by certain countries to enhance their negotiation power. In Indonesia, there was a significant number of tweets attributing the source of misinformation to domestic individuals or regions, indicating a fair amount of misinformation being produced and dis- seminated within the country. This has implications for public agencies, emphasizing the importance of not only addressing misinformation from external sources but also engaging with internal sources. On the other hand, in Nigeria, there were numerous tweets expressing distrust towards the government political elites, suggesting that the government would misuse vaccines for personal or party-related purposes. This is contrary to the focus misin- formation regarding foreign countries pharma- ceutical companies in the other two countries. serves as an important example demonstrating how different content within the misinformation theme can gain attention due to local political factors.
new COVID-19 cases increase misinformation rate over time for Brazil and Indonesia. We find pos- itive association misinformation rates between Brazil, Indonesia, and Nigeria. Additionally, our qualitative content analysis on tweets shows that vaccine misinformation can be contingent on local factors. We believe our dataset would allow pub- lic health communicators and researcher to discern the types of misinformation that are prevalent in each country and facilitate targeted communica- tion strategies that are specifically tailored local circumstances.
Limitations
Twitter ban in Nigeria
We acknowledge that there may be inaccuracies in analysis of data from Nigeria for the period from 2021 to 2022 due to the Twitter ban imposed by the Nigerian government ($5.2). Therefore, we plan to collect additional data on countries the African region that are geographically, economically, and culturally similar to Nigeria (e.g.. Ghana) in the fu- ture. This will enable comparative analysis of how communication social media regarding misin- formation differs between Nigeria, where there external intervention in Twitter usage, and coun- tries where such intervention did not occur.
Furthermore, we discover distinct variations in the content of tweets belonging to the vaceine reagents/ingredients misinformation theme across countries. In Brazil and Nigeria, considerable number of tweets discussed the topic of microchips embedded in vaccines to read people's minds or track individuals. In contrast, in Indonesia, there was unique trend where tweets mentioned the use of vaccine ingredients that were religiously pro- hibited (not halal). Such misinformation reflects specific anxicties within the Indonesian context, where halal certification holds great importance for the Muslim population. It serves as a reminder that combating vaccine misinformation requires tailored and culturally sensitive approaches.
Suboptimal Model
Furthermore, while our models have outperformed competitive baselines including GPT-3 models and while have qualitatively examined large num- ber of our model's predictions in our qualitative analysis ($ 5.4), we recognize that the model's per- formance may still be suboptimal. This is likely be- cause our current methodology relies solely on the linguistic and semantic features of tweets. There- fore, in the future, we plan to benchmark method- ologies that leverage external knowledge (Hu et al.. 2021) or consider the external context surround- ing tweets (Sheng et al., 2022), in addition to their inherent properties, on our data.
6 Conclusion
In this work, we present multilingual dataset of tweets from three middle income countries anno- tated with their relevance to COVID-19 vaccines, the presence of misinformation, and themes of misinformation. Our method leveraging domain- specific pre-training and text augmentation demon- strate performance improvement for detecting mis- information and its themes. We also show appli- cations of our COVID-19 vaccine misinformation system by conducting extensive analysis on large corpus of tweets from 2020 to 2022. Distributed lag model indicates that % changes in the number of
Data Alterations for Analysis in $5.2 and $5.3
26 observations are interpolated to generate % change in new COVID-19 cases variable and to estimate HAC standard errors. These are 25 ob- servations with zero new cases (2 from Brazil and 23 from Nigeria) and missing observation (from Brazil). We used smoothed number of new cases to replace the zeros, and took an average of neigh-

---

boring number of new cases to replace the missing value. However, we believe that the influence of this alteration scheme should be marginal at best. Observations from Nigeria suffers more severely from the issue of zero new cases after the first quar- ter of 2022, but these were not used for the analysis.
Young Auna Argyris. Nan Zhang, Bidhan Bashyal, and Pang-Ning Tan. 2022. Using deep learning to iden- tify linguistic features that facilitate inhibit the propagation of anti-and pro-vaceine content on social media. In 2022 IEEE International Conference on Digital Health (ICDIT), pages 107-116. IEEE.
Archana Asundi, Colin O'Leary, and Nahid Bhadelia. 2021. Global covid-19 vaccine inequity: The scope, the impact, and the challenges. Cell host & microbe, 29(7):1036-1033
Causal Interpretation for Analysis $5.2 and $5.3
The coefficients in $5.2 be interpreted as
(lagged) effects in a causal sense, given that the standard assumptions for the model are satisfied. We have made efforts to ensure that some of the assumptions are satisfied by truncating the analysis periods and running relevant tests. We do not in- tend the result in $5.3 to be interpreted as causal, because existence of confounding factors cannot be ruled out.
BBC News. 2022. Twitter agrees to Nigeria's 's demands to end seven-month ban. BBC News.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems, 33: 13:7797-1901.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Out, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross- lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Ethics Statement
The study protocol reviewed by and deter- mined not pertain human subjects research by the authors' Institutional Review Board. To comply with Twitter Content Redistribution policy, we will only release Tweet IDs/URLs and their annotations in our dataset. No other personal or identifiable information will be shared or published. All data were anonymized and de-identified prior to amal- ysis to protect the privacy of the individuals who posted the tweets. The primary aim of this research is contribute to the understanding of COVID-19 vaccine misinformation, and to inform strategies for public health communication.
Alexandre De Figueiredo, Clarissa Simas, Emilie Karafillakis, Pauline Paterson, and Heidi Larson. 2020. Mapping global trends in vaccine confidence and investigating barriers vaccine uptake: a large- scale retrospective temporal modelling study. The Lancer, 396( 10255))898-908
Matthew R Derna,, Francesco Pierri, Bao Tran Truong. John Bollenbacher, David Axelrod, Niklas Loynes, Christopher Torres- Lugo, Kai-Cheng Yang. Filippo Menczer, and John Bryden. 2021. Covaxxy: A col- lection of english-language twitter posts about covid- 19 vaccines. In Proceedings of the International AAAI Conference Web and Social Media, vol- ume 15. pages 992-999.
Acknowledgements
We thank anonymous reviewers for their helpful feedback on this work. This research was supported by Boston University's Center for Emerging Infec- tious Diseases Policy& & Research (CEID), NSF CCF-2200052, and NSF grant 1838193. Any opin- ions, findings, conclusions, or recommendations expressed here those of the authors and do not necessarily reflect the view of the sponsor.
Eve Dubé, Caroline Laberge, Maryse Guay, Paul Bra- madat, Réal Roy, and Julie A Bettinger. 2013. Vac- cine hesitancy: an overview. Human vaccines & immunotherapeutics, 1833
Jieyu D Featherstone, George Barett,, Jeanette B Ruiz, Yurong Zhuang, and Benjamin J Millam. 2020. Exploring childhood anti-vaccine and pro-vaccine communities on twitter-a a perspective from influ- ential users. Online Social Networks and Media, 20:100105.
References
Mahmoud Arafat, Syed A Morshed, Md Sultan Ali, and
Giorgia Guglielmi. 2022. Pandemic drives huge drop in child vaceinations.
Md Asif Raihan. 2021. Hydrating large-scale coro- navirus pandemic tweets: A review of software for transportation research. In International Conference on Transportation and Development 2021, pages 353-.
Lei Guo, Kate Mays, Yiyan Zhang, Derry Wijaya, and Margrit Betke. 2021. What makes gun violence a (less) prominent issue? a computational analysis of
363.

---

compelling arguments and selective agenda setting. Mass communication and society, 24(5):651-677
Siyi Liu, Lei Guo, Kate Mays, Margrit Betke, and Derry Tanti Wijaya. 2019. Detecting frames in news headlines and its application to analyzing news fram- ing trends surrounding us gun violence. In Proceed- ings of the 23rd conference on computational natural language learning (CoNLL.), pages 504-514.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't I stop pretraining: Adapt language models domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Brendan W Mason and Peter D Donnelly. 2000. Im- pact of local newspaper campaign on the uptake of the measles mumps and rubella vaceine. Journal of Epidemiology & Community Health, 54(6)473-477.
Kilem Li Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psy- chology, 61(1):29 48.
Hamdy Mubarak, Sabit Hassan, Shammur Absar Chowdhury, and Firoj Alam. 2022. Arcovidvac: Analyzing arabic tweets about covid-19 vaccination. arXiv preprint arXiv:2201.0644
Emily Hagg, V Susan Dahinten, and Leanne M Currie. 2018. The emerging use of social media for health- related purposes in low and middle-income countries: A scoping review. International journal of medical informatics, 115:92-103 105.
Sadiq Muhammed and Saji K Mathew. 2022. The disaster of misinformation: review of research in social media. International journal of data science and analytics, 13(4):27
Goran Muric, Yusong Wu, and Emilio Ferrara. 2021. Covid-19 vaccine hesitancy on social media: building public twitter data set of antivaccine content, vac- cine misinformation, and conspiracies. JMIR public health and surveillance, 7(1 (())3330
Kadhim Hayawi, Sakib Shahriar, Mohamed Adel Ser- hani, Ikbal Taleb, and Sujith Samuel Mathew. 2022. Anti-vax: a novel twitter dataset for covid-19 vaccine misinformation detection. Public health, 203:23-30 30.
Traci Hong, Zilu Tang, Manyuan Lu, Yunwen Wang. Jiaxi Wu, and Derry Wijaya. 2023. Effects of# coro- navirus content moderation on misinformation and anti-asian hate on instagram. New Media & Society, page 1461 04442231187229
Dr Greg Nyilasy. Fake news in the age of covid-19.
Greg Nyilasy. 2019. Fake news: When the dark side of persuasion takes over. International Journal of Advertising, 38(2)333--34.
Linmei Hu, Tianchi Yang. Lubao Zhang, Wanjun Zhong, Duyu Tang, Chuan Shi, Nan Duan, and Ming Zhou. 2021. Compare to the knowledge: Graph neural fake news detection with external knowledge. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11 International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pages 763-763, Online. Association for Computational Linguistics.
OWID. Data covid- 19 by our world in data.
Oxford Languages. 2021. Oxford Word of the Year 2021 Oxford Languages.
Francesco Pierri, Brea I. Perry, Matthew R De Kai-Cheng Yang, Alessandro Flammini, Filippo Menczer, and John Bryden. 2022. Online misinfor- mation is linked to early covid-19 vaccination hesi- tancy and refusal. Scientific reports, 12(1):5966.
Md Saiful Islam, Abu-Hena Mostofa Kamal, Alamgir Kabir, Dorothy L. Southern, Sazzad Hossain Khan, SM Murshid Hasan, Tonmoy Sarkar, Shayla Sharmin, Shiuli Das, Tuhin Roy, al. 2021. Covid-19 vaccine rumors and conspiracy theories: The need for cogni- tive inoculation against misinformation to improve vaccine adherence. PloS one, 16(5):e025166) 1605.
Jacob Poushter, Caldwell Bishop, and Hanyu Chwe. 2018. Social media use continues to rise devel- oping countries but plateaus across developed ones. Pew research center, 22:2-19.
Neha Puri. Eric A Coomes. Hourmazd Haghbayan, and Keith Gunaratne. 2020. Social media and vaccine hesitancy: new updates for the era of covid-19 19 and globalized infectious discases. Human vaccines & immunotherapeutics, 16(1 (6(1)::55622592
Ayodele Samuel Jegede. 2007. What led to the nigerian boycott of the polio vaccination campaign? PLoS medicine, 4(3)
Klaus Krippendorff 2018, Content analysis: An intro- duction to its methodology. Sage publications.
Gaurav Sahu, Pau Rodriguez, Issam Laradji, Parmida Atighehchian, David Vazquez, and Dzmitry Bah- danau. 2022. Data augmentation for intent classi- fication with off- shelfff- large language models. In Proceedings of the 4th Workshop on NLP for Conver- sational AI, pages 47-57, Dublin, Ireland. Associa- tion for Computational Linguistics.
Richard Landis and Gary G Koch. 1977. The mea- surement of observer agreement for categorical data. biometrics, pages 159-174.
Sun Kyong Lee, Juhyung Sun, Seulki Jang, and Shane Connelly. 2022. Misinformation of covid-19 vac- cines and vaccine hesitancy. Scientific Reports, 12(1):
Shannon Schumacher and Nicholas Kent. 2020. 8 charts on internet use around the world as countries grapple with covid-19.

---

Qiang Sheng, Juan Cao, Xueyao Zhang, Rundong Li. Danding Wang. and Yongchun Zhu. 2022. Zoom out and observe: News environment perception for fake detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pages 4543-445 4556, Dublin, Ireland. Association for Computational Lin- guistics.
Alan Smith, Joanne Yarwood, and David M Salisbury. 2007. Tracking mothers' attitudes to mmr immunisa- tion 1996-2006. Vaccine, 299202:3996-400.
Amir Sodikin. 2022. When mass media creates infor- mation disaster, an analysis of twitter conversations regarding rumors surrounding the effects of the covid- 19 vaccine. Master's thesis, Paramadina University.
Maxwell Weinzierl and Sanda Harabagiu. 2022a. Iden- tifying the adoption or rejection of misinformation targeting covid-19 19 vaccines in twitter discourse. In Proceedings of the ACM Web Conference 2022, pages 3196-3205.
Maxwell Weinzierl and Sanda Harabagiu. 2022b. Vac- cinelies: A natural language resource for learning recognize misinformation about the covid- 19 and hpv vaccines. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 6967-6975.
Maxwell A Weinzierl and Sanda M Harabagiu. 2021. Automatic detection of 1ovid-19 vaccine misinforma- tion with graph link prediction. Journal of biomedi- cal informatics, 103955.
WHO. 2006. Polio eradication the african region: Progress report.
World Bank. 2022. Middle income countries overview: Development news, research, data world bank.

---

A Appendix
A.2 Input Prompts for GPT-3 zero-shot and few-shot classification
A.1 Vaccine-Related Query Terms
Figure and Figure 7 show the examples of the input prompts for detecting misinformation and themes of misinformation, respectively. For brevity, the few-shot examples and the corresponding labels replaced with placeholders.
We used the following search query: "vaccine OR vaccines OR vaccination OR vaccinate OR vac- cinated OR jab OR jabs OR vax OR vaxxed OR anti-vax OR anti-vaxxer OR antivax OR antivaxxer OR inoculate OR vaxes OR vaxing OR vaxxes OR vaxxing OR vaxxic OR (shot NOT (party OR par- ties OR drinking OR drunk OR booze OR vodka OR tequila OR whisky OR alcohol OR wine OR beer OR hangover OR wasted OR hungover OR COc (shots NOT (party OR parties OR drinking OR drunk OR booze OR vodka OR tequila OR whisky OR alcohol OR wine OR beer OR hangover OR wasted OR hungover OR cocktail))" We also included country-specific terms including "vacina," "injecao." " and "Zé Gotinha" for Brazil; and "vaksin" " and "suntik" for Indonesia per the Ox- ford Languages Word for Vax (Oxford Languages, 2021).
Input Prompt
Classification
Does the tweet contain misinformation about COVID- 19 vacoines? Answer is ana of you or no.
Why print 3D guns the internet when you can just not vacoinato your kids and kill al?? answer:
tweet :
Prompt for Feew-shot Clessification
Dove: the boaret contiin misinformttion about COVID-19 vaccinee Answerr one no.
tweet (exammle tweet answorr: : (yssmo)
treet (jezample tweet 21 answor: (ycsino)
treet bweet answer: (yossno)
tweet: : (target twwed)) answer
Figure 6: Input prompts for misinformation classifica- tion
Input Prompt for Zero-shot Clasallicttion
following tweet
misinformation COVID-
Infertility
their babies before the govemment aoknowlodges that voccines should
tweet How pregrant
NEVER be given during pregrancy lal la (p Hipp smrower:
Input Prompt for Few Classification
The followingg tweets contains misinformation about COVID-I 19 vaccines, Answer whether the tweet about Infertiltty Anawer one of yes or no.
twwel (exzample bweet 1J
answor: (yosino)
(yosho)
teweet (eosmple bmeer! 10) answor: (yosho)
twset: (larget tweed))
Figure 7: Input prompts for classifying themes of misin- formation


Reviews:
[{"review_id": "SCxCQ6uNxx", "reviewer": "Reviewer_cr9A", "summary": "", "questions": "Did you consider the probable confounders in Sec 5.2?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive recommendation and feedback!\n\n***Comment 1- The causal effect of (% change in) COVID-19 cases on misinformation confounders***\n\nIt is recognized that the application of linear regression may not elucidate causal relationships unless the requisite assumptions underpinning the model are met. Consequently, we will rephrase it in the following way: \"A positive correlation has been identified between ..., which may potentially be construed as indicative of a causal link.\"\n\nA recommended course of action, as suggested by the reviewer, entails the inclusion of control variables to effectively address the issue of confounding. However, identifying suitable control variables has proven challenging, with the exception of policy stringency, as any variable that does not vary over time is absorbed into the model's constant term. Furthermore, even in instances where a variable exhibits temporal variance, its dynamic relationship presents formidable complexities. For instance, the impact of policy stringency on Covid cases might not manifest instantaneously but could materialize over the course of several days or weeks. Consequently, the decision was made to adopt a parsimonious model, while still taking into account potential heterogeneity and autocorrelation using Newey-West standard errors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This research is really meaningful. Previous research mainly focus on the vaccine misinformation in specific languages in specific countries or cultures. This research first propose to focus on middle income countries. \n\n2. The experiments are extensive (although with some issues that I will point out in the reasons to reject). The decriptions are detailed and provide sufficient information to understand\n\n3. The annotations are detailed and will be very helpful for future works on this direction.", "reasons_to_reject": "My main concern is in the experiment:\n\nThe causal effect of COVID-19 cases on misinformation. The authors claims that their analysis shows that increased COVID-19 cases are  positively affecting the vaccine misinformation ratio. This is problematic. My understanding is that their linear regression model does reveals a correlation. However, correlation may not be the causality. In causality area, the experiment that the authors did is treatment effect estimation. It is usually done after we have exclude the hypothesis of correlation, i.e. there might be some confounders that jointly influence the x and y. To make the claim in the paper hold, we need to first exclude the hypothesis of correlation. For example, we can do an IPTW to rebalance the data to make them ``seem\" to be drawn from random controlled trial to reduce the influence of counfounders. Or the authors can rephrase their expression as \"We found a positive correlation between ......\". I provided two papers in the missing reference that may be helpful. [1] did some analysis to vaccine hesistancy and misinformation of different states in the U.S., but they did not claims misinformation ``affect\" accine hesistancy because polictic trend could be a counfounder. Their following work [2] studies on some causal effect on social media. \n", "questions_to_authors": "Did you consider the probable confounders in Sec 5.2?", "paper_topic_and_main_contributions": "This paper presents a multilingual COVID-19 vaccine misinformation in middle income countries. It mainly studies the public Twitter posts from three contries. The authors provide detailed annotations to a subset of the posts, such as domain-expert misinformation labelling and identified themes. They applied this annotated subset to train a misinformation detection model and then analyze the rest unlabelled data with the model. Their analysis reveals some findings about misinformation of COVID-19 vaccines and other factors.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "The authors claims that their work is approved by their IRB."}, "extra_text": {"Missing_References": "[1] Karishma Sharma, Yizhou Zhang, and Yan Liu. Covid-19 vaccine misinformation campaigns and social media narratives. In Proceedings of the International AAAI Conference on Web and Social Media, pages 920–931.\n\n[2] Yizhou Zhang, Defu Cao, and Yan Liu. Counterfactual neural temporal point process for estimating causal influence of misinformation on social media. In Advances in Neural Information Processing Systems 2022."}}, {"review_id": "L7fPLAUdzz", "reviewer": "Reviewer_XBsM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback!\n\n***Comment 1 - “Inter annotator agreement is oddly calculated and seems quite low”***\n\nBoth Cohen’s Kappa and Gwet’s AC1 could be employed to evaluate the degree of agreement among raters. However, Gwet’s AC1 offers advantages over Cohen’s Kappa when dealing with binary and imbalanced data. Given that the majority of our coding scheme consists of questions utilizing binary response options (yes/no) and class imbalance is observed, we adopted Gwet’s AC1 over Cohen’s Kappa to ensure robust calculation of inter-annotator agreement.\n\n***Comment 2 and 3 - The goal and narrative of the paper***\n\nThe purpose of this study is not only to introduce a new dataset, but also to establish the initial baselines for the research that utilizes this dataset and to demonstrate practical applications of this dataset. To this end, several baseline methods as well as methods employing domain-specific pre-training and text augmentation were evaluated on the dataset. The best-performing model was then applied to a large set of unlabeled tweets for extensive analysis.\n\t\nWhile we have outlined the purpose and contributions of this study in the abstract and in the final paragraph of the introduction, we will also make this clearer in the revision so as to avoid confusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting multi-country and language dataset\n- Methodical iterative use of tweets", "reasons_to_reject": "- Inter annotator agreement is oddly calculated and seems quite low\n- Extensive evaluation presented with data augmentation seems to be beyond the point of the individual dataset\n- The paper is more than just a dataset release paper, making the narrative confusing", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this work the authors present a multi-language dataset for vaccine misinformation in middle income countries. The dataset is broken down into several smaller pieces that help with the global categorization of tweets to a more fine-grained level of different topics of vaccine misinformation. The evaluation is well designed and very methodical.  ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xW22tKx3hm", "reviewer": "Reviewer_Knzu", "summary": "", "questions": "A. In the modeling section, are there drawbacks to using a 4-step model pipeline, since errors could propagate? Did you consider merging some of the steps, eg, 1+2 -> directly predict if a tweet is about the COVID vaccine, or 3+4 -> directly predict if a COVID vaccine tweet is about a certain misinformation theme?\nB. What is the motivation for predicting the impact of COVID cases on vaccine misinformation rates? Intuitively, I would think the relationship should be the other way around, ie, misinformation -> lower vaccine rates -> higher COVID cases? \nC. I was a bit confused by lines 227-229: \"To balance the data for misinformation, we identify more tweets that contain misinformation, resulting in the final annotated dataset of N=5,952.\" – what does this mean? How were more tweets identified?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive recommendation and detailed review!\n\n***Comment 1 - Relatively small dataset***\n\nWe acknowledge that the dataset we have introduced might be relatively small. However, our multi-country dataset contains rich annotations, including relevance to vaccines and COVID-19 vaccines, presence of misinformation, and themes of misinformation, which requires greater annotation effort for each individual tweet. We believe these attributes of the dataset bestow it with significant value for the research of COVID-19 vaccine misinformation in middle-income countries.\n\nAdditionally, you pointed out the scarcity of certain misinformation themes (e.g., infertility) in our dataset. This is mainly because our annotated dataset follows the theme distribution in the original population of tweets (i.e., tweets are randomly sampled and annotated from the total tweets we have scraped). The scarcity of observations in specific categories of misinformation themes is reflective of the data. To address this inherent limitation of the dataset (imbalance, some small classes), we introduce a text augmentation technique using a large language model, demonstrating strong performance improvements compared to the baseline models.\n\n***Comment 2 & 3 - Confidence intervals & Evaluation per country***\n\nWe appreciate your suggestion. We first provide the confidence intervals and evaluation per country only for the best models in our paper due to the space limit. We will include the full version of the table in the revision. \n\n|  | Confidence Interval (95%) | Brazil | Indonesia | Nigeria |\n|---|---|---|---|---|\n| Q1   | 97.85 $\\pm$ 0.44 | 97.26 | 96.79 | 98.31 |\n| Q2   | 92.89 $\\pm$ 0.94 | 94.38 | 91.24 | 92.58 |\n| Q3   | 81.25 $\\pm$ 1.88 | 72.25 | 76.99 | 81.05 |\n| Q4a | 80.06 $\\pm$ 2.85 | 79.67 | 68.49 | 76.27 |\n| Q4b | 80.16 $\\pm$ 1.19 | 77.54 | 75.44 | 82.19 |\n| Q4c | 87.66 $\\pm$ 7.73 | 96.34 | 77.77 | 89.87 |\n| Q4d | 79.15 $\\pm$ 3.15 | 80.80 | 74.95 | 81.84 |\n| Q4e | 85.04 $\\pm$ 3.05 | 75.84 | 79.08 | 79.98 |\n| Q4f  | 83.84 $\\pm$ 2.47 | 69.87 | 77.85 | 83.93 |\n| Q4g | 81.83 $\\pm$ 3.95 | 78.23 | 76.54 | 85.66 |\n| Q4h | 75.36 $\\pm$ 15.15 | 78.79 | 76.34 | 73.49 |\n\n***Comment 4 - Small and inconsistent effect sizes across countries***\n\nConcerning effect sizes, variations are observed across different countries. In the context of Indonesia, the (dynamic) effect size of 0.0029 is construed as signifying that a 1% escalation in the tally of new Covid cases brings about a 0.0029 increase in the misinformation rate (which ranges from 0 to 1) over an initial 6-day interval – an effect size deemed substantial. Illustrated in Figure 2 is a period at the onset of 2022 wherein the percentage surge in new case counts surpasses 200, translating to a dynamic effect size exceeding 0.58. In the case of Brazil, where the estimated effect size is very small, we can still learn that the Covid cases do not affect misinformation.\n\n***Comment 5 - Drawbacks to using a 4-step pipeline since the errors could propagate***\n\nWe thought this is an interesting idea, but followed the 4-step pipeline mainly due to the following reasons :\n\t\nFirstly, one of the main objectives of this paper is to introduce a new dataset and evaluate various models on this dataset. For this purpose, we developed and evaluated models following the 4-step coding scheme used in dataset construction. Additionally, while grouping steps as you suggested might reduce the propagation of errors, the problem of class imbalance could be even compounded. More importantly, following the 4-step pipeline enables a more comprehensive and detailed analysis on the ~19 million tweets as we could obtain predictions for all four steps.\n\n***Comment 6 - Motivation for predicting the impact of (% change in) COVID cases on vaccine misinformation rates***\n\nOur approach of examining the influence of Covid cases on misinformation is based on the observed concurrent fluctuations in the (% change in) Covid cases and the rate of misinformation. The identical dynamic model which was employed to investigate the impact of Covid cases on misinformation was employed once more to investigate the relationship the other way around, yielding inconclusive results. This outcome was omitted due to space limitations, but we can add this information in the revision.  \n\n***Comment 7 - Clarification of how additional tweets were identified***\n\nAs described in Section 3, the initial annotated dataset consisted of a total of 5,500 instances. However, considering the relatively low proportion of tweets containing misinformation, 452 tweets with misinformation were additionally incorporated into the dataset, resulting in a final dataset of 5,952 tweets. To include only tweets with misinformation, two coders identical to the previous round of annotation assessed the presence of misinformation in randomly sampled tweets from the collected dataset. Only those tweets containing misinformation were added to the annotated dataset (we balance the real/misinformation classes, not the themes). This approach is in line with previous works to balance real/fake classes in misinformation dataset (Sheng et al., 2022). \n\nReference: Qiang Sheng, Juan Cao, Xueyao Zhang, Rundong Li, Danding Wang, and Yongchun Zhu. 2022. Zoom Out and Observe: News Environment Perception for Fake News Detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4543–4556, Dublin, Ireland. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This is a thorough study, including a new dataset, predictive models, and analyses.\n- The new dataset is richly labeled and interannotator agreement is high.\n- The models achieve strong performance, with some nice techniques incorporated to outperform baselines.\n- As the authors note, much of research on COVID-19 vaccine hesitancy focuses on the US, and it is important to study other countries.", "reasons_to_reject": "- The new dataset is relatively small. While it contains 5,500 labeled tweets, only 3,011 are related to the COVID vaccine and 1,119 contain COVID vaccine misinformation. Furthermore, within that set, some themes are barely represented, eg, only 26 tweets for Q4C (infertility).\n- Evaluation results in Section 4.5 are missing confidence intervals, which are especially important with small data. \n- It would also be useful to provide evaluation per country, since the numbers here seem to be aggregated over all three countries. It is important to understand whether the model performs equally well across countries, especially if the model is then applied to analyze trends across countries.\n- Unclear takeaways from the Analysis section. The effect sizes in 5.2 are very small and inconsistent across countries, and I'm not sure why we should expect COVID-19 rates to predict misinformation, instead of the other way around.", "questions_to_authors": "A. In the modeling section, are there drawbacks to using a 4-step model pipeline, since errors could propagate? Did you consider merging some of the steps, eg, 1+2 -> directly predict if a tweet is about the COVID vaccine, or 3+4 -> directly predict if a COVID vaccine tweet is about a certain misinformation theme?\nB. What is the motivation for predicting the impact of COVID cases on vaccine misinformation rates? Intuitively, I would think the relationship should be the other way around, ie, misinformation -> lower vaccine rates -> higher COVID cases? \nC. I was a bit confused by lines 227-229: \"To balance the data for misinformation, we identify more tweets that contain misinformation, resulting in the final annotated dataset of N=5,952.\" – what does this mean? How were more tweets identified?", "paper_topic_and_main_contributions": "This paper studies COVID-19 vaccine misinformation in three middle-income countries: Brazil, Indonesia, and Nigeria. The contributions are that the authors 1) curated a new dataset of tweets from these countries, labeled for COVID vaccine misinformation and its associated themes, 2) trained models to predict misinformation and its themes, 3) applied their trained models to all their collected tweets (~19 million) to analyze misinformation trends. \n\nDataset: first, they scrape all vaccine-related tweets from these countries (Jan 2020-Dec 2022). Then, they randomly sample a subset (N=5,500) to label. Each tweet is labeled for whether it is vaccine-related, whether it mentions non-COVID vaccine, whether it contains vaccine misinformation, and the theme of misinformation. Inter-coder reliability (ICR) is measured on 550 tweets that both annotators label; ICR is reasonably high. \n\nModels: following their labeling scheme, they train a series of models to 1) predict if the tweet is vaccine-related, 2) predict if it is about the COVID vaccine, 3) predict if it contains misinformation, 4) predict the theme of misinformation (each theme is its own binary classifier). They employ two techniques to improve model performance: pre-training on domain-specific tweets and data augmentation via GPT-3. Their final model outperforms the base XLM-RoBERTA model.\n\nAnalyses: they apply their trained models to all their collected tweets (~19 million). First, they use a regression model to measure the impact of COVID-19 case rate on misinformation rate. They find small effects on some lags of COVID-19 case rate. Second, they find association between rates of misinformation of each country. Finally, they do a qualitative analysis and observe differences in language across countries, within the same theme (eg, political and economic motives).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well-written."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes the creation of a novel COVID-19 multilingual dataset that covers multiple middle-income countries. The dataset creation is sound and with high potential for impact given the lack of datasets of this nature.

The paper is primarily a dataset paper with benchmark experiments using off-the-shelf methods. The novel contributions on the experimental side are more limited, but the insights derived from these experiments as well as the benchmark performance scores are useful for future work, in addition to the key contribution of a dataset.

I would suggest the authors to take into account reviewer comments in a further revision by toning down the correlation claims and by adding a discussion on per-country performance scores.
Decision: Accept-Main