SlowGuess/NAACL_2022_OCR huggingface dataset:
poland—multifactorial impact trajectories. Vaccines, 9(8):876.
Shivashankar Subramanian, Trevor Cohn, and Timothy Baldwin. 2018. Hierarchical structured model for fine-to-coarse manifesto text analysis. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1964-1974, New Orleans, Louisiana. Association for Computational Linguistics.
Fabio Tagliabue, Luca Galassi, and Pierpaolo Mariani. 2020. The "pandemic" of disinformation in COVID-19. SN comprehensive clinical medicine, 2(9):1287-1289.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579-2605.
Dominik Wawrzuta, Mariusz Jaworski, Joanna Gotlib, and Mariusz Panczyk. 2021. What arguments against COVID-19 vaccines run on facebook in poland: Content analysis of comments. Vaccines, 9(5):481.
Maxwell Weinzierl, Suellen Hopfer, and Sanda M Harabagiu. 2021. Misinformation adoption or rejection in the era of pandemic-19. In Proceedings of the International AAAI Conference on Web and Social Media, volume 15, pages 787-795.
Maxwell A Weinzierl and Sanda M Harabagiu. 2021. Automatic detection of Covid-19 vaccine misinformation with graph link prediction. Journal of biomedical informatics, 124:103955.
Manuel Widmoser, Maria Leonor Pacheco, Jean Honorio, and Dan Goldwasser. 2021. Randomized deep structured prediction for discourse-level processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1174-1184, Online. Association for Computational Linguistics.
Mohammadzaman Zamani, H Andrew Schwartz, Johannes Eichstaedt, Sharath Chandra Guntuku, Adithya Virinchipuram Ganesan, Sean Clouston, and Salvatore Giorgi. 2020a. Understanding weekly Covid-19 concerns through dynamic content-specific lda topic modeling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2020, page 193. NIH Public Access.
Mohammadzaman Zamani, H. Andrew Schwartz, Johannes Eichstaedt, Sharath Chandra Guntuku, Adithya Virinchipuram Ganesan, Sean Clouston, and Salvatore Giorgi. 2020b. Understanding weekly COVID-19 concerns through dynamic content-specific LDA topic modeling. In Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science, pages 193–198, Online. Association for Computational Linguistics.
A.1 Reasons and Phrases
Tables 10 and 11 show the full list of phrases for anti-vax and pro-vax reasons. The interactive task interface is presented in Figures 5 and 6. Bar plots for reason assignments before and after interaction are shown in Figure 7.
(a) Theme: VaccineDanger
(b) Theme: VaccineSafe
Figure 8: Wordclouds for reasons before interaction.
Figure 9: Wordclouds for reasons after interaction.
(a) Theme: VaccineDanger
(b) Theme: VaccineSafe
A.2 Data Collection
To create the list of keywords used to collect tweets about the Covid-19 vaccine, we read multiple articles about Covid mentioning vaccination status, vaccine hesitancy, misinformation, vaccine constraints, health issues, religious sentiment and other vaccine-related debates, and made a list of repeating statements. Then, we consulted three researchers, two in Computational Social Science and one in Psychology, and constructed a list of relevant keywords that are indicative of morally charged discussions. The full list of keywords can be observed in Table 12.
covid vaccine, covid vaccination, covid vaccine tyranny,  
covid vaccine oppression, covid vaccine mandate, covid vaccine conspiracy,  
covid vaccine anti-vax, covid vaccine religion, covid vaccine satan,  
covid vaccine god, covid vaccine jesus, covid vaccine islam,  
covid vaccine muslim, covid vaccine christianity, covid vaccine christian,  
covid vaccine hindu, covid vaccine jews, covid vaccine catholic,  
covid vaccine buddhism, covid vaccine religious, covid vaccine biden failure,  
covid vaccine passport, covid vaccine loyalty, covid vaccine cheating,  
covid vaccine freedom, covid vaccine betrayal, covid vaccine liberty,  
covid vaccine black people, covid vaccine propaganda, covid vaccine hesitancy,  
covid vaccine hesitant, covid vaccine microchip, covid vaccine bill,  
covid vaccine pregnancy, covid vaccine pregnant, covid vaccine approval,  
covid vaccine biden, covid vaccine fda, covid vaccine cdc,  
covid vaccine fauci, Covid-19 china, vaccine passport,  
vaccination mandate, covid vaccine death, covid vaccine military,  
experimental covid vaccine, covid vaccine authorization,  
vaccine oppression, vaccine satan, covid vaccine bill gates,  
covid vaccine side effect, covid vaccine adverse events
Table 12: List of the keywords for data collection.
A.3 Data Annotation Task
The steps for annotating tweets using our graphical interface are (See Figure 10).
1. Select the moral foundation of the text using the checkbox  $\square$ . You can see the definition of each moral foundation by hovering your mouse over them. If the tweet does not make any moral judgement, check  $\square$  "none". In this case, you don't have to highlight actor-target polarity.
2. After selecting any moral foundation other than "none", text highlighting for actor-target role with polarity will be visible below. If you select a moral foundation other than "none", you can highlight the actor-target polarity.
3. Choose the color-coded label Positive Actor/Positive Target/Negative Actor/Negative Target to highlight the text with the color of the selected label. You can see the definition of actor-target-polarity role by hovering your mouse over them.
4. Highlight words, phrases, or sections of the text for the actor-target role with polarity of the corresponding moral foundation.
5. If you made any mistakes in highlighting, select the "Unhighlight" button to unhighlight the previously highlighted text.
6. Finally, click the "Submit" button to submit the task.
We provided eight examples (Figure 11) covering six moral principles and non-moral cases to make our annotation task more understandable. Annotators could see the explanations for choosing a moral foundation and an. actor-target polarity by clicking the "See Explanation" button.
Annotators had to complete two practice examples before starting the real task. If they made any mistake, our practice session provided them the correct result with an explanation. Figure 12 shows the interface for one of the two practice examples.
A.4 Out-of-Domain Datasets
For moral foundation prediction, we use the dataset proposed by Johnson and Goldwasser (2018), consisting of 2K tweets by US congress members annotated for the five core moral foundations. We also use the Moral Foundation Twitter Corpus
Themes	Overarching Patterns
GovDistrust	Add phrases with strong word for distrust
“Good at being bad”
Explicit negations
GovTrust	Hedging phrases (sort-of trust)
VaxDanger	Closer connection between vaccine words and danger words (related to sickness, bad effects)
Explit negations
Rhetorical questions
Refusing the vaccine for medical reasons
VaxSafe	Explicit mentions of safety
Explicit negations
CovidFake	Stronger relevant negative words (fake, scam, hoax)
Explicit negations
CovidReal	Trust the science
References to Covid hospitalization on the rise, explicit mentions of hospitals
Explicit negations
VaxOppression	Legal language
Explicit mentions of discrimination and oppression
Sarcasm
VaxNotOppression	Justifying mandates
Freedom to be protected
Criticizing others using “you/people” language, focus freedom on me/my/I
BigPharmaAnti	Stronger words against pharmaceutical companies (corrupt, evil)
Not accountable / irresponsible past behavior
Mentions of negative side-effect of other products (cancer)
BigPharmaPro	Trust science/research and vaccine development process
Language about intent, the vaccine was created to do something good, explicit names of companies
NaturalImmunityPro	The vaccine is not enough
Explicit mentions to population immunity, herd immunity and antibodies
NaturalImmunityAnti	Emphasis on global look, collective entities, society
Natural immunity characterized as dangerous or not effective
Mentions of experts and trusting science
VaxAgainstReligion	I put it in god hands (god is deciding)
Treating pro-vax as another religion
VaxNotAgainstReligion	“Religious” in quotes
Bugus exemptions
“Where is your faith”
Call to action: get tested/get vaccinated/put a mask on (mentions of compassion)
No religion ask members to refuse vaccine
VaxDoesntWork	Reference to “magic vaccine”
“Never developed”, “doesn’t work”
Questions: why are deaths high? Why is corona not going away? Why are vaccinated people dying?
VaxWorks	“ask a doctor”, consult with an expert
Research on the vaccine is good/has been going on for a long time
Capture differences, e.g. “good trials” vs. rushed ones.
VaxNotTested	Language suggesting “rushed through trials” and “experimental vaccine”
VaxTested	trust the research and development process
Testing can be confused with COVID-test, use other language.
Table 9: Overarching argumentation patterns uncovered by coders during interaction
Table 10: AntiVax Themes and phrases for Covid-19 talking points. Themes that were added during interaction are shown in blue. Themes that were removed during interaction are shown in red. The original explanations/examples are presented in bold.
Table 11: ProVax reasons and phrases. Reasons that were added during interaction are shown in blue. Reasons that were removed during interaction are shown in red. The original explanatory phrases are presented in bold.
Figure 5: Interactive task interface.
(Hoover et al., 2020), consisting of 35k tweets annotated for moral foundations. The topics across these two datasets span political issues (e.g. gun control, immigration) and events (e.g. Hurricane Sandy, Baltimore protests). Given that neither of these two datasets contain examples for the liberty/oppression moral foundation, we curate a small lexicon by looking for synonyms and antonyms of the words liberty and oppression. Then, we use this lexicon to annotate the congressstweets dataset  $\dagger$ . We annotate a tweet as liberty/oppression if it contains at least four keywords, which results in around 2K tweets. The derived lexicon for liberty/oppression can be seen in Table 13
To learn to predict roles, we use the subset of Johnson and Goldwasser (2018) dataset annotated for roles by Roy et al. (2021), which contains roughly 3K tweet-entity-role triplets. For polarity, we combine the Roy et al. (2021) dataset with the MPQA 3.0 entity sentiment dataset (Deng and Wiebe, 2015), which contains about 1.6K entity-sentiment pairs.
liberty, independence, freedom, autonomy, sovereignty  
self-government, self-rule, self-determination, home-rule  
civil liberties, civil rights, human rights, autarky,  
free-rein, latitude, option, choice, volition, democracy,  
oppression, persecution, abuse, maltreatment, ill treatment,  
dictator, dictatorship, autocracy, tyranny, despotism,  
repression, suppression, subjugation, enslavement,  
exploitation, dependence, constraint, control, totalitarianism
Table 13: Liberty/Oppression Lexicon.
For stance, we annotate our dataset of 85K unlabeled pandemic tweets using a set of prominent antivirus and provax hashtags. For the antivirus case, we rely on the hashtags proposed by Muric et al. (2021). For the provax case, we manually annotate hashtags that have a clear provax message, and that are used in at least 50 tweets in our unlabeled dataset. The full set of hashtags used can be found in Tables 14 and 15.
$\dagger$  https://github.com/alexlitel/congressstweets
Figure 6: After querying the themes (i.e., CovidFake, CovidReal), interface shows the wordcloud.
(a) Without threshold before
(c) threshold  $\leq 0.3$  before
(b) Without threshold after
(d) threshold  $\leq 0.3$  after
Figure 7: Cluster assignment before and after refining arguments interactively.
What is the moral foundation of the following tweet?
neg act
neg tar
The government is forcing us to risk our health with these experimental COVID-19 vaccine.
care/harm fairness/cheating loyalty/betrayal authority/subversion sanctity/degradation liberty/oppression none
First pick the color.
Second highlight the text for actor-target role with polarity associated with corresponding moral foundation.
Positive Actor
Positive Target
Negative Actor
Negative Target
Unhighlight
After finishing the task, please click Submit button.
Figure 10: Annotation task interface.
Following we show simple examples (with explanation) for each category of moral foundation:
Example 1: People in poor countries are dying from COVID and need our help.
What's the moral Foundation of the above text? Answer: care/harm. because people from poor countries are getting harmed by COVID.
Highlight the text for actor-target role with polarity: People in poor countries are dying from COVID and need our help.
Negative Actor: COVID, Negative Target: People in poor countries. Explanation: because people from poor countries are target who are getting harmed (negative polarity) by COVID (actor).
Example 2: Black people have suffered disproportionately from the pandemic.
What's the moral Foundation of the above text? Answer: fairness/cheating. Because people from specific race (black) are suffering more from pandemic due to lack of facilities, which is not fair.
Highlight the text for actor-target role with polarity: Black people have suffered disproportionately from the pandemic
Negative Actor: pandemic, Negative Target: Black people. Explanation: because black people are suffering more from pandemic due to lack of facilities, which is not fair.
Example 3: Don't give evidence against your fellow workers.
What's the moral Foundation of the above text? Answer: loyalty/betrayal. See Explanation
Highlight the text for actor-target role with polarity: Don't give evidence against your fellow workers. See Actor Target Polarity
Example 4: I trust the doctors.
What's the moral Foundation of the above text? Answer: authority/subversion. See Explanation
Highlight the text for actor-target role with polarity: trust the actor. See Actor Target Polarity
Example 5: I only eat halal/kosher.
What's the moral Foundation of the above text? Answer: sanctity/degradation. See Explanation
Highlight the text for actor-target role with polarity: only eat halal/kosher. See Actor Target Polarity
Example 6: The government should not force me to wear a mask.
What's the moral Foundation of the above text? Answer: liberty/oppression. See Explanation
Highlight the text for actor-target role with polarity: The government should not force me to wear a mask. See Actor Target Polarity
Example 7: According to the CDC, the mortality rate in South America due to covid is higher than developed countries.
What's the moral Foundation of the above text? Answer: none. See Explanation
As there is no moral foundation, no need to highlight text for actor-target-polarity
Example 8: I got vaccinated today. Love pfizer vaccine. #nosideeffect #vaccinationdone.
As there is no moral foundation, no need to highlight text for actor-target-polarity.
Show Instruction
Hide Instruction
Figure 11: Examples provided to the annotators.
Figure 12: One of the two practice examples provided to the annotators before starting the real task.
Table 14: ProVax Hashtags
Table 15: AntiVax Hashtags