Venue: NAACL 2022

# A Holistic Framework for Analyzing the COVID-19 Vaccine Debate

Maria Leonor Pacheco\({}^{1}\)1Tunazzina Islam\({}^{1*}\)Monal Mahajan\({}^{1}\)

Andrey Shor\({}^{1}\) Ming Yin\({}^{1}\)Lyle Ungar\({}^{2}\)Dan Goldwasser\({}^{1}\)

\({}^{1}\)CS, Purdue University, USA, \({}^{2}\)CIS, University of Pennsylvania, USA

{pachecog,islam32,mmahaja,ashor,mingyin,dgoldwas}@purdue.edu

ungar@cis.upenn.edu

Equal contribution

Footnote 1: footnotemark:

###### Abstract

The Covid-19 pandemic has led to infodemic of low quality information leading to poor health decisions. Combating the outcomes of this infodemic is not only a question of identifying false claims, but also reasoning about the decisions individuals make. In this work we propose a holistic analysis framework connecting stance and reason analysis, and fine-grained entity level moral sentiment analysis. We study how to model the dependencies between the different level of analysis and incorporate human insights into the learning process. Experiments show that our framework provides reliable predictions even in the low-supervision settings.

## 1 Introduction

One of the unfortunate side-effects of the Covid-19 pandemic is a global infodemic flooding social media with low quality and polarizing information about the pandemic, influencing public perception on it Tagliabue et al. (2020). As studies have shown Montagni et al. (2021), these influences have clear real-world implications, in terms of public acceptance of treatment options, vaccination and prevention measures.

Most computational approaches tackling the Covid-19 infodemic view it a misinformation detection problem. In other words, they look at identifying false claims and analyzing reactions to them on social media Hossain et al. (2020); Alam et al. (2021); Weinzierl et al. (2021). This approach, while definitely a necessary component in fighting the infodemic, does not provide policy makers and health-professionals with much needed information, characterizing the reasons and attitudes that underlie the health and well-being choices individuals make.

Our goal in this paper is to suggest a holistic analysis framework, providing multiple interconnected views of the opinions expressed in text. We specifically focus on a timely topic, attitudes explaining vaccination hesitancy. Figure 1 describes an example of our framework. Our analysis identifies the _stance_ expressed in the post (anti-vaccination) and the _reason_ for it (distrust of government). Given the ideologically polarized climate of social media discussion on this topic, we also aim to characterize the moral attitudes expressed in the text (oppression), and how different entities mentioned in it are perceived ("Biden, Government" are oppressing, "citizens, us" are oppressed). When constructing this framework we tackled three key challenges.

**1. How should these analysis dimensions be operationalized?** While stance prediction is an established NLP task, constructing the space of possible arguments justifying stances on a given topic, and their identification in text, are still open challenges. _We take a human-in-the-loop approach to both problems_. We begin by defining a seed set of relevant arguments based on data-driven studies Weinzierl et al. (2021); Sowa et al. (2021), where each reason is defined by a single exemplar sentence. In a sequence of interactions, we use a pre-trained textual inference model to identify paraphrases in a large collection of Covid-19 vaccination tweets, and present a visualization of the results to humans. Humans then perform an error

Figure 1: Holistic Analysis Framework of Social Media Posts, Connecting entity-level Moral Perspectives, Stance and Arguments Justifying it.



analysis, and either add more sentences to help characterize the existing reason better, or add and characterize additional reasons, based on examples retrieved from the large corpus. We explain this process in detail in Section 2.

Our morality analysis is motivated by social science studies (Pagliaro et al., 2021; Diaz and Cova, 2021; Chan, 2021) that demonstrate the connection between moral foundation preferences (Haidt and Graham, 2007; Graham et al., 2009) and Covid-related health choices. For example, studies show that the endorsement of the _fairness_ and _care_ moral foundations is correlated with trust in science. To account for fine-grained patterns, we adapt the recently proposed morality-frame formalism (Roy et al., 2021) that identifies moral roles associated with moral foundation expressions in text. These roles correspond to actor/target roles (similar to agent/patient) and positive or negative polarity, which should be understood in the context of a specific moral foundation. In Figure 1_"Biden"_ is the negative actor in the context of Oppression, making him the oppressor. We explain this formalism in Section 3.

**2. How should the dependencies between these dimensions be captured and utilized?** The combination of stance, reason and moral attitudes provides a powerful source of information, allowing us to capture the moral attitudes expressed in the context of different stances and their reasons. These connections can also be used to help build expectations about likely attitudes in the context of each stance. As a motivating example, consider the reason "distrustingovernment", which can be associated with the "oppression" moral foundation _only_ when its actor is an entity related to government functions (rather than opposition from Covid-19 illness). We model these expectation as a probabilistic inference process (Pacheco and Goldwasser, 2021), by incorporating consistency constraints over the judgements made by our model, and predicting the most likely analysis jointly, consisting of all analysis dimensions. The full model, described using a declarative modeling language, is provided in Section 5.

**3. How can text analysis models be adapted to this highly dynamic domain, without extensive and costly manual annotation?** While our analysis in this paper focuses on a specific issue, vaccination hesitancy, we believe that our analysis framework should be easily adaptable to new issues. Relying on human insight to characterize and operationalize stance and reason identification is one aspect, that characterizes _issue-specific_ considerations. Moral Foundation Theory, by its definition, abstracts over specific debate topics, and offers a general account for human morality. However, from a practical perspective, models for predicting these highly abstract concepts are trained on data specific to a debate topic and might not generalize well. Instead of retraining the model from scratch, we hypothesize that given an initial model constructed using out-of-domain data, and a small amount of in-domain labeled data, we can obtain acceptable performance by modeling the interaction between reasons, stances and moral foundations. We study these settings, along with the fully supervised setting in Section 6.

The data, code and tools used in this paper are publicly available*.

Footnote *: [https://gitlab.com/mlpacheco/covid-moral-foundations](https://gitlab.com/mlpacheco/covid-moral-foundations)

## 2 Opinion Analysis

To analyze opinions about the Covid-19 vaccine, we model the vaccination stance expressed in each tweet (i.e. pro-vaccine, anti-vaccine, neutral) and the underlying reason behind such stance. For example, in Figure 1 the tweet expresses an anti-vaccine stance, and mentions their distrust of the Biden administration as the reason to take this stance.

There are three main challenges involved in this analysis: 1) predicting the stance, 2) constructing the space of possible reasons, and 3) mapping tweets to the relevant reasons. Stance prediction is an established NLP classification task (Glandt et al., 2021). However, uncovering latent themes from text automatically remains an open challenge, traditionally approached using noisy unsupervised techniques such as topic models (Zamani et al., 2020b), or by manually identifying and annotating them in text (Hasan and Ng, 2014).

Instead, we combine computational and qualitative techniques to uncover the most frequent reasons cited for pro and anti vaccination stances. We build on previous health informatics studies that characterized the arguments made against the Covid-19 vaccine in social media (Wawrzuta et al., 2021). In this work, researchers come up with a code-book of 12 main themes, frequently used as reasons to refuse or cast doubt on the vaccine. **We 

propose an interactive, humans-in-the-loop protocol** to learn representations for these 12 initial reasons, ground them in data, evaluate their quality, and refine them to better capture the discussion. To do this, we build a tool to explore repeating arguments and their reasons in the Covid-19 vaccine debate. The tool consists of an interactive Google Colab notebook equipped with a custom API to query current arguments, ground them in data, and visualize them. To initialize the system, we use the 12 reasons suggested by Wawrzuta et al. (2021), and represent them using the one-sentence explanation provided. Our main goal is to ground these reasons in a set of approximately 85,000 unlabeled tweets about the Covid-19 vaccine (details in Section 4). To map tweets to reasons, we use the similarity between their SBERT embeddings Reimers and Gurevych (2019). The interaction is centered around the operations outlined in Table 1. Intuitively, the first six operations allow humans to diagnose how reasons map to text, and the last three allows them to act on the result of this diagnosis, by adding and removing reasons, and modifying the phrases characterizing each reason.

We follow a simple protocol during interaction, where three human coders use the operations above to explore the initial reasons. The coders start by looking at the global picture: the reasons distribution, the 2D visualizations van der Maaten and Hinton (2008) and the silhouette score Rousseeuw (1987). Then, they query the reasons one by one, looking at the word cloud (characterizing the distribution of short phrases over all texts assigned to the reason) and the 10 closest tweets to each reason. Following these observations, there is a discussion phase in which the coders follow a thematic analysis approach Braun and Clarke (2012) to uncover the overarching themes that are not covered by the current set of reasons, as well as the argumentation patterns that the method fails to identify. Then, they are allowed to add and remove reasons, as well as explanatory phrases for them in natural language. Every time a reason or phrase is added or removed, all tweets are reassigned to their closest reasons. This process was done over two one-hour sessions. The coders were NLP and Computational Social Science researchers, two female and one male, between the ages of 25 and 40.

In the first session, the coders focused on adding new reasons and removing reasons that were not prevalent in the data. For example, they noticed that the initial set of reasons contained mostly antivaccine arguments, and added a positive reason for each negative reason (e.g. _government distrust_\(\Rightarrow\)_government trust_). In addition to this, they broke down the reason "Conspiracy Theory" into specific conspiracy theories, such as _Bill Gates' micro chip_, _the vaccine contains fetal tissue_, and _the vaccine makes you sterile_.They also removed infrequent reasons, such as _the swine flu vaccine_. The final set of reasons can be observed in Table 2.

In the second session, the coders focused on identifying the argumentative patterns that were not being captured by the original reason explanations, and came up with overarching patterns to create new examples to improve the representation of the reasons. For example, in the case of the _government distrust_ reason, the coders found that phrases with strong words were needed (e.g. _F the government_), examples that suggested that the government was "good at being bad" (e.g. _the government strong record of screwing things up_), and examples with explicit negations (e.g. _the government does not work logically_). Once patterns were identified, each coder contributed a set of 2 to 5 examples, which were introduced to the reason representation.

\begin{table}
\begin{tabular}{l|c} \hline \hline
**show\_reasons()** & \multicolumn{1}{c}{lists the current list of reasons (e.g. Government Distrust, Natural Immunity.)} \\
**show\_closest\_tweets(reason, K)** & \multicolumn{1}{c}{lists the k tweets} \\  & closet to a given reason, based on their embedding similarity. \\ \hline
**wordcloud(reason)** & Renders a word cloud to visualize the arguments associated to a given reason, based on bigram and trigran TF-ID features. \\ \hline
**show\_assignments(threshold)** & Renders a bar plot showing the assignment of tweets to reasons, based on embedding similarity. An optional threshold can be used to limit assignments. \\ \hline
**tsne(threshold)** & Renders a visualization of the reason clusters in a 2D map. Threshold is optional. \\ \hline
**silhouette\_score(threshold)** & Measures the overlapping degree between clusters. Threshold is optional. \\ \hline
**add\_reason(reason, phrase)** & Adds a new reason with aphrase that characterizes it in natural language \\
**remove\_reason(reason)** & Removes a given reason \\ \hline
**add\_phrase(reason, phrase)** & Adds an additional phrase \\ \hline \end{tabular}
\end{table}
Table 1: Interactive API Operations

\begin{table}
\begin{tabular}{p{369.9pt}|p{369.9pt}} \hline \hline
**Pro** & government distrust, vaccine dangerous, covid fake, vaccine compression, blarma bad, natural immunity effective, vaccine against religion, vaccine dos not work, vaccine not tested, bill gates’ mircep thip, vaccines tested on dogs, vaccine has fetal tissue, vaccine makes you sterile \\ \hline
**Anti Vax** & government trust, vaccine safe, covid real, vaccine not oppression, pharma good, natural immunity ineffective, vaccine not agist religion, vaccine works, vaccine tested \\ \hline \hline \end{tabular}
\end{table}
Table 2: Resulting Reasons 

In Appendix A.1, we include screenshots of the interactive notebook, and tables enumerating the full list derived patterns and phrases. To visualize the impact of interaction, we also show the overall distribution of reasons before and after interaction, and word clouds for a select set of reasons. The methodology and tool we developed are broadly applicable for diagnosing NLP models.

## 3 Morality Frame Analysis

Moral Foundations Theory (Haidt and Graham, 2007) suggests that there are at least six basic foundations that account for the similarities and recurrent themes in morality across cultures, each with a positive and negative polarity (See Table 3).

To analyze moral perspectives in tweets, we build on the definition of mortality frames proposed by Roy et al. (2021), where moral foundations are regarded as frame predicates, and associated with positive and negative entity roles.

While Roy et al. (2021) defined different roles types for each moral foundation (e.g. _entity causing harm_, _entity ensuring fairness_), we aggregate them into two general role types: **actor** and **target**, each with an associated polarity (positive, negative). An **actor** is a "do-er" whose actions or influence results in a positive or negative outcome for the **target** (the "do-ee"). For each moral foundation in a given tweet, we identify the "entity doing good/bad" (positive/negative actor) and "entity benefiting/suffering" (positive/negative target). For example, the statement "We are suffering from the pandemic" expresses **harm** as the moral foundation, where "pandemic" is a **negative actor**, and "we" is a **negative target** (i.e. the entity suffering from the actor's actions). There can be zero, one or multiple actors and targets in a given tweet. Entities can correspond to specific individuals or groups (e.g., I, democrats, people of a given demographic), organizations (e.g., political parties, CDC, FDA, companies), legislation or other political actions (e.g., demonstrations, petitions), disease or natural disasters (e.g., Covid, global warming), scientific or technological innovations (e.g., the vaccine, social media, the Internet), among others.

We break down the task of predicting morality frames into four classification tasks. For each tweet, our goal is to predict whether it is making moral judgement or not, and identify its prominent moral foundation. For each entity mentioned in the tweet, we predict whether it is a target or a role, and whether it has positive or negative polarity.

## 4 Data Collection and Annotation

There is no existing corpus of arguments about the Covid-19 vaccine annotated for morality frames and vaccination stance, so we collected and annotated our own. First, we searched for tweets between April and October of 2021 mentioning specific keywords, such as _covid Vaccine_ and _vaccine mandate_. The full list of keywords, as well as the procedure to obtain them, can be seen in Appendix A.2.

Then, we created an exclusive web application for annotating our task. Moral foundation and vaccination stance labels can be annotated directly. To identify entities, annotators were able to highlight the relevant text spans, and choose its role label (i.e. positive/negative actor or target). We annotated our dataset using three in-house annotators pursuing a Ph.D. in Computer Science. We awarded the annotators \(\$\,0.75\) per tweet and bonus (\(2*\$\,0.75=\$1.5\)) for completing two practice examples. Our work is IRB approved, and we follow their protocols.

To ensure quality work, we provided annotators with eight examples covering all six moral foundations and non-moral cases. Before starting the annotation task, the annotators had to read the instructions, go through the examples, and annotate two practice questions. The annotation interface, examples and practice questions can be seen in Appendix A.3.

Inter-annotator agreementWe calculated the agreement among annotators using Krippendorff's \(\alpha\)Krippendorff (2004), where \(\alpha=1\) suggests perfect agreement, and \(\alpha=0\) suggests chance-level

\begin{table} max width= 

\end{table}
Table 3: Moral Foundations Haidt and Graham (2007) 

agreement. We found \(\alpha=60.82\) for moral foundations, and \(\alpha=78.71\) for stance. For roles, we calculated the character by character agreement between annotations. For example, if one annotator marked "Dr Fauci" as a target in a tweet, and another marked "Fauci", it was considered to be an agreement on the characters "Fauci" but disagreement on "Dr". Doing this, we found \(\alpha=83.46\). When removing characters marked by all three annotators as "non-role", the agreement dropped to \(\alpha=67.15\).

Resulting annotated datasetWe used a majority vote to get moral foundation and vaccination stance labels, and obtained 750 annotated tweets. Similarly, we defined a text span to be an entity mention E, having a moral role R and polarity P, in a tweet T, if it was annotated as such by at least two annotators. Our resulting dataset contains 891 (T,E,R,P) tuples. Statistics can be seen in Table 4.

To evaluate the correlation between the different dimensions of analysis, we calculate the Pearson correlation matrices and present them in Figure 2. We can interpret reasons as distributions over moral foundations and stances (and vice-versa). This analysis provides a useful way to explain each of these dimensions. For example, we see that _care/harm_ is strongly correlated with reasons such as _covid is real_, _the vaccine works_, and _natural immunity is ineffective_. Other expected trends emerge, such as _purity/degradation_ being highly correlated with _vaccine against religion_. To evaluate the modeling advantage of our opinion analysis framework, we look at the correlation between stance, moral foundations and topics extracted in an unsupervised fashion using Latent Dirichlet Allocation (LDA) Blei et al. (2003). We find that the reasons extracted interactively have higher correlations with both vaccination stance and moral foundations. The LDA correlation matrices can be seen in Figure 3.

In Table 5 we show the top four reasons for _fairness/cheating_. We choose this moral foundation given that it is evenly split among stances and it is active for different reasons. We show the top two (E,R,P) tuples for each reason. We can appreciate that while this moral foundation is used by people on both sides, the reasons offered and entities used vary. On the anti-vax side, authority figures and vaccine trials are portrayed as negative actors, while women and children are portrayed as targets. On the pro-vax side, Covid and unvaccinated people are portrayed as negative actors, and the general public is portrayed as a target.

Unlabeled Covid-19 vaccine corpusIn addition to our annotated dataset, we collected a corpus of 85,000 tweets in English mentioning the covid vaccine, uniformly distributed between January and October of 2021. These tweets are unlabeled, and are used to ground arguments (Section 2) and to augment data for indirect supervision (Section 5).

\begin{table}
\begin{tabular}{l l|c c c c} \hline \hline
**Moral** & **Num.** & \multicolumn{4}{c}{**Vaccination Stance**} \\
**Foundation** & **Tw.** & Pro & Anti & Neut & No Agree \\ \hline Care/Harm & 96 & **77** & 17 & 2 & 0 \\ Fairness/Cheating & 75 & **33** & 28 & 14 & 0 \\ Loyalty/Bertayal & 33 & **26** & 2 & 5 & 0 \\ Authority/Subversion & 14 & 26 & **72** & 13 & 3 \\ Prurity/Degradation & 24 & 2 & **22** & 0 & 0 \\ Liberty/Oppression & 93 & 9 & **78** & 6 & 0 \\ Non-moral & 304 & **188** & 68 & 44 & 4 \\ No Agreement & 11 & 6 & 5 & 025 & 0 \\ \hline Total & 750 & 367 & 292 & 84 & 7 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Dataset Summary

Figure 2: Correlation Heatmaps





## 5 Joint Probabilistic Model

We propose a joint probabilistic model that reasons about the arguments made, their morality frames, stances, reasons, and the dependencies between them. We implement our model using DRaiL Pacheco and Goldwasser (2021), a declarative modeling framework for specifying deep relational models. Deep relational models combine the strengths of deep neural networks and statistical relational learning (SRL) to model a joint distribution over relational data. This hybrid modeling paradigm allow us to leverage expressive textual encoders, and to introduce contextualizing information and model different interdependent decisions. SRL methods have proven effective to model domains with limited supervision Johnson and Goldwasser (2018); Subramanian et al. (2018), and approaches that combine neural networks and SRL have shown consistent performance improvements Widmoser et al. (2021); Roy et al. (2021).

Following the conventions of statistical relational learning models, we use horn-clauses of the form \(p_{0}\wedge p_{1}\land...\wedge p_{n}\Rightarrow h\) to describe relational properties. Each logical rule defines a probabilistic scoring function over the relations expressed in its body and head.

Base rules/classifiersWe define three base rules to score whether a tweet t\({}_{i}\) has a moral judgment, what is its prominent moral foundation m, and what is its vaccination stance.

\[r_{0}:\texttt{Tweet}(\texttt{t}_{i}) \Rightarrow\texttt{IsMoral}(\texttt{t}_{i}) \tag{1}\] \[r_{1}:\texttt{Tweet}(\texttt{t}_{i}) \Rightarrow\texttt{HasMF}(\texttt{t}_{i},\texttt{m})\] \[r_{2}:\texttt{Tweet}(\texttt{t}_{i}) \Rightarrow\texttt{VarXStance}(\texttt{t}_{i},\texttt{s})\]

To score the moral role of an entity e\({}_{i}\) mentioned in tweet t\({}_{i}\), we write two rules. The first one scores whether the entity e\({}_{i}\) is an actor or a target, and the second one scores its polarity (positive or negative).

\[r_{3}:\texttt{Mentions}(\texttt{t}_{i},\texttt{e}_{i}) \Rightarrow\texttt{HasRole}(\texttt{e}_{i},\texttt{r}) \tag{2}\] \[r_{4}:\texttt{Mentions}(\texttt{t}_{i},\texttt{e}_{i}) \Rightarrow\texttt{EntPolarity}(\texttt{e}_{i},\texttt{p})\]

Note that these rules do not express any dependencies. They function as base classifiers that map tweets and entities to their most probable labels.

Dependency between roles and moral foundationsThe way an entity is portrayed in a tweet can be highly indicative of its moral foundation. For example, people are likely to mention _children_ as a _negative actor_ in the context of _care/harm_. To capture this, we explicitly model the dependency between an entity, its moral role, and the moral foundation.

\[r_{5}:\texttt{Mentions}(\texttt{t}_{i},\texttt{e}_{j})\land \texttt{HasRole}(\texttt{e}_{i},\texttt{r}) \tag{3}\] \[\land\texttt{EntPolarity}(\texttt{e}_{i},\texttt{p})\Rightarrow \texttt{HasMf}(\texttt{t}_{i},\texttt{m})\]

Dependency between stances and moral foundationsAs we showed in Section 4, there is a significant correlation between the stance of a tweet with

Figure 3: Correlation Heatmaps for LDA Topics

respect to the vaccine debate, and its moral foundation. For example, people who oppose the vaccine are more likely to express the liberty/oppression moral foundation. To capture this, we model the dependency between the stance of a tweet and its moral foundation.

\[r_{6}:\mathtt{VaxStance(t_{i},s)}\Rightarrow\mathtt{HasMf(t_{i},m)} \tag{4}\]

Dependency between reasons and moral foundations/stancesExplicitly modeling the dependency between repeating reasons and other decisions can help us add inductive bias into our model, potentially simplifying the task. For example, we can enforce the difference between two opposing views that use similar wording, and that could otherwise be treated similarly by a text-based model (e.g. _"natural methods of protection against the disease are better than vaccines"_ vs. _'vaccines are better than natural methods of protection against the disease"_). We add two rules to capture this dependency, one between reasons and moral foundations, and one between reasons and stances.

\[r_{7}:\mathtt{Mentions(t_{i},r)} \Rightarrow\mathtt{HasMf(t_{i},m)} \tag{5}\] \[r_{8}:\mathtt{Mentions(t_{i},r)} \Rightarrow\mathtt{VaxStance(t_{i},s)}\]

Hard constraintsTo enforce consistency between different decisions, we add two unweighted rules (or hard constraints). These rules are not associated with a scoring function and must always hold true. We enforce that, if a tweet is predicted to be moral, then it needs to also be associated to a specific moral foundation. Likewise, if a tweet is not moral, then no moral foundation should be assigned to it.

\[c_{0}:\mathtt{IsMoral(t_{i})}\Rightarrow\neg\mathtt{HasMf(t_{ i},none)} \tag{6}\] \[c_{1}:\neg\mathtt{IsMoral(t_{i})}\Rightarrow\mathtt{HasMf(t_{i}, none)}\]

Whenever different tweets have the same stance, we include a constraint to enforce consistency between the polarity of different mentions of the same entity. Roy et al. (2021) showed that enforcing consistency for mentions of the same entity within a political party was beneficial. Given the polarization of the Covid-19 vaccine, we use the same rationale.

\[c_{3}:\mathtt{Mentions(t_{i},e_{i})}\wedge\mathtt{Mentions(t_{j}, e_{j})} \tag{7}\] \[\land\mathtt{SameStance(t_{i},t_{j})}\land\mathtt{EntPolarity(e_{i },p)}\] \[\Rightarrow\mathtt{EntPolarity(e_{j},p)}\]

Learning and inferenceThe weights for each rule \(w_{r}:p_{0}\wedge p_{1}\wedge...\wedge p_{n}\Rightarrow h\) measure the importance of each rule in the model and can be learned from data. For example, when attempting to predict _care/harm_ for a tweet t\({}_{i}\), we would like the weight of rule instance IsTweet(t\({}_{i}\)) \(\Rightarrow\mathtt{HasMf(t_{i},care/harm)}\) to be greater than the weight of rule instance IsTweet(t\({}_{i}\)) \(\Rightarrow\mathtt{HasMf(t_{i},loyalty/betrayal)}\). In DRaiL, these weights are learned using neural networks with parameters \(\theta_{r}\). The collection of rules represents the global decision, and the solution is obtained by running a maximum a posteriori (MAP) inference procedure. Given that horn clauses can be expressed as linear inequalities corresponding to their disjunctive form, the MAP inference problem can be written as a linear program. DRaiL supports both locally and globally normalized structured prediction objectives. Throughout this paper, we used the locally normalized objective. For details about the learning procedure, we refer the reader to the original paper (Pacheco and Goldwasser, 2021).

Learning with low-supervisionTo learn DRaiL models in the low-supervision setting, we use an Expectation-Maximization style protocol, outlined in Algorithm 1. First, we initialize the parameters of base rules using distant supervision classifiers. For moral foundations, we use the Johnson and Goldwasser (2018) dataset and the Moral Foundation Twitter Corpus (Hoover et al., 2020). For roles, we use the Roy et al. (2021) dataset. For polarity, we combine the Roy et al. (2021) dataset with the MPQA 3.0 entity sentiment dataset (Deng and Wiebe, 2015). For vaccination stances, we annotate our 85K unlabeled tweets using a set of prominent antivax and provax hashtags. Details about these datasets are provided in Appendix A.4.

Once the base rules have been initialized using distant supervision, we turn our attention to learning DRaiL models over the Covid-19 dataset presented in Section 4. We alternate between MAP inference to obtain training labels (expectation step), and training the neural nets using these labels (maximization step). We receive an optional parameter \(k\) indicating the amount of direct supervision to be used. When \(k\) is provided, \(k\%\) of the annotated labels are seeded during inference.

## 6 Experimental Evaluation

The goal of our framework is to identify morality frames and opinions in tweets by modeling them 

jointly. In this section, we perform an exhaustive experimental analysis to evaluate the performance of our model and each of its components.

Experimental settingsIn DRailL, each rule \(r\) is associated with a neural architecture, which serves as a scoring function to obtain the rule weight \(w_{r}\). We use BERT-base-uncased Devlin et al. (2018) for all classifiers. For the rules that model dependencies (Eqs. 3, 4, 5), we concatenate the CLS token with a one-hot vector of the symbols on the left hand side of the rule (i.e. role, sentiment, stance and reason), before passing it through a classifier. For rules that have the entity on the left-hand side (Eqs. 2, 3), we use both the tweet and the entity as an input to BERT, using the SEP token. We trained supervised models using local normalization in DRaiL, and leveraged distant supervision using protocol outlined in Algorithm 1. In all cases, we used a learning rate of \(2e-5\), a maximum sequence length of \(100\), and AdamW. In all experiments, we perform 5-fold cross-validation over the annotated dataset and report the micro-averaged results.

General resultsTable 6 shows our general results for morality frames and vaccination stance. We evaluate our base classifiers and show the impact of modeling dependencies using DRaiL. The joint model results in a significant improvement for morality, moral foundation and vaccination stance. For entities, role and polarity remain stable. We also measure the impact of explicitly modeling reasons (Eq. 5) and present results in Table 7. We show the performance for the initial reasons proposed by Wawrzuta et al. (2021), which are all from the anti-vaccine perspective, and the impact of our two rounds of interaction, expanding and refining reasons (round 1) and augmenting argumentative patterns (round 2). We find that moral foundations improve from 60.07 to 62.27 and vaccination stance improves from 67.72 to 72.53 after interaction.

Ablation studyWe show an ablation study in Table 8. First, we can see how all dependencies contribute to the performance improvement, role-MF being the most impactful. We can also see that explicitly modeling morality constraints improves both the morality prediction and the moral foundation prediction, suggesting an advantage to breaking down this decision. We observe that the stance-polarity constraint does not have a significant impact, but does not hurt performance either, suggesting that our classifiers already capture this information. Lastly, we can see that the performance for roles and polarity remains stable, potentially because these classifiers have a strong starting point.

Distant supervisionIn Figure 4 we evaluate the impact of our indirect supervision protocol by slowly augmenting the amount of direct supervision available. We can see that by leveraging out of domain-data and dependencies, we can obtain a

\begin{table}
\begin{tabular}{l|c c|c c|c c|c c|c c} \hline \hline

- \\ \hline Base (distant sup.) & 69.77 & 68.88 & 28.79 & 41.27 & 71.94 & 72.05 & 63.88 & 74.30 & 69.46 & 70.35 \\ Base (direct sup.) & 68.94 & 69.71 & 35.28 & 42.92 & **84.71** & **84.75** & **72.92** & **84.31** & 66.91 & 67.36 \\ + **Joint Model** & **80.53** & **81.17** & **53.29** & **62.27** & 84.60 & 84.64 & 71.53 & 83.35 & **72.06** & **72.53** \\ \hline \hline \end{tabular}
\end{table}
Table 6: General Results (F1 Scores). NM: Non Moral

\begin{table}
\begin{tabular}{l l l} \hline \hline
**Model** & **MF** & **Var. Sax. Stance** \\ \hline
**ALL** (-Reasons) & 60.07 & 67.72 \\ + Reasons-Original & 61.51 & 72.62 \\ + Reasons-Interaction-1 & 61.21 & **73.83** \\ + Reasons-Interaction-2 & **62.27** & 72.53 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Contribution of reasons at different interaction rounds (Weighted F1) 

competitive model using just 25% of the annotated labels, and we can outperform the fully supervised classifiers using 50% of the annotations.

## 7 Related Work

Recent studies have noted the prevalence of rumors and misinformation in the context of the Covid-19 pandemic Loomba et al. (2021); Shahi et al. (2021); Lazarus et al. (2021); Ahmed et al. (2020). Following this trend, several computational approaches have been proposed to detect misinformation related to Covid in news outlets and social media Weinzierl and Harabagiu (2021); Bang et al. (2021); Serrano et al. (2020); Al-Rakhami and Almari (2020). In this paper, we take a different approach and look at the problem of identifying opinions surrounding the Covid-19 vaccine, and explicitly modeling the rationale and moral sentiment that motivates them.

Some recent works also look at analyzing arguments about Covid and vaccine hesitancy more broadly. In most cases, they either take a traditional classification approach for predicting stances Alilieibi et al. (2021); Lyu et al. (2021), or use topic modeling techniques to uncover trends in word usage Skeppstedt et al. (2018); Lyu et al. (2021); Sha et al. (2020); Zamani et al. (2020). In contrast, we propose a holistic framework that combines different methodological techniques, including human-in-the-loop mechanisms, classification with distant supervision, and deep relational learning to connect stance prediction, reason analysis and fine-grained entity moral sentiment analysis.

## 8 Discussion

We introduce a holistic framework for analyzing social media posts about the Covid-19 vaccine. We model morality frames and opinions jointly, and show that we can obtain competitive performance. The main limitation of our work is the size of the annotated dataset studied. Annotating for morality is a difficult and costly task, as it requires significant domain expertise. This motivates the need for methods that perform well under limited supervision, and that can leverage external and unlabeled resources. We took a first step in this direction by combining a wide range of methodological strategies. Given the amount of data generated daily about Covid, there are broader opportunities for exploiting these resources than what were explored in this paper. While we provided a preliminary analysis of the correlation between stances, reasons and morality, our current work looks at leveraging this framework to analyze opinions at scale.

We also presented a first step towards interactive exploration of opinions on social media. While we explored this approach in a limited scenario, there is a lot of potential for using this paradigm for diagnosing NLP models and adapting to new domains. More research is required to devise protocols and evaluation strategies for this process.

## 9 Acknowledgements

We gratefully acknowledge our in-house annotators for their work annotating the dataset presented in this paper. We also thank the anonymous reviewers of this paper for their insightful comments. This project was partially funded by a Microsoft Research Dissertation Grant, a Purdue Graduate School Summer Research Grant and an NSF CAREER award IIS-2048001.

## References

* Ahmed et al. (2020) Wasim Ahmed, Josep Vidal-Alaball, Joseph Downing, Francesc Lopez Segui, et al. 2020. Covid-19 and the 5g conspiracy theory: social network analysis of twitter data. _Journal of medical internet research_, 22(5):e19458.
* Bensouss et al. (2016)

Figure 4: Performance in low-supervision settings



Mabrook S Al-Rakhami and Atif M Al-Amri. 2020. Lies kill, facts save: detecting covid-19 misinformation in twitter. _Ieee Access_, 8:155961-155970.
* Alam et al. (2021) Firoj Alam, Fahim Dalvi, Shaden Shaar, Nadir Durrani, Hamdy Mubarak, Alex Nikolov, Giovanni Da San Martino, Ahmed Abdelali, Hassan Sajjad, Kareem Darwish, et al. 2021. Fighting the covid-19 infodemic in social media: A holistic perspective and a call to arms. In _Proceedings of the International AAAI Conference on Web and Social Media_, volume 15, pages 913-922.
* Alllieibi et al. (2021) Fahad M Alllieibib, Abdulfattah Omar, and Nasser Alkorais. 2021. Opinion mining of saudi responses to covid-19 vaccines on twitter. _International Journal of Advanced Computer Science and Applications_, 12(6):72-78.
* Bang et al. (2021) Yejin Bang, Etsuko Ishii, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. 2021. Model generalization on covid-19 fake news detection. _arXiv preprint arXiv:2101.03841_.
* Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. _Journal of machine Learning research_, 3(Jan):993-1022.
* Braun and Clarke (2012) Virginia Braun and Victoria Clarke. 2012. _Thematic analysis._, pages 57-71.
* Chan (2021) Eugene Y Chan. 2021. Moral foundations underlying behavioral compliance during the covid-19 pandemic. _Personality and individual differences_, 171:110463.
* Deng and Wiebe (2015) Lingjia Deng and Janyce Wiebe. 2015. MPQA 3.0: An entity/event-level sentiment corpus. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1323-1328, Denver, Colorado. Association for Computational Linguistics.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Diaz and Cova (2021) Rodrigo Diaz and Florian Cova. 2021. Reactance, morality, and disgust: The relationship between affective dispositions and compliance with official health recommendations during the covid-19 pandemic. _Cognition and Emotion_, pages 1-17.
* Glandt et al. (2021) Kyle Glandt, Sarthak Khanal, Yingjie Li, Doina Caragea, and Cornelia Caragea. 2021. Stance detection in COVID-19 tweets. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1596-1611, Online. Association for Computational Linguistics.
* Graham et al. (2009) Jesse Graham, Jonathan Haidt, and Brian A Nosek. 2009. Liberals and conservatives rely on different sets of moral foundations. _Journal of personality and social psychology_, 96(5):1029.
* Haidt and Graham (2007) Jonathan Haidt and Jesse Graham. 2007. When morality opposes justice: Conservatives have moral intuitions that liberals may not recognize. _Social Justice Research_, 20(1):98-116.
* Hasan and Ng (2014) Kazi Saidul Hasan and Vincent Ng. 2014. Why are you taking this stance? identifying and classifying reasons in ideological debates. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 751-762, Doha, Qatar. Association for Computational Linguistics.
* Hoover et al. (2020) J. Hoover, G. Portillo-Wightman, L. Yeh, S. Havaldar, A.M. Davani, Y. Lin, B. Kennedy, M. Atari, Z. Kamel, M. Mendlen, G. Moreno, C. Park, T.E. Chang, J. Chin, C. Leong, J.Y. Leung, A. Mirinjian, and M. Dehghani. 2020. Moral foundations twitter corpus: A collection of 35k tweets annotated for moral sentiment. _Social Psychological and Personality Science_, 11(8):1057-1071.
* Hossain et al. (2020) Tamanna Hossain, Robert L. Logan IV, Arjunta Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh. 2020. COVIDLies: Detecting COVID-19 misinformation on social media. In _Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020_, Online. Association for Computational Linguistics.
* Johnson and Goldwasser (2018) Kristen Johnson and Dan Goldwasser. 2018. Classification of moral foundations in microblog political discourse. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 720-730, Melbourne, Australia. Association for Computational Linguistics.
* Krippendorff (2004) Klaus Krippendorff. 2004. Measuring the reliability of qualitative text analysis data. _Quality and quantity_, 38:787-800.
* Lazarus et al. (2021) Jeffrey V Lazarus, Scott C Ratzan, Adam Palayew, Lawrence O Gostin, Heidi J Larson, Kenneth Rabin, Spencer Kimball, and Ayman El-Mohandes. 2021. A global survey of potential acceptance of a covid-19 vaccine. _Nature medicine_, 27(2):225-228.
* Loomba et al. (2021) Sahil Loomba, Alexandre de Figueiredo, Simon J Piatek, Kristen de Graaf, and Heidi J Larson. 2021. Measuring the impact of covid-19 vaccine misinformation on vaccination intent in the uk and usa. _Nature human behaviour_, 5(3):337-348.
* Lyu et al. (2021) Joanne Chen Lyu, Eileen Le Han, and Garving K Luli. 2021. Covid-19 vaccine-related discussion on twitter: topic modeling and sentiment analysis. _Journal of medical Internet research_, 23(6):e24435.
* Montagni et al. (2021) Ilaria Montagni, Kevin Ouazzani-Touhami, A Mebarki, N Texier, S Schuck, Christophe Tzourio, et al. 2021. Acceptance of a covid-19 vaccine is associated with ability to detect fake news and health literacy. _Journal of public health (Oxford, England)_ 

Goran Muric, Yusong Wu, and Emilio Ferrara. 2021. COVID-19 Vaccine Hesitancy on Social Media: Building a Public Twitter Dataset of Anti-vaccine Content, Vaccine Misinformation and Conspiracies.
* Pacheco and Goldwasser (2021) Maria Leonor Pacheco and Dan Goldwasser. 2021. Modeling content and context with deep relational learning. _Transactions of the Association for Computational Linguistics_, 9:100-119.
* Pagliaro et al. (2021) Stefano Pagliaro, Simona Sacchi, Maria Giuseppina Pacilli, Marco Brambilla, Francesca Lionetti, Karim Bettache, Mauro Bianchi, Marco Biella, Virginie Bonnot, Mihaela Boza, et al. 2021. Trust predicts covid-19 prescribed and discretionary behavioral intentions in 23 countries. _PloS one_, 16(3):e0248334.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
* Rousseeuw (1987) Peter Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. _J. Comput. Appl. Math._, 20(1):53-65.
* Roy et al. (2021) Shamik Roy, Maria Leonor Pacheco, and Dan Goldwasser. 2021. Identifying morality frames in political tweets using relational learning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 9939-9958, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Serrano et al. (2020) Juan Carlos Medina Serrano, Orestis Papakyriakopoulos, and Simon Hegelich. 2020. Nlp-based feature extraction for the detection of covid-19 misinformation videos on youtube. In _Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020_.
* Sha et al. (2020) Hao Sha, Mohammad Al Hasan, George Mohler, and P Jeffrey Brantingham. 2020. Dynamic topic modeling of the covid-19 twitter narrative among us governors and cabinet executives. _arXiv preprint arXiv:2004.11692_.
* Shahi et al. (2021) Gautam Kishore Shahi, Anne Dirkson, and Tim A Majchrzak. 2021. An exploratory study of covid-19 misinformation on twitter. _Online social networks and media_, 22:100104.
* Skeppostedt et al. (2018) Maria Skeppostedt, Andreas Kerren, and Manfred Stede. 2018. Vaccine hesitancy in discussion forums: computer-assisted argument mining with topic models. In _Building Continents of Knowledge in Oceans of Data: The Future of Co-Created eHealth_, pages 366-370. IOS Press.
* Sowa et al. (2021) Pawel Sowa, Lukasz Kisszkiel, Piotr Pawel Laskowski, Maciej Alimowski, Lukasz Szczerbinski, Marlena Paniczko, Anna Moniuszka-Malinowska, and Karol Kaminski. 2021. Covid-19 vaccine hesitancy in poland--multifactorial impact trajectories. _Vaccines_, 9(8):876.
* Subramanian et al. (2018) Shivashankar Subramanian, Trevor Cohn, and Timothy Baldwin. 2018. Hierarchical structured model for fine-to-coarse manifesto text analysis. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1964-1974, New Orleans, Louisiana. Association for Computational Linguistics.
* Tagliabue et al. (2020) Fabio Tagliabue, Luca Galassi, and Pierpaolo Mariani. 2020. The "pandemic" of disinformation in covid-19. _SN comprehensive clinical medicine_, 2(9):1287-1289.
* van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. _Journal of Machine Learning Research_, 9:2579-2605.
* Wawrzuta et al. (2021) Dominik Wawrzuta, Mariusz Jaworski, Joanna Gotlib, and Mariusz Panczyk. 2021. What arguments against covid-19 vaccines run on facebook in poland: Content analysis of comments. _Vaccines_, 9(5):481.
* Weinzierl et al. (2021) Maxwell Weinzierl, Suellen Hopfer, and Sanda M Harabagiu. 2021. Misinformation adoption or rejection in the era of covid-19. In _Proceedings of the International AAAI Conference on Web and Social Media_, volume 15, pages 787-795.
* Weinizierl and Harabagiu (2021) Maxwell A Weinzierl and Sanda M Harabagiu. 2021. Automatic detection of covid-19 vaccine misinformation with graph link prediction. _Journal of biomedical informatics_, 124:103955.
* Widmoser et al. (2021) Manuel Widmoser, Maria Leonor Pacheco, Jean Honorio, and Dan Goldwasser. 2021. Randomized deep structured prediction for discourse-level processing. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 1174-1184, Online. Association for Computational Linguistics.
* Zamani et al. (2020) Mohammad Zamani, H Andrew Schwartz, Johannes Eichstaedt, Sharath Chandra Guntuku, Adithya Virinchipuram Ganesan, Sean Clouston, and Salvatore Giorgi. 2020a. Understanding weekly covid-19 concerns through dynamic content-specific lda topic modeling. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, volume 2020, page 193. NIH Public Access.
* Zamani et al. (2020) Mohammad Zamanzi, H. Andrew Schwartz, Johannes Eichstaedt, Sharath Chandra Guntuku, Aditya Virinchipuram Ganesan, Sean Clouston, and Salvatore Giorgi. 2020b. Understanding weekly COVID-19 concerns through dynamic content-specific LDA topic modeling. In _Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science_, pages 193-198, Online. Association for Computational Linguistics 

Appendix

### Reasons and Phrases

Tables 10 and 11 show the full list of phrases for anti-vax and pro-vax reasons. The interactive task interface is presented in Figures 5 and 6. Bar plots for reason assignments before and after interaction are shown in Figure 7.

### Data Collection

To create the list of keywords used to collect tweets about the Covid-19 vaccine, we read multiple articles about Covid mentioning vaccination status, vaccine hesitancy, misinformation, vaccine constraints, health issues, religious sentiment and other vaccine-related debates, and made a list of repeating statements. Then, we consulted three researchers, two in Computational Social Science and one in Psychology, and constructed a list of relevant keywords that are indicative of morally charged discussions. The full list of keywords can observed Table 12.

### Data Annotation Task

The steps for annotating tweets using our graphical interface are (See Figure 10).

1. **Select** the moral foundation of the text using the checkbox \(\not{\Box}\). You can see the definition of each moral foundation by hovering your mouse over them. If the tweet does not make any moral judgement, **check**\(\not{\Box}\) "none". In this case, you don't have to highlight actor-target polarity.
2. After selecting any moral foundation other than "none", text highlighting for actor-target role with polarity will be visible below. If you select a moral foundation other than "none", you can highlight the actor-target polarity.
3. **Choose** the color-coded label Positive Actor/Positive Target/Negative Actor/Negative Target to highlight the text with the color of the selected label. You can see the definition of actor-target-polarity role by hovering your mouse over them.
4. **Highlight** words, phrases, or sections of the text for the actor-target role with polarity of the corresponding moral foundation.
5. If you made any mistakes in highlighting, select the "Unhighlight" button to unhighlight the previously highlighted text.
6. Finally, click the "**Submit**" button to submit the task.

We provided eight examples (Figure 11) covering six moral principles and non-moral cases to make our annotation task more understandable. Annotators could see the explanations for choosing a moral foundation and an. actor-target polarity by clicking the "**See Explanation**" button.

Annotators had to complete two practice examples before starting the real task. If they made any mistake, our practice session provided them the correct result with an explanation. Figure 12 shows the interface for one of the two practice examples.

### Out-of-Domain Datasets

For moral foundation prediction, we use the dataset proposed by Johnson and Goldwasser (2018), consisting of 2K tweets by US congress members annotated for the five core moral foundations. We also use the Moral Foundation Twitter Corpus

Figure 8: Wordclouds for reasons **before** interaction.

\begin{table}
\begin{tabular}{|p{227.6pt}|} \hline covid vaccine, covid vaccination, covid vaccine tyannay, covid vaccine mrandate, ovoid vaccine summary, crowd vaccine amit-vax, corvid vaccine ethnic, convid vaccine stem, cavid vaccine path, evoid vaccine stima, civil vaccine minhall, ovoid vaccine chrithan, ocvid vaccine chrithan, oval Vaccine chrithan, covid vaccine chrithan, cvoid vaccine etholcholch, covid vaccine rhelatum, cosvid vaccine religions, covid vaccine bilden failure, cavid vaccine lenggport, court vaccine leglty, covid vaccine cheaning, cavid vaccine rheolvant, cavid vaccine liberaty, cavid vaccine liberaly, cavid vaccine liberacy, cavid vaccine brill, cavid vaccine brebruary, cavid vaccine pregrant, covid vaccine approval, cavid vaccine glot, cavid vaccine flat, cavid vaccine edc, cavid vaccine faaci, Cavid-19-thna, vaccine passport, vacination mandate, cavid vaccine death, cavid vaccine military, experimental evido vaccine, cavid vaccine authorization, vaccine epprusent, vaccine stimuli, over vaccine bill garets, cavid vaccine side effect, cavid vaccine adverse events

Figure 9: Wordclouds for reasons **after** interaction.

 



\begin{table}
\begin{tabular}{|l|p{284.5pt}|} \hline  & Themes \\ \hline
**GovDistrust** & Add phrases with strong word for distrust \\  & “Good at being bad” \\  & Explicit negations \\ \hline
**GovTrust** & Hedging phrases (sort-of trust) \\ \hline
**VaxDanger** & Closer connection between vaccine words and danger words (related to sickness, bad effects) \\  & Explit negations \\  & Rhetorical questions \\  & Refusing the vaccine for medical reasons \\ \hline
**VaxSafe** & Explicit mentions of safety \\  & Explicit negations \\ \hline
**CovidFake** & Stronger relevant negative words (fake, scam, hoax) \\  & Explicit negations \\ \hline
**CovidReal** & Trust the science \\  & References to Covid hospitalization on the rise, explicit mentions of hospitals \\  & Explicit negations \\ \hline
**VaxOppression** & Legal language \\  & Explicit mentions of discrimination and oppression \\  & Sarcasm \\ \hline
**VaxNotOppression** & Justifying mandates \\  & Freedom to be protected \\  & Criticizing others using “you/people” language, focus freedom on me/my/I \\ \hline
**BigPharmaAnti** & Stronger words against pharmaceutical companies (corrupt, evil) \\  & Not accountable / irresponsible past behavior \\  & Mentions of negative side-effect of other products (cancer) \\ \hline
**BigPharmaPro** & Trust science/research and vaccine development process \\  & Language about intent, the vaccine was created to do something good, explicit names of companies \\ \hline
**NaturalImmunityPro** & The vaccine is not enough \\  & Explicit mentions to population immunity, herd immunity and antibodies \\ \hline
**NaturalImmunityAnti** & Emphasis on global look, collective entities, society \\  & Natural immunity characterized as dangerous or not effective \\  & Mentions of experts and trusting science \\ \hline
**VaxAgainstReligion** & I put it in god hands (god is deciding) \\  & Treating pro-vax as another religion \\ \hline
**VaxNotAgainstReligion** & “Relibigus" in quotes \\  & Bugus exemptions \\  & “Where is your faith” \\  & Call to action: get tested/get vaccinated/put a mask on (mentions of compassion) \\  & No religion ask members to refuse vaccine \\ \hline
**VaxDoesntWork** & Reference to “magic vaccine” \\  & “Never developed”, “doesn’t work” \\  & Questions: why are deaths high? Why is corona not going away? Why are vaccinated people dying? \\ \hline
**VaxWorks** & “ask a doctor”, consult with an expert \\  & Research on the vaccine is good/has been going on for a long time \\  & Capture differences, e.g.. “good trials” vs. rushed ones. \\ \hline
**VaxNotTested** & Language suggesting “rushed through trials” and “experimental vaccine” \\ \hline
**VaxTested** & trust the research and development process \\  & Testing can be confused with covid-test, use other language. \\ \hline \end{tabular}
\end{table}
Table 9: Overarching argumentation patterns uncovered by coders during interaction



\begin{table}
\begin{tabular}{|p{28.5pt} p{341.7pt}|} \hline \multicolumn{2}{c|}{Tenses} & \\ \hline
**GovDistrust** & “**tack of trust in the government”**, “Fuck the government ”, The government is a total failure”, \\  & “Never trust the governmentment”, “Biden is a failure”, “Bladen legal people die”, \\  & “The government and Fauci have been dishonest”, “The government always lies”, \\
**GovDistrust** & “The government has a strong record of screwing things up", “The government is good at screwing things up”, “The government only cares about money”, \\  & “The government doesn’t work logically”, “Do not trust the government”, \\  & “The government doesn't care about people’s health”, “The government won’t tell you the truth about the vaccine” \\ \hline
**VaxDanger** & “**the vaccine will be dangerous to health”**, “Covid vaccines can cause blood clots”, \\  & “The vaccine is a greater danger for our children’s health than COVID itself”, \\  & “The vaccine will kill you’, “The experimental covid vaccine is a death jab”, \\  & “The socialy vaccine causes cancer”, The void vaccine is harmful for pregnant women and kids”, \\  & “The vaccine increases health risk”, “The vaccine isn’ts safe”, \\  & “What are vaccines good for? Nothing, rather it increases risk.” \\  & “I and many others have medical exemptions”, “The vaccine is dangerous for people with medical conditions”, \\  & “I won’t take the vaccine due to medical reasons”, “The vaccine has dangerous side effects” \\ \hline
**CavOipfake** & “**Covid-19 disease does not exist”**, “Covid is fake”, “covid is a hoax”, “covid is a scam”, \\  & “covid is propaganda”, “the pandemic is a lie”, “covid isn’t real”, “Id don’t think that could is real”, \\  & “I don’t buy that corbiór said i real”, “I don’t think there is a panomicle”, \\  & “I don’t think the pandemics’ next, “I don’t buy that there is a pandemic” \\ \hline
**VaxOppression** & “**If do not want to be vaccinated because I have freedom of choice”** \\  & “Forcing people to take experimental vaccines is opposition”, \\  & “The vaccine has nothing to do with Covid-19; it’s about the vaccine passport and tyranny”, \\  & “The vaccine mandate is unconstitutionian”,“I choose not to take the vaccine”, “My body my choice”, “Tm not against the vaccine but I am against the mandate”, \\  & “I have freedom to choose not to take the vaccine”, “I am free to refuse the vaccine” \\  & “It is not about court, it is about control”, “Medical segregation based on vaccine mandates is discrimination”, \\  & “The vaccine mandate violates my rights”, “Falsely labeling the injection as a vaccine is illegal”, \\  & “Friring over vaccine mandates is oppression”, “Vaccine passports are medical tyranny”, \\  & “I won’t let the government tell me what I should do with my body”, “I won’t have the government tell me what to do” \\ \hline
**BigPharmaAnti** & “**the vaccine was created only for the profit of pharmaceutical companies”, \\  & “We are the subjects of massive experiments for the Modern and Pfizer Vaccines”, \\  & “Pharmaceutical companies are corrupt”, “The pharmaceutical industry is rotten”, Big Pharma is evil”, \\  & “How would you trust big pharma with the COVID vaccine? They haven’t been liable for vaccine harm in the past”, \\  & “COVID vaccines are not doing what the pharmaceutical companies promised”, \\  & “Parameteric companies have a history of irresponsible behavior”, \\  & “I don’t trust Johnson \& Johnson after knowing their baby powder caused cancer for decades” \\ \hline
**NaftImmunityPro** & “**natural methods of protection against the disease are better than vaccines**”, \\  & “Herd immunity is broad, protective, and durable”, \\  & “Natural immunity has higher level of protection than the vaccine”, “Embrace population immunity”, \\  & “I trust my immune system”, ”I have antibodies I do not need the vaccine”, “Natural immunity is effective” \\ \hline
**VaxAgainstReligion** & “The vaccine is against my religion”, “The vaccines are the mark of the least”, “The vaccine is a tool of Satan”, \\  & “The vaccine is harnym”, “The vaccine is not hualz”, \\  & “I will protect my body from a manbac vaccine”, “I put it all in God’s hands”, “God will decide our fate”, \\  & “The vaccine contains bovine, which conflicts with my religion”, \\  & “The vaccine contains aborted fetal tissue which is against my religion”, \\  & “The vaccine contains aborted fetal tissue which is against my religion”, \\  & “The vaccine contains gross, muslims can’t take the vaccine ”, ’Jesus will protect me”, \\  & “The vaccine doesn’t protect own fighting getting or spreading Covid, God does”, “The covid vaccine is another religion” \\ \hline
**VaxDoesntWork** & “**the vaccine does not work”**, “covid vaccines do not stop the spread”, \\  & “If the vaccine works, why are deaths so high?”, “Why are vaccinated people dying?”, \\  & “If the vaccine works, why is costly not going away?” \\ \hline
**VaxNotTested** & “**the Vaccine is not properly tested, it has been developed too quickly”**, \\  & “Covid-19 vaccines have not been through the same rigorous testing as other vaccines”, \\  & “The Covid vaccine is experimental”, “The covid vaccine uses raushed through trials”, \\  & “The approval of the experimental vaccine was rashed”, “How was the vaccine developed so quickly?” \\ \hline
**VaxExperimentDogs** & “Animal shelters are empty because Dr Fauct allowed experimenting of various Covid vaccines/drug ones and other domestic pets”, \\  & “Faucit routers dogs and puppies” \\ \hline
**BillGatesMicroChip** & “The covid vaccine is a ploy to microchip people”, \\  & “Bell Gates Watro using it use vaccines to implant microchips in people”, \\  & “Globalists support a cover mass chip implantation through the covid vaccine” \\ \hline VaxFastFlassue & “There is aborted fetal tissue in the Covid Vaccines”, “the Covid vaccines contain aborted fetal cells” \\ \hline VaxMakeYouSterile & “The covid vaccine will make you sterile”, “Covid vaccine will affect your fertility” \\ \hline NoResponsibility & no one is responsible for the potential side effects of the vaccine \\ \hline SwinPetVisar & mentioning the past development of the swine flu vaccine \\ \hline VaxResistance & the vaccine has existed before the Covid-19 epidemic, now there is too much resistance \\ \hline ConspiracyThoriesories & conspiracy theories, hidden vaccine effects (e.g., chips) \\ \hline \end{tabular}
\end{table}
Table 10: AntiVax Themes and phrases for Covid-19 talking points. Themes that were added during interaction are shown in blue. Themes that were removed during interaction are shown in red. The original explanations/examples are presented in bold.





\begin{table}
\begin{tabular}{|l|p{284.5pt}|} \hline Themes & \\ \hline  & “We trust the government”, “The government cares for people”, \\  & “We are thankful to the government for the vaccine availability”, \\
**GovTrust** & “Hats off to the government for tackling the pandemic’; \\  & “It is a good thing to be skeptical of the government, but they are right about the covid vaccine”, \\  & “It is a good thing to be skeptical of the government, but they haven’t lied about the covid vaccine”, \\  & “The government can be corrupt, but they are telling the truth about the covid vaccine”, \\  & “The government can be corrupt, but they are not lying about the covid vaccine” \\ \hline  & “The vaccine is safe”, “Millions have been vaccinated with only mild side effects”, \\  & “Millions have been safely vaccinated against covid”, ”The benefits of the vaccine outweigh its risks”, \\
**VaxSafe** & “The vaccine has benefits”, “The vaccine is safe for women and kids”, ("The vaccine won’t make you sick”, \\  & “The vaccine isn’t dangerous”, “The vaccine won’t kill you”, \\  & “The covid vaccine isn’t a death jab”,” The covid vaccine doesn’t harm women and kids* \\ \hline  & “Covid is real”, “I trust science”, “Covid death is real”, \\  & “The science doesn’t lie about covid”, “Scientist know what they are doing”, \\
**CovidReal** & “Seientist know what they are saying”, “Covid hospitalizations are on the rise”, \\  & “Covid hospitalizations are climbing as output stage surge continues”, \\  & “Covid's death toll has grown faster”, “Covid is not a hoax”, “The pandemic is not a lie”, \\  & “The pandemic is not a lie, hospitalizations are on the rise” \\ \hline  & “The vaccine mandate is not oppression because vaccines lower hospitalizations and death rates”, \\  & “The vaccine mandate is not oppression because it will help to end this pandemic”, \\  & “The vaccine mandate will help us end the pandemic”, \\  & “We need a vaccine mandate to end this pandemic”, “I support vaccine mandates”, \\  & “If you don’t get the vaccine based on your freedom of choice, \\
**VaxNotOppression** & don’t come crawling to the emergency room when you got COVID”, \\  & “If you refuse a free FDA-approved vaccine for non-medical reasons, then the government shouldn’t continue to give you free COVID tests”, \\  & “You are free net too take the vaccine businesses are also free to deny you entry”, \\  & “You are free not to take the vaccine, businessess are free to protect their customers and employees”, \\  & “If you choose not to take the vaccine, you have to deal with the consequences”, \\  & “If it is your body your choice, then insurance companies should stop paying for your hospitalization costs for COVID* \\ \hline  & “I trust the science and pharmaceutical research”, “Pharmaceutical companies are not hiding anything”, \\  & “The research behind covid vaccines is public”, “The Pfizer vaccine is saving lives”, \\ BigPharmaPro & “The Moderna vaccines are helping stop the spread of covid*, \\  & “The Johnson and Johnson vaccine was created to stop covid*, \\  & “Pharmaceutical companies are seeking FDA approval”, Pharmaceutical companies are following standard protocols* \\ \hline  & “Only the vaccine will end the pandemic”, \\ NatImmunityAnti & “ Vaccines will allow us to defeat covid without death and sickness*, \\  & “The vaccine has better long term protection than to natural immunity”, “Natural immunity is not effective”, \\  & “Natural immunity would require a lot of people getting sick”, \\  & “Exports recommend the vaccine over natural immunity” \\ \hline  & “The vaccine is not against religion, get the vaccine”, “No religion ask members to refuse the vaccine”, \\ VaxReligionOk & “Religious exemptions are bogus”, \\  & “When turning in your religious resorption forms for the vaccine, remember ignorance is not a religion”, \\  & “Disregard for others”lives isn’t part of your religion”, \\  & “Jeaus is trying to protect us from covid by drinly inspiring scientists to create vaccines” \\ \hline  & “The vaccine works”, “vacces do work, ask a doctor or consult with an expert”, \\
**VaxWorks** & “The covid vaccine helps to stop the spread”, “Unvaccinated people are dying at a rapid rate from Covid-19”, \\  & “There is a lot of research supporting that vaccines work”, \\  & “The research on the covid vaccine has been going on for a long time* \\ \hline  & “Covid vaccine research has been going on for a while*, “Plenty of research has been done on the covid vaccine”, \\
**VaxTested** & “The technologies used to develop the Covid-19 vaccines have been in development for years to prepare for outbreaks of infectious viruses*, \\  & “The testing processes for the vaccines were thorough didn’t skip any steps”, “The Vaccine received FDA approval” \\ \hline
**ProVax** & **positive attitude** \\ \hline \end{tabular}
\end{table}
Table 11: ProVax reasons and phrases. Reasons that were added during interaction are shown in blue. Reasons that were removed during interaction are shown in red. The original explanatory phrases are presented in bold.



(Hoover et al., 2020), consisting of 35k tweets annotated for moral foundations. The topics across these two datasets span political issues (e.g. gun control, immigration) and events (e.g. Hurricane Sandy, Baltimore protests). Given that neither of these two datasets contain examples for the _liberty/oppression_ moral foundation, we curate a small lexicon by looking for synonyms and antonyms of the words _library_ and _oppression_. Then, we use this lexicon to annotate the congressstweets dataset +. We annotate a tweet as _libertry/oppression_ if it contains at least four keywords, which results in around 2K tweets. The derived lexicon for liberty/oppression can be seen in Table13

Footnote †: [https://github.com/alexlitel/congressstweets](https://github.com/alexlitel/congressstweets)

To learn to predict roles, we use the subset of Johnson and Goldwasser (2018) dataset annotated for roles by Roy et al. (2021), which contains roughly 3K tweet-entity-role triplets. For polarity, we combine the Roy et al. (2021) dataset with the MPQA 3.0 entity sentiment dataset (Deng and Wiebe, 2015), which contains about 1.6K entity-sentiment pairs.

For stance, we annotate our dataset of 85K unlabeled covid tweets using a set of prominent antivax and provax hashtags. For the antivax case, we rely on the hashtags proposed by Muric et al. (2021). For the provax case, we manually annotate hashtags that have a clear provax message, and that are used in at least 50 tweets in our unlabeled dataset. The full set of hashtags used can be found in Tables14 and 15.

\begin{table}
\begin{tabular}{|p{199.2pt}|} \hline liberty, independence, freedom, autonomy, sovereignty self-government, self-rule, self-determination, home-rule civil liberties, civil rights, human rights, autarky, free-rein, latitude, option, choice, volition, democracy, compression, persecution, abuse, maltreatment, ill treatment, dictator, dictatorship, autocracy, tyranny, despotism, repression, suppression, subjugation, enslacurement, exploitation, dependence, constraint, control, totalitarianism \\ \hline \end{tabular}
\end{table}
Table 13: Liberty/Oppression Lexicon.

Figure 5: Interactive task interface.

 

* [25] S. K. G. Wilson, J. M. R. Walsh, and D. W. T. Watts, "A new approach to the computation of the number of nodes in a network," _Journal of Computational Physics: Series B_**106**(4), pp. 379-488.
* [26] A. L. C. De Silva, P. F. de Oliveira, and E. J. Newman, "Theoretical analysis of the number of nodes in a network," _Computational Science & Engineering_**106**(1), pp. 1-10.
* [27] A. L. C. De Silva, P. F. de Oliveira, and E. J. Newman, "On the number of nodes in a network," _Computer Methods in Computer Analysis_**106**(1), pp. 1-10.

[MISSING_PAGE_POST]

 

What is the **moral foundation** of the following tweet?

The government is forcing us to risk our health with these experimental COVID-19 vaccine.

\begin{tabular}{ 

* [19] J.-M. Chen, M.-H. Lee, and S.-J. Kim, "A new approach to the detection of a large number of people in an image using deep neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 2048-2053, 2016.
* [20] A. Dosovitskiy, V. Khosla, and L. Krausek, "Deep residual learning for image recognition with convolutional neural networks," in _IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 779-783, 2016.
* [21] C.-S. Chan, R. Girshick, and K. Zisserman, "Image classification by object detection: A survey," in _Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, Springer, 2016.
* [22] H.-C. Cheng, Y.-L. Lin, and X.-F. Wang, "Imagenet classification by object detection via deep convolutional neural networks," in _International Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, Springer, 2016.
* [23] G.-Y. Li, P.-W. Liu, and W.-T. Huang, "Learning feature representations from visual data using deep belief nets," in _Advances in Neural Information Processing Systems (IJCNN)_, vol. 10, no. 1, pp. 1-10, Springer, 2016.
* [24] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualizing features from optical images," in _European conference on computer vision and pattern recognition_, pp. 1-1, Springer, 2016.
* [25] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [26] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualization of human objects in a scene," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [27] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [28] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [29] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [30] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [31] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [32] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [33] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [34] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [35] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [36] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [37] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [38] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [39] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [40] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [41] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [42] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [43] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [44] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [45] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [46] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [47] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [48] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [49] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [50] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [51] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [52] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [53] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [54] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [55] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [56] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [57] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [58] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [59] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [60] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [61] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [62] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-1, Springer, 2016.
* [63] T.-P. Ma, B.-B. Park, and F.-R. Wu, "Visualized image representation from optical images," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CV