SlowGuess/NAACL_2022_OCR huggingface dataset:
To score the moral role of an entity  $\mathsf{e_i}$  mentioned in tweet  $\mathsf{t_i}$ , we write two rules. The first one scores whether the entity  $\mathsf{e_i}$  is an actor or a target, and the second one scores its polarity (positive or negative).
$$
r _ {3}: \text {M e n t i o n s} \left(t _ {i}, e _ {i}\right) \Rightarrow \text {H a s R o l e} \left(e _ {i}, r\right)
$$
$$
r _ {4}: \text {M e n t i o n s} \left(\mathrm {t} _ {\mathrm {i}}, \mathrm {e} _ {\mathrm {i}}\right) \Rightarrow \text {E n t P o l a r i t y} \left(\mathrm {e} _ {\mathrm {i}}, \mathrm {p}\right) \tag {2}
$$
Note that these rules do not express any dependencies. They function as base classifiers that map tweets and entities to their most probable labels.
Dependency between roles and moral foundations The way an entity is portrayed in a tweet can be highly indicative of its moral foundation. For example, people are likely to mention children as a negative actor in the context of care/harm. To capture this, we explicitly model the dependency between an entity, its moral role, and the moral foundation.
$$
r _ {5}: \text {M e n t i o n s} \left(t _ {i}, e _ {j}\right) \wedge \text {H a s R o l e} \left(e _ {i}, r\right) \tag {3}
$$
$$
\wedge \operatorname {E n t P o l a r i t y} \left(e _ {i}, p\right) \Rightarrow \operatorname {H a s M f} \left(t _ {i}, m\right)
$$
Dependency between stances and moral foundations As we showed in Section 4, there is a significant correlation between the stance of a tweet with
respect to the vaccine debate, and its moral foundation. For example, people who oppose the vaccine are more likely to express the liberty/oppression moral foundation. To capture this, we model the dependency between the stance of a tweet and its moral foundation.
$$
r _ {6}: \operatorname {V a x S t a n c e} \left(t _ {i}, s\right) \Rightarrow \operatorname {H a s M f} \left(t _ {i}, m\right) \tag {4}
$$
Dependency between reasons and moral foundations/stances Explicitly modeling the dependency between repeating reasons and other decisions can help us add inductive bias into our model, potentially simplifying the task. For example, we can enforce the difference between two opposing views that use similar wording, and that could otherwise be treated similarly by a text-based model (e.g. "natural methods of protection against the disease are better than vaccines" vs. "vaccines are better than natural methods of protection against the disease"). We add two rules to capture this dependency, one between reasons and moral foundations, and one between reasons and stances.
$$
\begin{array}{l} r _ {7}: \text {M e n t i o n s} \left(t _ {i}, r\right) \Rightarrow \text {H a s M f} \left(t _ {i}, m\right) \\ r _ {8}: \text {M e n t i o n s} \left(t _ {i}, r\right) \Rightarrow \text {V a x S t a n c e} \left(t _ {i}, s\right) \tag {5} \\ \end{array}
$$
Hard constraints To enforce consistency between different decisions, we add two unweighted rules (or hard constraints). These rules are not associated with a scoring function and must always hold true. We enforce that, if a tweet is predicted to be moral, then it needs to also be associated to a specific moral foundation. Likewise, if a tweet is not moral, then no moral foundation should be assigned to it.
$$
\begin{array}{l} c _ {0}: \operatorname {I s M o r a l} \left(t _ {i}\right) \Rightarrow \neg \operatorname {H a s M f} \left(t _ {i}, \text {n o n e}\right) \\ \begin{array}{l}c _ {0}: \neg \text {I s M o r a l} (t _ {i}) \rightarrow \text {H a s M f} (t _ {i}, \text {n o n e})\\c _ {1}: \neg \text {I s M o r a l} (t _ {i}) \Rightarrow \text {H a s M f} (t _ {i}, \text {n o n e})\end{array}\tag {6} \\ \end{array}
$$
Whenever different tweets have the same stance, we include a constraint to enforce consistency between the polarity of different mentions of the same entity. Roy et al. (2021) showed that enforcing consistency for mentions of the same entity within a political party was beneficial. Given the polarization of the Covid-19 vaccine, we use the same rationale.
$$
\begin{array}{l} c _ {3}: \text {M e n t i o n s} \left(t _ {i}, e _ {i}\right) \wedge \text {M e n t i o n s} \left(t _ {j}, e _ {j}\right) \\ \wedge \text {S a m e S t a n c e} \left(t _ {i}, t _ {j}\right) \wedge \text {E n t P o l a r i t y} \left(e _ {i}, p\right) \\ \Rightarrow \operatorname {E n t P o l a r i t y} \left(\mathrm {e} _ {\mathrm {j}}, \mathrm {p}\right) \tag {7} \\ \end{array}
$$
Learning and inference The weights for each rule  $w_{r} : p_{0} \wedge p_{1} \wedge \ldots \wedge p_{n} \Rightarrow h$  measure the importance of each rule in the model and can be learned from data. For example, when attempting to predict care/harm for a tweet  $t_{i}$ , we would like the weight of rule instance  $\text{IsTweet}(t_{i}) \Rightarrow \text{HasMf}(t_{i}, \text{care/harm})$  to be greater than the weight of rule instance  $\text{IsTweet}(t_{i}) \Rightarrow \text{HasMf}(t_{i}, \text{loyalty/betrayal})$ . In DRaiL, these weights are learned using neural networks with parameters  $\theta_{r}$ . The collection of rules represents the global decision, and the solution is obtained by running a maximum a posteriori (MAP) inference procedure. Given that horn clauses can be expressed as linear inequalities corresponding to their disjunctive form, the MAP inference problem can be written as a linear program. DRaiL supports both locally and globally normalized structured prediction objectives. Throughout this paper, we used the locally normalized objective. For details about the learning procedure, we refer the reader to the original paper (Pacheco and Goldwasser, 2021).
Learning with low-supervision To learn DRaiL models in the low-supervision setting, we use an Expectation-Maximization style protocol, outlined in Algorithm 1. First, we initialize the parameters of base rules using distant supervision classifiers. For moral foundations, we use the Johnson and Goldwasser (2018) dataset and the Moral Foundation Twitter Corpus (Hoover et al., 2020). For roles, we use the Roy et al. (2021) dataset. For polarity, we combine the Roy et al. (2021) dataset with the MPQA 3.0 entity sentiment dataset (Deng and Wiebe, 2015). For vaccination stances, we annotate our  $85\mathrm{K}$  unlabeled tweets using a set of prominent antivirus and provax hashtags. Details about these datasets are provided in Appendix A.4.
Once the base rules have been initialized using distant supervision, we turn our attention to learning DRaiL models over the Covid-19 dataset presented in Section 4. We alternate between MAP inference to obtain training labels (expectation step), and training the neural nets using these labels (maximization step). We receive an optional parameter  $k$  indicating the amount of direct supervision to be used. When  $k$  is provided,  $k\%$  of the annotated labels are seeded during inference.
The goal of our framework is to identify morality frames and opinions in tweets by modeling them
MODEL	MORAL/NM	MORAL FOUND.	ACTOR/TARGET	ENT. POLARITY	VAX STANCE
Macro	Weighted	Macro	Weighted	Macro	Weighted	Macro	Weighted	Macro	Weighted
Random	54.96	55.36	11.07	15.15	45.57	45.72	34.63	36.69	49.16	49.23
Majority Class	37.05	43.62	8.33	23.98	34.63	36.69	46.54	58.15	35.77	39.84
Lexicon Matching	58.97	60.01	25.28	35.85	-	-	-	-	-	-
Base (distant sup.)	69.77	68.88	28.79	41.27	71.94	72.05	63.88	74.30	69.46	70.35
Base (direct sup.)	68.94	69.71	35.28	42.92	84.71	84.75	72.92	84.31	66.91	67.36
+ Joint Model	80.53	81.17	53.29	62.27	84.60	84.64	71.53	83.35	72.06	72.53
Algorithm 1 Low Supervision Learning Protocol
1: Random initialization for all  $\theta_r$
2: for  $r \in$  base rules do
3:  $\theta_r\gets$  distant supervision classifier
4: end for
5: while not converged do
6:  $Y_{\mathrm{gold}} \gets \mathrm{DRaiL\_MAP\_inference(k)}$
7: Train all rules locally using  $Y_{\mathrm{gold}}$
8: end while
jointly. In this section, we perform an exhaustive experimental analysis to evaluate the performance of our model and each of its components.
Experimental settings In DRaiL, each rule  $r$  is associated with a neural architecture, which serves as a scoring function to obtain the rule weight  $w_{r}$ . We use BERT-base-uncased (Devlin et al., 2018) for all classifiers. For the rules that model dependencies (Eqs. 3, 4, 5), we concatenate the CLS token with a one-hot vector of the symbols on the left hand side of the rule (i.e. role, sentiment, stance and reason), before passing it through a classifier. For rules that have the entity on the left-hand side (Eqs. 2, 3), we use both the tweet and the entity as an input to BERT, using the SEP token. We trained supervised models using local normalization in DRaiL, and leveraged distant supervision using protocol outlined in Algorithm 1. In all cases, we used a learning rate of  $2e - 5$ , a maximum sequence length of 100, and AdamW. In all experiments, we perform 5-fold cross-validation over the annotated dataset and report the micro-averaged results.
General results Table 6 shows our general results for morality frames and vaccination stance. We evaluate our base classifiers and show the impact of modeling dependencies using DRaiL. The joint model results in a significant improvement for morality, moral foundation and vaccination stance.
For entities, role and polarity remain stable. We also measure the impact of explicitly modeling reasons (Eq. 5) and present results in Table 7. We show the performance for the initial reasons proposed by Wawrzuta et al. (2021), which are all from the anti-vaccine perspective, and the impact of our two rounds of interaction, expanding and refining reasons (round 1) and augmenting argumentative patterns (round 2). We find that moral foundations improve from 60.07 to 62.27 and vaccination stance improves from 67.72 to 72.53 after interaction.
Table 6: General Results (F1 Scores). NM: Non Moral
MODEL	MF	VAX. STANCE
ALL (-Reasons)	60.07	67.72
+ Reasons-Original	61.51	72.62
+ Reasons-Interaction-1	61.21	73.83
+ Reasons-Interaction-2	62.27	72.53
Table 7: Contribution of reasons at different interaction rounds (Weighted F1)
Ablation study We show an ablation study in Table 8. First, we can see how all dependencies contribute to the performance improvement, role-MF being the most impactful. We can also see that explicitly modeling morality constraints improves both the morality prediction and the moral foundation prediction, suggesting an advantage to breaking down this decision. We observe that the stance-polarity constraint does not have a significant impact, but does not hurt performance either, suggesting that our classifiers already capture this information. Lastly, we can see that the performance for roles and polarity remains stable, potentially because these classifiers have a strong starting point.
Distant supervision In Figure 4 we evaluate the impact of our indirect supervision protocol by slowly augmenting the amount of direct supervision available. We can see that by leveraging out of domain-data and dependencies, we can obtain a
MODEL	M/NM	MF	ACT/TAR	POLAR.
BERT	69.71	42.92	84.75	84.31
+RoleMF	69.71	55.54	84.64	84.13
+RoleMF+MC	79.00	57.68	84.64	84.13
+StanceMF	69.71	47.85	84.75	84.31
+StanceMF+MC	72.37	48.63	84.75	84.31
+StanceMF+MC+SPC	72.32	48.63	84.75	84.35
+ReasonMF	69.71	53.15	84.75	84.31
+ReasonMF+MC	72.60	53.41	84.75	84.31
+ReasonStance+SPC	69.71	42.92	84.64	83.26
+ALL	81.17	62.27	84.64	83.26
Table 8: Ablation Study (Weighted F1). MC: Morality Constraint, SPC: Stance-Polarity Constraint
Figure 4: Performance in low-supervision settings
competitive model using just  $25\%$  of the annotated labels, and we can outperform the fully supervised classifiers using  $50\%$  of the annotations.
Recent studies have noted the prevalence of rumors and misinformation in the context of the Covid-19 pandemic (Loomba et al., 2021; Shahi et al., 2021; Lazarus et al., 2021; Ahmed et al., 2020). Following this trend, several computational approaches have been proposed to detect misinformation related to Covid in news outlets and social media (Weinzierl and Harabagiu, 2021; Bang et al., 2021; Serrano et al., 2020; Al-Rakhami and Al-Amri, 2020). In this paper, we take a different approach and look at the problem of identifying opinions surrounding the Covid-19 vaccine, and explicitly modeling the rationale and moral sentiment that motivates them.
Some recent works also look at analyzing arguments about Covid and vaccine hesitancy more broadly. In most cases, they either take a traditional classification approach for predicting stances (Al-liheibi et al., 2021; Lyu et al., 2021), or use topic modeling techniques to uncover trends in word usage (Skeppstedt et al., 2018; Lyu et al., 2021; Sha et al., 2020; Zamani et al., 2020a). In contrast, we
propose a holistic framework that combines different methodological techniques, including human-in-the-loop mechanisms, classification with distant supervision, and deep relational learning to connect stance prediction, reason analysis and fine-grained entity moral sentiment analysis.
We introduce a holistic framework for analyzing social media posts about the Covid-19 vaccine. We model morality frames and opinions jointly, and show that we can obtain competitive performance. The main limitation of our work is the size of the annotated dataset studied. Annotating for morality is a difficult and costly task, as it requires significant domain expertise. This motivates the need for methods that perform well under limited supervision, and that can leverage external and unlabeled resources. We took a first step in this direction by combining a wide range of methodological strategies. Given the amount of data generated daily about Covid, there are broader opportunities for exploiting these resources than what were explored in this paper. While we provided a preliminary analysis of the correlation between stances, reasons and morality, our current work looks at leveraging this framework to analyze opinions at scale.
We also presented a first step towards interactive exploration of opinions on social media. While we explored this approach in a limited scenario, there is a lot of potential for using this paradigm for diagnosing NLP models and adapting to new domains. More research is required to devise protocols and evaluation strategies for this process.
We gratefully acknowledge our in-house annotators for their work annotating the dataset presented in this paper. We also thank the anonymous reviewers of this paper for their insightful comments. This project was partially funded by a Microsoft Research Dissertation Grant, a Purdue Graduate School Summer Research Grant and an NSF CAREER award IIS-2048001.
Wasim Ahmed, Josep Vidal-Alaball, Joseph Downing, Francesc López Segui, et al. 2020. Covid-19 and the 5g conspiracy theory: social network analysis of twitter data. Journal of medical internet research, 22(5):e19458.
Mabrook S Al-Rakhami and Atif M Al-Amri. 2020. Lies kill, facts save: detecting Covid-19 misinformation in twitter. *IEEE Access*, 8:155961–155970.
Firoj Alam, Fahim Dalvi, Shaden Shaar, Nadir Durrani, Hamdy Mubarak, Alex Nikolov, Giovanni Da San Martino, Ahmed Abdelali, Hassan Sajjad, Kareem Darwish, et al. 2021. Fighting the pandemic in social media: A holistic perspective and a call to arms. In Proceedings of the International AAAI Conference on Web and Social Media, volume 15, pages 913-922.
Fahad M Alliheibi, Abdulfattah Omar, and Nasser Al-Horais. 2021. Opinion mining of saudi responses to Covid-19 vaccines on twitter. International Journal of Advanced Computer Science and Applications, 12(6):72-78.
Yejin Bang, Etsuko Ishii, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. 2021. Model generalization on Covid-19 fake news detection. arXiv preprint arXiv:2101.03841.
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022.
Virginia Braun and Victoria Clarke. 2012. Thematic analysis., pages 57-71.
Eugene Y Chan. 2021. Moral foundations underlying behavioral compliance during the COVID-19 pandemic. *Personality and individual differences*, 171:110463.
Lingjia Deng and Janyce Wiebe. 2015. MPQA 3.0: An entity/event-level sentiment corpus. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1323-1328, Denver, Colorado. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Rodrigo Diaz and Florian Cova. 2021. Reactance, morality, and disgust: The relationship between affective dispositions and compliance with official health recommendations during the COVID-19 pandemic. Cognition and Emotion, pages 1-17.
Kyle Glandt, Sarthak Khanal, Yingjie Li, Doina Caragea, and Cornelia Caragea. 2021. Stance detection in COVID-19 tweets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1596–1611, Online. Association for Computational Linguistics.
Jesse Graham, Jonathan Haidt, and Brian A Nosek. 2009. Liberals and conservatives rely on different sets of moral foundations. Journal of personality and social psychology, 96(5):1029.
Jonathan Haidt and Jesse Graham. 2007. When morality opposes justice: Conservatives have moral intuitions that liberals may not recognize. *Social Justice Research*, 20(1):98-116.
Kazi Saidul Hasan and Vincent Ng. 2014. Why are you taking this stance? identifying and classifying reasons in ideological debates. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 751-762, Doha, Qatar. Association for Computational Linguistics.
J. Hoover, G. Portillo-Wightman, L. Yeh, S. Havaldar, A.M. Davani, Y. Lin, B. Kennedy, M. Atari, Z. Kamel, M. Mendlen, G. Moreno, C. Park, T.E. Chang, J. Chin, C. Leong, J.Y. Leung, A. Mirinjian, and M. Dehghani. 2020. Moral foundations twitter corpus: A collection of 35k tweets annotated for moral sentiment. Social Psychological and Personality Science, 11(8):1057-1071.
Tamanna Hossain, Robert L. Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh. 2020. COVID Lies: Detecting COVID-19 misinformation on social media. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online. Association for Computational Linguistics.
Kristen Johnson and Dan Goldwasser. 2018. Classification of moral foundations in microblog political discourse. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 720-730, Melbourne, Australia. Association for Computational Linguistics.
Klaus Krippendorff. 2004. Measuring the reliability of qualitative text analysis data. *Quality and quantity*, 38:787-800.
Jeffrey V Lazarus, Scott C Ratzan, Adam Palayew, Lawrence O Gostin, Heidi J Larson, Kenneth Rabin, Spencer Kimball, and Ayman El-Mohandes. 2021. A global survey of potential acceptance of a Covid-19 vaccine. Nature medicine, 27(2):225-228.
Sahil Loomba, Alexandre de Figueiredo, Simon J Piatek, Kristen de Graaf, and Heidi J Larson. 2021. Measuring the impact of Covid-19 vaccine misinformation on vaccination intent in the uk and usa. Nature human behaviour, 5(3):337-348.
Joanne Chen Lyu, Eileen Le Han, and Garving K Luli. 2021. Covid-19 vaccine-related discussion on twitter: topic modeling and sentiment analysis. Journal of medical Internet research, 23(6):e24435.
Ilaria Montagni, Kevin Ouazzani-Touhami, A Mebarki, N Texier, S Schuck, Christophe Tzourio, et al. 2021. Acceptance of a Covid-19 vaccine is associated with ability to detect fake news and health literacy. Journal of public health (Oxford, England).
Goran Muric, Yusong Wu, and Emilio Ferrara. 2021. COVID-19 Vaccine Hesitancy on Social Media: Building a Public Twitter Dataset of Anti-vaccine Content, Vaccine Misinformation and Conspiracies.
Maria Leonor Pacheco and Dan Goldwasser. 2021. Modeling content and context with deep relational learning. Transactions of the Association for Computational Linguistics, 9:100-119.
Stefano Pagliaro, Simona Sacchi, Maria Giuseppina Pacilli, Marco Brambilla, Francesca Lionetti, Karim Bettache, Mauro Bianchi, Marco Biella, Virginie Bonnot, Mihaela Boza, et al. 2021. Trust predicts Covid-19 prescribed and discretionary behavioral intentions in 23 countries. PloS one, 16(3):e0248334.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Peter Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math., 20(1):53-65.
Shamik Roy, Maria Leonor Pacheco, and Dan Goldwasser. 2021. Identifying morality frames in political tweets using relational learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9939-9958, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Juan Carlos Medina Serrano, Orestis Papakyriakopoulos, and Simon Hegelich. 2020. Nlp-based feature extraction for the detection of Covid-19 misinformation videos on youtube. In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020.
Hao Sha, Mohammad Al Hasan, George Mohler, and P Jeffrey Brantingham. 2020. Dynamic topic modeling of the Covid-19 twitter narrative among us governors and cabinet executives. arXiv preprint arXiv:2004.11692.
Gautam Kishore Shahi, Anne Dirkson, and Tim A Majchrzak. 2021. An exploratory study of COVID-19 misinformation on twitter. Online social networks and media, 22:100104.
Maria Skeppstedt, Andreas Kerren, and Manfred Stede. 2018. Vaccine hesitancy in discussion forums: computer-assisted argument mining with topic models. In *Building Continents of Knowledge in Oceans of Data: The Future of Co-Created eHealth*, pages 366–370. IOS Press.
Paweł Sowa, Łukasz Kiszkiel, Piotr Paweł Laskowski, Maciej Alimowski, Łukasz Szczerbiński, Marlena Paniczko, Anna Moniuszko-Malinowska, and Karol Kamiński. 2021. Covid-19 vaccine hesitancy in