SlowGuess/NAACL_2021_OCR huggingface dataset:
In masked language modeling, we fine-tune the model using a neural layer on top with the learning objective to predict masked tokens. In this step, we substitute that layer with a new neural layer as a stance classifier layer. Its weights are arbitrarily initialized. The prediction equation is similar to Equation 4 but now the input is not corrupted, and the output is a vector of the normalized probability of the three stance classes. We use a cross-entropy loss function and the objective is to minimize it. We use the Adam optimizer (Kingma and Ba, 2015) with five different learning rates, including  $2e - 5$ ,  $1e - 5$ ,  $5e - 6$ ,  $2e - 6$  and  $1e - 6$ . The batch
size is constantly set to 32 during the classification learning process.
We train and test our models on each candidate independently with five different learning rates. The best model is determined by the best macro average F1 score over three classes among five learning rates. Because the weights of the classifier layer are randomly initialized, we run each model five times. The average F1 score is reported in Table 2 as the classification performance.
After describing our data set (Section 5.1), we present our experimental evaluation, both quantitative (Section 5.2), and qualitative (Section 5.3).
5.1 Data Sets
For this study, our research team collected English tweets related to the 2020 US Presidential election. Through the Twitter Streaming API, we collected data using election-related hashtags and keywords. Between January 2020 and September 2020, we collected over 5 million tweets, not including quotes and retweets. These unlabeled tweets were used to fine-tune all of our language models.
Our specific stance task is to determine the stance for the two presidential candidates, Joe Biden and Donald Trump. For each candidate, we had three stance classes: support, opposition, and neutral. We consider two stance-labeled data sets, one for each candidate, Biden and Trump. Our data were labeled using Amazon Mechanical Turk (MTurk) workers (Crowston, 2012). These workers were not trained. Instead, we provided a set of examples for each stance category that they could refer to as they conducted the labeling task. Examples of statements presented to MTurk workers are presented in Table 3. We asked annotators to carefully review each tweet  $t_c^i$  from the tweet set  $T_{C} = \{t_{c}^{1},t_{c}^{2},\ldots \}$  and determine whether the tweet  $t_c^i$  is (i) clearly in support of  $C$ , (ii) clearly in opposition to  $C$  or (iii) not clearly in support or opposition to  $C$ , where  $t_c^i \in T_C$  and  $C \in \{\text{Donald Trump}, \text{Joe Biden}\}$ . To increase the labeling yield, we verify that two tweet sets  $T_{C=Donald Trump}$  and  $T_{C=Joe Biden}$  are mutually exclusive. Each tweet was labeled by three annotators and the majority vote is considered to be the true label. If all three annotators vote for three differ
<sup>6</sup>Our definition of stance labels is consistent with the definition from (Mohammad et al., 2016a)
Table 2: The average F1 scores over five runs. The confidence intervals for the macro F1 scores are computed based on a significance level of 0.05, meaning a  $95\%$  confidence level. The highest scores are shown in boldface.
Model	Biden	Trump
F1-Support	F1-Oppose	F1-Neutral	F1-macro	F1-Support	F1-Oppose	F1-Neutral	F1-macro
o-BERT	0.7324	0.6875	0.7151	0.7117 (±0.0063)	0.7574	0.8101	0.6955	0.7543 (±0.0069)
f-BERT	0.7743	0.7226	0.7347	0.7439 (±0.0049)	0.7921	0.8147	0.6961	0.7677 (±0.0084)
a-BERT	0.7905	0.7234	0.7432	0.7523 (±0.0049)	0.8090	0.8154	0.6926	0.7724 (±0.0078)
SKEP	0.7923	0.7153	0.7349	0.7475 (±0.0047)	0.7852	0.8169	0.7151	0.7724 (±0.0067)
KE-MLM-	0.7618	0.7303	0.7380	0.7434 (±0.0040)	0.7854	0.7968	0.7083	0.7635 (±0.0081)
KE-MLM	0.7927	0.7329	0.7475	0.7577 (±0.0032)	0.8094	0.8184	0.7354	0.7877 (±0.0075)
Table 3: Sample of stance examples presented to MTurk labelers.
Candidate	Statement	Stance
Biden	Biden will be a great president. I am voting for him in November.	Support
Biden has handled the pandemic poorly.	Oppose
Biden spoke in Pennsylvania.	Neutral
Trump	Trump has been a great president. I am voting for him in November.	Support
Trump has handled the pandemic poorly.	Oppose
Trump held a rally yesterday.	Neutral
ent classes, we assume the tweet's label is neutral because the stance is ambiguous.
Our data set contains 1250 stance-labeled tweets for each candidate. The stance label distributions are shown in Table 4. The distributions of both candidates are skewed towards the opposition label. Overall, the stance class proportions vary from  $27\%$  to  $39\%$ . The inter-annotator agreement scores from different metrics are shown in Table 5. The task-based and worker-based metrics are recommended by the MTurk official site (Amazon, 2011), given their annotating mechanism. All scores are range from  $86\%$  up to  $89\%$ , indicating the high inter-rater reliability for these data sets.
Table 4: Stance distribution for Biden and Trump.
%SUPPORT	%OPPOSE	%NEUTRAL
Biden	31.3	39.0	29.8
Trump	27.3	39.9	32.8
Table 5: Mechanical Turk inter-annotator agreement for Biden and Trump.
Metric	Biden	Trump
Task-based	0.8693	0.8920
Worker-based	0.8915	0.8969
We conducted experiments on train-test sets using a 70:30 split for both the Biden and Trump data
sets. $^{7}$  We evaluate the classification performance using the macro-average F1 score along with the F1 score of each class. The results presented in Table 2 show the average F1 scores over five runs with different random seeds. The highest score for each evaluation metric is highlighted in bold.
For Biden-stance, every fine-tuning method (f-BERT, a-BERT, SKEP, KE-MLM- and KE-MLM) improves the average F1 score from the original pre-trained model by  $3.2\%$ ,  $4.1\%$ ,  $3.6\%$ ,  $3.2\%$  and  $4.6\%$ , respectively. For Trump-stance, the average F1 scores are also improved by  $1.3\%$ ,  $1.8\%$ ,  $1.8\%$ ,  $0.9\%$  and  $3.3\%$ . The improvement is twice as much for Biden than for Trump. This is an indication that the additional background knowledge is more important for detecting stance for Biden than for Trump. In general, our knowledge enhanced model performs better than all the other models and outperforms the original BERT by three to five percent. a-BERT performs similarly to SKEP for Trump, but its performance is better for Biden. The model's overall performances are second-best with only a difference of  $0.5\%$  and  $1.5\%$  in the average F1-macro score when compared to KE-MLM for Biden and Trump, respectively. These results further highlight the importance of incorporating stance tokens into the tokenizer. While adding stance to the tokenization is important, the additional improvement of KE-MLM comes from focusing attention on both the stance tokens and the general election data. The result also supports our hypothesis that training KE-MLM- alone for two epochs would result in better accuracy than original BERT (o-BERT), but a lower accuracy than normally fine-tuned BERT (f-BERT) because it learns stance knowledge but lacks in-domain election knowledge.
To better understand the robustness of our models, we analyze the variance in the F1 scores across
<sup>7</sup>Because we do not have sufficient unlabeled election data from 2016, we cannot fairly test our model with the SemEval 2016 stance data.
the different runs. Figure 1 shows the box plots of the macro average F1 scores for each model. The scores of both candidates follow a similar pattern. For Biden, the highest F1 score and the lowest variance is KE-MLM. For Trump, the highest F1 score is KE-MLM, but the variance is comparable to the other models. The model with the lowest variance is SKEP. These figures further emphasize KE-MLM's ability to detect stance better than normally fine-tuning methods. Interestingly, a-BERT performs second-best (see gray boxes in Figure 1), further highlighting the importance of not ignoring stance tokens. Forcefully adding unseen stance tokens to the BERT tokenizer with random initial weights benefits overall classification performance.
(a) Biden
(b) Trump
Figure 1: The distribution of macro average F1 scores from five independent runs.
Additionally, we conducted a sensitivity analysis on different sizes of unlabeled data for pretraining to verify that the large unlabeled data is actually beneficial. We fine-tune f-BERT using different sizes of data (100K, 500K, 1M, 2M) and compare the results to those of BERT with zero-fine-tuning (o-BERT) and fine-tuning using the entire 5M tweets (f-BERT). We train each pre-trained language model on training and test on testing data
set five times. The average F1 scores are shown in Fig 2. For Biden, the average F1 score is  $3\%$  lower when there is no fine-tuning compared to using all 5M tweets. For Trump, the score only improves a little over  $1\%$ . Interestingly, as the size of the unlabeled data increases, the F1 score also increases even though the increase is not always large. Therefore, pre-training using a smaller size unlabeled data set does still produce benefits, but when possible, using a large sample does lead to improvement.
Figure 2: The model performance by f-BERT pretrained on different sizes of unlabeled data. We train each model five times and report the average F1 scores.
5.3 Qualitative Analysis of the Effect of Stance Knowledge
While we see from Table 2 that KE-MLM outperforms all baselines on average, we are interested in understanding when there is labeling disagreement between other methods and KE-MLM, what features are driving the disagreement. Therefore, we manually investigate samples in which f-BERT and a-BERT produced incorrect predictions, while KE-MLM produced correct ones. On average over multiple runs,  $28.8\%$  and  $38.5\%$  of misclassified tweets by f-BERT are correctly predicted by KEMLM for Biden and Trump, respectively. For a-BERT, they are  $22.5\%$  and  $25.7\%$  on average. As a case example, Table 6 illustrates the attention distribution of the sequence representation learned by each language model for a few unlabeled tweets. Significant words are colored. The color darkness is determined by the attention weights of the representation learned for the classification token. $^{8}$  The
The representation of classification tokens produced by a transformer encoder is usually referred to as [CLS]. Please see (Devlin et al., 2019) for details about the attention weight calculation.
Table 6: Visualization of selected samples with attention weight distribution by color darkness.
Candidate	Model	Sampled Sentence	Prediction
Biden	f-BERT	The democrats and @joebiden believe in the power of the government.	Neutral
The #GOP and @realdonaldtrump believe in the power of the american people. #maga
a-BERT	The democrats and @joebiden believe in the power of the government.	Neutral
The #GOP and @realdonaldtrump believe in the power of the american people . #maga
KE-MLM	The democrats and @joebiden believe in the power of the government.	Opposition
The #GOP and @realdonaldtrump believe in the power of the american people. #maga
Trump	f-BERT	Covid -19 was Trump's biggest test. He failed miserably . #demconvention	Neutral
a-BERT	Covid -19 was Trump's biggest test. He failed miserably. #demconvention	Neutral
KE-MLM	Covid -19 was Trump's biggest test. He failed miserably . #demconvention	Opposition
darker the color the more important the word. From the selected samples, we know from the knowledge mining step that the word "maga" and "dem convention" are two of the most distinguishing stance words (see Table 1), but both f-BERT and a-BERT fail to identify these strong stance words and therefore, produced incorrect predictions. In contrast, KE-MLM produces the correct predictions by paying reasonable attention to the stance information, further supporting the notion that KE-MLM is using meaningful, interpretable tokens.
Intuitively, a language model fine-tuned using indomain unlabeled data should result in better classification performance than using the vanilla pretrained BERT. Since our goal is to maximize the accuracy of a specific classification task, we train an attention-based language model to pay attention to words that help distinguish between the classes. We have shown that for stance detection, using the log-odds-ratio to identify significant tokens that separate the classes is important knowledge for this classification task. Once these important tokens are identified, forcing the language model to pay attention to these tokens further improves the performance when compared to using standard data for fine-tuning. To the best of our knowledge, our approach is better than the other state-of-the-art approaches for stance detection. Additionally, we are releasing our data set to the community to help other researchers continue to make progress on the stance detection task. We believe this is the first stance-labeled Twitter data for the 2020 US Presidential election.
There are several future directions of this work. First, to relax the trade-off between learning election semantics in general and learning stance knowledge, instead of fine-tuning one epoch with
the normal fine-tuning method and another epoch with KE-MLM, we could reduce the masking probability of stance distinguishing words from  $100\%$  to something lower based on the distinguishability of the token. Theoretically, this would give a higher weight to words that are more polarizing. This also relaxes the potential overfitting that may occur when learning only stance knowledge and lets the model randomly learn more tokens. Another future direction is to test our language modeling method on other classification tasks (e.g. sentiment analysis, spam detection). Also, this paper uses BERT as the base language model. There are many variations of BERT that can be further investigated (e.g. RoBERTa). Finally, we view stance as an important task for understanding public opinion. As our models get stronger, using them to gain insight into public opinion on issues of the day is another important future direction.
This research was funded by National Science Foundation awards #1934925 and #1934494, and the Massive Data Institute (MDI) at Georgetown University. We would like to thank our funders, the MDI staff, and the members of the Georgetown DataLab for their support. We would also like to thank the anonymous reviewers for the detailed and thoughtful reviews.
Abeer Aldayel and Walid Magdy. 2019. Your stance is exposed! analysing possible factors for stance detection on social media. The ACM on Human-Computer Interaction, 3(CSCW):1-20.
Abeer AlDayel and Walid Magdy. 2020. Stance detection on social media: State of the art and trends. arXiv preprint arXiv:2006.03644.
Amazon. 2011. Hit review policies. https://docsAWS.amazon.com/AWSMechTurk/latest/AWSMturkBAPI/APIReference_HITReviewPolicies.html.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics (TACL), 5:135-146.
Gerlof Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. In Proceedings of the biennial Conference of the German Society for Computational Linguistics and Language Technology (GSCL).
Ceren Budak. 2019. What happened? the spread of fake news publisher content during the 2016 us presidential election. In Proceedings of the World Wide Web Conference (WWW).
Peng Chen, Zhongqian Sun, Lidong Bing, and Wei Yang. 2017. Recurrent attention network on memory for aspect sentiment analysis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Kevin Crowston. 2012. Amazon mechanical turk: A research tool for organizations and information systems scholars. In Shaping the Future of ICT Research. Methods and Approaches.
Marco Del Tredici, Diego Marcheggiani, Sabine Schulte im Walde, and Raquel Fernandez. 2019. You shall know a user by the company it keeps: Dynamic representations for social media users in NLP. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Kuntal Dey, Ritvik Shrivastava, and Saroj Kaushik. 2017. Twitter stance detection—a subjectivity and sentiment polarity inspired two-phase approach. In IEEE international conference on data mining workshops (ICDMW).
Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017. Stance classification with target-specific neural attention networks. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI).
Shalmoli Ghosh, Prajwal Singhania, Siddharth Singh, Koustav Rudra, and Saptarshi Ghosh. 2019. Stance detection in web and social media: a comparative study. Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 75-87.
Marco Guerini, Lorenzo Gatti, and Marco Turchi. 2013. Sentiment analysis: How to derive prior polarities from SentiWordNet. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. 2011. Target-dependent twitter sentiment classification. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2019. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics (TACL), 8:64-77.
Dan Jurafsky, Victor Chahuneau, Bryan R Routledge, and Noah A Smith. 2014. Narrative framing of consumer sentiment in online restaurant reviews. First Monday.
Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, and Minlie Huang. 2020. SentiLARE: Sentiment-aware language representation learning with linguistic knowledge. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR).
Dilek Kucuk and Fazli Can. 2020. Stance detection: A survey. ACM Computing Surveys (CSUR), 53(1).
Yoav Levine, Barak Lenz, Or Dagan, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2020. SenseBERT: Driving some sense into BERT. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Margaret Mitchell, Jacqui Aguilar, Theresa Wilson, and Benjamin Van Durme. 2013. Open domain targeted sentiment. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016a. A
dataset for detecting stance in tweets. In Proceedings of the International Conference on Language Resources and Evaluation (LREC).
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016b. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the International Workshop on Semantic Evaluation (SemEval).
Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. 2008. Fightin' words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372-403.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Lahari Poddar, Wynne Hsu, and Mong Li Lee. 2017. Author-aware aspect topic sentiment model to retrieve supporting opinions from reviews. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI.
Martin Schmitt, Simon Steinheber, Konrad Schreiber, and Benjamin Roth. 2018. Joint aspect and polarity classification for aspect-based sentiment analysis with end-to-end neural networks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Anirban Sen, Manjira Sinha, Sandya Mannarswamy, and Shourya Roy. 2018. Stance classification of multi-perspective consumer health information. In Proceedings of the ACM India Joint International Conference on Data Science and Management of Data.
Umme Aymun Siddiqua, Abu Nowshed Chy, and Masaki Aono. 2019. Tweet stance detection using an attention based neural ensemble model. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Parinaz Sobhani, Saif Mohammad, and Svetlana Kiritchenko. 2016. Detecting stance in tweets and analyzing its interaction with sentiment. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics (*SEM).
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. ERNIE: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223.
Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang, and Feng Wu. 2020. SKEP: Sentiment knowledge enhanced pre-training for sentiment analysis. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Rui Wang, Deyu Zhou, Mingmin Jiang, Jiasheng Si, and Yang Yang. 2019. A survey on opinion mining: From stance to product aspect. IEEE Access, 7:41101-41124.
Wan Wei, Xiao Zhang, Xuqin Liu, Wei Chen, and Tengjiao Wang. 2016. pkudblab at SemEval-2016 task 6: A specific convolutional neural network system for effective stance detection. In Proceedings of the International Workshop on Semantic Evaluation (SemEval).
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems (NeurIPS).
Guido Zarrella and Amy Marsh. 2016. Mitre at semeval-2016 task 6: Transfer learning for stance detection. In Proceedings of the International Workshop on Semantic Evaluation (SemEval).
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Jie Zhou, Junfeng Tian, Rui Wang, Yuanbin Wu, Wenming Xiao, and Liang He. 2020. SentiX: A sentiment-aware pre-trained model for cross-domain sentiment analysis. In Proceedings of the International Conference on Computational Linguistics (COLING).