SlowGuess/NAACL_2021_OCR huggingface dataset:
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37-46.
Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, and Nigel Collier. 2020. Will-they-won't-they: A very large dataset for stance detection on Twitter. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1715-1724, Online. Association for Computational Linguistics.
Kareem Darwish, Walid Magdy, and Tahar Zanouda. 2017. Improved stance prediction in a user similarity feature space. In Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017, ASONAM '17, page 145-148, New York, NY, USA. Association for Computing Machinery.
Kareem Darwish, Peter Stefanov, Michael Aupetit, and Preslav Nakov. 2019. Unsupervised user stance detection on twitter.
Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. SemEval-2017 task 8: RumourEval: Determining rumour veracity and support for rumours. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 69-76, Vancouver, Canada. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Kuntal Dey, Ritvik Shrivastava, and Saroj Kaushik. 2018. Topical stance detection for twitter: A two-phase LSTM model using attention. In Advances in Information Retrieval, pages 529-536, Cham. Springer International Publishing.
Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017. Stance classification with target-specific neural attention. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 3988-3994.
Chris Dulhanty, Jason L. Deglint, Ibrahim Ben Daya, and Alexander Wong. 2019. Taking a stance on fake news: Towards automatic disinformation assessment via deep bidirectional transformer language models for stance detection.
Martin Fajcik, Pavel Smrz, and Lukas Burget. 2019. BUT-FIT at SemEval-2019 task 7: Determining the rumour stance with pre-trained deep bidirectional
transformers. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 1097-1104, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1163-1168, San Diego, California. Association for Computational Linguistics.
Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1161-1166, Hong Kong, China. Association for Computational Linguistics.
Shalmoli Ghosh, Prajwal Singhania, Siddharth Singh, Koustav Rudra, and Saptarshi Ghosh. 2019. Stance detection in web and social media: A comparative study. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 75-87, Cham. Springer International Publishing.
Genevieve Gorrell, Elena Kochkina, Maria Liakata, Ahmet Aker, Arkaitz Zubiaga, Kalina Bontcheva, and Leon Derczynski. 2019. SemEval-2019 task 7: RumourEval, determining rumour veracity and support for rumours. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 845–854, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6325-6334.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735-1780.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Dilek Kucuk and Fazli Can. 2020. Stance detection: A survey. ACM Comput. Surv., 53(1).
Dilek Küçük and Fazli Can. 2018. Stance detection on tweets: Ansvm-based approach.
Mirko Lai, Viviana Patti, Giancarlo Ruffo, and Paolo Rosso. 2018. Stance Evolution and Twitter Interactions in an Italian Political Debate. In Natural Language Processing and Information Systems, Lecture Notes in Computer Science, pages 15-27, Cham. Springer International Publishing.
Jieh-Sheng Lee and Jieh Hsiang. 2019. Patentbert: Patent classification with fine-tuning a pre-trained bert model.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pretrained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Zhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li, and Jun Zhao. 2020. Finbert: A pre-trained financial language representation model for financial text mining. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 4513-4519. International Joint Conferences on Artificial Intelligence Organization. Special Track on AI in FinTech.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016a. A dataset for detecting stance in tweets. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 3945-3952, Portoorž, Slovenia. European Language Resources Association (ELRA).
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016b. SemEval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31-41, San Diego, California. Association for Computational Linguistics.
Jonas Mueller and Aditya Thyagarajan. 2016. Siamese recurrent architectures for learning sentence similarity. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, page 2786-2792. AAAI Press.
Martin Müller, Marcel Salathe, and Per E Kummervold. 2020. Covid-twitter-bert: A natural language processing model to analyse COVID-19 content on twitter.
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. Bertweet: A pre-trained language model for english tweets.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 115-124, Ann Arbor, Michigan. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8026-8037. Curran Associates, Inc.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Dean Pomerleau and Delip Rao. 2017. Fake news challenge.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902-4912. Association for Computational Linguistics.
T. Y.S.S. Santosh, Srijan Bansal, and Avirup Saha. 2019. Can siamese networks help in stance detection? In Proceedings of the ACM India Joint International Conference on Data Science and Management of Data, CoDS-COMAD '19, page 306-309,
New York, NY, USA. Association for Computing Machinery.
Benjamin Schiller, Johannes Daxenberger, and Iryna Gurevych. 2020. Stance detection benchmark: How robust is your stance detection?
Parinaz Sobhani, Diana Inkpen, and Xiaodan Zhu. 2017. A dataset for multi-target stance detection. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 551-557, Valencia, Spain. Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Jannis Vamvas and Rico Sennrich. 2020. X-stance: A multilingual multi-target dataset for stance detection.
Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 18-22.
Z. Wang, Q. Wang, C. Lv, X. Cao, and G. Fu. 2020. Unseen target stance detection with adversarial domain generalization. In 2020 International Joint Conference on Neural Networks (IJCNN).
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32, pages 5753-5763. Curran Associates, Inc.
Shaodian Zhang, Lin Qiu, Frank Chen, Weinan Zhang, Yong Yu, and Noémie Elhadad. 2017. "We make choices we think are going to save us": Debate and stance identification for online breast cancer CAM discussions. Proceedings of the ... International World-Wide Web Conference. International WWW Conference, 2017:1073-1081.
Elena Zotova, Rodrigo Agerri, Manuel Nunez, and German Rigau. 2020. Multilingual stance detection: The catalonia independence corpus.
We release our code and pre-training for section §2 at this url- https://github.com/ Ayushk4/bias-stance. Our dataset and baselines for section §3 have been released this url - https://github.com/Ayushk4/ stance-dataset. The Readme for the respective repositories contain instructions to set up, environment, replicating the codebase and the links to pre-trained models.
In this appendix, we first discuss the baselines  $\S \mathbf{B}$ . Followed by our experimental setup  $\S \mathbf{C}$  and the datasets considered  $\S \mathbf{D}$
- Bert for Stance Detection (Ghosh et al., 2019) is shown in Figure 3a. It takes both tweet and target sentence as input separated by [SEP] token. The final [CLS] token representation is used for stance classification. It delivered state of the art performance across two datasets.
- Target oblivious Bert is shown in Figure 3b. It takes only the tweet sentence as input enclose within [CLS] and [SEP] tokens. The final [CLS] token representation is used for stance classification.
- SiamNet architecture (Santosh et al., 2019) is shown in Figure 4a. It uses siamese networks (Bromley et al., 1993) to learn representations each for tweet and targets and classify using the bottleneck of a single scalar being the similarity function. Similar to the WT-WT paper (Conforti et al., 2020) we find that the similarity function output scalar alone isn't strong enough feature for a classifier. Hence we concatenate the tweet and target representation vectors of the similarity function (inverse exponential of Manhattan distance) following (Mueller and Thyagarajan, 2016). We replaced the Glove and BiLstm with Bert embedding, where we obtain the tweet and target representations we taking the '[CLS]' vector representations from bert for those sentences.
- TAN (Du et al., 2017) is shown in Figure 4b. It uses a target-specific attention extraction over the tweet features obtained from BiLstm similar to (Dey et al., 2018). We use the same
(a) Bert Stance Detection
(b) Target oblivious Bert
Figure 3: Bert Models
approach as them except that we substitute the LSTM features for better features from Bert.
All our experiments were performed using Pytorch (Paszke et al., 2019), wandb (Biewald, 2020) and Huggingface (Wolf et al., 2019). The optimization algorithm used was the Adam optimizer (Kingma and Ba, 2014). We keep Bert layers and embedding trainable. In case of SiamNet and TAN, the Bert parameters are hard-shared. Experiments takes less than 10 minute per epoch and less than 5 GB GPU memory on a Tesla P100 GPU. The total model parameters for the Bert models are the same as the Bert (including being approximately same for SiamNet and TAN). Following the previous works demonstrating that domain-specific weights
(a) SiamNet Model
(b) TAN Model
Figure 4: SiamNet and TAN Baselines
result in improved performance (Gururangan et al., 2020; Lee et al., 2019; Beltagy et al., 2019; Lee and Hsiang, 2019; Liu et al., 2020; Müller et al., 2020), we use the Bert weights fine tuned on tweets (Nguyen et al., 2020) with the exception of the target aware Bert in tWT-WT where it was found to be unstable. So, we used bert-base-cased instead.
C.1 Hyperparameters
We use the Huggingface's Bert default config for our experiments. For the experiments, we tuned the learning rate from 5 values -  $\{1\mathrm{e} - 6,3\mathrm{e} - 6,1\mathrm{e} - 5,3\mathrm{e} - 5,1\mathrm{e} - 4\}$  and number of epochs from 3 values -  $\{2, 5, 10\}$  on the development set. The batch-size was fixed to 16. For the datasets with no development split, we use 5-fold cross validation. We evaluated and trained our model in the same settings as proposed for their respective datasets. Table 6 lists the hyper-parameters.
All Models
Batch size	16
Num epochs	2, 5, 10
Optimizer	Adam
Bert dropout	0.1
Max tokens	99
Classifier dropout	0.1
Bert trainable	Yes
Learning rate	{1e-6, 3e-6, 1e-5, 3e-5, 1e-4}
5-fold cross-valid	Macro-F1
SiamNet
Final mlp hidden	786
Distance metric	Inverse exponential of Manhattan distance
TAN
Final mlp hidden	786
Table 6: Hyperparameters for Bert.
C.2 Preprocessing
We use ekphrasis library (Baziotis et al., 2017) to perform the preprocessing. We do word tokenization and spelling correction. We also remove URLs, Emoji, non-ascii characters and do normalization to limit the length of inputs. For the RumourEval2019 dataset we trimmed the input to 99 tokens, since Reddit text can even cross 500 characters length.
For the datasets that released only the tweet ids, we obtain the tweet text using the Twitter API. However, some tweets are not accessible over time as accounts or tweets get banned/blocked/deleted etc. Table lists the full statistics 7 for the datasets.
- Will-They-Won't-They (WT-WT)
Dataset: (Conforti et al., 2020) is a cross-target stance detection dataset. It has 50k tweet-target pairs from financial domain. It has 5 targets, each a fixed Merger and Acquisition (M&A) operation. Four of the five M&A targets are from health domain - {CVS-Aetna, Cigna-Esrx, Anthem-Cigna, Aetna-Humana} and one from entertainment - {Disney-Fox} serving as an out-of-domain target. For the experimental settings, the model is trained on three health mergers and tested on the fourth. For out of domain
<sup>3</sup>https://developer/twitter.com/
Dataset	Download Link
WT-WT	https://github.com/cambridge-wtwt/acl2020-wtwt-tweets
SE16	https://alt.qcri.org/semeval2016/task6/
M-T	https://www.site.uottawa.ca/~diana/resources/stance_data/
RE17	https://alt.qcri.org/semeval2017/task8/
RE19	https://competitions.codalab.org/competitions/19938
Encryption	https://github.com/aseelad/The-Encryption Debate
Table 7: Links to download the datasets.
merger, the model is trained on four health mergers and tested on the entertainment domain. Its stance label set is - {Support, Refute, Comment, Unrelated}. As part of pre-processing, we normalize the health merger company names /acronyms, for example 'Anthem Inc'  $\rightarrow$  'Anthem', 'Antm'  $\rightarrow$  'Anthem'.
- SemEval 2016 Task 6-A (Mohammad et al., 2016b) is based on a stance detection dataset (Mohammad et al., 2016a) of 4163 tweets. The targets are fixed entities (politicians, movements, policy etc.). Specifically, these are from the set - {Atheism, Climate-Change, Hillary Clinton, Feminism, Legalizing Abortion}. Each of the tweet-target pair is labelled for one of the 3 stance from {Against, None, Favor}. The dataset has somewhat balanced class distribution and the metrics considered for this were Accuracy and  $F_{1}$ . In this task, the models were evaluated on same targets as they were trained.
- Multi-target (M-T) stance dataset (Sobhani et al., 2017) contains 4455 tweets from political domain. Target was a fixed entity pair containing two political entities - Hillary-Sanders, Hillary-Trump, Cruz-Trump. This tweet-target pair has two stances - one for each target from - {Against, None, Favor}. Thus, in pairs of two, it leads to 9 total possible combinations of the 3 labels. We treat the pair as two separate problems, and trained two separate models for it. The dataset has somewhat balanced class distribution and the models were evaluated on the same targets (pairs) as they were trained using the Accuracy and  $F_{1}$  metrics.
- RumourEval 2017 (Derczynski et al., 2017), was a rumour-stance detection task that proposed a new dataset for the task. The dataset consisted of 285 rumoured tweet threads with
a total of 4519 tweets. The root node of each thread was the rumour target, for which users replied and created a response thread exhibiting a tree structure. The tweet-targets pairs were labelled from one of the four stance classes being - {Support, Query, Comment, Deny}. The dataset has a very skewed distribution with the majority class (Comment) having about  $80\%$  examples. So, Macro-Averaged  $F_{1}$  score is a suitable metric. Here the models were evaluated on different threads (and hence different targets) as they were trained.
- RumourEval 2019 (Gorrell et al., 2019) was similar to the RumourEval 2017 task. It extended the dataset to include Reddit threads from selected sub-reddits. The resulting dataset has a total of 8574 datapoints. The tweet-targets pairs were labelled from the same labelset from one of the four stance classes being - {Support, Query, Comment, Deny}. This dataset has a very skewed distribution with the majority class (Comment) having about  $80\%$  examples. So, Macro-Averaged  $F_{1}$  score is a suitable metric for the dataset.
- Encryption Debate dataset (Addawood et al., 2017) consists of 2999 tweets labelled for three stances - {For, Against, Neutral} on one encryption debate topic. We observed repeated entries in the dataset, including some having conflicting labels for the same tweet-target pair; we excluded such tweets for our experiments. Additionally, only 5 tweets from the dataset belonged to the 'against' class. Since 5 examples is a very small number for most machine learning models to learn, we exclude this class for our analysis. The dataset has a very skewed distribution with the majority class (neutral) having about  $86\%$  examples. So, Macro-Averaged  $F_{1}$  score is a suitable metric. The dataset has only one target for training and evaluating the models.