SlowGuess/ACL_2020_OCR huggingface dataset:
In this study, we want to evaluate the models' performance on the stance polarity and intensity prediction problem. We separated the dataset into training and testing sets using a 75-25 split. For the neural network models (pkudblab-PIP and T-PAN-PIP), we separated out  $10\%$  of the training set as a validation set to detect over-fitting. The split was performed randomly without consideration of the discussion issue. Each issue was represented proportionally in the training and testing data sets with a maximum discrepancy of less than  $1\%$ .
For evaluation, we want to see how well the regression models are able to predict the continuous agreement value for a post. We report the root-mean-squared error (RMSE) for the predicted results.
5751
6.2 Agreement Prediction Models for Stance Detection
We wanted to investigate whether training models for agreement prediction would degrade their performance for stance detection. Ideally, these models should learn to identify both stance intensity without impacting their ability to identify stance polarity.
To test this, we compared each model to their original stance classification models described in their source papers. Thus, ridge-H is compared with an SVM trained on the same feature set (SVM-H), ridge-S is compared to a Linear-SVM trained on the same feature set (SVM-S), SVR-RF-R is compared to a majority-voting ensemble of a linear-SVM, Random Forest, and Naïve Bayes classifier using the same feature set (SVM-RF-NB), pkudblab-PIP is compared to the original pkudblab model trained using a softmax cross-entropy loss function, and T-PAN-PIP is compared to the original T-PAN model trained using a softmax cross-entropy loss function. We trained the classification models for stance detection by converting the continuous agreement values into categorical polarity values. When converted into categorical values, all of the positive agreement values are classified as Favoring, all negative values are classified as Opposing, and zero values are classified as Neutral. In the dataset, 12,258 arguments are Favoring  $(54\%)$ , 8962 arguments are Opposing  $(40\%)$ , and 1386 arguments are Neutral  $(6\%)$ . To assess the stance detection performance of the models trained for agreement prediction, we converted the predicted continuous agreement values output by the models into the categorical values using the same method.
For evaluation, we report both the accuracy value of the predictions and the macro-average F1-scores for the Favoring and Opposing classes on the testing set. This scoring scheme allows us to treat the Neutral category as a class that is not of interest (Mourad et al., 2018).
7 Evaluation Results
7.1 Agreement Prediction Results
The results for agreement prediction are shown in Table 1. A mean prediction baseline model is shown in the table to demonstrate the difficulty associated with the problem. The neural network models perform worse than both the ridge regression and ensemble models. Ridge-S performed slightly better than Ridge-M due to the sum word
Model	RMSE
Baseline (Mean)	0.718
Ridge-M	0.620
Ridge-S	0.615
SVR-RF-R	0.596
pkudblab-PIP	0.657
T-PAN-PIP	0.623
Table 1: The results of the regression models for the Agreement prediction task. The best result is bolded.
embedding features. The best performing model was the SVR-RF-R model with an RMSE of 0.596.
We performed feature analysis on the SVR-RF-R model using ablation testing (i.e., removing one feature set from the model). Results showed that removing a single features set for each type of feature (Word N-grams, Character N-grams, POS N-grams, Topic features, Lexicon features, word embedding features, and cosine similarity feature) impacted the RMSE of the model by less than 0.005. Using only the N-gram features resulted in an RMSE of 0.599, which is only a 0.0047 decrease from the total. This result matches the difference between Ridge-M (only uses N-gram features) and Ridge-S (includes N-gram and word embedding features). Since the N-gram features contain most of the textual information, it had the most impact on the model, while the additional features had smaller effects on the model accuracy.
7.2 Agreement Prediction models for Stance Detection Results
We compare the models trained on the agreement prediction task to their classification model counterparts in terms of performance on the stance detection task. Tables 2 and 3 show the comparison between the models in terms of accuracy and (macro) F1-score.
SVR-RF-R has the best accuracy and F1-score for stance detection, which outperformed its classifier counterpart (SVM-RF-NB) by  $2.12\%$  in accuracy and  $+0.016$  in F1-score. Three of the models trained for stance polarity and intensity prediction, SVR-RF-R, Ridge-S, and T-PAN-PIP, outperformed their classifier counterparts in accuracy by  $1 - 2\%$  and F1-score by  $+0.009$  on average. Two of the models trained for stance polarity and intensity prediction, Ridge-H and pkudblab-PIP, slightly underperformed their classifier counterparts in accuracy by  $-0.36\%$  and F1-score by  $-0.011$  on average.
5752
Stance Polarity Prediction Model	Polarity and Intensity Prediction Model	Diff
Model	Accuracy	Model	Accuracy
Baseline (Most Frequent)	54.36%	Baseline (Mean)	54.36%	0.00%
SVM-H	68.48%	Ridge-H	68.16%	-0.32%
SVM-S	67.63%	Ridge-S	68.84%	+1.21%
SVM-RF-NB	68.31%	SVR-RF-R	70.43%	+2.12%
pkudblab	67.28%	pkudblab-PIP	66.89%	-0.39%
T-PAN	65.55%	T-PAN-PIP	66.64%	+1.09%
Table 2: The classification accuracy of the stance polarity prediction models and the stance polarity and intensity prediction models for Stance Detection (polarity only) classification.
Stance Polarity Prediction Model	Polarity and Intensity Prediction Model	Diff
Model	F1-Score	Model	F1-Score
Baseline (Most Frequent)	0.352	Baseline (Mean)	0.352	0.000
SVM-H	0.701	Ridge-H	0.695	-0.006
SVM-S	0.697	Ridge-S	0.703	+0.006
SVM-RF-NB	0.705	SVR-RF-R	0.721	+0.016
pkudblab	0.688	pkudblab-PIP	0.672	-0.016
T-PAN	0.673	T-PAN-PIP	0.678	+0.005
Table 3: The F1-scores of the stance polarity prediction models and the stance polarity and intensity prediction models for Stance Detection (polarity only) classification.
The models behaved very similarly on the agreement prediction problem, where the difference between the best performing model and the worst performing model is only 0.061. Overall, the best model received an RMSE of 0.596, which is reasonably good but can be improved.
T-PAN-PIP had the worst performance, which is surprising, as it was the only model to include the parent post's information into its prediction, which should have helped improve its performance. It is possible that its architecture is unsuitable for agreement prediction; other architectures have been deployed that include a post's parent and ancestors into a stance prediction, which might be more suitable for agreement prediction. Future model designs should better incorporate a post's parent information into their predictions.
The difference in performance between the agreement prediction models and the classification models on the stance detection task was small and sometimes better. This demonstrates that the models learning to identify stance intensity do so without significant loss of performance in identifying stance polarity.
Larger gains in performance will likely require information about the post's author. Some post
authors will state strong levels of agreement in their statements, but annotate their argument with weaker agreement levels. For example, one author wrote, "Agree completely. Government should stay out of healthcare." and annotated that argument with an agreement value of  $+0.6$ . The authors were instructed on how to annotate their posts, but the annotations themselves were left to the post's author's discretion. Thus including author information into our models would likely improve the stance polarity and intensity prediction results.
We introduce a new research problem called stance polarity and intensity prediction in a responsive relationship between posts, which predicts both an online post's stance polarity and intensity value toward another post. This problem encapsulates stance detection and adds the additional difficulty of detecting subtle differences in intensity found in the text. We introduced a new large empirical dataset for agreement prediction, collected using a cyber argumentation platform. We implemented five models, adapted from top-performing stance detection models, for evaluation on the new dataset for agreement prediction. Our empirical results demonstrate that the ensemble model SVR-RF-R performed the best for agreement prediction and
5753
models trained for agreement prediction learn to differentiate between intensity values without degrading their performance for determining stance polarity. Research into this new problem of agreement prediction will allow for a more nuanced annotation and analysis of online debate.
We would like to acknowledge Md Mahfuzer Rahman and Najla Alhuniyan for their efforts in developing the ICAS platform and planning the empirical studies. We are also grateful to the anonymous reviewers for their constructive input during the review process.
Leila Amgoud and Jonathan Ben-Naim. 2018. Weighted Bipolar Argumentation Graphs: Axioms and Semantics. In *IJCAI'18 Proceedings of the 27th International Joint Conference on Artificial Intelligence*, pages 5194–5198, Stockholm, Sweden.
Ravi S. Arvapally, Xiaoqing Frank Liu, Fiona Fui-Hoon Nah, and Wei Jiang. 2017. Identifying outlier opinions in an online intelligent argumentation system. *Concurrency and Computation: Practice and Experience*, page e4107.
Ravi Santosh Arvapally and Xiaoqing (Frank) Liu. 2013. Polarisation assessment in an intelligent argumentation system using fuzzy clustering algorithm for collaborative decision support. 4(3):181-208.
Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. SENTIWORDNET 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining. In Proceedings of the Seventh International Conference on Language Resources and Evaluation, volume 10, pages 2200 - 2210, Valletta, Malta.
Philip Bell and Marcia C. Linn. 2000. Scientific arguments as learning artifacts: designing for learning from the web with KIE. International Journal of Science Education, 22(8):797-817.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research, 3(Jan):993-1022.
Filip Boltužić and Jan Šnajder. 2014. Back up your Stance: Recognizing Arguments in Online Discussions. In Proceedings of the First Workshop on Argumentation Mining, pages 49-58, Baltimore, Maryland. Association for Computational Linguistics.
Clinton Burfoot, Steven Bird, and Timothy Baldwin. 2011. Collective Classification of Congressional Floor-debate Transcripts. In Proceedings of the 49th
Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT '11, pages 1506-1515, Stroudsburg, PA, USA. Association for Computational Linguistics. Event-place: Portland, Oregon.
N. Chanda and X. F. Liu. 2015. Intelligent analysis of software architecture rationale for collaborative software design. In 2015 International Conference on Collaboration Technologies and Systems (CTS), pages 287-294.
Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 69-76, Vancouver, Canada. ArXiv: 1704.05972.
Kuntal Dey, Ritvik Shrivastava, and Saroj Kaushik. 2018. Topical Stance Detection for Twitter: A Two-Phase LSTM Model Using Attention. In Advances in Information Retrieval, Lecture Notes in Computer Science, pages 529-536. Springer International Publishing.
Rui Dong, Yizhou Sun, Lu Wang, Yupeng Gu, and Yuan Zhong. 2017. Weakly-Guided User Stance Prediction via Joint Modeling of Content and Social Interaction. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM '17, pages 1249-1258, New York, NY, USA. ACM. Event-place: Singapore, Singapore.
Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017. Stance classification with target-specific neural attention networks. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence.
Steffen Eger, Johannes Daxenberger, and Iryna Gurevych. 2017. Neural End-to-End Learning for Computational Argumentation Mining. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 11–22, Vancouver, Canada. Association for Computational Linguistics. ArXiv: 1704.06104.
Heba Elfardy and Mona Diab. 2016. CU-GWU Perspective at SemEval-2016 Task 6: Ideological Stance Detection in Informal Text. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 434-439, San Diego, California. Association for Computational Linguistics.
Andreas Hanselowski, Avinesh PVS, Benjamin Schiller, Felix Caspelherr, Debanjan Chaudhuri, Christian M. Meyer, and Iryna Gurevych. 2018. A Retrospective Analysis of the Fake News Challenge Stance Detection Task. arXiv:1806.05180 [cs]. ArXiv: 1806.05180.
Kazi Saidul Hasan and Vincent Ng. 2013. Stance Classification of Ideological Debates: Data, Models, Features, and Constraints. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 1348 - 1356.
Kazi Saidul Hasan and Vincent Ng. 2014. Why are you taking this stance? identifying and classifying reasons in ideological debates. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 751-762. Association for Computational Linguistics.
Minqing Hu and Bing Liu. 2004. Mining and Summarizing Customer Reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '04, pages 168-177, New York, NY, USA. ACM. Event-place: Seattle, WA, USA.
Yuki Igarashi, Hiroya Komatsu, Sosuke Kobayashi, Naoaki Okazaki, and Kentaro Inui. 2016. Tohoku at SemEval-2016 Task 6: Feature-based Model versus Convolutional Neural Network for Stance Detection. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 401-407, San Diego, California. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs]. ArXiv:1412.6980.
Mark Klein. 2011. How to Harvest Collective Wisdom on Complex Problems: An Introduction to the MIT Deliberatorium.
Igor Kononenko, Edvard Simec, and Marko Robnik-Sikonja. 1997. Overcoming the Myopia of Inductive Learning Algorithms with RELIEFF. Applied Intelligence, 7(1):39-55.
Werner Kunz and Horst W J Rittel. 1970. Issues as elements of information systems. volume 131, Berkeley.
Gina-Anne Levow, Valerie Freeman, Alena Hrynkevich, Mari Ostendorf, Richard Wright, Julian Chan, Yi Luan, and Trang Tran. 2014. Recognition of stance strength and polarity in spontaneous speech. In 2014 IEEE Spoken Language Technology Workshop (SLT), pages 236-241. ISSN: null.
Anders Edelbo Lillie and Emil Refsgaard Middelboe. 2019. Fake News Detection using Stance Classification: A Survey. arXiv:1907.00181 [cs]. ArXiv: 1907.00181.
Marco Lippi and Paolo Torroni. 2016. Argumentation Mining: State of the Art and Emerging Trends. ACM Trans. Internet Technol., 16(2):10:1-10:25.
X. Liu, R. Wanchoo, and R. S. Arvapally. 2011. Empirical study of an intelligent argumentation system in MCDM. In 2011 International Conference on Collaboration Technologies and Systems (CTS), pages 125-133.
Xiaoqing (Frank) Liu, Eric Christopher Barnes, and Juha Erik Savolainen. 2012. Conflict detection and resolution for product line design in a collaborative decision making environment. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW '12, pages 1327-1336. ACM. Event-place: Seattle, Washington, USA.
Xiaoqing (Frank) Liu, Ekta Khudkhudia, Lei Wen, Vamshi Sajja, and Ming C. Leu. 2010. An Intelligent Computational Argumentation System for Supporting Collaborative Software Development Decision Making. In Farid Meziane, Sunil Vadera, and Ivan Giannoccaro, editors, Artificial Intelligence Applications for Improved Software Engineering Development: New Prospects, Advances in Computational Intelligence and Robotics, pages 167 - 180. IGI Global.
Xiaoqing (Frank) Liu, Samir Raorane, and Ming C. Leu. 2007. A web-based intelligent collaborative system for engineering design. In W. D. Li, Chris McMahon, S. K. Ong, and Andrew Y. C. Nee, editors, Collaborative Product Design and Manufacturing Methodologies and Applications, Springer Series in Advanced Manufacturing, pages 37-58. Springer London.
Edward Loper and Steven Bird. 2002. NLTK: The Natural Language Toolkit. In Proceedings of the ACL Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, pages 63 - 70. ArXiv: cs/0205028.
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.
Andrew Kachites McCallum. 2002. Mallet: A Machine Learning for Language Toolkit.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781 [cs]. ArXiv: 1301.3781.
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 Task 6: Detecting Stance in Tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31-41, San Diego, California. Association for Computational Linguistics.
5755
Saif M. Mohammad. 2016. Sentiment Analysis: Detecting Valence, Emotions, and Other Affectual States from Text. In Herbert L. Meiselman, editor, Emotion Measurement, pages 201-237. Woodhead Publishing.
Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 321-327. ArXiv: 1308.6242.
Sara S. Mourad, Doaa M. Shawky, Hatem A. Fayed, and Ashraf H. Badawi. 2018. Stance Detection in Tweets Using a Majority Vote Classifier. In The International Conference on Advanced Machine Learning Technologies and Applications (AMLTA2018), Advances in Intelligent Systems and Computing, pages 375-384. Springer International Publishing.
Huy Nguyen and Diane Litman. 2016. Context-aware Argumentative Relation Mining. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1127-1137, Berlin, Germany. Association for Computational Linguistics.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournaepau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12:2825-2830.
Isaac Persing and Vincent Ng. 2016. End-to-End Argumentation Mining in Student Essays. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1384–1394, San Diego, California. Association for Computational Linguistics.
Joseph Sirrianni, Xiaoqing (Frank) Liu, and Douglas Adams. 2018. Quantitative Modeling of Polarization in Online Intelligent Argumentation and Deliberation for Capturing Collective Intelligence. In 2018 IEEE International Conference on Cognitive Computing (ICCC), pages 57-64.
Parinaz Sobhani, Diana Inkpen, and Stan Matwin. 2015. From Argumentation Mining to Stance Classification. In Proceedings of the 2nd Workshop on Argumentation Mining, pages 67-77, Denver, CO. Association for Computational Linguistics.
Parinaz Sobhani, Diana Inkpen, and Xiaodan Zhu. 2019. Exploring deep neural networks for multi-target stance detection. Computational Intelligence, 35(1):82-97.
Parinaz Sobhani, Saif Mohammad, and Svetlana Kiritchenko. 2016. Detecting Stance in Tweets And Analyzing its Interaction with Sentiment. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pages 159-169, Berlin, Germany. Association for Computational Linguistics.
Christian Stab and Iryna Gurevych. 2017. Parsing Argumentation Structures in Persuasive Essays. Computational Linguistics, 43(3):619-659.
Manfred Stede and Jodi Schneider. 2018. Argumentation Mining, volume 11 of Synthesis Lecutres on Human Language Technologies. Morgan & Claypool Publishers.
Qingying Sun, Zhongqing Wang, Qiaoming Zhu, and Guodong Zhou. 2018. Stance Detection with Hierarchical Attention Network. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2399-2409, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Stephen E. Toulmin. 2003. The Uses of Argument. Cambridge University Press, Cambridge. Google-Books-ID: 8UYgegaB1S0C.
Martin Tutek, Ivan Sekulic, Paula Gombar, Ivan Paljak, Filip Culinovic, Filip Boltuzic, Mladen Karan, Domagoj Alagic, and Jan Šnajder. 2016. TakeLab at SemEval-2016 Task 6: Stance Classification in Tweets Using a Genetic Algorithm Based Ensemble. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 464-468, San Diego, California. Association for Computational Linguistics.
Prashanth Vijayaraghavan, Ivan Sysoev, Soroush Vosoughi, and Deb Roy. 2016. DeepStance at SemEval-2016 Task 6: Detecting Stance in Tweets Using Character and Word-Level CNNs. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 425-431, San Diego, California. ArXiv: 1606.05694.
Wan Wei, Xiao Zhang, Xuqin Liu, Wei Chen, and Tengjiao Wang. 2016. pkudblab at SemEval-2016 Task 6: A Specific Convolutional Neural Network System for Effective Stance Detection. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 384-388, San Diego, California. Association for Computational Linguistics.
Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 347-354.
Guido Zarrella and Amy Marsh. 2016. MITRE at SemEval-2016 Task 6: Transfer Learning for Stance
Detection. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 458 - 463, San Diego, California. ArXiv: 1606.03784.
A.1 Extended Model Description
The following sections give a more detailed description for some of the models used in our research. The models were written using the Sci-kit learn (Pedregosa et al., 2011) and TensorFlow libraries (Martín Abadi et al., 2015).
A.1.1 SVR-RF-R Feature Set Description
The SVR-RF-R model used a total of 2855 features. They are listed below.
Linguistic Features: