SlowGuess/ACL_2020_OCR huggingface dataset:
Supervised Methods. We fine tuned the BERT-base-uncased and BERT-large-uncased models (Devlin et al., 2019) to predict matches between argument and key point pairs. We added a linear fully connected layer of size 1 followed by a sigmoid layer to the special [CLS] token in the BERT model, and trained it for three epochs with a learning rate of 2e-5 and a binary cross entropy loss.
NLI Transfer Learning. We also experimented with transfer learning from NLI to our task of argument-to-key point match classification. This was motivated by the similarity between these tasks (as discussed in Section 2.2), as well as the availability of large-scale NLI labeled datasets. We con
4034
sidered the Stanford (SNLI) and the Multi-Genre (MNLI) datasets (Bowman et al., 2015; Williams et al., 2018), each comprising hundreds of thousands of labeled premise-hypothesis pairs. Pairs labeled as ENTAILMENT were considered positive instances, while the rest of the pairs, labeled as NEUTRAL or CONTRADICTION were considered negative. We trained BERT-base and BERT-large models on each of these datasets, following the procedure described above.
4.1.2 Match Classification
In the match classification step we select the matching key points for each argument, based on their respective matching scores. The classification can be done locally, treating each pair individually, or globally, by examining all possible key points for each argument. We compared the following policies for selecting matching key points for a given argument.
Threshold. For each fold, we find the threshold on the match score that maximizes the F1 score for the positive (matching) class. Pairs whose score exceeds the learned threshold are considered matched.
Best Match (BM). Using a threshold is not optimal for our data, where most arguments have at most one matched key point. A natural solution is to select the best matching key point. For each argument, we consider all key points for the same topic and stance as candidates and predict only the candidate with the highest match score as matched to the argument and the rest as unmatched. Note that this is the only fully unsupervised selection policy, as it does not require labeled data for learning a threshold.
$\mathbf{BM} + \mathbf{Threshold}$ . The  $BM$  policy always assigns exactly one key point for each argument, while  $27.5\%$  of the arguments in our data are not matched to any key point. To address this, we combine the two former policies. The top matching key point is considered a match only if its match score exceeds the learned threshold.
Dual Threshold. In order to account for arguments with more than one matching key point, two thresholds are learned. If two key points exceed the lower threshold and at least one of them exceeds the upper threshold, both will be matched. Otherwise, it works the same as the  $BM+Threshold$
policy using only the lower threshold. This allows for zero to two matches per argument.
Thresholds are learned from the development set for supervised match scoring methods, and from both train and development set for unsupervised match scoring methods.
4.2.1 Match Scoring Methods
Table 4 compares the various match scoring methods, all using the Threshold key point selection policy. Results are obtained by micro-averaging over the argument-key point pairs in each fold, and averaging over the different folds. We consider Precision, Recall and F1 of the positive class, as well as the overall accuracy. We also list for reference the majority class baseline that always predicts "no match", and the random baseline, which randomly predicts the positive class according to its probability in the training data.
The unsupervised models fail to capture the relation between the argument and the key points. Tf-Idf and Glove perform the worst, showing that simple lexical similarity is insufficient for this task. BERT embedding does better but still reaches a relatively low F1 score of 0.4.
In contrast to the unsupervised models, supervised models are shown to perform well. BERT with fine tuning leads to a substantial improvement, reaching F1 score of 0.657 with the BERT-base model, and 0.684 with the BERT-large model.
BERT Models trained on NLI data are considerably better than the unsupervised methods, with the best model reaching F1 of 0.526, yet their performance is still far below the supervised models trained on our ArgKP dataset. This may reflect both the similarities and the differences between NLI and the current task. We have also experimented with combining these two types of data in cascade: BERT was first trained on a large NLI dataset (SNLI, MNLI or their union), and was then fine-tuned on the smaller ArgKP data. However, it did not improve the supervised results.
Error Analysis. By analyzing the top errors of the supervised classifier (BERT-large), we found several systematic patterns of errors. In most cases, non-matching arguments and key points received a high match score in one of the following cases:
- They share some key phrases. For example: "It is unfair to only subsidize vocational education. Achieving a more advanced education
Acc	P	R	F1
Majority Class	0.793	0.000
Random Predictions	0.679	0.206	0.200	0.203
Unsupervised Methods	Tf-Idf	0.512	0.246	0.644	0.352
Glove Embeddings	0.346	0.212	0.787	0.330
BERT Embeddings	0.660	0.319	0.550	0.403
Supervised Methods	BERT-base (ArgKP)	0.844	0.609	0.718	0.657
BERT-large (ArgKP)	0.868	0.685	0.688	0.684
NLI Transfer Learning	BERT-base (SNLI)	0.777	0.472	0.514	0.485
BERT-base (MNLI)	0.772	0.470	0.558	0.505
BERT-large (SNLI)	0.765	0.456	0.533	0.487
BERT-large (MNLI)	0.792	0.518	0.542	0.526
Table 4: Comparison of match scoring methods, using the Threshold selection policy. P, R and F1 refer to the positive class. Acc is the accuracy.
All Arguments	Single Key Point	Multiple Key Points	No Key Points
Acc	P	R	F1	P	R	F1	P	R	F1	Acc
Threshold	.868	.685	.688	.684	.720	.686	.701	.904	.690	.782	.933
Best Match	.876	.696	.711	.703	.836	.747	.789	.936	.448	.606	.839
BM+Threshold	.890	.772	.665	.713	.856	.699	.769	.941	.421	.580	.915
Dual Threshold	.887	.721	.740	.730	.784	.752	.767	.945	.656	.773	.908
Table 5: Comparing key point selection policies, using BERT-large trained on the ArgKP dataset for match scoring.
is very expensive and it would also need to be subsidized." and "Subsidizing vocational education is expensive".
- They share a large portion of the sentence, but not the main point, for example: "Women should be able to fight if they are strong enough" and "Women should be able to serve in combat if they choose to".
- They are at least partially related, but labeled as non-matching due to a better fitting key point for the same argument. For example: "We should subsidize space exploration because it increases the knowledge of the universe we are in" and "Space exploration improves science/technology" can be considered matched, but were labeled as unmatched due to the key point "Space exploration unravels information about the universe". Using the Best Match policy helps in these cases.
For arguments and key points that were labeled as matched but received a low match score, the relation was in many cases implied or required some further knowledge, for examples: "Journalism is an essential part of democracy and freedom of expression and should not be subsidized by the state." and "government intervention has the risk of inserting bias/harming objectivity".
4.2.2 Key Point Selection Policies
Table 5 compares different key point selection policies, all using the best performing match scoring method: BERT-large fine-tuned on ArgKP. We report the results over the whole dataset ("all arguments"), as well as the subsets of arguments having none, single or multiple matching key points according to the labeled data. In case of no matches we present the accuracy, as recall and F1 scores are undefined. When considering all the arguments, the Dual Threshold policy achieves the best F1 score of 0.73. The Threshold method performs well for arguments with no matches or multiple matches. When there is exactly one match (the common case in our data), it has lower precision. The Best Match policy performs well when there is a single match, but is not able to cope with arguments that have no matches or have multiple matches. The  $BM+Threshold$  method combines the two and is useful when there are no matching key points or a single matching key point, but still have lower recall when there are multiple matching key points. The Dual Threshold method improves the recall and therefore the F1 score for multiple matches while maintaining good performance for arguments with single or no matches.
Figure 2 shows Precision-Recall trade-off for the various policies, using the different possible thresholds, computed for one of the folds. For each policy, we specify the best F1 score, as well
Figure 2: Precision/Recall trade-off for different key point selection policies. For each method, the highest F1 score, as well as the F1 score for the chosen threshold are specified. For the Best Match + Threshold policy, these two scores coincide.
as the F1 score obtained for the selected threshold, which was optimized over the development set. The Threshold policy allows to control recall, up to one (where the threshold is zero), at the price of low precision. The  $BM+Threshold$  policy generates the highest precision, but low recall, since at most one candidate is selected. Note that when the threshold is zero, the  $BM+Threshold$  policy is equivalent to the  $BM$  policy. The Dual Threshold policy offers the best trade-off, for mid-range precision and recall.
This work addressed the practical problem of summarizing a large collection of arguments on a given topic. We proposed to represent such summaries as a set of key points scored according to their relative salience. Such summary aims to provide both textual and quantitative views of the argument data in a concise form. We demonstrated the feasibility and effectiveness of the proposed approach through extensive data annotation and analysis. We showed that a domain expert can quickly come up with a short list of pro and con key points per topic, that would capture the gist of crowd-contributed arguments, even without being exposed to the arguments themselves. We studied the problem of automatically matching arguments to key points, and developed the first large-scale dataset for this task, which we make publicly available.
Our experimental results demonstrate that the
problem is far from trivial, and cannot be effectively solved using unsupervised methods based on word or sentence-level embedding. However, by using state of the art supervised learning methods for match scoring, together with an appropriate key point selection policy for match classification, we were able to achieve promising results on this task.
The natural next step for this work is the challenging task of automatic key point generation. In addition, we plan to apply the methods presented in this work also to automatically-mined arguments. Finally, detecting the more implicit relations between the argument and the key point, as seen in our error analysis, is another intriguing direction for future work.
We would like to thank the anonymous reviewers for their helpful comments and suggestions.
Yamen Ajjour, Milad Alshomary, Henning Wachsmuth, and Benno Stein. 2019. Modeling frames in argumentation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2922-2932, Hong Kong, China. Association for Computational Linguistics.
Roy Bar-Haim, Indrajit Bhattacharya, Francesco Dinuzzo, Amrita Saha, and Noam Slonim. 2017. Stance
classification of context-dependent claims. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 251-261. Association for Computational Linguistics.
Filip Boltužić and Jan Šnajder. 2014. Back up your stance: Recognizing arguments in online discussions. In Proceedings of the First Workshop on Argumentation Mining, pages 49-58, Baltimore, Maryland. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Elena Cabrio and Serena Villata. 2013. A natural language bipolar argumentation approach to support users in online debate interactions. Argument & Computation, 4(3):209-230.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364.
Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing Textual Entailment: Models and Applications. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Charlie Egan, Advaith Siddharthan, and Adam Wyner. 2016. Summarising the points made in online political debates. In Proceedings of the Third Workshop on Argument Mining (ArgMining2016), pages 134-143, Berlin, Germany. Association for Computational Linguistics.
Liat Ein-Dor, Eyal Shnarch, Lena Dankin, Alon Halfon, Benjamin Sznajder, Ariel Gera, Carlos Alzate, Martin Gleize, Leshem Choshen, Yufang Hou, Yonatan Bilu, Ranit Aharonov, and Noam Slonim. 2020. Corpus wide argument mining - a working solution. In AAAI.
Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378-382.
Shai Gretz, Roni Friedman, Edo Cohen-Karlik, Assaf Toledo, Dan Lahav, Ranit Aharonov, and Noam Slonim. 2020. A large-scale dataset for argument quality ranking: Construction and analysis. In AAAI.
Kazi Saidul Hasan and Vincent Ng. 2014. Why are you taking this stance? identifying and classifying reasons in ideological debates. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 751-762, Doha, Qatar. Association for Computational Linguistics.
J. R. Landis and G. G. Koch. 1997. The measurements of observer agreement for categorical data. Biometrics, 33:159-174.
John Lawrence and Chris Reed. 2020. Argument mining: A survey. Computational Linguistics, 45(4):765-818.
Ran Levy, Yonatan Bilu, Daniel Hershcovich, Ehud Aharoni, and Noam Slonim. 2014. Context dependent claim detection. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1489-1500, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
Ran Levy, Ben Bogin, Shai Gretz, Ranit Aharonov, and Noam Slonim. 2018. Towards an argumentative content search engine using weak supervision. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2066-2081. Association for Computational Linguistics.
Ran Levy, Shai Gretz, Benjamin Sznajder, Shay Hummel, Ranit Aharonov, and Noam Slonim. 2017. Unsupervised corpus-wide claim detection. In Proceedings of the 4th Workshop on Argument Mining, pages 79-84. Association for Computational Linguistics.
Amita Misra, Brian Ecker, and Marilyn Walker. 2016. Measuring the similarity of sentential arguments in dialogue. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 276-287, Los Angeles. Association for Computational Linguistics.
Marie-Francine Moens, Erik Boiy, Raquel Mochales Palau, and Chris Reed. 2007. Automatic detection of arguments in legal texts. In Proceedings of the 11th International Conference on Artificial Intelligence and Law, ICAIL '07, pages 225-230, New York, NY, USA. ACM.
Nona Naderi. 2016. Argumentation mining in parliamentary discourse. In Proceedings of the 11th International Conference of the Ontario Society for the Study of Argumentation, pages 1-9.
Joonsuk Park and Claire Cardie. 2014. Identifying appropriate support for propositions in online user comments. In Proceedings of the First Workshop on Argumentation Mining, pages 29-38, Baltimore, Maryland. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Isaac Persing and Vincent Ng. 2016. End-to-end argumentation mining in student essays. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1384-1394, San Diego, California. Association for Computational Linguistics.
Nils Reimers, Benjamin Schiller, Tilman Beck, Johannes Daxenberger, Christian Stab, and Iryna Gurevych. 2019. Classification and clustering of arguments with contextualized word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 567-578, Florence, Italy. Association for Computational Linguistics.
Ruty Rinott, Lena Dankin, Carlos Alzate Perez, Mitesh M. Khapra, Ehud Aharoni, and Noam Slonim. 2015. Show me your evidence - an automatic method for context dependent evidence detection. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 440-450, Lisbon, Portugal. Association for Computational Linguistics.
Christian Stab, Johannes Daxenberger, Chris Stahlhut, Tristan Miller, Benjamin Schiller, Christopher Tauchmann, Steffen Eger, and Iryna Gurevych. 2018a. Argumentext: Searching for arguments in heterogeneous sources. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 21-25. Association for Computational Linguistics.
Christian Stab and Iryna Gurevych. 2017. Parsing argumentation structures in persuasive essays. Computational Linguistics, 43(3):619-659.
Christian Stab, Tristan Miller, Benjamin Schiller, Pranav Rai, and Iryna Gurevych. 2018b. Crosstopic argument mining from heterogeneous sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3664-3674. Association for Computational Linguistics.
Assaf Toledo, Shai Gretz, Edo Cohen-Karlik, Roni Friedman, Elad Venezian, Dan Lahav, Michal Jacovi, Ranit Aharonov, and Noam Slonim. 2019. Automatic argument quality assessment - new datasets
and methods. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5624-5634, Hong Kong, China. Association for Computational Linguistics.
Henning Wachsmuth, Martin Potthast, Khalid Al Khatib, Yamen Ajjour, Jana Puschmann, Jiani Qu, Jonas Dorsch, Viorel Morari, Janek Bevendorff, and Benno Stein. 2017. Building an argument search engine for the web. In Proceedings of the 4th Workshop on Argument Mining, pages 49-59. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Adam Wyner, Raquel Mochales-Palau, Marie-Francine Moens, and David Milward. 2010. Approaches to text mining arguments from legal cases. In Semantic Processing of Legal Texts: Where the Language of Law Meets the Law of Language, page 60-79, Berlin, Heidelberg. Springer-Verlag.