SlowGuess/ACL_2020_OCR huggingface dataset:
Similarity and Dissimilarity Wachsmuth et al. (2018) presented a method for retrieving the best counter argument to a given argument, based on capturing the similarity and dissimilarity between an argument and its counter. At its core, their method is based on two similarity measures between pairs of texts: (i) A word-based similarity, which is defined by the inverse Manhattan distance between the normalized term frequency vectors of the texts (where terms were as mentioned above); (ii) An embeddings-based similarity which used pretrained ConceptNet Number-batch word embeddings (Speer et al., 2017) to represent the words of the texts, averaged those embeddings to obtain a vector representing each text, and calculated the inverse Word Mover's distance (Kusner et al., 2015) between these vectors.
Previously, these measures were used to predict the relations between a pair of argumentative units. Since our speeches may contain multiple arguments, and their location within the text is unknown, we defined this method at the speech level by considering every supporting speech sentence and every candidate counter speech sentence. For each measure, the similarities of one supporting speech sentence to all candidate counter speech sentences were aggregated by applying a function  $f$ , yielding a sentence-to-speech similarity. These sentence-to-speech similarities were aggregated using another function  $g$ , yielding a speech-to-speech similarity. We denote these speech-to-speech measures by  $w_{fg}$  for word-based similarities and  $e_{fg}$  for embedding-based similarities. As aggregation functions, the maximum  $(\uparrow)$ , minimum  $(\downarrow)$ , average  $(+)$  and product  $(\times)$  were considered. For example,  $w_{\uparrow +}$  denotes taking the maximal word-based similarity of each supporting speech sentence to all candidate counter speech sentences, and averaging those values.
Lastly, following Wachsmuth et al. (2018) once more, the similarity  $(SD)$  between a supporting
7078
Setting	Train	Validation	Test
#Su	#Op	av. #C	Pos	#Su	#Op	av. #C	Pos	#Su	#Op	av. #C	Pos
All	649	1021	5.2	31%	218	340	5.2	31%	235	347	5.2	31%
Explicit	159	542	5.6	24%	58	208	6.0	22%	51	188	5.4	25%
Implicit	556	999	5.0	30%	194	337	5.0	30%	201	343	5.1	30%
Table 2: Data split statistics for each experimental setting (see §5.1): The number of supporting (#Su) and opposing (#Op) speeches, the average number of candidate counter speeches for each supporting speech (av. #C), and the percentage of those candidates which are counter speeches (Pos).
speech and a candidate counter is defined as
$$
\alpha \cdot s i m - (1 - \alpha) \cdot d i s s i m
$$
where  $sim$  and  $dissim$  are of the form  $w_{fg} + e_{fg}$ , both  $f$  and  $g$  are aggregation functions,  $sim \neq dissim$  and  $\alpha$  is a weighting factor. In this scoring model  $sim$  aims to capture topic similarity, whereas subtracting  $dissim$  seeks to capture the dissimilarity between arguments from opposing stances. Admittedly, this method is more appropriate for some of the settings explored in Wachsmuth et al. (2018), in which the candidate counter arguments to a given argument may be discussing other topics, and their stance towards the discussed topic is unknown. We include their method here for completeness, and to allow a comparison to their work.
The hyper-parameters, namely, the aggregation functions and the value of  $\alpha$  (from the range  $\{1,0.9,0.8\}$  used by Wachsmuth et al. (2018)) were tuned on the validation set. An additional variant  $(SD - e)$  based solely on the embeddings-based similarity was also considered, since it carries the advantage of not requiring any vocabulary to be derived from the training set. This allowed tuning the hyper-parameters on a larger set comprised from both the training and validation sets.
BERT Devlin et al. (2018) presented the BERT framework which was pre-trained on the masked language model and next sentence prediction tasks. Assuming that an argument and its counter are coherent as consecutive sentences, and that the first sentences of the candidate speech reference the last sentences of the supporting speech, those parts were scored using the pre-trained next-sentence prediction model with (BERT-T) and without (BERT) fine-tuning. The considered sentences from each speech were limited to at most 100 words, since the pre-trained model is limited to 512 word pieces (assuming about two word
pieces per word). Specifically, from the first speech we took the greatest number of sentences from the end of the speech such that their total length was less than 100 words, and similarly for the second speech for its starting sentences. For fine-tuning, we used the supporting speeches with each of their true counter speeches as positive sentence pairs, and added an equal number of negative pairs where the supporting speech appears with a randomly sampled opposing speech that is not its counter.
gram-based The methods described so far assign a score to a supporting speech and a candidate counter without considering the other candidates. Using that content can aid in detecting key phrases or arguments which best characterize the connection between the supporting speech and its counter – these are the ones which are shared between those speeches and are not mentioned in any of the other candidates. Having many such phrases or arguments may be an indication that a candidate is a true counter speech. Indeed, the quote and mention reason categories account for more than  $20\%$  of the reasons selected by the crowd annotators when answering correctly (see Table 2).
To capture this intuition, ngrams containing between 2 to 4 tokens were extracted from each speech. Those containing stopwords, and those fully contained within longer ngrams, were removed. The set of ngrams which appear in both the supporting speech and the candidate - but not in any of the other candidates - was calculated, and the total length of the ngrams it contains was used as the score of the candidate (ngrs).
Mutual Information The speeches were represented as bag-of-terms binary vectors, where the terms are stemmed unigrams (excluding stopwords). Each candidate counter was scored using the mutual information between its vector and the
7079
vector of the supporting speech  $(MI)$ .
In addition, the mutual information between those vectors, conditioned by the presence of terms in the other candidate counters  $(c - MI)$ , was calculated as follows. Let  $v_{s}$  be a vector representing a supporting speech and  $\{v_{c}\}_{c = 1}^{n}$  be a set of  $n$  vectors representing its candidate counters. Let  $c$  be such a candidate counter, and  $o_c$  represent the concatenation of the vectors of the other candidates excluding  $c$ . Let  $v_{c|k}$  denote the vector of values from  $v_{c}$  at the indices where the entries of  $o_c$  are  $k$  (for  $k = 1$  or 0), and let  $v_{s|k}$  be defined similarly. Then, the conditional mutual information of the candidate  $c$  is given by
$$
\sum_ {k = 0} ^ {1} p (k) I \left(v _ {s \mid k}; v _ {c \mid k}\right)
$$
where  $p(k)$  is the percentage of entries of  $o_c$  with the value  $k$ , and  $I(\cdot, \cdot)$  is mutual information. Intuitively, this measure aims to quantify the information shared between a supporting speech and a candidate, after observing the content of all other candidates, and thus is similar in spirit to the ngram-based method mentioned above.
Table 3 presents the results obtained by the different methods in our three experimental settings. These results show that there is a large performance gap between the implicit and explicit settings – in favor of the latter – for all methods (except BERT), suggesting it is an easier setting. This is consistent with the results of our annotation experiment.
While the best performing methods (JS and  $c-MI$ ) surpass the performance of individual crowd annotators (see Table 1), which testifies to the difficulty of the annotation task, the human experts clearly do better, suggesting there is still much room for improvement.
Error analysis We have manually analyzed the top 3 implicit and explicit speeches for which the differences in mutual information between the predicted counter speech and the true counter speech were the greatest. Analysis revealed that such counter speeches are characterized by argumentative material that is thematically similar to the material of the input speech. Depending on the use case, such results are not necessarily errors, since if the goal is to find relevant opposing content it is beneficial to present such speeches, even
Method	All	Explicit	Implicit
A	M	A	M	A	M
JS	51.1	0.69	80.4	0.88	41.2	0.62
c-MI	50.6	0.69	72.5	0.84	42.7	0.63
MI	48.5	0.68	68.6	0.81	40.7	0.62
ngrs	45.1	0.65	60.1	0.73	38.8	0.61
SD-e	42.1	0.63	60.8	0.76	35.3	0.58
Cos	40.0	0.62	49.0	0.70	35.8	0.58
BERT	36.4	0.57	21.6	0.44	33.7	0.57
SD	31.9	0.57	52.9	0.70	31.3	0.56
BERT-T	32.2	0.56	49.0	0.70	35.2	0.58
Rand	31%	-	25%	-	30%	-
Table 3: Experimental results on the test set for each method and experimental setting: The average accuracy of the top prediction (A) and the average mean reciprocal rank (M) of the true counter with the highest score. The methods are ordered by their M score in the All setting. The last row (Rand) shows the accuracy of the random baseline.
if they were not authored in response to the input speech. However, in some instances a thematically similar argument may be an irrelevant counter as arguments can share a theme without being opposing. For example, an input text may discuss an argument pertaining to the rights of a disenfranchised group, while the counter may revolve around pragmatic outcomes to the same disenfranchised group. While these arguments are likely to share the theme of disenfranchisement they are not necessarily opposing.
6 Further Research Potential
The data presented here was collected to facilitate the development of Project Debater, and we chose the novel counter speech detection task to showcase this data and make it available to the community. However, the unique properties of our data - recorded speech which is more organized and carefully construed than everyday speech - make it interesting to revisit well-known NLP and NLU tasks. Several examples are listed below.
Author attribution: All speeches in the dataset are annotated for the debater who recorded them. It could be particularly interesting to study author attribution on our dataset as it contains persuasive language, relevant to opinion writing and social media. Additionally, we provide voice recordings
7080
and transcripts for all speeches, enabling to study multi-modal methods for this task.
Topic identification: This is a well established research area which can be examined here in various aspects, including clustering speeches by topic, matching speeches to topics or extracting the topic of a speech without prior knowledge.
Whereas previous work often requires annotating the topics of texts and deducing a consensual label, in our data the topic of a speech is given by design.
Sentence ordering or local coherence: The sentence ordering task (Barzilay and Lapata, 2005) is concerned with organizing text in a coherent way and is especially relevant for natural language generation. Our dataset allows us to study this using spoken natural language of a persuasive nature, that often relies on a careful development of an argumentative intent. The data also provides a unique opportunity to study the interplay between a coherent arrangement of language and the associated prosodic cues.
Other tasks The large scale of the dataset, over 200 hours of spoken content and their manually-corrected transcripts, enables its use in other speech-processing tasks that require such data. Some examples include speech-to-text, text-to-speech, and direct learning from speech of word (Chung and Glass, 2018) or sentence (Haque et al., 2019) embeddings. Such tasks often use large scale datasets of read content (e.g. Panayotov et al. (2015)), and our data allows their exploration in the context of spoken spontenous speech.
In addition, with further annotations of the dataset, it lends itself to other potential tasks. One example is the extraction of the main points of a speech or article. This can facilitate various downstream tasks, such as single document summarization in the context of spoken language. Another example is the annotation of named entities within the transcript texts, facilitating direct identification of those entities in the audio, similarly to the work of Ghannay et al. (2018).
We presented a novel NLU task of identifying a counter speech, which best counters an input speech, within a set of candidate counter speeches.
As previous studies have shown, and consistent with our own findings, obtaining data for such
a task is difficult, especially considering that labeling at scale of full speeches is an arduous effort. To facilitate research of this problem, we recast the proposed general task in a defined debate setup and construct a corresponding benchmark data. We collected, and release as part of this work, more than 3,600 debate speeches annotated for the proposed task.
We presented baselines for the task, considering a variety of contemporary NLP models. The experiments suggest that the best results are achieved using Jensen-Shannon similarity, for speeches that contain explicit responses (accuracy of  $80\%$ ) and using conditional mutual-information on speeches that respond to the input speech in an implicit way (accuracy of  $43\%$ ).
We established the performance of humans on this task, showing that expert humans currently outperform automatic methods by a significant margin — attaining an accuracy of  $92\%$  on speeches with an explicit true counter, and  $76\%$  on speeches with an implicit one. Noteworthy is that some of the automatic methods outperform the results achieved by the crowd, suggesting that the task is difficult, and may require a level of expertise beyond layman-level.
The reported gap between the performance of expert humans and the results achieved by NLP models demonstrate room for further research. Future research may focus on the motivation we described, but may also utilize the large speeches corpus we release as part of this work to a variety of additional different endeavors.
We wish to thank the many debaters and transcribers that took part in the effort of creating this dataset, and the anonymous reviewers for their insightful comments, suggestions, and feedback.
Frank Arute, Kunal Arya, Ryan Babbush, Dave Bacon, Joseph C Bardin, Rami Barends, Rupak Biswas, Sergio Boixo, Fernando GSL Brandao, David A Buell, et al. 2019. Quantum supremacy using a programmable superconducting processor. Nature, 574(7779):505-510.
Regina Barzilay and Mirella Lapata. 2005. Modeling local coherence: An entity-based approach. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05),
7081
pages 141-148, Ann Arbor, Michigan. Association for Computational Linguistics.
Elena Cabrio and Serena Villata. 2012. Combining textual entailment and argumentation theory for supporting online debates interactions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 208-212, Jeju Island, Korea. Association for Computational Linguistics.
Elena Cabrio and Serena Villata. 2018. Five years of argument mining: a data-driven analysis. In *IJCAI*, pages 5427-5433.
Yu-An Chung and James Glass. 2018. Speech2vec: A sequence-to-sequence framework for learning word embeddings from speech. arXiv preprint arXiv:1803.08976.
Oana Cocarascu and Francesca Toni. 2017. Identifying attack and support argumentative relations using deep learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1374-1379, Copenhagen, Denmark. Association for Computational Linguistics.
T. Edward Damer. 2009. Attacking Faulty Reasoning: A Practical Guide to Fallacy-Free Arguments. Wadsworth/Cengage Laerning.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Phan Minh Dung. 1995. On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. Artificial intelligence, 77(2):321-357.
Nikita Duseja and Harsh Jhamtani. 2019. A sociolinguistic study of online echo chambers on twitter. In Proceedings of the Third Workshop on Natural Language Processing and Computational Social Science, pages 78-83, Minneapolis, Minnesota. Association for Computational Linguistics.
Kiran Garimella, Gianmarco De Francisci Morales, Aristides Gionis, and Michael Mathioudakis. 2018. Political discourse on social media: Echo chambers, gatekeepers, and the price of bipartisanship. In Proceedings of the 2018 World Wide Web Conference (WWW), pages 913-922, Republic and Canton of Geneva, Switzerland. International World Wide Web Conferences Steering Committee.
Sahar Ghannay, Antoine Caubrière, Yannick Esteve, Nathalie Camelin, Edwin Simonnet, Antoine Laurent, and Emmanuel Morin. 2018. End-to-end named entity and semantic concept extraction from speech. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 692-699. IEEE.
Trudy Govier. 1991. A Practical Study of Argument. Wadsworth Pub. Co.
Albert Haque, Michelle Guo, Prateek Verma, and Li Fei-Fei. 2019. Audio-linguistic embeddings for spoken sentences. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7355-7359. IEEE.
Xinyu Hua, Zhe Hu, and Lu Wang. 2019. Argument generation with retrieval, planning, and realization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2661-2672, Florence, Italy. Association for Computational Linguistics.
Xinyu Hua and Lu Wang. 2018. Neural argument generation augmented with externally retrieved evidence. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 219-230, Melbourne, Australia. Association for Computational Linguistics.
Gil Kalai. 2019. The argument against quantum computers. arXiv preprint arXiv:1908.02499.
Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. 2015. From word embeddings to document distances. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37, pages 957-966. JMLR.org.
Tamar Lavee, Lili Kotlerman, Matan Orbach, Yonatan Bilu, Michal Jacovi, Ranit Aharonov, and Noam Slonim. 2019a. Crowd-sourcing annotation of complex NLU tasks: A case study of argumentative content annotation. In Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP, pages 29-38, Hong Kong. Association for Computational Linguistics.
Tamar Lavee, Matan Orbach, Lili Kotlerman, Yoav Kantor, Shai Gretz, Lena Dankin, Shachar Mirkin, Michal Jacovi, Yonatan Bilu, Ranit Aharonov, and Noam Slonim. 2019b. Towards effective rebuttal: Listening comprehension using corpus-wide claim mining. 6th Workshop on Argument Mining.
John Lawrence and Chris Reed. 2019. Argument mining: A survey. Computational Linguistics, 0(ja):1-55.
David M. J. Lazer, Matthew A. Baum, Yochai Benkler, Adam J. Berinsky, Kelly M. Greenhill, Filippo Menczer, Miriam J. Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, Michael Schudson, Steven A. Sloman, Cass R. Sunstein, Emily A. Thorson, Duncan J. Watts, and Jonathan L. Zittrain. 2018. The science of fake news. Science, 359(6380):1094-1096.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. Cite arxiv:1806.08730.
7082
Shachar Mirkin, Michal Jacovi, Tamar Lavee, Hong-Kwang Kuo, Samuel Thomas, Leslie Sager, Lili Kotlerman, Elad Venezian, and Noam Slonim. 2018a. A recorded debating dataset. In Proceedings of LREC.
Shachar Mirkin, Guy Moshkowich, Matan Orbach, Lili Kotlerman, Yoav Kantor, Tamar Lavee, Michal Jacovi, Yonatan Bilu, Ranit Aharonov, and Noam Slonim. 2018b. Listening comprehension over argumentative content. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 719-724. Association for Computational Linguistics.
Elena Musi, Debanjan Ghosh, Mark Aakhus, Smaranda Muresan, and Nina Wacholder. 2017. Building an ontology of (dis) agreement space for argument mining. SIGDIAL/SEMDIAL 2017 Joint Session on Negotiation Dialog (position paper).
Matan Orbach, Yonatan Bilu, Ariel Gera, Yoav Kantor, Lena Dankin, Tamar Lavee, Lili Kotlerman, Shachar Mirkin, Michal Jacovi, Ranit Aharonov, and Noam Slonim. 2019. A dataset of general-purpose rebuttal. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5590-5600, Hong Kong, China. Association for Computational Linguistics.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An asr corpus based on public domain audio books. pages 5206-5210.
Andreas Peldszus and Manfred Stede. 2015a. An annotated corpus of argumentative microtexts. In Proceedings of the First Conference on Argumentation, Lisbon, Portugal.
Andreas Peldszus and Manfred Stede. 2015b. Towards detecting counter-considerations in text. In Proceedings of the 2nd Workshop on Argumentation Mining, pages 104-109.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language models are unsupervised multitask learners. OpenAI Blog.
Sara Rosenthal and Kathy McKeown. 2015. I couldn't agree more: The role of conversational structure in agreement and disagreement detection in online discussions. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 168-177.
Robert Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-First AAAI Conference on Artificial Intelligence.
Dhanya Sridhar, James Foulds, Bert Huang, Lise Getoor, and Marilyn Walker. 2015. Joint models of disagreement and stance in online debate. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 116-125.
Stephen E. Toulmin. 2003. The Uses of Argument. Cambridge University Press.
Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. Retrieval of the best counterargument without prior topic knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 241-251, Melbourne, Australia. Association for Computational Linguistics.
Marilyn A Walker, Jean E Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012. A corpus for research on deliberation and debate. In LREC, pages 812-817. Istanbul.
Douglas Walton. 2009. Objections, rebuttals and refutations. Argument Cultures: Proceedings of the 8th OSSA Conference.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv preprint 1905.00537.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
7083
This appendix contains the guidelines used in all the data generation and annotation tasks described in the paper: 1) speech authorship guidelines, 2) identifying the response speech from a list of candidates, 3) identifying the response speech speech from a list of candidates and providing a reason.
Following the guidelines are two examples of full response speeches - an explicit counter speech and an implicit counter speech (see §3).
B Speech Authoring Guidelines
For supporting speeches: