SlowGuess/ACL_2020_OCR huggingface dataset:
Matan Orbach, Yonatan Bilu, Assaf Toledo, Dan Lahav, Michal Jacovi, Ranit Aharonov and Noam Slonim
{matano,yonatanb,michal.jacovi,noams}  $@$  il.ibm.com
{assaf.toledo,dan.lahav, ranit.aharonov}  $@$  ibm.com
An educated and informed consumption of media content has become a challenge in modern times. With the shift from traditional news outlets to social media and similar venues, a major concern is that readers are becoming encapsulated in "echo chambers" and may fall prey to fake news and disinformation, lacking easy access to dissenting views. We suggest a novel task aiming to alleviate some of these concerns – that of detecting articles that most effectively counter the arguments – and not just the stance – made in a given text. We study this problem in the context of debate speeches. Given such a speech, we aim to identify, from among a set of speeches on the same topic and with an opposing stance, the ones that directly counter it. We provide a large dataset of 3685 such speeches (in English), annotated for this relation, which hopefully would be of general interest to the NLP community. We explore several algorithms addressing this task, and while some are successful, all fall short of expert human performance, suggesting room for further research. All data collected during this work is freely available for research<sup>1</sup>.
Recently, a publication on Quantum Computing described a quantum computer swiftly performing a task that arguably would require 10,000 years to be solved by a classical computer (Arute et al., 2019). A non-expert reader is likely to consider this claim as a hard-proven fact, especially due to the credibility of the venue in which this publication appeared. Shortly afterwards, a contesting blog written by other experts in that field<sup>2</sup>
argued, among other things, that the aforementioned problem can be simulated on a classical computer, using proper optimizations, in 2.5 days. Clearly, out of potentially many texts questioning the promise of Quantum Computers (e.g. Kalai (2019)), making readers of the former publication aware of that specific blog post, which directly contests the claims argued in that publication, will provide them with a more informed view on the issue.
Broadly, argumentative texts, such as articles that support a certain viewpoint, often lack arguments contesting that viewpoint. This may be because those contesting arguments are not known to the author of the text, as they might not even have been raised at the time of writing. Alternatively, authors may also deliberately ignore certain known arguments, which might undermine their argumentative goal. Regardless of the reason, this issue places readers at a disadvantage. Lacking familiarity with opposing views that specifically challenge a given perspective, may lead to uninformed decisions or establishing opinions based on partial or biased information. Therefore, there is merit to developing a system that can automatically detect such opposing views.
Motivated by this scenario, we propose a novel natural language understanding task: Given an input text and a corpus, retrieve from that corpus a counter text which includes arguments contesting the arguments raised in the input text. While contemporary systems allow fetching texts on a given topic, and can employ existing tools to discern its stance – and so identify texts with an opposing view – they lack the nuance to identify the counter text which directly contests the arguments raised in the input text.
The potential use-cases of the proposed system exist in several domains. In politics, it can present counters to partisan texts, thus promot
1https://www.research.ibm.com/ haifa/dept/vst/debating_data.shtml# DebateSpeechAnalysis
$^{2}$ https://www.ibm.com/blogs/research/2019/10/on-quantum-supremacy/
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7073-7086 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ing more informed and balanced views on existing controversies. In social media, it can alleviate the bias caused by the "echo chamber" phenomenon (Garimella et al., 2018), by introducing opposing views. And in the financial domain, it can potentially help analysts find relevant counter-texts to predictions and claims made in earning calls. It may also help authors to better present their stance, by challenging them with counter texts during their writing process. Lastly, it may aid researches to examine relevant citations by annotating which papers, out of potentially many, hold opposing views. Note, however, that this paper focuses on counter text detection - a useful tool for these worthy goals, but not a complete solution.
To pursue the aforementioned task, one needs a corresponding benchmark data, that would serve for training and evaluating the performance of an automatic system. For example, one may start with an opinion article, find a set of opinion articles on the same topic with an opposing stance, and aim to detect those that most effectively counter the arguments raised in the opinion article we started with. This path represents a formidable challenge; for example, reliable annotation of long texts is notoriously difficult to obtain (Lavee et al., 2019a), to name just one reason out of many.
To overcome this issue, here we focus on a unique debate setup, in which the goal of one expert debater is to generate a coherent speech that counters the arguments raised in another speech by a fellow debater. Specifically, as part of Project Debater<sup>3</sup>, we collected more than 3,600 debate speeches, each around four minutes long, recorded by professional debaters, on a wide variety of controversial topics, posed as debate motions (e.g. we should ban gambling). With this paper, we make this data available to the community at large. Each motion has a set of supporting speeches, and another set of opposing speeches, typically recorded in response to one – and only one – of the supporting speeches. Correspondingly, our task is defined as follows. Given a motion, a supporting speech, and a set of candidate opposing speeches discussing the same motion, identify the opposing speeches recorded in response to the supporting speech.
We analyze human performance on this chal
lenging task, over a sample of speeches, and further report systematic results of a wide range of contemporary NLP models. Our analysis suggests that expert humans clearly outperform the examined automatic methods, by employing a potentially non-trivial mix of heuristics.
In summary, our main contributions are as follows: (1) Introducing a novel NLU task, of identifying the long argumentative text that best refutes a long argumentative text given as input. (2) Suggesting to simulate the proposed general task in a well-framed debate setup, in which one should identify the response speech(es) that rebuts a given supporting speech. (3) Sharing a large collection of more than 3,600 recorded debate speeches, that allow to train and evaluate automatic methods in our debate-setup task. (4) Providing empirical results for a variety of contemporary NLP models in this task. (5) Establishing the performance of humans in this task, conveying that expert humans currently outperform automatic methods.
Most similar to our work is the task of retrieving the best counter argument to a single given argument (Wachsmuth et al., 2018), also within the debate domain. However, in that setting counterarguments may discuss different motions, or have the same stance towards one motion. In our setting, identifying speeches discussing the same motion can be done using existing NLP methods, and being of opposing stances may be explored with various sentiment analysis techniques. Our focus is on identifying the response to a supporting speech within a set of opposing speeches, all discussing the same motion. Other than the different setup, our task also handles a more complex premise - speeches which are substantially longer than any single argumentative unit, and include multiple such units.
An alternative to our approach is breaking the problem into three stages: (1) identifying specific arguments made in each debate speech; (2) establishing counterargument relations between such arguments found in different speeches; (3) choosing the best response speech based on these argument-level relations. The first subproblem has been recently explored in Mirkin et al. (2018b); Lavee et al. (2019b); Orbach et al. (2019). The second is related to a major research area within computational argumentation
<sup>3</sup>https://www.research.ibm.com/artificial-intelligence/project-debater/
7074
(see recent surveys by Cabrio and Villata (2018); Lawrence and Reed (2019)). Such research includes detecting attack relations between arguments (Cabrio and Villata, 2012; Rosenthal and McKeown, 2015; Peldszus and Stede, 2015b; Cocarascu and Toni, 2017; Wachsmuth et al., 2018), modeling them (Sridhar et al., 2015), depicting these relations (Walker et al., 2012; Peldszus and Stede, 2015a; Musi et al., 2017), generating counter-arguments (Hua and Wang, 2018; Hua et al., 2019), and establishing a theoretical framework for engagement (Toulmin, 2003; Govier, 1991; Dung, 1995; Damer, 2009; Walton, 2009).
A major drawback of the above approach is that it requires a considerable labeling effort – the annotation of arguments mentioned within speeches – which has been shown to be a challenge (Lavee et al., 2019a). Another is that the methods in the above studies which focus on establishing relations at the individual argument level may be limited when aiming to evaluate the perspective of long texts. Specifically, a response speech may contain multiple arguments that relate to the supporting speech in different ways. For instance, the speaker in such a speech may choose to concede an argument, while still maintaining an opposite view. Therefore simply mapping argument level relations may fall short when trying to generalize and assess full speeches. Our task complements the above endeavors by facilitating a framework that would allow extending their granularity from the argument level to a full-text level. Also, our main motivation is different – detecting whole long counter speeches, and not the exact counter arguments within the counter speech. The latter, perhaps more challenging goal, is out of scope for this work.
New neural models have recently driven performance improvements across many NLP tasks (Devlin et al., 2018; Radford et al., 2018), surpassing the level of non-expert humans in a diverse set of benchmark tasks (Wang et al., 2018; McCann et al., 2018). To facilitate the progress of further research Wang et al. (2019) introduced a benchmark aiming to pose a new series of rigorous tests of language understanding which are challenging for cutting-edge NLP technologies. Our work is consistent with the motivation behind these benchmarks, as it suggests a challenging new NLU task, accompanied by a corresponding dataset and benchmarks.
The rise of deliberate disinformation, such as fake news, highlights the erosion in the credibility of consumed content (Lazer et al., 2018), and situations where one is exposed only to opinions that agree with their own, as captured by the notion of echo chambers, are becoming more prevalent (Garimella et al., 2018; Duseja and Jhamtani, 2019). The task proposed in this work seems timely in this context.
We now detail the process of collecting the speeches, the structure of the dataset, and how it is used for our task.
Dataset structure Each speech in the dataset discusses a single motion and is either a supporting speech – in which a single speaker is arguing in favor of the discussed motion, or an opposing speech – in which the speaker is arguing against the motion, typically in response to a supporting speech for that motion. As described below, debaters recording an opposing speech typically listen to a given recorded supporting speech, and then design and record their own speech in response to it. This counter speech is either explicit – including a rebuttal part in which the speaker directly addresses arguments raised in the rebutted speech, or implicit – including no such dedicated rebuttal section, but tacitly relating to the issues raised in the supporting speech they respond to. The data contains multiple counter speeches to each supporting speech, among which some, none or all may be explicit or implicit. Figure 1 depicts the structure of this dataset. Examples of one explicit and one implicit counter speeches are included in the Appendix.
Recording speeches The supporting speeches were produced by a team of professional debaters, using a procedure similar to the one described in Mirkin et al. (2018a): The debaters were each given a list of motions, accompanied by relevant background materials (taken from an online resource such as Wikipedia). They were allowed ten minutes of preparation time to review a motion's background material, after which they recorded a speech arguing in favor of that motion, which was around four minutes long. Through this process, 1797 supporting speeches were recorded, discussing 460 motions.
To record an opposing speech, the debaters were
7075
Figure 1: The speeches data structure for two motions ( $M1$  and  $M2$ ): Each motion has several supporting (Sup.) and opposing (Op.) speeches. Opposing speeches which constitute an explicit/implicit counter speech to a supporting speech are connected to it with a solid/dashed line. In the data, each supporting speech has zero or more counters, and each opposing speech is the counter of at most one supporting speech.
first given ten minutes to review the background material for the motion, as in the recording of a supporting speech. Then, they listened to a supporting speech (recorded by a fellow debater) and recorded a counter speech of similar length. Due to different debate styles popular in different parts of the world, some debaters recorded explicit counter speeches while others recorded implicit ones. To expedite the pace of the recording process, towards its end, few opposing speeches were recorded without requiring the debater to respond to a specific supporting speech. Instead, the debaters were instructed to think of supporting arguments themselves, and respond to these arguments. In total, 1887 opposing speeches were recorded: 348 are explicit counters, 1389 are implicit, and the other 150 are not the counter speech of any supporting speech. The full guidelines used by the debaters during the recordings are included in the Appendix.
The recorded audios were automatically transcribed into text using Watson's off-the-shelf Automatic Speech to Text (STT) $^{4}$ . Human transcribers listened to the recorded speeches, and manually corrected any errors found in the transcript texts produced by the STT system. On average, each speech transcript contains 28.2 sentences, and averages 738.6 tokens in length.
For the purpose of this work, the manually-corrected transcripts are used. The full data of 3685 speeches, including the recorded audios, the STT system outputs and the manually-corrected
transcripts are available on our website<sup>5</sup>. For comparison, the previous release of Project Debater's speeches dataset (Lavee et al., 2019b) included a smaller subset of 400 speeches. Further details on the format of the full data and the recordings process are available in Mirkin et al. (2018a).
Usage As noted above, our task input is comprised from a supporting speech and several candidate opposing speeches all discussing the same motion. Some candidates are counters of the supporting speech, and others are typically counters of other supporting speeches for the same motion. The goal is to identify those counter speeches made in response to the supporting speech. Opposing speeches produced by the speaker of the supporting speech were excluded from the candidates set, as in the real world it is unexpected for one to simultaneously support both sides of a discussion.
4 Human Performance
Recently, with deep learning techniques achieving human performance on several NLU tasks, and even surpassing it, there is growing interest in raising the bar (Wang et al., 2019). That is, to facilitate advancing NLU beyond the current state-of-the-art, there is a need for novel tasks which are solvable by humans, yet challenging for automatic methods. To assess our proposed task in this context, we performed an annotation experiment, as described below.
Setup Each question presented one supporting speech and between 3 to 5 candidate opposing speeches, all discussing the same motion. Annotators were instructed to read the speeches, and select one opposing speech which they thought was a counter speech of the supporting speech. When they could not identify such a counter, they were asked to guess and mention that they had done so.
60 questions were randomly sampled and given to 3 English-proficient expert annotators, who have successfully worked with our team in other past annotation experiments. Following their choice of a counter speech, they were asked to explain their choice in free form language.
Following this step, one of the authors read the explanations provided by the experts and formed
4https://www.ibm.com/cloud/watson-speech-to-text
$^{5}$ https://www.research.ibm.com/haifa/dept/vst/debating_data.shtml#DebateSpeechAnalysis
7076
All	Explicit	Implicit
A	R	A	R	A	R
Ex	85.6	31.4	91.7	37.4	76.4	30.2
Cr	48.9	28.4	55.6	29.5	38.9	26.6
Table 1: Annotation results showing, for each annotation setting, the average accuracy (A) obtained by the experts (Ex) and crowd annotators (Cr), along with the accuracy of randomly guessing the answer (R).
a set of reason categories. Then, another 60 questions were sampled and given to 3 crowd annotators, using the Figure-Eight crowdsourcing platform. The crowd annotators were from a dedicated group which regularly participates in annotations done by our team. After choosing a counter speech, they were instructed to choose the reason (or multiple reasons) for their choice from the set of reason categories. The crowd payment was set to  $2.5\mathrm{~s~}$  per question. To encourage thorough work, a post-processing bonus was given for each correct answer, doubling that pay.
The full guidelines given to the expert and crowd annotators are provided in the Appendix.
Results Performance was evaluated by calculating the accuracy of each annotator, and averaging over annotators. These results are presented in Table 1. Overall, the experts obtained an average accuracy of  $86\%$  (Ex row), considerably better than randomly guessing the answer which yielded an accuracy of  $31\%$ . The accuracy of the crowd annotators  $(\mathbf{Cr})$  was lower, yet distinctly better than random. This suggests that the task is difficult, and may require a level of dedication or expertise beyond what is common for crowd-annotators. Fortunately, the dataset is constructed in such a way that human annotation is not required to label it - it is clear by design which opposing speech counters which supporting speech.
To establish whether identifying explicit counters is easier than identifying implicit ones, the average annotator accuracy was separately computed for these two types. Noteworthy, the accuracy of the experts drops from a near perfect score of  $92\%$  on questions with an explicit true counter, to  $76\%$  on questions with an implicit one. Some of the drop may be explained by the smaller chance of guessing the correct answer at random over
Figure 2: The distribution of reasons for the correct and wrong answers of crowd annotators (who overall had accuracy  $\geq 60\%$ ).
this set, but not all<sup>7</sup>. This suggests that, as may be expected, identifying implicit counter speeches is more challenging than identifying an explicit counter. Still, the performance of both types of annotators, over both types of speeches, was better than random.
Reasons analysis The explanations provided by the experts revealed several best-practices for this task, which we categorized as follows: The true counter speech quotes a phrase from the supporting speech; mentions a specific case or argument from the supporting speech; is more comprehensive and addresses more issues raised in the supporting speech than the other candidates; addresses those issues in the same order as they appear in the supporting speech; discusses similar issues; deals with the main issue raised in the supporting speech. Another reason was elimination - discarding the other candidates since they responded to issues or arguments which were not raised in the supporting speech. The last two categories were guess and other (which required writing a reason in free form language).
Focusing on crowd annotators who did the task relatively well (accuracy  $\geq 60\%$ ), Figure 2 presents the distribution of the reasons they gave for their answers, separated between cases when they were correct and when they were wrong. Overall, the reasons distribution suggests that correctly solving this task requires balancing between the various heuristics. While some of these reasons, such as similarity, correspond to existing algorithmic ideas, others (e.g. order or main issue)
Suppose that when answering, annotators answer correctly a fraction  $f$  of the time, and guess  $1 - f$  of the time, with probability of success equal to the random baseline. Then in the explicit case  $f = 0.87$  and in the implicit  $f = 0.67$ .
$^{6}$ www_figure-eight.com
7077
could inspire future research.
5 Counter Speech Identification
Having established that experts perform well on this task, the question remains whether present NLP methods can match that performance.
Data A supporting speech was included in the experiments if (a) there was an opposing speech addressing it; and (b) there was at least one additional opposing speech discussing its motion which was produced either in response to another supporting speech, or without responding to any specific supporting speech. Supporting speeches not meeting these criteria were excluded from the analysis. With these criteria, the data used in the experiments comprised 1102 supporting speeches and 1708 opposing speeches, pertaining to 329 motions.
Split The motions were randomly split into train  $(60\%)$ , validation  $(20\%)$  and test  $(20\%)$  sets, and their speeches were partitioned accordingly.
Settings To separately evaluate the ability to detect explicit and implicit counters, the experiments were performed in three settings. The first utilized the entire data - given a supporting speech, all of the opposing speeches discussing its motion were considered as candidate counters. In the second setting, the true counter speeches were limited to explicit counters. Supporting speeches without any explicit counter were excluded. Similarly, in the last setting, the true counter speeches were limited to implicit counters, and supporting speeches without such counters were excluded. For example, a supporting speech with one explicit counter, one implicit counter and whose motion is associated with two other opposing speeches (which are not its counters), is considered with all four opposing speech candidates in the first setting and three such candidates in the second and third settings - the two non-counters and the one counter of the type relevant to the setting. Table 2 details the statistics of each data split and experimental setting.
Evaluation The methods described next score each of the candidate counters. We report the average accuracy of the top predictions (A) and the average mean reciprocal rank (M), defined as  $1 / r$  where  $r$  is the highest rank of a true counter.
Document similarity Our first method represented speeches as bag-of-terms vectors, where terms are stemmed unigrams appearing in at least  $1\%$  of the speech-pairs in the training set, and the term counts are normalized by the total count of terms in the speech. Given two vectors, their similarity was computed using the Cosine similarity  $(Cos)$  or the inverse Jensen-Shannon divergence  $(JS)$ .