SlowGuess/ACL_2020_OCR huggingface dataset:
Milad Alshomary
milad.ashomary@mail.upb.de
Shahbaz Syed
shahbaz.syed@uni-leipzig.de
Martin Potthast
martin.potthast@uni-leipzig.de
Henning Wachsmuth
henningw@upb.de
In argumentation, people state premises to reason towards a conclusion. The conclusion conveys a stance towards some target, such as a concept or statement. Often, the conclusion remains implicit, though, since it is self-evident in a discussion or left out for rhetorical reasons. However, the conclusion is key to understanding an argument, and hence, to any application that processes argumentation. We thus study the question to what extent an argument's conclusion can be reconstructed from its premises. In particular, we argue here that a decisive step is to infer a conclusion's target, and we hypothesize that this target is related to the premises' targets. We develop two complementary target inference approaches: one ranks premise targets and selects the top-ranked target as the conclusion target, the other finds a new conclusion target in a learned embedding space using a triplet neural network. Our evaluation on corpora from two domains indicates that a hybrid of both approaches is best, outperforming several strong baselines. According to human annotators, we infer a reasonably adequate conclusion target in  $89\%$  of the cases.
The conclusion (or claim) of a natural language argument conveys a pro or con stance towards some target, such as a controversial concept or statement (Bar-Haim et al., 2017). It is inferred from a set of premises. Conclusions are key to understanding arguments, and hence, critical for any downstream application that processes argumentation. The task of identifying conclusions has been studied intensively in the context of argument mining (Stab and Gurevych, 2014) and automatic essay assessment (Falakmasir et al., 2014). In genres other than essays, however, conclusions often remain implicit, since they are clear from the context of a discussion (Habernal and Gurevych, 2015) or hidden on pur-
Figure 1: Illustration of our full model of generating an argument's conclusion from its premises. This paper focuses on the identification and inference of targets.
pose for rhetorical reasons, as is often the case in news editorials (Al Khatib et al., 2016). This alters the task entirely to become a synthesis task: Given an argument's premises, generate its conclusion.
As detailed in Section 2, research on argumentation synthesis is still limited. Existing approaches focus on generating single claims (Bilu and Slonim, 2016), new arguments (Reisert et al., 2015), counterarguments (Hua et al., 2019), or argumentative texts (Wachsmuth et al., 2018). Closer to conclusion generation, Egan et al. (2016) summarized the main points of online debates, and Wang and Ling (2016) worked on identifying the main claim of an argument through abstractive summarization. To our knowledge, however, no approach so far reconstructs an argument's conclusion from its premises.
In general, we consider the synthesis task outlined above. Conceptually, we decompose this task into three steps, as depicted in Figure 1: (1) inferring the conclusion's target from the premises, (2) inferring the conclusion's stance, and (3) generating the conclusion's text with the inferred stance
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4334-4345
and the inferred target. In this paper, we focus on the first step by proposing two computational approaches for conclusion target inference.
As sketched in Figure 1, we hypothesize that the conclusion target is related to the targets of the argument's premises. To obtain premise targets, we train a state-of-the-art sequence labeling model (Akbik et al., 2018) on target-annotated claims (Bar-Haim et al., 2017). Since the exact relation of premise and conclusion targets is unknown, we develop two complementary inference approaches: One approach ranks premise targets based on their likelihood of being a conclusion target. The other one employs a triplet neural network (Hoffer and Ailon, 2015) that generates a conclusion target embedding from the premise targets in a learned target embedding space. A unique facet of the latter is the integration of the network with a knowledge base of targets (built from any training set), namely, the approach returns the known target whose embedding is closest to the generated embedding.
We compare the approaches against several baselines, including an existing sequence-to-sequence model for argument summarization (with and without encoded premise targets). For evaluation purposes, we study argument corpora from two genres where the correct conclusions are given: student essays (Stab and Gurevych, 2014) and debate portals (Wang and Ling, 2016). On these corpora, we empirically test how often an inferred target matches the target found in the ground-truth conclusion. Moreover, we let human annotators manually check the adequacy of the inferred targets.
In our experiments, both approaches consistently outperform sequence-to-sequence generation, justifying the explicit modeling of the relation between premise and conclusion targets. According to manual evaluation, a combined version of the two approaches infers an at least somewhat adequate target in  $89\%$ , and a fully adequate target in  $55\%$  of the cases, indicating the practical applicability of our target inference in conclusion generation.
1. A conceptual model of the task of generating an argument's conclusion from its premises.
2. Two complementary approaches that infer a conclusion's target from premises effectively.
3. Empirical evidence for the importance of modeling targets in conclusion generation.
Arguments have been modeled in different ways, focusing on the roles of their components (Toulmin, 1958), their inference scheme (Walton et al., 2008), or the interplay between their pro and con components (Freeman, 2011). On an abstract level, the models all share that they consider an argument as a conclusion (in terms of a claim) and a set of premises (reasons to support or object the claim). We restrict our view to this abstract model here.
Even though this paper is about inferring conclusion targets, our ultimate goal is to reconstruct the whole conclusion of an argument. Computational approaches to identify conclusions in a text have been pioneered research on student essay assessment (Burstein and Marcu, 2003). Falakmasir et al. (2014) show the importance of essay conclusions in applications, whereas Jabbari et al. (2016) specifically target an essay's overall conclusion, i.e., its thesis (also known as major, main, or central claim). Given the importance of theses, we dedicate one experiment particularly targeting them below.
The classification of argument components (as theses, conclusions, premises, etc.) is a core task in argument mining (Stede and Schneider, 2018) and has been approached for different genres (Stab and Gurevych, 2014; Peldszus and Stede, 2015). As Habernal and Gurevych (2015) observe, though, real-world arguments often leave the conclusion implicit, particularly where it is clear in the context of a discussion. In genres such as news editorials, conclusions may even be left out on purpose, in order to persuade readers in a "hidden" manner (Al Khatib et al., 2016). If an implicit conclusion is needed, it hence needs to be synthesized.
Argumentation synthesis research is on the rise. Early argument generation approaches relied on rule-based discourse planning techniques (Zukerman et al., 2000). Later, Reisert et al. (2015) generalized target-stance relations from claims and used them to automatically create new arguments. The relations were curated manually, though. An approach that finds the best conclusion for generation among a set of candidate claims was presented by Yanase et al. (2015). Sato et al. (2015) built upon this approach to phrase texts with multiple arguments. Others recycled targets and predicates of claims in new claims (Bilu and Slonim, 2016), generated arguments with specific inference schemes for user-defined content (Green, 2017), modeled rhetorical aspects in synthesis (Wachsmuth et al.,
2018), and composed arguments that follow a strategy (El Baff et al., 2019). All these methods synthesize new argumentative content. In contrast, we aim for the missing components of given arguments.
As such, our task resembles enthymeme reconstruction. An enthymeme is an implicit premise, usually the warrant (or major premise) that clarifies how a conclusion is inferred from the given premises (Walton et al., 2008). Motivated by the importance of finding the thesis, Boltuzic and Snajder (2016) study how to identify such enthymemes given the other components. Similarly, Habernal et al. (2018) present the task of identifying the correct warrant from two options, and Rajendran et al. (2016) aim to generate the premise connecting an aspect-related opinion to an overall opinion. Instead of missing premises, we aim to synthesize (parts of) an argument's conclusion.
For any text generation task, a candidate technique is sequence-to-sequence models (Sutskever et al., 2014). Relevant in the given context, Hua and Wang (2018) used such models to generate counter-arguments, and Hua et al. (2019) extended this approach by planning and retrieval mechanisms. With a comparable intention, Chen et al. (2018) modified the bias of news headlines from right-to-left or vice versa. Closest to our work is the approach of Wang and Ling (2016) whose sequence-to-sequence model generates summaries for opinionated and argumentative text. Like us, the authors face the problem of varying numbers of input components, and tackle this using an importance-based sampling method. For their evaluation, they crawled arguments from idebate.org. We use this dataset in our experiments. Unfortunately, their manual evaluation considers opinionated text only, leaving the semantic adequacy of the generated argument summaries unclear.
The exact connection to summarization is unclear, which is why we include an approximation of the model of Wang and Ling (2016) as a baseline in our experiments. General research on summarization is manifold and beyond the scope of this work. For a survey, we refer the reader to Gambhir and Gupta (2017). In recent work, we summarize the core of an argument to be used as a snippet in the context of argument search by a two-sentence extract (Alshomary et al., 2020) and Egan et al. (2016) create abstractive summaries of the main points in a debate. We hypothesize a dependency between the target and stance of a conclusion and
those of the premises. At a high level, this resembles the work of Angelidis and Lapata (2018) where aspects and sentiments are modeled for the extractive summarization of opinions.
We focus on the inference of conclusion targets in this work. Our approach builds upon ideas of Bar-Haim et al. (2017), who classify the stance of premises to a conclusion. To do so, they identify and relate targets in these components, and model stance with sentiment. We do not explicitly tackle stance inference here, because our focus is a conclusion's target. To identify premise targets, we first train a state-of-the-art sequence tagger using contextualized word embeddings (Akbik et al., 2018) on the corpus of Bar-Haim et al. (2017). From these premise targets, we then infer the conclusion target, as explained below.
Before discussing our target inference approach in Section 4, this section briefly introduces the datasets that we use in our analyses and experiments. To allow for evaluating the given task, the conclusion is always given in these datasets.
3.1 Wikipedia Claims with Targets
The Claim Stance Dataset (Bar-Haim et al., 2017) contains 2,394 claims referring to 55 topics from Wikipedia articles. Not only the stance of premises towards their topics is manually annotated, also a phrase is marked in each claim as being a target. We use this dataset to train and evaluate a target phrase tagging model for the purpose of identifying targets in the given premises of an argument. As Bar-Haim et al., we take all premises associated to 25 conclusions for training and the rest for testing.
3.2 Debate Portal Conclusions
The iDebate Dataset (Wang and Ling, 2016) consists of 2,259 pro and con points for 676 controversial issues from the online debate portal idebate.org. Each point comes with a one-sentence conclusion (called central claim by the authors) and an argumentative text supporting the conclusion. Each sentence is seen as one premise of the conclusion (called argument), resulting in a total of 17,359 premises. We use this dataset for training, optimizing, and evaluating all approaches to conclusion target inference. Following its authors, we split the dataset based on debates: 450 debates for training, 67 for validation, and 150 for testing.
3.3 Essay Theses and Conclusions
The Argument Annotated Essays corpus (Version 2; Stab and Gurevych (2014)) includes 402 persuasive student essays. Each essay was segmented manually into subsentence-level argument components: theses (called major claims), conclusions (claims), and premises. We use this corpus to study target inference in a second domain. To analyze different types of argument relations, we derive two datasets from the corpus: Essay Conclusions for conclusions and their premises with 1,530 training, 256 validation, and 234 test cases, and Essay Theses for theses and the underlying conclusions with 300 training, 50 validation, and 52 test cases.
We now present our approach to infer the target of an argument's conclusion from its premises. Based on a premise target identifier, it employs two complementary sub-approaches: One ranks premise targets by their potential representativeness for the (later unknown) conclusion, and then picks the top-ranked premise target. The other predicts candidate embeddings for the conclusion target from the top-ranked premise targets, and then picks the conclusion target from a knowledge base of targets whose embedding is most similar to those embeddings.
4.1 Premise Target Identification
To model the relation between premises and conclusion target, we first identify the premises' targets. The task of identifying target phrases in argumentative text has been introduced by Bar-Haim et al. (2017). We here tackle it as BIO sequence labeling, classifying each token as being the beginning, inside, or outside of a target. Since premise target identification is not our main focus, we simply train a state-of-the-art neural sequence tagger (Ak-bik et al., 2018) on the claim stance dataset and then use it to automatically annotate targets in all input premises.
4.2 Inference by Premise Target Ranking
A reasonable hypothesis is that one of the premise targets of an argument represents an adequate conclusion target. Our first sub-approach thus simplifies the given task into selecting the premise target that most likely represents the conclusion target.
Figure 2: Percentage of training arguments in the given datasets where the conclusion target matches any of the premise targets, assuming a match when either a certain minimum token overlap (solid lines) or some embedding cosine similarity (dashed lines) is given.
Since there is no training data that reflects this likelihood, we follow the idea of importance sampling of Wang and Ling (2016): Given the output of our target identifier on a training instance, we use the percentage of content tokens overlapping between premise targets and the conclusion target as a representativeness label (quantified as Jaccard distance). Then, we learn a ranking model to predict the representativeness of a candidate premise target based on four features:
1. The average embedding cosine similarity of the candidate to the other candidates,
2. the number of words in the candidate,
3. the relative start and end character position of the candidate in the covering premise, and
4. the number of sentiment words (positive, negative, and neutral) in that premise.
The input of the ranking model are premise targets grouped by argument. During training, a probability is learned to reflect the ordering between each pair of premise targets in an argument with respect to conclusion target representativeness. Then, the model utilizes a cross-entropy loss function to minimize the difference between learned and the desired probability.
The effectiveness of this approach is naturally limited by the percentage of cases where the conclusion target actually matches any premise target.
2Despite domain differences to the other datasets, we see in Section 5 that the tagger works rather reliably across datasets.
Figure 3: Sketch of the target embedding space transformation. The distance from the averages  $\mathbf{s}_1, \mathbf{s}_2, \ldots$  of the premise targets to the correct conclusion target  $\mathbf{c}$  is minimized, the distance to other targets  $\mathbf{c}'$  maximized.
For a rough estimation, Figure 2 shows, based on two different similarity measures, how often at least one premise target matches the conclusion target in the three given training sets. Naturally, it is unclear in general how high the similarity needs to be for actual semantic equivalence.
4.3 Inference by Target Embedding Learning
To overcome the outlined shortcoming of being restricted to premise targets, we investigate a second hypothesis: An adequate conclusion target can be found in other arguments. To this end, we integrate a neural model with a knowledge base of targets in a novel way.
In particular, our second sub-approach tackles the given task by producing candidate conclusion target embeddings from the (top-ranked) premise targets, and then picking the target from a knowledge base whose embedding is most similar to the candidates. In principle, the knowledge base can be built from any corpus of argumentative texts based on our target identifier. In our experiments, we simply use all conclusion targets extracted from the training split of the datasets.
Now, to predict a conclusion target embedding, we first get the top  $k > 1$  premise targets using our ranking approach and create average embeddings  $\mathbf{s}_1, \mathbf{s}_2, \ldots$  of all  $\binom{k}{m}$  possible subsets of these targets with  $m > 1$ . Then, we learn a function  $f$  on training arguments that maps each  $\mathbf{s}_i$  to a transformed embedding space where it resembles the correct conclusion target  $\mathbf{c}$  and differs more from other targets  $\mathbf{c}'$ . Figure 3 sketches this idea. The best  $k$  and  $m$  are found by tuning in validation.
As depicted in Figure 4, we model  $f$  as a triplet neural network (Hoffer and Ailon, 2015) with three vectors as an input: an anchor  $\mathbf{s}_i$ , a positive  $\mathbf{c}$ , and a
Figure 4: Our approach to learn conclusion target embeddings. The triplet neural network makes the average embedding  $\mathbf{s}$  of a subset of the premise targets similar to the correct embedding  $\mathbf{c}$ , and dissimilar to others.
negative  $\mathbf{c}^{\prime}$ , where  $\mathbf{c}^{\prime}$  is a randomly sampled target from the target knowledge base. During training, we create  $\binom{k}{m}$  triplets from each argument. Based on these, we utilize the following triplet loss function to minimize the cosine distance  $d$  between  $\mathbf{s}_i$  and  $\mathbf{c}$ , and to maximize  $d$  between  $\mathbf{s}_i$  and  $\mathbf{c}^{\prime}$ :
$$
\max \left\{d (f (\mathbf {s} _ {i}), f (\mathbf {c})) - d (f (\mathbf {s} _ {i}), f (\mathbf {c} ^ {\prime})) + d _ {m a x}, 0 \right\}
$$
Here,  $d_{max}$  represents the maximum distance to be considered, also determined during validation.
During prediction, we employ the trained network to map the average embeddings  $\mathbf{s}_1, \mathbf{s}_2, \ldots$  of all premise target subsets to the transformed embedding space, and compute the average  $avg(f(\mathbf{s}_i))$  of all mapped embeddings  $f(\mathbf{s}_i)$ . Then, we pick the conclusion target  $\mathbf{c}$  from the knowledge base whose mapped embedding  $f(\mathbf{c})$  has the minimum cosine distance to  $avg(f(\mathbf{s}_i))$ . This way, we ensure that we always end up with a meaningful target. Figure 5 sketches the conclusion target inference on the left and exemplifies it on the right.
4.4 A Hybrid of Both Sub-Approaches
The reasonableness of the conclusion target inferred by the second sub-approach depends on the quality of the knowledge base. To avoid inferring fully unrelated targets, we also consider a simple hybrid of our two approaches below: If the target inferred by the embedding learning approach overlaps with the (full) text of any premise in at least one content token, it is taken. Otherwise, the target inferred by the premise ranking is taken. More elaborated heuristics are left to future work.
Figure 5: Sketch of inferring a conclusion target from an argument's premises. Given a knowledge base of candidates, the target is chosen whose learned embedding  $f(\mathbf{c})$  is closest to the learned average  $avg(f(\mathbf{s}_i))$  of premise targets. An example is shown on the right.
In this section, we report on empirical experiments, along with their results, performed to evaluate our approaches to target inference.
5.1 Premise Target Identification
We implemented the target identifier as a BiLSTM-CRF with hidden layer size 256, using the pretrained contextual string embedding model of Akbik et al. (2018). We trained the model on the training set of the Claim Stance Dataset with batch size 16 and a learning rate of 0.1 for five epochs.
Results On the Claim Stance test set, the identifier achieved an F1-score of 0.77. To assess its effectiveness in other domains, we let human annotators evaluate the identified targets of a random sample of 100 conclusions from the iDebate dataset. Each instance was evaluated by three annotators. Based on the majority agreement, the tagger identified  $72\%$  of the cases correctly. $^3$
5.2 Conclusion Target Inference
To evaluate target inference, we use the iDebate Dataset and the two essay datasets. As no groundtruth conclusion targets are provided, we used our target identifier to extract targets from the conclusions and compared them to the output of our approaches. In some cases, particularly where targets
were not explicitly phrased, our target identifier did not annotate any token. Hence, we eliminated those cases from the test set. $^4$
Approaches For the premise target ranking approach, we trained LambdaMART (Burges, 2010) on each training set with 1000 estimators and a learning rate of 0.02. We refer to this approach below as Premise Targets (ranking).
For target embedding learning, we used the pretrained FastText embeddings with 300 dimensions (Bojanowski et al., 2017) to initially represent each target. To obtain a knowledge base of candidate targets, we applied the target identifier to all conclusions of all training sets. The resulting lexicon contains 1,780 targets, each is represented by its FastText embedding. We implemented the triplet neural network as three feed-forward neural networks, each with two layers and shared weights. We call this approach Target Embedding (learning).
The simple hybrid of both approaches introduced above is denoted Hybrid (ranking & embedding).
Baselines On one hand, we compare to the state-of-the-art sequence-to-sequence argument summarizer of Wang and Ling (2016). Since its code is not available, we approximately reimplemented it. Specifically, we replicated the importance sampling with the same features (also on five premises) but no regularization. For generation, we used three LSTM layers with hidden size 150 and a pretrained embedding of size 300. Extra features of the original approach were left out, as they did not help much in our case. We trained the model with batch size 48 and learning rate 0.1 using the Adagrad optimizer (Duchi et al., 2011). For translation, we followed Wang and Ling. To identify targets in the generated summaries, we employed our target identifier. We refer to this baseline as Seq2Seq.