SlowGuess/ACL_2020_OCR huggingface dataset:
To test our hypothesis on the relation of premise and conclusion targets, we extended Seq2Seq by a pointer generator (See et al., 2017) and an extra binary feature that encodes whether a token belongs to a target or not, allowing the model to learn this relation. We call this Seq2Seq (w/ premise targets).
On the other hand, we complemented our approaches with simpler variants, in order to check whether learning is needed. Instead of premise tar
<sup>3</sup>In terms of Fleiss'  $\kappa$ , the agreement was 0.39, which is not high but still seems reasonable, given that we did not train annotators. Notice that this agreement value has no effect at all on the evaluation of our target inference approaches below.
Example conclusion where no target was identified: "It makes it more difficult for extremists to organize and spread their message when blocked".
5 More elaborated knowledge bases are left to future work.
The authors did not respond to our requests.
#	Approach	Scenario	iDebate dataset	Essay Conclusions	Essay Theses
bleu	meteor	accur.	bleu	meteor	accur.	bleu	meteor	accur.
b1	Seq2Seq	-	0.7	0.01	0%	-	-	-	-	-	-
b2	Seq2Seq (w/ premise targets)	-	4.4	0.07	5%	-	-	-	-	-	-
b3	Premise Targets (random)	-	3.9	0.11	8%	2.2	0.09	3%	8.8	0.19	17%
b4	Target Embedding (average)	Optimistic	7.2	0.16	18%	8.3	0.12	8%	15.3	0.24	21%
Pessimistic	6.4	0.15	17%	4.1	0.12	6%	15.3	0.24	21%
a1	Premise Targets (ranking)	-	9.7	0.16	17%	4.1	0.11	5%	17.3	0.25	24%
a2	Target Embedding (learning)	Optimistic	9.2	0.15	18%	8.3	0.12	8%	27.9	0.29	27%
a2	Pessimistic	7.2	0.13	16%	3.4	0.09	5%	13.6	0.23	21%
a1&a2	Hybrid (ranking & embedding)	Optimistic	10.0*	0.16	20%*	8.2	0.13	8%	27.9	0.29	27%
Pessimistic	8.1	0.15	18%	3.4	0.10	5%	13.6	0.23	21%
Oracle (theoretic upper bound)	Optimistic	94.3	0.85	100%	98.9	0.95	100%	98	0.90	100%
Pessimistic	35.8	0.58	65%	34.2	0.59	49%	26	0.52	48%
Table 1: Effectiveness of the evaluated target inference approaches in terms of BLEU, METEOR, and accuracy on the test sets of the iDebate dataset and the two essay datasets. The best value in each column is marked bold. Values of  $a1 \& a2$  marked with * are significantly better than the best baseline  $b4$  at  $p < 0.05$  (student  $t$ -test). The bottom rows show the effectiveness of an oracle that selects those conclusion targets, which maximize each score.
get ranking, our baseline Premise Targets (random) simply chooses a premise target randomly. Instead of target embedding learning, we simply pick the target from the target space whose embedding is most similar to the average premise target embedding, called Target Embedding (average).
Measures We use two common complementary evaluation measures, BLEU (Papineni et al., 2002) and METEOR (Lavie and Agarwal, 2007). BLEU counts n-gram matches (we include 1- and 2-grams) focusing on precision, while METEOR is recall-oriented. Following the idea of Figure 2, we also report accuracy, where a given target is correct if it has  $50\%+$  content overlap with the ground truth.
Experiments We tuned all approaches on the respective validation sets, and then evaluated them on the test set. Since Seq2Seq requires much training data, we evaluated both variants on iDebate only.
Before the inference of Target Embedding (learning), the corresponding premise targets were added to the knowledge base as candidates for a conclusion target. Below, we consider two scenarios, an optimistic and a pessimistic one: In the former, the ground-truth target is added to the knowledge base, in the latter not. The optimistic scenario thus reflects the effectiveness of the approach regardless of the limitations of the knowledge base.
Results Table 1 lists the results. Clearly, encoding premise targets into Seq2Seq boosts its effectiveness, indicating the importance of modeling premise targets. However, both Seq2Seq variants perform poorly compared to our approaches. While
Figure 6: Histogram of the number of arguments with a specific number of premises in the three given datasets.
the limited training data size is one reason, this also indicates that pure sequence-to-sequence generation may not be enough.
On iDebate, both approaches are better than all baselines in terms of BLEU score. The best results are achieved by Hybrid (ranking & embedding) in terms of all measures (significantly for BLEU and accuracy). Even in the pessimistic scenario, its BLEU score of 8.1 outperforms all baselines.
In the optimistic scenario on the essay datasets, Target Embedding (learning) is strongest for most scores. The hybrid approach hardly achieves any improvement. Due to the small dataset size, no significance was found, though. In the pessimistic scenario, Premise Target (ranking) seems more suitable. The lower scores on Essay Conclusions can be attributed to the low number of premises (see Figure 6), which makes finding an adequate conclusion target among the premise targets less likely.
#	iDebate dataset	Essay Conclusions	Essay Theses
New	Exact	New	Exact	New	Exact
b4	5%	6%	3%	2%	0%	6%
a1	0%	9%	0%	1%	0%	9%
a2	24%	7%	25%	6%	12%	12%
a1&a2	9%	8%	15%	6%	12%	12%
Table 2: Percentage of test cases where each approach picked a new target (not a premise target) and where the picked target is an exact match of the ground-truth target. The highest value in each column is marked bold.
(a) Premise targets	how to use the mobile phone
Phones
Having a mobile phone
the internet phones
Conclusion target	Mobile phones	Phones	Mobile Phones
Ground-truth	Inference of a1	Inference of a2
(b) Premise targets	Relocating to the best universities
Improving the pool of students
Online courses
Stanford University's online course on Artificial Intelligence
Conclusion target	Online courses	Online courses	distance-learning
Ground-truth	Inference of a1	Inference of a2
(c) Premise targets	saving the use of that kinds of languages
in this case
to be respected and preserved
language
Conclusion target	the government	language	language acquisition
Ground-truth	Inference of a1	Inference of a2
As Table 1 shows, all approaches are much worse than theoretically possible (oracle) in terms of automatic metrics. However, the manual evaluation below reveals that the inferred conclusion targets actually compete with the ground truth.
Analysis To illustrate the behavior of selected approaches, Table 2 compares the percentages of cases where they pick a new target as well as where they pick the exact ground-truth conclusion target (in the optimistic scenario). Befittingly, target embedding learning  $(a2)$  is most "exploratory" regarding new targets. On the essay datasets, where the conclusion target only sometimes occurs in the premises,  $a2$  is also best in inferring the exact target. Still, premise target ranking  $(a1)$  may pick the ground truth, if it matches any premise target. The hybrid seems a suitable balance between both.
Figure 7(a) exemplifies the ability of  $a2$  to infer the correct conclusion target even if it does
Figure 7: Three examples of premise targets from the datasets, the associated ground-truth conclusion target, and the conclusion targets inferred by our approaches.
#	Scenario	Fully	Somewhat	Not	Majority
b2	-	5%	18%	76%	93 / 100
a1	-	56%	33%	11%	91 / 100
a2	Optimistic	50%	28%	22%	92 / 100
Pessimistic	49%	27%	24%	93 / 100
a1&a2	Optimistic	55%	34%	11%	89 / 100
Pessimistic	56%	32%	12%	90 / 100
Ground-truth	62%	29%	10%	84 / 100
Table 3: Majority agreement for how adequate (fully, somewhat, not) are the conclusion targets of baseline  $b2$ , our approaches, and the ground truth. The right column lists the number of cases where majority is given.
not match a premise target exactly. Example (b) stresses the limitation of automatic evaluation: “distance-learning” (inferred by  $a2$ ) does not overlap with the ground truth, but it semantically matches well. In (c), the ground-truth target was barely inferable from the premise targets.[7]
6 Manual Evaluation
To assess the actual quality of the inferred conclusion targets, we manually evaluated our approaches (optimistic and pessimistic scenario) and the baseline  $b2$  (Seq2Seq (w/ premise targets)) in comparison to the ground-truth targets using Amazon Mechanical Turk. For this, we sampled 100 random instances from the iDebate test set. In a single task, an argument's premises were given along with the conclusion target of either approach. Annotators had to judge the adequacy of the target for the given premises as fully, somewhat, or not adequate. Each instance was judged by five annotators. No one judged multiple targets for the same argument.[8]
Table 3 shows the distribution of majority judgments for each approach. Only  $23\%$  of the  $b2$  targets were considered fully or somewhat adequate, i.e., pure text generation seems insufficient. In contrast, our sub-approaches' targets are competitive to the ground truth, which was not always adequate either (likely due to errors in target identification). The high performance of  $a1$  (Premise Targets (ranked)) might be explained by the inferred targets being part of the premises, affecting annotators' preferences. Still, the targets of  $a2$  (Target Embedding (learning)) are seen as adequate in  $78\%$  of the cases ( $50\%$  fully), with the ability of infer
${}^{7}$  Full example arguments found in supplementary material.  ${}^{8}$  We paid  $\$ {0.40}$  per task,restricting access to annotators with an approval rate of at least 95% and 5000 approved tasks. To ensure correct annotations, a reason had to be given.
ring conclusion targets that are not explicitly stated in the premises. Even in the pessimistic scenario, the inferences of  $a1$  and  $a1 \& a2$  remain stable.
An argument's conclusion comprises its stance towards the target it discusses. Still, the conclusion is often left implicit in real life, because it is clear for humans or hidden for rhetorical reasons. We have conceptualized the task of reconstructing the conclusion from the argument's premises as (1) inferring the conclusion's target, (2) inferring its stance, and (3) phrasing its actual text. Then we have focused on the first step in which we infer the conclusion target given a set of premises.
Hypothesizing that the conclusion target depends on the premise targets, we have developed two new and complementary target inference approaches: Premise Targets (ranking) returns the premise target that is most likely adequate for the conclusion, while Target Embedding (learning) generates a conclusion target embedding from the premises and matches it against a target knowledge base.
On three datasets from two domains (debate portals and student essays), our approaches outperform several baselines, including a state-of-the-art neural sequence-to-sequence summarizer. The latter also benefits from modeling premise targets, additionally supporting our hypothesis. In terms of BLEU, METEOR, and accuracy, Target Embedding (learning) and a hybrid of both approaches turned out particularly strong, whereas Premise Targets (ranking) was best in a manual evaluation. Overall, we manage to infer an at least somewhat adequate conclusion target in  $89\%$  of all cases, indicating the practical applicability of our approaches.
Combining target inference with stance classification in future work, we can already generate basic conclusions, say, "Raising the school leaving age is good". A more elaborate phrasing approach may take over context information from the premises.
This work was partially funded by the German Research Foundation (DFG) within the collaborative research center "On-The-Fly Computing" (SFB 901/3, project no. 160364472). We thank students from Paderborn University for evaluating our target identifier: Denis Kuchelev, Christin Loe, Natalie Lüke, Avishek Mishra, Enri Ozuni, René Scherf, Harsh Shah, Nikit Srivastava, Martin Wegmann.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638-1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Khalid Al Khatib, Henning Wachsmuth, Johannes Kiesel, Matthias Hagen, and Benno Stein. 2016. A news editorial corpus for mining argumentation strategies. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3433-3443. The COLING 2016 Organizing Committee.
Milad Alshomary, Nick Dusterhus, and Henning Wachsmuth. 2020. Extractive snippet generation for arguments. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, to appear.
Stefanos Angelidis and Mirella Lapata. 2018. Summarizing opinions: Aspect extraction meets sentiment prediction and they are both weakly supervised. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3675-3686, Brussels, Belgium. Association for Computational Linguistics.
Roy Bar-Haim, Indrajit Bhattacharya, Francesco Dinuzzo, Amrita Saha, and Noam Slonim. 2017. Stance classification of context-dependent claims. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 251-261, Valencia, Spain. Association for Computational Linguistics.
Yonatan Bilu and Noam Slonim. 2016. Claim synthesis via predicate recycling. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 525-530, Berlin, Germany. Association for Computational Linguistics.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Filip Boltuzic and Jan Šnajder. 2016. Fill the gap! Analyzing implicit premises between claims from online debates. In Proceedings of the Third Workshop on Argument Mining (ArgMining2016), pages 124-133. Association for Computational Linguistics.
Christopher J.C. Burges. 2010. From RankNet to LambdaRank to LambdaMART: An overview.
Jill Burstein and Daniel Marcu. 2003. A machine learning approach for identification of thesis and conclusion statements in student essays. Computers and the Humanities, 37(4):455-467.
Wei-Fan Chen, Henning Wachsmuth, Khalid Al Khatib, and Benno Stein. 2018. Learning to flip the bias of news headlines. In Proceedings of the 11th International Conference on Natural Language Generation, pages 79-88. Association for Computational Linguistics.
John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121-2159.
Charlie Egan, Advaith Siddharthan, and Adam Wyner. 2016. Summarising the points made in online political debates. In Proceedings of the Third Workshop on Argument Mining (ArgMining2016), pages 134-143, Berlin, Germany. Association for Computational Linguistics.
Roxanne El Baff, Henning Wachsmuth, Khalid Al Khatib, Manfred Stede, and Benno Stein. 2019. Computational argumentation synthesis as a language modeling task. In Proceedings of the 12th International Conference on Natural Language Generation, pages 54-64, Tokyo, Japan. Association for Computational Linguistics.
Mohammad Hassan Falakmasir, Kevin D Ashley, Christian D Schunn, and Diane J Litman. 2014. Identifying thesis and conclusion statements in student essays to scaffold peer review. In International Conference on Intelligent Tutoring Systems, pages 254-259. Springer.
James B. Freeman. 2011. Argument Structure: Representation and Theory. Springer.
Mahak Gambhir and Vishal Gupta. 2017. Recent automatic text summarization techniques: a survey. Artificial Intelligence Review, 47(1):1-66.
Nancy L. Green. 2017. Argumentation scheme-based argument generation to support feedback in educational argument modeling systems. International Journal of Artificial Intelligence in Education, 27(3):515-533.
Ivan Habernal and Iryna Gurevych. 2015. Exploiting debate portals for semi-supervised argumentation mining in user-generated web discourse. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2127-2137. Association for Computational Linguistics.
Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. The argument reasoning comprehension task: Identification and reconstruction of implicit warrants. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1930-1940, New Orleans, Louisiana. Association for Computational Linguistics.
Elad Hoffer and Nir Ailon. 2015. Deep metric learning using triplet network. In International Workshop on Similarity-Based Pattern Recognition, pages 84-92. Springer.
Xinyu Hua, Zhe Hu, and Lu Wang. 2019. Argument generation with retrieval, planning, and realization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2661-2672, Florence, Italy. Association for Computational Linguistics.
Xinyu Hua and Lu Wang. 2018. Neural argument generation augmented with externally retrieved evidence. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 219-230. Association for Computational Linguistics.
Fattaneh Jabbari, Mohammad Hassan Falakmasir, and Kevin D. Ashley. 2016. Identifying thesis statements in student essays: The class imbalance challenge and resolution. In Proceedings of the Twenty-Ninth International Florida Artificial Intelligence Research Society Conference, pages 220-225.
Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228-231, Prague, Czech Republic. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Andreas Peldszus and Manfred Stede. 2015. Joint prediction in MST-style discourse parsing for argumentation mining. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 938-948. Association for Computational Linguistics.
Pavithra Rajendran, Danushka Bollegala, and Simon Parsons. 2016. Contextual stance classification of opinions: A step towards enthymeme reconstruction in online reviews. In Proceedings of the Third Workshop on Argument Mining (ArgMining2016), pages 31-39. Association for Computational Linguistics.
Paul Reisert, Naoya Inoue, Naoaki Okazaki, and Kentaro Inui. 2015. A computational approach for generating Toulmin model argumentation. In Proceedings of the 2nd Workshop on Argumentation Mining, pages 45-55. Association for Computational Linguistics.
Misa Sato, Kohsuke Yanai, Toshinori Miyoshi, Toshihiko Yanase, Makoto Iwayama, Qinghua Sun, and
Yoshiki Niwa. 2015. End-to-end argument generation system in debating. In Proceedings of ACLIJCNLP 2015 System Demonstrations, pages 109-114. Association for Computational Linguistics and The Asian Federation of Natural Language Processing.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Christian Stab and Iryna Gurevych. 2014. Identifying argumentative discourse structures in persuasive essays. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 46-56. Association for Computational Linguistics.
Manfred Stede and Jodi Schneider. 2018. Argumentation Mining. Number 40 in Synthesis Lectures on Human Language Technologies. Morgan & Claypool.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS'14, pages 3104-3112, Cambridge, MA, USA. MIT Press.
Stephen E. Toulmin. 1958. The Uses of Argument. Cambridge University Press.
Henning Wachsmuth, Manfred Stede, Roxanne El Baff, Khalid Al Khatib, Maria Skeppstedt, and Benno Stein. 2018. Argumentation synthesis following rhetorical strategies. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3753-3765. Association for Computational Linguistics.
Douglas Walton, Christopher Reed, and Fabrizio Macagno. 2008. Argumentation Schemes. Cambridge University Press.
Lu Wang and Wang Ling. 2016. Neural network-based abstract generation for opinions and arguments. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 47-57, San Diego, California. Association for Computational Linguistics.
Toshihiko Yanase, Toshinori Miyoshi, Kohsuke Yanai, Misa Sato, Makoto Iwayama, Yoshiki Niwa, Paul Reisert, and Kentaro Inui. 2015. Learning sentence ordering for opinion generation of debate. In Proceedings of the 2nd Workshop on Argumentation Mining, pages 94-103, Denver, CO. Association for Computational Linguistics.
Ingrid Zukerman, Richard McConachy, and Sarah George. 2000. Using argumentation strategies in automated argument generation. In INLG'2000 Proceedings of the First International Conference on Natural Language Generation, pages 55-62, Mitzpe Ramon, Israel. Association for Computational Linguistics.
Supplementary material: Example arguments with inferred conclusion targets
Argument: Relocating to the best universities is a budgetary concern, but also family and social relations concern for many people, which prevents all the best people from even applying to universities that would suit them the best. Online courses can recruit students from anywhere in the world much easier than traditional universities can because students do n't need to travel far away for the best education. This then ensures that universities have better access to the brightest people. For instance, Stanford University's online course on Artificial Intelligence enabled people from 190 countries to join, and none of students receiving a score of 100 percent where from Stanford -LSB-14-RSB-. Improving the pool of students would automatically result in better academics, professionals and science, which would benefit the society better.
Ground truth conclusion: Online courses are a way to higher academic excellence
Premise Targets (ranked): Online courses
Target Embedding (learning): distance-learning
Argument: Having a mobile phone helps us to learn in a lot of different ways. First we learn about technology; about how to use the mobile phone. Second most phones today have apps -LRB- programs -RRB- to enable learning using the phone, or else through the internet. Phones can access online courses and lessons which can be provided in fun ways and can in some cases instantly tell you if you have the right answer. It may even sometimes be possible to do homework on a phone and send it to your teacher. Even without the internet phones can be used to provide short assignments, or to provide reminders to study.
Ground truth conclusion: Mobile phones help us to learn
Premise Targets (ranked): Phones
Target Embedding (learning): Mobile phones
Argument: students who used to prepare Microsoft PowerPoint presentation for their school projects, get an edge over others at an early stage of their career When children are allowed to play around with computer from a very early age, they get acquainted with the previously mentioned skills and become expert before facing professional world computers enable people to prepare presentations, draw complex graphs and pictures, document thesis in a simple though efficient way
Ground truth conclusion: it's clear that computer has a positive effect on the children
Premise Targets (ranked): students who used to prepare Microsoft PowerPoint presentation for their school projects