SlowGuess/ACL_2020_OCR huggingface dataset:
To maximize its usefulness, the sample of arguments for our argument quality corpus was drawn from the recently published args.me corpus (Stein
and Wachsmuth, 2019), a collection of 387,606 arguments crawled from various debate portals. To ensure some topic diversity and relevance of the arguments to the topic, while keeping the amount of judgments within our budget limits, we (1) indexed the args.me corpus using three retrieval models from the Terrier information retrieval library (Ounis et al., 2006) (namely BM25, DPH, and DirichletLM), (2) retrieved texts for 20 topic queries at a depth of 50 texts per topic per model, (3) and pooled all 3000 retrieved texts to remove overlap between the different models. In total, 1,610 unique spans of text remained for the 20 topics.
Using Amazon's Mechanical Turk, in a first preprocessing step, we tasked crowd workers with deciding whether or not a given retrieved item actually contained argumentative text. Each text was judged by five crowd workers, using majority vote as a decision rule. To ensure quality, we recruited only workers for the task with an approval rate of at least  $95\%$ , like Swanson et al. (2015).
Of the 1,610 input texts, 339 were flagged as non-arguments. Most of these texts are noise resulting from using debate platforms as data source; examples include statements of acceptance for a debate ("I accept this debate on [Topic] and will go first [...])", statements with no argumentative value ("I think [Topic] is good"). definitions, and in some cases even jokes or personal attacks. 1,271 arguments remained for quality annotation.
In a second step, all remaining arguments were annotated for argument quality via a sample of pairwise judgments on which we applied our model. For each pair of arguments, a crowd worker was tasked to select the one that exhibits a higher quality compared to the other with regard to a given description of the respective quality dimensions. The annotation was repeated separately for each of 20 topics and each of the three aforementioned quality dimensions. To make the task accessible to workers without prior knowledge of argumentation theory, the quality dimensions were operationalized as follows: "Which text has the better logical structure?" (logical quality), "Which text has the better style of speech?" (rhetorical quality), and "Which text would be more useful in a debate?" (dialectical quality). Examples were given to annotators as guidance. Table 2 shows exemplary arguments for each of the three quality dimensions alongside a brief explanation why this argument lacks the specified quality. In each task, one such
Dimension	Argument	Explanation
Rhetorical quality	“Gender is a social construct cuse we are told when we are first born by a dude what gender but if he didnt tell us that we wouldnt have a gender its only cuse he told us that gender that we are that gender.”	This argument is of low rhetorical quality, as it lacks proper sentence structure, uses informal speech, has typos, and its use of ellipsis makes it hard to follow.
Logical quality	“I support an abortion ban. We must not forget that abortion opposes the principle of sanctity of life. Women are blessed with the gift of giving birth to another life and hence, should accept it with responsibility.”	Even though this argument has a clearly stated claim, the evidence used to support it is insufficient. Key con- cepts are not defined (What is ‘sanctity of life’? Why does it apply to unborn fetuses?) and the conclusion (‘Women should accept it with responsibility.’) does not necessarily follow from the evidence.
Dialectical quality	“Banning abortion would mean that there is more peo- ple in the world. This leads to overpopulation which is a major problem and 842 million people are under-nourished every year. More people only causes more problems.”	This argument is not very convincing since the evidence (overpopulation) presented in support of the conclusion (abortion ban) is not very relevant to the issue. It can easily be invalidated by, for example, offering better solutions to overpopulation than abortion. Thus, the argument does not make a meaningful contribution to resolving the debate conflict.
Table 2: Example arguments from the args.me corpus with accompanying explanation for why each argument lacks the specified quality.
negative example as well as one positive example was provided with explanations, ensuring no topic overlap between annotated material and example.
Five comparisons were presented together as one task. The comparison sets of five were compiled randomly to minimize order effects. A cyclic group comparison strategy as described in Section 3.5 with  $k = 8$  was employed, with each pair annotated by one worker. On average, a topic pooling consists of  $n = 64$  unique arguments, with  $c_{\text{sampled}} = 698$  and  $c_{\text{exhaustive}} = 2,043$ . The mean sample rate compared to the exhaustive comparison set therefore is 0.342. Erring on the safe side, we chose to maximize correlation with what can be expected from an exhaustive comparison as guided by Table 1. In total, 2,797 HITs were carried out. A reward of $0.08 per HIT was paid, amounting to $268.54 per quality dimension and $805.54 total while ensuring an hourly rate of at least $8.
In comparison to the setup of Habernal and Gurevych (2016), who carried out an exhaustive comparison with a factor of  $x = 5$  crowd workers per pair, the annotation effort for our study could be reduced by  $93.17\%$  based on our model. Nevertheless, if a higher accuracy is deemed necessary in future experiments, our comparison set can easily be extended by adding additional votes per comparison or by increasing the group size.
Compared to a traditional annotation setup using graded scales, having five workers annotate each item on a scale from 0 (low) to 4 (high) would put the total annotation cost for one quality dimension
at around  $150.00, supposing a reward of$ 0.08 per HIT. Although cheaper, the annotation quality would be much lower as per the reliabilities reported in previous work. Moreover, the highly increased level of detail in the quality scores produced by our new approach is worth the extra cost. In cases where annotation quality is not as important, sampling at higher values of  $k$  still achieves acceptable correlation scores, rendering our method even cheaper than the traditional approach.
5 Corpus Analysis
In this section, we carry out a statistical analysis of our new corpus. First, we study the distribution and the correlation effects between the different quality dimensions, and between the quality dimensions and text length, to draw comparisons to the prior work of Wachsmuth et al. (2017). Then, we explore the hypothesis of overall quality being a latent variable by analyzing the influence of quality dimensions.
5.1 Distribution
Distributions of scores for all three different quality dimensions are shown in Table 3a. Additionally, the distribution of text length in the corpus as well as scatterplots for text length and quality are given. All three dimensions exhibit a similar distribution, centered at zero. Given that the dummy item of the model, i.e., an item that is defined to have exactly one win and one loss against every other item, has a fixed score of 1, this indicates that the majority
5778
(b)
Dimension	Rhetorical	Logical	Dialectical
Our Expert	Our Expert	Our Expert
Rhetorical	0.63	0.81	0.63	0.81	0.61 0.75
Logical	0.55 0.78
Dialectical	0.61	0.75	0.55	0.78
Pro	Con	Pro	Con	Pro Con
Rhetorical	0.62	0.63	0.60 0.61
Logical	0.62	0.63	0.60 0.51
Dialectical	0.60	0.61	0.60	0.51
(c)
Dimension	Overall	l>100
Rhetorical	0.65	0.37
Logical	0.64	0.37
Dialectical	0.63	0.28
(d)
Step	Variance	Rhetorical	Logical	Dialectical
1	0.73	-0.5866	-0.5715	-0.5738
2	0.15	0.1050	0.6489	-0.7536
3	0.12	-0.8031	0.5023	0.3206
Sample Sizes
Overall	l > 100
1271	869
Pro	Con
675	596
Args.	Non-args.
1271	339
Table 3: (a) Distributions and scatterplots for quality scores and text length in the corpus. (b) Pearson  $\rho$  correlation coefficient cross-tabulation for different attribute combinations, full set and per stance. Expert values are taken from Wachsmuth et al. (2017), Table 3. Maximum per column in bold. (c) Correlation between quality dimensions and text length, full and only for texts longer than 100 words. (d) Component vectors and explained variance for PCA steps on argument quality. (e) Sample sizes for (b) and (c).
of texts in our corpus are only of mediocre argumentative quality. Also, all three distributions are slightly asymmetric, with the lower end extending more. The distribution of text lengths is fairly similar across all lengths, with only one apparent spike around 0-50 words.
5.2 Correlation
Table 3b shows correlation coefficients for the three quality dimensions when compared to each other, and when compared to argument stance. The interquality correlation as given by Wachsmuth et al. is also included, and found to be commensurate with their figures. Although the correlation is lower in total, which is expected given the much bigger sample size and different annotation methodology, the general pattern is reproduced, with one slight deviation: dialectical quality appears to correlate slightly more with rhetorical quality in our corpus, but with logical quality in their data. However, given that the two correlation coefficients are nearly equal in the data of Wachsmuth et al., and the value differ
ences between the different quality dimensions in our data are also too small to draw any conclusions regarding whether two of the three are more intertwined than the third, this effect is not problematic.
The correlation between the quality dimensions being fairly high hints at them being dependent on a latent variable, which could be the overall argumentation quality. When computing the scores separately for each stance, no systematic difference is apparent. Following the reasoning of Potthast et al. (2019), this may indicate that the scoring method is not prone to assessor bias.
A correlation of quality and text length (measured as word count), as also noted by Swanson et al. (2015), is evident (Table 3c). While this could hint at a data bias, with crowd workers just voting for longer texts in the comparison but not actually reading all of it, the effect is much less pronounced when only measuring the correlation in texts longer than 100 words ( $n = 869$ ). Thus, much of the pronounced effect can be explained by short texts receiving justified low scores rather
5779
than longer texts being voted higher regardless of content. From a qualitative point of view, a correlation effect between length and quality would also be expected, since a solid argumentative reasoning (claim and justification) usually requires at least some amount of text. The scatterplots in Table 3a additionally corroborate the correlation effects: only a very minor trend is apparent, with longer text receiving slightly higher scores. Given the accumulation of texts towards the lower end of the length spectrum, and these receiving lower scores further explains the lower overall correlation when only measuring in texts over 100 words.
5.3 Overall Argument Quality
For some applications a scalar value for overall argument quality is warranted. As it has been argued that an overall argument quality is hard to measure, the three different explored quality dimensions could be combined to derive such a rating. The high correlation of the different quality dimensions implies such a latent variable. As a working hypothesis, the overall argument quality could be interpreted as a three-dimensional vector in a space spanned by the three quality dimensions. Based on this, two essential questions have to be explored: (1) Are the different dimensions equally influential on the overall argument quality? (2) How can a scalar quality value for overall quality be derived from such a vector?
To address the first question, principal component analysis (PCA) was carried out to measure the influence of each quality dimension on the hypothesized latent variable. Results are given in Table 3d. The first step of the PCA accounts for  $73\%$  of the data variance, and is equally influenced by all three quality dimensions. Therefore, evidence is given towards the hypothesis. As for how to derive a numerical value for this overall argument quality, since the influence of all dimensions is equal, the euclidean vector length is proposed. However, since the quality scores derived in this work are positive as well as negative, the length of a vector is the same as that of its negative counterpart. To account for this, the score distributions are equally shifted into the positive domain. Thus, a standardized scalar value for overall argument quality can be calculated.
A novel approach for annotating argument quality based on stochastic transitivity modeling has been proposed, outperforming existing approaches in terms of annotation effort and annotation detail, while maintaining a high annotation quality. The overall workload in comparison to previous approaches within the same class of approaches was reduced by  $93.17\%$  through an efficient sampling method. Sampling at even higher rates is possible, resulting in the new framework operating at the same cost as the traditional approach relying on graded scales.
The collected data and a reference implementation of our model are made available in form of the Webis-ArgQuality-20 corpus, one of the largest and most detailed corpora for pairwise argument quality. The collected corpus can be used for a multitude of purposes—especially in the emerging field of argument retrieval, it is suitable as basis for retrieval evaluation, or to train new learning to rank models. A second field of application is debate systems, where a dataset can be of use for training a system to formulate new arguments. The developed annotation approach is also not only limited to rate argument quality: it can easily be transferred to other questions or criteria that can be rated by comparison. Even though the annotation cost can be slightly higher compared to the traditional absolute rating approach, the derived data is much more detailed and allows for conclusions with higher statistical power.
Insight into argument quality was derived on a larger scale than in previous studies. It has been shown that the three quality dimensions can be successfully annotated by laymen when using the described annotation procedure. The correlation patterns found in previous studies were reproduced, showing the quality dimensions to be equally correlating with each other. This is likely due to them being dependent on a latent overall quality, a hypothesis that was supported using a PCA analysis of derived quality vectors. A procedure to derive a scalar value for overall quality was introduced, proposing Euclidean vector length to combine the different dimension scores.
Alan Agresti. 2003. Categorical data analysis, 2 edition. John Wiley & Sons, Hoboken, NY.
Ralph Allan Bradley and Milton E. Terry. 1952. Rank analysis of incomplete block designs: The method of paired comparison. Biometrika, 39(3-4):324-345.
Michael L Burton. 2003. Too many questions? The uses of incomplete cyclic designs for paired comparisons. Field Methods, 15(2):115-130.
Xi Chen, Paul N. Bennett, Kevyn Collins-Thompson, and Eric Horvitz. 2013. Pairwise ranking aggregation in a crowdsourced setting. In Proceedings of the sixth ACM international conference on web search and data mining, WSDM '13, pages 193-202, New York, NY. ACM.
Ivan Habernal and Iryna Gurevych. 2016. Which argument is more convincing? Analyzing and predicting convincingness of web arguments using bidirectional LSTM. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1589-1599, Berlin, Germany. Association for Computational Linguistics.
David R. Hunter. 2004. MM algorithms for generalized Bradley-Terry models. The annals of statistics, 32(1):384-406.
Iadh Ounis, Gianni Amati, Vassilis Plachouras, Ben He, Craig Macdonald, and Christina Lioma. 2006. Terrier: A High Performance and Scalable Information Retrieval Platform. In Proceedings of ACM SIGIR'06 Workshop on Open Source Information Retrieval (OSIR 2006).
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The pagerank citation ranking: Bringing order to the web.
Thomas Pfeiffer, Xi Alice Gao, Yiling Chen, Andrew Mao, and David G Rand. 2012. Adaptive polling for information aggregation. In Twenty-Sixth AAAI Conference on Artificial Intelligence.
Leslie Gross Portney, Mary P Watkins, et al. 2009. Foundations of clinical research: Applications to practice, volume 892. Pearson/Prentice Hall, Upper Saddle River, NJ.
Martin Potthast, Lukas Gienapp, Florian Euchner, Nick Heilenkötter, Nico Weidmann, Henning Wachsmuth, Benno Stein, and Matthias Hagen. 2019. Argument Search: Assessing Argument Relevance. In 42nd International ACM Conference on Research and Development in Information Retrieval (SIGIR 2019). ACM.
P.V. Rao and Lawrence L. Kupper. 1967. Ties in paired-comparison experiments: A generalization of the Bradley-Terry model. Journal of the American Statistical Association, 62(317):194-204.
Edwin D. Simpson and Iryna Gurevych. 2018. Finding convincing arguments using scalable bayesian preference learning. Trans. Assoc. Comput. Linguistics, 6:357-371.
Benno Stein and Henning Wachsmuth, editors. 2019. 6th Workshop on Argument Mining (ArgMining 2019) at ACL. Association for Computational Linguistics, Berlin Heidelberg New York.
Reid Swanson, Brian Ecker, and Marilyn Walker. 2015. Argument mining: Extracting arguments from online dialogue. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 217-226, Prague, Czech Republic. Association for Computational Linguistics.
Assaf Toledo, Shai Gretz, Edo Cohen-Karlik, Roni Friedman, Elad Venezian, Dan Lahav, Michal Jacovi, Ranit Aharonov, and Noam Slonim. 2019. Automatic argument quality assessment - New datasets and methods. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 5624-5634. Association for Computational Linguistics.
Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberdingk Thijm, Graeme Hirst, and Benno Stein. 2017. Computational argumentation quality assessment in natural language. In 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2017), pages 176-187.
Ting Yan, Jinfeng Xu, and Yaning Yang. 2011. Grouped sparse paired comparisons in the Bradley-Terry model. arXiv preprint.
Peng Ye and David Doermann. 2013. Combining preference and absolute judgements in a crowd-sourced setting. In ICML Workshop.
5781