SlowGuess/ACL_2020_OCR huggingface dataset:
Lukas Gienapp<sup>1</sup> Benno Stein<sup>2</sup> Matthias Hagen<sup>3</sup> Martin Potthast<sup>1</sup>
1 Leipzig University
$^{2}$  Bauhaus-Universität Weimar
<sup>3</sup> Martin-Luther-Universität Halle-Wittenberg
We present an efficient annotation framework for argument quality, a feature difficult to be measured reliably as per previous work. A stochastic transitivity model is combined with an effective sampling strategy to infer high-quality labels with low effort from crowdsourced pairwise judgments. The model's capabilities are showcased by compiling Webis-ArgQuality-20, an argument quality corpus that comprises scores for rhetorical, logical, dialectical, and overall quality inferred from a total of 41,859 pairwise judgments among 1,271 arguments. With up to  $93\%$  cost savings, our approach significantly outperforms existing annotation procedures. Furthermore, novel insight into argument quality is provided through statistical analysis, and a new aggregation method to infer overall quality from individual quality dimensions is proposed.
For a broad variety of tasks, such as argument mining, argument retrieval, argumentation generation, and question answering, compiling labeled data for argument quality remains an important prerequisite, yet, also a difficult problem. Most commonly, human assessors have been presented with one argument at a time and then asked to assign labels on a graded quality scale  $\langle 0,1,2\rangle$  with label descriptions such as (0) "low quality", (1) "medium quality" and (2) "high quality" for guidance.
In previous work, this was usually done concurrently for multiple orthogonal sub-dimensions of argument quality; judging the overall quality of an argument has been deemed complex (Wachsmuth et al., 2017). But on closer inspection, even the more specialized quality dimensions considered are difficult to be assessed as evidenced by the low reliability scores reported. Especially crowdsourcing suffers from assessors often having different reference frames to base their judgments on and task instructions being nondescript and therefore
unhelpful in ensuring consistency. Employing experts, however, not only comes at a significantly higher cost per label; despite their expertise, even experts did not achieve more reliable judgments.
We pursue an alternative approach: stochastic transitivity modeling based on pairwise judgments of arguments. This enables the employment of laymen; the decisions required from them remain comparably simple and expect neither prior knowledge nor a common reference frame, while the labels that can be derived from their judgments still exhibit a high accuracy and informativeness. Though pairwise judgment has already been considered for assessment of argument quality (Habernal and Gurevych, 2016; Toledo et al., 2019), its significant cost overhead has hindered widespread application.
We explore the lower bound of effort needed to infer labels of sufficient quality. We combine a pairwise model with a highly effective offline sampling strategy to minimize the set of needed pairwise judgments, saving up to  $93\%$  of the effort of an exhaustive comparison. As part of this work, we release the Webis Argument Quality Corpus 2020, which includes a total of 41,859 pairwise judgments between 1,271 arguments across the three dimensions of rhetorical, logical, and dialectical quality. Further, inferred scalar scores for the three dimensions as well as overall quality and topic relevance are provided, alongside a reference implementation of our model. $^{1}$
Carrying out a first analysis of the statistical properties of the corpus, we validate both the new annotation method and the corpus by drawing comparisons to previous work. Since judging overall quality by itself is a difficult task, based on our statistical analysis, we find that euclidean vector length adequately combines scores from the three aforementioned specialized quality dimensions into a single overall quality score.
<sup>1</sup>Resources: https://webis.de/publications.html?q=ACL+2020
Corpus: https://zenodo.org/record/3780049
Code base: https://github.com/webis-de/ACL-20
5772
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5772-5781 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Wachsmuth et al. (2017) surveyed many facets of argument quality that are distinguished in argumentation theory, organizing them within three major dimensions: logical quality (the argument's structure and composition), rhetorical quality (persuasive effectiveness, vagueness, and style), and dialectical quality (contribution to the discourse). Further, they built the first comprehensive argument quality corpus, tasking three experts with annotating arguments with respect to all 15 (sub-)dimensions in their taxonomy. Each dimension has been annotated on a scale from 1 (low) to 3 (high), reaching Krippendorff's  $\alpha$  values between 0.26 and 0.51, depending on the quality dimension. Despite a rigorous setup, the low agreement is evidence that even experts have difficulties to reliably judge argument quality.
Potthast et al. (2019) explore the use of argument quality as an evaluation criterion beyond relevance for argument retrieval, thus needing to collect quality judgments for their evaluation task. Based on the taxonomy of Wachsmuth et al., they had the three major dimensions annotated on graded scales ranging from 1 (low) to 4 (high), reproducing the findings of Wachsmuth et al. They recruited highly educated students of at least bachelor's level education from a national foundation for gifted students who have a strong interest in societal issues. Still, reliable annotation was difficult to achieve due to the highly subjective, complex, and nuanced nature of argument quality.
Each study operates on rather small amounts of data; they only annotate 320 (Wachsmuth et al., 2017) and 437 (Potthast et al., 2019) individual arguments. Both setups become nonviable for larger annotation tasks, since the associated labor costs in such (semi-)expert studies are usually high.
This is not an issue in crowdsourced settings, where judgments can be collected in abundance for a comparatively cheap price. However, the problem of annotation quality is more severe here: argument quality might be even more difficult to judge without prior domain-specific knowledge, creating the need for annotation frameworks that can still maintain a sufficiently high data quality. Judging from the agreement scores given by Wachsmuth et al. and Potthast et al., obtaining reliable data using classic graded scales proves infeasible, an effect that should be even more pronounced in a crowdsourced setting.
Swanson et al. (2015) measure an arguments' quality as the amount of context or inference required for it to be understood, describing an annotation setup where assessors judge seven individual quality dimensions on a 0-1slider. Recruiting assessors on Amazon Mechanical Turk (MTurk), they use intra-class correlation to estimate inter-rater agreement, with an average value of 0.42 over all topics, thus also indicating a poor reliability (Portney et al., 2009). They further observe a correlation with sentence length, prompting them to remove all sentences shorter than four words.
All three studies indicate that absolute rating (i.e., having assessors label a single argument on a given absolute scale without the context of other arguments) performs unfavorably. This rating method, also known as Likert scale or Mean Opinion Score, is known to have two major drawbacks (Ye and Doermann, 2013): (1) Absolute rating is often treated as if it produces data on an interval scale. However, assessors rarely perceive labels as equidistant, thus producing only ordinal data. This leads to a misuse of statistical tests and results in low statistical power of subsequent analyses. (2) Absolute rating is difficult for assessors without prior domain knowledge, since they may be unsure which label to assign. This results in noisy, inconsistent, and unreliable data.
As an alternative, preference rating (i.e., a relative comparison by showing two arguments to an assessor and letting them declare their preference towards one of them) has been considered by Habernal and Gurevych (2016), who compile an exhaustive set of pairwise comparisons to infer labels for argument convincingness. For 1,052 arguments on 32 issues, each of the over 16,000 total comparisons was annotated by five different crowd workers on MTurk. While no  $\alpha$  statistics are provided, the authors do conclude that preference ratings in a crowdsourced setting are sufficiently accurate, since the best-ranked rater for each pair achieves 0.935 accuracy compared to a gold label.
The indicated reliability of pairwise annotation for argument quality is further corroborated by Toledo et al. (2019), who compile a large dataset of about 14,000 annotated argument pairs, and absolute ratings in the 0-1-range for about 6,300 arguments. Pairwise annotations were made in regard to the overall quality of arguments, operationalized as "Which of the two arguments would have been preferred by most people to support/contest the
5773
topic?" Using a strict quality control, they show that the annotated relations consistently reproduce the direction implied by absolute ratings. Yet, annotating quality as a single feature is problematic, since (1) it is hard to capture the multi-facet nature of argument quality in that way (Wachsmuth et al., 2017) and the chosen operationalization is similar to the facet of dialectical quality, neglecting the other major two; and (2) scores for individual quality dimensions are warranted for in-depth training and evaluation for a broad range of argumentation technology (Potthast et al., 2019).
Although preference rating seems promising based on the reported reliability, it creates the need for a model that infers score labels from the collected comparison data. Habernal and Gurevych propose the use of PageRank (Page et al., 1999). This is problematic, since cycles in the comparison graph may form rank sinks, distorting the latent rankings. Habernal and Gurevych deal with this problem by constructing a directed acyclic graph (DAG) from the collected data prior to applying PageRank, assuming that argument convincingness exhibits the property of total order. However, no prior evidence for this property is apparent. Simpson and Gurevych (2018) note further problems with PageRank and propose the use of Gaussian process preference learning instead, demonstrating a high scalability.
However, for a practical approach, an effective strategy to minimize the number of needed comparisons is warranted, since, to build the DAG, exhaustive comparison data is required. This is inefficient; at worst  $\binom{n}{2}$  comparisons have to be obtained for  $n$  arguments. Also, no data was collected on how the PageRank method performs on incomplete or sparse comparison data.
Chen et al. (2013) also propose an online sampling strategy based on the Bradley-Terry model (Bradley and Terry, 1952). They implement an online Bayesian updating scheme, which, contrary to previous work such as presented by Pfeiffer et al. (2012), does not require retraining the whole model when new comparisons are added. After each comparison added to the total set of annotated pairs, they identify the next pair to be compared by calculating which comparisons would reduce the overall model uncertainty the most. Simpson and Gurevych (2018) opt for a similar active learning approach, but note that that it is prone to overfitting, causing accuracy to decrease.
While online learning uses an approximately minimal amount of comparisons, additional drawbacks besides overfitting can be noted: (1) The updating scheme diminishes the reusability of the collected data, since such a specific method of choosing pairs introduces data bias for other applications. (2) Online sampling is complicated to implement on a crowdsourcing platform, preventing multiple workers from making judgments in parallel. (3) In the case of Chen et al. (2013), the model is not equipped to handle comparison ties, i.e., an assessor declaring no preference. Yet, ties frequently occur in real-world annotation tasks.
Overall, the Bradley-Terry model appears to be a promising candidate for our purposes: its robustness and statistical properties have been studied in great detail (Hunter, 2004), and it can be efficiently computed (Chen et al., 2013). However, an alternative offline sampling method has to be formulated, which we introduce in the following section.
3 Pairwise Quality Annotation
In this section, we define a model to aggregate pairwise judgments into scalar ranking scores and combine different sampling strategies to form a highly efficient annotation framework.
3.1 The Bradley-Terry Model
Let  $D = \{d_{1},\ldots ,d_{n}\}$  denote a set of  $n$  items (e.g., arguments) for which a latent ranking is assumed according to a scale-invariant set  $\Gamma = \{\gamma_1,\dots ,\gamma_n\}$  of real-valued "merits", where the  $i$ -th item  $d_{i}$  has merit  $\gamma_{i}$ . When independently comparing pairs of items  $(d_{i},d_{j})$  from  $D$ , the probability of item  $d_{i}$  beating item  $d_{j}$  is defined as follows:
$$
P \left(d _ {i} \succ d _ {j}\right) = \frac {\gamma_ {i}}{\gamma_ {i} + \gamma_ {j}}. \tag {1}
$$
Using exponential score functions  $p_i = e^{\gamma_i}$  reduces the model to a logistic regression on pairs of individuals (Agresti, 2003):
$$
P \left(d _ {i} \succ d _ {j}\right) = \frac {p _ {i}}{p _ {i} + p _ {j}}. \tag {2}
$$
The merits  $\Gamma$  can thus be inferred with maximum likelihood optimization (Hunter, 2004) and the following log-likelihood equation for a pool of pairwise comparisons  $C$ , a multiset of pairs  $(i,j)$ , where  $i$  and  $j$  are drawn from  $[1,n]$ :
$$
\mathcal {L} (\Gamma , C) = \sum_ {(i, j) \in C} \log P (d _ {i} \succ d _ {j}). \tag {3}
$$
5774
3.2 Incorporating Ties
Pairs of items  $(d_i, d_j)$  may exist whose merit difference is below a threshold  $\tau$  so that assessors cannot decide which is better. Rao and Kupper (1967) incorporate such ties into the model as follows:
$$
P \left(d _ {i} \succ d _ {j}\right) = \frac {p _ {i}}{p _ {i} + p _ {j} \theta} \tag {4}
$$
for the probability of preference of  $d_{i}$  over  $d_{j}$ , and
$$
P (d _ {i} \approx d _ {j}) = \frac {p _ {i} p _ {j} (\theta^ {2} - 1)}{(p _ {i} + p _ {j} \theta) (p _ {i} \theta + p _ {j})} \quad (5)
$$
for the probability of no preference between the two, where  $\theta = e^{\tau}$ . For  $\tau = 0$ , i.e., assessors being able to differentiate every item pair, these equations reduce to the standard Bradley-Terry model.
3.3 Regularization
The maximization is guaranteed to converge to the unique maximum likelihood estimator in finite steps under the assumption that in every possible partition of the items into two nonempty subsets, some subject in the second set beats some subject in the first set at least once (Hunter, 2004). Thus, a pairwise comparison experiment is restricted in two ways: (i) The matrix formed by the comparisons must construct a strongly connected graph; (ii) The comparisons between the partitions cannot all be won by subjects from the same group, i.e., no item has losses or wins exclusively.
Even though the adherence becomes asymptotically likely given an appropriate experiment design (Yan et al., 2011), the problem can be regularized to increase robustness. The regularization term
$$
\mathcal {R} (\Gamma) = \sum_ {i = 1} ^ {n} \left[ \log \left(\frac {e ^ {1}}{e ^ {1} + p _ {i}}\right) + \log \left(\frac {p _ {i}}{p _ {i} + e ^ {1}}\right) \right], \tag {6}
$$
weighted by a regularization parameter  $\lambda$ , is added to model a dummy item  $d_0$  with merit  $\gamma_0 = e^1$ , which is defined to compare against every item with exactly one win and one loss (Chen et al., 2013). Convergence is now ensured as the graph is guaranteed to be strongly connected. Additionally, the merits  $\Gamma$  are no longer scale-invariant, since the merit of the dummy item is fixed at 1.
3.4 Log-Likelihood Maximization
The log-likelihood equation, with regularization parameter  $\lambda$  and merit threshold  $\tau$  takes the form
$$
\begin{array}{l} \mathcal {L} (\Gamma , \tau , \lambda , C) = \\ \sum_ {(i, j) \in C} \log \left[ \left\{ \begin{array}{l l} P \left(d _ {i} \succ d _ {j}\right) & \text {i f} d _ {i} \succ d _ {j} \\ P \left(d _ {i} \approx d _ {j}\right) & \text {i f} d _ {i} \approx d _ {j} \end{array} \right. \right] \tag {7} \\ + \lambda \mathcal {R} (\Gamma). \\ \end{array}
$$
$\Gamma$  is initialized with 1's by convention. Chen et al. (2013) propose  $\lambda \in [0.1, 10]$ , inferring rankings similar to the unregularized problem for sufficiently small values, while regularized rankings for larger  $\lambda$  values often outperform the baseline for a broad range of applications. The maximization was solved using BFGS optimization.
3.5 Sparsification
Sampling strategies are needed to reduce the amount of comparisons as obtaining an exhaustive set of  $\binom{n}{2}$  comparisons becomes infeasible with larger item counts. Nevertheless, sampling strategies should preserve a high annotation quality.
Burton (2003) describes a strategy where items are arranged in a cyclical way. A main feature is that each item is required to appear in the same number of pairs in order to gain the same amount of information about each item. For a random permutation of the items in  $D$ , the  $i$ -th is compared with the  $(i + 1)$ -th item for  $i < n$ , and item  $d_{n}$  with item  $d_{1}$ , thus completing the cycle. This can be generalized to higher step sizes  $s$ : for instance, if  $s = 2$ , all items that are separated by two positions around the ring are compared. However, this strategy suffers from the major drawback that for some step sizes, the resulting graph has multiple unconnected components, thus violating the restriction that the comparison matrix must form a strongly connected graph. Therefore, complex combinations of different step sizes are needed, resulting in needlessly complicated experimental setups.
Alternatively, Yan et al. (2011) proposed a method of sparse grouped comparisons, where the set of all items  $D$  is partitioned into  $m$  equized disjoint subsets  $D_{k}$ , where  $k \in [1,m]$ , so that the following constraints hold true:
(i) for each  $D_{k}$ ,  $(i,j)\in C$  when  $d_i,d_j\in D_k,i\neq j$  , and
(ii)  $(i,j)\in C$  when  $d_{i}\in D_{k},d_{j}\in D_{k + 1}$  for  $k = 1,\ldots ,m - 1$
5775
Figure 1: Comparison matrices for  $n = 32$  and different values of  $k$ . Variables  $i$  and  $j$  denote the matrix indices as used in the constraints.
Figure 2: Example cyclic group design for six groups of four items.
However, in this approach not every item has the same amount of comparisons. To make the strategy of Yan et al. (2011) consistent with this requirement, both strategies can be combined to derive a cyclical group by also including comparisons between group  $D_{1}$  and group  $D_{k}$ , as reflected in the additional constraint
(iii)  $(i,j)\in C$  when  $d_{i}\in D_{1},d_{j}\in D_{m}$
This way, the design adheres to the principle that every item should have the same number of comparisons but the overall construction of the experiment remains simple. All combinations of items in the same group, and the Cartesian product of adjacent groups are included. Therefore  $k \cdot \binom{n/k}{2}$  intra-group comparisons and  $k \cdot \left( \frac{n}{k} \right)^2$  inter-group comparisons are needed. Thus, the total amount of comparisons  $c$  is
$$
c = k \left(\left(\frac {n}{k}\right) ^ {2} + \binom {n / k} {2}\right) = \frac {3 n ^ {2}}{2 k} - \frac {n}{2}. \tag {8}
$$
If multiple independent judgments per unique comparison are collected, as is frequently done in crowdsourcing,  $c$  has to be multiplied by a factor  $x$  denoting how many unique judgments are collected per comparison.
Figure 2 shows an exemplary cyclic group design for six groups of four items. Shown on the left is the overall design, with intra-group comparisons (Constraint (i)) depicted in the middle, and
inter-group comparisons between adjacent groups (Constraints (ii) and (iii)) on the right.
Example comparison matrices for  $n = 32$  and different values of  $k$  are shown in Figure 1 to provide a visual intuition of the sampling process. Although the comparison matrix is inherently symmetric, to reflect the true count of comparisons, only the lower half is depicted. All comparisons introduced by Constraint (i) are colored in light gray, Constraint (ii) in medium gray, and dark gray for Constraint (iii). Note that for the special case of  $k = 2$ , Constraints (ii) and (iii) are equal.
3.6 Model Evaluation
To test the accuracy trade-off between exhaustive comparison and sparse comparison on real-world data, twenty topics of 32 arguments each were randomly selected from the UKPCovArg1 corpus (Habernal and Gurevych, 2016), which includes an exhaustive pairwise comparison set for argument convincingness. With each comparison having five independent annotations, a baseline was established by fitting the model on the full set. Then, different values for  $k$  and  $x$  were used to sample a subset of the comparisons and the proposed model was fitted with each of the sampled comparison sets. The obtained merit ranking was compared against the baseline ranking using Pearson's  $\rho$ , with confidence intervals calculated using bootstrapping  $(n = 10,000)$ . For each of the resulting rankings, the amount of used comparisons and the correlation with the baseline ranking are detailed in Table 1.
The following interesting properties are apparent: (1) Collecting multiple judgments per unique comparison, as is usual practice in methods relying on graded scales is not sufficiently beneficial. Increasing the annotation effort by factor 5 (from  $x = 1$  to  $x = 5$ ) results in only a minimal gain in ranking accuracy of 0.06 for  $k = 4$ . For higher sampling rates, decreasing  $k$  yields a larger net increase
5776
x	k	Our approach
Judgments	Judgments %	ρ	95% CI
5	2	2480	100	1.00	1.00	1.00
4	1904	76	0.99	0.99	1.00
8	944	38	0.96	0.95	0.97
16	464	18	0.88	0.85	0.90
32	224	9	0.67	0.61	0.72
4	4	1520	61	0.99	0.99	0.99
8	752	30	0.95	0.94	0.96
16	368	14	0.86	0.83	0.88
32	176	7	0.64	0.60	0.69
3	2	1488	60	0.99	0.99	0.99
4	1136	45	0.98	0.98	0.99
8	560	22	0.93	0.82	0.95
16	272	10	0.82	0.79	0.85
32	128	5	0.65	0.60	0.69
2	2	992	40	0.98	0.97	0.99
4	752	30	0.97	0.96	0.97
8	368	14	0.91	0.89	0.93
16	176	7	0.78	0.75	0.81
32	80	3	0.59	0.56	0.63
1	2	496	20	0.95	0.94	0.96
4	368	14	0.92	0.90	0.93
8	176	7	0.82	0.79	0.86
16	80	3	0.66	0.61	0.71
32	32	1	0.47	0.40	0.54
Table 1: Our model's performance under sparsification.  $x$  denotes the number of judgments collected per unique comparison,  $k$  the group factor along the resulting total number of judgments in absolute and relative terms, compared to an exhaustive comparison and corresponding  $\rho$ -correlations with the baseline ranking.
in ranking accuracy than increasing  $x$ , at the same cost. By example, going from  $x = 1$ ,  $k = 16$  to  $x = 1$ ,  $k = 4$  ends up at the same number of comparisons as  $x = 2$ ,  $k = 8$ , but has a slightly higher ranking accuracy. Therefore, it is more economical to increase the sampling rate until the required accuracy is met than collecting multiple judgments. (2) The proposed model and comparison strategy are able to produce near-perfect rankings ( $x = 1$ ,  $k = 4$ ,  $\bar{\rho} = 0.92 \pm 0.02$ ) using only  $14\%$ , and acceptable rankings ( $x = 1$ ,  $k = 8$ ,  $\bar{\rho} = 0.82 \pm 0.04$ ) using only  $7\%$  of the full comparison set. This significant reduction is a promising sign for employing our model in crowdsourced studies at scale. However, the specific choice of  $k$  depends on scale and domain of the data as well as trustworthiness of comparisons. Therefore, we refrain from making a general suggestion for the choice of  $k$ . Thus, if the model is to be adapted to drastically different domains or item counts, exploratory studies are advised to estimate the quality tradeoff for a specific use case.
4 The Webis Argument Quality Corpus