SlowGuess/ACL_2020_OCR huggingface dataset:
Enrique Amigo
enrique@lsi.uned.es
Julio Gonzalo
julio@lsi.uned.es
Stefano Mizzaro
University of Udine
mizzaro@uniud.it
Jorge Carrillo-de-Albornoz
jcalbornoz@lsi.uned.es
In Ordinal Classification tasks, items have to be assigned to classes that have a relative ordering, such as positive, neutral, negative in sentiment analysis. Remarkably, the most popular evaluation metrics for ordinal classification tasks either ignore relevant information (for instance, precision/recall on each of the classes ignores their relative ordering) or assume additional information (for instance, Mean Average Error assumes absolute distances between classes). In this paper we propose a new metric for Ordinal Classification, Closeness Evaluation Measure, that is rooted on Measurement Theory and Information Theory. Our theoretical analysis and experimental results over both synthetic data and data from NLP shared tasks indicate that the proposed metric captures quality aspects from different traditional tasks simultaneously. In addition, it generalizes some popular classification (nominal scale) and error minimization (interval scale) metrics, depending on the measurement scale in which it is instantiated.
In Ordinal Classification (OC) tasks, items have to be assigned to classes that have a relative ordering, such as positive, neutral, negative in sentiment analysis. It is different from n-ary classification, because it considers ordinal relationships between classes. It is also different from ranking tasks, which only care about relative ordering between items, because it requires category matching; and it is also different from value prediction, because it does not assume fixed numeric intervals between categories.
Most research on Ordinal Classification, however, evaluates systems with metrics designed for
those other problems. But classification measures ignore the ordering between classes, ranking metrics ignore category matching, and value prediction metrics are used by assuming (usually equal) numeric intervals between categories.
In this paper we propose a metric designed to evaluate Ordinal Classification systems which relies on concepts from Measurement Theory and from Information Theory. The key idea is defining a general notion of closeness between item value assignments (system output prediction vs gold standard class) which is instantiated into ordinal scales but can be also be used with nominal or interval scales. Our approach establishes closeness between classes in terms of the distribution of items per class in the gold standard, instead of assuming predefined intervals between classes. We provide a formal (Section 4) and empirical (Section 5) comparison of our metric with previous approaches, and both analytical and empirical evidence indicate that our metric suits the problem best than the current most popular choices.
2 State of the Art
In this section we first summarize the most popular metrics used in OC evaluation campaigns, and then discuss previous work on OC evaluation.
2.1 OC Metrics in NLP shared tasks
OC does not match traditional classification, because the ordering between classes makes some errors more severe than others. For instance, misclassifying a positive opinion as negative is a more severe error than as a neutral opinion. Classification metrics, however, have been used for OC tasks in several shared tasks (see Table 1). For instance, Evalita-16 (Barbieri et al., 2016) uses  $F_{1}$ , NTCIR
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3938-3949
July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Table 1: Metrics used for OC in evaluation campaigns
Acc	F1	AvgRec	Pearson	R/S	MAEM	MSE
NTCIR-7	✓
REPLAB-13	✓
SEM15-T11	✓
EVALITA-16	✓
STS-16	✓
SEM17-T4	✓	✓
7 (Kando, 2008) uses Accuracy, and Semeval-17 Task 4 (Rosenthal et al., 2017) uses Macro Average Recall.
OC does not match ranking metrics either: three items categorized by a system as very high/high/low, respectively, are perfectly ranked with respect to a ground-truth high/low/very_low, but yet no single item is correctly classified. However, ranking metrics have been applied in some campaigns, such as R/S for reputation polarity and priority in Replab-2013 (Amigo et al., 2013a).
OC has also been evaluated as a value prediction problem – for instance, SemEval 2015 Task 11 (Ghosh et al., 2015) – with metrics such as Mean Average Error (MAE) or Mean Squared Error (MSE), usually assuming that all classes are equidistant. But, in general, we cannot assume fixed intervals between classes if we are dealing with an OC task. For instance, in a paper reviewing scale strong_accept/accept/weak_accept/undecided/weak_reject/reject/strong_reject, the differences in appreciation between each ordinal step do not necessarily map into predefined numerical intervals.
Finally, OC has been also considered as a linear correlation problem. as in the Semantic Textual Similarity track (Cer et al., 2017). An OC output, however, can have perfect linear correlation with the ground truth without matching any single value.
This diversity of approaches – which do not happen in other types of tasks – indicates a lack of consensus about what tasks are true Ordinal Classification problems, and what are the general requirements of OC evaluation.
2.2 Studies on Ordinal Classification
There is a number of previous formal studies on OC in the literature. First, the problem has been studied from the perspective of loss functions for ordinal regression Machine Learning algorithms.
In particular, in a comprehensive work, Rennie and Srebro (2005) reviewed the existing loss functions for traditional classification and they extended them to OC. Although they did not try to formalize OC tasks, in further sections we will study the implication of using their loss function for OC evaluation purposes.
Other authors analyzed OC from a classification perspective. For instance, Waegeman et al. (2006) presented an extended version of the ROC curve for ordinal classification, and Vanbelle and Albert (2009) studied the properties of the Weighted Kappa coefficient in OC.
Other authors applied a value prediction perspective. Gaudette and Japkowicz (2009) analysed the effect of using different error minimization metrics for OC. Baccianella et al. (2009) focused on imbalanced datasets. They imported macro averaging (from classification) to error minimization metrics such as MAE, MSE, and Mean Zero-One Error.
Remarkably, a common aspect of all these contributions is that they all assume predefined intervals between categories. Rennie and Srebro assumed, for their loss function, uniform interval distributions across categories. In their probabilistic extension, they assume predefined intervals via parameters in the join distribution model. Waegeman et al. explicitly assumed that "the misclassification costs are always proportional to the absolute difference between the real and the predicted label". The predefined intervals are defined by Vanbelle and Albert via weighting parameters in Kappa. The MAE and MSE metrics compared by Gaudette and Japkowicz also assume predefined (uniform) intervals. Finally, the solution proposed by Baccianella et al. is based on "a sum of the classification errors across classes".
In our opinion, assuming and adding intervals between categories to estimate misclassification errors violates the notion of ordinal scale in Measurement Theory (Stevens, 1946), which establishes that intervals are not meaningful relationships for ordinal scales. Our measure and our theoretical analysis are meant to address this problem.
3 Closeness Evaluation Measure (CEM)
3.1 Measure Definition
Evaluation metrics establish proximity between a system output and the gold standard (Amigo and Mizzaro, 2020). In ordinal classification we have to compare the classes assigned by the system with
Figure 1: In the left distribution, weak accept vs. weak reject would be a strong disagreement between reviewers (i.e., the classes are distant), because in practice these are almost the extreme cases of the scale (reviewers rarely go for accept or reject). In the right distribution the situation is the opposite: reviewers tend to take a clear stance, which makes weak accept and weak reject closer assessments than in the left case.
the true classes in the gold standard.
A key idea in our metric is to establish a notion of informational closeness that depends on how items are distributed in the rank of classes. The idea is that two items  $a$  and  $b$  are informationally close if the probability of finding an item between the two is low. As an example, Figure 1 illustrates the intuition of how item distribution affects informational closeness in the context of paper reviewing. This is similar in spirit to, for instance, comparing the quality of two journals according to their quartiles in the rank of journals of comparable topics. With this notion of informational closeness, proximity between classes adapts to the way in which classes are used in a given dataset.
This idea of informational closeness can be implemented using Information Theory: the more unexpected it is to find an item between  $a$  and  $b$ , the more information such event provides, and the more  $a$  and  $b$  are informationally closer. Let  $P(x \preceq_{\mathrm{ORD}}^b a)$  be the probability that, sampling an item  $x$  from the space of items,  $x$  is closer to  $b$  than  $a$  in the ordinal scale of classes. Then we can define Closeness Information Quantity (CIQ) between  $a$  and  $b$  as the Information Quantity of the event  $x \preceq_{\mathrm{ORD}}^b a$ , as follows:
$$
\operatorname {C I Q} ^ {\mathcal {O R D}} (a, b) \equiv - \log \left(P \left(x \preceq_ {\mathcal {O R D}} ^ {b} a\right)\right). \tag {1}
$$
Let us now apply this concept for the evaluation of system outputs. Let  $\mathcal{D}$  be the item collection,  $\mathcal{C} = \{c_1,\ldots ,c_n\}$  a set of sorted classes such that  $c_{1} < c_{2} < \dots < c_{n}$ , and  $g,s:\mathcal{D}\longrightarrow \mathcal{C}$  the gold standard and a system output. Given the classes  $g(d),s(d)$  assigned to an
item  $d\in \mathcal{D}$  by the gold standard and the system output,  $\mathrm{CIQ}^{\mathrm{ORD}}(s(d),g(d))$  measures the closeness between the assigned class and the gold standard class:
$$
\operatorname {C I Q} ^ {\mathsf {O R D}} (s (d), g (d)) = - \log (P (x \preceq_ {\mathsf {O R D}} ^ {g (d)} s (d))).
$$
Our proposed evaluation measure consists in adding CIQ values for all items  $d \in \mathcal{D}$ , and normalizing the sum by its maximal value, which is the one obtained by a system output that matches the gold standard perfectly. This is what we call Closeness Evaluation Measure,  $\mathrm{CEM}^{\mathrm{ORD}}$ :
$$
\mathrm {C E M} ^ {\mathsf {O R D}} (s, g) = \frac {\sum_ {d \in \mathcal {D}} \mathrm {C I Q} ^ {\mathsf {O R D}} (s (d) , g (d))}{\sum_ {d \in \mathcal {D}} \mathrm {C I Q} ^ {\mathsf {O R D}} (g (d) , g (d))}.
$$
In an ordinal scale, the condition  $x \preceq_{\mathrm{ORD}}^b a$  ( $x$  is closer to  $b$  than  $a$ ) implies that  $x$  is between  $a$  and  $b$  ( $a \geq x \geq b$  or  $a \leq x \leq b$ ). Therefore, if  $n_i$  is the amount of items assigned to class  $c_i$  in the gold standard, and  $N$  is the total amount of items, the formula above turns into:
$$
\mathrm {C E M} ^ {\mathcal {O R D}} (s, g) = \frac {\sum_ {d \in \mathcal {D}} \operatorname {p r o x} (s (d) , g (d))}{\sum_ {d \in \mathcal {D}} \operatorname {p r o x} (g (d) , g (d))}
$$
where  $\mathrm{prox}(c_i, c_j) = -\log \left(\frac{\frac{n_i}{2} + \sum_{k=i+1}^{j} n_k}{N}\right)$ .
Note that the term  $\mathrm{prox}(c_i, c_j)$ , which is the core of the metric, reflects the informational closeness that the metric assigns to a pair of classes  $c_i, c_j$ . Note also that half of the ties (elements in the class  $i$ ) are included in the computation. Every time the system assigns the class  $c_i$  and the ground truth is  $c_j$ , the contribution of that assignment to the final
value of  $\mathrm{CEM}^{\mathrm{ORD}}$  is proportional to the informational closeness between both classes.
As an example, let us consider the two ground truth distributions in Figure 1. The proximity between the classes weak_accept and weak_reject for the left distribution is:
$$
- \log \left(\frac {9 0 / 2 + 1 9 3 + 1 0 5}{4 0 2}\right) = 0. 2 3
$$
and for the right distribution is:
$$
- \log \left(\frac {1 0 / 2 + 3 + 1 0}{3 7 6}\right) = 4. 3 8.
$$
A mistake between these two classes is more heavily penalized by the metric in the left distribution. Note also that correct predictions have different weights -  $\mathrm{prox}(c_i, c_i)$  - which are higher for infrequent classes. For instance, a correct guess for a reject ground truth in the left distribution has a weight of  $\mathrm{prox}(\mathrm{reject}, \mathrm{reject}) = 6.84$ , because it is a rare class (7/402 items); but a correct guess for an undecided item has only a weight of 2.06 because the class is very frequent in the ground truth (193/402 items). This is an effect of using Information Theory to characterize closeness: an infrequent class has more information than a frequent class.
Overall,  $\mathrm{CEM}^{\mathrm{ORD}}$  rewards exact matches, considers ordinal relationships, and does not assume predefined intervals between classes (instead, intervals depend on the distribution of items into classes in the gold standard). Appendix A shows detailed examples of how to compute  $\mathrm{CEM}^{\mathrm{ORD}}$  from the confusion matrix for a system output.
3.2 Formalization of CEM on Different Scales
We have specified our measure  $\mathrm{CEM}^{\mathrm{ORD}}$  at ordinal scale to address OC tasks, but it could be used at any scale. In this section we briefly investigate this generalization. In Measurement Theory, at least in Stevens's model (1946), all measures map items to real numbers, and measurement equivalence at different scales is determined by permissible transformation functions. Permissible transformations are bijective functions in nominal scale ( $\mathcal{F}_{\mathrm{NOM}}$ ), strictly increasing functions in ordinal scale ( $\mathcal{F}_{\mathrm{ORD}}$ ), and linear functions for the interval scale ( $\mathcal{F}_{\mathrm{INT}}$ ).
Starting from the notion of  $|a - b|$  as the standard algebraic distance between numbers, we define closeness at a certain measurement scale  $T$  if
it fits for at least one permissible transformation in  $\mathcal{F}_{\mathrm{T}}$
Definition 1 (Closeness for a Scale Type) Being three numbers  $x$ ,  $a$ , and  $b$ , we say that  $x$  is closer to  $b$  than  $a$ ,  $(x \preceq_{\mathrm{T}}^{b} a)$  for a certain scale type  $\mathrm{T}$ , if and only if:
$$
\exists f \in \mathcal {F} _ {\mathrm {T}} \left(\left| f (x) - f (b) \right| \leq \left| f (a) - f (b) \right|\right).
$$
The conditions for  $x \preceq_{\mathbb{T}}^{b} a$  at ordinal scale ( $\mathrm{T} = \mathrm{ORD}$ ) are  $(b \geq x \geq a) \lor (a \geq x \geq b)$  (see proof in the supplementary material). That is, at ordinal scale,  $x$  must be located between  $a$  and  $b$  to be closer to  $a$  than  $b$ . The condition for nominal scale ( $\mathrm{T} = \mathrm{NOM}$ ) is  $(b = x \lor b \neq a)$ . At interval scale ( $\mathrm{T} = \mathrm{INT}$ ), the condition matches the standard algebraic closeness between numbers:  $(|b - x| \leq |b - a|)$ .
We can generalize  $\mathrm{CIQ}^{\mathrm{ORD}}$  and  $\mathrm{CEM}^{\mathrm{ORD}}$  to consider closeness at any scale  $\mathbf{T}$ , simply replacing  $x \preceq_{\mathrm{ORD}}^{b} a$  with  $x \preceq_{\mathrm{T}}^{b} a$ . We denote these generalizations as  $\mathrm{CIQ}^{\mathrm{T}}$ ,  $\mathrm{CEM}^{\mathrm{T}}$ . The  $\mathrm{CEM}^{\mathrm{T}}$  metric generalizes some of the most popular metrics in classification.
Proposition 1 Assuming that categories in  $g$  follow a uniform distribution, then Accuracy is proportional to CEM at nominal scale. Formally, whenever  $P(g(d) = c)$  is equal for all categories  $c \in \mathcal{C}$ , then:
$$
\operatorname {A c c} (s, g) \propto \operatorname {C E M} ^ {\text {N O M}} (s, g).
$$
Macro Average Accuracy can be also defined by aggregating  $\mathrm{CIQ}^{\mathrm{NOM}}(s(d), g(d))$  in the corresponding manner. Also, under the same statistical assumptions, Precision and Recall for a category  $c$  can be defined in terms of aggregated CIQs of items in the system or gold category respectively.
Proposition 2 Whenever  $P(g(d) = c)$  is equal for all categories  $c \in \mathcal{C}$ , then:
$$
\operatorname{Pre}_{g,c}(s)\propto \sum_{d\in \mathcal{D}:s(d) = c}\operatorname{CIQ}^{\text{NOM}}(s(d),g(d))
$$
$$
\operatorname{Rec}_{g,c}(s)\propto \sum_{d\in \mathcal{D}:g(d) = c}\operatorname{CIQ}^{\mathsf{NOM}}(s(d),g(d)).
$$
Exact match between Precision, Recall and the CIQ aggregation is achieved when values are normalized with respect to the maximum.
On the other hand, if we do not assume a uniform distribution of items into classes in the gold standard, then we obtain a classification metric  $\mathbf{CEM}^{\mathrm{NOM}}(s,g)$  which gives more (logarithmic) weight to errors in infrequent classes.
Finally, at interval scale,  $\mathbf{CEM}^{\mathrm{INT}}$  would be equivalent to a logarithmic version of MAE whenever items are uniformly distributed across classes.
We leave a more detailed formal and empirical analysis of CEM at other scales for future work, as it is not the primary scope of this paper.
4 Theoretical Evidence
Following a methodology previously applied for Classification (Sebastiani, 2015; Sokolova, 2006), Clustering (Dom, 2001; Meila, 2003; Amigo et al., 2009), and document ranking tasks (Moffat, 2013; Amigo et al., 2013b), here we define a formal framework for OC via desirable properties to be satisfied, which are illustrated in Figure 2 and introduced below.
4.1 Metric Properties
The first property states that an effectiveness metric  $\operatorname{Eff}(s, g)$  should not assume predefined intervals between classes, i.e., it should be invariant under permissible transformation functions at ordinal scale.
Property 1 (Ordinal Invariance) An effectiveness metric Eff has ordinal invariance if it is invariant under strictly increasing functions  $f_{\mathsf{ORD}} \in \mathcal{F}_{\mathsf{ORD}}$  applied to both the system output and the gold standard:
$$
\operatorname {E f f} (s, g) = \operatorname {E f f} \left(f _ {\text {O R D}} (s), f _ {\text {O R D}} (g)\right).
$$
For instance,  $\operatorname{Eff}((1,2,2),(1,2,3))$  should be equivalent to  $\operatorname{Eff}((11,24,24),(11,24,39))$ , by considering the (strictly increasing) permissible transformation function  $f_{\mathrm{ORD}}(x) = 10x + x^2$ .
Although we can not compare intervals at ordinal scale, we know, e.g., that "neutral" is closer to "positive" than "negative". Therefore we need another property to verify monotonicity with respect to category closeness.
Property 2 (Ordinal Monotonicity) Changing system predictions closer to the true category should result in a metric increase:
$$
I f \exists d. (s (d) \neq s ^ {\prime} (d)) \wedge
$$
$$
(\forall d. ((s (d) > s ^ {\prime} (d) \geq g (d)) \lor (s (d) = s ^ {\prime} (d))))
$$
then  $\operatorname{Eff}(s', g) > \operatorname{Eff}(s, g)$ .
The formalization of ordinal monotonicity states that if all predictions by system  $s'$  are better or equal than predictions by  $s$ , and at least one is
Figure 2: Illustration of desirable formal properties for Ordinal Classification. Each bin is a system output, where columns represent ordered classes assigned by the system, and colors represent the items' true classes, ordered from black to white. "=" means that both outputs should have the same quality, and "> " that the left output should receive a higher metric value than the right output.
strictly better, then the metric score of  $s'$  must be higher.
Finally, in order to manage the effect of imbalanced data sets, another desirable property is that an item classification error in a frequent class should have less effect than a classification error in a small class (Fatourechi et al., 2008). In order to formalize this property, we use  $g_{d \to c}$  to denote the result of moving the item  $d$  to the class  $c$  in the gold standard.
Property 3 (Imbalance) Distancing items from a small class has more effect than distancing items from a large class. Let  $(c_{1}, c_{2}, c_{3})$  be three contiguous classes such that  $c_{1}$  is larger than  $c_{3}$ , and  $d_{1}, d_{3}$  two items such that  $g(d_{1}) = c_{1}$  and  $g(d_{3}) = c_{3}$ . Then
$$
\operatorname {E f f} \left(g _ {d _ {1} \rightarrow c _ {2}}, g\right) > \operatorname {E f f} \left(g _ {d _ {3} \rightarrow c _ {2}}, g\right).
$$
4.2 Metric Analysis
Table 2 displays the properties satisfied by metrics grouped by families. Classification metrics are ordinal invariant, but they do not satisfy ordinal monotonicity. Attempts to mitigate this limitation include (i) Accuracy at n (Gaudette and Japkowicz, 2009) which relaxes Accuracy with an ordinal margin error, and (ii) ignoring the neutral class (Rosenthal et al., 2014). However, both approaches are insensitive to some types of error. Some classification metrics such as MAAC, Cohen's Kappa or F-measure averaged across classes satisfy the imbalance constraint.
See the supplementary material for proofs and counter examples where appropriate.
3942
Table 2: Constraint-based Metric Analysis
Metric family Metrics	Constraints Ord.	Ord.	Imb.
Inv.	Mon.
Classification	Acc	✓	-	-
Acc with n	✓	-	-
Metrics	Macro Avg Acc, Cohen's κ	✓	-	✓
F-measure avg. across classes	✓	-	✓
Value Prediction	MAE, MSE	-	✓	-
Macro Avg. MAE/MSE	-	✓	✓
Weighted κ	-	✓	✓
Rennie & Srebro loss function	-	✓	-
Cosine similarity	-	✓	-
Correlation Coefficients	Linear correlation	-	-	-
Ordinal: Kendall (tau-b), Spea.	✓	-	✓
Kendall-(Tau-a)	✓	-	-
Reliability and Sensitivity	✓	-	✓
Clustering	MI, Purity and Inv. Purity	✓	-	✓
Path based	Ordinal Classification Index	✓	-	-
CEM	CEM\( ^{NOM} \)	✓	-	✓
CEM\( ^{INT} \)	-	✓	✓
CEM\( ^{ORD} \)	✓	✓	✓
The most popular Value Prediction metrics are Mean Absolute Error (MAE) and Mean Square Error (MSE). They both assume a predefined fixed numerical value for each category. Therefore, ordinal invariance is violated. The imbalance property is satisfied by the Macro Average versions  $\mathbf{MAE}^m$ $\mathrm{MSE}^m$  (Baccianella et al., 2009). The weighted Kappa can be monotonic whenever the accumulated weights are consistent with the ordinal structure (Vanbelle and Albert, 2009). In addition, it can satisfy imbalance depending on the weighting scheme. However, ordinal invariance is not satisfied. The loss function for ordinal classification proposed by Rennie and Srebro (2005) is, in the same way as MAE, grounded on category differences, and therefore does not satisfy ordinal invariance. Finally, the cosine similarity has also been employed to evaluate OC (Ghosh et al., 2015), where documents are dimensions and categories are vector values. Just like any other geometric measure, it is not ordinal invariant and it does not satisfy imbalance.
In general, correlation coefficients do not satisfy monotonicity, given that exact matching of gold standard values is not required to achieve the maximum score. Unlike linear correlation, ordinal correlation coefficients (i.e., Kendall or Spearman) are ordinal invariant. Kendall can be computed in different ways depending on how ties are man
aged. In Tau-a, only discordant pairs are considered  $(g(d_{1}) > g(d_{2})$  and  $s(d_{1}) < s(d_{2}))$  and imbalance is not satisfied. The most popular Kendall coefficient approach (Tau-b) and Spearman both satisfy imbalance. Pearson coefficient does not, due to the interval effect. Reliability and Sensitivity metrics, which extend the clustering metric BCubed, are essentially an ordinal correlation metric, being invariant but failing in monotonicity, with the advantage of satisfying imbalance due to the precision/recall notions.
By definition, clustering metrics are ordinal invariant, because they are not affected by the cluster of category descriptors. In addition, most of them, such as Mutual Information (MI) or Purity and Inverse Purity, satisfy imbalance. However, they are not ordinal monotonic, given that they do not consider any ordinal relationship between categories.