SlowGuess/ACL_2020_OCR huggingface dataset:
Finally, we must include the approach by Cardoso and Sousa (2011), a path based metric called Ordinal Classification Index which is designed specifically for OC problems. This is a metric that integrates aspects from the previous three metric families, including two parameters  $\beta_{1}$  and  $\beta_{2}$  to combine different components. Therefore, this metric can capture the different quality aspects involved in the OC process. However, the metric inherits the lack of invariance of MAE and MSE when computing the ordinal distance between categories, and monotonicity can be violated depending on the effect of discordant item pairs.
The table ends with our proposed metric CEM, which is either a classification, error minimization, or OC metric depending if it is instantiated into nominal (CEM $^{\text{NOM}}$ ), interval (CEM $^{\text{INT}}$ ), or ordinal measurement scale (CEM $^{\text{ORD}}$ ). CEM $^{\text{ORD}}$  is the only metric that satisfies the three properties, provided that there are no empty classes in the gold standard (see Appendix A.2).
Meta-evaluating metrics is not straightforward. A common criterion is robustness, defined as consistence (correlation) of system rankings across data sets. However, although robustness is relevant – and we do report it at the end of this section – it does not reflect to what extent a metric captures the quality aspects of systems.
As many authors have pointed out, an OC metric should capture diverse aspects of systems: class matching, ordering, and imbalance. In our experi
iments, in addition to robustness, we select three complementary metrics, each focused on one of these partial aspects, and we evaluate to what extent existing OC metrics are able to capture all these aspects simultaneously.
The selected metrics are: (i) Accuracy, as a partial metric which captures class matching; (ii) Kendall's correlation coefficient Tau-a (without counting ties), in order to capture class ordering $^2$ ; and (iii) Mutual Information (MI), a clustering metric which reflects how much knowing the system output reduces uncertainty about the gold standard values. This metric accentuates the effect of small classes (imbalance property).
5.1 Meta-evaluation Metric
In order to quantify the ability of metrics to capture the aspects reflected by these three metrics, we use the Unanimous Improvement Ratio (UIR) (Amigó et al., 2011). While robustness focuses on consistence across data sets, UIR focuses on consistence across metrics. It essentially counts in how many test cases an improvement is observed for all metrics simultaneously. Being  $\mathcal{M}$  a set of metrics, and  $\mathcal{T}$  a set of test cases, and  $s_t$  a system output for the test case  $t$ , the Unanimous Improvement Ratio  $\mathrm{UIR}_{\mathcal{M}}(s,s')$  between two systems is defined as:
$$
\frac {\left| \left\{t \in \mathcal {T} : s _ {t} \geq_ {\mathcal {M}} s _ {t} ^ {\prime} \right\} \right| - \left| \left\{t \in \mathcal {T} : s _ {t} ^ {\prime} \geq_ {\mathcal {M}} s _ {t} \right\} \right|}{\left| \mathcal {T} \right|},
$$
where  $s_t \geq_{\mathcal{M}} s_t'$  represents that system  $s$  improves system  $s'$ , on the test case  $t$ , unanimously for every metric:
$$
s _ {t} \geq_ {\mathcal {M}} s _ {t} ^ {\prime} \equiv \big (\forall m \in \mathcal {M} \big (m (s _ {t}) \geq m (s _ {t} ^ {\prime}) \big).
$$
Therefore, UIR reflects to what extent a system outperforms another system for several metrics simultaneously. Then, we define our meta-evaluation measure Coverage for a single metric  $m$  as the Spearman correlation (over system output pairs  $s, s'$  in the set of system outputs) between differences in  $m$  and unanimous improvements over the reference metric set. Being  $\mathcal{M}$  the reference metric set:
$$
\operatorname {C o v} _ {\mathcal {M}} (m) = \operatorname {S p e a} \bigg (m (s) - m (s ^ {\prime}), \operatorname {U I R} _ {\mathcal {M}} (s, s ^ {\prime}) \bigg).
$$
The more the coverage of a metric  $m$  is high with respect to a reference metric set  $\mathcal{M}$ , the more an improvement according to  $m$  reflects all quality aspects represented by  $\mathcal{M}$ .
5.2 Compared Metrics
We evaluate the coverage of  $\mathrm{CEM}^{\mathrm{ORD}}$  and other metrics with respect to the reference metric set Accuracy, Kendall, and MI. In the empirical study we have considered most metrics used in practice to evaluate OC problems; we have excluded a few metrics which are included in the theoretical study, either because they have not been used previously to evaluate OC problems (such as clustering metrics) or because they have internal parameters and therefore a range of variability that requires a dedicated study (such as weighted Kappa and Ordinal Index). In order to check the need for the logarithmic scaling in  $\mathrm{CEM}^{\mathrm{ORD}}$  (which comes from the application of Information Quantity), we also include an alternative metric  $\mathrm{CEM}_{\text{flat}}^{\mathrm{ORD}}$ , which is similar to CEM but without the logarithmic scaling.
5.3 Experiments on Synthetic Data
In order to play with a representative and controlled amount of classes and distributions, we first experiment with synthetic data. Let us consider a synthetic dataset with 100 test cases and 200 documents per test case, classified into 11 categories. In order to study different degrees of imbalance, we assign ground truth labels to documents according to a normal distribution with average 4 and a typical deviation between 1 and 3. The imbalance grade (deviation) varies uniformly across topics. The majority class is therefore the fourth class. Finally, we discretize the resulting values into their closest category in  $\{1,2,\ldots ,11\}$ .
We generate synthetic system outputs according to the following behaviour: each system makes mistakes in a certain ratio  $r$  of value assignments, where  $r \in \{0.1, 0.2, \dots, 0.9, 1\}$ . Then we distinguish between five kinds of mistakes, thus obtaining  $10 \times 5$  possible system configurations. The five alternative mistakes are:
1. Majority class assignment: Assign the most frequent category:  $s_{maj}(d) = 4$ .
2. Random assignment: Assign classes randomly:  $s_{rand}(d) = v$  with  $v \sim U(1, 11)$ .
4We selected class 4 instead of 6 in order to have asymmetry in the distribution.
$^{2}$ en.wikipedia.org/wiki/Kendall_rank_corrlation_coefficient  
 $^{3}$ We use the non parametric coefficient Spearman instead of Pearson. This focuses the meta-evaluation on system score ordering rather than particular scale properties of metrics.
Table 3: Metric Coverage: Spearman Correlation between single metrics and the UIR combination of Mutual Information, Accuracy, and Kendall across system pairs in both the synthetic and real data sets.
Synthetic data	Real data
all systems	minus sRand	minus sprox	minus smaj	minus stDisp	minus soDisp	Replab 2013	SEM-2014 T9-A	T9-B	SEM-2015 T10-A	T10-B	T10-C
Reference metrics in UIR	Accuracy	0.81	0.77	0.78	0.78	0.94	0.77	0.75	0.90	0.98	0.85	0.94	0.80
Kendall	0.84	0.81	0.82	0.82	0.93	0.82	0.88	0.94	0.98	0.84	0.97	0.88
MI	0.84	0.82	0.84	0.82	0.93	0.82	0.91	0.97	0.99	0.93	0.98	0.93
Classification metrics	F-measure	0.83	0.80	0.82	0.81	0.93	0.81	0.66	0.90	0.98	0.91	0.98	0.92
MAAC	0.83	0.81	0.82	0.79	0.91	0.81	0.84	0.86	0.97	0.84	0.95	0.82
Kappa	0.81	0.78	0.79	0.77	0.94	0.77	0.44	0.95	0.99	0.93	0.98	0.97
Acc with 1	0.79	0.75	0.77	0.80	0.85	0.79	0.23	0.82	0.60	0.31	0.35	-0.19
Error minimization	MAE	0.84	0.82	0.83	0.87	0.86	0.84	0.81	0.96	0.95	0.95	0.87	0.56
MAEm	0.74	0.73	0.74	0.80	0.76	0.73	0.73	0.95	0.88	0.91	0.74	0.30
MSE	0.89	0.87	0.87	0.88	0.93	0.88	0.28	0.87	0.98	0.63	0.97	0.93
MSEm	0.83	0.80	0.80	0.82	0.90	0.83	0.10	0.85	0.94	0.48	0.91	0.52
Correlation coefficients	Pearson	0.77	0.79	0.74	0.73	0.83	0.79	0.91	0.97	0.98	0.96	0.97	0.79
Spearman	0.72	0.67	0.69	0.77	0.76	0.70	0.07	0.96	0.98	0.97	0.98	0.80
Measurement theory	CEMORD	0.91	0.89	0.90	0.90	0.95	0.89	0.94	0.96	0.99	0.98	0.99	0.96
CEMord flat	0.87	0.84	0.86	0.88	0.89	0.87	0.82	0.96	0.96	0.94	0.92	0.65
3. Tag displacement: Assign the next category:
$$
s _ {t D i s p} (d) = g (d) + 1.
$$
4. Ordinal displacement: Being  $\operatorname{ord}(d)$  the ordinal position of  $d$  in a sorting of documents in concordance with category values  $(g(d) > g(d') \Rightarrow \operatorname{ord}(d) > \operatorname{ord}(d'))$ , the system displaces the document  $\frac{n}{10}$  positions:
$$
s _ {o D i s p} \left(d _ {i}\right) = g \left(d ^ {\prime}: o r d \left(d ^ {\prime}\right) = o r d (d) + \frac {n}{1 0}\right).
$$
5. Proximity assignment: The assignment is closer to the gold standard than a random one: it assigns a category between a randomly selected one and the gold standard:
$$
s _ {p r o x} (d) = g \left(d ^ {\prime}: o r d \left(d ^ {\prime}\right) = \frac {o r d (d) + r P o s}{2}\right)
$$
with  $rPos \sim U(1, n)$  (a random position between 1 and  $n$ ).
We discretize the resulting values in the same way than the gold standard. The synthetic outputs are designed to produce trade-offs between evaluation metrics. For instance, a total displacement  $(s_{tDisp}^{r=1})$  achieves the maximal Kendall correlation but the lowest Accuracy. On the contrary, a  $30\%$  of random assignments  $s_{\{r=0.3, rand\}}$  can decrease substantially the ordinal relationships, but keeping a  $70\%$  of Accuracy. Also,  $s_{rand}^{r=0.3}$  outperforms  $s_{prox}^{r=0.5}$  in terms of accuracy, but not necessarily in terms of error minimization metrics. Finally,  $s_{rand}^{r=0.3}$  can be outperformed by  $s_{maj}^{r=0.4}$  given that
the second system assigns documents to the majority class, but not in terms of MI, which accounts for the imbalance effect.
Table 3 (left part) shows the results. The metric coverage can vary substantially when changing the distribution of systems. For this reason, we first consider every synthetic output and then we repeat the experiment removing each of the system types. As the table shows,  $\mathrm{CEM}^{\mathrm{ORD}}$  improves all other metrics, including the individual metrics used as a reference via UIR (MI, Kendall, and Accuracy). Note that the flat (not logarithmic) version  $\mathrm{CEM}_{\text{flat}}^{\mathrm{ORD}}$  performs systematically worse than the original metric, which supports the use of the logarithmic, information-theoretic formula to compute similarity.
5.4 Experiments on NLP shared tasks
Let us now study how metrics behave with actual data from evaluation campaigns, where we cannot control the amount and types of error. We use data from six OC evaluation campaigns for which system outputs are publicly available.
The first data set comes from the Replab 2013 reputational polarity task (Amigo et al., 2013a). It consists of 61 companies with 1,500 tweets each; tweets are annotated as positive, negative, or neutral for the company's reputation.
All the other five datasets are sentiment analysis subtasks from SemEval for which system outputs are available online: SemEval-2015 task 10A
(1680 samples, 13 systems), task 10B (8985 samples, 51 systems) and task 10C (3097 samples, 11 systems) (Rosenthal et al., 2015); and SemEval2014, tasks 9A (2392 samples, 48 systems) and 9B (2396 samples, 7 systems). All these tasks contain three categories. Given that SemEval tasks do not distribute samples in test cases, we emulate 10 test cases by dividing randomly the data sets into 10 partitions in order to compute UIR.
Table 3 (right part) shows the results.  $\mathrm{CEM}^{\mathrm{ORD}}$  is the top performer in four datasets, and the second best (with a minimal difference of 0.01 with respect to the best metric) in the other two. The non-logarithmic version of  $\mathrm{CEM}^{\mathrm{ORD}}$  is, again, worse than the logarithmic version in all cases (except one, SemEval 2014 task 9A, where they both give the same result).
Some metrics are able to achieve a high coverage in some data sets, but not in a consistent manner. For instance, Kappa maximizes the coverage in the last dataset in the table, but achieves an extremely low result for RepLab. In general, the table also shows that the relative coverage performance of metrics varies depending on the dataset characteristics.
Finally, we also computed metrics robustness in terms of Spearman correlation between system rankings produced by the metric for topics (or data set partition) pairs in the campaigns. The highest robustness (0.57) is achieved by  $\mathrm{CEM}^{\mathrm{ORD}}$ , Accuracy and F-measure; and the lowest robustness (0.49) is achieved by Accuracy with 1 and Macro Average MAE.  $\mathrm{CEM}^{\mathrm{ORD}}$  is more robust than its non-logarithmic version  $\mathrm{CEM}_{flat}^{\mathrm{ORD}}$  (0.57 vs 0.55), again supporting the use of the information-theoretic logarithmic formula.
Our findings can be summarized as follows: (i) metrics commonly used for Ordinal Classification problems are highly heterogeneous and, in general, inconsistent with the notion of ordinal scale in Measurement Theory; (ii) the notion of closeness between classes can be modelled in terms of Measurement Theory and Information Theory and particularized for different scales; and (iii) our proposed Ordinal Closeness Evaluation Measure  $(\mathrm{CEM}^{\mathrm{ORD}})$  is the only one that satisfies all desirable formal properties, it is as robust as the best state-of-the-art metrics, and it is the one that better captures the different quality aspects of OC problems in our ex
perimentation, with both synthetic and naturalistic datasets.
From a methodological perspective, the evidence that we have presented covers the four approaches pointed out in Amigo et al. (2018): we have compared metrics in terms of desirable formal properties to be satisfied (theoretic top-down), we have generalized existing approaches (theoretic bottom-up), and we have compared effectiveness on human assessed and on synthetic data (empirical bottom-up and top-down). Future work includes the application of CEM at scales other than the ordinal.
Code to compute CEM will be available at github.com/EvALLTEAM/EvALLToolkit.
This research has been partially supported by grants Vemodalen (TIN2015-71785-R) and MIS-MIS (PGC2018-096212-B-C32) from the Spanish government, as well as by the Google Research Award Axiometrics: Foundations of Evaluation Metrics in IR.
Enrique Amigo, Jorge Carrillo de Albornoz, Irina Chugur, Adolfo Corujo, Julio Gonzalo, Tamara Martín, Edgar Meij, Maarten de Rijke, and Damiano Spina. 2013a. Overview of RepLab 2013: Evaluating online reputation monitoring systems. In Information Access Evaluation. Multilinguality, Multimodality, and Visualization, pages 333-352.
Enrique Amigó, Hui Fang, Stefano Mizzaro, and ChengXiang Zhai. 2018. Are we on the right track?: An examination of information retrieval methodologies. In Proceedings of ACM SIGIR'18, pages 997-1000.
Enrique Amigó, Julio Gonzalo, Javier Artiles, and Felisa Verdejo. 2009. A comparison of extrinsic clustering evaluation metrics based on formal constraints. Information Retrieval, 12(4):461-486.
Enrique Amigó, Julio Gonzalo, Javier Artiles, and Felisa Verdejo. 2011. Combining evaluation metrics via the unanimous improvement ratio and its application to clustering tasks. Journal of Artificial Intelligence Research, 42:689-718.
Enrique Amigó, Julio Gonzalo, and Felisa Verdejo. 2013b. A general evaluation measure for document organization tasks. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, pages 643-652.
Enrique Amigo and Stefano Mizzaro. 2020. On the Nature of Information Access Evaluation Metrics: A
Unifying Framework. Information Retrieval Journal. To appear.
S. Baccianella, A. Esuli, and F. Sebastiani. 2009. Evaluation measures for ordinal regression. In 2009 Ninth International Conference on Intelligent Systems Design and Applications, pages 283-287.
Francesco Barbieri, Valerio Basile, Danilo Croce, Malvina Nissim, Nicole Novielli, and Viviana Patti. 2016. Overview of the Evalita 2016 SENTiment POLarity Classification Task. In Proceedings of Third Italian Conference on Computational Linguistics (CLiC-it 2016).
Jaime S. Cardoso and Ricardo Sousa. 2011. Measuring the performance of ordinal classification. International Journal of Pattern Recognition and Artificial Intelligence, 25(8):1173-1195.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of SemEval-2017, pages 1-14.
B. Dom. 2001. An information-theoretic external cluster-validity measure. IBM Research Report.
M. Fatourechi, R. K. Ward, S. G. Mason, J. Huggins, A. Schlögl, and G. E. Birch. 2008. Comparison of evaluation metrics in classification applications with imbalanced datasets. In 2008 Seventh International Conference on Machine Learning and Applications, pages 777-782.
Lisa Gaudette and Nathalie Japkowicz. 2009. Evaluation methods for ordinal classification. In Canadian AI 2009, volume 5549, pages 207-210.
Aniruddha Ghosh, Guofu Li, Tony Veale, Paolo Rosso, Ekaterina Shutova, John Barnden, and Antonio Reyes. 2015. SemEval-2015 task 11: Sentiment analysis of figurative language in twitter. In Proceedings of SemEval 2015, pages 470-478.
Noriko Kando, editor. 2008. Proceedings of the 7th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering and Cross-Linguual Information Access, NTCIR-7, National Center of Sciences, Tokyo, Japan, December 16-19, 2008. National Institute of Informatics (NII).
Marina Meila. 2003. Comparing clusterings. In Proceedings of COLT 03.
Alistair Moffat. 2013. Seven numeric properties of effectiveness metrics. In AIRS'13, pages 1-12.
Jason Rennie and Nathan Srebro. 2005. Loss functions for preference levels: Regression with discrete ordered labels. Proceedings of the IJCAI Multidisciplinary Workshop on Advances in Preference Handling.
Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of SemEval '17. ACL.
Sara Rosenthal, Preslav Nakov, Svetlana Kiritchenko, Saif Mohammad, Alan Ritter, and Veselin Stoyanov. 2015. SemEval-2015 task 10: Sentiment analysis in Twitter. In Proceedings of SemEval 2015, pages 451-463.
Sara Rosenthal, Alan Ritter, Preslav Nakov, and Veselin Stoyanov. 2014. SemEval-2014 task 9: Sentiment analysis in Twitter. In Proceedings of SemEval 2014, pages 73-80.
Fabrizio Sebastiani. 2015. An axiomatically derived measure for the evaluation of classification algorithms. In Proceedings of ICTIR 2015, pages 11-20. ACM.
Marina Sokolova. 2006. Assessing invariance properties of evaluation measures. In Proceedings of NIPS'06 Workshop on Testing Deployable Learning and Decision Systems.
Stanley Smith Stevens. 1946. On the theory of scales of measurement. Science, 103 (2684):677-80.
S. Vanbelle and A. Albert. 2009. A note on the linearly weighted kappa coefficient for ordinal scales. Statistical Methodology, 6(2):157-163.
Willem Waegeman, Bernard De Baets, and Luc Boullart. 2006. A comparison of different ROC measures for ordinal regression. In Proceedings of the CML 2006 workshop on ROC Analysis in Machine Learning.
Appendix A. Example computation of CEM
Figure 3 illustrates the computation of CEM for two systems (A and B) on the same ground truth with the three usual classes in sentiment analysis: negative, neutral, positive. The ground truth distribution is 10, 60 and 30 items, respectively, which is all the information needed to compute proximity between classes. Note that proximity of one class with respect to other is  $-\log$  of the amount of items that lie between them (including all items in the ground truth class and half of the items in the system-predicted class) divided by the total number of items. The lowest score corresponds to the proximity between the two extreme cases (in the example, the negative and positive classes), because all items except half of the items in the system-predicted class lie between them, and therefore the  $-\log$  value is minimal.
System A and System B in the figure both have the same accuracy (0.70), but system B receives a higher  $\mathrm{CEM}^{\mathrm{ORD}}$  score (0.76 vs 0.71). The main reason is that system A makes more mistakes between distant classes (positive and negative). Another reason is that system A makes more positive/neutral than negative/neutral mistakes; and positive/neutral errors are more penalized by the metric than negative/neutral. The reason is that, together, the positive and neutral classes represent  $90\%$  of the items in the dataset, and therefore are considered less close from an information-theoretic point of view.
Appendix B. Metric Properties Counter Examples
Here we provide examples of how certain metrics fail to satisfy some of the properties proposed in the paper.
Ordinal Monotonicity. Let us consider the set of categories  $\mathcal{C} = \{1,2,3,4,5\}$ . All classification metrics and correlation coefficients fail to satisfy ordinal monotonicity, given that for all of them:
$$
\operatorname {E f f} ((1, 2, 3), (3, 4, 5)) = \operatorname {E f f} ((2, 3, 4), (3, 4, 5)).
$$
But, according to the ordinal monotonicity property, the system output (2, 3, 4) should receive a higher value than (1, 2, 3), because all predicted classes are closer to the ground truth labels.
Ordinal Invariance Pearson correlation, and every error minimization metric fails to satisfy ordi
nal invariance, given that for all of them:
$$
\operatorname {E f f} ((1, 2, 3), (3, 4, 5)) \neq
$$
$$
\operatorname {E f f} ((f (1), f (2), f (3)), (f (3), f (4), f (5))
$$
being  $f$ , for instance, the strict (not linear) increasing function  $f(x) = 10 + x^3$ .
Imbalance. According to the imbalance property,
$$
\operatorname {E f f} ((1, 2, 2, 3), (1, 1, 2, 3)) >
$$
$$
\operatorname {E f f} ((1, 1, 2, 2), (1, 1, 2, 3)).
$$
Metrics that do not satisfy this restriction are Accuracy  $\left(\frac{3}{4},\frac{3}{4}\right)$ , Accuracy with 1 (1,1), MAE and MSE  $\left(-\frac{1}{4}, - \frac{1}{4}\right)$ , cosine similarity (0.973,0.979) and Pearson (0.85,0.9).
Appendix C. Proofs
Here we provide proofs for the properties satisfied by metrics in our study. For the sake of brevity, we do not include formal complete proofs, but their explanations.
Proof for closeness conditions at ordinal scale: Focusing on the ordinal scale, if  $x$  is located between  $y$  and  $r$  ( $y \leq x \leq r$  or  $r \leq x \leq y$ ), then  $|f(x) - f(r)| \leq |f(y) - f(r)|$  for any strict increasing function  $f$ . In other case, that is, if  $x < y \wedge x < r$  or  $y < x \wedge r < x$  we can define a strict increasing function that invalidates  $|f(x) - f(r)| \leq |f(y) - f(r)|$ . The reasoning for the strict case is similar.
Proof for  $\mathrm{CEM}^{\mathrm{ORD}}$  properties:  $\mathrm{CEM}^{\mathrm{ORD}}$  is computed over ordinal comparisons  $(g(d') \preceq_{\mathrm{ORD}}^{g(d)} s(d))$ . By definition, closeness at ordinal scale is invariant under ordinal transformation. Therefore,  $\mathrm{CEM}^{\mathrm{ORD}}$  is ordinal invariant. Monotonicity is also satisfied given that approaching the predicted category to the ground truth category necessarily reduces the amount of documents appearing in intermediate categories (provided there is no empty category in the gold standard), and therefore increases the similarity weight used by the metric. Finally, imbalance is also satisfied given that, being  $g(d_i) = c_i$  and being  $c_i$  and  $c_j$  contiguous classes:
$$
\begin{array}{l} \mathrm {C E M} ^ {\mathrm {O R D}} \left(g _ {d _ {i} \rightarrow c _ {j}}, g\right) - \mathrm {C E M} ^ {\mathrm {O R D}} (g, g) \\ \propto - \log \left(\frac {n _ {i} + \frac {n _ {j}}{2}}{N}\right) - \left(- \log \left(\frac {\frac {n _ {i}}{2}}{N}\right)\right). \\ \end{array}
$$
system A	ground truth
neg	neu	pos	total
\( neg_A \)	5	5	7	17
\( neu_A \)	1	50	8	59
\( pos_A \)	4	5	15	24
total	10	60	30	100
ground truth
neg	neu	pos	total
system B	\( neg_B \)	7	12	4	23
\( neu_B \)	1	45	8	54
\( pos_B \)	2	3	18	23
total	10	60	30	100
class proximity
neg	neu	pos
neg	4.32	0.62	0.07
neu	1.32	1.74	0.74
pos	0.23	0.42	2.74