Mathematical Characterization of Better-than-Random Multiclass Models
Authors: Sébastien Foulle
Venue: Accepted by TMLR ()

Abstract: A binary supervised model outperforms chance if and only if the determinant of the confusion matrix is positive. This is equivalent to saying that the associated point in the ROC space is above the random guessing line. This also means that Youden's J, Cohen's $\kappa$ and Matthews' correlation coefficient are positive. We extend these results to any number of classes: for a target variable with $m \geq 2$ classes, we show that a model does better than chance if and only if the entries of the confusion matrix verify $m(m-1)$ homogeneous polynomial inequalities of degree 2, which can be expressed using generalized likelihood ratios. We also obtain a more theoretical formulation: a model does better than chance if and only if it is a maximum likelihood estimator of the target variable. When this is the case, we find that the multiclass versions of the previous metrics remain positive. If  $m>2$, we notice that no-skill classifiers are only a small part of the topological boundary between better-than-random models and bad models. For $m=3$, we show that bad models occupy exactly 90\% of the ROC space, far more than the 50\% of the two-class problems. Finally, we propose to define weak multiclass classifiers by conditions on these generalized likelihood ratios.

Published in Transactions on Machine Learning Research (06/2025)
Mathematical Characterization of Better-than-Randon Multiclass Models
Sébastien Foulle
Marketing, Customer Experience and Institutional Relations Department
Abeille Assurances
80 Avenue de I'Europe
92270 Bois-Colombes France
Reviewed on OpenReview: https: // openrevee.. net/ for/ormm? id- VdW9SkALSd
Abstract
binary supervised model outperforms chance if and only if the determinant of the confusion matrix is positive. This is equivalent to saying that the associated point in the ROC space is above the random guessing line. This also means that Youden's J, Cohen's 's and Matthews' correlation coefficient are positive. We extend these results to any number of classes: for a target variable with m 2 2 classes, we show that a model does better than chance if and only if the entries of the confusion matrix verify m(m - 1) homogencous polynomial inequalities of degree 2, which can be expressed using generalized likelihood ratios. We also obtain a more theoretical formulation: a model does better than chance if and only if maximum likelihood estimator of the target variable. When this the case, we find that the multiclass versions of the previous metrics remain positive. If m > 2, notice that no-skill classifiers are only small part of the lopological boundary between beller-than-randon models and bad models. For m 3, we show that bad models occupy exactly 90% of the ROC space, far more than the 50% of the two-class problems. Finally, we propose to define weak multiclass classifiers by conditions on these generalized likelihood ratios.
1 Introduction
In machine learning, supervised multiclass modeling seeks to predict as correctly as possible the values of a categorical larget variable that lakes at least. m distinct values. The necessary and sufficient condition for correct binary modeling is well known: we must have sensitivity 2 1 1-11-ppp specifiity, equality corresponding to random models. The ROC curve of probabilistic model must therefore lic above the line segment of no-skill classifiers, and the area under the ROC curve is between that of the perfect model and the area under this line.
The conditions for correct modeling have not received much attention when m 2 3. Practitioners have successfully focused on designing good multiclass algorithms, with model performance measured, for example, by macro-averaging classical binary metrics. But this doesn't provide a clear understanding of the multiclass case, especially when comes to generalizing theoretical concepts such as AUC other metrics. From more algorithmic perspective, ensemble learning combines weak classifiers, that is models that perform slightly better than random guessing. Weak multiclass classifiers have been built by avoiding, for example, conditions on accuracy that are too strong or too weak (Schapire & Freund 2012 Chapter 10). Knowing the exact conditions to be satisfied can thus help in the design of these algorithms. Our main objective is to obtain a mathematical characterization multiclass models that do better than chance, which we call decent models. However, it beyond the scope of this article to study cost-sensitive models, multiclass probabilistic classification or which multiclass metric is best suited a given use case, be it Matthews' correlation coefficient, Cohen's k the volume under the ROC surface, for example.

---

Published in Transactions on Machine Learning Research (06/2025)
Researchers have concentrated on the actual construction of the ROC hypersurface and the calculation of the associated volume (see Kleiman & Page 2019), which do not require defining what better-than--andom model is. The only notable exception is in Ferri et al. (2003), which characterizes these models when m = 3. Edwards et al. (2005) state a second thcoretical result: for any m > 2, and with a certain choice of axes in the ROC space, the hypervolumes under the ROC hypersurface of perfect model ('perfect ideal observer") and a no-skill classifier ("guessing observer') are equal to 0. The authors therefore observe that we no longer have a continuous variation of the AUC when the difficulty of the task to be modelled varies from the simplest to the most complex, unlike binary classification. And Powers (2011) proposes multiclass generalizations of several binary metrics, for example in the form of fractions with a positive denominator and as numerator the determinant of the confusion matrix. In what follows, we show that some of these results are incorrect, that others are confusing, and that the definitions of some multiclass metrics are unconvincing.
To get to the multiclass sctting, we need concepts that are simpler to state and understand when m = 2. The first section recalls the definitions of classical metries as well as those of lifts, which are metrics mainly used in marketing. We then describe how to characterize decent binary models using these different metrics.
The second section formalizes well-known property of certain binary metrics, the invariaaee. These metrics do not depend on prevalence and therefore mcasure model characteristics. We establish links between lifts, which depend on prevalence, and positive and negative likelihood ratios, which are prevalence- invariant. These links are decisive in achieving our main result.
The third section is devoted to the new results we derive for the multiclass case. We first characterize random models and generalize some of the metries seen in the two-elass tasks. We relate these metrics to binary One versus Rest models, and we state the theorem that links lifts and gencralized likelihood ratios. This result makes it possible to define decent (or random) multiclass models as follows: all likelihood ratios must be greater than or equal to one. It is easy to reformulate these conditions for any m 2 2: a model does better than chance if and only if it is a maximum likelihood estimator of the target variable. The propertics of these models are illustrated on different examples. In particular, we show that a bad model can look good on one data set and bad on the over- or undersampled data set. This phenomenon does not appear to have been noted before and only occurs for m 2. We then check that multiclass versions of Cohen's K, Matthews' correlation coefficient and a generalization of Youden's J take positive values on decent models. And we prove that the multiclass balanced accuracy of decent model is always greater than 4 this bound being optimal. We also establish that our multiclass version of Youden's J satisfies a property not verified by an analogous formula proposed by Powers (0011). In the next section, we characterize the topological boundary between decent and bad models. In particular, we note that random models are only a small subset of this boundary, which leads us to qualify EEwards et al. (2005)'s 's conclusions. We then examine an earlier criterion in Ferri et al. (2003) and show that it is too restrictive in its characterization of bad thrre-class models. This leads us to recalculate the volume occupied by bad three-class models in the six-dimensional space of all three-class models. This volume represents 90% of the total volume, much more than the 50% of the ROC space when m = 2. We conclude with proposed definition of weak multiclass models based on generalized likelihood ratios. In particular, we derive minimum and maximum values for the multiclass balanced accuracy of a weak classifier.
2 Reminders on Binary Models and Metrics
We review how to characterize binary models that do better than chance, using several metrics that we'll find again in the multiclass case.
Definition 1.
A binary target variable is a function y: : O + {0, 1) defined on a set of observations, and the class of interest is y 1. A binary predictive model of y is function I O + {0 obtained by supervised learning algorithm.
The confusion motria mat(a, b, e, d) counts the crossed values of y and 9J:
= 0
ÿ 1
==0
0
a
1=1
c
2

---

Published in Transactions on Machine Learning Research (06/2025)
The number of obsercations is n c
The prevalence is À = P(y =) = ctd We assume that 0 < X < 1, that is y(0) = {0, 1} (in other
words, each row of the confusion matria contains at least one non-zero entry).
The detection prevalence (or positivity rate) is = P(ğ = 1) = btd
2.1 Classical Metrics
We recall the definitions of several well-known metrics. For two events A and B, P(A B) Р(AПB) F(B)
is the
conditional probability of A knowing B and A O is the complementary of A.
Definition 2.
d
d/n
sensitivity Pl y 1)
c+ X
a/n
specificity = Pf = 0 0)
a+b 1- X
a
d d/n
precision F(y 9 1)
b+d
a/n
precision, = Ply = 0 | و 0) c
1
Youden's index J = sensitivity + specificity - (ad - bc)/n²
X(1 A)
Matthews' correlation coefficient MCC = corr(y. 0) (ad - bc)/n2
VX(1- N)
Cohen's K Ply () Ap (1 A) - 11)
2(ad bc)/n2
X(1 (1) + uu(1 X)
X(1 - 11) + 11(1 - X)
Positive likelihood ratio LR, sensitivily
d(1 A)
specificity
bd
Negative likelihood ratio LR_ 1 sensitivity
c(1 X)
specificity Diagnostic odds ratio DOR LR+ ad LR_ bc
aA
precision
d(1 11)
Relative risk RR 1 precisiong
qu
The definition of x requires some explanation: we start with the probability P(y 9) = P(y = f 1) + P(y = y 0) aid Then we subtract this same expression when and 9 are independent, which is Ply = 1)P = 1) + F(y 0)P(O = 0) = Ap (1 - X) - 1)). And we divide by 1 minus this expression to normalize the result. The denominator is therefore 1 - Ay - () = 2-p--2p= x(1=))+((-AA)
Finally, we recall the definitions of the ROC curve (or Roceiver Operating Charactoristic curve) and the ROC space (see for example Davis & Goadrich 2006). The ROC space is the square [0, [0, in which each binary model is represented by the point with coordinates (1 specificity, sensitivity). It is made up of three distinct parts: the straight line with equation J = 0 corresponds to random models. The upper triangular region defined by J > 0 corresponds to models that do better than chance, also called "decent models" in the following. And the lower triangular region defined by J < 0 corresponds to models that do RR a predictive score y, for any threshold t we define a binary model j. by de 1 if p 2 and ýe otherwise. In ROC space, the ROC curve connects points de and AUC stands for Area Under the ROC Curve.
worse than chance. pf
2.2 Lifts
In addition to the above, will need the lift. (see Berrar 2019 here's reminder of its main characteristics.
3

---

Published in Transactions on Machine Learning Research (06/2025)
Definition 3. For two events A and B with non-zero probability, lift is defined by
P(ANB) P(A B) P(B A)
Lift(A, F(A)P(() P(A) P(B)
Events A and B are said to altract (resp. are independent, repel) LifM ( B) 1 (resp. =1, < <1), And for any model i of a target variable y such that μ 0, note lift Lift(y = 1. 8 1) for short.
To say that A and B attract is equivalent to saying that we have P(A B) > P(A) and P(B A) > P(B). In other words, one of the two events occurs, the probability observing the other event increases.
Lift has the following properties:
Lift is symmetric: Lift(A, B) = Lift(B, A)-
Lift is easily interpreted: 'model 9 of y detects the positive class lift times better than chance'; f = 1) because Ppy 1
= lift Pyy = 1).
Lemma [ shows that Lift(A, B) and (if B) have opposite signs.
Lemma 1. For bwo events A and B wilh 0 < (AA) < and 0 < P(BB), we have ((A) (Lift(A, B) - 1) = PP (A (1 - (A B))
So for < and < u <
Lift = 1, v = 1) - 1=
(1 Lift = 0. f = 1))
X
1 - 11 (1 Lift(y 1.9 0))
1
(1
A)(1 - 11)
(Lift(y 0, 0= 0)
df
Proof. By definition
P(AnB)
1) = P(A B) TP())
(AA)(I B) 1) -P(A)
P(A))(()
Replacing A by A gives P(A9) (Lif((A9 -) = IPA [B)-P(A9) Then P(A B) = 1-P(AA1 B) B)-
P(A) = (PA9)) - P(A B), giving ((A) (Lift(A, B) - 1) = -P((4) (Lift(A B) - 1) = PA9) (1 Lift(A (B,,)A B))
We can have μ or μ 1 when 9 is constant model equal to or 1. If μ = 0, Lift = 1, 1) and Lift = 0, 9 = 1) do not exist. If p===1 1, Lift(y 1, 0) and Lift 0, á = 0) do not exist. Proposition 1. We have the following equalities (when all members are well defined):
Liftly = 1, = 1) (ad - be) /n²
(ad - be:) /n2
1 1 -Liiftyy = 0,0 = 1)
0,
(1 2)
(ad bc)/n²
Lift(y = 0, = 0)
(1 X)(1 11)
(ad - bc)//r)
1 - -iift(y = 1.9 = 0) =
X(1 11)
Proof. By definition, Ap(lift 1) Ply = و = 1) Ap d(a do+ n
ed-bee From Lemma wc n²
can deduce the formulas for the other lifts, assuming yu(1 11) to be non-zero. By positing μ = then f = 1, we can check that the formulas remain true (when both members of the equation are well defined).
4

---

Published in Transactions on Machine Learning Research (06/2025)
2.3 Decent Binary Models
Models fall into three categories:
Those that do better than chance.
Models that work like chance.
Models that do worse than chance.
For a model of the first category, we want the events {v = 1} and (0 = 1} to attract, and similarly for the events fy=00 0} and {Đ- f0=0) 0}. Proposition shows that this equivalent to ad - be > 0. Models that procood randomly are those that are independent of the larget variable, which is equivalent are equal 1, that is ad be O. The constants 0 and 1 are part of these models, and constants are not random phenomena in the usnal sense of the term. We therefore prefer to refer to them uninformative rather than random models. The other models are characterized by ad be 0. For these, the events {y } and (ú 0} attract each other, well as (y= = 0} and (Ø = 1}. In other words, these models tend to predict 0 instead of and vice versa. This is most often the result of an accidental permutation of the class labels. This leads us to the next definition.
saying that all lifts
Definition 4.
A model is decent if ad - be > 0.
model uninformative if ad - be 0.
model erroneous if ad be < 0.
An erroncous model can be transformed into a docent model by replacing ý by 1 9 (we will find out on an example that in the multiclass setting, swapping the values taken by doesn't 't transform every bad model into decent one). And an uninformative model ğ is a model characterized by the value p P(ğ = 1). It has sensitivity P(ğ = 1 y 1) PPO 1) y and specificity IP(g 0 0) P(g 0) = /4.
The next proposition shows that the category of binary model can be deduced from the values taken by any of the usual metrics. We'll see later that a single metric is no longer sufficient in the multiclass case.
Proposition 2. The following statements are equivalent:
Model i of y is decent
Lyft(y 1, 0 1)
Lift(y 0, 0)
Lift(y 1, 0) <
Lift(y - 0, - <1
J>0 0
MCC >
LR4 >
LR_ <1
DOR >
If any of these inequalilies is equality, the model is uninformative and all inequalities equalilies.

---

Published in Transactions on Machine Learning Research (06/2025)
Proof. This is obvious for the four lifts and metrics J, MCC, к and DOR, from Definition and Proposition Using J, model is decent (resp. uninformative) if and only if sensitivity > specificity (resp. = 1 - specificity) that is specificity > 1 - sensitivity (resp. = I sensitivity). By division we deduce the
desired result for L and LR...
There are two other ways of interpreting the three categories of models, based on the sign of or MCC:
In the ROC space, decent (resp. uninformative, erroncous) model is above (resp. on, below) the diagonal line with equation sensitivity = -specificity-
We have models that are positively correlated with the target variable, those that are independent of it, and finally models that are negatively correlated with the target variable.
3 Links between Prevalence and Metrics
Suppose have binary target variable y O + {0,1}. If we train model O and apply it to test data set T, the statistical distribution of the characteristics of observations verifying y (resp. must be approximately the same on O and T. On the other hand, the prevalence of the test data set may differ from that of the training data set: we sometimes over- or under-sample the data to train the model. In practice, therefore, a model is applied to different data sets, which differ from each other only in their prevalence (apart from random variations). So a model & applied to two data sets will have approximately confusion matrices mat(a, b, c, d) and mat(ua, ub, vo, vd) with and 0.
0)
3.1 Prevalence-Invariancee
We formalize well-known property here, and obtain first new result: Youden's J allows us to characterize models with poor predictive ability, whatever the values of À and μ.
Definition 5. A metric is gevalence-invarian (also noted X-inwariant) if it does not depend on prevalence. In other words, the metric applied to the confusion matria below must not depend on the positive numbers u and v.
9 = 0 9 ==
0 1
ub
vd
This equivalent to saying that such metric is function of and - (by posing _ and = 4). As specificity _ and sensitivity I+5 Among the metrics we've ve already encountered, J, LR+, LR_ and DOR are well known for being X-invariant.
1 X-invariant metrics functions of sensitivity and specificity.
We note that the sign of ad - be does not change if we multiply the rows of the matrix by u > and > 0. In other words, whether a model is docent, uninformative or erroncous does not depend on prevalence. In the multiclass case, we'll see that this property isn't 't automatic and that it's an integral part of the definition of decent model. In general, X-invariant metrics be used to measure the absolute quality of model, as shown in Example
Example 1. Consider a first data set D, a binary target variable and a model 9 of y described by the following confusion matria. We claim that this model is excellent, albeit imperfect.
0
=
= 0
999
1
= 1 1
999
We now consider second data E, obtained from the first by replicating the negative class millions of times. The model i applied to this second data has the following confusion matrix. Model precision is equal to 0.999 D and 0.000998 F.

---

Published in Transactions on Machine Learning Research (06/2025)
9
y
999 000 DO0 1 0D0 000
y
999
Does this mean that model can be both good and bad? Obviously not. But if we use non X-invariant metric, even the best model will look bad on a very unbalanced data set. On the other hand, with a A- invariant metric, the model quality istimate is independent of the prevalence of the data set used. It is therefore a model-specific quality that is mcasured.
Suppose we want to detect poorly predictive models. This a property of models, so only X-invariant metric can reveal it. The metric J fits the bill perfectly, as the next proposition shows.
Proposition 3. Let K = max((sensitivitty pll /specificity 1 + ul). We have
S K &
| S 2K
Proof. Given Φ = sensitivity- and & = specificity 1+ we have & 1 1)
1 p P(v=(=1) /
(od-be)/nn (1 A)J according to Proposition 1 We deduce vb sensitivity + / v and these three quantities have the same
p lift- μ
J - A1- XJ, hence 0 s K < J]. Furthermore J sign, that of ad bc. So have [ |o) + |&ا which gives [J| < 2K.
According to this proposition, J is small if and only if the sensitivity and specificity of the model are close to their values for uninformative models. This result had been observed numerically in Chicco et al. (2021)
3.2 Relationships between Likelihood Ratios and Lifts
Likelihood ratios are also X-invariant metrics, and the next proposition shows their narrow links with lifts (see Vu et al. 2019 for similar formulas): we can express likelihood ratios as functions of lifts and vice versa. These formulas will be used to define generalized likelihood ratios and generalized diagnostic odds ratios in the multiclass setting.
Proposition 4. We have the following equations:
LR+ Lift 1, 1)
Lift(y 0,
Lift(y 1, 0)
LR_
Lift(y = 0, 0)
Lift(y 1, L)Lift(y 0, y 0)
DOR
Lift = 1, 9 = Lif = 0, 0 = 1)
LR -1
lift - 1 = (1 A) A(LR - 1) +1
1 - -LR_
- Lift(y 1, = 0) = (1 - A)
1 1 X(1 LLR
Proof. By definition Lift(y 1, 1= =) d/n Lift = 0, 9 1) b/n Lift 1,0 0) c/n (1-X)a²
x1-=) and
Lift(y = 0, ÿ = 0) = [ーーXI1-p] We then obtain the first three cqualitics by applying Definition 2 We also
a/n
have μ PP 1) = P(l0 1} by 1}) P(10 1) tv 0}) Ppy 1)P 1 y P(y
0)P(O 11 = 0) = sensitivvty + (1 () specificity). This results in
lift = sensitivity
sensitivity
Xsensitivity + (1 A)(1 - specificity)
LR.
LR, - X(LR - 1) 11 (1- (LR (1-))(LR,
ALR +1
X(LR - 1)+1 +1
X(LR - +1
7

---

Published in Transactions on Machine Learning Research (06/2025)
We deduce the second equation by exchanging the values 0 and 1 of g.
We can make metric A-invariant by making À tend towards 0.
Corollary 1. If sensitivity and specificity are fized:
LR+ lim lift
X-
LR_ lim Lift 1, 0)
X-0
Life(y 1, = X-0 Lift(y = 1.0 = 0) X-+0
DOR = lim
lim RR
Proof- We make À tend towards 0 in the last two equations of Proposition to obtain the first two results. We then apply the formulas DOR LR+ and RR d(1-p) op Ly
Lif(g=1.j=1)
LR_
3.3 Normalized Likelihood Ratios
This section is not useful for preparing the definition of decent multiclass models. But the three new metrics defined here may be of interest to practitioners of likelihood ratios and odds ratios by applying the same transformation to LR+, LR_ and DOR, we obtain metrics with a wider definition domain, which are zero for uninformative models and take the value the model perfect. We briefly need the following notations, with 'CO' standing for "centered odds".
Definition 6.
CO(LR4) LR., -
J
LR +
sensitivity + specificity
1 LR
CO(LR-) 1 LR
- sensitivity specificity DOR ad bc CO(DOR) = DOR +1 ad + be
Another simple way make metric X-invariant to set A 0.5. Using Proposition we obtain
2 lift = CO(LR+)) LR+ +
This metric has larger domain of definition than LR., it is well defined as soon as μ # 0. And it's clear that the best of a set of models for LR+ is also the best for this new metric. They have two similar geometric interpretations. Let M be point the ROC curve and O the origin:
(R) is the tangent of the angle between the line segment [OM] and the æ-axis.
CO(LR+)(M)) is the tangent of the angle between the line segment [OM] and the line of equation y because tan (w- - 4)
tan(w)-1 [
We also have 1 - Liftly = 9 = 00 = CO(LR-) = I+LR_ 1. This metric is defined as soon as 1 and here, too, the best model for one of the two metrics is the best for the other. Finally, the metric CO(DOR) = DOR+1 is defined for μ(1 1---)) μ)) * 0 and both metrics always have the same best model. It clear that CO(LR+), CO(LR_) and CO(DOR) lie ween 0 and for decent models. These transformations can be interpreted in terms of odds: for any x 2 0, there exists E [0, 1] such that Th,
2
2
and then have + 2p E [-1

---

Published in Transactions on Machine Learning Research (06/2025)
4 Multiclass Models and Metrics
We consider an integer m 2 3, a target variable y O + {0, 1, ..., m - 1 such that y(0) = {0, 1, m 1
and a model 0: O + 10,1,...; m - -1}. Let nx, be the number of observations in the set {u = 1] 0 (0 - 5
and (nu.))., the confusion matrix. We note ni. = DE Ra,j, n. = E; ni.j. n = Σ, n.j nン = >, mi x = _ and
Py %. By definition we have A > 0 for any i, and when m = 2 we get Ao = 1 1 X, A = X, 10=1-1 and μ1 μ.
All the results in this section are new, apart from the next characterization of uninformative models and the multiclass definitions of Matthews' correlation coefficient and Cohen's k.
4.1 Models
An uninformative model is one that tells us nothing about the target variable, that is y and á are independent variables. We can rephrase this condition as follows (see the 'guessing" observer in Edwards et al. 2005). Proposition 5. The model أ of is uninformative if and only if the confusion matrir is of rank 1.
Proof. If ÿ is uninformative, we have P(t = it n - j}) Pyy = i)P(C = j) for any and j, which is equivalent to ns n.R,,y for any i and j. All the columns of the confusion matrix are then proportional to the non-zero vector (nt.); (and the rows are proportional to the vector (m,)))); which implies that the confusion matrix is of rank 1.
Conversely, if this matrix is of rank 1, the non-zero columns of the matrix are proportional to each other, and therefore proportional to their sum (nt.). So there exists family of numbers (3,) such that 14, for any i and j. Summing this equations over gives n.j 8ym, hence na n. for any i and j. ]
n4.3, 3
This proposition shows that the model 9 of y is uninformative if and only if there are two families of numbers (as); and (3z) such that P(fv= = i) n =}}] = a,3, for any i and j. On a balanced data set, the confusion matrix of an uninformative model has its rows equal, and its rate of well-classifedd observations is therefore m. We'll see later how to generalize this result to any data set, by replacing accuracy with balanced accuracy. It may be tempting to extend the metric formulas of the binary case by replacing ad be by the determinant of the confusion matrix (naj)ss (see for example Powers 2011). but this is not good idea: this determinant is zero if and only if (oss)s. is of rank < m, which is a much weaker condition than the one in the previous proposition. We will see later an example of a decent multiclass model with a confusion matrix whose determinant is zero.
1.
We generalize part of Proposition f to define the following pointwise metrics. Definition 7. For any and j, let
Lift(y- = i = j) P(yy = i} to= (1)
Xijti
P(ğ = j y =
Liftly -.0 - j) Njjj Ai Lift = i =
LR
PPO = i-
nijj X3
DOR,, = DORG, = LRagLRy, ne,
P(j = j =lbj= 1)
P(j
For m = 2 we find LRo,1 = LR+, LR LR and DOR = DOR. The next result generalizes Proposition and shows a direct link between diagonal lifts Lift(y = i, i) and One versus Rest binary models.
Definition 8. let be largel variable and & model of For any i and j, we define the binary larget variable y; which is the indicator of event {y = i}, and the binary model Ú of yi which is the indicator of event {ύ- j}. When i = j, each model 0; of the target variable yj is referved to as a "One versus Rest" model.
Proposition 6. Using the notations of Definition g we have
Lift 1, 03 1) Lif =
9

---

Published in Transactions on Machine Learning Research (06/2025)
Proof. We want to calculate Lift(A, B) with A = {ys 1} and B {ūg = 1}. We can also write A {y i} and B = (0- j}, which is enough to conclude.
We have seen with Proposition 4 and Corollary 1 that lifts can be expressed in terms of likelihood ratios and converge towards them. This is always true in the multiclass case.
Theorem 1. For any j.
P -) 14 Σ (x(((==)l P(O = 1))
Equivalently, if all maj are non-zero:
A
Li - 1.9 j)
LRis
Proof. We have P(O = j y = 5) - P(- -) j) = P(g = = j) - = - y = i) i) -,, = #1y y= = j) - P2O = j = =)). By dividing by P(O = =jly
y = j) and subtracting 1, we deduce the second
formulation.
Corollary 2. If all probabilities P(ğ k I y = 1) are fixed, for any and j we have
lim (P(j = و = =) - وو) P0 = j 1y= 5) P(O P(- [i-=jlv =
Equivalently, if all P(O k y yy=1 1) are non-zero:
lim Lift(y j.9 = 5) = LRRij A=-
In the following, we sometimes write Lift = -, = =) 2 1 or LR 2 1 because these expressions are simple, but they are not always well defined. The reader can mentally replace them with P(ğ = j y = i) N μy or PD - i y = j) 2 P j i) which are equivalent and always defined.
Intuitively, if multiclass model is decent, we want the events {y= =i and (0 i) to attract each other. In other words we want to have Lift i, y = i) 2 1 for any i, which is equivalent to saying that all One versus Rest binary models are decent according to Propositions 2 and 6 And being decent must be characteristic of the model, independent of the A, values. Corollary then implics that the likelihood ratios are greater than or equal to 1. Conversely, if the likelihood ratios are greater than or equal to 1, the previous theorem shows that we have Lift(y = i, = i) 2 for any i, which leads us the next definition. It does not involve the vector of prevalences (Az)a, so it is indeed characteristic of the model, independent of any oversampling or undersampling of the data set under consideration.
Definition 9 (Decent multiclass model). A model 9 of y is decent if the following conditions are met:
For any j, we have max, PpO j l i) P = j jly- 1y= = j).
There are i and j distinct such that PPğ = j y = 5) < P(O =j,y=
A model that satisfies the first condition but not the second is uninformative, since we then have P(f0 = i} n {v = =)) = Ply = i)P(( j y = i) = Ply = :)P(O j y = j) = an33, for any i and j, using the notations from a previous remark. We can rewrite the first condition in a more familiar way, noting x the observed value of 9, 0 the value to be estimated of y and 0(x) the set argmazzP(2 8): for any x we have x E =0(z). We deduce that a model is decent or uninformative if and only if it is a marimum likelihood estimator of the target variable. When m = 2 we obtain that model is decent or uninformative if we have specificity 2 1 1 sensitivity and sensitivity 2 specificity, that is J 2 O. In the multiclass situation, several equivalences become implications: model is decent. or uninformative if and only if LRa, 2 1 for
10

---

Published in Transactions on Machine Learning Research (06/2025)
any i + J. j, which implies DOR., 2 1. According to Theorem this also leads to j. for any j and one of these inequalities is strict if the model is decent. Applying Lemma 1 we finally have Lyft(y + 1. = 5) S 1 and Lift(y = j. ū + j) S 1 for any j.
=)
Models that are neither decent nor uninformative cannot be called erroneous as in the two-class tasks: a simple permutation of the confusion matrix columns is not enough to make them decent general, as we shall see little further on. In other words, permuting the values predicted by the model does not correct the model's shortcomings. Any model that is not decent or uninformative in the multiclass setting is simply referred to as 'bad model". Being decent multiclass model is rather demanding condition: if there is class k such that 14 > O and R = 0, the model is bad. Poor model performance on one of the classes cannot therefore be compensated for by performance on the other classes.
The following two examples show that accuracy is a misleading metric when m > 2, even if we restrict ourselves to balanced data sets: it is not sufficient on own determine the quality of a model.
00
3
0
Example 2. The confusion matrir B =
2
0
is that of a bad model according the previous remork.
0
3
However, 5 of the 9 observations are correctly classified. And we noticed after Proposition that on a balanced data set with m = 3, an uninformative model correctly classifies only a third of the observations.
11
10
9
100
10
10)
Example 3. Let D
10
10
10
and U =
10
10
10
The confusion matria D is of rank 3, so
12
10
10
10/
the model is not unénformative, and have nJ, 30 for any When the sample is balanced, as it is here, verifying that a model is decent or uninformative is easy: we need only check that each diagonal entry is equal to the marimum volue of its column, which is clear. The percentage of well- pell-classified observations is 33 = 36, 66... which may seem low, and we 'Il come back to this rate in the next section. But our model is 90 clearly better than the uninformative model with confusion matria U.
The model studied here looks good on one data set and bad on the oversampled data set, which shows that examining binary One versus Rest models is not enough to judge the quality a model either.
2
2 1 2
Example 4. Let B1
and B₂
9
6
0
The confusion matrix B1 defined on a
0
4
balanced dala sel, and the model is clearly bad. However, if we place the three binary One versus Rest models in the ROC space, they are all above the line of uninformative models. The confusion matrix B2 is obtained by oversampling the class "1", whose numbers have been tripled, and the "0" versus Rest associated model has
9+0) This model is slightly below the random guessing line of the ROC space, as (1 - specificity, scnsitivity) = (2 5) (0.45, 0.4). Note that an over- or undersampling that lends to such an apparently contrudiclory siluation is impossible if m 2. What the two confusion matrices 11 A < 1.
the confusion malrir (6 16+0+ 0++0+1 1+1
1 1+2
have in common is the value of the likelihood ratio LR1,0 ne.o
We should also point out that the mazimun value of each column of B₁ never in the first row. So no permutation of the columns con transform this bad model into a decent one.
We now study a confusion matrix with a zero determinant, but corresponding a decent model. For any confusion matrix, have lifts greater than or equal to 1 and less than equal to 1 in each row and each column: for any i we have E2 i} n 10- 5}) = X = E; Aijl,, that is 0 E, = i} n (ō j))->uzg), and similarly in each column. In the case of a decent model, one might think that the off-diagonal lifts are less than 1, but the following example shows that this is not true.
2
1
1
1.5
0.6
Example 5. We denote D
2
1
and L
0.75
1.2
The confusion matriz is of rank 2 1
2
1
0.75
1.2
1
(the sum of the first two columns is proportional to the third column), is therefore not that of a uninfor- malive model and its determinent. is zero. The data set is balanced and the model is clearly decent. We have
11

---

Published in Transactions on Machine Learning Research (06/2025)
Ao - = Po 3. M1 p1= off-diagonal lift Ly 2, 1) is greater than 1.
12. M2 = and (Lift(y i.9 9 j))ss L. Note in particular that the
3
If a model is decent, Proposition B shows that merging all but one of the classes results a decent One versus Rest model. But this not always the case if the mergers are partial.
3
1
2
2
2
2
2
2
Example 6. Let D
The confusion matrir D corresponds to a decent model on a balanced
2
2
2
3
2
1
2/
data set. If we merge classes and and classes 2 and 3, we obtain the confusion matric 7 of a bad
model.
We could try to imitate the definition of a decent model to compare two models: say that a decent model D1 the target variable is better than a decent model ÿ2 if all the diagonal lifts of ý1 are greater than those of 82, whatever the prevalences. This would be equivalent to saying that all likelihood ratios of ýt are greater than those of d2. But already when m this path leads nowhere: consider a convex ROC curve (we know that we can always reduce ourselves to this situation). Tet A be the point associated with model ộ1 and B with model Ú2, and let O, P. Q, R. be the points with coordinates (0,0), (1,0), (1,1) and (0,1) respectively. We have (01) > (12)
and only if AOP > BOP, and (01) < (f2) if and only if AQR < BQR. This is equivalent to saying that AOQ BOQ and AQO BQO. We leave it to the reader to convince himself that this is impossible due to convexity, unless A-B.
4.2 Global Metrics
To compare several models of the same target variable, we keep the decent models and estimate their respective overall performance with a metric. Multiclass versions of Matthews' correlation coefficient or Cohen's τ can be used, for example. Definition 10.
MCC E,( P-P(y=0-1 -= ArH1)) V1 -1
(P 9 =) ArHle)
K 1 1- Ap14;
The denominator of к never cancels, and that of MCC cancels only for constant models. Theorem [ shows that the numerator of these two metrics is positive if the model decent. We have seen that lifts minus 1 are interpreted with One versus Rest models of Definition g
MCC and λ are forms of macro-average.
We can extend Youden's JJ to the multiclass case. Tf we still want J to detect decent but not very informative models, we can set J = - (E, P(O = -) -) = - (P(( = = -4)) As is decent or uninformative, J lies between 0 and and all terms of the last sum are non-negative, so ÿ is close to an uninformative model and only if all terms are small, that is if J small. In this way, we' ve generalized all the binary results of the Propositions and 3 replacing certain equivalences with implications. The last Asja)). When all X are equal (and therefore equal to ), we obtain & = J m- P ). Thus docent multiclass models have a rate of well-classified observations greater than is not true when m > 2. Multiclass balanced accuracy is casy to define: let balanced accuracy = = E, P(j l i). which is X-invariant. On balanced data set, this is the rate of well-classified observations. On any data set, we have J = -- (m balanced accuracy 1), and so decent. multiclass models always have a balanced accuracy greater than m. It should be noted that extension of Youden's J differs from that of Powers (2011) which is given by the formula B(R, P) Σ, Prev(1) = =, A (e(t.)d((.a)-MM.)e(.))))) (;(1-A;)
formula can be rewritten as J m-I > (P
m
balanced data sets, but we in Example that the converse
Σ, Agrig Lifly= E, (P = i il = -)) But this formula doesn't detect almost uninformative models. We summarize these considerations in the following definition.
1-X;
12

---

Published in Transactions on Machine Learning Research (06/2025)
Definition 11.
1
Balanced accuracy = m
Σ PQ = i y = =)
1
P(y =i) dip
m (m balanced accuracy - 1) = m
X
4.3 Boundary between Decent and Bad Models in the ROC Space
When = 2, uninformative models separate decent and bad models in the ROC space. We show that this is no longer the case if m 2 3 by describing precisely the boundary between these two types of models. The ROC space defined below as subset of [0, 10. with the usual Euclidean topology. And for any subset B of IRm its boundary B n B where denotes the closure of B.
Definition 12. The set of row stochastic matrices, that is square matrices with m rows and m columns, with non-negative coefficients and whose sum of each row is 1, is called ROC space. For any model 9 of a target variable y, we call 'normalized confusion matrie' the matrir of conditional probabilities (P(( = =ily i)),,, which is an element of the ROC space. By abuse of terminology, we refer lo decent (resp. uninformalive, bad) model in the ROC space to designate the normalized confusion matriz associated with such model.
A first geometric remark can be made about decent uninformative models: they form convex subset of the ROC space, can be seen immediately from the first condition of Definition g
Edwards et al. 2005 show that the hypervolumes under the ROC hypersurface of a perfect model and a random classifier are equal to 0 when m 2 3 in a particular coordinate system: the diagonal elements of the matrix (P(() = j = =))sss are linear combinations of the off-diagonal entries, so they use the m(m - 1) values (P( = j y y= i with different from j as the coordinate system in ROC space (with constraints: the sum of off-diagonal entries in each row is less than or equal to 1). When (I -1- specificity, 1 sensitivity). According to the authors, "this suggests that hypervolume may not be a useful performance metric in N-class classification task for N > 2'. This conclusion needs to be qualified: in the proposition that follow, we show that uninformative models are only a tiny subset of the topological boundary between decent models and bad models when m 2 3. It is therefore more appropriate to compare the volumes occupied by these two calegories of models, which we do in the next section when m 3.
2, these coordinates are
We can see from Example 3 that it's easy distort an uninformative model a little into decent or bad model, so these models are subset of the boundary between decent and bad models. And they form low-dimensional set: according to Proposition 5 the normalized confusion matrices of these models have all their rows equal. The example below shows that the boundary considered is strictly larger.
21 3 3
Example 7. Let =
be a normalized confusion matria with 0 Ostsl. is never the 2)
matrir of an uninformative model, since the last two decent. model for any 1² 0.5 and that of a bad model otherwise. So Co.5 is boundary point of the set of decent or uninformatice models in the ROC space, and it's not an uninformative model. Note also that we have K = J= 2 0. This shows that in the mulliccaass case, classical metric can be zero or posilive even when the model is bad.
not proportional. The matriz Ct is that of
The proposition below formalizes the observations made in the previous example.
Definition 13. In the ROC space, we denote E the set of decent or uninformative models, that is E = ((Ma))a,yV} max, Ma = M,5). And for any lo and I & k. let Ex,1 = {(Mss)ss E E|M = M,,). Proposition 7. The boundary of E Usyt Exa and the set of uninformative models is Nkpp Ex.
Proof. Let M be an element of the boundary of the set of docent or non-informative models. This means that there is a sequence (D(n)), of decent or uninformative models such that lim D(n) M and a sequence [ (n)),, of bad models such that lim B(n) M. Denoting (D(n)ss)ss the normalized confusion matrix
13

---

Published in Transactions on Machine Learning Research (06/2025)
of D(n), we have the equations max, D D for any j. By letting n tend to infinity, we deduce that M is decent or uninformative.
Moreover, for any there exist i(n) and j(n) e i(m) such that (()))))))( > 5(0)50))) There are at most m(m 1) distinct pairs (t(n),j(n)), there is therefore at least one pair JI for which there exists an infinite subsequence of (i(n).((n)), taking this value JJ, We replace the sequence (i(n),(n))) by this subsequence, giving B(n)r,J > B(n),, for any n. When goes infinity, we get Mr 2 My and since M is decent or uninformative, we actually have MI My and M Er,J.
Conversely, if we have M E EI,,, for any integer n 2 1 we define a matrix (B(n))a as follows:
If Mr.J = MJ = 1 (and therefore Ma5 = 0 for any j + J), we pose BJ, = MJ. - 1, B1 and Bx,1 = Ms. otherwise. In particular, we have BI = MI.J- = =13 BJ,.-
If MI,J Ma,J < 1, there exists k + J with ML. > 0. Let BI = MI., BI,J = MI,1 + Mr.J = MJ = BIJ, BI = MI - - and BEd = Ma,1 otherwise. For n large enough we have B1,J 5 1 and By 20. O.
We thus obtain a sequence of row stochastic matrices corresponding to bad models, such that lim, B(n) M. Moreover, by definition, the point M is an element of E, so it's a boundary point of E. Finally, the statement Nezt Ex, clear.
In the binary case, we have Fa,1 = F and the boundary between decent and bad models is equal to the set of uninformative models.
4.4 Volume Under the ROC Surface for Three Classes
In this section, we allow ourselves an abuse of language: to follow the notations and terminology of Ferri let al. (2003), we speak of volume under the ROC surface. This is in fact the volume occupied by bad models in the ROC space, never consider particular ROC surface associated with predictive score.
To our knowledge, there is only one article in the literature that mentions necessary and sufficient conditions for a multiclass model to be decent. In Ferri et al. (003), we find a characterization of bad models for m 3,
Y/
03
noting (P(ý- = 1 = 1)) =
F1
yb
F8
there exists ha 2 0. hy 2 0, he 2 0 with ha + hB + he =
x2
Na
Ye
such that 22 2 hay x22 has Tz hb, FA
1b, V5 2 hes To 2 hee
Models that meet these criteria are indeed uninformative or bad models: we find = 1 11-x33 x3 1- he = has and similarly yo I hy and Ye S he- So Yo I m2n( Yb S min(r3, .44) and ye I min (Zy, zx)
But these criteria are incorrect because they are too restrictive, as we can see with the balanced confusion
matrix (0) 2 1)
1
The proportion of correctly classified observations is equal to 3 < m we have J <
(1 1
and this model is neither decent nor uninformative. The reader can casily check that all diagonal lifts are strictly less than (in other words, all three One versus Rests models are wrong), which also implies that MCC and are negative. But we have 11 = min (#1, x²) = min(0, 1) O, r2 = min(xs, x44 = min(2 (2 1) 3 r3 min(as, x²) = min( f. 2) = and rı + r2 hs, he) that satisfies the above conditions.
r3- = 3 < It therefore not possible to find triplet
When m = 2, the maximum (resp. minimum) area under ROC curve is that of a perfect probabilistic (resp. random) model, and is (resp. 0.5). For the three-class case, their analogues are VUSTue and VUST noting VUS as the Volume Under the ROC Surface: VUS is the volume of the set of all models in the ROC space and VUSTán is the volume of the set of all bad models. The maximum volume calculated in Ferri et al. (2003) VUS = 1/ = 0.125 We also find a Monte- Monto-Carlo estimate of VUS ~ 0.0055, but the latter value is too low because the number of bad models is underestimated. We obtain a new Monte-Carlo
14

---

Published in Transactions on Machine Learning Research (06/2025)
estimate for 100,000,000 points: VUSS 0.1125. Its exact value is calculated below, and shows that bad models occupy 90% of the space of all three-class models.
Proposition 8.
9 = 0.9 VUSY
VUST
80
3000
201
3502
Proof. Consider the normalized confusion matrix X
210
211
212
subject to the following conditions:
220
221
D/22
0 S Tay S 1 for any and j and Ey Tiy the Tij are all distinct. All these matrices occupy a hypersurface of dimension of volume VUSg We want to calculate the volume VUSg of matrices X where the maximum value of cach column is reached in row j. By symmetry, we have V = 6 VUSS where V is the volume of matrices X where the maximum values of each column are reached in distinct rows. So W = VUS V is the volume of matrices X where the maximum values of each column are reached in two of the three rows. Indeed, we can't 't have the three maximum values of each column in single row, because of the constraints E, Ziz 1. Again by symmetry, have W 9U with U the volume of matrices X where the maximun values of the first two columns are reached in the first row. Finally, we have VUS VUS VUS
for any i. In the volume calculations below, we can impose that
6 VUSg
3.. All that
remains is to calculate U, the constraints being 0 S Tı0 < z00: 0 S < 200, 0 S æι < 201, 0 S 22 201, 200 + 201 < 1, 210 + #11 < 1 and Z20 + S 1. The last two inequalities are consequences of the other incqualitics, which gives
1-x00
Fel
U
droo
dro
dr1o
d0
drn
dr
0
!' wo do00
1-=100
1
T1 daoi
x=
1 dy
x x² - xx)²
dr
3
J
[ (x²
1
1
1
1
3x3
+
+x+
x. dx
3
3
3
4
5
6
3
6
20
180
We deduce V +
3 1
15
1 27
68
2 180
12
12
10
12 20
80
There are several possible extensions the ROC curve to the multiclass setting and other definitions of ROC space, VUS and multiclass AUC, which may be more useful in practice (see Kleiman & Page 2019 Aguilar-Ruiz & Michalak 2024 for example). We won't discuss these variants, as they stray too far from the subject of our article.
4.5 Weak Multiclass Classifiers
In this final section, we show how inequalities on likelihood ratios can be used to define weak multiclass classifiers. We state the next proposition using balanced accuracy, which is a linearly transformed version of Youden's J according to Definition
I1 (but in no case do we assume that the data sets considered are balanced). We have already pointed out in the discussion following Definition 10 that if we restrict ourselves to decent models, value of J close to that. is not very informative. And if the balanced accuracy is is nearly equal to its maximum value 1, then the normalized confusion matrix is very close to the diagonal identity matrix and the model is almost perfect.
balanced accuracy close +. means that the model
Proposition 9. Let 8 and & be such that + 8 < LRi,j S 7 for any distinct i and j. Then
1 m
< balanced accuracy S m +
1
7 m
m(m m + 5)
Proof. We have LR,,, = so LR,, S for any i and j. Theorem leads to Lift j. = S y, that is PC = i | = j) S Ma for any We deduce balanced accuracy by summing over j.
15

---

Published in Transactions on Machine Learning Research (06/2025)
For i fixed and any + we have = ( 2)
y= i), and we also have P(g i =il y
(1+))P(f = i y = i) - (p = i = i). Summing over j, we get m balanced accuracy 2 1+8- 135--911 -p i) y i). Then we sum over i, which gives m² balanced accuracy 2 m(1 + 5) - ôm balanced accuracy.
Dividing by m we finally find (m + 5) balanced accuracy 2 1
The assumptions in the previous proposition guarantee that the model does better than chance, and they allow us to control how close or how far the balanced accuracy of the model is to that of a random model. Such inequalities can therefore be a good starting point for defining weak multiclass classifiers used, for example, by boosting algorithm. These incqualities could also remove theoretical barrier: in [Schapire
1 0
0
0 1
0
& Freund (2012), we find an example of two weak classifiers hn =
1 0
0 and h =
0
1 0
'with
0
0
1
0
0
1
accuracy siguificantly better than the random guessing rate of m but. for which no boosting algorithm
can exist that uses such weak classifiers to compute a combined classifier with perfect (training) accuracy'. The authors add that "This difficulty turns out to be provably unavoidable when the performance of the weak learner is measured only in terms of error rale". This led them to completely change their approach, abandoning weak multiclass classifiers and reducing the multiclass problem to several binary problems. We immediately check that (h1) = LRo, (h2) 1. So this counter-example doesn't apply if we no longer use accuracy but rather likelihood ratios as in the previous proposition to define the weakness of classifier It may then be possible to design a multiclass boosting algorithm as close to perfection as desired on the training sample.
5 Conclusion
We obtain a characterization of decent multiclass models based on two assumptions: being decent is a characteristic of the model, independent of the frequency of each class of the target variable. And a decent model must have decent One versus Rest binary versions. This gives us some very simple necessary and sufficient conditions, which generalize in a natural way various well-known criteria for two-class problems. These criteria have enabled to correct relativize several previous results, and they also pave the way for the design of new algorithms for aggregating multiclass weak classifiers.
Decent models are more complex to understand intuitively when m > 2. Even if we restrict ourselves to balanced data sets, the rate of correct classification no longer tells us whether a model is decent or not. Merging certain classes can transform a decent model into a bad one, unless we apply a One versus Rest merger. Uninformative models no longer form the boundary between decent and bad models. And there's no natural way to generalize binary metries into global multiclass metrics, unless you want to preserve particular properties of the metric.
Unexpectedly, we show that the classical metrics seen in the binary case actually hide two groups of metrics. We have familics of pointwise metrics, associated with each cell of the confusion matrix: lifts, diagnostic odds ratios and likelihood ratios. The latter are used to define decent multiclass models. We also have global metrics such as Matthews' correlation coefficient, Cohen's x and Youden's J., they take on positive values on decent models.
Acknowledgments
I'd like to thank my colleagues from the Customer Knowledge team who attend my Python training courses. This forces me go into greater depth on certain subjects to ensure that they are as accurate and complete as possible. I'd also like to thank Célia Etien for her patient proofreading and valuable feedback.
References
J.S. Aguilar-Ruiz and M. Michalak. Classification performance assessment for imbalanced multiclass data. Scientific Reports, 14:1-10, May 2024.
16

---

Published in Transactions on Machine Learning Research (06/2025)
D. Berrar. Performance measures for binary classification. In Encyclopedia of Bioinformatics and Compulational Biology, pp. 546-560. Academic Press, Oxford, 2019. doi:
19978-0-1 12--0-33---- 3.20051-..
D. Chicco, N. Tötsch, and G. Jurman. The Matthews correlation coefficient (MCC) reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation. BioDala Mining, 14(1):1 22, February 2021.
J. Davis and M. Goadrich. The relationship between precision-recall and ROC curves. ICML '06: Proceedings of the 23rd international conference Machine learning, pp. 233 240, New York, NY, USA, 2006. Association for Computing Machinery.
D.C. Edwards, C.E. Metz, and R.M. Nishikawa. The hypervolume under the roc hypersurface of 'near- guessing' and 'near-perfect" observers n-class classification tasks. IEEE Transactions on Medical Imag- ing, 24(3):293 299, 2005.
C. Ferri, J. Hernández-Orall and M.A. Salido. Volume under the ROC surface for multi-class problems. In Machine Learning: ECML 2003, pp. 108 108-120, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg.
R. Kleiman and D. Page. AUCμ: A performance metric for multi-class machine learning models. In Pro- ceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3439 3447. PMLR, 2019.
D. Powers. Evaluation: From precision, recall and f-measure ROC, informedness, markedness and corre- lation. Journal of Machine Learning Technologies, 2(1):37 63, 2011.
R. Schapire and Y. Freund. Boosting: Foundations and Algorithms. The MIT Press, 2012.
K. Vu, R. Clark, C. Bellinger, G. Erickson, A. Osornio Osornio-Vargas, data mining has close relationship with the association measure relative risk in epidemiological studies. BMC Medical Informatics Decis, Mak., 19(1):112:1-11258 2019.
O. Zaïane, and Y. Yuan. The index lift. in
17


Reviews:
[{"review_id": "9BFEud9uWH", "reviewer": "Reviewer_vfzM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- Extending the notion of model decency from binary to multiclass classification is non-trivial, as several properties do not carry over directly. This paper addresses these challenges thoughtfully, clearly discussing the distinctions between the two settings.\n\n- The problem tackled is relevant to practitioners, particularly those dealing with imbalanced multiclass datasets. The framework proposed offers a principled and consistent approach to evaluating model performance in such contexts.\n\n**Weaknesses:**\n\n- I would expect some (preliminary) experiments for the proposed aggregating multiclass weak classifiers (e.g., in boosting context) to see if there is any advantage it might offer in practice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a formal characterization of decent models—those that perform better than random guessing—for multiclass classification tasks, based on an analysis of their confusion matrices. The authors build their argument on two key principles: (1) decency should be an intrinsic property of the model, independent of class imbalance in the dataset, and (2) a decent model in a multiclass setting should also exhibit decency when evaluated using the One-vs-Rest (OvR) approach, a well-studied binary classification framework.  These principles culminate in a formal definition of decency (Definition 9). The authors also analyze the ROC space, highlighting a critical difference from the binary case: in multiclass classification, uninformative models occupy only a small region on the boundary separating decent and bad models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I don't have many suggestions for changes to this paper. However, in Section 4.3, I recommend that the authors provide a more systematic introduction to the ROC space in the binary case, offering the necessary background before extending the discussion to the multiclass case ($m \\geq 3$).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nC0GzIl0xp", "reviewer": "Reviewer_KkCr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Comments:\n\nThe authors may want to define \"ROC curve\" and \" ROC surfaces\" explicitly. It is a bit odd that the authors emphasize elementary properties like Proposition 2 in details, while assuming that the readers are all familiar with ROC curves and surfaces, which are considered to be advanced concepts.\n\nIn addition, it is not rigorous to use \"ROC space\" and \"ROC hypersurfaces\" interchangeably, since \"ROC space\" (Clémençon et.al, 2008, AoS) can also refer to the collection of ROC curves associated with a family of classifiers in formal research. Clémençon and Robbiano (2015, JNS) have established similar results for ROC curves from the perspective of ranking. It will be helpful if the authors can comment how this result shed light on the understanding of ROC curves in the classification scenario.\n\nSection 1: It is friendly to introduce \"One versus Rest\" when moving from binary to multi-class classification context, in particular, the elementary examples in Section 2 can be removed, and the room could be used to have detailed examples in Section 4, where the novelty lives. For the prerequisite concepts that are not used, we can refer the readers to relevant literature instead of listing all concepts. This revision can also help reader to understand claims \"If a model is decent, merging all but one of the classes results in a decent One versus Rest model.\" (P11) later, which itself requires a proof.\n\nExample 2: The phrase “rate of well classified is nonetheless not negligible” is confusing. Is the author attempting to say: “rate of correctly classified observations” or “well‑classified rate”?\n\nSection 2.3: For \"Those that do worse than chance\", practitioners usually just use its complement model. Namely, if the model predicts class A, then we would make the decision of choosing class B. Therefore, the complement model will be a classifier that do better than chance. Is there any reason why we shall consider them separately?\n\nSection 3.2: It is unclear why Proposition 4 provides new insghts. As pointed out by the authors, Vu et. al(2019) has already stated similar results; It seems like that we are describing a Bayes factor for testing the hypothesis that the sample belongs to class 0 or 1, is that correct?\n\nSection 3.3: If this section is not directly related (as stated by the authors), it can be postponed to the appendix. Or the author may want to give examples to illustrate how normalized likelihood ratios are different from the unnormalized ones, since this CO-type metrics seem to be a linear transformed version of original metrics.\n\nSection 4.3: \"And from proposition 5, the normalized confusion matrices of uninformative models have m − 1 degrees of freedom. We can use the values (P(ˆy = j | y = i))i,j with i different from j as the coordinate system in ROC space.\" This is quite loose a claim since Prop 5 is for binary model and this is a claim for multi-class model. I am not sure if this is correct, and the concept of \"degrees of freedom\" comes out from nowhere.\n\nExample 7: This is also confusing. From Definition 12, the normalized confusion matrix needs to have legit row probabilities, and it has to be row stochastic, then what does it mean by \"we can move continuously from a decent model with all its likelihood ratios greater than 1 to a bad model\". If you move in the space of parameter t, then isn't the boundary just a point? if you move in the space of row stochastic matrices, it is not clear this boundary is one of those geometric boundaries in that space (perhaps need better explanation of the geometric interpretation in previous Sec 3.2-3.3).\n\nProposition 7: How do we define boundary here? And the meaning of ROC space requires a rigorous math definition and topology, then we can introduce the concept of boundary. Or I may not fully understand what the author meant here.\n\nSection 4.5: The discussion seems to be focused on the balanced multi-class classification problem, this is quite different from the generality claimed in Section 1, it needs to be clarified how well these analysis can be generalized to the unbalanced scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper fully characterizes when multi-class models truly outperform chance (i.e., a classifier decision made by random coin flipping) via algebraic conditions on the confusion matrix. This extends core binary metrics, maps out the ROC‐space geometry, and even lays theoretical groundwork for multi-class scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the comments above, all these comments should be carefully addressed.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oJ9nc1nr0d", "reviewer": "Reviewer_wubw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I must admit I'm not an expert in the subject matter, but I really enjoyed reading this paper. Given my non-expert nature, I may not be aware of other similar existing results. That being said, the propositions and theorems that the authors present are very interesting, and to some extent even surprising (I'm referring especially to the boundary in the multiclass setting being a superset of the collection of uninformative models). I'm happy to recommend for an acceptance after minor revisions, that are spelled out in the next text box.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors extend the existing results on supervised models outperforming chance to the multiclass setting, and discover many interesting properties.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It is odd to refer to sections of the paper as chapters. Perhaps this work is extracted from a thesis. Please consider changing \"chapter\" to \"section\".\n\n$\\lambda > 0$ and  $\\lambda <1$ hold when $c+d <n$, right? That is, when at least one between $a$ and $b$ are positive.\n\nThe writing can be polished in some parts. For example, in Proposition 4 it would be customary to write something like \"The following hold\" before presenting the formulae. In addition, sometimes Cohen's $\\kappa$ is referred to as Cohen's kappa (this is indeed very minor).\n\nI must confess I don't quite understand the need right after Definition 9 to introduce new notation to claim that a model is decent or uninformative if and only if it is a maximum likelihood estimator of the target variable. Was it to better relate to the most commonly used notation $P(x \\mid \\theta)$ for the likelihood distribution?\n\nPlease do not use contraction as \"let's\".", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a characterization of multiclass models that perform better than chance. The results are non-trivial and could be of interest to practitioners.
Decision: Accept (poster)