Statistical Multicriteria Benchmarking via the GSD-Front
Authors: Christoph Jansen
Georg Schollmeyer
Julian Rodemann
Hannah Blocher
Thomas Augustin
Venue: NeurIPS 2024 spotlight (2024)
Keywords: multicriteria benchmarking
robust statistics
statistical test
imprecise probabilities
reliability
non-standard scales of measurement
decision theory
Primary area: evaluation
TL;DR: We propose the GSD-front for reliable multicriteria benchmarking of classifiers, give conditions for its consistent estimability, propose (robust) statistical tests for checking if a classifier is contained, and illustrate it on two benchmark suites.

Abstract: Given the vast number of classifiers that have been (and continue to be) proposed, reliable methods for comparing them are becoming increasingly important. The desire for reliability is broken down into three main aspects: (1) Comparisons should allow for different quality metrics simultaneously. (2) Comparisons should take into account the statistical uncertainty induced by the choice of benchmark suite. (3) The robustness of the comparisons under small deviations in the underlying assumptions should be verifiable. To address (1), we propose to compare classifiers using a generalized stochastic dominance ordering (GSD) and present the GSD-front as an information-efficient alternative to the classical Pareto-front. For (2), we propose a consistent statistical estimator for the GSD-front and construct a statistical test for whether a (potentially new) classifier lies in the GSD-front of a set of state-of-the-art classifiers. For (3), we relax our proposed test using techniques from robust statistics and imprecise probabilities. We illustrate our concepts on the benchmark suite PMLB and on the platform OpenML.

Statistical Multicriteria Benchmarking via the GSD-Front
Christoph Jansen' c. ansenelaacasserr ac uk
Georg Schollmeyer?* georg schollmeyer@stat. wnw.uuii-mnn. ami-muenchen.de .e
Julian Rodemann 2*
Hannah Blocher
julian@stat. moni-mmenchencco de
hannah. blocher@stat. wnw.unii-mu -muemchen.dee
Thomas Augustin
thomas augustin@stat. uni www-mmuueeeeecemm de
School of Computing & Communications
Lancaster University Leipzig
Leipzig, Germany
"Department of Statistics Ludwig- www.aiiliann--nivvaaaaaa München Munich, Germany
Abstract
Given the vast number of classifiers that have been (and continue to be) proposed, reliable methods for comparing them becoming increasingly important. The desire for reliability is broken down into three main aspects: (1) Comparisons should allow for different quality metries simultaneouslly. (2) Comparisons should take into account the statistical uncertainty induced by the choice of benchmark suite. (3) The robustness of the comparisons under small deviations in the under- lying assumptions should be verifiable. To address (1), we propose to compare classifiers using generalized stochastic dominance ordering (GSD) and present the GSD-front as an information-efficcentt alternative to the classical Pareto-front. For (2), propose consistent statistical estimator for the GSD-front and construct a statistical test for whether (potentially new) classifier lies in the GSD-front of set of state-of-the- tfe--e-rrt classifiers. For (3), relax our proposed techniques from robust statistics and imprecise probabilities. We illustrate our concepts on the benchmark suite PMLB and on the platform OpenML.
using
Introduction
The comparison of classifiers in machine learning is usually carried out using quality metrics C x D - [0, 1], i.e., bounded functions assigning real number every pair (C, D) of classifier and data set from suitable domain D, where, by construction, higher numbers indicate better quality. However, in many applications, the choice of a unique quality metric used for the comparison is not self-evident. Instead, competing quality metrics are available, each of which can be well- motivated but may lead to different ranking of the analyzed classifiers. One attempt safeguard against this effect is to use multidimensional quality metrics: instead of a single metric, one chooses set of metries +- (#1,..., Φn)) C + 10m that taken together provide balanced
marks equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

foundation for assessing the quality of classifiers. Generally, we distinguish two (related but) different motivations for choosing multidimensional quality metries:
Performance is a latent construct: The application at hand suggests a very clear evaluation concept, which, however, is too complex to be expressed in terms of single metric. In this case, the latent construct to evaluate is operationalized with a set of quality metrics (that serve as an approximation). For example, the latent construct of robust accuracy can be operationalized by taking together the following three quality metrics: Accuracy of a classifier (i.e., the proportion of correctly predicted labels), and robustness of this proportion under weak perturbations of the data in either the features or the target variable. This will be exemplified in Section 5.2 using the PMLB benchmark suite.
Quality is multidimensional concept: Even if the application at hand suggests evaluation criteria that can be perfectly expressed using quality metrics, it can still be desirable to compare the classifiers under consideration in terms of various contentual dimensions. For example, one can be interested in how well a classifier performs in the trade-off between accuracy and computation time in the training and the test phase: Clearly distinguishable contentual dimensions are included and the analysis aims at investigating how the different classifiers under consideration trade-off between these dimensions. This will be exemplified in Section 5.1 using one of OpenML's benchmark suites.
Regardless of the motivation for considering multidimensional quality metrics, their interpretative advantage naturally comes at price: Without further assumptions, classifiers will often be incom- parable, as the quality metrics in the different dimensions contradict each other in their ranking Already one data set, a multidimensional quality metric only induces a (natural yet potentially incomplete) preorder: a classifier is rated at least as good as a competitor if (and only if) it receives at least the same metric value in each dimension. The problem of incomparability becomes even more severe for multiple data sets (as considered here). In this case, one of the following analysis paths is often chosen: (I) An expected weighted sum (for example, weighted by importance) of the individual quality metrics is considered and the problem is then analyzed on this new pooled quantity F (II) The problem is analyzed based on the Pureto-front par($), i.e., the set of all classifiers that not component-
coonnenttwiieeeee wise (strictly) dominated by any competitor, whose definition followed by an illustrative
example are included for reference.
Definition 1. Let D C 'D be some set of data sets. The D-Pareto front par(4, D) of d is given by
{c e C/AC e CVD e D: 4CC", D) D)),
where & is the strict part of the component-wit wise 2 2-relation on R". Set par( :=par( D). Example 1. Consider the following schematic example of three classifiers C = {C1, Cz C2} evaluated for a fictitious population of four data sets 'D = =D2, D2, Ds. D1). Every entry gives the -io-dimensional evaluation & (CC, D) of a classifier on a data set w.r.. predictive accuracy and the computation time for training in three ordinal categories fast, medium and s slow
data set
D1
D2
D3
D2
classifier
C1 C2
(0.7, s low)
(0.8, medium) (0.85, fast)
(0.9, fast) (0.91, fast)
(0.95, slow) (0.96, s slow)
(0.75, s slow) (0.99, s slow)
C3
(0.91, fast)
(0.85, fast)
(0.75, slow)
Here, it holds that (C2, Dz) > & (C1,, for all i 1, 4, i.e., C1 is component-wise wise (strictly) dominated by C2. Classifiers C2 and C3 are not component-wise (strictly) dominated. Thus, the Pareto-front is given by par ④) = (C2, Cz).
Both approaches are extreme in certain sense: (I) reduces the multidimenssonall information structure of the problem to one single real-valued score. Any selection of classifier based on this score will heavily depend the choice of the weights in the score and, therefore, becomes dubious once
"This effect is usually more pronounced under the second motivation: Whereas in the first case the different metrics attempt to formalize the same latent construct, here different quality dimensions are actually to be covered. E.g.. improvement in accuracy may often be accompanied by a deterioration in computation time. 'There, one interprets the data sets as realizations of a random variable 1' : Q + D on some probability space (S,S, S, v). chooses weights Wi...., Mr E R.. and assigns each C' E C the value Em., w. (O. (C)

---

this choice is not perfectly justified. This seems even more severe for problems where some of the involved quality metrics might only allow for an ordinal interpretation, e.g., feature sparseness as proxy for interpretability [75]. risk levels in the EU Al act (50) or other regulatory frameworks robustness (see experiments Section 5.2 or runtime levels (Section [5T]. Opposed to this, (II) seems to be very conservative: By considering classifiers that are in the Pareto-front par($), one (potentially) completely ignores both information encoded in the cardinally interpretable dimensions and information about the distribution of the data sets. As trade-off between these two extremes, which utilizes the complete available information but avoids the choice of weights, it has recently been proposed to compare classifiers under multidimensional quality metries using generalized stochastic dominance (GSD) (45). The rough idea of this approach is to first embed the range of the multivariate performance measure in special type of relational structure, so-called preference system, which then allows for also formalizing the entire information originating from the cardinal dimensions of the quality metric. A classifier is then judged at least as good competitor (similar to classic stochastic dominance). if its expected utility is at least as high with respect to every utility function representing (both the ordinal and the cardinal parts of) the preference system (also see Definition (5.. Although GSD also induces only a preorder, the set of not strictly dominated classifiers will generally be considerably smaller than under the classical Pareto analysis. Furthermore, avoids potentially difficult to justify assumptions about the weighting of the different quality metrics. Therefore, working with the GSD-front, as introduced below, will prove to be very promising analysis option: it combines the advantages of the conservative Pareto analysis with those of the liberal comparison of weighted sums.
1.1 Our contribution
We introduce the concept of the GSD-front (see, after some preparations. Definition 6 and characterize it in Theorem 2 as more discriminative than the Pareto-front. In this sense, the GSD-front is an information-efficient way to handle the multiplicit/implici multidimensionality of quality criteria, powerfully exploiting their ordinal and quantitative components.
GSD-Front:
Since typically the available data sets are just sample of the corresponding universe, empirical counterparts of the major concepts are needed to do justice to the underlying statistical uncertainty. In particular, we give a sound inference framework: Firstly, we propose a set-valued estimator for the GSD-front and provide sufficient conditions for its consistency (see Theorem I and Remark 33. Secondly, we develop static and dynamic statistical permutation-test if a classifier is in the GSD-front and prove their level-a- validity and their consistency (see Theorem
Proper handling of statistical uncertainty; estimating and testing:
Additionally, we recognize the fact that the underlying assumption of identically and independently distributed (i.i.d) sampling is questionable in many benchmarking studies. Thus, in Section 4.2 we quantify how robust the test decisions are under such deviations.
Robustification:
Experiments with benchmark suites and implementation: We illustrate the concepts and corrob- orate their relevance with experiments run over two benchmark suites (PMLB and OpenML, see Section 5) based on an implementation that is freely available and easily adaptable to comparable problems We consider experiments with mixed-scaled (ordinal and cardinal) multidimensional quality metrics, also incorporating (potentially) ordinal criteria.
1.2 Related work
Specifically, benchmarking classifiers over multiple data sets is a much-studied problem in machine learning, as it enables practitioners to make informed choices about which methods to consider for given data set. Furthermore, also proposals for novel classifiers must often first demonstrate their potential
Benchmarks are the foundations of applied machine learning research 27 90 78 65
for improvement in benchmark studies. Examples include (58] 40 31 57 12]. recent years, in recognition of the fact that the benchmark suite under consideration is only a sample of data sets, especially focusing on statistically significant differences between classifiers has received great for Bayesian variants). An R implementation of some of these tests is described in [15]. whereas use-cases in the context of time series and
interest (see, e.g., [24] 35 34 19 45
or, e.g., 19 22
'Implementtaiions of all methods and scripts to reproduce the experiments tttps:///itthbb com/ hannahblo/Statisttie MullllMiMlltiiiirreeaeaa www.llicrrrtrra---accaaarrnnnn via-thheeeeeeee taaetthe-SSi Front
3

---

neural networks for regression are discussed in [4 [6]. The diversity and the associated problem
of selecting quality metries (e.g.. (5T)) is currently attracting great deal of interest (e.g.. (89) )
Consequently, finding ways for comparing classifiers in terms of multidimensional quality metrics is
intensively studied, ranging from multidimensional interpretability measures (e.g.. (59) over classical
(e.g.. (3T) to embeddings in the theory of data depth (e.g.. [13 71). While utilizing
variants of stochastic dominance in statistics is quite common (e.g., (56] 61 6 76 67), the same seems not to hold for machine learning. Exceptions include [23] in an optimization context, (47]
48), who investigate special types of stochastic orders, and [45]. utilizing already GSD-relations for
classifier comparisons without the GSD-front. Finally, relying imprecise probabilities (e.g..
85
86 3) to robustify statistical hypotheses follows the tradition of 66 42
41 2]. see also, e.g.. [5
25
6 48 For application to Bayesian networks, see, e.g, [55 54 and 88
69
70
52
30 16 17 among others, for robustified machine learning in this spirit.
2
Deiisicn- preliminaries
The relevant basic concepts in order theory collected in Appendix A.I Based on these we can make the following definition, originating from the decision -thooretic context discussed in [46).
Definition 2. Let A be non-empty set, Rí S Ax A a preorder on A, and R2 C Ri x Rı a preorder on R2. The triplet A = [A, R1 R2] is then called a preference system on A. The preference system A' [A', R1, R2y is called subsystem of Af A' C A, R2 Rg, and R2 C R2.
In our context, R2 formalizes the ordinal information, i.e., the information about the ranking of the objects in A, whereas R2 describes the cardinal information, i.e., the information about the intensity of certain rankings. To ensure that Ra and R2 are compatible, we use a consistency criterion relying on the idea of simultaneous representability of both relations. For this, for a preorder R, we denote by JR its indifference and by PR its strict part (see A.
Definition 3. The preference system A [4, R1, R2] consistent if there exists a representation u: A + R such that for all a, b, c, d E A we have:
(a, E Rí u(aa 2 u(b) with equality iff (a, b) E IR.
ii (), d)) E R₂ + u(a)= u(b)= 2 u(c) u(d) with equality iff ((a,b), d)) E IR. The set of all representations of A is denoted by UA
Finally, we need to recall the concept of generalized stochastic dominance (GSD) (see, e.g.. 48) ).
which is crucial for the concepts presented in this paper: For a probability space (12, x) and
consistent preference system A, we define by F(A.z) the set of all X E A such that u o X E
L S2,, T) for all u E UA. We then can define the DSD-preorder on F(A.T) as follows.
Definition 4. Let A = [A, R1, Ra] be consistent. For X,Y E F(Aa), say X (A, T. -dominatess Y if
(u e X)= 2 E, (u o Y) for all u E UA Denote the induced GSD-preorder on F(A.)) by R(Aa).
3 GSD for classifier comparison
We return to the initial problem: Assume we are given a finite set C of classifiers, arbitrary set D of data sets and n quality metrics $1,..., &n C x D > [0, 1]. combined to the multidimensional quality metric & (Ø1,..., Φx) C x D [0, 1l". As metrics, we assume that, for 0 < 2 S n, the metrics $1. Φ. are of cardinal scale (differences may be interpreted), while the remaining ones are purely ordinal (differences are meaningless apart from sign). We embed the range (C x D) of & in the following preference system:
also want allow ordinal quality
p [[0, 1]" Ri, R2). where
R = (x, リソ Vj < n and R2 ((o,,))
Ri is the usual component-wise r-relation. For Ris, one pair of consequences preferred to another if. in the ordinal dimensions, the exchange associated with the first pair is not a deterioration to the

---

exchange associated with the second pair and, in addition, there is component wise dominance of the differences of the cardinal dimensions. In order transfer the GSD-relation from Definition to the case of comparing classifiers under multidimensional performance metrics, we interpret the data sets in 'D as realizations of random variable T S - on some probability space (92, S, T). We then associate each classifier C' E C with the random variable &c == Φ(C, T()) on S2 and compare classifiers by comparing the associated random variables by means of GSD.
Definition 5. Denote by Pa the preference system obtained by restricting P to 4 x D). Further, let C be such that {&c C' E C} S F(( For C, C' E C, say that C dominates C', abbreviated with C x C², whenever (Dc, Pc)) E R(Po.))-
In the application situation, instead of the true GSD-order 2 among classifiers, we will often have to get along with its empirical analogue, i.e., the GSD-relation where sample of data sets is treated like the underlying population and the true probability measure is replaced by the corresponding empirical ones. More precisely, we assume that we have sampled i.id. copies Ti,..., Ts of T and then define the set Z5 := (C, T:) i A E C}2 of (random) observations under the different classifiers. We then use W to denote the (random) subsystem of P that arises when P is restricted to the (random) set Z, For C, C" EC C we define the random variable
d, C') : infuetww ELez, ((=)(fc((=)) #cc. ((x)))
where, for M S [0, 1]", we set (c(M) == f ((C, M) For concrete sample associated to wh E Ω we then say that C empirically GSD-ddminntes C', if (C, (w)( ) 2 0. Intuitively, d, can thus be used to check whether the classifier C' empirically dominates the classifier C" with respect to GSD in the samples at hand (i.e., in the benchmark suite under investigation).
Based on these concepts, we can now define the sets of (empirically) GSD-undominated classifiers Definition 6. Let C be such that {&c C' E CC} S F(Past) Let denote .ı.... T, i.idd. copies of T.
i) The GSD-front is the set gsd(C) = (C E #C EC C'
where > denotes the strict part of k.
ii Let E [0, The e-empirical GSD-front is the (random) subset of C defined by
d_(C", (CC" 2 - (C, <
egsd, (C) = C: 3C' est.
(C) is always non-empty. In contrast, egsd, may very well be empty if's > O. Note that choosing values of > 0 is intended to make egsd,, (C) less prone to sampling noise.
Remark 1.
Remark 2. Some words on the semantics of the GSD-front: From a decision-theon theoretic point of view, classifier C strictly GSD-dominates classifier C' iff C has least as high expected utility as C regarding any compatible utility representation of all the metrics considered, and stricly higher for at least one such utility. The GSD-front then simply collects all classifiers from C which are not strictly GSD-ddoinated by any competitor, ie., which potentially can be optimal in expectation.
Example 2. Consider again the situation of Example I and recall that par($) = (C2, Cs} leaves C2 and C3 incomparable. However, if considering only the distribution of the (multivariate)) performance of the classifiers (while assuming a uniform distribution over DD, C3 is clearly dominating O2 w.r.t. GSD: Matching dataset D, with dataset D5-; creates (strict) pointwise dominance of C3 over Cz (where the strict dominance is due to D2 and D2). Thus, gsd(C) = (Ca) & par(T) = (C2, Ca)-
The following two theorems show that the e-empirical GSD-front fulfills two very natural require- ments: First, under some regularity conditions, it is a consistent statistical estimator for the true GSD-front (Theorem [T.. This is important because in practical benchmarking we almost never have access to the GSD-front of the whole population, i.e., the benchmarking results on all possible datasets from specific problem class D. Second, is ensured that neither the e-empirical nor the true GSD-front can ever become larger than the respective Pareto- front, irrespective of the choice of e (Theorem 2. This important as it guarantees our analysis does never conflict with, but potentially more information--cfficiee than a Pareto-type analysis. Proofs are given in B. and B.2
5

---

Theorem 1. Denote by Lt the set of all sets {a u(a) 2 c}, where e E [0, 10 and E Up. Assume that 2 antisymmetric. If the VC-dimensior of Is is funite and if : - [0, 1] converges with rate at most (11 V/8,, then (egsd (C)) SEN is a consistent statistical estimator, i.e.,
. ofWEES S:: lim egsd,( ( gsd(c))) =1,
where set convergence is defined via the trivial metric.
Remark The assumption of a finite VC dimension only necessary to ensure that the e-empirical GSD front does not become too large. In particular, the following does hold without this assumption:
π z{fu E Q: lim egsd, (a)d)(()(C) (C) gsd(C))) =1.
80000
Thus, the e-empirical GSD-front almost surely converges to superset of the true GSD-front.
Theorem 2. Assume C with { : C C) S F(P.s). Let further denote T1,...; Tg i.i.d. copies of T and let 61 S 82 E [0, 1]. It then holds that i) gsd(C) par(()). Moreover, it holds that ii) egsd"" (C) C egsd (C) C par( Φ,, ,
4 Statistical testing
We saw the e-empirical GSD-front can be consistent statistical estimator and that both the empirical and the true GSD-front are compatible with the Pareto-front. We now address statistical testing,
4.1 A test for the GSD-front
From now on, make the (technical) assumption that the order 2 among the classifiers from C is additionally antisymmelric, transforming it from a preorder into a partial order Equipped with this assumption, want address the question how statistically lest if given classifier C' E C is an element of the true GSD-front gsd(C). To achieve this, we formulate the question of actual interest as the alternative hypothesis of the test, i... we obtain the hypothesis pair:
Ho C $ gsd(C) vs. H1 : C E gsd(C)
possible motivation for developing tests on the hypothesis pair (Ho. -Ho) Ho) is the following: One would like to compare the quality of a newly developed classifier C' for problem class D with the classifiers in C {C} that are considered state-of-the-as for this problem class, see application in Section 5.2 If suitable statistical test would allow the above null hypothesis to be rejected, then one could draw the conclusion (subject to statistical uncertainty) that the new classifier C' on the problem class D could potentially improve the state-of the-art. As first step, note that (under asymmetry) the null hypothesis Ho can be equivalently rewritten as Ho 3C' EC {C} This reformulation makes obvious that Ho is false if and only if for every C" E (C) the auxiliary
: C' ž C.
hypothesis HJ" : C' 2 C is false. Statistical tests for hypothesis pairs of the form (HG - were proposed (in the context of statistical inequality analysis) in [48]: The authors there showed how exact statistical tests under i.i.d. sampling can be constructed by using (non parametric) permutation test based a regularized version CC' C) of d. (C', C') as test statistic. The strength regularization of the test statistic is there controlled by a parameter increase reduces the number of representation functions over which the infimum in the test statistic is formed, while equally attenuating all quality metrics Due to space limitations, we omit to recall an exact description of the testing scheme in the main text and instead refer Appendix A.2
[0, 1], whose
The idea is then replace the global test for (Ho. -H) with == C 1 tests of hypotheses - and to reject the null hypothesis at significance level
tests reject their individual null hypotheses HC" at the same significance level a. Call this the static GSD-test. Clearly, this test tends to be conservative, as it ignores potential correlations of the test statistics for different pairs of classifiers. Moreover, a slightly modified test in the context of the GSD-front is directly
'The VC-dimension of set system S is the largest cardinality of set A with 2 = (AnS: S: : S E S}. 'This not very restrictive, only assumes to consider classifiers are not already equivalent w.r.t. GSD. In both applications in Section 5 the tests are based on the unregularized statistics (C", C),, the regularization performed in [88 aims reaching goal which is primarily relevant for paper (see A.2.5

---

derivable: If one is rather interested in identifying the maximal subset Smax of C for which C significantly lies in the GSD-front, i.c., in testing Hg C & gsd(S) vs. H C E gsd(S) for all S S C with C' simultaneously, the following alternative test a statistically valid level-o test: First, perform all individual tests for (HS -H6 with level g. Then identify Smax as the set of all classifiers from C for which the individual hypotheses are rejected. The (random) alternative hypothesis H C E gsd(S is then statistically valid in the sense of being false only with a probability bounded by a. Call this the dynamic GSD-test. We have the following theorem, demonstrating that the proposed tests are indeed reasonable statistical tests (see Bor for the proof).
Theorem 3. Let the assumptions of Theorem Srreeem@@hllt hold. Then, both the static and dynamic GSD-test are valid level-a tests. Additionally, both tests are consistent in the sense that under the corresponding alternative hypothesis, i.e., H1 : C E gsd(C) resp. H1 3S S C C E S 2 2,C' E gsd((S), the probability of rejecting the corresponding null hypothesis converges to 1 as -> oc.
4.2 Checking robustness under non-ii.d,-scenaroo
We argue that meaningful benchmark studies should abstain from treating the sample of data sets in the suite as a complete survey. That is, benchmark analyses should aim at statements about a well-defined population and regard the benchmark suite non-degenerate sample thereof. A major practical problem in this context is that often little is known about the inclusion criteria for data sets or test problems in the respective benchmark suite (see, e.g., the discussions in [8 53
37). For instance, the popular platform OpenML 82] allows users to upload benchmark results for machine learning models with varying hyperparameters. harming representativity, see Section 5.1 and Appendix [C.]] The absence of methods to randomly sample from the set of all problems or data sets is identified as unsolved issue in [57] Section 2). This calls the common i.i.d. sampling assumption into question, which our (and most other) tests are based upon, and raises the issue as to what extent statistically significant results depend on this assumption. We now address precisely this question.
In [48) it was shown how the binary tests on the hypothesis pairs (HI" -Hg"" (11"" discussed in Sec- tion 1.T can be checked for robustness against deviations from the underlying i. aad.-aasuuption The idea here is to deliberately perturb the empirical distributions of the performances for the different classifiers and to analyze the permutation test used under the most extreme yet compatible casst-case. The perturbation of the empirical distribution is carried out here using a --comtamminatoon model (see, e.g., [85] p. 147]). which is widely used in robust statistics. We now want to adapt a similar robustness check for the global hypothesis pair (Ho, -Ho) discussed here. For this, suppose we have sample Ti..... T, of data sets (i.e., the benchmark suite). We further assume that k S 8 of these variables (where it is not known which ones) are not sampled i.i.d. but come from an arbitrary distribution about which nothing else is known. We then know, for every fixed C E C, that its associated true empirical measure TC based on the true (uncontaminated) sample would have be contained in
Mc {(1 1 + + probability measure).
where #20" denotes the empirical measure based on the contaminated sample Tı, Ti,..., T,. Note that Mc is by definition a 7-contamination model with central distribution #gg" and contamination degree 7 = :.
& In this setting, (48) show that to ensure that their permutation tests used for hypothesis
pairs (H6" -HO",-H6" -H(") only advise rejection of the null hypothesis if this is justifiable for any empirical distribution compatible with the contaminated sample, i.e.. for every combination of measures (TI, R2) E MM x MMr. one has compare the most pessimistic value of the test statistic for the concrete sample at hand with the most optimistic value of the test in each of the resamples. Moreover, they show that the (approximate) observed p-values for a concrete contaminated sample Ti (wp)..... ................................ To (wo) E D associated with wo, E Ω of this robustified test can be expressed by a function in the number of contaminations k. given by
(k) 1 N EIEIN (17- (DW())> 28 }
1
where N denotes the number of resamples, IN is the corresponding set of resamples, and df is the test statistic evaluated for the resample associated to I. Due space limitations, we omit an exact description of the robustness check for the test on the hypothesis pairs (Hf' - as well as a derivation of the function f(o .c) in the main text and instead refer to Appendix A.3
7

---

Similar as shown in Section 4.1 it is straightforward to calculate an (approximate) observed p-value for the static GSD-test for (H. -Ho): We calculate the maximal observed p-value among all C' e C') "M{C',, i.c set (k( := max(ficr.o degree of contamination i can be carried out as follows: Calculate (k(kk) and reject Ho if (k I This indeed gives us a valid level-a-test for the desired global hypothesis Ho : C & gsd(C) under the additional freedom that up to & of the variables in the sample might be contaminated. Note, however, that also this test tends to be conservative as both performing the individual tests at level a as well as the adapted resampling scheme of the permutation test worst-case analyses. Finally, also the robustified dynamic GSD-test can be obtained straightforwardly: Under up to & contaminations. the (random) alternative hypothesis 11, Sua : C' E gsd(Sman is statistically valid with level or if all individual robustified tests reject H6" at level a. i.e., if (k) S :.
C' E CV(C)). The robustified static GSD-test for the
We end the section with short comment on computation: The test statistics for the permutation test and the robustified variant can be calculated using linear programming. We are guided here by the linear programs proposed in [48] Propositions 4 and 5j. There are two computational bottlenecks in the actual evaluation: (1) the creation and storage of the constraint matrices of the linear programs and (2) the repeated need to solve large linear programs. An efficient, well-commented implementation that can be quickly transferred to similar applications is made available on GitHub (see Footnote 4
5 Benchmarking experiments
We demonstrate our concepts on two well-established benchmark suites: OpenML 82
11 and
PMLB (64). While for PMLB we compare classifiers w.r.t. the latent quality metric robust accuracy (see the first motivation in Section D. for OpenML we use a multidimensional metric that includes accuracy and computation time as unidimensional metries (see the second motivation in Section [ The analysis of PMLB is kept short in the main text and detailed in Appendix [ Since the metrics in both applications are composed of one continuous and two (finitely) discrete metrics, we have (see (B.)::
Corollary 1. In both applications, the E-empirical GSD-front is a consistent estimator for the true GSD-front (provided is chosen as in Theorem Z
5.1
Experiments on OpenMI.
We select 80 binary classification datasets (according to criteria detailed in Appendix C.I from
OpenML. (82) to compare the performance of Support Vector Machine (SVM) with Random Forest (RF), Decision Tree (CART), Logistic Regression (LRR), Generalized Linear Model with Flastic net (GL) Extreme Gradient Boosting (xGBoost), and k-Nearest Neighbors (KNN) 8 Our multidimensional quality metric is composed of predictive accuracy, computation time on the test data, and computation time on the training data. Since the computation time depends strongly on the used computing environment (e.g. number of cores or free memory), we discretize the time- related metrics and treat them as ordinal. Accuracy is not affected by this and is therefore treated as cardinal. For details, see Appendix [.I] To gain a purely descriptive impression, we computed the empirical GSD relation. For this, we calculated deo (C.C") for C * C" E C := (SVM, RF, CART, LR, GLMNet, xGBoost, kNN) (see Hasse graph in Figure 2 in Appendix C.I We see that CART (strictly) empirically GSD-dominates xGBoost, SVM, IR, and GLMNet. All other classifiers are pairwise incomparable. Three classifiers are not strictly empirically GSDD-dominated by any other, namely RF, CART, and KNN. Thus, the 0-empirical GSD-front is formed by these. While at first glance this result might seem rather unexpected, a closer look on the performance evaluations provided by OpenML indeed confirms the dominance structure found, see Appendix C.I details.
To move to reliable inferential statements that take into account the statistical uncertainty, we exemplarily test (at level a 0.05) if SVM significantly lies in the GSD-front of some subset of C. As described in Section 4.1 we therefore perform six pairwise permutation tests for the hypothesis pairs (HB ; -H6" (where := SVM and C' e eC\ (SVM)) at level
case of the static GSD-test or at level in case of the dynamic GSD-test!" 17 That is, we test six auxiliary null hypotheses each stating that SVM is GSD-dominated by KNN, xGBoost, RF, CART, LR, and GLMNet, respectively.
$ Foor benchmarking deep learning classifiers or optimizers, we refer to future work discussed Section As explained in Footnote 7 we base the tests in Sections 5.1 and 5.2 on the unregularized (C C).
8

---

Decsin
Pepecosbe
Cd5
Contamenatd Sargres
Figure 1: Left: Densities of resampled test statistics for pairwise permutation tests of SVM vs. six other classifiers on 80 datasets from OpenML. Big (small) vertical lines depict observed (resampled) test statistics. Rejection regions for the static (dynamic) GSD-test are highlighted red (dark red). Right: Effect of Contamination: p-values for pairwise tests of SVM versus GL.MNet, KNN, RF and xGBooot. Red lines mark significance levels of 0.05 (dark red: SVM being in the GSD-front remains stable under contamination of up to 7 of 80 datasets.
0.05 Significance of
The distributions of the test statistics are visualized on the left of I (densities) and Figure (CDFs) in C.I They show that the pairwise tests of SVM versus kNN, xGBoost, RF, and GLMNet reject at level and, thus, that SVM significantly (at level a) lies in the GSD-front of the subset composed of SVVM and these four classifiers. In other words, we conclude that SVM is significantly (a 0.05) not outperformed by KNN, xGBoost, RF, and GLMNet regarding all compatible utility representation of accuracy, training and test runtime. Finally, as discussed in Section 4.2 we turn to the third aspect reliability (besides multiple criteria and statistical uncertainty): We analyze how robust this test decision is under contamination of the benchmark suite, i.c., deviations from i.i.d. The results are visualized on the right of Figure ! It can be seen that the tests at level 0.5 of SVM against GLMNet, kNN, RF and xGBoost cease to be significant from contamination 6 (approximately) 7, 8, 11, and 11 of 80 data sets, respectively. That is, the results on up to 7, 8, 11, and 11 datasets could be arbitrarily redistributed, while maintaining significance of rejection. Since the significance of the dynamic GSD-test's ' decision depends on all pairwise tests being significant at 6. we can conclude that SVM would still have been significantly in the GSD-front of (SVM, kNN, xGBoost, RF, GNNNet). even if 7 out of 80 data sets had been contaminated. Summing up, our proposed testing scheme not only allowed for meaningful statistical benchmarking of SVM versus competitors regarding accuracy, test time, and train time; it also enabled us to quantify as to what degree our conclusions remained stable under contamination of the benchmark suite.
of
level 0.5
The results highlight the advantages the GSD-front over existing approaches. Applying first-order stochastic dominance (a special case of GSD where R is the trivial preorder) on the same set-up, yields that no classifier significantly larger than (or incomparable to) any other classifier, based 5% significance level. This illustrates that the GSD-approach accounts for accuracy being a cardinal measure. In contrast, the Pareto-front here contains all considered classifiers. Thus, the Pareto front is much less informative than the GSD-front, which is also reflected in Theorem [ Unlike the Pareto-front, the GSD-front is based on the distribution of the multidimensional quality metric and not only on the pairwise comparisons, and can use this knowledge to define the front. Thus, the GSD front is balance between the conservative Pareto analysis and the liberal weighted sum comparison. Finally, we want to compare our method with
Method comparison:
an approach based on extending the test for single quality metrics proposed in [24] to the multiple metric setting. We therefore perform all possible single- metric tests as in (24) and define the marginal front as those classifiers that are not statistically significantly worse than another classifier on all metrics. However, this procedure can not used to define hypothesis test. Therefore, only comparison with the empirical GSD-front is meaningful. For OpenML, this marginal front consists of all classifiers and is less exploratory than the empirical GSD-front. More details on the results of these other approaches and how these compare to the GSD front can be found in Appendix C.I

---

5.2 Experiments on PMLB
We select 62 datasets from the Penn Machine Learning Benchmark (PMLB) suite [64] according to criteria explained in Appendix C.2 The following analysis shall exemplify how our proposed statistical tests can aid researchers in benchmarking newly developed classifiers against state-of of-the- art ones. To this end, we compare a recently proposed classifier based on compressed rule ensembles of trees (CRE) [62] w.r.t. robust accuracy against five well-established classifiers, namely CART, RF, SVM with radial kernel, kNN and GL.NNet. We operationalize the latent quality criterion of robust accuracy through i) classical accuracy (metric), ii) robustness of accuracy w.r.t. noisy features (ordinal), and iii) robustness of accuracy w.r.t. noisy classes (ordinal). Computation of i) is straightforward; in order to retrieve ii) and ii), we follow [92 93] by randomly perturbing a share (here: 20 %) of both classes and features and computing the accuracy subsequently, detailed in Appendix C.2 Since there exist competing definitions of robustness [43 10, 72] and due to the share's iss i arbitrary size, treat ii) and iii) ordinal and discretize the perturbated accuracy in the same way as for the runtimes in the openML experiments. Detailed results and visualization thereof C.2 In nutshell, and the dynamic GSD-test at significance level a = 0.05. In particular, we do not reject any of the pairwise auxiliary tests for hypothesis pairs (H6², -HO") with C == CRE and C" E (CRE)) for neither a nor 5. Our analysis hence concludes that we cannot rule out significance level a 0.05 that the newly proposed classifier CRE is dominated by the five state-of-the-r classifiers w.r.t. all compatible utility representation of the latent criterion robust accuracy.
can be found in Appendix
find evidence reject the null of both the static
5.3 Additional recommendations for the end-user
We end the section with a few brief general notes for and-users of our benchmark methodology. This should make it easy to decide whether a GSD-based analysis is appropriate in given use-case.
1. GSD-based studies do not primarily aim to identify the best algorithm for given benchmark suite. Often, the GSD front contains more than one element. They are rather intended for checking whether a newly proposed classifier for certain problem class can potentially improve on the state-of-the-ar classifiers, or whether it disqualifies itself from the outset.
2. GSD-based studies allow statements with inferential guarantees by providing appropriate sta- tistical tests: Assuming an i.i.d. benchmark suite, a judgment about an algorithm represents statement about an underlying population and not just this specific suite.
3. GSD-based studies enable the robustness of the results to be quantified under the deviation from the i...d. assumption: It can be checked which share of the benchmark suite may be contaminated without affecting the obtained inferential statements.
4. GSD- studies allow algorithms to be compared w.r.t. multiple metrics simultaneously. They enable the full exploitation of the information contained in differently scaled metrics.
Concluding remarks
We introduced the GSD-front for multicriteria comparisons of classifiers, gave conditions for its consistent estimability and proposed a statistical test for checking if classifier belongs to it. We illustrated our concepts using two well-established benchmark suites. The results came with threefold reliability: They included several quality metrics, representation of statistical uncertainty. and quantification of robustness under deviations from the assumptions.
Summary:
Limitations and future research:
Two specific limitations open promising avenues: 1.) Comparing other types of algorithms: We restricted ourselves to comparing classifiers. However, any situation which objects are to be compared on the basis of different (potentially differently scaled) metrics over random selection of instances can be analyzed using these ideas. For instance, applications of our framework to the multicriteria deep learning benchmark suite DAWNBench (2I) or the bi-criteria optimization benchmark suite DeepOBS (74) appear straighforward. 2.) Extension to regression-tye type analysis: Analyses based on the GSD-front do not account for meta propertics of the data sets. A straightforward extension to the case of additional covariates for the data sets is to stratify by these for the GSD comparrnii..... This would allow for a situatiionspeciife GSD-an anallssssss,, presumably yielding more informative results.
10

---

Acknowledgements
We thank the anonymous reviewers and the area chair for providing valuable feedback. HB sincerely thanks the Evangelisches Studienwerk Villigst V. for the funding and support of her doctoral studies. Support by the Federal Statistical Office of Germany within the co-operation project "Machine Learning in Official Statistics" (JR and TA), by the Bavarian Academy of Sciences (BAS) through the Bavarian Institute for Digital Transformation (bidt, JR) and by the LMU Mentoring Program (JR and HB) is gratefully acknowledged.
References
[1] J. Abellan, C. Mantas, J. Castellano, and S. Moral-Garcia. "Increasing diversity in random forest learning algorithm via imprecise probabilities". In: Expert Systems with Applications 97 (2018), pp. 228-243.
[2] T. Augustin. "Neyman-Pearsee testing under interval probability by globally least favorable pairs: Reviewing Huber-Strassen theory and extending it to general interval probability". In: Journal of Statistical Planning and Inference 105 (2002), pp. 149-173.
[3] T. Augustin, F. Coolen, G. de Cooman, and M. Troffaes, eds. Introduction to Imprecise Probabilities. Wiley, 2014.
[4] T. Augustin and G. Schollmeyer. "Comment: On focusing, soft and strong revision of Choquet capacities and their role in statistics". In: Statistical Science 36.2 (2021), pp. 205-209. [5] T. Augustin, G. Walter, and F. Coolen. "Statistical Inference". In: Introduction to Imprecise Probabilities. Ed. by T. Augustin, F. Coolen, G. de Cooman, and M. Troffaes. Wiley, 2014, pp. 135-189.
[6] G. Barrett and S. Donald. "Consistent tests for stochastic dominance", In: Econometrica 7 11.1 (2003), pp. 71-104.
[7] D. Bates and M. Maechler. Package 'Matrix' [Accessed: 13.05.2024). 2010. URL: http:
7/ //cran. 020:2/2000rrr.
[8] A. Benavoli, G. Corani, Demšar, and M. Zaffalon. "Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis". In: Journal of Machine Learning Research 18.77 (2017), pp. 1-36.
[9] Benavoli, Corani, and F. Mangili. "Should we really use post-hoe tests based on mean- ranks?" In: Journal of Machine Learning Research 17.1 (2016), pp. 152-161.
[10] D. Bertsimas, J. Dunn, C. Pawlowski, and Y. Zhuo. "Robust classification". In: INFORMS
Journal on Optimization 1.1 (2019), pp. 2-34.
[11] B. Bischl, G. Casalicchio, M. Feurer, P. Gijsbers, F. Hutter, M. Lang, R. Mantovani, J. van Rijn, and J. Vanschoren. "OpenML: A benchmarking layer on top of OpenML. to quickly create, download, and share systematic benchmarks". In: NeurIPS - Track on Datasets and Benchmarks (2021).
[12] B. Bischl, P. Kerschke, L. Kotthoff, M. Lindauer, Y. Malitsky, A. Fréchette, H. Hoos, F. Hutter, K. Leyton- Brownn, K. Tierney, and J. Vanschoren. "ASlib; A benchmark library for algorithm selection". In: Artificial Intelligence 237 (2016), pp. 41-58.
[13] H. Blocher. G. Schollmeyer, C. Jansen, and M. Nalenz. "Depth functions for partial orders with a descriptive analysis of machine learning algorithms". In: Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications. Vol. 215.
Proceedings of Machine Learning Research. PMLR, 2023, pp. 59-71.
[14] R. Cabanas, A. Antonucci, D. Huber, and M. Zaffalon. "CREDICI: A Java library for causal inference by credal networks". In: International Conference on Probabilistic Graphical Models. Ed. by M. Jaeger and T. Nielsen. Vol. 138. PMLR. 2020, pp. 597-600.
[15] B. Calvo and G. Santafé. "scmamp: Statistical comparison of multiple algorithms in multiple problems". In: The R Journal 8.1 (2016), pp. 248-256.
[16] M. Caprio, Y. Sale, E. Hüllermeier, and Lee. "A Novel Bayes' Theorem for Upper Probabili- ties". In: Epistemic Uncertainty in Artificial Intelligence - First International Workshop, Epi UAI 2023, Pittsburgh, PA, USA, August 4, 2023, Revised Selected Papers. Ed. by F. Cuzzolin and M. Sultana. Vol. 14523. Lecture Notes in Computer Science. Springer, 2024, pp. 1-12.

---

[17] M. Caprio, M. Sultana, E. Elia, and F. Cuzzolin. Credal Learning Theory. 2024. arXiv:
2402. 00957 URL: https:///axxaii org/ abs/2402 00957
[18] Y. Carranza and S. Destereke. "Imprecise Gaussian discriminant classification". In: Pattern Recognition 112 (2021), p. 107739.
[19] L. Chang. "Partial order relations for classification comparisons". In: Canadian Journal of Statistics 48.2 (2020), pp. 152-166.
[20] T. Chen, T. He, M. Benesty, V. Khotilovich, Y. Tang, H. Cho, K. Chen, R. Mitchell, I. Cano, T. Zhou, M. Li, J. Xie, M. Lin, Y. Geng, Li, and J. Yuan. Package 'xghoost' [Accessed: 13.05.2024). 2023. URL: https: // /ra.ran.... project org/ web httpb//ccagss////a xgboost/
xgboost. pdf
[21] C. Coleman, D. Narayanan, D. Kang. T. Zhao, J. Zhang, L. Nardi, P. Bailis, K. Olukotun, C. Ré, and M. Zaharia. "Dawnbench: An end-to-end deep learning benchmark and competition". In: Training 100.101 101 (2017), p. 102.
[22] G. Corani, A. Benavoli, J. Demšar, F. Mangili, and M. Zaffalon. "Statistical comparison of classifiers through Bayesian hierarchical modelling". In: Machine Learning 106. (2017),
pp. 1817-1837.
[23] H. Dai, Y. Xue, N. He, Y. Wang. N. Li, D. Schuummans, and B. Dai. "Learning to optimize for stochastic dominance constraints". In: International Conference on Artificial Intelligence and Statistics. Ed. by F. Ruiz, J. Dy, and J. van de Meent. Vol. 206. PMLR. 2023, pp. 8991-9009. [24] J. Demšar. "Statistical comparisons of classifiers over multiple data sets". In: Journal of Machine Learning Research 7 (2006), pp. 1-30.
[25] S. Destercke, I. Montes, and E. Miranda. "Processing distortion models: A comparative study".
In: International Journal of Approximate Reasoning 145 (2022), pp. 91-120.
[26] E. Dimitriadou, K. Hornik, F. Leisch, D. Meyer, and A. Weingessel. Package 'e1071' [Ac- cessed: 13.05.2024]. 2010. URL: https://cran rr http//eeran.poojjjt............. wtwpjjecttrrggwbbbeee wttp
e1071. pdf
[27] D. Donoho. "Data Science at the Singularity" In: 2023 IMS International Conference on Statistics and Data Science (ICSDS). Ed. by R. Liu and A. Qu. 2023, p. 3.
[28] R. Dudley. "Central limit theorems for empirical measures". In: The Annals of Probability 6.6 (1978), pp. 899-929.
[29] R. Durrett. Probability: Theory And Examples. Vol. 49. Cambridge University Press, 2019.
[30] S. Dutta, M. Caprio, V. Lin, M. Cleaveland, K.J. Jang. I. Ruchkin, O. Sokolsky, and I. Lee. Distributionally Robust Statistical Verification with Imprecise Neural Networks. 2023. arXiv:
2308. 14815 [C] URL: (ttp:::ttpss/. //tpvv.rrvvvv.//aa org/ abs abs/2308. 2308. 14815)
[31] M. Eugster, T. Hothorn, and F, Leisch. "Domain-basse -based benchmark experiments: Exploratory and inferential analysis". In: Austrian Journal of Statistics 41.1 (2012), pp. 5-26.
[32] J. Friedman, T. Hastie, R. Tibshirani, Narasimhan, K. Tay, N. Simon, and J. Qian. Package gllmnet. [Accessed: 13.05.2024). 2021. URL: https : // cran r project org/ web pdd]
[33] M. Friedman. "The use of ranks to avoid the assumption of normality implicit in the analysis of variance". In: Journal of the American Statistical Association 32.200 (1937), pp. 675-701.
[34] S. García, Fernández, Luengo, and Herrera. "Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power". In: Information Sciences 180.10 (2010), pp. 2044-2064.
[35] S. Gareía and F. Herrera. "n extension on "Statistical comparisons of classifiers over multiple cata sets" for all pairwise comparisons". In: Journal of Machine Learning Research 9 (2008),
pp. 2677-2694
[36] M. Graczyk, т. Lasota, Z. Telec, and B. Trawinski. "Nonparametric statistical analysis of machine learning algorithms for regression problems", In: International Conference on Knowledge-Bau Based and Intelligent Information and Engineering Systems. Ed. by R. Setchi, I. Jordanov, R. Howlett, and L. Jain. Springer. 2010, pp. 111-120.
[37] N. Hansen, A. Auger, D. Brockhoff, and Tušar. "Anytime performance assessment in blackbox optimization benchmarking".. In: IEEE Transactions Evolutionary Computation 26.6 (2022), pp. 1293-1305.
12

---

[38] K. Hechenbichler and K. Schliep. Weighted k-Nearest-Neighhor Techniques and Ordinal Classification. Technical Report, LMU. 2004. URRL: http: htt///nnnn-aaa/ reeoovvvng de/ /urn/
resolver, resolver.pl'urncee pl bvb: 19 19bb--17766
[39] K. Hornik, C. Buchta, T. Hothorn, A. Karatzoglou, D. Meyer, and A. Zeileis. Package 'rweka'. .
[Accessed: 033.552023]) 2007. URL: pattp::/cran.r-paaa
Wleka/index. html
[40] T. Hothorn, F. Leisch, A. Zeileis, and K. Hornik. "The design and analysis of benchmark experiments". In: Journal of Computational and Graphical Statistics 14.3 (2005), pp. 675-699.
[41]
P. Huber. Robust Statistics. New York: Wiley, 1981.
[42] P. Huber. "The use of Choquet capacities in statistics", In: Proceedings of the 39th Session of
the International Statistical Institute 45 (1973), pp. 181-191.
[43] S. Ishii and D. Ljunggren. "A comparative analysis of robustness to noise in machine learning classifiers" PhD thesis. KTH Royal Institute of Technology, 2021.
[44] H. Ismail Fawaz, G. Forestier, J. Weher, L.. Idoumghar, and P.-A. Muller. "Deep learning for time series classification: a review". In: Data Mining and Knowledge Discovery 33.4 (2019), pp. 917-963.
[45] C. Jansen, M. Nalenz, G. Schollmeyer, and T. Augustin. "Statistical comparisons of classifiers by generalized stochastic dominance". In: Journal of Machine Learning Research 24 (2023),
pp. 1-37.
[46] C. Jansen, G. Schollmeyer, and T. Augustin. "Concepts for decision making under severe uncertainty with partial ordinal and partial cardinal preferences". In: International Journal of Approximate Reasoning 98 (2018), pp. 112-131.
[47] C. Jansen, G. Schollmeyer, and T. Augustin. "Multi-target decision making under conditions of severe uncertainty". In: Modeling Decisions for Artificial Intelligence. Ed. by V. Torra and Y. Narukawa. Springer, 2023, pp. 45-57.
[48] C. Jansen, G. Schollmeyer, H. Blocher, J. Rodemann, and T. Augustin. "Robust statistical comparison of random variables with locally varying scale of measurement". In: Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence. Ed. by R. Evans and
I. Shpitser. Vol. 216. Proceedings of Machine Learning Research. PMLR, 2023, pp. 941-952.
[49] M. Kuhn, Packuge 'caret'. Accessed:: 13.05. 2023]. 2015. URL: httph:tt/////ttt r- proojecct
org/ http://wwwpaaee///////// caret/ index.ndxxi.. html
[50] J. Laux, S. Wachter, and B. Mittelstadt. "TTrustwothyy artificial intelligence and the European Union Al act: On the conflation of trustworthiness and acceptability of risk". In: Regulation & Governance 18.1 (2024), pp. 3-32.
[51] N. Lavesson and Davidsson. "Evaluating learning algorithms and classifiers", In: Interna- tional Journal of Intelligent Information and Database Systems 1 (2007), pp. 37-52.
[52] J. Lienen and E. Hüllermeier. "Credal Self- self-Suuervised Learning". In: Advances in Neural Infor- mation Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 14, 202 virtual. Ed. by MA. Ranzato, A. Beygelzimer, Y.N. Dauphin, P. Liang, and J. W. Vaughan. 2021, pp. 14370-14382.
[53] D. Mattos, L. Ruud, J. Bosch, and H. Holmström Olsson. On the assessment of benchmark suites for algorithm comparison. 2021. arXiv: 2104. 07381 [cs NE]]
[54] D. Maua and F. Cozman. "Thirty years of credal networks: Specification, algorithms and complexity". In: International Journal of Approximate Reasoning 126 (2020), pp. 133-157. [55] D. Maua and C. de Campos. "Editorial to: Special issue on robustness in probabilistic graphical models". In: International Journal of Approximate Reasoning 137 (2021). p. 113.
[56] D. McFadden. "Testing for stochastic dominance", In: Studies in the Economics of Uncertainty. Ed. by T. Fomby and T. Seo. Springer, 1989, pp. 1 113-134.
[57] O. Mersmann, M. Preuss, H. Trautmann, B. Bischl, and C. Weihs. "Analyzing the BBOB results by means of benchmarking concepts". In: Evolutionary Computation 23 (2015), pp. 161-185.
[58] D. Meyer, F. Leisch, and K. Hornik. "The support vector machine under test". In: Neurocom- puting 55.1 (2003), pp. 169-186.
[59] C. Molnar, G. Casalicchio, and B. Bischl. "Quantifying model complexity via functional decomposition for better post-hoe interpretability". In: Machine Learning and Knowledge Discovery in Databases. Ed. by P. Cellier and K. Driessens. Springer International Publishing, 2020. pp. 193-204.
13

---

[60] I. Montes, E. Miranda, and S. Destercke. "Unifying neighbourhood and distortion models: Part II new models and synthesis". In: International Journal of General Systems 49 (2020).
pp. 636-674.
[61] K. Mosler. "Testing whether two distributions are stochastically ordered not". In: Grundla- gen der Statistik und ihre Anwendungen: Festschrift für Kurt Weichselberger. Ed. by H. Rinne, B. Rüger, and H. Streeker. Physica-Verlag, 1995, pp. 149-155.
[62] M. Nalenz and 'T. Augustin. "Compressed rule ensemble learning". In: Proceedings of The 25th International Conference on Artificial Intelligence and Statistics. Ed. by G. Camps- Valls, F. Ruiz, and Valera. Vol. 151, Proceedings of Machine Learning Research. PMLR, 2022,
pp. 9998-10014.
[63]
P. Nemenyi. "Distribution free Multiple Comparisons". PhD thesis. Princeton University, 1963.
[64] R. Olson, W. La Cava, P. Orzechowski, R. Urbanowicz, and Moore. "PMLB: a large benchmark suite for machine learning evaluation and comparison". In: BioData Mining 10 (2017), p. 36.
[65] S. Ott, A. Barbosa-Silva, K. Blagec, J. Brauner, and M. Samwald. "Mapping global dynamics of benchmark creation and saturation in artificial intelligence". In: Nature Communications 13.1 (2022), P. 6793.
[66] Huber P. and V. Strassen. "Minimax tests and the Jeyman-Pearson lemma for capacities". In: The Annals of Statistics 1 (1973), pp. 251-263.
[67] T. Range and L. Østerdal. "First-order dominance: stronger characterization and bivariate checking algorithm". In: Mathematical Programming 173 (2019), pp. 193-219.
[68] B. Ripley and W. Venables. Package 'nner'. [Accessed: 13.05.2024). 2016. URL: https wwnl///nt/t/////eeoeeeoeenn pdf [69] J. Rodemann and T. Augustin. " "Accounting for Gaussian process imprecision in Bayesian optimization". In: International Symposium on Integrated Uncertainty in Knowledge Modelling and Decision Making (IUKM). Springer. 2022, pp. 92-104.
// /staff.. fmi uvt. wot.oooo/daiia daniela. zaharie/ e ddw0mmm/0/mmmmam
[70] J. Rodemann, Jansen, G. Scholmmeyr, and T. Augustin. "In all likelihoods: Robust selec- tion of labule-labeled data". In: Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications. Ed. by E. Miranda, I. Montes, E. Quaeghe- beur, and B. Vantaggi. Vol. 215. Proceedings of Machine Learning Research. PMLR, 2023, pp. 412-425.
[71] Julian Rodemann and Hannah Blocher. "Partial Rankings of Optimizers". In: International Conference on Learning Representations (ICLR), Tiny Papers Track. 2024.
[72] J. Sáez, J. Luengo, and F. Herrera. "Evaluating the classifier behavior with noisy data consider- ing performance and robustness: The equalized loss of accuracy measure". In: Neurocomputing 176 (2016), pp. 26-35.
[73] L. Schmitt. "Mapping global Al governance: nascent regime in a fragmented landscape". In: Al and Ethics 2.2 (2022), pp. 303-314.
[74] F. Schneider, L. Balles, and P. Hennig. "DeepOBS: A deep learning optimizer benchmark suite". In: International Conference Learning Representations. 2018.
[75] L. Schneider, B. Bischl, and J. Thomas. "Multi-ojective optimization of performance and interpretability of tabular supervised machine learning Mmodels". In: Proceedings of the Genetic and Evolutionary Computation Conference. 2023, pp. 538-547.
[76] G. Schollmeyer, C. Jansen, and T. Augustin. Detecting stochastic dominance for valutd random variables as an example of linear programming on closure systems. [Accessed: (3.05.2024) 2017. uRl: /tttps://epub... ub. uni - mwenchen.ncennccmm /p/.d///////.. ///444//1///TR11000 2df [77] M. Shaked and G. Shanthikumar. Stochastic orders. Springer, 2007.
[78] A. Shirali, R. Abebe, and M. Hardt. "A theory of dynamic benchmarks". In: The Eleventh International Conference on Learning Representationss. 2023.
[79] T. Therneau, B. Atkinson, and B. Ripley. Puckage 'rpart'. [Accessed: 15.02.2023). 2015. URL:
/ttp://cran ma. ic. ac. uk/ wwlk//ww/pppae httpb//aaaaaass/spaaaaaeeeeeee aappart/rpaat /rraar..attttt
[80]
L. Utkin. "An imprecise deep forest for classification". In: Expert Systems with Applications
141 (2020), p. 119978.
[81] L. Utkin and A. Konstantinov. "Attention-based random forest and contamination model". In:
Neural Networks 154 (2022), pp. 346-359.
14

---

[82] J. Van Rijn, B. Bischl, L. Torgo, B. Gao, V. Umaashankar, S. Fischer, P. Winter, B. Wiswedel, M. Berthold. and J. Vanschoren. "OpenML: A collaborative science platform". In: Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part III 13. Springer. 2013, Pp. 645-649.
[83] H. Vandierendonck and K. De Bosschere. "Experiments with subsetting benchmark suites". In: IEEE International Workshop on Workload Characterization, 2004. WWC-7. 2004. 2004, pp. 55-62.
[84]
V. Vapnik. The Nature Of Statistical Learning Theory. Springer, 1999.
[85]
P. Walley. Statistical Reasoning with Imprecise Probabilities. London: Chapman and Hall,
1991.
[86] K. Weichselberger. Elementare Grundbegriffe einer allgemeineren Wahrscheinlichkeit- rechnung I: Intervalwwhrrschiniickkee als unfassendes Konzept [Elementary Foundations of More General Calculus of Probability I: Interval Probability as Comprehensive Concept]. Physica, Heidelberg, 2001.
[87] H. Wickham, R. François, L.. Henry, and K. Müller. Package 'dplyr'. [Accessed: 13.05.2024).
2019. URL: luttps://cran. rrrojnnr-prrject.. http://eb//aacaags////pp//////ee
html
[88] M. Wright and A. Ziegler. "ranger: A fast implementation of random gorests for high dimen- sional data in C++ and R" In: Journal of Statistical Software 77.1 (2017), pp. 1-17.
[89] B. Yu and K. Kumbier. "Veridical data science". In: Proceedings of the National Academy of Science 117.8 (2020), pp. 3920-3929.
[90] G. Zhang and M. Hardt. Inherent Trade-Ofi between Diversity and Stability in Multi-Tuu Task Benchmark. 2024. arXiv: 2405. 01719 [CS.LG]
[91] J. Zhang, M. Harman, L. Ma, and Y. Liu. "Machine learning testing: Survey, landscapes and horizons". In: IEEE Transactions Software Engineering 48.1 (2020), pp. 1-36.
[92] Zhu and Wu. "Class noise vs. attribute noise: quantitative study". In: Artificial Intelligence Review 22 (2004), pp. 177-210.
[93] X. Zhu, X. Wu, and Y. Yang. "Error detection and imesct-sensitive instance ranking in noisy datasets". In: Proceedings of the National Conference on Artificial Intelligence. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999. 2004, pp. 378-384.
15

---

A Mathematical background
A.1 Basic definitions from order theory
A binary relation A on a set M is subset of the Cartesian product of M with itself, i.c., R M x M. transitive, if (a,b), (b. E
(a, R. antisymmetric. if (a, 6), (6, a) E R + a = b, and complete, if (a, 6) R or (b, E R (or both) for arbitrary elements a, b, C E M. preference relation is binary relation that is complete and transitive; preorder is binary relation that is reflexive and transitive; a linear order is a preference relation R is a preorder, we denote by PR S M x M its strict purt and by IR C M x M its indifference pari, defined by (a, b) E PR + (a, e RA (b, a) & R, and (a, b) E In + (a.b) RA (b, a) R.
R is called reflexive, if (a,0)
that antisymmetric; a partial order is a preorder that is antisymmetric.
A.2 Detailed description of the permutation test from Section 4.1
In this section we describe in detail the statistical test for the hypothesis pair (H6", -H16") discussed in S4. and first introduced in (48). Moreover, we give further details on our proposed extension of this test to the global hypothesis pair (Ho, -Ho) in both the static and the dynamic variant.
A.2.1 Preliminaries
Before we can describe the test from Section 4.] in detail, we first need to recall two more definitions.
Definition 7. Let A = [A, R1, Ry] be a preference system. We call A bounded, if there exist a,,a" a² E A such that (a" a) R1, and (a, a.) Rı for all A. and (a',ax) PR.-
Definition 8. Let A = [A, R1, R2] be a consistent and bounded preference system with a, a" as before. Define
NA = {u E UA u(a.) = A u(a") = =].
For d E [0, 1), denote by N the set of all u E -N with
u(a) u(b) 2 8 A u(e) u(d) u(e) + +u() 2 8
for all (a. E PR. for all ((c, d), (e, f)) E PR.
We start by describing an adapted version of the permutation test for the hypothesis pairs (HG", -H(") proposed in [48]. For conerete realization of the i.id.-sample of data sets D₁ Ti T((w))..... D2 == T. (wo) ED with s E N associated with wo E Ω, we define the set (C, 2 = (#(C, D.) is $) U ( D.) i (0,1)
where 0 is the vector containing zeros and 1 is the vector containing ones. Denote by Pwu, the restriction of P to (C, C3) Jwy.
then easy to verify that Pas, is a consistent and bounded preference system with o. == 0 and a* == 1. For testing the hypothesis pair (H6", - ) defined and discussed in Section of the main text, then the following regularized test statistic for the specific sample induced by wp:
dC (C))) = inf
Σ
u(z) (R (1z)) #C ({=)))
aEAths Puo sE(C,O")
with 8 E [0, 1] and μs == 8- supls: W * 0}. and ne0 resp. # are the emiprical probability measures of the performances of C resp. C" for the specific sample induced by wi-
A.2.2 Testing scheme for -H
We denote our samples follows:
x = (#.....,, xz) = ()CC
y (V1..... Us) (C( Di) ......... (CC,, Dz))
The concrete testing scheme for the permutation test for hypothesis pair (H -HH' then looks as follows:
16

---

Step 1: Take the pooled data sample: w = (W1..... W2e) = (01.... 91, ,............................... Ms)
Step 2: Take all r = (2)) index sets [ {1,..., 28) of size 8. Evaluate dy. (C", C) for (wi)ier and (ws)ie (mm..am))) instead of x and y. Denote the evaluations by dy.
Step 3: Sort all de in increasing order to get d ................................ (r)
Step 4: Reject Hy if dC, (wo) is strictly smaller than df with & == La r] and a the significance level.
Note that, for large (22), we can approximate the above resampling scheme by computing dř only for a large number N of randomly drawn 1. Moreover, note that only the i.i.d. assumption is needed for the above test to be valid.
A.2.3 Static GSD-test
As argued in the Section 4.1 of the main part of the paper, if we want to obtain a valid statistical test at the significance level a E [0, 1] for hypothesis pair (Ho, -H,), we can simply perform all pairwise tests of hypothesis pairs (HTg -HH² at this same significance level a. We can then reject the hypothesis Hu at level a if we can reject each hypothesis H8" at level a or, in other words,
( N
mine
EiEYTyy "(s10),0000)at)) : : :'' sVeecto) 21-a.
We call this the static GSD-test.
To see that this procedure indeed gives valid level-a test for the global hypothesis pair (Ho, -Ho). observe that assuming Ho to be true the probability of Ho being rejected equals the probability of all hypothesis H." being rejected simultaneously. The latter probability still assuming Hg to be true is obviously bounded from above by the probability that one specific hypothesis Ho" is rejected, which itself is bounded from above by the significance level a by construction.
A.2.4 Dynamic GSD-test
As discussed in the main text and reprinted here again for convenience of the reader, slightly modified test in the context of the GSD-front is directly derivable: If one is rather interested in identifying the maximal subset Smax of C for which C significantly lies in the GSD-from i.e., in testing the hypothesis pairs (HS. -H ) for all S C C simultaneously, the following alternative test would be a statistically valid level-a test: First, perform all individual tests for - with level g. Then identify Smax as the set of all classifiers from C for which the individual hypotheses are rejected. The (random) alternative hypothesis ATSuum C E gsd(Smat is then statistically valid in the sense of being false only with a probability bounded by a. We call this the dynamic GSD-test.
To see that this procedure indeed gives a valid level-a test for the (random) hypothesis pair -Hi, observe that - under the null hypothesis - the probability of C lying in the GSD- front of some random subset S of C is bounded from above by the sum of probabilities of lying in the GSD-front of {C,S), where summation is over all S S. As each of these probabilities is bounded from above by by construction, the corresponding sum is bounded from above by [S] g. Finally, as |S] < c, this gives the desired upper bound of a.
A.2.5 Computation and regularization
Note that the test statistic dg (C", (wwo) can be computed by solving a linear optimization problem (see 48 Proposition 4]) and, hence, the test just described is computationally tractable.
Moreover, note that in both applications in Section 5 the tests are based on the unregularized statisties (C" CC) as the regularization performed in [48] aims at reaching a goal which is not primarily relevant for the present paper: The authors there are primarily interested in significantly detecting GSD of one variable over the other. Consequently, their regularization aims at making the test more sensitive for exactly this purpose. In contrast, in our study we are primarily interested in significantly detecting incomparabilities between variables, making the regularization by far less natural.
17

---

A.3 Detailed description of the robustness check from Section 4.2
In this section we describe in detail the robustification of the statistical test for the hypothesis pair (H& -H6 discussed in Section 4.2 and first introduced in [48). Moreover, we give further details on our proposed extension of this robustified statistical test to the global hypothesis pair (Ho, -Ho)
in both the static and the dynamic variant.
A.3.1 Preliminiaries
If we assume, as done in Section 4.2 that up to k S. s of the observations in our sample Ti.... T. might be contaminated and, accordingly, follow any arbitrary distribution, then we have to base the permutation test for hypothesis pair (H) -H6 on a worst-case analysis of between the measures contained in .Mc and Mc. defined instead of the true empirical measures of the two samples induced by the classifiers C' and C". If again D1 := Ti (wo),... ..... D, T, (wp) D is concrete (now potentially contaminated) sample associated with wp Ω and we again define x and as in Section A.2 then the observed contamination models of C and C" look as follows:
Mc(wo) {{(1
) ) & probability measure]
aoo + E probability measure). Mcr(wo) = { $ C
A.3.2
Testing scheme for robustified test on -HO'
If we set
dC (wo) ==
sup (M
inf uENS Pwg =E(C.C))w
Σ
u(z) (TT(( me((=)))
inf (C,,))(w)) mi (e0))
inf HENES Pwg =E(C,C")=g
(TI(2(( x2((2)))
then the concrete testing scheme for the permutation test for hypothesis pair (116 -11"") under at most k contaminated sample members looks as follows:
Step Take the pooled data sample: w : (W1...., W2s) == (#1...... 2s .......... ys)
Step 2: Take all (2') index sets I C {1, (1,...,2s) 28) of size Evaluate d (C", C) for (wi)ier and (1....t) instead of x and y. Denote the evaluations by dy.
Step 3: Sort all d in increasing order to get de (1):..... d d'(r)
Step 4: Reject H&" if (C", (CC,)))((w)) is strictly smaller than Leo with == [a rr] and a the significance level.
The adapted testing scheme just described gives a valid (yet conservative) level-a-test for the hypothesis pair -H" under at most & contaminated sample members.
Moreover, it directly follows from the discussions in Part C of the supplementary material the (approximate) observed p-value of this test is given by
48 that
f(o)o)(k) = N
Σ 1 (1)-)(())))))) ) IEIN
where again N denotes the number of resamples, IN the corresponding set of resamples, and dē is the test statistic evaluated for the resample associated to I.
A.3.3 Robustified static GSD-test
As already argued in the main text, it is now easy calculate an (approximate) observed p-value for our global hypothesis pair (Hu, -Ho): We simply calculate the maximal observed p- p-value among all C" E C. (C). i.e. set
Fo(k) (ix C' ec (C]).
18

---

The robustified test for the degree of contamination & can be carried out as follows: Calculate (c and reject Ho if Fc S a, i.e., if the maximal (approximate) p-value of the pairwise tests still lower or equal than the significance level.
The argument that the testing procedure just described indeed produces valid level-a test of the global hypothesis pair (Ho. ~Ho) under up to k contaminated data sets in the sample, can be carried completely analogous done in Appendix A.2.3
A.3.4 Robustified dynamic GSD-test
Finally, as discussed in the main text and reprinted here again for convenience of the reader, also the robustified dynamic GSD-test can be obtained in straightforward manner: Under up contaminated data sets in the sample, the (random) alternative hypothesis H C gsd(Smax) from before is statistically valid with level a if all individual robustified tests reject H6" at level e. i.., if (k(k) S -.
The argument that the testing procedure just described indeed produces valid level-a test for the (random) hypothesis pair (H1'-. -Hg under up to k contaminated data sets in the sample, can be carried out completely analogous as done in Appendix A.2.4
A.3.5
Computation and regularization
Note that also the robustified test statistic (C', (w)) can be computed by solving a linear optimization problem (see [88] Proposition 6]) and, hence, the test just described is computationally tractable.
Again, note that the tests in Section 5 Jree based the unregularized test statistics with ō 0. The reason for this is the same as discussed at the end of Appendix A.2
B Proofs
B.1 Proof of Theorem 1
Proof. First, note that for C, C'C' E C, we have that C x C' and only if
D(C,C") inf (Ex 2c) Ex (u 2c 2 0.
"Elips
Thus, the GSD-from equivalently be rewritten as
gsd(C) CEC: AC e s.t.
DCC,,)) 2 0 D(C,C") 0
Now, let e : N [0,1] + 1/ We show that:
Ce E gC) = lim egsd
(C) (C) Ta.. and
(3)
$-220
C gC) + C & lim egsd (*) (C) T-a.s.
(4)
+-000
Note that the proof immediately translates to the more general case of E(s) 0 (s) stated in Theorem Denote with É the expectation w.r.t. the empirical measure associated with the i.i.d. 10
sample (T .....,,7 Ts). For Implication (3). assume that C' E gsd(C). Then for every other classifier C' there exists utility function u E ltp, with E, (u Φc) E=(u &cy) (Otherwise would have D C) 2 and D"C,C") < 0. where the second statement due to antisymmetry). For these corresponding utility functions, because of the strong law of large numbers, we would get (C", C) B(u (c+) É(u Φc) a.S, < Since C consists only of finitely many classifiers. egsd:(*)( (C) will almost surely not contain asymptotically. Note that for Implication (3) to hold,
10 Note that assuming only an exchangeable sample would also suffice. Note further that we have to assume the measurability of the involved infimum type statistics. For more details this issue, see, e.g.. 228
19

---

it is only necessary that E(s) converges to zero as goes to infinity. The order of convergency as (1/ (S) is only needed for Implication (4).
For Implication (4) assume that C' & gsd(C). Then there exists a classifier C' with D(C", C) 2 0
and D(C, C*) 0. An analog argumentation like above shows that dg (C, C*) converges almost surely to value smaller than zero. It remains to analyze D(C", C). For this, we have to show that dC' C) + s(s)
B. O. We utilize uniform convergence: For arbitrary & > 0, 84 192
Theorem 5. 5.1] gives us
e e-2s
P sup nEldrg
B(u 2c) B(u o 2c) > 8
exp ( (-E2s}
where &. = { 1/s and h is the VC dimension of IT. inequality then gives
The same holds for Do.. The triangle
p
sup DEllPs
B(u @c) B(u ( > >25
8
c-
exp (-(2s].
For (s) = 1/ (1/8) and large enough, we have Ex(8) = =(8) - 1/8 2 E {8) and therefore
e 28
h
P sup WEllrs
(c) - Biuo > 20(8)
s 8
exp (+)28)
h
1-28
h
exp { (-c(s)88/4}
h
This implies
<8 / e 28
P sup uelte
B(uo Pc) -B(u =(8)/2 (()) >
exp {-{(s)2//6))
(5)
8 E 28
exp V3/64}
(6)
If the VC dimension h is finite, the term 8 ($22)* is polynomially growing in v (or s). whereas
the term exp (-V3/64) is exponentially decreasing in v (or s). Therefore, the right hand side of Inequality (5) converges to zero, which shows that
[(uo -((o))))BB(o)))) s(s)
sup nelirs
converges in probability to a value e S O or equivalently, that d, dC' C) =(s) converges to a value e 1 Since the right hand side of Inequality (5) converges exponentially s, the Borel-Cantelli theorem (cf., e.g., (29] p.67fff) gives also strong convergency, which completes the proof. Note that it not necessary to specify s()) concretely as 1/ f8. It would be sufficient to define e(s) as of the order of ((1/ (8).
B.2 Proof of Theorem
Proof. i) Assume that C & p)r( Then, by definition of par($), there exists C² such that for all D E D it holds that (CC', D) > ((C, D). This implies that for all D E D holds that Up.. Since u then, by definition, is strictly isotone
(#(C", D), D)) with respect to PRy. this allows to conclude that the function u u(D(C", -)) u(((C,.)) is strictly positive, i.e. we have u(O(C", D)) u(D(C, D)) 0 for arbitrary D E D.
D)) E PRŞ- Now, choose
We compute:
I u(D((', (())))() -
Ex (u o 2cr) - Ex (u 4c)
[ u(D(CC - di dw) >0
20 for all WEf, since T(w)E
20

---

This gives E, (x) Ex (u bc). by definition of the GSD-front, we have C' & gsd(C).
As was chosen arbitrarily, this implies that C" > C. Hence,
ii) First, note that both postulates are statements involving random sets (i.e., sets dependent on the realizations of the variables Ti...., T.)) Thus, we have prove both statements for arbitrary realizations of these variables. So let D1 := Ti (wo)..... D. ;= Ta (wo) E D be an arbitrary realisation. For this concrete realization of the sample, the first statement is immediate, since if there is no C" such that d, (C", (wo) 2 -22 there is also no C" such that d,(C", (w))) 2 =1 (as the latter is harder to satisfy due to 51 S E2).
Again for the chosen concrete realization of the sample, the second postulate is an immediate consequence of statement i) from above. As in both situations the realization of the variables was chosen arbitrarily, this implies the statement.
B.3 Proof of Theorem 3
To see that the static test is valid level-a test for the global hypothesis pair (Ho. -Ho), observe that - assuming Ho to be true - the probability of Ho being rejected equals the probability of all hypothesis H6" being rejected simultaneously. The latter probability - still assuming Ho to be true - is obviously bounded from above by the probability that one specific hypothesis H6" is rejected, which itself is bounded from above by the significance level by construction.
Furthermore, the reason for the consistency of the static test is the following: First, note that under the assumption of Theorem [ (because of the finite VC dimension), we have that d,(C" C*) converges to D(C", C)) in probability for every abrbitrary classifier C" + C. Therefore, for fixed C" and under the null hypothesis Hg" we have d, C)) converges in probability to a value larger than or equal to zero. This implies that under this null hypothesis the implicit critical values of the permutation test. become arbirarily close to a values larger than or equal to zero
Now, let C' be in the GSD-front. Then, due to antisymmetry of X. for every other classifier C", there exists a utility for which the expectation of u &e is larger than the expectation of u o Pcr. Because of the weak law of large numbers, this translates to the empirical expectations with an arbitrarily high probability if only the sample size is large enough. Therefore, all- all-together, as s converges to infinity, the test rejects the null hypothesis in this situation with arbitrary high probability. Finally, since have only a finite number of hypothesis of the static test, this also translates to the static test itself. Therefore the static test is indeed consistent level-a test.
To see that also the dynamic test is valid level-a test for the (random) hypothesis pair observe that under the null hypothesis - the probability of lying in the GSD- front of some random subset S of C is bounded from above by the sum of probabilities of C' lying in the GSD-front of (C,S), where summation over all S E S. As each of these probabilities is bounded from above by by construction, the corresponding sum is bounded from above by |8| g. Finally, as |S] & c, this gives the desired upper bound of a.
Finally, also the consistency of the dynamic test follows from the fact that it is constructed from finite set of consistent tests for every possible set S _ C.
B.4 Proof of Corollary !
Assume that (C x D) C M x S1 x S2, where S1, S2 C [0, 1] are finite, and M C [0, 1] is arbitrary. This is possible since by definition of & we have M dI(C D) and S₁ C Φz(CC D) and S2 S dgsCC x D). and the metrics d2 and 6s are assumed to be finitely discrete. We show that the width" of the restriction of R₁ to &(C x D) is finite. then follows directly from e.g. (76) Proposition 2] that the VC-dimension of
Io = { (a: u(a) 2 ( {te)
is also finite. The claim then follows from Theorem I
"The width of a preordered set is the maximal cardinality of an antichain, i.c. the maximal number of pairwise incomparable elements.
21

---

To show the finiteness of the width, assume wlog that | < oo and | = h < oc. Assume, for contradiction, that there exists an antichain Q S ( D) within the restriction of Ri to (CC 'D) of cardinality strictly greater than 9 h. Then there exist = 42, y (y1, V2, U/) E Q such that 222 = 1/2 and z3 = Ys (as there are only g h different combinations the second and the third component). However, since the first component is completely ordered by 2. this implies either (x, y) or (y. x) (or both) is contained in the restriction of Ri to & x D). This is contradiction to and y being elements of the same antichain Q, completing the argument.
C Further results on the applications
This section provides further information on the benchmarking examples in Section
C.1 Experiments with OpenML
This sections gives further insight to the example on the OpenML data analysed in Section 5..1 We start with giving more details on the data set with all the computation settings of the classifier algorithms, Afterwards, we provide more graphics and explanations of the analysis.
C.1.1
Data
Overall, we are comparing the performance of Support Vector Machine (SVM) to further 6 classifier algorithms on 80 data sets. The data sets as well as the performance evaluation given by the OpenML library [82 13 The analysis is restricted to binary classification problems. We selected those data sets of MP that evaluated the predictive accuracy, truin data time computation and test data time computation (both measured in milliseconds) for all of the 7 algorithms. Since the computation times depend on the environment, i.c. the number of cores used or the free memory, we discretized the computation times and considered them ordinal. Therefore, into ten categories, where category one contains the 10% highest times, and so on. Moreover, we restricted analysis on data sets with more than 450 and less than 10000 observations. This gives us in total 80 data sets.
divided cach computation time
The algorithms discussed are:
Support Vector Machine (SVM) algorithm implemented in the e1071 library 226
Random Forests (RF) algorithm is implemented the ranger library (88).
Decision Tree (CART) algorithm implemented via the rpart library [79].
Logistic regression (LR) algorithm implemented via the nnet library (68).
eXtreme Gradient Boosting (xGBoost) algorithm is implemented in the xgboost library
(20).
Elastic net (GLMNet) algorithm is implemented through the glmnet library 32 and
k-nearest neighbors (kNN) algorithm is implemebted via the kknn library 338
C.1.2 Detailed results of the GSD-based analysis
We started our analysis in Section 5.1 by computing the empirical GSD-front. This gives the Hasse graph where a top-down edge from C' to C' states that deo C") 2 holds.
In addition to the left of Figures I (densities of resampled test statistics) and the right of Figure I (effect of contamination on v-vauues) the main paper, we include the cumulative distribution functions (CDFs) in Figure Since we do not include the values of the observed test statistics here, the differences in distributions are visible to greater extent. We observe the resampled test statisties' distributions for SVM vs. xGBoost and GLMNet to be left-shifted compared to SVM vs. CART, xGBoost, and LR. A visual analysis of the test decision, however, is not possible in the absence of the observed test statistics. This is why include their values in the caption of Figure 3
12 antichain preordered set (M, R) a subset C M such that for m1, ma A it holds (m1, m2) & and (m2, m1) & R. Last OpenML access: 24/10/2024
22

---

CART
KNN
go (C)
xGBoost
GMMNet
Figure 2: The blue shaded region symbolizes the 0-empirical GSD-front for the OpenML data sets.
130
-0.1
4055
LR
KNN
Test: SVM vs.
GLMnet
XGBoast
CART
Figure 3: Cumulative Distribution Functions (CDFs) of resampled test statistics for hypothesis tests SVM LR, RF, kNN, GLMNet, xGBoost, and CART, respectively, OpenML's benchmarking suite. As opposed to Figure T the main paper, values of observed test statistics are not included. They are: 0.0125 (CART), -0.3875 (kNN), 00.4375 (xGBoost), -0.41875 (RF), -0.3375 (GLM- Net), and -0.04897227 (LR). It becomes evident that the resampled test statistics' distributions for SVM xGBoost and GLMNet left-shifted compared SVM vs. CART, xGBoost, and LR. This is also visible in Figure in the main paper, albeit less clearly.
C.1.3 Detailed results of state-of-theeard analyses and comparison to GSD-front
This section provides the detailed computation of the state-of-the offthe-arttt art approaches and the comparison with the GSD approach. Here, we go step by step through all the methods touched Section C.I
First-order stochastic dominance Analogously to the GSD-front, one can define the front based on (multivariate) first-order stochastic dominance (see, e.g., (77). Note that classical first-order stochastic dominance is special case of our generalized stochastic dominance (GSD) in the case that all quality metrics are (treated as) of ordinal scale of measurement. Given the test logic followed by. for example, (6). for the OpenML data it turns out that no classifier is significantly stochastically larger than (or incomparable to) any other classifier (based on a significance level %). Compared the results we obtain from our GSD-front analysis, this indiscriminative result is much less informative.
Pareto-front Both for the PMLB benchmark suite and the OpenML setup the Pareto-front contains all considered classifiers and is therefore not very informative. This shows the advantage of our approach because our approach is generally more informative (see Theorem ( In particular, by using generalized stochastic dominance one refrains from solely relying on pointwise comparisons of classifiers over datasets. Instead one only looks at the distribution of the multidimensional quality metric. Beyond this, compared to both Pareto analysis and a classical first order stochastic dominance analysis (see above), the GSD approach does justice to the fact that the dimension accuracy is cardinal and, at the same time, the fact that the other dimensions are of ordinal scale of measurement. Additionally, compared to an approach that only looks at the marginal distributions of every single quality metric separately, the GSD approach takes also the dependence structure
23

---

between the different quality metrics into account. This is of particular interest if one has different performance dimensions that are anticorrelated.
Weighted sum approach The GSD>-ased approach has advantages over an approach of weighted summation of the various quality metrics especially when it is not clear how specifically the weights are to be chosen. Specifically, each weighting leads to a total ordering among the classifiers under consideration. A clear best classifier can, therefore, be identified for each weighting. Thus, different weightings generally lead to different best classifiers. As consequence, if one chooses a specific weighting, one should really be convinced that domain knowledge thoroughly justifies it, as even small changes in the weighting can completely change the resulting ranking. In contrast, the GSD-based approach can be used if no weighting of the involved metrics available, but still more information (e.g., from the cardinal metrics) is available than required for a Pareto-tyr type analysis. In summary, we emphasize that our method and the weighted summation should be used under different conditions and, therefore, complement rather than compete with each other.
Marginal-front A highly popular testing scheme for benchmark analysis the one proposed by 22).. We compare our approach against using a marginal front that directly results from following this scheme. This marginal front is defined as function of (a) statistical test(s), i.e. classifiers are in it depending on the test results. We emphasize that this front is not directly comparable to the GSD-front, since the GSD-front is a theoretical object (like the Pareto front) that can be used to formulate hypotheses that can then be tested by statistical tests like the ones proposed in this paper. Thus, we compare the marginal front to the empirical GSD front as reported Fig. of the paper for the application on OpenMI..
We multiple single-objective evaluations and include in the marginal-front the classifiers that not statistically significantly worse than another classifier on all metrics. For the single-objective tests, we follow the well-established procedure of [24). That is, we first run a global Friedman test (see (33) for the null hypothesis that all classifiers have no differences with respect to the quality metric under investigation. In case we reject this null hypothesis, we can run post hoc Nemenyi pairwise tests (see [63]), comparing the performance of algorithms pairwise, with the null hypothesis being that there no difference between their performances w.r.t. the multidimensional quality metric considered. We would like to emphasize that such an approach does not take into account the dependence structure among the quality metrics. other words, it only considers the marginal distribution (hence the term marginal-front) of the classifiers w.r.t. the individual quality metrics separately, not their joint distribution. In the following, we conduct the suggested marginal analysis for OpenML w.r.t. the three-dimensional quality metric considered (accuracy, computation time on training data, computation time on test data): Accuracy
Global Friedman Test: Friedman rank sum test (33 rejects global null of no differences (p-value = 3.986e-14) This means we can conduct (two-sided) pairwise post hoc tests (a = 0.05) with no difference as the null hypothesis.
Post Hoc Nemenyi Test: Table below shows the pairwise comparisons of algorithm performance with the Nemenyi test (63] P-values below 0.05 are highlighted and indicate statistically significant differences in performance.
Table 1: Pairwise comparisons of algorithm performance with the Nemenyi test based accuracy. Underlined values indicate differences significant at a = 0.05 level.
LR
RF
CART
SVM
xGBoost
GLMNet
kNN
RF
3.9e 11
CART
0.19622
6.9e - 05
SVM
0.00055
0.06513
0.55264
xGBoost
0.92896
5.7e 08
0.85263
0.03259
GLMNet
0.92341
6.4e 08
0.86095
0.03446
1,00000
KNN
0.98454
9.2e 09
0.68760
0.01261
0.99995
0.99993
24

---

Computation time on training data
Global Friedman Test: Friedman rank sum test 33 rejects the global null hypothesis of no differences (p-value < 2.2c-16). This means we can conduct pairwise post hoc tests 0.05) with the null hypothesis of difference.
Post Hoc Nemenyi Test 63
Table
Table 2:
Pairwise comparisons of algorithm performance with the Nemenyi test based on computation time the training data. Underlined values indicate differences significant 0.05 level.
LR
RF
CART
SVM
xGBoost
GL.Net
kNN
RF
9.1e
CART
0.13788
< 2e 16
SVM
3.4c - 05
0.00037
4.0e 12
xGBoost
5.9c - 14
0.97584
< 2e 16
5.1e 06
GLMNet
0.03081
5.1e 08
2.9c 07
0.62723
1.6e 10
KNN
1.3e 08
< 2e 16
0.00541
5.8e 14
< 2e
16
7.2e 14
Computation time on test data
Global Friedman Test Friedman rank sum test (33) rejects the global null hypothesis of differences (p-value < 2.2c-16). This means we conduct pairwise post hoc tests (a. = 0.05) with the null hypothesis of no difference.
Post Hoc Nemenyi Test 633. see Table
Table 3: Pairwise comparisons of algorithm performance with the Nemenyi test based on computing time on testing data. Underlined values indicate differences significant at a 0.05 level.
LR
RF
CART
SVM
xGBoost
GLMNet
kNN
RF
2e 16
CART
0.676
< 2e
16
SVM
0.652
6.5e 14
0.019
xGBoost
<. 2e 16
0.996
< 2e 16
7.2e 14
GLMNet
3.2e 09
6.2e 06
1.2e 13
4.0e 05
1.9c 07
KNN
9.1c 14
0.177
6.8c 14
2.3c 12
0.034
0.106
Table 4 provides the mean results of the classifier comparisons. (Recall that for train/test time: the lower, the better)
Table Mean results of the classifier comparisons.
LR
RF
CART
SVM
xGBoost
GLMNet
kNN
Accuracy
0.761
0.854
0.831
0.8113
0.820
0.763
0.789
Train Time
0.370
7.019
0.199
1.866
9.561
1.491
0.012
Test Time
0.062
0.458
0.055
0.106
0.407
0.184
0.291
As becomes evident from the mean values of the three quality metrics and the single-criterion test results presented above, there is no classifier that significantly dominated by another classifer w.r.t. all three quality metrics. Hence, the marginal- front would contain all classifiers and would be rather indiscriminative compared to the empirical GSD-front that we present in the paper, see Figure Appendix C.I. which contains random forest (RF), trees (CART), and k-nearest neighbor (kNN). This is in line with our explanation of OpenML results above. Since the quality metrics accuracy, train time, and test time only weakly (if at all) correlated due to trade-off between speed and accuracy, the marginal-fin
based on single-criterine comparisons does not facilitate practitioners' decision-making, while empirical GSD-front provides valuable insights.
For the sake of completeness, we also report the results these multiple single-ohjective evaluations on the PMLB benchmark suite in tables 5 and The interpretation is completely analogous
25

---

Table 5: Post Hoc Nemenyi Test (Accuracy) on PMLB.
cre
svmRadial
J48
ranger
knn
glmnet
svmRadial
0.74628
J48
0.78740
0.08257
ranger
0.00106
0.09912
2.2c-06
knn
0.00227
4.2e-06
0.13239
2.0e-13 13
glmnet
1.00000
0.67200
0.84844
0.00064
0.00360
Table 6: Post Hoc Nemenyi Test Summary (Accuracy with Noisy X) on PMLB.
cre
svmRadial
J48
ranger
knn
glmnet
svmRadial
0.8130
J48
0.4971
0.0323
ranger 0.3063
0.9647
0.0019
knn
0.0072
3.8e-05
0.5290
5.1e-07
glmnet
0.7173
0.0826
0.9994
0.0067
0.3195
to the interpretation of the results on OpenML above. Note that the Friedman rank sum test rejects global null of no differences for all three criteria. This means we can conduct pairwise post hoc tests (a 0.05) with (two-sided) null of no difference.
C.1.4 Discussion of the unexpected results
Recall the discussion in Section 5.] about the unexpected results. We want emphasize that these have a high degree originality and should be of particular interest to practitioners. This shows that experience and intuition with method can also be misleading if only the evaluation framework slightly modified: multidimensional quality metric that seeks the optimal trade-off between different, potentially confficting metrics will generally rank differently than unidimensional one. In the following, we show that the dominance of CART over xGBoost, SVM, LR, and GLMNet is indeed consistent with the quality metrics provided by the OpenML repository.
First all, here, we are interested in the trade-off between accuracy and computation time, (e-g., the better the accuracy, the higher/worse the computation time). We now look at the comparison between SVM and CART to demonstrate that the results are indeed in line with the data. We obtain:
For 27 datasets, CART outperforms SVM on all dimensions (c.g., prediction accuracy, computation time on test data, and computation time on training data) at once.
For 9 datasets, CART dominates SVM for at least quality metric and for all other quality metrics the performance CART is not worse.
For 41 datasets, SVM's prediction accuracy is better than CART's. At the same time, CART outperforms SVM for at least one of the two computation times. The two classifiers are therefore incomparable for these datasets.
For 3 datasets CART outperforms SVM based accuracy, but least of the computa- tion times of SVM is below the one of CART.
Overall, there exists no dataset where SVM dominates CART in all dimensions at once. Either the two classifiers are incomparable, CART dominates SVM. Furthermore, CART dominates SVM
Table 7: Post Hoc Nemenyi Test Summary (Accuracy with Noisy Y) on PMLB.
cre
svmRadial
J48
ranger
knn
glmnet
svmRadial
1.00000
J48
0.03722
0.04911
ranger
0.06405
0.04911
1.7e-07
knn 0.00096
0.00141
0.90728
2.3e-10
glmnet
0.67200
0.73193
0.68732
0.00031
0.12513
26

---

Table 8: Mean Results (Accuracy and Noisy Data) on PMLB
cre
svmRadial
J48
ranger
knn
glmnet
Accuracy
0.7807
0.8494
0.8347
0.8629
0.7780
0.8106
Accuracy with noisy x
0.7307
0.7823
0.7679
0.7924
0.7339
0.7570
Accuracy with noisy y
0.7346
0.7776
0.7638
0.7984
0.7237
0.7640
for nearly half of the datasets (27 + 9 36 of 80). Thus, the dominance structure provided by our method is in line with the performance evaluation values provided by OpenML.
A second issue that may have influenced the unexpected performance structure obtained in the paper the way performance is evaluated by OpenML. OpenML is based on the uploads of its users. Each user is free to decide which hyperparameter settings to use. Thus, as there might be different goal on the hyperparameter setting in each dataset, the results are not representative for the best performance of each algorithm. This aspect should be included in any further discussion. Especially since some algorithms are more dependent on hyperparameter settings/tuning than others. For an example involving hyperparameter tuning that is fixed in advance, see Section 5.2
C.2 Experiments with PMLB
This sections give further insight the exemplary benchmarking analysis the Penn Machine Learning Benchmarks (PMLB) in Section in the main paper. We start by giving more details on the data sets with all the computation settings of the classifier algorithms. Afterwards, we provide more figures and explanations of the analysis.
C.2.1 Data
Penn Machine Learning Benchmarks (PMLB) is collection of curated benchmark datasets for evaluating and comparing supervised machine learning algorithms (64). We select all datasets from PMLB for binary classification tasks with 40 1000 observation and less than 100 features. On these 62 [s recently proposed classifier based on compressed rule ensemble learning [62] is compared w.r.t. robust accuracy against five well-established classifiers, namely classification tree (CART), random forest (RF), support vector machine with radial kernel (SVM), k-nearest neighbour (kNN), and generalized linear model with elastic net (GLMNet). In detail, we deploy
Support Vector Machine (SVM) algorithm as implemented in the e1071 library 26
Random Forests (RF) algorithm as implemented in the ranger library library 26 and dplyr [87]
88
requiring e1071
Decision Tree (CART) algorithm (C4.5-like trees) as implemented in the RWeka library
39
Elastic net (GLMNet) algorithm implemented through the glmnet library
32
requiring
library Matrix (7). and
k-nearest neighbors (kNN) algorithm as implemented in the kknn library
38
Note that we used the respective methods in the caret library (49] for hyperparameter tuning and cross-validation to retrieve i) through iii), as detailed below.
We operationalize the latent quality criterion of robust accuracy through i) classical accuracy (met- ric). ti) robustness w.r.t. noisy features (ordinal), and iii) robustness w.r.t. noisy classes (ordinal). Computation is straightforward; in order to retrieve ii) and iii), we follow [92 93) by randomly perturbing share (here: 20 %) of the data points. We randomly selected data points with selection probability of 20% and replaced the values by random draw from the marginal distribution of the corresponding variable. (This is slight difference to [92 93] who replaced the data points by random draw from a uniform distribution the corresponding support of the marginal distribution.)
We then tune the six classifiers' hyperparameters on (multivariate) grid of size 10 following 49 for each of the 62 datasets and eventually compute i) to iii) through 10-fold cross validation.
49 requires least 4 data points in the test set, which translates to mininmal 40, since we deploy 10-fold cross validation.
15 Last access of PMLB: 12/05/24.
27

---

RF
CART
KNN
GLMNet
SVM
egsd
62 (C)
GRE
Figure 4: Hasse graph of the empirical GSD-relatiio for the PMLB data sets. The blue shaded region symbolizes the 0-empirical GSD-front, Definition
C.2.2 Detailed results of the GSD-based analysis
To initially obtain a purely descriptive overview, we construct the Hasse graph illustrating the empirical GSD relation. In this process, we calculate the value dc2 C*) for C' * C' C = {CRE, SVM, RF, CART. GLMNet, kNN) and connect to C² with a top-down edge whenever de2( d62(C, ((,CC)))) C*) 2 O. The resulting graph is portrayed in Figure 4 It is evident from the graph that RF (strictly) empirically GSD-dominates the classifier CRE. ATl other classifiers are pairwise incom- parable. Five classifiers, namely RF, CART, KNN, GLMNet, and SVM are not strictly empirically GSD-dominated by any other considered classifier and, thus, form the O-empirical GSD-front.
This latter purely descriptive analysis already hints at the CRE not belonging to the GSD-Front. In order transition to inferential statements, we aim statistically test (at level a 0.05) whether CRE significantly lies in the GSD-front of some subset of C. As detailed in Section 4.1 we conduct
five pairwise tests for the hypothesis pairs (H -HO (where C == CRE and C" E C {CRE)) at a level of e as explained in Section 4 other words, we test five auxiliary null hypotheses, each asserting that CRE is ESD-dominated by SVM, RF, CART, GLMNet, and kNN, respectively.
The results of these tests are visualized in T and Figure (cumulative distribution functions) 17 They indicate that the pairwise tests of CRE versus SVM, RF, CART, GLMNet, and level of 5 nor at level a. Hence, conclude that based on the observed
kNN do not reject benchmark results we cannot conclude at significance level a 0.05 that CRE lies in the GSD-front of any subset of In other words, we have no evidence to rule out that CRE is in the GSD-front, i.c. we cannot confirm based on the data that CRE is not outperformed by SVM, RF, CART, GLMNet, and KNN with respect to all compatible utility representation of robust accuracy. As can be seen in Figure 5 testing CRE vs. CART results in the smallest p-value of all pairwise tests, which appears plausible, since CRE is CART-based method. On the other hand, the observed test statistic of CRE v8. RF is far away from the critical value and the test cleary does not reject, even though RF is also tree-based method.
Finally, as discussed in Section 4.2 we further analyze the robustness of this test decision under contamination of the benchmark suite, i.c., deviations from the i.i.d. -ssuumpton.. As opposed to our OpenML analysis in Section 5.1 see also Appendix C.I contamination does not affect the test decisions here, since none of the tests rejects already for uncontaminated samples. Increasing contamination only drives p-values further. The results are visualized in Figure 7 It is observed that the tests are neither significant at level of 0.05 nor 0.05 and this clearly does not change with growing size of contaminated benchmark data sets.
In summary. the PMLB experiments demonstrated how apply our benchmarking framework to the problem of comparing a newly proposed classifier to a set of state-of-the-as ones. Furthermore, it illustrated our tests' applications multiple criteria of mixed scales (ordinal and cardinal) that operationalize a latent performance measure, namely robust accuracy. It became evident that our framework allows statistically assess whether the novel classifier CRE can compete with existing ones that is, whether CRE lies in the GSD-front of some state-of-the-ar classification algorithms. In
16 clarified Footnote the tests in Sections 5.1 and 5.2 are based on the unregularized test statistics (C",C)
"For generating these plots, used quantile functions from both base and the ggplot library. As the underlying quantile functions definitions differed slightly, we relied on the latter and corrected the quantiles from the other manually. Detailed documentation of all computations involved in generat- ing the visualizations in the paper. we refer the interested reader to htttps://githuu com hoom/hnnnabblo/ Statistical Matiiaaiiiiiiiiiiiieiiiiei Satulliiettttr--BBaaaakkniiee via- Che-GSDDFrooi
28

---

CRE VSL
-039
-0.5
EXE 083
Decision:
Reject:o
Ragadeer
Figure Densities of resampled test statistics for pairwise tests of CRE vs. six other classifiers on 62 datasets from PMLB. Big (small) vertical lines depict observed (resampled) test statistics. Rejection regions for the static (dynamic) GSD-test are highlighted red (dark red). As becomes evident, we cannot reject any of the pairwise tests for neither significance level.
6.1
Test CRE
5VN GLNMe
CART
Figure 6: Cumulative Distribution Functions (CDFs) of resampled test statistics for hypothesis tests on PMLB benchmark suite of CRE vs. SVM, GLMNet, RF, kNN, and CART, respectively. As opposed to Figure aboov, values of observed test statistics are not included. They are: -0.1031746 (CART), -0.08730150 (kNN), 0.02380952 (RF), -0.05555556 (GLMNet). -0.07936508 (SVM). It becomes evident that the resampled test statisties' distributions more similar to each other than in the case of testing SVM vs. competitors in the OpenML benchmark suite.
29

---

Test
p-
Contaminated Samples
Figure 7: Effect of Contamination: p-values for pairwise tests of CRE versus the five competitors PMLB benchmark suite application. Analogous to Figure [ dotted red lines mark significance levels of a = 0.05 (dark red: a = _ ). Since none of the tests reject for a = 0.05 under no contamination, this obviously does not change with contaminated samples.
this case, the test decisions of both static and dynamic GSD-tests was not to reject the null hypothesis of CRE being outperformed by RE, CART, SVM, GLMNet, and kNN w.r.t. robust accuracy.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract and introduction of the paper reflect the paper's contribution and scope, covering both theoretical and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The limitations of the work are discussed in Sections 4.2 and 6 of the main paper.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
31

---

Answer: [Yes]
Justification:
The main theoretical results of the paper are stated in Theorems 1, 2 and well Corollary For each of these theorems, full sets of assumptions are provided. Moreover, complete proofs for all these statements are provided in the paper's appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper fully discloses all information needed to reproduce the experimental results in Section 5 of the main paper and Section C of the paper's appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
32

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All data and code needed to reproduce the paper's experiments are openly accessible via GitHub repository. A link this repository provided in Footnote of the main paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https www.eeeeeeoeeemmmeeeeeeeeeee
/ /twpps.ccppppaee.. wwtpcccppppccccceeeoec
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All necessary background information to understand the papers experimental results be found in Section 5 of the main paper and Section C of the paper's appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All experimental results the paper are statistically tested for significance. Moreover, the test decisions are checked with respect to their robustness towards the as- sumptions underlying the respective tests. Compare Section 4 for theoretical considerations testing and Sections and Appendix C for statistical test results of the applications.
Guidelines:
The answer NA means that the paper does not include experiments.
33

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Information computer resources is provided in the GitHub repository referenced to in Footnote 3 of the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwwppppppppp//aaaaaacaaa
Answer: [Yes]
Justification: The research conducted in the paper is, in every aspect, conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
34

---

We do not foresee direct negative societal impact from the current work. Positive societal impact in form of making benchmarking results more robust to unjustified assumptions are discussed in Section 4.2 of the paper.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
35

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
36

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "8ovGAqFPVa", "reviewer": "Reviewer_1fNM", "summary": "This submission studies the problem of comparing multiple classifiers under multiple evaluation criteria. It presents the construction of the GSD-front, i.e., the set of GSD-undominated classifiers, and an empirical variant called the $\\epsilon$-empirical GSD-front.\n\nThen, theoretical aspects of the GSD-front and the $\\epsilon$-empirical GSD-front, such as satistical consistency, and the inclusion concerning the respective Pareto-front, are investigated. \n\nTwo statistical tests, namely static GSD-test and dynamic static GSD-test, on whether a given classifier $C \\in \\mathcal{C}$ is an element of the true GSD-front are proposed.  Both tests are shown to be valid level-$\\alpha$ tests. A robustified static GSD-test is also proposed to deal with the setting where the available data sets are drawn under non-i.i.d.-scenarios.\n\nAn empirical study is conducted to assess the proposed tests. It covers 80 binary classification data sets from OpenML [70]. The set of classifiers consists of Support Vector Machine (SVM) with Random Forest (RF), Decision Tree (CART), Logistic Regression (LR), Generalized Linear Model with Elastic net (GLMNet), Extreme Gradient Boosting (xGBoost), and k-Nearest Neighbors (kNN). The set of evaluation metrics consists of predictive accuracy, computation time on the test data, and computation time on the training data. Empirical evidence suggests that the GSD-approach tends to be more contrastive, compared to Pareto-front and another test that aggregates the single-metric tests as in [19]. \n\nAnother empirical study is conducted using 62 datasets from the Penn Machine Learning Benchmark (PMLB). The set of classifiers consists of rule ensembles of trees (CRE) [53],  CART, RF, SVM with radial kernel, kNN and GLMNet.  The set of evaluation metrics consists of classical accuracy (metric), ii) robustness of accuracy w.r.t. noisy features (ordinal), and iii) robustness of accuracy w.r.t. noisy classes (ordinal). The results provided by GSD-tests are discussed.", "questions": "Q1: Could you elaborate more on the case where only one training data is taken into account? This is related to the comment in W2. \n\nQ2: Regarding W3, should one randomly choose one element of the GSD-front when having to make predictions? Yet, one might consider doing ensemble learning on the set of promising classifiers.  In safety-critical applications, such a strategy might lower the interpretability and explainability of the predictive system. Do you have any recommendations in such cases?\n\nQ3: Could you also discuss the Pareto-front and results of the test that aggregates the single-metric tests as in [19] for the empirical study on PMLB data sets?", "limitations": "Please refer to \"Weaknesses\" and \"Questions\" for detailed comments.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The submission is well-written and organized. \n\nS2: Theoretical aspects of the GSD-front and GSD-tests are carefully investigated. \n\nS3: The authors also take into account an interesting aspect that the different evaluation metrics may have different levels of impact. Non-i.i.d.-scenarios are also taken into account.\n\nS4: Empirical evidence seems to suggest that GSD-tests can complement the existing statistical tests for comparing classifiers meaningfully.", "weaknesses": "W1: The authors might consider mentioning more scenarios where comparing multiple classifiers under the multiple evaluation criteria is crucial. An example might be multi-label classification, where multiple evaluation metrics have been proposed to evaluate multi-label classifiers. The set of evaluation metrics can also be extended with others, such as training time, storage memory, and so on. \n\nW2: The motivation to compare multiple classifiers using evidence on multiple data sets, which can come from very different application domains might need to be strengthened. For example, it would be not easy to convince practitioners in safety-critical applications, such as lung cancer detection to use some classifier that has shown to be promising given evidence aggregated from multiple domains.     \n\nW3: Additional recommendations for the end-user might be beneficial. For example, what should one do if the GSD-front contains multiple classifiers?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and thoughtful consideration. We appreciate the suggestions to include end-user recommendations and comparative studies for the PMLB datasets. We follow both in the revision (recommendations go to the main paper, studies to the appendix). We are pleased you find our paper \"well-written and organized\" and our framework \"carefully investigated\".\n\nWe now address your comments:\n\n**W1:** We are grateful for your idea to add more scenarios, which we do in the revised version. The multilabel classification case is very interesting: For classes $i=1,...,K$, one can use a collection of classical metrics for multiclass classification like Hamming distance, Jaccard similarity etc. Within our approach, a more direct way may be even more interesting: Take a separate metric for every class (e.g., accuracy) and then treat all metrics together as a K-dimensional multi-cardinal metric. With the flexibility of the $R_2$ relation, one can implement different strengths of the underlying scale of measurement, e.g. by imposing only cardinal commensurability within each class, or by additionally requiring interclass commensurability (which presumably leads to an approach very similar to using only a classical (cardinal) one-dimensional metric for the entire multiclass performance). This can be accompanied, similar to our experiments, with further criteria such as training time or memory. \n\n**W2:** We fully share your concern about benchmarking on datasets from multiple domains different from the one of intended use, e.g., lung cancer detection. We consider this a very relevant problem in benchmarking generally (not only limited to our GSD-analysis). It is the very reason why we robustified our tests against non-i.i.d. selection of datasets in section 4.2. We observe it to be common practice in benchmarking to simply test methods on huge benchmark suites that include datasets from a myriad of domains. Statistically, such a collection can hardly be considered *i.i.d.* from the population of interest.\nOur robustification allows for valid conclusions even if some heterogeneity/asymmetry in the selection processes is allowed, i.e., if it deviates from *i.i.d*. E.g., for the OpenML study, the observed conclusions remain valid under contamination (i.e. can come from very different data domains) of 7, 8, 11 and 11 out of 80 datasets respectively (lines 335-344). This exactly addresses your concern: Our methodology allows for valid statistical conclusions even if the datasets are not *i.i.d*.\nFinally, from a different angle, this point is an excellent motivation to extend our results to stratified and regression-like situations, as mentioned in Section 6. Clearly, the area from which a dataset originates is an important meta-information to include.\n\n**Q1:** Our method can also be applied to one dataset. It then translates to the partial order resulting from the classifiers' performances w.r.t. a multidimensional metric on this dataset, and, in the further special case of only one metric, even results in a total order. This is a sensible thing to do if one is interested in a purely descriptive (as opposed to inferential) analysis of classifier performance on a single dataset. Then, however, much of our conceptual work regarding statistical properties of our estimator and tests is not needed; the tiny sample size does not allow for meaningful inferences. Nevertheless, such an application of our methodology is possible in principle and leads to the known special cases.\n\n**Q2 and W3:** We agree it is good to add a paragraph with end-user recommendations. We use part of the extra page for such an explanation and will add it to the beginning of Section 5. Specifically, we plan to explain that our method is not primarily intended to identify the best classifier for a problem class. (In this sense, it is not a problem if the GSD front contains more than one element.) It is rather for checking whether a newly proposed classifier for a certain problem class can potentially improve on the state-of-the-art classifiers, or whether it disqualifies itself from the outset. Furthermore, we want to emphasize that our framework allows us to provide such statements with inferential guarantees by means of appropriately constructed statistical tests, and even to examine these inferential guarantees in terms of their robustness to deviations from the i.i.d assumption.\n\nWe are very grateful for your idea about ensemble learning with the GSD-front. Although our original motivation was different, this sounds practically quite rewarding, but of course needs more careful attention than can be paid in the context of this paper under review. Speculating a bit, it seems tempting to apply aggregation rules to the outputs of the different classifiers in the GSD front. Most in line with our work is to rely then on a credal point of view, collecting, for every unit, all the predicted classes (similar to E-admissibility in credal decision making). In safety-critical applications, this may help to avoid overoptimism, and it informs the user about whether there is consensus among the non-dominated classifiers. \n\n**Q3:** We add the results of the Pareto-analysis and the aggregated single-metric tests for the PMLB study to the appendix. Also for the PMLB case, the Pareto front contains all classifiers and is not informative. The pdf page presenting the results of the single-metric tests as in [19] is attached to the “global response” above. Furthermore, note that – for all three metrics considered – the Friedman rank sum test rejects the global null of no differences such that we can conduct (two-sided) pairwise post hoc tests (α=0.05) with no difference as their null. The interpretation of the results is in complete analogy to the one of the OpenML study in Ap. C.1.3. \n\nWe thank you once more for your review and your concrete suggestions and questions allowing us to further improve our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ICT6g57OrZ", "reviewer": "Reviewer_3sEe", "summary": "The authors propose a method for multicriteria evaluation of classifiers which is more informative than the Pareto front. The new GSD-front and is based on the previously proposed generalized stochastic dominance ordering (GSD) for classifiers. The authors also provided a sound inference framework, included hypothesis tests for whether a new classifier would be in the GSD-front of a benchmark.", "questions": "1. How does the running time of a Pareto-front benchmark compare with a GSD-front one?\n    1. If the Pareto-front is much cheaper to obtain, would it make sense to start with it, and only look for what classifiers in that pre-selection would be in the GSD-front?\n2. How were different hyper-parameter values treated in the experiments? Were all results of each classifier for a dataset aggregated or did the authors use specific settings for each classifier, e.g. default values from the classifier's implementation?", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The method is sound and extremely useful\n2. The paper is very well written; notation is consistent throughout \n3. Given the large (and increasing) number of potential models that can be applied to any given task, this method is very significant for identifying which models tended to be among the best for other similar tasks\n4. The hypothesis tests derived from the framework are also very useful to add significance to these benchmarks\n5. All the information needed to fully understand the paper is in the main body of text", "weaknesses": "1. There could have been maybe a toy example to show the differences between GSD-front and Pareto-front in a more didactic way\n2. The method seems computationally expensive. The pmlb_permutation_tests.R script needs 5 days to run and this does not include training of the evaluated models, as these are collected results", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we would like to thank the reviewer for their time and thoughtful consideration of our paper. We are grateful for the suggestion to add a toy example showing the differences between GSD-front and Pareto-front in a more didactic way and the suggestion to add more details on hyperparameter tuning in the PMLB experiment. We follow both suggestions in the revision of our manuscript (the example will be added to the main text, the tuning details to the corresponding paragraph in the appendix). \n\nWe are very pleased that the reviewer appreciates our method as “extremely useful” and the paper as “very well written”, with notation being “consistent throughout”.\nWe will now address the reviewer's comments in turn:\n\n\n**Weakness 1: Missing toy example**\n\nWe agree that a concrete (minimal) example could help to quickly and intuitively grasp the differences between Pareto and GSD-Front. We are happy to use parts of the additional page to include such an example. Specifically, we plan to build the example directly after Definition 1 to illustrate the Pareto front and then continue it after Definition 6 to contrast the GSD front. To further strengthen the reader's intuition right from the start, we also plan to refer to the \"Method comparison\" from Section 5.1 at the end of these examples. Thanks for the proposal, we think this is an excellent idea! \n\n\n**Weakness 2: Computation time**\n\nWe agree that the computation time is relatively long. However, we believe that this is not really a major problem for the settings in which our approach is intended to be applied. Firstly, we are in a benchmark setup, so our method only needs to be run once, not several times. This is in contrast to the intensive hyperparameter tuning of models that is often part of everyday work. Even more important, our method can provide the information that an algorithm is dominated by others and, therefore, should not be considered in later tasks. So, starting with a rather expensive benchmark experiment can save a lot of time later on, in particular, as the results provided by our method come with (additionally robustified) inferential guarantees and are not purely descriptive.\n\nSecondly, the expensive part is the computation of the test (as it is a permutation test) and not the computation of the empirical GSD front. The computation of the test is indeed expensive, but unlike other benchmarking approaches, we obtain from it a statistically sound inferential statement. If we restrict our analysis to the empirical, non-inferential part, the computation time is much smaller. \n\n\n**Question 1: Pareto-front vs empirical GSD-front**\n\nWe think the idea to first compute the Pareto-front (as a kind of pre-processing step) and then proceed with our GSD-based analysis is excellent. Thank you very much for that suggestion! We will include this idea in the new paragraph with end-user recommendations (see the answer to 1fNM) in the revised manuscript.\nConcerning the computation of the Pareto vs GSD-front: Since the Pareto-front simply checks whether one classifier is strictly dominated by another, the Pareto-front is simply based on counting the existence of observed dominances. This can be done in milliseconds. As we pointed out above, the computation of the empirical GSD-front is much faster than the entire inference part, but still more expensive than the computation of the Pareto-front. Similar to above, we think that the further insight that the empirical GSD-front provides more than justifies the higher computational time – in particular, as it can save computation time later on.\n\n\n**Question 2: Hyper-parameter**\n\nThanks for this important question. In the case of the Open ML platform, we used the performance evaluation provided by the OpenML library, see line 838-846. In the case of the PMLB benchmarking suite, we tuned the six classifiers’ hyperparameters on a (multivariate) grid for each of the 62 datasets and eventually computed all evaluation metrics through 10-fold cross validation, see lines 999-1008 in the paper for more details. Concretely, we tuned\n\n- “sigma” and “C” for support vector machines\n- “mtry”, “splitrule” and “min.node.size” for ranger\n- “k” for knn\n- “alpha” and “lamba” for glmnet\n- “CF”, “M”, and “U” for CART (J48)\n- “eta” and “k” for compressed rule ensembles (cre)\n\n\n\nWe thank you once more for your review and your concrete suggestions and questions allowing us to further improve our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e7uXLA4R3c", "reviewer": "Reviewer_n7Tx", "summary": "The authors propose a novel way of comparing classifiers to assess their effectiveness. They posit (1) that comparisons should allow for different quality metrics simultaneously, (2) that comparisons should take into account the statistical uncertainty induced by the choice of benchmark suite, and (3) that the robustness of the comparisons under small deviations in the underlying assumptions should be verifiable. They illustrate their proposed method on the benchmark suite PMLB and on the platform OpenML.", "questions": "When introducing imprecise probabilities, the authors fail to reference a few papers in such a literature. In particular, the authors leverage the idea of $\\epsilon$-contamination in section 4.2. Other papers in the imprecise probabilistic machine learning literature dealing with, and advancing the knowledge of, $\\epsilon$-contaminations are [1-5]. In addition, [6] uses IPs to robustify Bayesian neural networks. More in general, we suggest the authors to read (and possibly cite) the works by Eyke Hüllermeier, Thierry Denoeux, Sebastien Destercke, Fabio Cuzzolin, Alessio Benavoli, and Michele Caprio on imprecise probabilistic machine learning, all of which have a distinct \"robustness\" flavor to them, and that might be interesting to the authors for future endeavors.\n\nThere is a typo in line 95: there's a period after superscript 3.\n\nWhile I understand the need of being as general as possible, wouldn't it be more beneficial to the reader to define $\\hat{\\pi}_c(\\{z\\})= 1/s \\left|\\lbrace\\{ i : i \\leq s \\text{, } \\Phi(C,T_i)=z \\rbrace\\}\\right|$ in line 167, instead of using a generic $M \\subseteq [0,1]^n$ for the definition?\n\nThere is a typo in line 167: last letter should be $C^\\prime$ and not $C$.\n\nLine 205: the question *of* how to [...]\n\n---\n\n[1] https://www.jstor.org/stable/2242055\n\n[2] https://onlinelibrary.wiley.com/doi/book/10.1002/9780470434697 (Chapter 10)\n\n[3] https://link.springer.com/chapter/10.1007/978-3-031-57963-9_1#:~:text=In%20their%20seminal%201990%20paper,bound%20to%20hold%20with%20equality.\n\n[4] https://arxiv.org/abs/2402.00957\n\n[5] https://arxiv.org/abs/2308.14815\n\n[6] https://arxiv.org/abs/2302.09656", "limitations": "See questions.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clear and easy to follow. The proposed method is innovative, it enjoys some desirable mathematical properties (that are well-presented), and it seems to work well on the selected benchmarks.", "weaknesses": "Some references to the existing imprecise probabilistic literature are missing, there are some small typos, and an extra effort can be put to make the definition of $d_s$ more accessible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we would like to thank the reviewer for their time and thoughtful consideration of our paper. We believe that the more direct mathematical notation suggested by the reviewer regarding the empirical measure as well as the suggested more detailed explanation of the test statistic will help to further improve the accessibility of the paper, which the reviewer otherwise describes as “easy to follow”. \n\nWe are pleased that the reviewer finds our method “innovative” and emphasizes that it “enjoys some desirable mathematical properties”, which they also describe as “well-presented”.\n\nWe will now address the reviewer's concerns and questions in turn:\n\n**Weakness 1: Missing references, different notions of robustness in IP for ML**\n\nWe agree with the reviewer that Huber's book on robust statistics should definitely be cited when introducing the contamination model and thank the reviewer for identifying this gap. We add a corresponding reference to the revision of our paper. \nWe are also very grateful for the further references provided by the reviewer; they for sure deserve proper consideration: We will go through them and use some part of the additional page to add appropriate references in the revision of our related work section and the introduction of contamination models in Section 4.2.\nIn this context, we will also pay special attention to sharpen the presentation of our understanding of robustness even more, also in contrast to related, but in detail subtly different concepts from current literature.\nThanks once more for raising the issue of slightly deviating robustness concepts and for all the hints to important further references.\n\n\n**Weakness 2: Extra effort to make the definition of $d_s$ more accessible**\n\nWe agree with the reviewer that a purely formal introduction of the test statistic $d_s$, which plays a central role in the paper, is not ideal. We are happy to use some of the additional space in the revised version of the manuscript to add an additional – less formal and instead more intuitive – description of $d_s$. We think this is an excellent point and thank the reviewer for raising it.\n\n**Questions:**\n\nFirst of all, thank you for your close reading, which made it possible to find three notational and grammatical typos scattered throughout the paper. We appreciate this very much. We agree that your proposed definition of the empirical measure is more direct and more easy to access. We will change this accordingly in the revised version. Thank you very much for pointing this out!\n\nWe thank you once more for your review and your concrete suggestions and questions allowing us to further improve our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Global response**

We sincerely thank all reviewers for their thorough, high-quality, and detailed assessment of our manuscript. We are encouraged by the very positive, affirmative reviews and feel most grateful for the very precise and constructive suggestions on how to improve our paper further. The reviews unanimously underscore the novel and innovative nature of our proposed benchmark methodology via the GSD-Front.
 
 
**Reviewer 1fNM** praises the paper for its "well-written and organized" presentation, emphasizing the thorough investigation into the theoretical aspects of the GSD-front and GSD-tests. The reviewer further welcomes the inclusion of different evaluation metrics and the consideration of non-i.i.d. scenarios as a crucial aspect of the methodology, noting that "the empirical evidence suggests that GSD-tests can complement the existing statistical tests for comparing classifiers meaningfully." 

**Reviewer 3sEe** highlights the method's soundness and utility by calling it "extremely useful" and “very significant”. The reviewer further praises the notation as “consistent throughout”. Generally, 3sEe applauds the "very well written" nature of the paper, which provides "all the information needed to fully understand the paper in the main body of text." 

Lastly, **Reviewer n7Tx** calls our work “innovative”, stating that the paper is "clear and easy to follow" with a methodology that "enjoys some desirable mathematical properties", which are “well presented”. The reviewer also appreciates the extensive empirical evidence in favor of our method, stating that it "seems to work well on the selected benchmarks.”

Generally, neither the theoretical soundness nor the practical relevance ("excellent" and “extremely useful”: Reviewer 3sEe) was questioned. The reviews rather focused on didactical aspects. As described in detail in the individual responses, we are most confident that we can properly address all issues in the camera ready version. The additional page will mainly be used to follow the reviewers’ suggestions to add some further didactical material illustrating and demonstrating our approach. In particular, we will include a paragraph with additional recommendations for the end-user (**Reviewer 1fNM**), add an example elaborating the differences between GSD-front and Pareto-front (**Reviewer 3sEe**), as well as extend the discussion of related work and additionally provide a more intuitive description of the central test statistic (**Reviewer n7Tx**).

**Attached pdf page**

Please note that the pdf page attached to this global comment is specific to Q3 of Reviewer 1fNM. The context of the contents of the page is explained in our corresponding answer.

Meta-review: The authors propose a new method for benchmarking classifiers on multiple criteria using the GSD-front. The paper provides an in depth treatment with theoretical results and statistical tests, backed up by extensive empirical experiments. On the down side, the paper is lacking in providing the reader with an intuitive understanding of the GSD-front, which the authors have promised to address via a didactic example. There is also a focus on benchmarking using large collections of datasets, which could be useful for academic comparisons of classifiers, but less useful for practitioners who are interested in selecting good classifiers for a particular problem.  Overall, all reviewers recommended accepting the paper.
Decision: Accept (spotlight)