Statistical Multicriteria Benchmarking via the GSD-Front

Authors: christoph  jansen
georg  schollmeyer
julian  rodemann
hannah  blocher
thomas  augustin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 14:45:28

Abstract: Given the vast number of classifiers that have been (and continue to be) proposed, reliable methods for comparing them are becoming increasingly important. The desire for reliability is broken down into three main aspects: (1) Comparisons should allow for different quality metrics simultaneously. (2) Comparisons should take into account the statistical uncertainty induced by the choice of benchmark suite. (3) The robustness of the comparisons under small deviations in the underlying assumptions should be verifiable. To address (1), we propose to compare classifiers using a generalized stochastic dominance ordering (GSD) and present the GSD-front as an information-efficient alternative to the classical Pareto-front. For (2), we propose a consistent statistical estimator for the GSD-front and construct a statistical test for whether a (potentially new) classifier lies in the GSD-front of a set of state-of-the-art classifiers. For (3), we relax our proposed test using techniques from robust statistics and imprecise probabilities. We illustrate our concepts on the benchmark suite PMLB and on the platform OpenML.

Main paper: 
 introduction: 
 The comparison of classifiers in machine learning is usually carried out using quality metrics ϕ : C × D → [0, 1], i.e., bounded functions assigning a real number to every pair (C, D) of classifier and data set from a suitable domain C × D, where, by construction, higher numbers indicate better quality. However, in many applications, the choice of a unique quality metric used for the comparison is not self-evident. Instead, competing quality metrics are available, each of which can be wellmotivated but may lead to a different ranking of the analyzed classifiers. One attempt to safeguard against this effect is to use multidimensional quality metrics: instead of a single metric, one chooses a set of metrics Φ := (ϕ 1 , . . . , ϕ n ) : C × D → [0, 1] n that -taken together -provide a balanced Regardless of the motivation for considering multidimensional quality metrics, their interpretative advantage naturally comes at a price: Without further assumptions, classifiers will often be incomparable, as the quality metrics in the different dimensions contradict each other in their ranking. 2Already on one data set, a multidimensional quality metric only induces a (natural yet potentially incomplete) preorder: a classifier is rated at least as good as a competitor if (and only if) it receives at least the same metric value in each dimension. The problem of incomparability becomes even more severe for multiple data sets (as considered here). In this case, one of the following analysis paths is often chosen: (I) An expected weighted sum (for example, weighted by importance) of the individual quality metrics is considered and the problem is then analyzed on this new pooled quantity. 3 The problem is analyzed based on the Pareto-front par(Φ), i.e., the set of all classifiers that are not component-wise (strictly) dominated by any competitor, whose definition followed by an illustrative example are included for reference. Here, it holds that Φ(C 2 , D i ) ⋗ Φ(C 1 , D i ) for all i = 1, 2, 3, 4, i.e., C 1 is component-wise (strictly) dominated by C 2 . Classifiers C 2 and C 3 are not component-wise (strictly) dominated. Thus, the Pareto-front is given by par(Φ) = {C 2 , C 3 }.Both approaches are extreme in a certain sense: (I) reduces the multidimensional information structure of the problem to one single real-valued score. Any selection of classifier based on this score will heavily depend on the choice of the weights in the sum score and, therefore, becomes dubious once this choice is not perfectly justified. This seems even more severe for problems where some of the involved quality metrics might only allow for an ordinal interpretation, e.g., feature sparseness as a proxy for interpretability [75] , risk levels in the EU AI act [50] or other regulatory frameworks [73] , robustness (see experiments in Section 5.2) or runtime levels (Section 5.1). Opposed to this, (II) seems to be very conservative: By considering classifiers that are in the Pareto-front par(Φ), one (potentially) completely ignores both information encoded in the cardinally interpretable dimensions and information about the distribution of the data sets. As a trade-off between these two extremes, which utilizes the complete available information but avoids the choice of weights, it has recently been proposed to compare classifiers under multidimensional quality metrics using generalized stochastic dominance (GSD) [45] . The rough idea of this approach is to first embed the range of the multivariate performance measure in a special type of relational structure, a so-called preference system, which then allows for also formalizing the entire information originating from the cardinal dimensions of the quality metric. A classifier is then judged at least as good as a competitor (similar to classic stochastic dominance), if its expected utility is at least as high with respect to every utility function representing (both the ordinal and the cardinal parts of) the preference system (also see Definition 5) . Although GSD also induces only a preorder, the set of not strictly dominated classifiers will generally be considerably smaller than under the classical Pareto analysis. Furthermore, it avoids potentially difficult to justify assumptions about the weighting of the different quality metrics. Therefore, working with the GSD-front, as introduced below, will prove to be a very promising analysis option; it combines the advantages of the conservative Pareto analysis with those of the liberal comparison of weighted sums.1.1 Our contribution GSD-Front: We introduce the concept of the GSD-front (see, after some preparations, Definition 6) and characterize it in Theorem 2 as more discriminative than the Pareto-front. In this sense, the GSD-front is an information-efficient way to handle the multiplicity/implicit multidimensionality of quality criteria, powerfully exploiting their ordinal and quantitative components.Proper handling of statistical uncertainty; estimating and testing: Since typically the available data sets are just a sample of the corresponding universe, empirical counterparts of the major concepts are needed to do justice to the underlying statistical uncertainty. In particular, we give a sound inference framework: Firstly, we propose a set-valued estimator for the GSD-front and provide sufficient conditions for its consistency (see Theorem 1 and Remark 3). Secondly, we develop static and dynamic statistical permutation-tests if a classifier is in the GSD-front and prove their level-αvalidity and their consistency (see Theorem 3) .Robustification: Additionally, we recognize the fact that the underlying assumption of identically and independently distributed (i.i.d.) sampling is questionable in many benchmarking studies. Thus, in Section 4.2 we quantify how robust the test decisions are under such deviations.Experiments with benchmark suites and implementation: We illustrate the concepts and corroborate their relevance with experiments run over two benchmark suites (PMLB and OpenML, see Section 5), based on an implementation that is freely available and easily adaptable to comparable problems. 4 . We consider experiments with mixed-scaled (ordinal and cardinal) multidimensional quality metrics, also incorporating (potentially) ordinal criteria.Benchmarks are the foundations of applied machine learning research [27, 90, 78, 65, 91] . Specifically, benchmarking classifiers over multiple data sets is a much-studied problem in machine learning, as it enables practitioners to make informed choices about which methods to consider for a given data set. Furthermore, also proposals for novel classifiers must often first demonstrate their potential for improvement in benchmark studies. Examples include [58, 40, 31, 57, 12] . In recent years, in recognition of the fact that the benchmark suite under consideration is only a sample of data sets, especially focusing on statistically significant differences between classifiers has received great interest (see, e.g., [24, 35, 34, 19, 45] or, e.g., [9, 22, 8] for Bayesian variants). An R implementation of some of these tests is described in [15] , whereas use-cases in the context of time series and neural networks for regression are discussed in [44, 36] . The diversity and the associated problem of selecting quality metrics (e.g., [51] ) is currently attracting a great deal of interest (e.g., [89] ). Consequently, finding ways for comparing classifiers in terms of multidimensional quality metrics is intensively studied, ranging from multidimensional interpretability measures (e.g., [59] ) over classical Pareto-analyses (e.g., [31] ) to embeddings in the theory of data depth (e.g., [13, 71] ). While utilizing variants of stochastic dominance in statistics is quite common (e.g., [56, 61, 6, 76, 67] ), the same seems not to hold for machine learning. Exceptions include [23] in an optimization context, [47, 48] , who investigate special types of stochastic orders, and [45] , utilizing already GSD-relations for classifier comparisons without the GSD-front. Finally, relying on imprecise probabilities (e.g., [85, 86, 3] ) to robustify statistical hypotheses follows the tradition of [66, 42, 41, 2] , see also, e.g., [5, 25, 4, 60, 48] . For application to Bayesian networks, see, e.g, [55, 14, 54] , and [81, 69, 18, 80, 1, 70, 52, 30, 16, 17] , among others, for robustified machine learning in this spirit. 
decision-theoretic preliminaries: 
 The relevant basic concepts in order theory are collected in Appendix A.1. Based on these we can make the following definition, originating from the decision-theoretic context discussed in [46] . Definition 2. Let A be a non-empty set, R 1 ⊆ A × A a preorder on A, andR 2 ⊆ R 1 × R 1 a preorder on R 1 . The triplet A = [A, R 1 , R 2 ]is then called a preference system on A. The preference systemA ′ = [A ′ , R ′ 1 , R ′ 2 ] is called subsystem of A if A ′ ⊆ A, R ′ 1 ⊆ R 1 , and R ′ 2 ⊆ R 2 .In our context, R 1 formalizes the ordinal information, i.e., the information about the ranking of the objects in A, whereas R 2 describes the cardinal information, i.e., the information about the intensity of certain rankings. To ensure that R 1 and R 2 are compatible, we use a consistency criterion relying on the idea of simultaneous representability of both relations. For this, for a preorder R, we denote by I R its indifference and by P R its strict part (see A.1). Definition 3.The preference system A = [A, R 1 , R 2 ] is consistent if there exists a representation u : A → R such that for all a, b, c, d ∈ A we have: i) (a, b) ∈ R 1 ⇒ u(a) ≥ u(b) with equality iff (a, b) ∈ I R1 ii) ((a, b), (c, d)) ∈ R 2 ⇒ u(a) -u(b) ≥ u(c) -u(d) with equality iff ((a, b), (c, d)) ∈ I R2The set of all representations of A is denoted by U A .Finally, we need to recall the concept of generalized stochastic dominance (GSD) (see, e.g., [48] ), which is crucial for the concepts presented in this paper: For a probability space (Ω, S, π) and a consistent preference system A, we define by F (A,π) the set of all X ∈ A Ω such that u • X ∈ L 1 (Ω, S, π) for all u ∈ U A . We then can define the GSD-preorder on F (A,π) as follows.Definition 4. Let A = [A, R 1 , R 2 ] be consistent. For X, Y ∈ F (A,π) , say X (A, π)-dominates Y if E π (u • X) ≥ E π (u • Y ) for all u ∈ U A . Denote the induced GSD-preorder on F (A,π) by R (A,π) .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The limitations of the work are discussed in Sections 4.2 and 6 of the main paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
gsd for classifier comparison: 
 We return to the initial problem: Assume we are given a finite set C of classifiers, an arbitrary set D of data sets and n quality metrics ϕ 1 , . . . , ϕ n :C × D → [0, 1], combined to the multidimensional quality metric Φ := (ϕ 1 , . . . , ϕ n ) : C × D → [0, 1] n .As we also want to allow ordinal quality metrics, we assume that, for 0 ≤ z ≤ n, the metrics ϕ 1 , . . . , ϕ z are of cardinal scale (differences may be interpreted), while the remaining ones are purely ordinal (differences are meaningless apart from sign). We embed the range Φ(C × D) of Φ in the following preference system:EQUATIONR * 1 = (x, y) : x j ≥ y j ∀j ≤ n , and R * 2 = ((x, y), (x ′ , y ′ )) :x j -y j ≥ x ′ j -y ′ j ∀j ≤ z x j ≥ x ′ j ≥ y ′ j ≥ y j ∀j > z . R * 1 is the usual component-wise ≥-relation. For R * 2 ,one pair of consequences is preferred to another if, in the ordinal dimensions, the exchange associated with the first pair is not a deterioration to the exchange associated with the second pair and, in addition, there is component-wise dominance of the differences of the cardinal dimensions. In order to transfer the GSD-relation from Definition 4 to the case of comparing classifiers under multidimensional performance metrics, we interpret the data sets in D as realizations of a random variable T : Ω → D on some probability space (Ω, S, π). We then associate each classifier C ∈ C with the random variable Φ C := Φ(C, T (•)) on Ω and compare classifiers by comparing the associated random variables by means of GSD. Definition 5. Denote by P Φ the preference system obtained by restricting P to Φ(C × D). Further, let C be such that{Φ C : C ∈ C} ⊆ F (PΦ,π) . For C, C ′ ∈ C, say that C dominates C ′ , abbreviated with C ≿ C ′ , whenever (Φ C , Φ C ′ ) ∈ R (PΦ,π) .In the application situation, instead of the true GSD-order ≿ among classifiers, we will often have to get along with its empirical analogue, i.e., the GSD-relation where a sample of data sets is treated like the underlying population and the true probability measure is replaced by the corresponding empirical ones. More precisely, we assume that we have sampled i.i.d. copies T 1 , . . . , T s of T and then define the set Z s := Φ(C, T i ) : i ≤ s ∧ C ∈ C , of (random) observations under the different classifiers. We then use W to denote the (random) subsystem of P that arises when P is restricted to the (random) set Z s . For C, C ′ ∈ C we define the random variabled s (C, C ′ ) := inf u∈U W z∈Zs u(z)(π C ({z}) -πC ′ ({z})), where, for M ⊆ [0, 1] n , we set πC (M ) := 1 s | i : i ≤ s ∧ Φ(C, T i ) ∈ M |.For a concrete sample associated to ω 0 ∈ Ω, we then say that C empirically GSD-dominatesC ′ , if d s (C, C ′ )(ω 0 ) ≥ 0.Intuitively, d s can thus be used to check whether the classifier C empirically dominates the classifier C ′ with respect to GSD in the samples at hand (i.e., in the benchmark suite under investigation).Based on these concepts, we can now define the sets of (empirically) GSD-undominated classifiers. Definition 6. Let C be such that{Φ C : C ∈ C} ⊆ F (PΦ,π) . Let denote T 1 , . . . , T s i.i.d. copies of T . i) The GSD-front is the set gsd(C) := C ∈ C : ∄C ′ ∈ C s.t. C ′ ≻ C ,where ≻ denotes the strict part of ≿.ii) Let ε ∈ [0, 1]. The ε-empirical GSD-front is the (random) subset of C defined by egsd ε s (C) = C : ∄C ′ ∈ C s.t. d s (C ′ , C) ≥ -ε d s (C, C ′ ) < 0 .Remark 1. egsd 0 s (C) is always non-empty. In contrast, egsd ε s (C) may very well be empty if ε > 0. Note that choosing values of ε > 0 is intended to make egsd ε s (C) less prone to sampling noise. Remark 2. Some words on the semantics of the GSD-front: From a decision-theoretic point of view, classifier C strictly GSD-dominates classifier C ′ iff C has at least as high expected utility as C regarding any compatible utility representation of all the metrics considered, and stricly higher for at least one such utility. The GSD-front then simply collects all classifiers from C which are not strictly GSD-dominated by any competitor, i.e., which potentially can be optimal in expectation. Example 2. Consider again the situation of Example 1 and recall that par(Φ) = {C 2 , C 3 } leaves C 2 and C 3 incomparable. However, if considering only the distribution of the (multivariate) performance of the classifiers (while assuming a uniform distribution over D), C 3 is clearly dominating C 2 w.r.t. GSD: Matching dataset D i with dataset D 5-i creates a (strict) pointwise dominance of C 3 over C 2 (where the strict dominance is due to D 1 and D 4 ). Thus, gsd(C)= {C 3 } ⊊ par(Φ) = {C 2 , C 3 }.The following two theorems show that the ε-empirical GSD-front fulfills two very natural requirements: First, under some regularity conditions, it is a consistent statistical estimator for the true GSD-front (Theorem 1). This is important because in practical benchmarking we almost never have access to the GSD-front of the whole population, i.e., the benchmarking results on all possible datasets from a specific problem class D. Second, it is ensured that neither the ε-empirical nor the true GSD-front can ever become larger than the respective Pareto-front, irrespective of the choice of ε (Theorem 2). This is important as it guarantees our analysis does never conflict with, but is potentially more information-efficient than a Pareto-type analysis. Proofs are given in B. 1 (C) = gsd(C) = 1,where set convergence is defined via the trivial metric. Remark 3. The assumption of a finite VC dimension is only necessary to ensure that the ε-empirical GSD front does not become too large. In particular, the following does hold without this assumption:π ω ∈ Ω : lim s→∞ egsd ε(s) s (C) ⊇ gsd(C) = 1.Thus, the ε-empirical GSD-front almost surely converges to a superset of the true GSD-front. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Information on computer resources is provided in the GitHub repository referenced to in Footnote 3 of the main paper.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
statistical testing: 
 We saw the ε-empirical GSD-front can be a consistent statistical estimator and that both the empirical and the true GSD-front are compatible with the Pareto-front. We now address statistical testing.From now on, we make the (technical) assumption that the order ≿ among the classifiers from C is additionally antisymmetric, transforming it from a preorder into a partial order. 6 Equipped with this assumption, we want to address the question how to statistically test if a given classifier C ∈ C is an element of the true GSD-front gsd(C). To achieve this, we formulate the question of actual interest as the alternative hypothesis of the test, i.e., we obtain the hypothesis pair: H 0 : C / ∈(C ′ , C) of d s (C ′ , C) as a test statistic. The strength of regularization of the test statistic is there controlled by a parameter δ ∈ [0, 1], whose increase reduces the number of representation functions over which the infimum in the test statistic is formed, while equally attenuating all quality metrics. 7 Due to space limitations, we omit to recall an exact description of the testing scheme in the main text and instead refer to Appendix A.2.The idea is then to replace the global test for (H 0 , ¬H 0 ) with c := |C| -1 tests of hypotheses (H C ′ 0 , ¬H C ′ 0 ) and to reject the null hypothesis at significance level α if all tests reject their individual null hypotheses H C ′ 0 at the same significance level α. Call this the static GSD-test. Clearly, this test tends to be conservative, as it ignores potential correlations of the test statistics for different pairs of classifiers. Moreover, a slightly modified test in the context of the GSD-front is directly derivable: If one is rather interested in identifying the maximal subset S max of C for which C significantly lies in the GSD-front, i.e., in testing HS 0 : C / ∈ gsd(S) vs. HS 1 : C ∈ gsd(S) for all S ⊆ C with C ∈ S simultaneously, the following alternative test is a statistically valid level-α test: First, perform all individual tests for (H C ′ 0 , ¬H C ′ 0 ) with level α c . Then identify S max as the set of all classifiers from C for which the individual hypotheses are rejected. The (random) alternative hypothesis HSmax 1 : C ∈ gsd(S max ) is then statistically valid in the sense of being false only with a probability bounded by α. Call this the dynamic GSD-test. We have the following theorem, demonstrating that the proposed tests are indeed reasonable statistical tests (see B.3 for the proof).Theorem 3. Let the assumptions of Theorem 1 hold. Then, both the static and dynamic GSD-test are valid level-α tests. Additionally, both tests are consistent in the sense that under the corresponding alternative hypothesis, i.e., H 1 : C ∈ gsd(C) resp. H1 : ∃S ⊆ C : C ∈ S, |S| ≥ 2, C ∈ gsd(S), the probability of rejecting the corresponding null hypothesis converges to 1 as s → ∞.We argue that meaningful benchmark studies should abstain from treating the sample of data sets in the suite as a complete survey. That is, benchmark analyses should aim at statements about a well-defined population and regard the benchmark suite as a non-degenerate sample thereof. A major practical problem in this context is that often little is known about the inclusion criteria for data sets or test problems in the respective benchmark suite (see, e.g., the discussions in [83, 53, 37] ). For instance, the popular platform OpenML [82] allows users to upload benchmark results for machine learning models with varying hyperparameters, harming representativity, see Section 5.1 and Appendix C.1. The absence of methods to randomly sample from the set of all problems or data sets is identified as an unsolved issue in [57, Section 2] . This calls the common i.i.d. sampling assumption into question, which our (and most other) tests are based upon, and raises the issue as to what extent statistically significant results depend on this assumption. We now address precisely this question.In [48] it was shown how the binary tests on the hypothesis pairs (H C ′ 0 , ¬H C ′ 0 ) discussed in Section 4.1 can be checked for robustness against deviations from the underlying i.i.d.-assumption. The idea here is to deliberately perturb the empirical distributions of the performances for the different classifiers and to analyze the permutation test used under the most extreme yet compatible worst-case. The perturbation of the empirical distribution is carried out here using a γ-contamination model (see, e.g., [85, p. 147] ), which is widely used in robust statistics. We now want to adapt a similar robustness check for the global hypothesis pair (H 0 , ¬H 0 ) discussed here. For this, suppose we have a sample T 1 , . . . , T s of data sets (i.e., the benchmark suite). We further assume that k ≤ s of these variables (where it is not known which ones) are not sampled i.i.d., but come from an arbitrary distribution about which nothing else is known. We then know, for every fixed C ∈ C, that its associated true empirical measure πtrue C based on the true (uncontaminated) sample would have to be contained inEQUATIONwhere πcont C denotes the empirical measure based on the contaminated sample T 1 , . . . , T s . Note that M C is by definition a γ-contamination model with central distribution πcont C and contamination degree γ := k s . In this setting, [48] show that to ensure that their permutation tests used for hypothesis pairs (H C ′ 0 , ¬H C ′ 0 ) only advise rejection of the null hypothesis if this is justifiable for any empirical distribution compatible with the contaminated sample, i.e., for every combination of measures(π 1 , π 2 ) ∈ M C × M C ′ ,one has to compare the most pessimistic value of the test statistic for the concrete sample at hand with the most optimistic value of the test in each of the resamples. Moreover, they show that the (approximate) observed p-values for a concrete contaminated sample T 1 (ω 0 ), . . . , T s (ω 0 ) ∈ D associated with ω 0 ∈ Ω of this robustified test can be expressed by a function in the number of contaminations k, given byf (C ′ ,C) (k) := 1 -1 N • I∈I N 1 d δ I -d δ s (C ′ ,C)(ω0)> 2k (s-k),where N denotes the number of resamples, I N is the corresponding set of resamples, and d δ I is the test statistic evaluated for the resample associated to I. Due to space limitations, we omit an exact description of the robustness check for the test on the hypothesis pairs (H C ′ 0 , ¬H C ′ 0 ) as well as a derivation of the function f (C ′ ,C) in the main text and instead refer to Appendix A.3. Similar as shown in Section 4.1, it is straightforward to calculate an (approximate) observed p-value for the static GSD-test for (H 0 , ¬H 0 ): We calculate the maximal observed p-value among allC ′ ∈ C \{C}, i.e. set F C (k) := max f (C ′ ,C) (k) : C ′ ∈ C \{C} .The robustified static GSD-test for the degree of contamination k can be carried out as follows: Calculate F C (k) and rejectH 0 if F C (k) ≤ α.This indeed gives us a valid level-α-test for the desired global hypothesis H 0 : C / ∈ gsd(C) under the additional freedom that up to k of the variables in the sample might be contaminated. Note, however, that also this test tends to be conservative as both performing the individual tests at level α as well as the adapted resampling scheme of the permutation test are worst-case analyses. Finally, also the robustified dynamic GSD-test can be obtained straightforwardly: Under up to k contaminations, the (random) alternative hypothesis HSmax1 : C ∈ gsd(S max ) is statistically valid with level α if all individual robustified tests reject H C ′ 0 at level α c , i.e., if F C (k) ≤ α c .We end the section with a short comment on computation: The test statistics for the permutation test and the robustified variant can be calculated using linear programming. We are guided here by the linear programs proposed in [48, Propositions 4 and 5] . There are two computational bottlenecks in the actual evaluation: (1) the creation and storage of the constraint matrices of the linear programs and (2) the repeated need to solve large linear programs. An efficient, well-commented implementation that can be quickly transferred to similar applications is made available on GitHub (see Footnote 4). 
benchmarking experiments: 
 We demonstrate our concepts on two well-established benchmark suites: OpenML [82, 11] and PMLB [64] . While for PMLB we compare classifiers w.r.t. the latent quality metric robust accuracy (see the first motivation in Section 1), for OpenML we use a multidimensional metric that includes accuracy and computation time as unidimensional metrics (see the second motivation in Section 1). The analysis of PMLB is kept short in the main text and detailed in Appendix C. Since the metrics in both applications are composed of one continuous and two (finitely) discrete metrics, we have (see B.4): Corollary 1. In both applications, the ε-empirical GSD-front is a consistent estimator for the true GSD-front (provided ε is chosen as in Theorem 1).We select 80 binary classification datasets (according to criteria detailed in Appendix C.1) from OpenML [82] to compare the performance of Support Vector Machine (SVM) with Random Forest (RF), Decision Tree (CART), Logistic Regression (LR), Generalized Linear Model with Elastic net (GLMNet), Extreme Gradient Boosting (xGBoost), and k-Nearest Neighbors (kNN). 8 Our multidimensional quality metric is composed of predictive accuracy, computation time on the test data, and computation time on the training data. Since the computation time depends strongly on the used computing environment (e.g. number of cores or free memory), we discretize the timerelated metrics and treat them as ordinal. Accuracy is not affected by this and is therefore treated as cardinal. For details, see Appendix C.1. To gain a purely descriptive impression, we computed the empirical GSD relation. For this, we calculated 2 in Appendix C.1). We see that CART (strictly) empirically GSD-dominates xGBoost, SVM, LR, and GLMNet. All other classifiers are pairwise incomparable. Three classifiers are not strictly empirically GSD-dominated by any other, namely RF, CART, and kNN. Thus, the 0-empirical GSD-front is formed by these. While at first glance this result might seem rather unexpected, a closer look on the performance evaluations provided by OpenML indeed confirms the dominance structure found, see Appendix C.1 for details.d 80 (C, C ′ ) for C ̸ = C ′ ∈ C := {SVM, RF, CART, LR, GLMNet, xGBoost, kNN} (see Hasse graph in FigureTo move to reliable inferential statements that take into account the statistical uncertainty, we exemplarily test (at level α = 0.05) if SVM significantly lies in the GSD-front of some subset of C. As described in Section 4.1, we therefore perform six pairwise permutation tests for the hypothesis pairs (H C ′ 0 , ¬H C ′ 0 ) (where C := SVM and C ′ ∈ C \{SVM}) at level α in case of the static GSD-test or at level α 6 in case of the dynamic GSD-test. 9 That is, we test six auxiliary null hypotheses each stating that SVM is GSD-dominated by kNN, xGBoost, RF, CART, LR, and GLMNet, respectively. of SVM against GLMNet, kNN, RF and xGBoost cease to be significant from a contamination of (approximately) 7, 8, 11, and 11 of 80 data sets, respectively. That is, the results on up to 7, 8, 11, and 11 datasets could be arbitrarily redistributed, while maintaining significance of rejection. Since the significance of the dynamic GSD-test's decision depends on all pairwise tests being significant at level 0.5 6 , we can conclude that SVM would still have been significantly in the GSD-front of {SVM, kNN, xGBoost, RF, GLMNet}, even if 7 out of 80 data sets had been contaminated. Summing up, our proposed testing scheme not only allowed for meaningful statistical benchmarking of SVM versus competitors regarding accuracy, test time, and train time; it also enabled us to quantify as to what degree our conclusions remained stable under contamination of the benchmark suite.The results highlight the advantages of the GSD-front over existing approaches. Applying first-order stochastic dominance (a special case of GSD where R * 2 is the trivial preorder) on the same set-up, yields that no classifier is significantly larger than (or incomparable to) any other classifier, based on a 5% significance level. This illustrates that the GSD-approach accounts for accuracy being a cardinal measure. In contrast, the Pareto-front here contains all considered classifiers. Thus, the Pareto front is much less informative than the GSD-front, which is also reflected in Theorem 2. Unlike the Pareto-front, the GSD-front is based on the distribution of the multidimensional quality metric and not only on the pairwise comparisons, and can use this knowledge to define the front. Thus, the GSD front is a balance between the conservative Pareto analysis and the liberal weighted sum comparison. Finally, we want to compare our method with an approach based on extending the test for single quality metrics proposed in [24] to the multiple metric setting. We therefore perform all possible single-metric tests as in [24] and define the marginal front as those classifiers that are not statistically significantly worse than another classifier on all metrics. However, this procedure can not be used to define a hypothesis test. Therefore, only a comparison with the empirical GSD-front is meaningful. For OpenML, this marginal front consists of all classifiers and is less exploratory than the empirical GSD-front. More details on the results of these other approaches and how these compare to the GSD front can be found in Appendix C.1.We select 62 datasets from the Penn Machine Learning Benchmark (PMLB) suite [64] according to criteria explained in Appendix C.2. The following analysis shall exemplify how our proposed statistical tests can aid researchers in benchmarking newly developed classifiers against state-of-theart ones. To this end, we compare a recently proposed classifier based on compressed rule ensembles of trees (CRE) [62] w.r.t. robust accuracy against five well-established classifiers, namely CART, RF, SVM with radial kernel, kNN and GLMNet. We operationalize the latent quality criterion of robust accuracy through i) classical accuracy (metric), ii) robustness of accuracy w.r.t. noisy features (ordinal), and iii) robustness of accuracy w.r.t. noisy classes (ordinal). Computation of i) is straightforward; in order to retrieve ii) and iii), we follow [92, 93] by randomly perturbing a share (here: 20 %) of both classes and features and computing the accuracy subsequently, as detailed in Appendix C.2. Since there exist competing definitions of robustness [43, 10, 72] and due to the share's arbitrary size, we treat ii) and iii) as ordinal and discretize the perturbated accuracy in the same way as for the runtimes in the openML experiments. Detailed results and visualization thereof can be found in Appendix C.2. In a nutshell, we find no evidence to reject the null of both the static and the dynamic GSD-test at significance level α = 0.05. In particular, we do not reject any of the pairwise auxiliary tests for hypothesis pairs (H C ′ 0 , ¬H C ′ 0 ) with C := CRE and C ′ ∈ C \ {CRE}) for neither α nor α 5 . Our analysis hence concludes that we cannot rule out at significance level α = 0.05 that the newly proposed classifier CRE is dominated by the five state-of-the-art classifiers w.r.t. all compatible utility representation of the latent criterion robust accuracy.We end the section with a few brief general notes for end-users of our benchmark methodology. This should make it easy to decide whether a GSD-based analysis is appropriate in a given use-case. 1. GSD-based studies do not primarily aim to identify the best algorithm for a given benchmark suite. Often, the GSD front contains more than one element. They are rather intended for checking whether a newly proposed classifier for a certain problem class can potentially improve on the state-of-the-art classifiers, or whether it disqualifies itself from the outset. 2. GSD-based studies allow statements with inferential guarantees by providing appropriate statistical tests: Assuming an i.i.d. benchmark suite, a judgment about an algorithm represents a statement about an underlying population and not just this specific suite. 3. GSD-based studies enable the robustness of the results to be quantified under the deviation from the i.i.d. assumption: It can be checked which share of the benchmark suite may be contaminated without affecting the obtained inferential statements. 4. GSD-based studies allow algorithms to be compared w.r.t. multiple metrics simultaneously.They enable the full exploitation of the information contained in differently scaled metrics. 
concluding remarks: 
 Summary: We introduced the GSD-front for multicriteria comparisons of classifiers, gave conditions for its consistent estimability and proposed a statistical test for checking if a classifier belongs to it. We illustrated our concepts using two well-established benchmark suites. The results came with threefold reliability: They included several quality metrics, representation of statistical uncertainty, and a quantification of robustness under deviations from the assumptions.Limitations and future research: Two specific limitations open promising avenues: 1.) Comparing other types of algorithms: We restricted ourselves to comparing classifiers. However, any situation in which objects are to be compared on the basis of different (potentially differently scaled) metrics over a random selection of instances can be analyzed using these ideas. For instance, applications of our framework to the multicriteria deep learning benchmark suite DAWNBench [21] or the bi-criteria optimization benchmark suite DeepOBS [74] appear straighforward. 2.) Extension to regression-type analysis: Analyses based on the GSD-front do not account for meta properties of the data sets. A straightforward extension to the case of additional covariates for the data sets is to stratify by these for the GSD-comparison. This would allow for a situation-specific GSD-analysis, presumably yielding more informative results.A.1 Basic definitions from order theory In this section we describe in detail the statistical test for the hypothesis pair (H C ′ 0 , ¬H C ′ 0 ) discussed in Section 4.1 and first introduced in [48] . Moreover, we give further details on our proposed extension of this test to the global hypothesis pair (H 0 , ¬H 0 ) in both the static and the dynamic variant.EQUATIONBefore we can describe the test from Section 4.1 in detail, we first need to recall two more definitions. Definition 7. Let A = [A, R 1 , R 2 ] be a preference system. We call A bounded, if there exist a * , a * ∈ A such that (a * , a) ∈ R 1 , and (a, a * ) ∈ R 1 for all a ∈ A, and (a * , a * ) ∈ P R1 . Definition 8. Let A = [A, R 1 , R 2 ] be a consistent and bounded preference system with a * , a * as before. DefineN A := u ∈ U A : u(a * ) = 0 ∧ u(a * ) = 1 . For δ ∈ [0, 1), denote by N δ A the set of all u ∈ N A with u(a) -u(b) ≥ δ ∧ u(c) -u(d) -u(e) + u(f ) ≥ δfor all (a, b) ∈ P R1 and for all ((c, d), (e, f )) ∈ P R2 .We now start by describing an adapted version of the permutation test for the hypothesis pairs (H C ′ 0 , ¬H C ′ 0 ) proposed in [48] . For a concrete realization of the i.i.d.-sample of data sets D 1 := T 1 (ω 0 ), . . . , D s := T s (ω 0 ) ∈ D with s ∈ N associated with ω 0 ∈ Ω, we define the set(C, C ′ ) ω0 = {Φ(C, D i ) : i ≤ s} ∪ {Φ(C ′ , D i ) : i ≤ s} ∪ {0, 1},where 0 is the vector containing n zeros and 1 is the vector containing n ones. Denote by P ω0 the restriction of P to (C, C ′ ) ω0 . It is then easy to verify that P ω0 is a consistent and bounded preference system with a * := 0 and a * := 1. For testing the hypothesis pair (H C ′ 0 , ¬H C ′ 0 ) defined and discussed in Section 4.1 of the main text, we then use the following regularized test statistic for the specific sample induced by ω 0 :d δ s (C ′ , C)(ω 0 ) := inf u∈N µ δ Pω 0 z∈(C,C ′ )ω 0 u(z) • (π ω0 C ′ ({z}) -πω0 C ({z}))with δ ∈ [0, 1] and µ δ := δ • sup{ξ : N ξ Aω ̸ = ∅}, and πω0 C resp. πω0 C ′ are the emiprical probability measures of the performances of C resp. C ′ for the specific sample induced by ω 0 .A.2.2 Testing scheme for (H C ′ 0 , ¬H C ′ 0 ) We denote our samples as follows:x := (x 1 , . . . , x s ) := (Φ(C, D 1 ), . . . , Φ(C, D s )) y := (y 1 , . . . , y s ) := (Φ(C ′ , D 1 ), . . . , Φ(C ′ , D s ))The concrete testing scheme for the permutation test for hypothesis pair (H C ′ 0 , ¬H C ′ 0 ) then looks as follows:Step 1: Take the pooled data sample: w := (w 1 , . . . , w 2s ) := (x 1 , . . . , x s , y 1 , . . . , y s )Step 2: Take all r := 2s s index sets I ⊆ {1, . . . , 2s} of size s. Evaluate d δ s (C ′ , C) for (w i ) i∈I and (w i ) i∈{1,...,n+m}\I instead of x and y. Denote the evaluations by d δ I .Step 3: Sort all d δ I in increasing order to get d δ (1) , . . . , d δ (r) .Step 4:Reject H C ′ 0 if d δ s (C ′ , C)(ω 0 ) is strictly smaller than d δ (ℓ), with ℓ := ⌊α • r⌋ and α the significance level.Note that, for large 2s s , we can approximate the above resampling scheme by computing d δ I only for a large number N of randomly drawn I. Moreover, note that only the i.i.d. assumption is needed for the above test to be valid.As argued in the Section 4.1 of the main part of the paper, if we want to obtain a valid statistical test at the significance level α ∈ [0, 1] for hypothesis pair (H 0 , ¬H 0 ), we can simply perform all pairwise tests of hypothesis pairs (H C ′ 0 , ¬H C ′ 0 ) at this same significance level α. We can then reject the hypothesis H 0 at level α if we can reject each hypothesis H C ′ 0 at level α or, in other words, ifmin 1 N • I∈I N 1 d δ s (C ′ ,C)(ω0)<d δ I : C ′ ∈ C \ {C} ≥ 1 -α.We call this the static GSD-test.To see that this procedure indeed gives a valid level-α test for the global hypothesis pair (H 0 , ¬H 0 ), observe that -assuming H 0 to be true -the probability of H 0 being rejected equals the probability of all hypothesis H C ′ 0 being rejected simultaneously. The latter probability -still assuming H 0 to be true -is obviously bounded from above by the probability that one specific hypothesis H C * 0 is rejected, which itself is bounded from above by the significance level α by construction.As discussed in the main text and reprinted here again for convenience of the reader, a slightly modified test in the context of the GSD-front is directly derivable: If one is rather interested in identifying the maximal subset S max of C for which C significantly lies in the GSD-front, i.e., in testing the hypothesis pairs ( HS 0 , ¬ HS 0 ) for all S ⊆ C simultaneously, the following alternative test would be a statistically valid level-α test: First, perform all individual tests for (H C ′ 0 , ¬H C ′ 0 ) with level α c . Then identify S max as the set of all classifiers from C for which the individual hypotheses are rejected. The (random) alternative hypothesis HSmax 1 : C ∈ gsd(S max ) is then statistically valid in the sense of being false only with a probability bounded by α. We call this the dynamic GSD-test.To see that this procedure indeed gives a valid level-α test for the (random) hypothesis pair ( HSmax 0 , ¬ HSmax 0 ), observe that -under the null hypothesis -the probability of C lying in the GSDfront of some random subset S of C is bounded from above by the sum of probabilities of C lying in the GSD-front of {C, S}, where summation is over all S ∈ S. As each of these probabilities is bounded from above by α c by construction, the corresponding sum is bounded from above by |S| • α c . Finally, as |S| ≤ c, this gives the desired upper bound of α.Note that the test statistic d δ s (C ′ , C)(ω 0 ) can be computed by solving a linear optimization problem (see [48, Proposition 4] ) and, hence, the test just described is computationally tractable. Moreover, note that in both applications in Section 5 the tests are based on the unregularized statistics d 0 s (C ′ , C), as the regularization performed in [48] aims at reaching a goal which is not primarily relevant for the present paper: The authors there are primarily interested in significantly detecting GSD of one variable over the other. Consequently, their regularization aims at making the test more sensitive for exactly this purpose. In contrast, in our study we are primarily interested in significantly detecting incomparabilities between variables, making the regularization by far less natural.In this section we describe in detail the robustification of the statistical test for the hypothesis pair (H C ′ 0 , ¬H C ′ 0 ) discussed in Section 4.2 and first introduced in [48] . Moreover, we give further details on our proposed extension of this robustified statistical test to the global hypothesis pair (H 0 , ¬H 0 ) in both the static and the dynamic variant.If we assume, as done in Section 4.2, that up to k ≤ s of the observations in our sample T 1 , . . . , T s might be contaminated and, accordingly, follow any arbitrary distribution, then we have to base the permutation test for hypothesis pair (H C ′ 0 , ¬H C ′ 0 ) on a worst-case analysis of between the measures contained in M C and M C ′ defined instead of the true empirical measures of the two samples induced by the classifiers C and C ′ . If again D 1 := T 1 (ω 0 ), . . . , D s := T s (ω 0 ) ∈ D is a concrete (now potentially contaminated) sample associated with ω 0 ∈ Ω, and we again define x and y as in Section A.2, then the observed contamination models of C and C ′ look as follows:M C (ω 0 ) = (1 -k s )π cont,ω0 C + k s µ : µ probability measure , M C ′ (ω 0 ) = (1 -k s )π cont,ω0C ′ + k s µ : µ probability measure .C ′ 0 , ¬H C ′ 0 ) If we set d δ s (C ′ , C)(ω 0 ) := sup π1∈M C ′ (ω0),π2∈M C (ω0) inf u∈N µ δ Pω 0 z∈(C,C ′ )ω 0 u(z) • (π 1 ({z}) -π 2 ({z})) , d δ s (C ′ , C)(ω 0 ) := inf π1∈M C ′ (ω0),π2∈M C (ω0) inf u∈N µ δ Pω 0 z∈(C,C ′ )ω 0 u(z) • (π 1 ({z}) -π 2 ({z})) ,then the concrete testing scheme for the permutation test for hypothesis pair (H C ′ 0 , ¬H C ′ 0 ) under at most k contaminated sample members looks as follows:Step 1: Take the pooled data sample: w := (w 1 , . . . , w 2s ) := (x 1 , . . . , x s , y 1 , . . . , y s )Step 2: Take all r := 2s s index sets I ⊆ {1, . . . , 2s} of size s. Evaluate d δ s (C ′ , C) for (w i ) i∈I and (w i ) i∈{1,...,n+m}\I instead of x and y. Denote the evaluations by d δ I .Step 3: Sort all d δ I in increasing order to get d δ (1) , . . . , d δ (r) .Step 4:Reject H C ′ 0 if d δ s (C ′ , C)(ω 0 ) is strictly smaller than d δ (ℓ), with ℓ := ⌊α • r⌋ and α the significance level.The adapted testing scheme just described gives a valid (yet conservative) level-α-test for the hypothesis pair (H C ′ 0 , ¬H C ′ 0 ) under at most k contaminated sample members. Moreover, it directly follows from the discussions in Part C of the supplementary material to [48] that the (approximate) observed p-value of this test is given byf (C ′ ,C) (k) := 1 -1 N • I∈I N 1 d δ I -d δ s (C ′ ,C)(ω0)> 2k (s-k), where again N denotes the number of resamples, I N is the corresponding set of resamples, and d δ I is the test statistic evaluated for the resample associated to I.As already argued in the main text, it is now easy to calculate an (approximate) observed p-value for our global hypothesis pair (H 0 , ¬H 0 ): We simply calculate the maximal observed p-value among allC ′ ∈ C \ {C}, i.e. set F C (k) := max f (C ′ ,C) (k) : C ′ ∈ C \ {C} .The robustified test for the degree of contamination k can be carried out as follows: Calculate F C (k) and reject H 0 if F C (k) ≤ α, i.e., if the maximal (approximate) p-value of the pairwise tests is still lower or equal than the significance level.The argument that the testing procedure just described indeed produces a valid level-α test of the global hypothesis pair (H 0 , ¬H 0 ) under up to k contaminated data sets in the sample, can be carried out completely analogous as done in Appendix A.2.3.Finally, as discussed in the main text and reprinted here again for convenience of the reader, also the robustified dynamic GSD-test can be obtained in a straightforward manner: Under up to k contaminated data sets in the sample, the (random) alternative hypothesis HSmax1 : C ∈ gsd(S max ) from before is statistically valid with level α if all individual robustified tests reject H C ′ 0 at level α c , i.e., if F C (k) ≤ α c .The argument that the testing procedure just described indeed produces a valid level-α test for the (random) hypothesis pair ( HSmax 0 , ¬ HSmax 0 ) under up to k contaminated data sets in the sample, can be carried out completely analogous as done in Appendix A.2.4.Note that also the robustified test statistic d δ s (C ′ , C)(ω 0 ) can be computed by solving a linear optimization problem (see [48, Proposition 6] ) and, hence, the test just described is computationally tractable.Again, note that the tests in Section 5 are based on the unregularized test statistics with δ = 0. The reason for this is the same as discussed at the end of Appendix A.2.Proof. First, note that for C, C ′ ∈ C, we have that C ≿ C ′ if and only ifD(C, C ′ ) := inf u∈U P Φ (E π (u • Φ C ) -E π (u • Φ C ′ )) ≥ 0.Thus, the GSD-front can equivalently be rewritten asgsd(C) = C ∈ C : ∄C ′ ∈ C s.t. D(C ′ , C) ≥ 0 D(C, C ′ ) < 0 . Now, let ε : N → [0, 1] : s → 1/ 4 √ s.We show that:EQUATIONEQUATIONNote that the proof immediately translates to the more general case of ε(s) ∈ Θ(1/ 4 √ s) as stated in Theorem 1. Denote with Ê the expectation w.r.t. the empirical measure associated with the i.i.d. sample 10 (T 1 , . . . , T s ). For Implication (3), assume that C ∈ gsd(C). Then for every other classifier C ′ there exists an utility function u ∈ U PΦ with E π (u• Φ C ) > E π (u • Φ C ′ ) (Otherwise we would have D(C ′ , C) ≥ 0 and D(C, C ′ ) < 0,where the second statement is due to antisymmetry). For these corresponding utility functions, because of the strong law of large numbers, we would getd s (C ′ , C) ≤ Ê(u • Φ C ′ ) -Ê(u • Φ C ) a.s.-→ c < 0. Since C consists only of finitely many classifiers, egsd ε(s) s (C) will almost surely not contain C asymptotically. Note that for Implication (3) to hold, it is only necessary that ε(s) converges to zero as s goes to infinity. The order of convergency as Θ(1/ 4 √ s) is only needed for Implication (4).For Implication (4) assume that C / ∈ gsd(C). Then there exists a classifier C ′ with D(C ′ , C) ≥ 0 and D(C, C ′ ) < 0. An analog argumentation like above shows that d s (C, C ′ ) converges almost surely to a value smaller than zero. It remains to analyze D(C ′ , C). For this, we have to show that d s (C ′ , C) + ε(s) a.s.-→ c ≥ 0. We utilize uniform convergence: For arbitrary ξ > 0, [84, p. 192 Theorem 5.1] gives usP sup u∈U P Φ E(u • Φ C ) -Ê(u • Φ C ) > ξ ≤ 8 e • 2s h h • exp -ξ 2 * s ,where ξ * = ξ -1/s and h is the VC dimension of I Φ . The same holds for Φ C ′ . The triangle inequality then givesP sup u∈U P Φ Ê(u • Φ C ) -Ê(u • Φ C ′ ) > 2ξ ≤ 8 e • 2s h h • exp -ξ 2 * s .For ε(s) = 1/ 4 (1/s) and s large enough, we have ε * (s) = ε(s) -1/s ≥ ε(s)/2 and therefore P supu∈U P Φ Ê(u • Φ C ) -Ê(u • Φ C ′ ) > 2ε(s) ≤ 8 e • 2s h h • exp -ε * (s) 2 s ≤ 8 e • 2s h h exp -ε(s) 2 s/4 .EQUATION= 8 e • 2s hEQUATIONIf the VC dimension h is finite, the term 8 e•2s h h is polynomially growing in √ s (or s), whereas the term exp {-√ s/64} is exponentially decreasing in √ s (or s). Therefore, the right hand side of Inequality (5) converges to zero, which shows thatsup u∈U P Φ Ê(u • Φ C ) -Ê(u • Φ C ′ ) -ε(s)converges in probability to a value c ≤ 0 or equivalently, that d s (C ′ , C) + ε(s) converges to a value c ≥ 0. Since the right hand side of Inequality (5) converges exponentially in s, the Borel-Cantelli theorem (cf., e.g., [29, p.67ff] ) gives also strong convergency, which completes the proof. Note that it is not necessary to specify ε(s) concretely as 1/ 4 √ s. It would be sufficient to define ε(s) as of the order of Θ( 1 We compute:E π (u • Φ C ′ ) -E π (u • Φ C ) = Ω u(Φ(C ′ , T (ω)))dπ(ω) - Ω u(Φ(C, T (ω)))dπ(ω) = Ω u(Φ(C ′ , T (ω))) -u(Φ(C, T (ω))) >0 for all ω∈Ω, since T (ω)∈D dπ(ω) > 0 This gives E π (u • Φ C ′ ) > E π (u • Φ C ).As u was chosen arbitrarily, this implies that C ′ ≻ C. Hence, by definition of the GSD-front, we have C / ∈ gsd(C).ii) First, note that both postulates are statements involving random sets (i.e., sets dependent on the realizations of the variables T 1 , . . . , T s ). Thus, we have to prove both statements for arbitrary realizations of these variables. So let D 1 := T 1 (ω 0 ), . . . , D s := T s (ω 0 ) ∈ D be an arbitrary realisation. For this concrete realization of the sample, the first statement is immediate, since if there is no C ′ such that d s (C ′ , C)(ω 0 ) ≥ -ε 2 there is also no C ′ such that d s (C ′ , C)(ω 0 ) ≥ -ε 1 (as the latter is harder to satisfy due to ε 1 ≤ ε 2 ).Again for the chosen concrete realization of the sample, the second postulate is an immediate consequence of statement i) from above. As in both situations the realization of the variables was chosen arbitrarily, this implies the statement. □To see that the static test is a valid level-α test for the global hypothesis pair (H 0 , ¬H 0 ), observe that -assuming H 0 to be true -the probability of H 0 being rejected equals the probability of all hypothesis H C ′ 0 being rejected simultaneously. The latter probability -still assuming H 0 to be true -is obviously bounded from above by the probability that one specific hypothesis H C * 0 is rejected, which itself is bounded from above by the significance level α by construction.Furthermore, the reason for the consistency of the static test is the following: First, note that under the assumption of Theorem 1 (because of the finite VC dimension), we have that d s (C ′ , C) converges to D(C ′ , C) in probability for every abrbitrary classifier C ′ ̸ = C. Therefore, for fixed C ′ and under the null hypothesis H C ′ 0 , we have d s (C ′ , C) converges in probability to a value larger than or equal to zero. This implies that under this null hypothesis the implicit critical values of the permutation test become arbirarily close to a values larger than or equal to zero. Now, let C be in the GSD-front. Then, due to antisymmetry of ≿, for every other classifier C ′ , there exists a utility for which the expectation of u • Φ C is larger than the expectation of u • Φ C ′ . Because of the weak law of large numbers, this translates to the empirical expectations with an arbitrarily high probability if only the sample size is large enough. Therefore, all-together, as s converges to infinity, the test rejects the null hypothesis in this situation with arbitrary high probability. Finally, since we have only a finite number of hypothesis of the static test, this also translates to the static test itself. Therefore the static test is indeed a consistent level-α test.To see that also the dynamic test is a valid level-α test for the (random) hypothesis pair ( HSmax 0 , ¬ HSmax 0 ), observe that -under the null hypothesis -the probability of C lying in the GSDfront of some random subset S of C is bounded from above by the sum of probabilities of C lying in the GSD-front of {C, S}, where summation is over all S ∈ S. As each of these probabilities is bounded from above by α c by construction, the corresponding sum is bounded from above by |S| • α c . Finally, as |S| ≤ c, this gives the desired upper bound of α. I Φ := {a : u(a) ≥ c} : c ∈ [0, 1] ∧ u ∈ U PΦis also finite. The claim then follows from Theorem 1.To show the finiteness of the width, assume -wlog -that|S 1 | = g < ∞ and |S 2 | = h < ∞.Assume, for contradiction, that there exists an antichain 12 Q ⊆ Φ(C × D) within the restriction of R * 1 to Φ(C × D) of cardinality strictly greater than g • h. Then there exist x = (x 1 , x 2 , x 3 ), y = (y 1 , y 2 , y 3 ) ∈ Q such that x 2 = y 2 and x 3 = y 3 (as there are only g • h different combinations of the second and the third component). However, since the first component is completely ordered by ≥, this implies either (x, y) or (y, x) (or both) is contained in the restriction of R * 1 to Φ(C × D). This is a contradiction to x and y being elements of the same antichain Q, completing the argument. □This section provides further information on the benchmarking examples in Section 5.This sections gives further insight to the example on the OpenML data analysed in Section 5.1.We start with giving more details on the data set with all the computation settings of the classifier algorithms. Afterwards, we provide more graphics and explanations of the analysis.Overall, we are comparing the performance of Support Vector Machine (SVM) to further 6 classifier algorithms on 80 data sets. The data sets as well as the performance evaluation is given by the OpenML library [82] . 13 The analysis is restricted to binary classification problems. We selected those data sets of OpenML that evaluated the predictive accuracy, train data time computation and test data time computation (both measured in milliseconds) for all of the 7 algorithms. Since the computation times depend on the environment, i.e. the number of cores used or the free memory, we discretized the computation times and considered them as ordinal. Therefore, we divided each computation time into ten categories, where category one contains the 10% highest times, and so on. Moreover, we restricted our analysis on data sets with more than 450 and less than 10000 observations. This gives us in total 80 data sets.The algorithms discussed are:• Support Vector Machine (SVM) algorithm is implemented in the e1071 library [26] • Random Forests (RF) algorithm is implemented in the ranger library [88] ,• Decision Tree (CART) algorithm is implemented via the rpart library [79] ,• Logistic regression (LR) algorithm is implemented via the nnet library [68] ,• eXtreme Gradient Boosting (xGBoost) algorithm is implemented in the xgboost library [20] ,• Elastic net (GLMNet) algorithm is implemented through the glmnet library [32] , and• k-nearest neighbors (kNN) algorithm is implemebted via the kknn library [38] .We started our analysis in Section 5.1 by computing the empirical GSD-front. This gives the Hasse graph 2, where a top-down edge from C to C ′ states that d 80 (C, C ′ ) ≥ 0 holds.In addition to the left of Figures 1 (densities of resampled test statistics) and the right of Figure 1 (effect of contamination on p-values) in the main paper, we include the cumulative distribution functions (CDFs) in Figure 3 . Since we do not include the values of the observed test statistics here, the differences in distributions are visible to a greater extent. We observe the resampled test statistics' distributions for SVM vs. xGBoost and GLMNet to be left-shifted compared to SVM vs. CART, xGBoost, and LR. A visual analysis of the test decision, however, is not possible in the absence of the observed test statistics. This is why we include their values in the caption of Figure 3 .12 An antichain of a preordered set (M, R) is a subset A ⊆ M such that for all m1, m2 ∈ A it holds (m1, m2) / ∈ R and (m2, m1) / ∈ R. 13 Last OpenML access: 24/10/2024 This is also visible in Figure 1 in the main paper, albeit less clearly.This section provides the detailed computation of the state-of-the-art approaches and the comparison with the GSD approach. Here, we go step by step through all the methods touched in Section C.1.First-order stochastic dominance Analogously to the GSD-front, one can define the front based on (multivariate) first-order stochastic dominance (see, e.g., [77] ). Note that classical first-order stochastic dominance is a special case of our generalized stochastic dominance (GSD) in the case that all quality metrics are (treated as) of ordinal scale of measurement. Given the test logic followed by, for example, [6] , for the OpenML data it turns out that no classifier is significantly stochastically larger than (or incomparable to) any other classifier (based on a significance level of 5 %). Compared to the results we obtain from our GSD-front analysis, this indiscriminative result is much less informative.Pareto-front Both for the PMLB benchmark suite and the OpenML setup the Pareto-front contains all considered classifiers and is therefore not very informative. This shows the advantage of our approach because our approach is generally more informative (see Theorem 2). In particular, by using generalized stochastic dominance one refrains from solely relying on pointwise comparisons of classifiers over datasets. Instead one only looks at the distribution of the multidimensional quality metric. Beyond this, compared to both a Pareto analysis and a classical first order stochastic dominance analysis (see above), the GSD approach does justice to the fact that the dimension accuracy is cardinal and, at the same time, the fact that the other dimensions are of ordinal scale of measurement. Additionally, compared to an approach that only looks at the marginal distributions of every single quality metric separately, the GSD approach takes also the dependence structure between the different quality metrics into account. This is of particular interest if one has different performance dimensions that are anticorrelated.Weighted sum approach The GSD-based approach has advantages over an approach of weighted summation of the various quality metrics especially when it is not clear how specifically the weights are to be chosen. Specifically, each weighting leads to a total ordering among the classifiers under consideration. A clear best classifier can, therefore, be identified for each weighting. Thus, different weightings generally lead to different best classifiers. As a consequence, if one chooses a specific weighting, one should really be convinced that domain knowledge thoroughly justifies it, as even small changes in the weighting can completely change the resulting ranking. In contrast, the GSD-based approach can be used if no weighting of the involved metrics is available, but still more information (e.g., from the cardinal metrics) is available than required for a Pareto-type analysis. In summary, we emphasize that our method and the weighted summation should be used under different conditions and, therefore, complement rather than compete with each other.Marginal-front A highly popular testing scheme for benchmark analysis is the one proposed by [24] . We compare our approach against using a marginal front that directly results from following this scheme. This marginal front is defined as a function of (a) statistical test(s), i.e. classifiers are in it depending on the test results. We emphasize that this front is not directly comparable to the GSD-front, since the GSD-front is a theoretical object (like the Pareto front) that can be used to formulate hypotheses that can then be tested by statistical tests like the ones proposed in this paper. Thus, we compare the marginal front to the empirical GSD front as reported in Fig. 1 of the paper for the application on OpenML.We run multiple single-objective evaluations and include in the marginal-front the classifiers that are not statistically significantly worse than another classifier on all metrics. For the single-objective tests, we follow the well-established procedure of [24] . That is, we first run a global Friedman test (see [33] ) for the null hypothesis that all classifiers have no differences with respect to the quality metric under investigation. In case we reject this null hypothesis, we can run post hoc Nemenyi pairwise tests (see [63] ), comparing the performance of algorithms pairwise, with the null hypothesis being that there is no difference between their performances w.r.t. the multidimensional quality metric considered. We would like to emphasize that such an approach does not take into account the dependence structure among the quality metrics. In other words, it only considers the marginal distribution (hence the term marginal-front) of the classifiers w.r.t. the individual quality metrics separately, not their joint distribution. In the following, we conduct the suggested marginal analysis for OpenML w.r.t. the three-dimensional quality metric considered (accuracy, computation time on training data, computation time on test data):• Global Friedman Test: Friedman rank sum test [33] rejects global null of no differences (p-value = 3.986e-14). This means we can conduct (two-sided) pairwise post hoc tests (α = 0.05) with no difference as the null hypothesis. • Post Hoc Nemenyi Test: Table 1 below shows the pairwise comparisons of algorithm performance with the Nemenyi test [63] . P-values below 0.05 are highlighted and indicate statistically significant differences in performance. A second issue that may have influenced the unexpected performance structure obtained in the paper is the way performance is evaluated by OpenML. OpenML is based on the uploads of its users. Each user is free to decide which hyperparameter settings to use. Thus, as there might be a different goal on the hyperparameter setting in each dataset, the results are not representative for the best performance of each algorithm. This aspect should be included in any further discussion. Especially since some algorithms are more dependent on hyperparameter settings/tuning than others. For an example involving hyperparameter tuning that is fixed in advance, see Section 5.2.This sections give further insight to the exemplary benchmarking analysis on the Penn Machine Learning Benchmarks (PMLB) in Section 5.2 in the main paper. We start by giving more details on the data sets with all the computation settings of the classifier algorithms. Afterwards, we provide more figures and explanations of the analysis.Penn Machine Learning Benchmarks (PMLB) is a collection of curated benchmark datasets for evaluating and comparing supervised machine learning algorithms [64] . We select all datasets from PMLB for binary classification tasks with 40 to 1000 observations 14 and less than 100 features. On these 62 datasets 15 , a recently proposed classifier based on compressed rule ensemble learning [62] is compared w.r.t. robust accuracy against five well-established classifiers, namely classification tree (CART), random forest (RF), support vector machine with radial kernel (SVM), k-nearest neighbour (kNN), and generalized linear model with elastic net (GLMNet). In detail, we deploy• Support Vector Machine (SVM) algorithm as implemented in the e1071 library [26] • Random Forests (RF) algorithm as implemented in the ranger library [88] , requiring e1071 library [26] and dplyr [87] • Decision Tree (CART) algorithm (C4.5-like trees) as implemented in the RWeka library [39] ,• Elastic net (GLMNet) algorithm is implemented through the glmnet library [32] requiring library Matrix [7] , and • k-nearest neighbors (kNN) algorithm as implemented in the kknn library [38] .Note that we used the respective methods in the caret library [49] for hyperparameter tuning and cross-validation to retrieve i) through iii), as detailed below.We operationalize the latent quality criterion of robust accuracy through i) classical accuracy (metric), ii) robustness w.r.t. noisy features (ordinal), and iii) robustness w.r.t. noisy classes (ordinal). Computation of i) is straightforward; in order to retrieve ii) and iii), we follow [92, 93] by randomly perturbing a share (here: 20 %) of the data points. We randomly selected data points with a selection probability of 20% and replaced the values by a random draw from the marginal distribution of the corresponding variable. (This is a slight difference to [92, 93] who replaced the data points by a random draw from a uniform distribution of the corresponding support of the marginal distribution.)We then tune the six classifiers' hyperparameters on a (multivariate) grid of size 10 following [49] for each of the 62 datasets and eventually compute i) to iii) through 10-fold cross validation. This latter purely descriptive analysis already hints at the CRE not belonging to the GSD-Front. In order to transition to inferential statements, we aim to statistically test (at level α = 0.05) whether CRE significantly lies in the GSD-front of some subset of C. As detailed in Section 4.1, we conduct five pairwise tests for the hypothesis pairs (H C ′ 0 , ¬H C ′ 0 ) (where C := CRE and C ′ ∈ C \ {CRE}) at a level of α 5 , as explained in Section 4. 16 In other words, we test five auxiliary null hypotheses, each asserting that CRE is GSD-dominated by SVM, RF, CART, GLMNet, and kNN, respectively.The results of these tests are visualized in Figure 5 (densities) and Figure 6 (cumulative distribution functions). 17 They indicate that the pairwise tests of CRE versus SVM, RF, CART, GLMNet, and kNN do not reject at a level of α 5 nor at level α. Hence, we conclude that based on the observed benchmark results we cannot conclude at significance level α = 0.05 that CRE lies in the GSD-front of any subset of C. In other words, we have no evidence to rule out that CRE is in the GSD-front, i.e., we cannot confirm based on the data that CRE is not outperformed by SVM, RF, CART, GLMNet, and kNN with respect to all compatible utility representation of robust accuracy. As can be seen in Figure 5 , testing CRE vs. CART results in the smallest p-value of all pairwise tests, which appears plausible, since CRE is a CART-based method. On the other hand, the observed test statistic of CRE vs. RF is far away from the critical value and the test cleary does not reject, even though RF is also a tree-based method.Finally, as discussed in Section 4.2, we further analyze the robustness of this test decision under contamination of the benchmark suite, i.e., deviations from the i.i.d.-assumption. As opposed to our OpenML analysis in Section 5.1, see also Appendix C.1, contamination does not affect the test decisions here, since none of the tests rejects already for uncontaminated samples. Increasing contamination only drives p-values further. The results are visualized in Figure 7 . It is observed that the tests are neither significant at a level of 0.05 5 nor at 0.05 and this clearly does not change with growing size of contaminated benchmark data sets.In summary, the PMLB experiments demonstrated how to apply our benchmarking framework to the problem of comparing a newly proposed classifier to a set of state-of-the-art ones. Furthermore, it illustrated our tests' applications to multiple criteria of mixed scales (ordinal and cardinal) that operationalize a latent performance measure, namely robust accuracy. It became evident that our framework allows to statistically assess whether the novel classifier CRE can compete with existing ones -that is, whether CRE lies in the GSD-front of some state-of-the-art classification algorithms. In Justification: Abstract and introduction of the paper reflect the paper's contribution and scope, covering both theoretical and experimental results.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification: The research conducted in the paper is, in every aspect, conform with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We do not foresee direct negative societal impact from the current work. Positive societal impact in form of making benchmarking results more robust to unjustified assumptions are discussed in Section 4.2 of the paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA]Justification: The paper does not use existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.This effect is usually more pronounced under the second motivation: Whereas in the first case the different metrics attempt to formalize the same latent construct, here different quality dimensions are actually to be covered. E.g., an improvement in accuracy may often be accompanied by a deterioration in computation time.There, one interprets the data sets as realizations of a random variable T : Ω → D on some probability space (Ω, S, π), chooses weights w1, . . . , wn ∈ R+ and assigns each C ∈ C the value n i=1 wiEπ(ϕi(C, T )).Implementations of all methods and scripts to reproduce the experiments:https://github.com/ hannahblo/Statistical-Multicriteria-Benchmarking-via-the-GSD-Front.The VC-dimension of a set system S is the largest cardinality of a set A with 2 A = {A ∩ S : S ∈ S}.This is not very restrictive, it only assumes to consider classifiers that are not already equivalent w.r.t. GSD.In both applications in Section 5 the tests are based on the unregularized statistics d 0 s (C ′ , C), as the regularization performed in[48] aims at reaching a goal which is not primarily relevant for our paper (see A.2.5).For benchmarking deep learning classifiers or optimizers, we refer to future work discussed in Section 6.As explained in Footnote 7, we base the tests in Sections 5.1 and 5.2 on the unregularized d 0 s (C ′ , C).Note that assuming only an exchangeable sample would also suffice. Note further that we have to assume the measurability of the involved infimum type statistics. For more details on this issue, see, e.g.,[28].The width of a preordered set is the maximal cardinality of an antichain, i.e., the maximal number of pairwise incomparable elements.[49] requires at least 4 data points in the test set, which translates to a mininmal n of 40, since we deploy 10-fold cross validation.Last access of PMLB: 12/05/24.As clarified in Footnote 7, the tests in Sections 5.1 and 5.2 are based on the unregularized test statistics d 0 s (C ′ , C).For generating these plots, we used quantile functions from both base r and the ggplot library. As the underlying quantile functions definitions differed slightly, we relied on the latter and corrected the quantiles from the other manually. Detailed documentation of all computations involved in generating the visualizations in the paper, we refer the interested reader to https://github.com/hannahblo/ Statistical-Multicriteria-Benchmarking-via-the-GSD-Front. 
 
 Appendix:  


Decision: The authors propose a new method for benchmarking classifiers on multiple criteria using the GSD-front. The paper provides an in depth treatment with theoretical results and statistical tests, backed up by extensive empirical experiments. On the down side, the paper is lacking in providing the reader with an intuitive understanding of the GSD-front, which the authors have promised to address via a didactic example. There is also a focus on benchmarking using large collections of datasets, which could be useful for academic comparisons of classifiers, but less useful for practitioners who are interested in selecting good classifiers for a particular problem.  Overall, all reviewers recommended accepting the paper.

Reviews: review: paper_summary: paper_summary: This submission studies the problem of comparing multiple classifiers under multiple evaluation criteria. It presents the construction of the GSD-front, i.e., the set of GSD-undominated classifiers, and an empirical variant called the $\epsilon$-empirical GSD-front.

Then, theoretical aspects of the GSD-front and the $\epsilon$-empirical GSD-front, such as satistical consistency, and the inclusion concerning the respective Pareto-front, are investigated. 

Two statistical tests, namely static GSD-test and dynamic static GSD-test, on whether a given classifier $C \in \mathcal{C}$ is an element of the true GSD-front are proposed.  Both tests are shown to be valid level-$\alpha$ tests. A robustified static GSD-test is also proposed to deal with the setting where the available data sets are drawn under non-i.i.d.-scenarios.

An empirical study is conducted to assess the proposed tests. It covers 80 binary classification data sets from OpenML [70]. The set of classifiers consists of Support Vector Machine (SVM) with Random Forest (RF), Decision Tree (CART), Logistic Regression (LR), Generalized Linear Model with Elastic net (GLMNet), Extreme Gradient Boosting (xGBoost), and k-Nearest Neighbors (kNN). The set of evaluation metrics consists of predictive accuracy, computation time on the test data, and computation time on the training data. Empirical evidence suggests that the GSD-approach tends to be more contrastive, compared to Pareto-front and another test that aggregates the single-metric tests as in [19]. 

Another empirical study is conducted using 62 datasets from the Penn Machine Learning Benchmark (PMLB). The set of classifiers consists of rule ensembles of trees (CRE) [53],  CART, RF, SVM with radial kernel, kNN and GLMNet.  The set of evaluation metrics consists of classical accuracy (metric), ii) robustness of accuracy w.r.t. noisy features (ordinal), and iii) robustness of accuracy w.r.t. noisy classes (ordinal). The results provided by GSD-tests are discussed.
strength_weakness: strength_weakness: W1: The authors might consider mentioning more scenarios where comparing multiple classifiers under the multiple evaluation criteria is crucial. An example might be multi-label classification, where multiple evaluation metrics have been proposed to evaluate multi-label classifiers. The set of evaluation metrics can also be extended with others, such as training time, storage memory, and so on. 

W2: The motivation to compare multiple classifiers using evidence on multiple data sets, which can come from very different application domains might need to be strengthened. For example, it would be not easy to convince practitioners in safety-critical applications, such as lung cancer detection to use some classifier that has shown to be promising given evidence aggregated from multiple domains.     

W3: Additional recommendations for the end-user might be beneficial. For example, what should one do if the GSD-front contains multiple classifiers?
questions: questions: Q1: Could you elaborate more on the case where only one training data is taken into account? This is related to the comment in W2. 

Q2: Regarding W3, should one randomly choose one element of the GSD-front when having to make predictions? Yet, one might consider doing ensemble learning on the set of promising classifiers.  In safety-critical applications, such a strategy might lower the interpretability and explainability of the predictive system. Do you have any recommendations in such cases?

Q3: Could you also discuss the Pareto-front and results of the test that aggregates the single-metric tests as in [19] for the empirical study on PMLB data sets?
limitations: limitations: Please refer to "Weaknesses" and "Questions" for detailed comments.
paper_summary: paper_summary: The authors propose a method for multicriteria evaluation of classifiers which is more informative than the Pareto front. The new GSD-front and is based on the previously proposed generalized stochastic dominance ordering (GSD) for classifiers. The authors also provided a sound inference framework, included hypothesis tests for whether a new classifier would be in the GSD-front of a benchmark.
strength_weakness: strength_weakness: 1. There could have been maybe a toy example to show the differences between GSD-front and Pareto-front in a more didactic way
2. The method seems computationally expensive. The pmlb_permutation_tests.R script needs 5 days to run and this does not include training of the evaluated models, as these are collected results
questions: questions: 1. How does the running time of a Pareto-front benchmark compare with a GSD-front one?
    1. If the Pareto-front is much cheaper to obtain, would it make sense to start with it, and only look for what classifiers in that pre-selection would be in the GSD-front?
2. How were different hyper-parameter values treated in the experiments? Were all results of each classifier for a dataset aggregated or did the authors use specific settings for each classifier, e.g. default values from the classifier's implementation?
limitations: limitations: Yes
paper_summary: paper_summary: The authors propose a novel way of comparing classifiers to assess their effectiveness. They posit (1) that comparisons should allow for different quality metrics simultaneously, (2) that comparisons should take into account the statistical uncertainty induced by the choice of benchmark suite, and (3) that the robustness of the comparisons under small deviations in the underlying assumptions should be verifiable. They illustrate their proposed method on the benchmark suite PMLB and on the platform OpenML.
strength_weakness: strength_weakness: Some references to the existing imprecise probabilistic literature are missing, there are some small typos, and an extra effort can be put to make the definition of $d_s$ more accessible.
questions: questions: When introducing imprecise probabilities, the authors fail to reference a few papers in such a literature. In particular, the authors leverage the idea of $\epsilon$-contamination in section 4.2. Other papers in the imprecise probabilistic machine learning literature dealing with, and advancing the knowledge of, $\epsilon$-contaminations are [1-5]. In addition, [6] uses IPs to robustify Bayesian neural networks. More in general, we suggest the authors to read (and possibly cite) the works by Eyke Hüllermeier, Thierry Denoeux, Sebastien Destercke, Fabio Cuzzolin, Alessio Benavoli, and Michele Caprio on imprecise probabilistic machine learning, all of which have a distinct "robustness" flavor to them, and that might be interesting to the authors for future endeavors.

There is a typo in line 95: there's a period after superscript 3.

While I understand the need of being as general as possible, wouldn't it be more beneficial to the reader to define $\hat{\pi}_c(\{z\})= 1/s \left|\lbrace\{ i : i \leq s \text{, } \Phi(C,T_i)=z \rbrace\}\right|$ in line 167, instead of using a generic $M \subseteq [0,1]^n$ for the definition?

There is a typo in line 167: last letter should be $C^\prime$ and not $C$.

Line 205: the question *of* how to [...]

---

[1] https://www.jstor.org/stable/2242055

[2] https://onlinelibrary.wiley.com/doi/book/10.1002/9780470434697 (Chapter 10)

[3] https://link.springer.com/chapter/10.1007/978-3-031-57963-9_1#:~:text=In%20their%20seminal%201990%20paper,bound%20to%20hold%20with%20equality.

[4] https://arxiv.org/abs/2402.00957

[5] https://arxiv.org/abs/2308.14815

[6] https://arxiv.org/abs/2302.09656
limitations: limitations: See questions.

Comments: title: Thanks for the reply
comment: Thank you for your reply and for taking into account our rebuttal in your assessment of the paper! 

We will include a discussion about the case where the GSD front contains multiple classifiers in the new paragraph with end-user recommendations.
Thanks for your reply. I think the response indeed complements the submission. The additional discussion on the case where GSD-front contains multiple classifiers might need to be further elaborated as it might be of practical relevance. In different applications of credal decision making, I guess one might expect that there are experts who know and can provide the true classes (by possibly paying additional costs). Yet, I think whether one should expect such experts in model selection/comparison might require more careful attention. However, I think the motivation for avoiding overoptimism is also interesting. After due consideration, I raised the rating to $7$.