Multivariate Stochastic Dominance via Optimal Transport and Applications to Models Benchmarking
Authors: Gabriel Rioux
Apoorva Nitsure
Mattia Rigotti
Kristjan Greenewald
Youssef Mroueh
Venue: NeurIPS 2024 poster (2024)
Keywords: Optimal Transport
Stochastic dominance
hypothesis testing
Central limit theorem
LLM benchmarking
Primary area: learning_theory
TL;DR: We extend the univariate first order stochastic dominance hypothesis testing to the multivariate case using entropic smooth optimal transport

Abstract: Stochastic dominance is an important concept in probability theory, econometrics and social choice theory for robustly modeling agents' preferences between random outcomes.  While many works have been dedicated to the univariate case,
little has been done in the multivariate scenario, wherein an agent has to decide between different multivariate outcomes. By exploiting a  characterization  of  multivariate first stochastic dominance in terms of couplings, we  introduce  a statistic that assesses multivariate almost stochastic dominance under the framework of Optimal Transport  with a smooth cost. Further, we introduce an entropic regularization of this statistic, and establish a central limit theorem (CLT) and consistency of the bootstrap procedure for the empirical statistic. Armed with this CLT, we propose a hypothesis testing framework as well as an efficient implementation using the Sinkhorn algorithm. We showcase our  method in comparing and benchmarking Large Language Models that are evaluated on multiple metrics. Our multivariate stochastic dominance test allows us to capture the dependencies between the metrics in order to make an informed and statistically significant  decision on the relative performance of the models.

Multivariate Stochastic Dominance via Optimal Transport and Applications to Models Benchmarking
Gabriel Rioux
Apoorva Nitsure MIT-IBM Watson AI Lab IBM Research
Mattia Rigotti MIT-IBM Watson AI Lab IBM Research
Center for Applied Mathematics Cornell University
Kristjan Greenewald MIT-IBM Watson AI Lab IBM Research
Youssef Mroueh MIT-IBM Watson Al Lab IBM Research
Abstract
Stochastic dominance is important concept in probability theory, econometrics and social choice theory for robustly modeling agents' preferences between random outcomes. While many works have been dedicated to the univariate case, little has been done in the multivariate scenario, wherein an agent has to decide between different multivariate outcomes. By exploiting characterization of multivariate first stochastic dominance in terms of couplings, we introduce a statistic that assesses multivariate almost stochastic dominance under the framework of Optimal Transport with smooth cost. Further, we introduce an entropic regularization this statistic, and establish a central limit theorem (CLT) and consistency of the bootstrap procedure for the empirical statistic. Armed with this CLT, we propose hypothesis testing framework well an efficient implementation using the Sinkhorn algorithm. We showcase our method in comparing and benchmarking Large Language Models that are evaluated on multiple metrics. Our multivariate stochastic dominance test allows us to capture the dependencies between the metrics in order to make an informed and statistically significant decision on the relative performance of the models.
Introduction
In choice theory, economics, finance, and models benchmarking, agents are faced with stochastic prospects that are (eventually) multivariate random variables which they wish to order according to a utility risk measure of interest. To formalize such notion of ordering of stochastic quantities, the concept of stochastic dominance can be utilized.
In the univariate case, standard notion of order is given by First order Stochastic Dominance (FSD) which can be expressed in terms of the quantiles of the underlying random variables [Ogryczak and Ruszezynski, 2002]. To wit, a random variable X dominates another random variable Y in FSD, if it has larger quantiles than Y across all percentiles. A weaker notion FSD, called almost stochastic dominance
introduced in del Barrio et al. [2018]. Their approach is based optimal transport (OT) and consists of measuring a ratio which quantifies how close X to dominating Y. del Barrio al. [2018] further lay the groundwork for principled statistical analysis of almost FSD by establishing a central limit theorem for the empirical ratio as well as consistency of the bootstrap. Dror al. [2018], Ulmer et al. [2022], Nitsure al. [2023] used the almost FSD testing framework benchmarking Large Language models to make statistically significant decisions regarding which model select when these models are evaluated with a metric of interest on data.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Our main motivation is to extend the testing framework for almost FSD of del Barrio et al. [2018] to the multivariate case as to enable applications with dependencies between metrics. For instance, the problem of multivariate portfolio selection in financial applications has been treated via reduction to univariate orders [Kouaissah, 2021]. Another application of interest is that of multi- metric benchmarking which is central nowadays in ranking and selecting Large Languages Models [Bommasani et al., 2023, Chang et al., 2023, Huang et al., 2023, MosaicLM, 2023, Wolf, 2023, Zhang and Hardt, 2024]. Current approaches such as Nitsure et al. [2023] use aggregation techniques to reduce the ordering to the univariate case thereby ignoring dependencies between metrics.
Our starting point is the so-called standard multivariate stochastic order (see Chapter 6 in Shaked and Shanthikumar, 2007). This order can be expressed terms of couplings between random vectors. This insight allows us to follow the approach of del Barrio et al. [2018] and define an almost multivariate FSD via OT in Section 2. This notion of stochastic dominance can be defined using multivariate violation ratios that are expressed as optimal transport problems with smooth costs [Manole and Niles-Weed, 2024, Hundrieser et al., 2022, Groppe and Hundrieser, 2023]. Given that empirical OT suffers from the curse of dimensionality, we resort in Section 3 to entropic regularization [Cuturi, 2013] to alleviate that issue and hence define Entropic Multivariate Violation Ratios. We establish in Section 3 convergence of these entropic violation ratios as the regularization parameter goes to zero, as well as a central limit theorem and bootstrapping consistency in Section using the functional delta method [Römisch, 2006]. We highlight that the delta method has seen general success for proving limit theorems with entropic OT [Hundrieser et al., 2024, Goldfeld et al., 2024a,b). Armed with this theory, we propose new framework for hypothesis testing of multivariate stochastic dominance and apply it to multi-metric benchmarking of L.Mss. Multivariate FSD) captures the dependencies between the metrics in this setting and leads more robust ordering.
The indicator function of a set C Rd is denoted L (x(x) and takes the value 1 if E A and 0 otherwise. We also adopt the following shorthand notation, R4, [0, oo), the maximum of two numbers a, b E I is denoted by V 6, and for vectors 2, y E Rd, we write x S y to indicate that I; S y/ for every i {1, [l.....d]] d}.
Notation
Throughout, P( is the set of all probability measures on Rd. A measure 1) P(IR)) is said to be Lub-Gaussian with parameter 72 2 O with respect to the 1-norm provided E,, [IX22 S 2.
Convergence in distribution of random variables is denoted by - (in the sense of Hoffmann-Jorgennen when necessary, see Chapter 1 in van der Vaart and Wellner, 1996).
2 Optimal transport and stochastic order
2.1 ESD and Almost Stochastic Dominance in One Dimension
To properly motivate our results on multivariate FSD, we first review some theory for the univariate setting. For random variables X, Y, it is said that X dominates Y in the stochastic order (denoted Y) if < t) < P(Y < t) for every t E R. Formally, this means that the inequality
X
FSD Y TSX generally holds for a given instantiation of these random variables. This condition can be cast, cquivalently, as Fy (t) S F for every E (0, 1), where Fx (t) and Fy (t) are the quantile functions for X and Y respectively. With this formulation in mind, del Barrio et al. [2018] propose the following index of almost stochastic dominance;
Ewg v) = (F (t) (1)
where WH v) = -('")>))'''000''>"""""""00 F7')'00-))'()))
(2 v)
a = (aw(X), = lY)(Y), and, for = E R, (=)2 = (0V 2)² denotes the squared hinge loss. Here, the numerator captures the degree to which X fails to dominate Y whereas the denominator serves as a normalizing constant so that Ews(n.)) E [0, 1]. Indeed, as (x - y) + (y - = = (x y)² we see that Ew, (H, v) = O precisely when F (t) < Fx for a.e. t E (0. 1) whereas Ew, (p, v) when the opposite inequality holds. del Barrio al. [2018] then propose to test the null hypothesis Ey3, (H, 2) S Ep for some sufficiently close to O, corresponding to the case where X almost dominates Y in the stochastic order, versus the alternative hypothesis ewy, (H, v) > Eq. To this end, they provide a central limit theorem for the statistic &W and propose to construct the rejection region via bootstrap estimation of the limiting variance. Similar results were obtained for a notion of almost second order stochastic dominance in Nitsure et al. [2023].

---

We highlight that the index EWs can be connected to OT. Indeed, by Theorem 2.9 in Santam- brogio [2015], the numerator and the denominator in Ewy (u.v) v) can be written, respectively, as inf, ref()))) ['V- - x) (1, v), and inl f'V - xx) =))ddr (x, v), where [T v) denotes the set of all couplings of (p, v). These problems are (univariate) instances of the well-studied OT problem
/ odT,
OT,(U,
inf =ET((L.))
where e I2 IR Rt is given cost function. We highlight that the costs considered herein are such that an optimal coupling always exists (see Theorem 4. in Villani, 2009). This connection will serve as our basis for extending the notion of almost stochastic dominance to the multivariate setting.
2.2 Multivariate FSD and its Relaxation via Optimal Transport with Compatible Costs
In the sequel, we provide a general framework for assessing multivariate almost FSD using a purely OT-based methodology. Following Chapter 6 and Theorem 6.B.1. in Shaked and Shanthikumar [2007], given the random vectors X,Y E R we say that X dominates Y in the usual stochastic order (denoted X > Y) provided that there exists a coupling (X, Y) of (x, satisfying IP(X 2 Y) =1 FSD
(i.e. for each i = 1....,d.. d, X2 2 Y; with probability one). This condition can be cast as follows:
Lemma 1. Letting (resp. v) denote the law of X (resp. Y), X > Y if W/OTelu, v) = 0, where e: TR.
TRdd + RR4 the cost function c(x, y) 1 (.,
Evidently, the cost (eSw) (z) in Lemma 1 can be replaced by any nonnegative cost function c(x, y) satisfying e(2z, y) 0 if and only if x S y and it still holds that X Y if OT, 0.
FSD
We denote the set of all such costs by Cg. Definition 1 (OT Costs Compatible with Multivariate FSD). The set of all cost functions which are compatible with multivatiate FSD in the sense that 2laww (X), law (Y)) )) = 0 implies that X b Y
FSD
given by cg= = {c: Rd x Rd + R, such that c(z, y) = if and only if x S y).
A simple recipe for generating cost functions in Eg is to take any univariate function h: RR with the property that h- ({0)) = (=o,, 0) and define c(z, v) = > h(y - -x;). For notational simplicty, we write OT, to denote the OT cost with this type of cost function even when the aforementioned property does not hold. The results presented in the following sections require some additional smoothness assumptions the function h discussed above which summarize presently.
Definition 2 (Smooth Costs). The function h: RR- + R.,, satisfies the smoothness condition (SCa) if h is Lipschitz continuous with constant L2 0 (that Ih(x) h(y) < L/x - y) for every x. y IR) and, for k- = [/2]] + 1, h is k-times continuously differentiable with derivatives of order 8 S k satisfying (x)) < C (1 + |=l)p* for some Cs < co and Pk: > which may depend on k.
We now discuss some examples of cost functions of interest.
Example 1 (Examples of OT Costs). 1) The function h(2)) = e for 20) and O otherwise is known to be smooth (see Example 1.3 in Tu, 2011) and satisfies h ({0)) = (-o0, O). It is easy to that all derivatives of h are O (- -o, 0). and decay 0 at infinity (and hence are bounded on IR) so that h satisfies (SCa) for any d E N, and induces cost function in Ez.
2) The squared hinge function l1(z) (=)² considered in Section 2.1 has linear growth, but is non- smooth. Although this function can be smoothed using e.g. mollification as introduced in Friedrichs [1944], this will result in a cost which is not element of Eg and may be costly implement due to the convolution operation used in mollification.
3) The logistic function h(z)) log(1 + e for B 0 has linear growth and derivative h((z)= 1 1tess= 3x(B22,, where <(z) i+e- is the sigmoid function. As <'(2) = <(2)(1-<(2)) is easy to see that all derivatives of f are bounded on E so that the assumption (SCa) is satisfied for any d E N. Although h does not induce cost in Cz, it is increasing and decays to 0 faster than
s= as -cc. Moreover, if the induced cost sutisfies c(x, v)) S Ea. then max
x;)) S B log - 1). Thus, if (1, v) StoSea) cdm' + [(0200 edx* = for an optimal plan "t and
3

---

some small ε > 0. one has that [(ese0) cdx S so that n rte > su}) S f/eo i.e at assigns at least mass 1 5/0 to points (x. for which max; - =z)= I log(e - 1). Hence, for large B. e enforces similar properties to a cost in e.. Note also that h can be viewed as a smooth surrogate for the 0/1 loss for large B.
At this point, multivariate analogue to the univariate almost stochastic domination could be defined by analogy with Section 2.1. However, we highlight two major impasses which make the entropically regularized index considered in the following sections far more palatable option in dimension > 1. First, is well-known that the expected rate of convergence of empirical OT generally suffers from the curse of dimensionality in statistical estimation; scaling as n 1/a (cf. e.g. Manole and Niles-Weed, 2024). Although Hundrieser et al. [2022] improve these rates to depend on the minimum of the intrinsic dimensions of μu, V/ in place of d, entropic optimal transport exhibits preferable parametric rate of convergence. Next, solving the OT problem numerically between two finitely discrete distributions supported on N points requires solving a linear program in N² variables which can be prohibitive for even moderately sized problems.
Entropic Regularization of OT with Multivariate FSD Compatible Costs
Before defining the regularized index, we first provide some background on entropic optimal transport (EOT) with cost Rdd x TRd RR+. EOT is defined by regularizing the OT problem (1)
/ cd' + + (TII
OT ((,V))))=== inf xEll(a.))
(2)
where A 2 0 is a regularization parameter and DKL is the Kullback-Leiin divergence defined by dp if p is absolutely continuous with respect to and DKL (p. 1) = +o0
DKL (p. 1) = f log
de an
otherwise. When À = O, we recover the standard OT problem. If (+ @ v), (2) admits a unique solution and is paired in strong duality with the problem
/ ppp + Vodv - ff 2in dp 2 v(z, +
sup (EI
(3)
Solutions to (3) are known to be almost surely unique up to additive constants (i.e. if (p. 1/), (yp', v/ solve (3), xp = o' + C' p-almost surely and db v C v-almost surely for some constant C' R) and are uniquely determined for μ-a.e. x and v-ae. y by the so-called Schrödinger system
v(v) of
du(y), e --/o = /
s(e) efe.m)
du((),,
(4)
which implies that fe =())+ du @ v(z,)) EOT potentials satisfying (4) on the whole space are known to exist and are unique up to additive constants (see Lemma 6). We refer the reader to Nutz [2021] for comprehensive introduction on EOT including all stated results.
Entropic regularization of OT problems was introduced the seminal work of Cuturi [2013] as means to accelerate computation by utilizing Sinkhorn-Knopp's matrix scaling algorithm which can be efficiently implemented on GPUs Interestingly, entropic regularization also alleviates the curse of dimensionality rates in statistical estimation inherent to standard OT; for instance Genevay et al. [2019] and Mena and Niles- Weed [2019] show that the plug-in estimator for the EOT cost with fixed À > 0 and cost c(z: y) ||I y/ achieves parametric expected rate of convergence with dimension dependent constant (see also Groppe and Hundrieser, 2023, Stromme, 2023 for related results with the dimension replaced by the minimum intrinsic dimension of μ and v); del Barrio et al. [2023] and Goldfeld et al. [2024b] further establish a central limit theorem this setting.
Given that EOT is meant to approximate OT, we derive the properties of OT (1,, v) and its solutions as À O. First, we quantify the rate of convergence of OTT, to OT under mild conditions, then show that solutions of OTA. converge to solutions of OT 6,0 as X 0 in suitable sense.
Theorem 1 (Stability as 4 0). Let μ, E ((R.d) have finite first moment, be a function satisfying (SCa), and fix 8 > 0. Then,

---

1. for any X E [0, 1) satisfying [A-1] 2 (20/6)* we have that
4TA S OTA,AAAA v) OT 0 v) I dd lAlog(!/))
+
(5C' + 20d)
for Cs I=
Σÿ= S |z,] dup
2. if x is the unique solution to x. v) for À > 0, then there exists a subsequence of (TT) which converges weakly solution of OT 01,,)
We highlight that the implications of Theorem require only the Lipschitz condition imposed in (SCa). The proof of the first result follows that of Theorem 3.3 in Eckstein and Nutz [2023] which controls the error of approximating the OT cost and OT plan using discretizations of the measures play. The main novelty in our approach is provide explicit constants and simple argument showing that the rate at which a general measure on I can be approximated by finitely discrete measure on at most n points under the 1- W-Wasserstein distance scales at worst as n 1/a for sufficiently large n. The second statement is proved using the machinery of --convergence (see Maso, 1993). Complete proofs are provided in Appendix C.2.
Entropic Multivariate FSD Violation Ratio and Testing
By analogy with the univariate case described in Section 2.1, consider a normalized index of stochastic order violation given by
OT (T,,)(
En,(K,, D)= = OT (H,
(5)
we adopt the convention that En, E v) = 0 whenever OT (1, v) = O. Here, OT (H, v) is the EOT problem with cost c(x, 3) Ei= h(u by h((;; (01(xx;00) y1). This cost function is induced by the function h(z) = hz2) + h(-2) and hence satisfies (SCa) provided that h satisfies (SCa). Moreover, A,, v) < OT h. (n, v) by construction so that (n, v) E [0, yielding a normalized index. The corresponding notion of entropic multivariate almost stochastic dominance can thus be defined.
Definition 3 (Entropic Multivariate Almost FSD). We define (h,), FSD, the entropic multi- variate almost FSD via the violation ratio as follows:
> (h.A,50)-FDD
vf tffn,)(U,)) S EQ.
In light of Theorem 1, limAJo En, (1,, OTr,l(m.2) oT,,o(a,0) with the convention that this latter quantity is zero when OT o v) O. In the case that h is the squared hinge function described in Example I and d 1, we recover the univariate index from Section 2. As aforementioned. the squared hinge function is not sufficiently smooth to enable us to characterize the asymptotic fluctuations of the empirical index in arbitrary dimensions. See Example 1 which lists other examples of costs satisfying (SCa); this condition is sufficient for the following statistical developments. We underscore that the choice of cost influences the notion of stochastic dominance reflected by the violation ratio. In applications where a practitioner wishes to formulate a domain-specift notion of dominance, data-driven approach can be employed by replacing the fixed cost in the previous development by collection of costs (ha.0, )d_ for each dimension where 0; is the corresponding parameter (e.g. B in the logistic function) and optimizing over these parameters. The results presented herein readily adapt this setting, but, for simplicity, restrict our attention to the case of fixed h throughout.
4.1 Statistical Properties
We now lay the groundwork for performing principled statistical inference with the empirical estimator of the entropic index ε,, Namely, we establish the asymptotic properties of the plug-in estimator En (pn, Pn), where An = Σ 6x, Dn = EØ= ōy, are the empirical distributions from n independent observations, (X+) and (Y,) of and respectively. Furthermore, we establish consistency of the bootstrap procedure. To this end, given sets of n iterations observations of l and (X and (Y above and sets (X
B) ) and (Y;3)) y of independent samples from
5

---

An and Dn. p - - and = , Ey are the corresponding bootstrap empirical distributions. p& denotes the conditional probability given the data.
Theorem 2 (Limit distribution and bootstrapping). Assume that Jr, u E P(TRO) are sub-Gaussian with shared parameter r² > O and that h satisfies (SCa)- Let (ph, tn)) and (9hr th%) be any pairs of optimal potentials for O (H, v) and (1,,)(,,)) v) respectively satisfying the Schrödinger system (4) on Rd x Rd_ Then, 8, > 0,
1. vn (En, 1/n) En,A(14, v)) 3 N((, mean-zero Gaussian with variance o² OTs,a(Mu
1 vary (TRAAAAN)) Phh (Th, Ph) + var, (OT thn OT& (1,0)) 4%)
1
(Tn,)(u)))
ofo² 00>000 0. suPteR
Da (VT(Ens(AR (R)) - (nn Bn)) st) - P(N(O, ( 0.
Observe that the limiting distribution in Theorem 2 is non-pivotal in the sense that the variance depends on the population distributions (H, v) rendering direct estimation of the limiting variance highly non-trivial. The bootstrap consistency result in the second point enables us to establish confidence intervals for (,,) v). Explicitly, if Çs denotes the smallest value of t R for which
pB (En. (i / S t) 2 1 - B3 for any ,3 E (0, 1), then (n, v) E [0, 2 pn) - Ca] with (0,1) 1) due to Lemma 23.3 in Van der Vaart [2000].
probability approaching 1 a for any
The proof of Theorem 2 based on the functional delta method [Römisch, 2006] which extends the standard delta method to functionals defined normed vector spaces, following the framework of Goldfeld et al. [2024b]. Formally, this approach consists of showing that the functional map- ping Saag-r----Gusssaee Gaussian distributions (17. p) to A (1) p) is directionally differentiable at (n. 1) and Lipschitz continuous in a suitable sense and that the relevant potentials lie in a space of sufficiently smooth functions using the assumption (SCa). Smoothness of the potentials is crucial ensure that the empirical processes vn((ān (t), Vn(2, v) converge when treated as functionals on the aforementioned space of smooth functions. Complete details are included in Appendix C.3, and primer on the functional delta method can be found in Appendix D.
Remark 1 (On Theorem 2). 1) We note that the condition that A u) > 0 in Theorem 2 is satis- fied except in certain degenerate settings. Indeed, for general nonnegative cost e, cTc,> (p. v) =
if and only if J cdp @ V = 0 as follows from the fact that ( μ @ v) 2 0 with equality if and only if T = p @ V. In particular, OT (14, v) = 0 if and only if h(a;; yt) h(y; xx,) = 0 for every x E spt(μ) and y E spt(v) and every i (1....,d) d). If chosen as to generate a cost function which is compatible with multivariate FSD (recall Definition 1), h- ((0)) = (-00,0) 0] so that OT O, (1, v) = 0 if and only if p and v are point masses at some shared a E Rdd
2) Theorem 2 is presented in the balanced case with empirical measures from n samples. In the case where fnn and im are empirical measures from n + m samples with n the implications of Theorem 2 are easily seen to hold with var,
atm + s E (0, 1),
mm
in place of vn and o²
n-m
(OTHINREO)PP Ph OTaAAaaoppPh) ph) var. (OTALNIA Un
.OT (OTnalne)
SOT, (TT.)(mM)) OTRA vn) in place of g².
4.2 Multivariate FSD Hypothesis Testing In ML Models Benchmarking
With the statistical properties of the violation ratio in hand, we now consider using the violation ratio in the context of statistical testing for (h, Eo) FSD. Consider two d-dimensional distributions μ4, v. In our application, these will correspond to the distributions of performance of two language models' responses evaluated on metrics. Given n, samples from f4, respectively, Theorem to create statistically valid tests comparing μ and v. Similarly to Nitsure et al. [2023], we consider both absolute and relative testing (see Nitsure et al., 2023 for a complete discussion).
apply
Absolute testing The most straightforward application of Theorem 2 to specify a desired threshold Eo and consider the following hypothesis test for (h, X, 5o)- FSD: Ho 1
versus the
(A.A..o)- -FSD
alternative H1 : (h.A.50)
Note that v/ dominating μ would be tested separately. Given desired

---

confidence 1 - aa, the central limit theorem and bootstrap results in Theorem 2 suggest rejecting Ho if
m + n
Eh, Pm) S Eo +
V mn
(f(n: )m "(a),
(6)
where of is the bootstrapped variance (See Algorithm 1) and & is the CDF of the standard normal distribution. By Theorem 2, this test will be asymptotically valid.
Relative testing A downside of absolute testing is that it requires specifying a threshold Eo. This threshold can be meaningful in pairwise comparisons, but when multiple distributions (e.g. multiple language models evaluations) are being compared for ranking purposes, it is difficult to determine a priori what threshold to use to ensure that the distributions can be separated. As in Nitsure et al. [2023], we therefore also present a relative test that compares each of k random vectors with . between the k measures: Egg (hh.A) = EA.A (14 Hj) for j, {1... k), i # j. Let M- = (pI.... Mk)., and define the one-versus-all violation ratio of the dominance of μs on all other variables μJ. 4 i: Es (1h (M) = - E E Eig
measures ,i,..., jig. using a one-versus-all violation ratio. First, consider all pairs of violations ratios
(h. We can then define relative stochastic dominance (h, R--R-FSD as (M) S ... S c,2 (1h))) Here the most dominating Ei (h.A
Min x Mig R-FSD
Mi,
R-FSD
model is the one with the lowest one-versus-all violation ratio. Testing for relative dominance of Ae
14 on 24 we can then compare their one-versus-all ratios via the following statistic:
(0) ij (M)
El (M) - e) (M).To test for (h. A)-R-FSS of μr versus By then, we have the null hypothesis Ho : (M)) 2 versus the alternative H1 Ae;((M) < 0. It is possible to extend the central limit theorem and bootstrapping results in Theorem 2 to this relative statistic under independence assumption (omitted for brevity). Let M, = (f.,m:---- ()).)) be the empirical measures for samples from each distribution. As in the absolute case, we then reject Ho with a confidence a if:
Aa (Ma) S / 22 (i1, i2) (ax) where of B. (41, i2) is the bootstrapped variance (see Algorithm for the variance expression).
Multitesting and Ranking To apply the above multivariate FSD violation ratio hypothesis tests to ranking of multiple distributions, we follow the approach of Nitsure et al. [2023]. We aggregate the set of all pairwise tests, ensure multitesting statistical validity via Family- Wise Error Rate (FWER) control, and, if needed, aggregate the pairwise test results into numerical ranking. Our approach is described below and summarized in Algorithm 1. The overall complexity of performing one pairwise test is dominated by the cost of computing the EOT cost with h and h. To this end, the Sinkhorn algorithm is used [Cuturi, 2013], which computes EOT between distributions on N points with a complexity of O(N2 + N2), where K (d) denotes the cost of complexity of computing the cost c(2. y) between x, y E TR...
Ranking multiple distributions In our experiments below, we seek to the pairwise tests above to obtain statistically valid ranking of a set of le random vectors (1) with measures yl((), e.g. samples of ( can be the per-sample evaluation metrics for a set of language models. To rank k models at a specified significance level a, we first test all 122 - A pairs (n()) ((()),,), i * j. employing a FWER correction (see next paragraph) to guarantee a valid control on the overall false rejection rate. This yields a set of trinary outcomes for each (i,j) pair, with 1 if the null is rejected in the positive direction, -1 if the null is rejected in the negative direction, and 0 if the null is not rejected. These pairwise rankings are then combined into single rank using simple Borda count [de Borda, 1781] rank aggregation algorithm.
Multitesting FWER control When running a family of T tests each at significance level 1 - o, the true probability that at least one test falsely rejects the null scales with T. If the output of all T tests needs to be trusted simultancously, instead it is desirable that the probability of any test falsely rejects the null is less than or equal to the specified a. Achieving this requires adjusting, or "correcting" the significance level of each of the T tests, in a process called Family-Wise Error Rate (FWER) control. In the present work, we use the Bonferroni correction, which sets the significance level of the ith test to - a; with a; = a/T. /T. Note that while the Bonferroni correction is known to be pessimistic, we choose it as it sets uniform significance levels for all tests, as opposed to other strategies such as the Holm correction [Holm, 1979] which is tighter but yields highly nonuniform statistical power across the family of tests. Exploring sensible ways to employ nonuniform FWER control approaches the context of performance ranking is interesting avenue for future work.
7

---

Experiments
All experiments on NVIDIA A100 S0GB GPUs using PyTorch [Ansel al., 2024] (v.2.3.0, BSD-3 license) and the Python Optimal Transport package [Flamary et al., 2021] (v.0.9.3, MIT license) to compute optimal transport distances with and without regularization. Code for these experiments is available at
5.1 Synthetic Data Experiment
In this section we analyze our method on synthetic toy dataset that enables us to parametrically control the level of the multivariate stochastic dominance between two random variables. Given dimension d, parameter p E [0, 1]. and mean and variance parameters μ., o², our synthetic dataset is generated by sampling from the multivariate random variables X, X.Y E IR:
.x ~ (N,, 0²) for 1,...,d . Y; = + (2- (y()) - 1)U, with B((p) = Bernoulli(p) E {0, 1} and U, = Uniform(O, 1).
These variables X and Y are designed in such way that parametrizes the dominance of Y over X. In partieular, X 7 Y p < 0.5, and Y x
X if p > 0.5.
(h.A)
(NS
As a baseline for our synthetic experiments, we also compute the violation ratio for the standard FSD framework using unregularized OT (EMD), i.e. Ehinge,O- as a function of p for fixed d = 5, 0, o² 1.0 and N = 100 samples from X and Y. We then investigate how well this baseline approximated by Elog.A>0. the entropically regularized ratio with a logistic cost in Example 1. Fig. shows that as the entropic regularization parameter X decreases towards 0 and as the gain of the logistic cost 3 increases, SlogAA>o converges towards Etinge.0 across all values of p E [0, 1]. In all cases, multivariate FSD violation ratio predicts linearly p. indicating that it is captures well the FSD violations. This experiment indicates that, to best approximate the standard FSD violation ratio, X should be taken as small as possible (c.f. Theorem 1) and / should be taken as large possible (c.f. Example 1). There is, in practice, a tradeoff that must be made when computing the regularized index, as Sinkhorn's 's algorithm requires the matrix C/A, where C' is the matrix of pairwise costs. As such, if the ratio 8/ A is too large, numerical underflow will occur and the algorithm will fail. As rule of thumb, is recommended set / first and increase the value of X if instability occurs.
logistic cost with B = 8.0 (varying A)
logistic cost with varying 8 (A = 0.1)
OTninos o (EMD)
OTog $ = 8.0
OTie a. f 4.0
OTog 8 -
clu4, [iy)
0.6
0.6
Tie 8 1.0
Miggs. (EMD)
0T A 0.1
0.2
OT A 0.5
0.2
TTgg
OT A 2.0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
p
p
Figure 1: Convergence of Aso towards Shinge.0 in the synthetic dataset introduced this section. Left panel: for fixed parameter B = 8 of the logistic cost, Elog,As0 converge towards Ehinge as is decreased toward 0. Right panel: for a fixed entropic regularization parameter À = 0.1, Blog,> converges towards Ehinge.0 the gain of the logistic cost 3 increases. All simulations were generated for d = 5, = 0, o² = 1.0 and N 100. Points and error bars indicate average and standard deviation across 100 repetitions.
We now assess the power of our proposed test with as a function of the number of samples and the dimension. We consider the same setup as the previous experiment and set p 0.65 so that
8

---

Y
X. We then estimate the type I and type II error of the relative test statistic by
(M.,00.5)-PS0
averaging across 100 repetitions, these results are compiled in Fig. 2.
Type error for Synthetic Dataset
Type error for Synthetic Dataset
rate Wpe errer (false posieves)
0.7
0.14
dim-20
0.6
dim-20
0.12
dim=50
dim=50
0.5
0.10
0.4
0.08
0.06
0.04
0.2
0.02
0.00
0.0
20
40
60
80
100
120
140
20
60
80
100
sample
sample
Figure 2: Type and type II error of the relative test statistic as a function of the sample size n in dimension E {10, 20, 50}. Here, 8 = 8 and À = 0.01d.
Sample dependent Model Ranking Comparison with ChatGPT FSD
Kendall Tau Similarity
Evl
Relative Testing FSD w/ P(IC)
Relative Testing FSD P(EC)
MultiVariate Relative Testing
0.55
Sample Size
Figure 3: Mix Instruct Results: Comparison of Multivariate FSD to Reduction to univariate FSD with aggregation across the dimensions.
5.2 LLM Benchmarking
To test our method on real world scenarios, we have chosen a current topic of significant interest to the community: LLM Benchmarking, We show through our experiments that our method can provide more holistic ranking of LLMs evaluated different metrics opposed to present strategies which involve mean win rate. To demonstrate our method's application to LLM Benchmarking, we have conducted assessments two different sets of data.
Mixinstruct For our first evaluation we use the dataset from Jiang et [2023] (MIT license) that consists of responses from 12 different instruction following LLMs, with each response evaluated on metrics such as BLEU, ROUGE, BERTScore, BARTScore, etc. The data has train (100K rows) and test (5k rows) split where each row consists of an instruction, input sentence, the expected output from users, as well as the responses of set of different LLMSs with their decoding parameters and evaluation scores different metries. However for the test set, Jiang et al. [2023] also did

---

pairwise evaluation of the responses from the models by asking ChatGPT which response was better. We use this test set and generate a ranking of the LLMs using Entropic Multivariate FSD (Algorithm 1). where for each LLM we construct an empirical measure on IR' using samples varying from 100 to 5000. We then compute the pairwise ratios for these empirical distributions using the logistic loss with 3 = 0.2, the regularization parameter X = 0.1, and utilize the relative testing procedure from Section 4.2 to rank the 12 LLMS (see Fig. 4 for the ranking obtained with n 5000 in appendix Appendix B). The confidence intervals are then generated using 1000 bootstrap repetitions. Finally, we compare the resulting ranking with the (univariate ranking) provided by ChatGPT scoring (which serves as a human proxy) function of the sample size n using Kendall Tau similarity. These results are presented in Fig. 3.
We then compare different methods that reduce multivariate ordering to univariate FSD via aggrega- tion. The first method is portfolio aggregation with Independent Copula P(IC) [Nitsure et al., 2023], where a dimension is normalized with a global univariate CDF across all models and geometric mean is performed across all dimensions. A univariate FSD then applied on the resulting univariate random variables. The second method, referred to as portfolio aggregation with Empirical Copula P(EC) [Ruschendorf, 1976, Ulan et al., 2021], I, estimates a global multivariate CDF across all models, and then assigns to each evaluation vector the value of its CDE, Similarly, a univariate FSD is applied on this one dimensional data.
Results We see from Fig. 3 that the multivariate FSD, is sample efficient and has the highest Kendall tau rank similarity with GPT score. We hypothesize that this thanks its ability to capture dependencies between the metrics. The independent copula P(IC), ignores the dependencies and hence lags little behind but is still sample efficient. Whilst the empirical copula P(EC) captures the dependencies, it suffers from the curse of dimension and is not sample efficient.
Conclusion
In this paper, we proposed entropic multivariate FSD violation ratio as a statistic for assessing multivariate first order dominance. We addressed the convergence of these ratios as the entropic regularization goes to zero and established central limit theorem and bootstrap consistency for this statistic. These statistical properties were leveraged in framework for multivariate FSD) testing which was applied to multi-metrics benchmarking machine learning models, showing its benefits in capturing the metric dependencies. Casting testing for stochastic order an optimal transport problem with a smooth cost and devising an entropic regularization to ensure beneficial statistical and computational properties is an interesting framework that we envision to be useful and versatile for other stochastic orders. For instance the ji-first order dominance Galichon and Henry [2012] uses optimal transport maps as multivariate quantiles [Carlier et al., 2014] and defines ju-tochastic dominance; our entropic violation ratio framework can be extended to that case and, upon proving central limit theorems on the OT potentials, will lead to similar central limit theorems to the one presented in this work. Similarly, for the multivariate Lorenz order [Fan et al., 2024] that is of interest when the agent making the choice is risk averse. The Lorenz order can be expressed in terms of optimal transport maps and can be extended to our statistical testing framework using the tools introduced in this paper. We leave these developments for future work.
Acknowledgments and Disclosure of F Funding
G. Rioux is partially supported by the NSERC postgraduate fellowship
References
Ambrosio, N. Gigli, and G. Savaré. Gradient flows: in metric spaces and the space of probability measures. 190009777777000 364333--88 87722-8
Birkhäuser Basel, Basel, 2005. URL https:
J. Ansel, E. Yang, H. He, N. Gimelshein, A. Jain, M. Voznesensky, B. Bao, P. Bell, D. Berard, Burovski, et al. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. In Proceedings of the 29th ACM International Conference on Architectral Support for Programming Languages and Operating Systems, Volume pages 929 947, 2024.
P. Billingsley. Convergence of probability measures. John Wiley & Sons, 2013.
10

---

R. Bommasani. P. Liang, and T. Lee. Holistic evaluation of language models. Annals of the New York Academy of Sciences, 2023.
G. Carlier, V. Chernozhukov, and A. Galichon. Vector quantile regression: an optimal transport approach. arXiv preprint arXiv:1400 4643, 2014,
Y. Chang, X. Wang, Wang, Y. Wu, K. Zhu, H. Chen, L. Yang, X. Yi, C. Wang, Y. Wang, al. A survey on evaluation large language models. arXiv preprint arXiv:2307.031 2023.
G. Constantine and T. Savits. A multivariate Faa di Bruno formula with applications. Transactions of the American Mathematical Society, 348(2):503-520, 1996.
M. Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors. Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.. 2013. URL http:: pttptppep//prrocedingg neurips.cc/papeee wwwpppppppppppeeeeeeeeeeee.eeeee ppapperfileepppeeeeeeeeaeeeeeeee ptt.e//ppppe//113// Papeeddd-ppp---- pdf.
J.-C. de Borda. Mémoire sur les élections au scrutin. Histoire de l'Académie Royale des Sciences, 1781.
E, del Barrio, J. A. Cuesta-Alhertos, and C. Matrán, An optimal transportation approach for assessing almost stochastic order. In E. Gil, E. Gil, J. Gil, and M. A. Gil, editors, The Mathematics of the Uncertain: A Tribute Pedro Gil, pages 33 44. Springer International Publishing, Cham, 2018. URL https://doi. org/ 10.
011779978-3-311--7888--8-8
E. del Barrio, A. G. Sanz, J.-M. Loubes, and J. Niles-Weed. An improved central limit theorem and fast convergence rates for entropic transportation costs. SIAM Journal on Mathematics of Data Science, 5(3): 639-669, 2023. URL https://dol: org/ 10. 0177/222249660
R. Dror, G. Baumer, S. Shlomav, and R. Reichart. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers), pages 1383 1392, 2018.
R. M. Dudley. Uniformm central limit theorems, volume 142. Cambridge university press, 2014.
S. Eckstein and M. Nutz. Convergence rates for regularized optimal transport via quantization. Mathematics of Operations Research, Articles in Advance, pages 1-18, 2023. URL https: http//ooi.oo/////////o 10. 1287/moor. 2022. 0245.
Y. Fan, M. Henry, B. Pass, and J. A. Rivero. Lorenz map, inequality ordering and curves based multidimen- sional rearrangements, 2024.
R. Flamary, N. Courty, A. Gramfort, M. Z. Alaya, A. Boisbunon, S. Chambon, L. Chapel. A. Corenflos, K. Fatras, N. Fournier, L. Gautheron, N. T. Gayraud, H. Janati, A. Rakotomamonjy, I. Redko, A. Rolet, A. Schutz, V. Seguy, D. J. Sutherland, R. Tavenard, A. Tong, and T. Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021. URL http://jmlr http://rrroorroo//o/////oo wrg/ httg:/pppr//v22220-eeee
.ttml. G. B. Folland. Real analysis: modern techniques and their applications, volume 40. John Wiley & Sons, 1999.
K. O. Friedrichs. The identity of weak and strong extensions of differential operators. Transactions of the American Mathematical Society, 15:132-151, 1944.
A. Galichon and M. Henry. Dual theory of choice with multivariate risks. Journal of Economic Theory, 147(4): 1516, 2012.
A. Genevay, L. Chizat, F. Bach, M. Cuturi, and G. Peyré. Sample complexity of sinkhorn divergences. In K. Chaudhuri and M. Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Staristics, volume 89 of Proceedings of Machine Larrnng Research, pages 1574- 1583. PMLR, 16-18 Apr 2019. URL httph:ttt///t.t...t hatppa/:///ccceiiggg.. mlr. wwwpp/rvs. genevay 19a. html.
Z. Goldfeld, K. Kato, G. Rioux, and R. Sadhu. Limit theorems for entropic optimal transport maps and sinkhorn divergence. Electronic Journal of Statistics, 18(1):900-041, 2024a.
Goldfeld, Kato, G. Rioux, and R. Sadhu. Statistical inference with regularized optimal transport. Informa- tion and Inference: A Journal of the IMA, 13(1)ia 2024b.
S. Graf and H. Luschgy. Foundations of Quantization for Probability Distributions. Springer Berlin Heidelberg. www00000600009955555555
Berlin, Heidelberg, 2000. URL
M. Groppe and S. Hundrieser. Lower complexity adaptation for empirical entropic optimal transport. arXiv preprint arXiv:2306. 133500, 2023.
Holm. simple sequentially rejective multiple test procedure. Scandinavian journaa of statistics, pages 65-70, 1979.
Y. Huang, Q. Zhang, L. Sun, al. Trustgpt: A benchmark for trustworthy and responsible large language models. arXiv preprint arXiv:2306. //507, 2023.
S. Hundrieser, T. Staudt, and A. Munk. Empirical optimal transport between different measures adapts to lower complexity. arXiv preprint arXiv:2202. 10434, 2022.

---

S. Hundrieser, M. Klatt. and A. Munk. Limit distributions and sensitivity analysis for empirical entropic optimal transport countable spaces. The Annals of Applied Probubility, 34(1B): 1468, 2024. doi: 10.1214/3-AAP1995 URL https: //ooi org/18 1214/23-AAP1995.
D. Jiang. X. Ren, and B. Y. Lin. LIm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.0255: 2023.
R. W. Keener. Theoretical Statistics: Topics for a Core Course. Springer New York, New York, NY, 2010. URL 1007 attps://does org/10. 1007 00000/99880--00 -3839993--
N. Kouaissah, Using multivariate stochastic dominance to enhance portfolio selection and warn of financial crises. The Quarterly Review of Economics and Finunce, 80:480-493, 2021.
Manole and J. Niles-Wece Sharp convergence rates for empirical optimal transport with smooth costs. The Annals of Applied Probability, 34(1B):1108-11 1135, 2024.
G. D. Maso. An introduction to T-comvegence. Birkhäuser Boston, Boston, MA, 1993. URL https: //doi org/ 11111111099777--11 1---61---3322
G. Mena and J. Niles-Weede Weed.. Statistical bounds for entropic optimal transport: sample complexity and the central limit theorem. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL cttps://proceeddiee pdf.
MosaicLM. Lim evaluation scores. https: httts://www.mossaccml mosaicml................mmm..m.. com/1lm eeeilmmmvllaaiionn 2023.
Nietert, Z. Goldfeld, and Kato. Smooth p-Wasserstein distance: structure, empirical approximation, and statistical applications. In International Conference on Machine Learning, pages 8172-8183. PMLR, 2021.
A. Nitsure, Y. Mrouch, M. Rigotti, K. Greenewald, B. Belgodere, M. Yurochkin, J. Navratil, I. Melnyk, and Ross. Risk assessment and statistical significance in the age foundation models. arXiv preprint arXiv:2310007332 2023.
M. Nutz. Introduction to entropic optimal transport. Lecture notes, Columbia University. 2021.
W. Ogryczak and A. Ruszezynski. Dual stochastic dominance and related mean-risk models. SIAM Journal on Optimization, 13(1):60-78, 2002.
W. Römisch. Delta method, infinite dimensional. In Encyclopedia of Statistical Sciences. John Wiley & Sons.
Ltd, 2006. URL https: //tp:::///d...e http////i..o/////////// 1002/0471667196 ess3139.
L. Ruschendorf. Asymptotic Distributions of Multivariate Rank Order Statistics, The Annals of Statistics, 4(5):
912 - 923, 1976.
Santambrogio. Optimal Transport for Applied Mathematicians: Calculus of Variations, PDEs, and Modeling. Springer International Publishing, Cham, 2015. URL https://doi.org: 1007/978-77333 3-19-- 200922022--222022 2.
M. Shaked and G. Shanthikumar. Stochastic Orders. Springer New York, New York, NY, 2007. URI. bttps://doi. org/10 107001000070-7777777 38778-----7733455555
A. Shapiro. On concepts of directional diffirentiabilitty Journal of oprimization theory and applications, 66: 477-487, 1990.
A. Shapiro. Asymptotic analysis of stochastic programs. Annals of Operations Research, 30(1):169-186, Dec. 1991.
A. J. Stromme. Minimum intrinsic dimension scaling for entropic optimal transport. arXiv preprint arX1v:2306.03338, 2023.
L. W. Tu. An Introduction to Manifolds. Springer New York, New York, NY, 2011. URL https://doi.orr 10. 1007 19789197--9----11111 7400--6
M. Ulan, W. Löwe, M. Ericsson, and A. Wingkvist. Copula-based software metrics aggregation. Software Quality Journal, 29)4):633-899, 2021. URL htttps://doi.orggee 11t.:0000010111110100--0----0000
D. Ulmer, C. Hardmeier, and Frellsen. leee-ssgnificanne-eee and meaningful statistical significance testing in the age of neural networks. arXiv preprint arXiv:2204.06815, 2022.
A. Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
A. W. der Vaart and A. Wellner. Weak Convergence and Empirical Processes: With Applications Statis- rics. Springer New York, New York, NY, 1996. URL https://doi.oz:
1978-1-4757-2545
V. S. Varadarajan. On the convergence of sample probability distributions. Sankhyā: The Indian Journal of Statistics (1933-1960), 1911/))23-266 1958.
C. Villani. Optimal Transport: Old and New. Springer Berlin Heidelberg, Berlin, Heidelberg, 2009. URL https://doll http://oo...r//1..0000//////// 740---000-9
12

---

E. B. CF...ISHHNNNRR......... T. Wolf. Open Ilm leaderboard. https::/huggiingface.. co/ opacee//uggging"aceee///o 1am_mmmmm_____ leaderbarrr 2023.
G. Zhang and M. Hardt. Inherent trade-offs between diversity and stability in multi-task benchmarks, 2024.
13

---

A Algorithm
Algorithm describes our muliitesting-based ranking procedure, for both the absolute and relative testing frameworks.
Algorithm Multivariate Stochastic Order Multi Aulti-testing (relative and absolute)
1: Input: M1, Pkx, &o models we want to rank corresponding to empirical measure Pr # -Ziz 62.,...Px PR 1 Ei=1 ógt. Threshold: т.
2: Input: Desired h, X, B number of bootstraps, K number of comparisons, significance level a.
3: Cache the bootstraps samples
4: for 1 to do
5: Pj P.
6: for 1 to B
7: p) < RESAMPLEWITHEE HIMMPLEWTHHRRPLLEEEENTTYY n)
8: Compute all violation ratios
9:
for to B do
10: for i = to k do
11:
for j to 1: do if i then
12:
Cost matrix [Ca]a. h([pTlx - lof])). {Use h(z) loogll + e0²). [pf]a is the kth
sample in ph.}
14:
[CElk, < [Cn]k. [ChlIA.
15:
14g. Sinkhorn(Ch, X, A.P, pf), 14 Sinkhorn (Cr, A, pp.
(Sinkhorn alg. with costs Ch. Cj and entropic reg. X.
16:
OT x
(p OT A. (p) Trace(Cr [..j.)) + ADkL (TT.,llllll @ p
Trace(C,, Hb + ADKL([I,, @ p))
18:
Eb,i,j En, xp..
in (5).
19: Eb,i,i 0,
20: Compute the sum statistics
21: for = to B do
22: fori= 1: do
23:
=/ - Σj Eb.i.
24: Compute the relative statisties
25: Asha = =ei <j, vb, i.j
26: Compute the Bootstrap Variance
27: for = 1 to & do
28: for j =
do
29:
i V B
(DE MEAN( AE .))2
30:
abs
B-1 ΣB Di (Eb.1, - MEAN(EOR.,.))
31:
Compute the test
32: Wing Win -0
33: for i to k do
34:
for
do
35:
if j and Aeo ij
vn - < then Win:; = (with confidence level 1 a//k2)
36:
37:
if i + j and Co.i. - - k2) < then Win abs 1 wwith confidence level 1 a/k² rank- BORDA(Win)) (with confidence level - a) rankgbs BORDA (Win
38:
(WRD #bsy (with confidence level 39: Return rank, rank
14

---

Model
one-versus-all violation ratio
1ess-sttt--pthiia-----epoo
12b-epoch-3.5
0.463332
chatglm-6b
0.472356
alpaca-native
0,474452
1ouuna-13b-1.1
0.475832
llama-
Jlaaa-7--f-baiiz--lor---ff
0.490481
mooss-moon-003-tt
0.499978
koala-7B-HF
0,506661
mpt-7b-instruct
0.508465
dolly-v2-12b
0,512816
stableim-tunee-alpphaae
0.521765
mpt-7b
0.522466
flan-t5-xxl
0.558775
Figure This table ranks the 12 models tested in the LLM benchmarking experiment using 5000
samples according to their one-versus-all violation ratio, E; (h.AA) (M) xー Ejtt Gj
(h.2)
(b.A)
where Egj
the pairwise violation ration of model compared with model j (lower is better).
B Ranking from the Mix Instruct experiment
C Proofs of main results
C.1 Proof of Lemma
Observe that the set {(x, y) E Rd x Rd : S y} closed, as any limit point, (z,y), of a sequence {I Vn) IneN satisfying Zn $ Yn is such that y the relevant inequalities preserved in the limit. follows that 1 (xSu) (I, a lower semicontinuous function and hence there exists coupling म E [1 v) for which infxen((r)) J I ( (I, (x, v) = (x v) y) for any choice of uuu E Rd (cf. e.g. Theorem 4.1 in Villani, 2009).
Assume that inf infxet((..) J (wsu)) (I,, dT (2, y) 0 such that there exists coupling ã for which 0 = [[ (2, y)d y) (x (X & Y) = 1 - P2(x Pr (X > 9) i.e. P ((X,8)= (X > Y) 2 P(x (X ÿ) = which, in light of Theorem 6.B.1. in [Shaked and Shanthikumar,
2007], implies that X > Y.C
FSD
C.2 Proof of Theorem
Throughout, we fix the metric ((rr, v), (x), E Rdad x Rdd + || x'111 + 1ly 5'Il on agdd We first show that costs induced by functions h satisfying (SCa) are Lipschitz continuous with respect with constant L.
Lemma 2. For any (x, v), (I', E End Rd. |e(æ, B) cla',a0 s s Ld((x, v). (z),1)).
Proof. As c(x,y) = Ei= M(:: 2x) for (x. v) Rdd x Rd and h is Lipschitz continuous with constant L 2 have that, for any x, y, x' Rdd
d
Mlvl +1)1 < L Σ 1y 1/ + x xz |e(2, v) - c('' |k - x;))
i=1
< L(lly 3/ |1 + || x'1l1).
Lemma 3. For any choice of (n, v), (1', E P(Rd)) x P(Rd) and π E M(, E MT(d)e))
have that
s (T,
where W1 WI n²) OTa(I, n²) is the 1-Wasserstein distance for the metric d.
15

---

Proof. Let v), (H', v/) P(R")x x xP())) be arbitrary and fix any I[ v) and x' E II(, Let y II( T') be arbitrary and consider
V cl (d.,))d v) - fclet cla', (d) o
/ dz. () c('', (od.v))dlle v. u..
- / le(x, v) - c(t' d2 v.
L / d((I,,)) (1).
As r is arbitrary, follows that |ʃ cd(a 7') & OTa(T,
The proof of Theorem 1 is based on the framework developed in [Eckstein and Nutz, 2023] and, in particular, the proof of Theorem 3.1 therein. As a byproduct of their proof techmique, it is demonstrated that, for any I1( v), there exists coupling x' E II(,,)) v)) which is independent of e satisfying
Jddt' oftn'-fdte cdd s 2T (0, A OT (o(",)))
(8)
provided that the cost satisfies the condition (7) (see Definition 3.1 in [Eckstein and Nutz, 2023] for weaker condition) for any choice of μ²,, n² p" (P*( the of all probability measures on Rd supported on at most points. We note that 11,0 is simply the 1-Wasserstein distance for the distance induced by the 1-norm and recall that has Lipschitz constant 1 due to the reverse triangle inequality (recall the notation OT,,, A from Section 3). then shown in their proof that
f cddn' - catn'- cdn S 4LCX, DKL - log -f
1
(
(9)
a
provided that there exists μ²,, 1n E Pm P" (Rd) satisfying (OT (1,11,0(4" .1) A OT 1,0(v" v)) I Cn for [a~ for some
(0,1) and C O.
Lemma 4. Fix 8 > 0 and assume that n, P(R") is not mutually singular with respect to the Lebesgue measure. Then, there exists measure 17"h pm (IRd) satisfying
20d
OT,,,, 7) S n 1/a
E, X1
1+8 +
8
1=1
for every 2 (20/5)*.
Proof. First, note that if Σd= E,, 1+8 co, the inequality holds vacuously. We hence assume that Σ E,, 1,| < <0.
Fix 8 > 0. Then, by Lemmas 6.6 and 6.7 in [Graf and Luschgy, 2000], there exists constants
C1, C2 C3 depending on 5 for which
inf OT OT n 1/a 20, Σ By [x 28 + 2dC2 PEP" (R4)
(10)
for every (2C))d. Here C
5 and C2 Cs 0 can be chosen any constants satisfying
28
((2))(8n-1) I(ón)
- <
Co C1 for every n 2 3. Observe that C2 = C33 10 satisfy these conditions.
5n
We conclude by noting that the infimum in (10) is achieved by Theorem 4.12 in [Graf and Luschgy, 2000].
Lemma 5. Fix 8 > 0 and E ((RRA). Then, there exists a measure 17"h E pp (RR)) satisfying
5
d
20d
OTH,,007", 77) s n 1/a
E,
X
1+8 +
8 1=1
8
for every > (20/8)².
16

---

Proof. Again, if Ei=1 | 1+6 = oo, the inequality trivially holds.
Assume that Σ By X |+8 < oo and, for t O, let 9e R R denote the density of an
isotropic mean-zero normal distribution with covariance ld.. Define the probability measure v 9r via
7 * (*(( = // La(x ())(())ddd(())=
= ( La(t))o(== - 2)dz) (0))))
dr(())
= = La(+) (f afllz - x) x)d((a) dz.
for any Borel measurable set A C R From the above display, 1) * gt. has density 2 E R + 9x(= =x)dn(x>)) with respect to the Lebesgue measure.
follows from minor modification of Lemma 7.1.10 in [Ambrosio et al., 2005], that, for any 1 S p < co, OT j-)) * 17) I t" 91= moments of all orders. Hence, limego OT OT1-1,((7> gr, n) 0 and Eyene Xx| E, as t (see Theorem 6.9 in [Villani, 2009]).
so normal distributions have finite absolute
Now, ry (R") be such that
5 8
d [x,12 1+5 + 8
20d
OT HLo00 77 9t) s
E Engg.
for every ni 2 (20/8)* as in Lemma 4. It follows for the triangle inequality for Wasserstein distances (see Chapter 6 in Villani, 2009) that
H.,,007 m) s (T. > + OT oooooonnn 77 * gt)-
The claimed result follows by applying the upper bound from the penultimate display and taking the limit t & 0 on both sides of the resulting inequality.
Lemma provides the worst case scaling for OT O..",. d can be replaced by suitable notion of intrinsic dimension for 17.
17). As noted in the text, is anticipated that
Proof of Theorem I. We begin with part The lower bound 0 S OT,, v/) due to the fact that the Kulback-Leibler divergence is non-negative. As for the upper bound, letting I( v) be an optimal plan for OT,,0 v). such a plan always exists due to Theorem 4.1 in [Villani, 2009]. It follows from (9) and Lemma 5 that there exists a plan x' E II( v) satisfying
oo v) is
I cdd'' + ADDLL @ v) - OT,o(u, S dd dd log
(1)
5 5O
d
20d
+ 4LA
Σ
Ey
1-6 +
t=1
provided that [A-d] 2 (20/2)d. By minimizing both sides of the above display with respect to r' M(p,, v), Part readily follows.
The proof Part will follow from the fact that the set of all couplings II(,, tight and hence admits limit point in the weak topology by Prokhorov's 's theorem (see Lemma 4.4 in Villani, 2009) and Corollary 7.20 in [Maso, 1993] once we establish that the functionals
sdd + ADKL (T] & v), if I((,))) otherwise,
FA : h E P(R Rd)
+00,
F:TEPPRY Rd) [edT,
if II(,,)), otherwise,
+00,
are such that Fx I connvrree to F when treated as functionals on the separable metric space (P(RR x d'), where d' is the Lévy-Prokhorov metric which metrizes the weak convergence of probability distributions (cf. e.g. p.72 in [Billingsley, 2013J), we refer the reader to [Maso, 1993] as a standard reference on I r-convergence. In light of Proposition 8.1 in [Maso, 1993] it suffices to show that:
for every æ E P(R (ed)) and any sequence P(RRd Rd)) a क - a with respect to d', F(TT) < lim inf (T(T)).
17

---

2. for every π E P(R x IR") there exists sequence P(R x (R 3 o - with respect to d' satisfying F(TT) limayo (T())
We start by proving the first statement. Fix т E P(R x Rd) and any sequence (Tx)aLO C P(R x converging to with respect to WT. By Lemma 4.4 in [Villani, 2009], [1 1)) is tight and hence precompact with respect to d' by Prokhorov's theorem. It is easy to see that [[( v) is closed under the weak convergence such that it is in fact compact with respect to d'. It follows that if π & II v), a & ([p,,)) 2/) for every À sufficienlly small, hence Fx (TA) + too = F(TT) as À 4 0. Now, [[( v),, compactness of [[ 2) implies that T E I[ v\ for every À sufficiently small. As J4, v have finite first moments and e Lipschitz continuous with constant I,, any E IT(,, v) satisfies
/ le(z,v))d7(z1 - [c(0, 0)| + L(I|=]] + 11el))d7((x
|c(0 0) f || + [1ullddul)) < co.
Conclude from Lemma 5.1.7 in [Ambrosio et al., 2005] that
(TA) -
f cdTT + ADKL 2
/ cdttx + cdt F(TT) FF(T),
proving the first condition.
As for the second condition, if π & I[(, the constant sequence TΛ = π satisfies (T(A FX(T)
+o0. If a I(,, let μe, E ("(RR) be such that (H, 14), OT 1110 (U" v) + 0 as spectively and - [|| - E E.
F(x)
0o (i.e. μ²'h and v" converge weakly to and re- + ExllY)]h). For instance, the empirical versions ju and constructed from independent samples satisfy this condition almost surely (see Theorem 3 in |Varadarajan, 1958] for weak convergence, convergence of the moments is due to the law of large numbers). By (8) and the surrounding discussion, there exists π E I[ v) satisfying
f cddx - fcdrr I 4L (OTH,O (ula A OT gjA-t,,)) - as A O.
From the proof of Theorem 3.1 in [Eckstein and Nutz, 2023], ADKL (Tx ( Alog( (X - "J] as À 0 and
(T(TA, s 2 (OT (alaT)) a OTH.00 (u(A-), + a
Conclude that
FX(TT) f
- cdtxx + ( + f cdt F(TT),
and, as W1 (TX, + O, Ta converges weakly ा (see [Villani, 2009]).Thi This concludes the proof.
C.3 Proof of Theorem 2
We first establish some useful properties of optimal potentials for this problem, there exists a pair Lemma 6. Fix X > and sub tab-Gaussion distributions (r, u P(RR)) with a shared constant 72 > Then, for any choice of h satisfying (SCa), there exists unique pair of continuous optimal potentials (q, 1b) for OT (,, v) for which the Schrödinger system (4) holds at all points (x. v) E Rd x Rd and (0)) O. Moreover, this pair of potentials satisfies the estimates
|x(x)| < CCL.IM1o (1 + |,
((bx)) S (1+
D()(())= S Ca..T.pp. (1 || FPp
ID DD)b())) S CaL.App + II
for any multi-index E Ng of order laa < == + 1 with the constants k. L. Ph from the (SCa) condition. The quantities in the subscripts of the constants indicate what parameters the constants depend
18

---

Proof. Let (ph: thb) be an arbitrary pair of optimal potentials for OT,,a so that
f (andu= + f Widd OT,, (1);
and, setting C= f dn (., we have that (fh) 1%) (ppn + C, 2h C) is another pair of optimal potentials, and J 1/ = S V/ A
We further consider the functions defined on Rd by
( de(v))
( du(x))
(x) = - - log
v((()) = -
log
We will show that (5%) 4%) are optimal potentials satisfying the claimed bounds. It follows by Jensen's inequality and Lemma 2 that
(x(x)) - ((E..v (1(l))vly) = c(0.0) +L
1 OT (TAAUED) (1vll + +||a=|1
=h(0)
Sv4.2
observing that 2 E., (exp( [x1li/2*)) 2 [11 such that flll 1lide s V472 (again by Jensen's inequality). follows that (x)) S Cd. (1 + ||=]|])) where Ca.L.r.A(o) depends
d, L, T, and the value of h(0). The same bound evidently holds for 4% on Rd and for 1iy on the support of v. Applying this bound and Lemma 2, it holds that
(fea
Ca livt1)
aCoy Diell 44v11
- (x(()) & A log
dv(v) S L||x||1 + Ca
where we have used the fact that T. [e S Ey
22,2
x
S for any / as follows
from Young's 's inequality and the sub-Gaussian assumption. The same argument implies that vo satisfies an analogous bound.
To that (10% v)() is pair of optimal potentials, observe that Jensen's inequality yields
/ (0% dh + / [(0% - - & Alog
/
du + Alog
dy
(hlottokhw-aamm X du @ v(z v)
>llog
Alog
dp @ v(x, 4) =0
as follows from the fact that (1%) vix) satisfy (4). Conclude that f 4h + dv OT , (H, v)
such that equality must hold (indeed f =(())+0[(w) du @ v(x, 3) = 1 by construction, so the left hand side of the ineqalit is the objective value in the dual form of the EOT problem) and, by strict concavity of the logarithm, 9% = 4/, fı-a.e and vi = vi/, v-aee such that (v% v/ are indeed optimal potentials for this problem.
Now, consider the potentials (sp, 1b) = (1p% - (0, vi + ()())), which satisfy (4) on Rd Rd (0) = O. The bounds for 66 and vy evidently carry over and tir so that there exists a constant C d.L. do depending only on d, L,,, v, and 1(0)) for which
|x(x)l S (o) (1 |
|((( S (o (1 Ilall))
for every (x, v) Rd Bdd Now, suppose that (7, )) is any other pair of potentials for which (4) holds Rd x Rd and (0)) As discussed in Section 3, one has that (yp, vb) and (5, () coincide μ- and v-a.e. that, for every Rd.
dv(v)
vivi cia.v)
dv(y) -1-
which shows that sp the equality of s and & follows analogously.
19

---

We now establish the bounds on the derivatives. By the multivariate Faà di Bruno formula (cf. e.g. Constantine and Savits, 1996), for any multi-index a Ný of order |a| 1, the derivative DD (assuming it exists) can be expressed as a linear combination of products of derivatives of the form
lod II j=1
D e (
dv(y)
(11)
ʃe (() dv(y)
where 3, E Ng and ky No satisfy the derivative commutes with the integral sign and, applying the same formula to the derivative we see that it can be expressed as linear combination of products of the form
E/= Byh; a. By the dominated convergence theorem,
D.: e -c(mm)
[8g1 II m=1
D m
Eh(y z1i)
(12)
where 7m E No. Im E No with Ele 7m = 3j. and we assume that all relevant derivatives exist.
As h satisfies (SCa), it is k-times continuously differentiable for k: = [4/2] + with derivatives of order < h satisfying (x)] S C&(1 |x])* )m for some Cj
and pk 1 which may depend on k. From this assumption, (12) is well defined provided that [3,1 S k and we observe that
T.m Ei h(y: x2) S Σ C1 + 1/s 2x1)² Sii1 C2 (1 + ly: mx( as Pa >
so that D e olaa) can be bounded as e Ca.A.pe (1 b1l = = 18, by appealing to (12).
<la.y)
Returning (11), infer that Da is well-defined for any |a| l and that
D Ic f x
-k(w)) du(y)
= =1ʃ wkin)
D
dla du(s)
/
sk
S CaMimaa
(1+ I1y z]11) pa1 du(y). pallsldyyy))
We will split this integral into the regions chosen later in the proof.
||y/|2 < T and [Blla 2 r for some 7 > which will be
In this first region,
J (
(1 11y x = S (1 + |[]|1)) / - dv(2))
by the triangle inequality so that
D&d S [[wller)) (()))((..)) dv(y)
< (1+
c(
dv(y)
For the second region, we have from Lemma 2 and the first part of the proof that
/
(1 + æ|11 pp (p))))
211
M00--M0++
(1 + 1/ æ|11 pallsldd(w)
1 (1 |y x pM 1p 18)l
21)
To bound this integral, note that
(1 + |/ - x 21112) pull8l < (1 + ||vlla + ||x]])) 2P11831-1 ( (1+ |l pxl051 + -1001(118.1)
20

---

and that
LV 2
20txil1 de(y)
dr(y)
21l
elllll
(Nmo 1du(()) (dd
e Lemp (
Se
by the Sauchy-Schwarce incquality. Similarly,
Jumniien 6 se 2
zveeR+ 12ll I1 dv())
de aului)
(Mmmmize)) I1ullpaalls de(y))
1
Now, f 111y11 dy S 2e due to the bound [elllllll) s E,
2 1x11?
S 2e 2,
which
holds for every t 2 O. Further,
J Idv(y) S ee=at /
1917 du(y) S 220- ,:.
e aad
(11a11233
[1lal112r
Ju (11i1127)
|B 2xxl, du(y) se ira f
Iull? e
||v|
19s1 dv()) S v2e ia
(fIlvl (
de(s))
where we have applied the Cauchy-Schwer inequality the second line. By sub- sub-Gaussianity
2 2 Exle 2 Ey
||x fr (2r2)pp
for any p E N so that / Bv [||X]] 4pu 19,1] s
22(272)pu133) (2pM |B,I Combining all of these bounds,
(1 + 118 - -111) dv(y)
2pp188111111
|0/|1 pall8sle EVaR2421E11
422g2 22 /2
x
37
2// 22/++
(2(222) 2
18,1
/ ((2p)188)1
The denominator can be bounded
(ʃe
dv(y)
S MO112 1 L as
follows from the first part of the proof. The ratio we wish to bound thus includes the exponential
140422y477.. +22 2x,222
- which we equate to by setting 7² -x² (h(() 2L 2L
.2
term e
2T. []]]) + 32 CL
16r? T.|101|12
All all,
(hill dia.v) D ʃ - du(y)
ʃe (((1)-l(o du(y)
Ca (1+ (Cr,,, + 16722~1 (I((x111)) 1/2 +|12111)
PEl
2//4+ + (22(27))ull1)) /
2P4lllp..2 + PA1B1))
2//4
(2px18,D!
the products in (11) can thus be bounded as Ca M.M00 (1 + [ for any |a| s The same argument establishes analogous bounds for 1/ As (4,, v) coincides with (v% v/f)) up to additive constants, the derivative bounds evidently transfer over.
21

---

In what follows, we always choose the unique solution for any given EOT problem described in Lemma 6.
As aforementioned, our approach to proving limit distributions based on the functional delta method (see Appendix D for summary of the method). As i also satisfies (SCa) with Lipschitz constant 2L. C& replaced by 2C2, and h()) 2h(0), there exists constant Ca, for which all bounds from Lemma 6 hold simultaneously for some choice of potentials (fPt, Vh) for OT A (p, v)) and (1% vn)) for OT (,, v). We thus instantiate the function classes
Fr. (s E C | sax)) S (1 + [(a]))) for k [4/2] + 1,
Va Ng with |a| s k, vo R²)
and
= -
where f @ g is understood as the function (x,y) E Rd IRd + f(x) + g(y). We further consider the class of probability distributions
Dt = (u (MOV: PE and are >a-sub-Gaussian)
for some 7 > 0. Throughout, we will treat S as a subset e0 (
7. the Banach space of bounded real functions on F.B endowed with the supremum norm || =0.
sup, sager" |&( @ g)]; the action of μ @ S on g is given by μ @ v(ʃ e g) = f fdu + gdv. It is easy see that @ v defines bounded function on this function class due to the growth bounds inherent to Fr,
Our approach is similar to that of Proposition I in [Goldfeld et al., 2024b] and, in particular. Section of that work. Namely, we prove type of directional differentiability and Lipschitz continuity for the EOT cost.
Lemma 7. Fix X > and Gab-Gaussian distributions 1, P(R") with constant 7² > Then, for any choice of optimal potentials (yp, v) solving OT,, and satisfying (4) on Rd x Rd.
O 1(p 1)). v + t(n v)) OT OT,,(U, = /dlpp - 1)+ + / vddnn - v).
lim e10
t
Proof. For t E [0, 1], let h4 := μ t(p - 11) and v4 == v + t(1 v) and, let (soluri)) vb(mesne)) denote the unique pair of optimal potentials for solving the Schrödinger system on IRd x Rd (see Lemma 6) and likewise for (sp(msie), golmer))) and (gpla.e), sb(mi)). Observe that
vn) satisfying pli).)) and
OTHA(UE OT -0T
= =TM v1) - OT,,)He,, +) OT
$ gliniee dyc + / (slures)de du. - / pllmeee ddue psleeree du
+ > w djh & v(xx1)--
+ pluaee) (oe f gslurcedde / plucee f ysluee)
+ > A/o eleeweeeieeeewelooeeaaa du v(z,y) ~x,
22

---

where we recall that the potentials satisfy the relevant Schrödinger systems (4) such that
dx(()) = 1 and f wo tol dv(y)) 1 on R so that fe (ot afx..)
OT a 4) -
-TMAllee v1) OTAAl - Mun v)
OT (TAaHu
I gluriee dye d
/ /sluneeddss colmiinn / pileeceed du
(13)
+ f plumeddes f /pllecedds / colanieddu solere)
=t (lmmi) 11) +t f splrreedddl - v)
and, analogously,
2))
2 pluv) 1) +t
plunii v).
(14)
It suffices, therefore, to show pointwise convergence of all relevant potentials to (ppaa,e, , wb(a.))) in the limit 4 O. Here we will only show convergence of (yo(mrite) sblue.et)), convergence of the other set of potentials follows analogously. For convenience set (4Pt, she) = (splmii))) as(m).)))
Let [0, in O be arbitrary and fix subsequence b./ By Lemma 6 we can apply the Arzelå- Ascoli theorem (cf. e.g. Theorem 4.44 in Folland, 1999) to infer that (tst, whr) converges to a pair of continuous functions (1p, 15) uniformly on compact sets and, in particular, pointwise along a further
subsequence t From Lemma 6,
29 o (6)-M) 2 (9) oor.9)
C2 (ill)) c0..0) Disih 121sl1
Se
where this final term is integrable with respect to any product of sub- Sab-Gaussian measures. By the dominated convergence theorem, for any (I. y) E Rd x Rd.
Pe (n)
4 (v)-n(n.))
d(u + bnn (n) + f ((s)-dla) du(y),
Mear (1)
(0)
d(u + tw 1))(2)) + 1 ((t)-c(a))) du(zz)
as ba 1. O such that the pair (sp, 16) satisfies the Schrödinger system (4) pointwise and hence is a
A. pair of optimal potentials for OT TT,, (1, and pp0)) lim Pt... (0) O, whence (se, 1h) (glimi) ()a))) Combining (13) and (14), we conclude that
O Me,.,) OT (.,,) t,p" and, as the limit is independent of the choice of original subsequence it follows that
I giw-)ao-- 11) / sllne) -) v),
lim 10
lim ( ut) - -T
I pluv)d 11) + / ws(nr) v)).
t
This limit is invariant under the transformation (solmnr), volure)) (splmie) + C, ssluee) C), proving the claim.
Lemma 8. Fix X > O and arbitrary sub-Gaussian distributions p. n, p. of E P(RR) with a shared constant 72 > 0. Then, we have that
(0Tn,((p.1) - OT (t..l/p./))11 l1p 8' 1'''oro
Proof. Let (gplam) (b(an)) (glle'n'), wolo' and (cplo' m)) yslo' be the optimal potentials for b. m). OT..(p' .7)), and OT (.xx'p' described Lemma 6. Then, by analogy with (13)
and (14)
OT,,)(,,)) OT oT / plant pd) +0)+ + wlnnddnn - n)),
OTH.(P, - -T glo'm)) 10 + f - n)),
23

---

such that
(OT - -OTal) &
{ a - -p)+
wi wi - n)
v clomi - p)+ + vilommad - -n)
As the constants from Lemma (and hence the constants defining Fr.h)) are independent of the choice of sub-Gaussian distributions (rather they depend only on the sub-Gaussian constant), (gol mn) wio n) and (vp(p.)) sple'))) m) are elements of Fith so that
(OTRA(A m) OT ,701 s ll & o' 2 2 1'lm,,
Lemma 9. Fix X > and sub- Gaussian distributions 14, V. E P(RR)) with a shared constant T² >> 0. Then, vil @ Pn - μ @ v) 4 Gusv in f (FFT.) where Gusv is a tight Gaussian process in e0 (FF for which Guzu (f @ g) = (1,, (f) (9)x
rhh
Proof. From the proof of Lemma 27 in [Goldfeld et al., 2024b] (see also Lemma 8 in Nietert et al.,
2021), see that the function class Fr,h .h p-Donsker and y-Donsker (i.e. the associated empirical processes Vn(ūa- and vn(2, -v) converge weakly to tight mean-zero Brownian bridge processes
in 80t (J,,), with respective covariance functions given by (f.g)= E Fr.h h F + cOv (f..))
and (f,g) E Fr,h x Fr, + cov.(f.g) respectively) provided that Er p pdtkps 119.-1pp 11 (I] >
1)4/2 < 00 for k = [4/2] + and likewise for v. By the Chernoff bound, Pp (IIXX 2
1) <
En [e] e 1((- for any t > 0. The standard inequality ||=||| I ||=|| for any z E Rd yields
En [ < E [c s 2e 2_,
(recall the proof of Lemma 6). It readily follows that the sum Ers pdtkpt--Ip (. [I|X 2 rr- 1) is finite, establishing Donskerness of the class with respect to Ja and hence v by the same argument.
By independence of the samples (X,)== and (Y1)²=1 we have by Example 1.4.6 in [van der Vaart and Wellner, 1996], Lemma 3.2.4 in [Dudley, 2014], and Donskerness of the class that
(Vri(an - 11), vn(0, - -)) 4 (G,,, G.) in e (Fr,) E (Frm),
where Gy and Gv are independent tight μ- and v-Brownian bridge processes. As the map (f, E 100 (Frri) x 8o0 (Fr,,,) + @ l' E &0 (FF))))) is continuous (indeed || @ al S ||]|,
(1())1),)) we have by the continuous mapping theorem that
VūlAmn 11) @ Vnlla- - -1) = (Mn 2 in -180) S GRow in e (FTR))
where Gaou(fo + f1) Gp(ft) + (f1) for any fo 1 f1 F T. proving the claim.
Proof of Theorem 2. Throughout, we fix some and observe that μ4, V are P--sub-Gauusii then they are also 7"-sub-Gaussian. From the proof of Proposition in [Goldfeld et al., 2024b] (see also Remark 4 of the same reference), Lemmas 7 and together imply that the functional & r7 E P + OT (treated as a convex subset of (FF).) with derivative 7 E Tpp (H 8 v) + r(p v) where (1,, v) denote any pair of optimal potentials for OT,,)(H, v) satisfying (4) on TRd x Rdd, and the derivative defined on the tangent cone to p at μ @ y which is defined as
(p, n) is Hadamard directionally differentiable at @ v tangentially to P
Tp (1 2 v) = {o E 80 (FF
F.hh: EPS C (Pn @ BRDNEN - ps 2 vn
and f., 4
s.t. 7 lim Pro @ mn
1 8 u t
see Appendix D for precise definitions. Note that we have identified O 17) with a functional on p such an identification is well-defined in light of the discussion following Proposition in [Goldfeld et al., 2024b].
24

---

The same implications hold for the functional & E P + E,, (p, in light of the discussion preceding Lemma 7, with corresponding derivative E p (u @ @v)
7 @ v), where (Ø, +) is
any pair of optimal potentials for (,,,, v) satisfying (4) on R RRd
R H 5/v such that the Note that E (14, vn) f (O,,,x(4, v), OT k,, v)) for f: (c. E R
chain rule (cf. e.g. Proposition 3.6 in Shapiro, 1990) guarantees that p 2 P + EA,A 7)) is Hadamard directionally differentiable at @ tangentially to p with derivative
1 OT h. (1,
OT
(EnW)ow Y (p
76 ① b))
OT F (M. w)² (p 1)
(15)
which is notably linear as function of y. will be useful to rewrite this expression in terms of single evaluation of y. To this end, observe that if f1 f2, @92 E F
F.h* then so too is (af1 1111-991 991 (a
892) for any a, B E IR with |a| + |3] S 1. Moreover, setting M OTR,AlaRt V a
(which (TAA(AM))
(
assumed to be nonzero by construction), we have that 0 S 2M OT OT
V 2M OT (
S
so that write
OT v)
(1p 2 w)
OT (T, (50 ( 3 v))
OT (1, v)
1
1
1
2M
+
() 1)
2M
OT
2M OT h.s
(T,,)(U,
1 OT (T,
2M
2M OT k,, v12 f
2M OT, i 2 2012
1
OT,X(H,D)
2M
2M (T
2M OT A. D)2
1
OT (,,,(1,, 2M OT 2/2
F. (1, v) v/
2M OT
F.,
where the final term in brackets is element of 7. Further, for any 7 E To (n 2 v), there exists sequence (pn MMNNEN C. P which converges to p in e0 (F and sequence t, 4 0 for which
7 1im Pn uu80
Thus, fo @ f1, @ g1 F are such that fo + go + fi + 91 E FF
we have that
7So @ f1) + r(go @ 91)
(f fudpn + / fidmm
-f fodp -
/ fidn)
lim (m) n=
(f soddnn f /giddin - godp f (gidn)
+ lim m n-400
ffo+soden godpn + +/ fi gidna / fo + godp - f fi + gndi)
= lim t n-900
=x(fo + 90 + g1).
Likewise, af @afi F. for some RR and fo + E F... then
( fuddpn fidmn - / fodp / fidn) an(fo f1).
7(afo @afi lim 1
With this, (15) can be written
(En.)) : : Tp v)
OT OTEA(U,,
OT OT,,)(,, v) 2MOT 22)
2My
2M0TT
2MOT, ( 2MOT 2MOT (HG
(16)
25

---

Given the above differentiability result and Lemma 9, part 1 of Theorem will follow from the functional delta method (see Lemma 10 in Appendix D) upon showing that f., & fn E P with probability approaching one and noting that Gpsv E Tpp (μ @ v) with probability one follows from the portmanteau theorem. To this end, note that, by the law of large numbers.
Eg., [exp(IIXili/2+*) = 1 E exp((IX:1 - En [exp(IIXili/2**) 2 <2
almost surely such that An is Suu-Guusi with probability approaching one. The same delibera- tions imply that in share the same property.
By applying the delta method, we obtain that
VTlEn. Pnn)) - EE,A v1) I (EnD) yeo (Gwov),
and using the explicit expression for the derivative from (16), we see that (EH,)) yoso (GHBw) ) is equal OT, x
in distribution to 2M, x² + w²), where v² var, (MMTT.AADD - 2MOT R.A (x. and ATHA( )): the 2.M terms in the variance and multiplying the w² var, 2 & 2MOT x
normal distribution evidently cancel out to give the desired formula for the limiting variance.
As for the bootstrap consistency result, since (15) is linear, it follows from Corollary 1 in [Goldfeld et al., 2024b] that (p, 1)) + An,A(P, 11) is Hadamard directionally differentiable at @ tangentially to spt(G As in the proof of Lemma 9, the class Fr is μr- and v-Donsker such that the bootstrapped empirical processes ( - (n)) and Vn(i
R - (a) are asymptotically measurable and converge conditionally in distribution to the μr- and v-Brownian bridge processes Gy, and G2 respectively (see Chapter 3.6 in van der Vaart and Wellner, 1996). By Lemma 1.4.4 and Example 1.4.6 in [van der Vaart and Wellner, 1996], (yn(a, - -in), Vn( -nn) is asymptotically measurable and converges conditionally in distribution to (G,,, Gy) as elements of go0 (ox x ('x As the map (e, (') E (F+) x (F+) + l & l' E (FF)), is continuous, (vπ(p, - (in) @ vn - D,)) asymptotically measurable and converges conditionally in distribution to GRSS as elements of e (FFAA) (F))) where Gyæev (fo @ fi) = (fo) + Gz(fi) for any fo @ fi E F7.h- T.h"
Bootstrap consistency then follows from Theorem 23.9 in [Van der Vaart, 2000] by applying the logic from the first half of the proof.
D The Functional Delta Method
Our strategy for deriving limit distributions and consistency of the bootstrap is based upon the functional delta method, which generalizes the standard delta method for functions of simple random variables. This section provides brief introduction to the functional delta method following the exposition of [Römisch, 2006]. Throughout, convergence in distribution is understood in the sense of Sollimann-Jorgensen when necessary (cf. e.g. Chapter 1 in [van der Vaart and Wellner, 1996]).
Much like the delta method which, identifies the distributional limit of vn(g(Xn) g(14) as N (0, o0, (())())))) n oc provided that (n((X, - 14) $ - (0, o²) and that g IR + differen- tiable at (see Proposition 1.14 in Keener, 2010), the functional delta method establishes the limit distribution of functional f θ C + IR, where D is a normed vector space. In this setting, the surrogate for the derivative in the standard delta method is the Hadamard directional derivative.
Definition (Hadamard directional derivative [Römisch, 2006, Shapiro, 1990]. Let D be a normed vector space and fix a non-empty set θ C D. The tangent cone to θ at 0 E θ is given by
On 0 for some On, E O O, + 0, 10.400 10.
(0) = h E D:
lim =0
In
A functional f + R is Hadamard directionally differentiable at E θ tangentially to θ if there exists map f/. (8) + R satisfying
f(O + hn - f(0) (b), tn
lim
(17)
12 100
for any h E To(0). tn 0, and bn + h in D with 0 + tnke e.
26

---

This notion of differentiability is compatible with distributional convergence of random elements of D in the sense that the following generalization of the delta method holds.
Lemma 10 (Functional delta method [Römisch, 2006, Shapiro, 1991]) Fix probability space (2, E. P) and let D be a normed vector space and J + C D + IR be Hadamard directionally differentialle at O ④ langenfially to (e) with derivative f' (a( RR. Let T2 Ω Q
be maps such that (n 0) 4 T for some norming sequence Vu. and measurable map T: Ω (0) c D. Then rn (T(T.) f())) 3 (T) and, if is convex, Tn - fb (Tn 0)) 0 in outer probability.
Whilst Lemma 10 is sufficient to derive limit distributions, bootstrap consistency typically requires the following notion of full Hadamard differentiability (see e.g. Theorem 23.9 in [Van der Vaart, 2000] or Theorem 3.9.11 in van der Vaart and Wellner, 1996). A functional f D - IR is said to be Hadamard differentiable at 0 tangentially to a vector subspace Du C if there exists continuous linear functional fg Do + I satisfying (17) for any h E (#).,, tn o 0, ta - 0, and h, - h D with 0 1, E θ. The following lemma enables a connection between full and directional Hadamard differentiability.
Lemma 11 (Lemma 2 in [Goldfeld et al., 2024b]). I O C D + R is Hadamard directionally differentiable at tangentially to (0) and fo linear on a subspace Do Te((), then is Hadamard differentiable at 9 tangentially to Do-
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction make the following claims: 1) we introduce statistic for accessing a notion of multivariate almost stochastic dominance 2) we establish CLT and bootstrapping for this statistic 3) we use this statistic to compare the relative performance of models. The first two points are addressed in Section 4 whereas Section 4.2 clarifies the testing framework and our approach is validated empirically in Section 5.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification: As we mention in the discussion section, a limitation of our work is that does not treat the case where choices are risk averse. However, we indicated how these situations can be addressed by combining our statistical testing framework with multivariate Lorenz order.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
28

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The complete proofs of all results are provided in the appendices along with short outline in the main text. All relevant assumptions are also clearly stated.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification; Algorithm 1, included in the appendix, provides the relevant pseudocode im- plement the proposed statistical framework. The descriptions of the numerical experiments provide all required details to reproduce the results provided, A complete code package is also included in the submission.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
29

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: As highlighted in the previous point, the code is included in the submission, and all relevant experimental details provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/ http///ii////deueeemeee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. htc//publc/gguueeeec Soaeee//ooSSubmisecc ciommisssoon wwwsnsolol.. onoolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All relevant experimental details are provided in the main text.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All experiments are provided with eroor bars and are computed using the bootstrap as justified by the main theoretical results.
Guidelines:
30

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided details the type of hardware and computation time needed in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eeuripp.eeeeeeeee cttpriipp.ccpppbiieeee htcppppicc/thiccsuddeeeeeeee
Answer: [Yes]
Justification: This work provides a theoretical framework for assessing statistical property of multivariate data. The application explored in this work is benchmarking of different models according to a vector of metrics. Such a framework can be broadly useful for identifying ethical concerns in models, but in itself poses such concerns.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
31

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
The statistical framework proposed in this paper is used to compare the outputs of different models based on metrics of interest. On own, this framework has no social impact.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
No new data or models were released in conjunction with this paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All relevant details regarding existing assets are provided in the main text.
Guidelines:
32

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets were released in conjunction with this paper.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer:
[NA]
Justification:
No crowdsourcing or research with human subjects was included in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing research with human subjects included in this paper.
33

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
34


Reviews:
[{"review_id": "7j4zR1fCda", "reviewer": "Reviewer_i6xT", "summary": "The paper studies the testing of multivariate stochastic dominance, i.e., deciding an order between two multivariate random variables. The authors generalized the notion of index of almost stochastic dominance, which is for uni-variate rv. The new index is based on regularized value of optimla transport problems. Convergence of plug-in estimate of that index, and its bootstrap theory is developed. As a main application, it is applied to LLM Benchmarking, where an LLM is evaluated on many metrics.", "questions": "- What does $\\hat \\mu$ $\\hat \\nu$ mean in the context of LLM benchmarking? Do we divide the dataset into multiple smaller datasets and evaluate LLM on them?\n- Typo at line 185. Should be \"whenever $OT_{\\bar h, \\lambda } = 0$\"", "limitations": "See above.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This generalized notion of stochastic order index for multivariate random variables is interesting. Its application to LLM benchmarking is also interesting. \n\n2. Many advanced tools from probability is used in the proof section. It is interesting to see that these theories find application in LLM evaluation.", "weaknesses": "1. There are two hyperparameters that need to be chosen to define the ratio statistics. The author should provide guidance on how to choose them.\n2. Can the authors discuss the computation complexity of computing the ratio statistics?\n3. The new notion of stochastic dominance is interesting. I think it would explain this new concept more if the authors could compile a list of examples that showcase this notion. For example the example in Sec 5.1 is intuitive. But it is too simplified. Since \"dominance\" is a complicated concept for multivariate distributions, explaining what kind of dominance is being captured and what not could be useful when practitioners apply the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful reading of the paper and for identifying sections which could use further clarification. This feedback is invaluable to us in improving the quality of our submission. We have corrected the typos identified and will further polish the paper to improve its readability.\n___\n**There are two hyperparameters that need to be chosen to define the ratio statistics. The author should provide guidance on how to choose them.**\n___\nFor the choice of hyperparameters, we recall that the parameter $\\lambda$ corresponds to the regularization strength in the entropic optimal transport distance, which is employed as a statistically efficient and computationally tractable proxy for the standard optimal transport distance. Given Theorem 1, it is desirable to set $\\lambda$ as small as possible in order to approximate the true optimal transport distance well, for example $\\lambda = 0.1$ as used in the numerical experiments. We remark that for practical purposes, $\\lambda$ cannot be chosen arbitrarily small due to possible underflow in the matrix $e^{-C/\\lambda}$, where $C$ is the matrix of pairwise costs, used when computing the entropic distance using Sinkhorn’s algorithm (see the following point for more details). Notably, Sinkhorn’s algorithm is a method tailored to positive matrices and cannot cope with zero entries. As such, a practitioner can start with a small value for $\\lambda$ and increase it gradually if numerical instability is encountered.\n\nAs for the choice of function, $h$, three examples are discussed in Example 1. For practical applications, the logistic function is preferred for ease of computation and since it satisfies the desired smoothness properties. As mentioned in Example 1 and demonstrated empirically in Section 5.1, the larger the gain, $\\beta$, of the logistic function is, the closer the logistic function is to being compatible with standard multivariate FSD as described in Definition 1. For the matrix $e^{-C/\\lambda}$ from the previous point to be well-conditioned, there is a tradeoff between $\\beta$ and $\\lambda$. In the experiments, these hyperparameters were set by first fixing the entropic parameter to $\\lambda = 0.1$, then $\\beta$ was set as large as possible subject to a computational budget constraint. Indeed, as described in the following point of this response, the number of iterations required for Sinkhorn’s algorithm to converge scales as $||C||_{\\infty}/\\lambda$.\n\nWe also highlight that the hyperparameter $\\varepsilon_0$ used when defining the notion of entropic multivariate almost FSD does not need to be set when performing relative testing; a discussion of this approach is included on lines 252 onwards. Notably, the empirical study we performed on ranking LLMs utilizes this relative testing framework to perform the rankings using the proposed ratio statistic and does not require one to set the parameter $\\varepsilon_0$.  \n\nWe will add these clarifications on the choice of hyperparameters to the appendix.\n___\n**Can the authors discuss the computation complexity of computing the ratio statistics?**\n___\nThis is indeed an important question, we acknowledge that a more complete discussion is warranted. The ratio requires computing two entropic optimal transport distances with regularization strength $\\lambda$ between two distributions on $\\mathbb R^d$ supported on $N$ points $x^{(i)},$$y^{(i)}$ $i=1,\\dots, N$.\n\nTo solve this problem numerically, we utilize the popular Sinkhorn scaling algorithm as implemented in the Python OT package. To compute one distance, we first construct a matrix of pairwise distances $C\\in\\mathbb R^{N\\times N}$ where $C_{ij}=c(x^{(i)},y^{(j)})$, which requires a one-time cost of  $N^2K(d)$ operations, where $K(d)$ is the complexity of computing the cost between two $d$-dimensional vectors), then an iterative scaling procedure is performed, which consists of iterated products of the fixed matrix $e^{-C/\\lambda}$ with two vector iterates so that each iteration runs in $O(N^2)$ time.  Following [1] Theorem 1, we see that the algorithm reaches its termination condition to a given precision $\\eta$ (the default value in the package used is 1e-9) in a number of iterations, $K$, bounded as \n$$\n\tK\\leq 2 +\\frac{-4\\log(e^{-\\|C\\|_{\\infty}/\\lambda }\\kappa)}{\\eta},\n$$\nwhere $\\kappa$ is the smallest value in the input probability vectors (remark that this quantity is at most $\\frac 1 N$ with equality when considering empirical distributions on distinct points). \n\nGiven that the ratio requires two evaluations of the entropic distance, the overall complexity is twice that of the above procedure. Moreover, if the numerator is computed first we can reuse the matrix C when computing the denominator’s pairwise cost matrix. \n\nWe will add a sentence to this effect in the main text and provide a thorough discussion in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W8VSONxPZW", "reviewer": "Reviewer_w1Lw", "summary": "In this paper authors propose a testing framework for  First order Stochastic Dominance (FSD) for multivariate random variables. To achieve this, the authors use ideas from optimal transport using Entropic Regularization to derive a hypothesis testing procedure for testing multivariate FSD. The proposed methodology includes theoretical results showing the distributional convergence of the test statistic.", "questions": "See above", "limitations": "The authors discuss the limitations in the discussion section.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper considers an important problem, which is well motivated. \n- Theoretical results back the methodology\n- The paper is well-presented", "weaknesses": "1. The experimental results do not present the Type I and II errors for the hypothesis testing methodology proposed. The authors should consider adding these results (with a synthetic setup if needed). \n2. The results should also investigate how these errors change with increasing $N$? And increasing $d$?\n\nOverall, I think more extensive empirical investigation is needed to show how fast the convergence takes place, and how well the methodology scales with increasing $d$ and $N$ (in terms of computational complexity)\n\n**Other comments:**\n\nLines 166-167:\n- Notation should be $OT_{h, 0}$ instead of $OT_{0, \\lambda}$\n- Kendall tau rank has not been defined\n- From the definition of $\\mathcal{E}_{\\mathcal{W}_2}$  below line 73, we should have that \n\n$\\mathcal{E}_{\\mathcal{W}_2}= 1$ \nwhen \n\n$F_Y^{-1}(t) \\leq F_X^{-1}(t)$ for a.e. t. However, line 77 states that in this case $\\mathcal{E}_{\\mathcal{W}_2} = 0$. I'm not sure if this is correct.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their support of this paper and for highlighting some additional experiments which would further clarify the performance of the method. We have run the proposed experiments and will add them to the appendix. We have also addressed the points from your other comments in the text.\n___\n**The experimental results do not present the Type I and II errors for the hypothesis testing methodology proposed. The authors should consider adding these results (with a synthetic setup if needed). The results should also investigate how these errors change with increasing $N$? And increasing $d$? Overall, I think more extensive empirical investigation is needed to show how fast the convergence takes place.**\n___\nWe agree that additional experiments would help to better illustrate the strengths and limitations of this approach. In line with your recommendations, we have computed the Type I and II errors for the synthetic  experiment in the paper and conducted an empirical study of how these errors vary as a function of $N$ and $d$. The results of these experiments are compiled in the attached pdf and demonstrate that the proposed testing methodology is sample efficient and scales well even in moderate dimensions.  As mentioned, these additional experiments will be added to the appendix.\n___\n**How well [does] the methodology scales with increasing $d$ and $N$ (in terms of computational complexity)**\n___\n\nThank you for bringing up this important point, we agree that the computational complexity of our approach should be more carefully discussed. We will add the following discussion to the appendix. \n\nThe ratio requires computing two entropic optimal transport distances with regularization strength $\\lambda$ between two distributions on $\\mathbb R^d$ supported on $N$ points $x^{(i)},y^{(i)}$, $i=1,\\dots,N$. \n\nTo solve this problem numerically, we utilize the popular Sinkhorn scaling algorithm as implemented in the Python OT package. To compute one distance, we first construct a matrix of pairwise distances $C\\in\\mathbb R^{N\\times N}$ where $C_{ij}=c(x^{(i)},y^{(j)})$, which requires a one-time cost of  $N^2K(d)$ operations, where $K(d)$ is the complexity of computing the cost between two $d$-dimensional vectors), then an iterative scaling procedure is performed, which consists of iterated products of the fixed matrix $e^{-C/\\lambda}$ with two vector iterates so that each iteration runs in $O(N^2)$ time.  Following [1] Theorem 1, we see that the algorithm reaches its termination condition to a given precision $\\eta$ (the default value in the package used is 1e-9) in a number of iterations, $K$, bounded as \n$$\n\tK\\leq 2 +\\frac{-4\\log(e^{-||C||_{\\infty}/\\lambda }\\kappa)}{\\eta},\n$$\nwhere $\\kappa$ is the smallest value in the input probability vectors (remark that this quantity is at most $\\frac 1 N$ with equality when considering empirical distributions on distinct points). \n\nGiven that the ratio requires two evaluations of the entropic distance, the overall complexity is twice that of the above procedure. Moreover, if the numerator is computed first we can reuse the matrix C when computing the denominator’s pairwise cost matrix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gQbXP4x8lg", "reviewer": "Reviewer_RTCD", "summary": "Insipred by the uni-dimensional case of First order Stochastical Dominance (FSD), the authors indtroduce multidimensional FSD which does away making approximations using aggregations and reductions of multi dimensional metrics and reduce the orderings to unidimensional case. This is done vi Optimal Transport framework but this comes with a caveat since, using empirical Optimal Transport framework in multi dimensional case suffers from the curse of dimensionality. This is dealt with using  approximations in form of entropic regularization using ideas from Cuturi 2013, to get Entropic Optimal Transport framework. \nTheorem 1 in the paper shows that the approximation error has a computable upper bound. The framework is tested on a simulated data and LLM metric evaluation case study.", "questions": "Some typos and writing mistakes\n1. Some references are repeated, - Del Barrio 2018\n2.  How to tune the hyperparameters: $\\beta$ and $\\lambda$ in Sec 5.2 ?\n3.  It will be helpful to clearly state that for Kendall tau statistic, lower values are better Fig 2.\n4. How reasonable it is to take consensus with ChatGPT as the ground truth ?", "limitations": "I am not an expert in theory and this paper is heavy on OT theory, and the readibility of the paper is low in many important places. It would have been great if the authors would have listed clearly what limitations do these methods bring, one can ofcourse be related to computation when using multidimenional statistics over aggregation to uni-dimensional case, at what dimension could the cost become too prohibitive ?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is great to see a method which can help LLM research community regarding proper evaluation and benchmarking. There is clearly a need for such metrics and methods, since most LLM's responses are highly stochastic, even when using same prompt and same parameters. \n2. It is now high time for LLM research community to turn to rigorous statistical analysis of their research methodology and this paper contributes to that so it is both timely and useful. \n3. The authors have applied the framework to two different domains: financial domain with portfolio allocation study and LLM evaluation.\n4. Clear comparison between OT and EOT frameworks, absoulute and relative tests.\n5. Figures are clear and support the claims made.", "weaknesses": "- Unclear writing in many parts of the paper which makes such a difficult technical paper harder to read and understand. I strongly advise the authors to take couple of polishing passes.\n- Line 287-290 unclear writing\n- Line 314-319 unclear\n- predicts linearly p, indicating that it is captures well \n- Figure 1 caption and title don't match.\n- Maybe in experiments use multiple LLMs and not just Chat GPT", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their support of this work. We take your feedback regarding unclear writing very seriously and will further polish the paper to minimize any ambiguities. We have corrected all of the typos you have identified and reworked the sections that were identified as unclear. \n\n___\n**How to tune the hyperparameters: $\\beta$ and $\\lambda$ in Sec 5.2?**\n___ \n\nThis is indeed an important point when implementing this methodology, we will expand on the discussion in the main text according to the following points.\n\nThe parameter $\\lambda$ corresponds to the regularization strength in the entropic optimal transport distance, which is employed as a statistically efficient and computationally tractable proxy for the standard optimal transport distance. Given Theorem 1, it is desirable to set $\\lambda$ as small as possible in order to approximate the true optimal transport distance well, for example $\\lambda = 0.1$ as used in the numerical experiments. We remark that for practical purposes, $\\lambda$ cannot be chosen arbitrarily small due to possible underflow in the matrix $e^{-C/\\lambda}$, where $C$ is the matrix of pairwise costs, used when computing the entropic distance using Sinkhorn’s algorithm. Notably, Sinkhorn’s algorithm is a method tailored to positive matrices and cannot cope with zero entries. As such, a practitioner can start with a small value for $\\lambda$ and increase it gradually if numerical instability is encountered.\n\nAs for the choice of $\\beta$, it is mentioned in Example 1 and demonstrated empirically in Section 5.1 that the larger the gain, $\\beta$, of the logistic function is, the closer the logistic function is to being compatible with standard multivariate FSD as described in Definition 1. For the matrix $e^{-C/\\lambda}$ from the previous paragraph to be well-conditioned, there is a tradeoff between $\\beta$ and $\\lambda$. In the experiments, these hyperparameters were set by first fixing the entropic parameter to $\\lambda = 0.1$, then $\\beta$ was set as large as possible subject to a computational budget constraint. Indeed, as described in the final point of this response, the number of iterations required for Sinkhorn’s algorithm to converge scales as $||C||_{\\infty}/\\lambda$.\n___\n**How reasonable it is to take consensus with ChatGPT as the ground truth ?**\n___ \n\nIn the experiments, the usage of ChatGPT for the purposes of evaluation is motivated by its use as a human proxy in the context of evaluating the quality of generated natural language  [2-5]. Such evaluations are common in the literature and we do not anticipate significant differences by using another equivalently sized LLM such as Claude, LLama2 70b etc. for the purpose of scoring. \n\nWe underscore that the advantage of our multivariate stochastic dominance over the ChatGPT-based scoring method or human evaluation is three-fold. First, it requires significantly less computational overhead as discussed in the following point. Second, there are no upfront monetary costs associated with evaluating the ratio statistic, this can be an important consideration for large-scale comparison tasks. Third, our approach can be run locally thereby eliminating the privacy concerns of exposing sensitive data on APIs running LLM as a judge, such as ChatGPT.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4CWEi4h4JP", "reviewer": "Reviewer_a9kp", "summary": "This paper considers the task of estimating the degree of stochastic dominance between two multivariate distributions. In the univariate setting, stochastic dominance is a useful tool for tasks such as benchmarking LLMs, where a practitioner may have estimates of some quality metric for responses. An efficient way to approximately estimate multivariate stochastic dominance could thus be helpful in setting where models may be evaluated on many metrics, rather than just a single value. The authors introduce a natural definition of near-stochastic dominance based on an existing definition in the univariate case and a connection to optimal transport, and show that while this notion may not allow for efficient estimation, adding an entropic regularization term allows for better guarantees. Through experiments, the authors demonstrate that their proposed metric converges to the true unregularized value in synthetic experiments, and for an LLM dataset, their approach outputs a ranking of models far more correlated with a \"ground-truth\" ranking.", "questions": "- Does your method also improve estimates of the degree of stochastic dominance (e.g. the first-ranked model is much better than the second-ranked model, but the second- and third-ranked models are comparable)? Is this something that can be demonstrated empirically?\n\n- If GPT-generated prompt comparisons are viewed as good enough to use as a ground-truth ranking, why not use this as the gold-standard benchmarking method? If not, is there a better baseline to compare the multivariate FSD approach to? \n\n- Moreover, is there a reason why we should expect a \"human\" ranking to match the true stochastic order induced by some set of metrics? It seems that in some cases we might actually want the order induced by a set of metrics to look somewhat different than what the average human might output as a ranking.", "limitations": "I think the authors adequately address the limitations of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written, and does a good job of explaining how their results rest on prior results in univariate stochastic domination and regularization methods. \n\n- The goal of obtaining good methods for estimating multivariate stochastic domination is very natural, and seems very relevant to many areas of practice, especially evaluating LLMs on multiple axes, as the authors mention.", "weaknesses": "- The experiments section is a bit confusing, in particular the explanations of the various approaches tested in the LLM experiment. The reasoning behind the choice of the ground-truth ranking is also a bit unclear to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments regarding the experiments. We will work to clarify the experimental setup and address the comments regarding comparison to ChatGPT. We believe that both of these are important to further improve the quality of our submission.\n___\n**The experiments section is a bit confusing, in particular the explanations of the various approaches tested in the LLM experiment. The reasoning behind the choice of the ground-truth ranking is also a bit unclear to me. \nIf GPT-generated prompt comparisons are viewed as good enough to use as a ground-truth ranking, why not use this as the gold-standard benchmarking method? If not, is there a better baseline to compare the multivariate FSD approach to?**\n___\n\nWe thank the reviewer for pointing out that the LLM benchmarking experiment should be more clearly explained to avoid ambiguities. We will amend this in the text. The experiment is conducted as follows:\n\n1. We use the dataset from [1], which compiles a training set of 100K prompts along with the resulting response from 12 different LLMs.\n\n2. For each prompt, the responses from the 12 different LLMs are evaluated according to 9 automatic metrics (e.g. BLEU, ROUGE, BERTScore, BARTScore, etc) and are stored as 12 vectors in $\\mathbb R^9$ $(x^{(i)})_{i=1}^{12}$, where $x^{(1)}$ is the vector containing the outputs of the 9 metrics for the 1st LLM and so on. $x^{(1)}$ is then viewed as a sample from the distribution of all possible responses from the 1st LLM evaluated according to these 9 metrics.\n\n3. This process is repeated for each of the 100K prompts so that one can construct the empirical measure $\\hat \\mu_N^{(1)}=\\frac{1}{N}\\sum_{i=1}^N\\delta_{x^{(1)}_i}$ for $N = 100K$ (here $\\delta_x$ is the Dirac measure at the point $x\\in\\mathbb R^9$).\n\n4. The empirical measures $\\hat \\mu_n^{(i)}$ and $\\hat \\mu_n^{(j)}$ for some $n\\leq 100K$ and $i\\neq j$ are then compared by computing the normalized index $\\varepsilon_{h,\\lambda}(\\hat \\mu_n^{(i)},\\hat \\mu_n^{(j)})=\\varepsilon_{ij}^{(h,\\lambda)}$. \n\n5. Given the pairwise ratios $\\varepsilon_{ij}^{(h,\\lambda)}$, we rank the 12 LLMS according to the relative testing procedure described in Section 4.2. The resampling procedure (bootstrap) is used to construct the confidence intervals. \n\n6. The results thus obtained are compared (via Kendall Tau Similarity) to a univariate ranking based on ChatGPT scoring i.e. ChatGPT is presented by the instruction and the response of each LLM as described in point 1 and produces a score that judges the quality of the response in terms of following the given instruction; the different models are then ranked according to their ChatGPT score (a univariate ranking) and compared to the multivariate ranking described above.\n\n\nIn the experiments, ChatGPT is being used for the purposes of evaluation only, motivated by its use as a human proxy in the context of evaluating the quality of generated natural language [1-4] . We show that the ranking resulting from multivariate stochastic dominance on automatic metrics correlates well with the ChatGPT ranking which, in turn, correlates well with human evaluation. The advantage of our multivariate stochastic dominance over ChatGPT or human evaluation is threefold. First, as discussed previously, it requires significantly less computational overhead. Second, there are no upfront monetary costs associated with evaluating the ratio statistic, this can be an important consideration for large-scale comparison tasks. Third, our approach can be run locally thereby eliminating the privacy concerns of exposing sensitive data on APIs running LLM as a judge, such as ChatGPT.\n\n___\n**Moreover, is there a reason why we should expect a \"human\" ranking to match the true stochastic order induced by some set of metrics? It seems that in some cases we might actually want the order induced by a set of metrics to look somewhat different than what the average human might output as a ranking.**\n___\nThis is a great and subtle point. The notion of an “ideal” ranking according to a given set of metrics depends largely on the preferences of the practitioner. One limitation with the approach discussed in the text is that all dimensions are treated equally, as the same function $h$ is used to compare each dimension. This may limit the utility of this methodology in applications where violations of the order in one particular dimension should be severely penalized (e.g. unsafe responses from LLMs). In such a case, one may modify the proposed framework to use a cost function of the form $c(x,y)=\\sum_{i=1}^d h_i(y_i-x_i)$ (i.e. we prescribed a different cost to violations in each dimension). Provided that each of the $h_i$ are nonnegative functions satisfying the smoothness condition, all of the theoretical results derived in the paper still go through. As such, a practitioner that wishes to formulate a domain-specific notion of almost stochastic dominance can adopt a data-driven approach to learn a new stochastic order tailored to the user’s preferences by letting the $h_i$ be parametrized costs (e.g. the logistic function) and optimizing over the parameters (in the previous example, the gain).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their careful reading of our submission. Their questions and comments have proved invaluable in improving the quality of the paper and helped us in identifying passages which were unclear and confusing. We briefly summarize the main comments brought up in the reviewers here and attach point by point detailed responses to each official review. 

* Q: **The complexity of computing the violation ratio is not stated explicitly. In particular, what is its dependence on the dimension $d$ when comparing two distributions on $N$ points in $\mathbb R^d$?**

A: The entropic optimal transport distances which are used to define the ratio are computed using Sinkhorn’s algorithm. The only step of this algorithm which depends on the dimension is the construction of the pairwise cost matrix $C$ which requires a one time cost of  $N^2K(d)$ operations, where $K(d)$ is the complexity of computing the cost between two $d$-dimensional vectors. The main loop of the algorithm consists of matrix-vector products between a fixed $N\times N$ matrix and $N$-dimensional vector iterates; the number of steps required for the loop to terminate is also characterized and independent of $d$.

* Q: **How should a practitioner set the various hyperparameters, $\lambda,\beta,\varepsilon_0$?**

A: The parameter $\lambda$ serves as the regularization strength for the entropic optimal transport distance. Given that the entropic distance is used as a computationally and statistically efficient proxy for the standard optimal transport distance, it is desirable to set $\lambda$ as small as possible. As Sinkhorn’s algorithm utilizes the matrix $e^{-C/\lambda}$, the ratio $(\max_{ij} C_{ij})/\lambda$ cannot be too large, otherwise numerical underflow will be encountered. If underflow occurs, Sinkhorn’s algorithm will fail, as it can only cope with positive matrices. 

As for $\beta$, the gain in the logistic cost function, it is discussed in the text and demonstrated in the experiments that $\beta$ should be taken as large as possible to best capture the notion of stochastic dominance. Given the discussion in the previous paragraph, there is a tradeoff regarding how large $\beta/\lambda$ can be for the purposes of numerical estimation. This ratio also figures in the number of Sinkhorn iterations required for convergence. As such, a practitioner may first fix a desired value of $\lambda$ (say $0.1$) and set $\beta$ to be large (but not so large as to cause underflow). If Sinkhorn iterations take too long to converge, the user may consider decreasing $\beta$ or increasing $\lambda$.

Finally, we underscore that, for the purpose of relative testing of models,  the threshold for multivariate entropic FSD, $\varepsilon_0$ does not need to be set.
 
* Q: **How reasonable is it to take consensus with ChatGPT as the ground truth?**

A: In the experiments, ChatGPT is being used for the purposes of evaluation as motivated by its use as a human proxy in the context of evaluating the quality of generated natural language [1-4] . It is shown that the ranking obtained via multivariate stochastic dominance on automatic metrics correlates well with the ChatGPT ranking which, in turn, correlates well with human evaluation. We underscore that our approach based on multivariate stochastic dominance is preferable to ChatGPT or human evaluation with regard to the following factors. First, as discussed previously, it requires significantly less computational overhead. Second, there are no upfront monetary costs associated with evaluating the ratio statistic, this can be an important consideration for large-scale comparison tasks. Third, our approach can be run locally thereby eliminating the privacy concerns of exposing sensitive data on APIs running LLM as a judge, such as ChatGPT.

* Q: **What notions of dominance can be captured using this methodology? If a practitioner wishes to implement a notion of stochastic ordering tailored to a particular application, can they do so using this methodology?**

One limitation of our approach is that all dimensions are treated equally as the same function $h$ is used to compare each dimension. This may limit the utility of this methodology in applications where violations of the order in one particular dimension should be severely penalized (e.g. unsafe responses from LLMs). However, the proposed framework can be adapted to use a cost function of the form $c(x,y)=\sum_{i=1}^d h_i(y_i-x_i)$ (i.e. we prescribed a different cost to violations in each dimension) provided that the $h_i$ satisfy the technical conditions described in the paper. With this, a practitioner aiming to formulate a domain-specific notion of almost stochastic dominance can adopt a data-driven approach to learn a new stochastic order tailored to the user’s preferences by letting the $h_i$ be parametrized costs (e.g. the logistic function) and optimizing over the parameters (in the previous example, the gain).


[1] Hada, R., Gumma, V., de Wynter, A., Diddee, H., Ahmed, M., Choudhury, M., ... & Sitaram, S. (2023). Are large language model-based evaluators the solution to scaling up multilingual evaluation?. arXiv preprint arXiv:2309.07462.

[2] Jiang, D., Ren, X., & Lin, B. Y. (2023). Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561.

[3] Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023). G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634.

[4] Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. and Zhang, H. (2024). Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36.

Meta-review: There is a general consensus among reviewers that this paper, although it could benefit from a better presentation and slightly more thorough experiments, should be accepted, as it provides a novel and sound approach to the important problem of estimating the degree of stochastic dominance between multivariate distributions. I concur with this assessment and recommend acceptance.
Decision: Accept (poster)