Title: Multivariate Stochastic Dominance via Optimal Transport and Applications to Models Benchmarking
Venue: NeurIPS 2024
Accepted: yes
Keywords: Optimal Transport

Stochastic dominance

hypothesis testing

Central limit theorem

LLM benchmarking
Abstract: Stochastic dominance is an important concept in probability theory, econometrics and social choice theory for robustly modeling agents' preferences between random outcomes.  While many works have been dedicated to the univariate case,
little has been done in the multivariate scenario, wherein an agent has to decide between different multivariate outcomes. By exploiting a  characterization  of  multivariate first stochastic dominance in terms of couplings, we  introduce  a statistic that assesses multivariate almost stochastic dominance under the framework of Optimal Transport  with a smooth cost. Further, we introduce an entropic regularization of this statistic, and establish a central limit theorem (CLT) and consistency of the bootstrap procedure for the empirical statistic. Armed with this CLT, we propose a hypothesis testing framework as well as an efficient implementation using the Sinkhorn algorithm. We showcase our  method in comparing and benchmarking Large Language Models that are evaluated on multiple metrics. Our multivariate stochastic dominance test allows us to capture the dependencies between the metrics in order to make an informed and statistically significant  decision on the relative performance of the models.

Multivariate Stochastic Dominance via Optimal Transport and Applications to Models Benchmarking

Abstract

Stochastic dominance is an important concept in probability theory, econometrics and social choice theory for robustly modeling agents' preferences between random outcomes. While many works have been dedicated to the univariate case, little has been done in the multivariate scenario, wherein an agent has to decide between different multivariate outcomes. By exploiting a characterization of multivariate first stochastic dominance in terms of couplings, we introduce a statistic that assesses multivariate almost stochastic dominance under the framework of Optimal Transport with a smooth cost. Further, we introduce an entropic regularization of this statistic, and establish a central limit theorem (CLT) and consistency of the bootstrap procedure for the empirical statistic. Armed with this CLT, we propose a hypothesis testing framework as well as an efficient implementation using the Sinkhorn algorithm. We showcase our method in comparing and benchmarking Large Language Models that are evaluated on multiple metrics. Our multivariate stochastic dominance test allows us to capture the dependencies between the metrics in order to make an informed and statistically significant decision on the relative performance of the models.

1 Introduction

In choice theory, economics, finance, and models benchmarking, agents are faced with stochastic prospects that are (eventually) multivariate random variables which they wish to order according to a utility or risk measure of interest. To formalize such a notion of ordering of stochastic quantities, the concept of stochastic dominance can be utilized.

In the univariate case, a standard notion of order is given by First order Stochastic Dominance (FSD) which can be expressed in terms of the quantiles of the underlying random variables [Ogryczak and Ruszczynski, 2002]. To wit, a random variable X dominates another random variable Y in FSD, if it has larger quantiles than Y across all percentiles. A weaker notion of FSD, called almost stochastic dominance was introduced in del Barrio et al. [2018]. Their approach is based on optimal transport (OT) and consists of measuring a ratio which quantifies how close X is to dominating Y . del Barrio et al. [2018] further lay the groundwork for principled statistical analysis of almost FSD by establishing a central limit theorem for the empirical ratio as well as consistency of the bootstrap. Dror et al. [2018], Ulmer et al. [2022], Nitsure et al. [2023] used the almost FSD testing framework in benchmarking Large Language models to make statistically significant decisions regarding which model to select when these models are evaluated with a metric of interest on test data.

Our main motivation is to extend the testing framework for almost FSD of del Barrio et al. [2018] to the multivariate case as to enable applications with dependencies between metrics. For instance, the problem of multivariate portfolio selection in financial applications has been treated via a reduction to univariate orders [Kouaissah, 2021]. Another application of interest is that of multimetric benchmarking which is central nowadays in ranking and selecting Large Languages Models [Bommasani et al., 2023, Chang et al., 2023, Huang et al., 2023, MosaicLM, 2023, Wolf, 2023, Zhang and Hardt, 2024]. Current approaches such as Nitsure et al. [2023] use aggregation techniques to reduce the ordering to the univariate case thereby ignoring dependencies between metrics.

Our starting point is the so-called standard multivariate stochastic order (see Chapter 6 in Shaked and Shanthikumar, 2007). This order can be expressed in terms of couplings between random vectors. This insight allows us to follow the approach of del Barrio et al. [2018] and define an almost multivariate FSD via OT in Section 2. This notion of stochastic dominance can be defined using multivariate violation ratios that are expressed as optimal transport problems with smooth costs [Manole and Niles-Weed, 2024, Hundrieser et al., 2022, Groppe and Hundrieser, 2023]. Given that empirical OT suffers from the curse of dimensionality, we resort in Section 3 to entropic regularization [Cuturi, 2013] to alleviate that issue and hence define Entropic Multivariate Violation Ratios. We establish in Section 3 convergence of these entropic violation ratios as the regularization parameter goes to zero, as well as a central limit theorem and bootstrapping consistency in Section 4 using the functional delta method [Römisch, 2006]. We highlight that the delta method has seen general success for proving limit theorems with entropic OT [Hundrieser et al., 2024, Goldfeld et al., 2024a,b]. Armed with this theory, we propose a new framework for hypothesis testing of multivariate stochastic dominance and apply it to multi-metric benchmarking of LLMs. Multivariate FSD captures the dependencies between the metrics in this setting and leads to a more robust ordering.

Notation The indicator function of a set A ⊂ R d is denoted ✶ A ( x ) and takes the value 1 if x ∈ A and 0 otherwise. We also adopt the following shorthand notation, R + = [0 , ∞ ) , the maximum of two numbers a, b ∈ R is denoted by a ∨ b , and for vectors x, y ∈ R d , we write x ≤ y to indicate that x i ≤ y i for every i ∈ { 1 , . . . , d } .

Throughout, P ( R d ) is the set of all probability measures on R d . A measure η ∈ P ( R d ) is said to be sub-Gaussian with parameter τ 2 ≥ 0 with respect to the 1 -norm provided E η [ exp( ∥ X ∥ 2 1 / 2 τ 2 ) ] ≤ 2 . Convergence in distribution of random variables is denoted by d → (in the sense of Hoffmann-Jørgensen when necessary, see Chapter 1 in van der Vaart and Wellner, 1996).

2 Optimal transport and stochastic order

2.1 FSD and Almost Stochastic Dominance in One Dimension

To properly motivate our results on multivariate FSD, we first review some theory for the univariate setting. For random variables X,Y , it is said that X dominates Y in the stochastic order (denoted X ≽ FSD Y ) if P ( X ≤ t ) ≤ P ( Y ≤ t ) for every t ∈ R . Formally, this means that the inequality Y ≤ X generally holds for a given instantiation of these random variables. This condition can be cast, equivalently, as F -1 Y ( t ) ≤ F -1 X ( t ) for every t ∈ (0 , 1) , where F -1 X ( t ) and F -1 Y ( t ) are the quantile functions for X and Y respectively. With this formulation in mind, del Barrio et al. [2018] propose the following index of almost stochastic dominance;

µ = law( X ) , ν = law( Y ) , and, for z ∈ R , ( z ) 2 + = (0 ∨ z ) 2 denotes the squared hinge loss. Here, the numerator captures the degree to which X fails to dominate Y whereas the denominator serves as a normalizing constant so that ε W 2 ( µ,ν ) ∈ [0 , 1] . Indeed, as ( x -y ) 2 + +( y -x ) 2 + = ( x -y ) 2 , we see that ε W 2 ( µ, ν ) = 0 precisely when F -1 Y ( t ) ≤ F -1 X ( t ) for a.e. t ∈ (0 , 1) whereas ε W 2 ( µ, ν ) = 1 when the opposite inequality holds. del Barrio et al. [2018] then propose to test the null hypothesis ε W 2 ( µ, ν ) ≤ ε 0 for some ε 0 sufficiently close to 0 , corresponding to the case where X almost dominates Y in the stochastic order, versus the alternative hypothesis ε W 2 ( µ, ν ) > ε 0 . To this end, they provide a central limit theorem for the statistic ε W 2 and propose to construct the rejection region via bootstrap estimation of the limiting variance. Similar results were obtained for a notion of almost second order stochastic dominance in Nitsure et al. [2023].

We highlight that the index ε W 2 can be connected to OT. Indeed, by Theorem 2.9 in Santambrogio [2015], the numerator and the denominator in ε W 2 ( µ, ν ) can be written, respectively, as inf π ∈ Π( µ,ν ) ∫ ( y -x ) 2 + dπ ( x, y ) , and inf π ∈ Π( µ,ν ) ∫ ( y -x ) 2 dπ ( x, y ) , where Π( µ, ν ) denotes the set of all couplings of ( µ, ν ) . These problems are (univariate) instances of the well-studied OT problem

where c : R d × R d → R + is a given cost function. We highlight that the costs considered herein are such that an optimal coupling always exists (see Theorem 4.1 in Villani, 2009). This connection will serve as our basis for extending the notion of almost stochastic dominance to the multivariate setting.

2.2 Multivariate FSD and its Relaxation via Optimal Transport with Compatible Costs

In the sequel, we provide a general framework for assessing multivariate almost FSD using a purely OT-based methodology. Following Chapter 6 and Theorem 6.B.1. in Shaked and Shanthikumar [2007], given the random vectors X,Y ∈ R d , we say that X dominates Y in the usual stochastic order (denoted X ≽ FSD Y ) provided that there exists a coupling ( ˆ X, ˆ Y ) of ( X,Y ) satisfying P ( ˆ X ≥ ˆ Y ) = 1

(i.e. for each i = 1 , . . . , d , X ≥ Y with probability one). This condition can be cast as follows:

i i Lemma 1. Letting µ (resp. ν ) denote the law of X (resp. Y ), X ≽ Y if OT c ( µ, ν ) = 0 , where c : R d × R d → R + is the cost function c ( x, y ) = ✶ { x ≤ y } ( x, y ) Evidently, the cost ✶ { x ≤ y } ( x, y ) in Lemma 1 can be replaced by any nonnegative cost function c ( x, y ) satisfying c ( x, y ) = 0 if and only if x ≤ y and it still holds that X ≽ FSD Y if OT c ( µ, ν ) = 0 . We denote the set of all such costs by C ≤ .

ˆ ˆ FSD .

Definition 1 (OT Costs Compatible with Multivariate FSD) . The set of all cost functions which are compatible with multivatiate FSD in the sense that OT c (law( X ) , law( Y )) = 0 implies that X ≽ Y

FSD d d + .

A simple recipe for generating cost functions in C ≤ is to take any univariate function h : R → R + with the property that h -1 ( { 0 } ) = ( -∞ , 0] and define c ( x, y ) = ∑ d i =1 h ( y i -x i ) . For notational simplicty, we write OT h to denote the OT cost with this type of cost function even when the aforementioned property does not hold. The results presented in the following sections require some additional smoothness assumptions on the function h discussed above which we summarize presently.

Definition 2 (Smooth Costs) . The function h : R → R + satisfies the smoothness condition (SC d ) if h is Lipschitz continuous with constant L ≥ 0 (that is, | h ( x ) -h ( y ) | ≤ L | x -y | for every x, y ∈ R ) and, for k = ⌊ d / 2 ⌋ + 1 , h is k -times continuously differentiable with derivatives of order s ≤ k satisfying | h ( s ) ( x ) | ≤ C k (1 + | x | ) p k for some C k < ∞ and p k > 1 which may depend on k .

We now discuss some examples of cost functions of interest.

Example 1 (Examples of OT Costs) . 1) The function h ( z ) = e -1 / z for z ∈ (0 , ∞ ) and 0 otherwise is known to be smooth (see Example 1.3 in Tu, 2011) and satisfies h -1 ( { 0 } ) = ( -∞ , 0] . It is easy to see that all derivatives of h are 0 on ( -∞ , 0] , and decay to 0 at infinity (and hence are bounded on R ) so that h satisfies (SC d ) for any d ∈ N , and induces a cost function in C ≤ .

2) The squared hinge function h ( z ) = ( z ) 2 + considered in Section 2.1 has linear growth, but is nonsmooth. Although this function can be smoothed using e.g. mollification as introduced in Friedrichs [1944], this will result in a cost c which is not an element of C ≤ and may be costly to implement due to the convolution operation used in mollification.

3) The logistic function h ( z ) = log(1 + e βz ) for β > 0 has linear growth and derivative h ′ ( z ) = β e βz 1+ e βz = βς ( βz ) , where ς ( z ) = 1 1+ e -z is the sigmoid function. As ς ′ ( z ) = ς ( z )(1 -ς ( z )) , it is easy to see that all derivatives of h are bounded on R so that the assumption (SC d ) is satisfied for any d ∈ N . Although h does not induce a cost in C ≤ , it is increasing and decays to 0 faster than e βz as z → -∞ . Moreover, if the induced cost satisfies c ( x, y ) ≤ ε 0 , then max d i =1 ( y i -x i ) ≤ 1 β log( e ε 0 -1) . Thus, if OT c ( µ, ν ) = ∫ { c ≤ ε 0 } cdπ ⋆ + ∫ { c>ε 0 } cdπ ⋆ = ε for an optimal plan π ⋆ and

some small ε > 0 , one has that ∫ { c>ε 0 } cdπ ⋆ ≤ ε so that π ⋆ ( { c > ε 0 } ) ≤ ε / ε 0 i.e. π ⋆ assigns at least mass 1 -ε / ε 0 to points ( x, y ) for which max d i =1 ( y i -x i ) ≤ 1 β log( e ε 0 -1) . Hence, for large β , c enforces similar properties to a cost in C + . Note also that h can be viewed as a smooth surrogate for the 0 / 1 loss for large β .

At this point, a multivariate analogue to the univariate almost stochastic domination could be defined by analogy with Section 2.1. However, we highlight two major impasses which make the entropically regularized index considered in the following sections a far more palatable option in dimension d > 1 . First, it is well-known that the expected rate of convergence of empirical OT generally suffers from the curse of dimensionality in statistical estimation; scaling as n -1 / d (cf. e.g. Manole and Niles-Weed, 2024). Although Hundrieser et al. [2022] improve these rates as to depend on the minimum of the intrinsic dimensions of µ, ν in place of d , entropic optimal transport exhibits a preferable parametric rate of convergence. Next, solving the OT problem numerically between two finitely discrete distributions supported on N points requires solving a linear program in N 2 variables which can be prohibitive for even moderately sized problems.

3 Entropic Regularization of OT with Multivariate FSD Compatible Costs

Before defining the regularized index, we first provide some background on entropic optimal transport (EOT) with a cost c : R d × R d → R + . EOT is defined by regularizing the OT problem (1) as

where λ ≥ 0 is a regularization parameter and D KL is the Kullback-Leibler divergence defined by D KL ( ρ, η ) = ∫ log ( dρ dη ) dρ if ρ is absolutely continuous with respect to η and D KL ( ρ, η ) = + ∞ otherwise. When λ = 0 , we recover the standard OT problem. If c ∈ L 1 ( µ ⊗ ν ) , (2) admits a unique solution and is paired in strong duality with the problem

Solutions to (3) are known to be almost surely unique up to additive constants (i.e. if ( φ, ψ ) , ( φ ′ , ψ ′ ) solve (3), φ = φ ′ + C µ -almost surely and ψ = ψ ′ -C ν -almost surely for some constant C ∈ R ) and are uniquely determined for µ -a.e. x and ν -a.e. y by the so-called Schrödinger system

which implies that ∫ e φ ( x )+ ψ ( y ) -c ( x,y ) λ dµ ⊗ ν ( x, y ) = 1 . EOT potentials satisfying (4) on the whole space are known to exist and are unique up to additive constants (see Lemma 6). We refer the reader to Nutz [2021] for a comprehensive introduction on EOT including all stated results.

Entropic regularization of OT problems was introduced in the seminal work of Cuturi [2013] as a means to accelerate computation by utilizing Sinkhorn-Knopp's matrix scaling algorithm which can be efficiently implemented on GPUs. Interestingly, entropic regularization also alleviates the curse of dimensionality rates in statistical estimation inherent to standard OT; for instance Genevay et al. [2019] and Mena and Niles-Weed [2019] show that the plug-in estimator for the EOT cost with fixed λ > 0 and cost c ( x, y ) = ∥ x -y ∥ 2 achieves a parametric expected rate of convergence with a dimension dependent constant (see also Groppe and Hundrieser, 2023, Stromme, 2023 for related results with the dimension replaced by the minimum intrinsic dimension of µ and ν ); del Barrio et al. [2023] and Goldfeld et al. [2024b] further establish a central limit theorem in this setting.

Given that EOT is meant to approximate OT, we derive the properties of OT h,λ ( µ, ν ) and its solutions as λ ↓ 0 . First, we quantify the rate of convergence of OT h,λ to OT h, 0 under mild conditions, then show that solutions of OT h,λ converge to solutions of OT h, 0 as λ ↓ 0 in a suitable sense.

Theorem 1 (Stability as λ ↓ 0 ) . Let µ, ν ∈ P ( R d ) have finite first moment, h be a function satisfying (SC d ) , and fix δ > 0 . Then,

1. for any λ ∈ [0 , 1) satisfying ⌊ λ -d ⌋ ≥ ( 20 / δ ) d , we have that

if π λ is the unique solution to OT h,λ ( µ, ν ) for λ > 0 , then there exists a subsequence of ( π λ ) λ ↓ 0 which converges weakly to a solution of OT h, 0 ( µ, ν ) .

We highlight that the implications of Theorem 1 require only the Lipschitz condition imposed in (SC d ) . The proof of the first result follows that of Theorem 3.3 in Eckstein and Nutz [2023] which controls the error of approximating the OT cost and OT plan using discretizations of the measures at play. The main novelty in our approach is to provide explicit constants and a simple argument showing that the rate at which a general measure on R d can be approximated by a finitely discrete measure on at most n points under the 1 -Wasserstein distance scales at worst as n -1 / d for sufficiently large n . The second statement is proved using the machinery of Γ -convergence (see Maso, 1993). Complete proofs are provided in Appendix C.2.

4 Entropic Multivariate FSD Violation Ratio and Testing

By analogy with the univariate case described in Section 2.1, we consider a normalized index of stochastic order violation given by

we adopt the convention that ε h,λ ( µ, ν ) = 0 whenever OT ¯ h,λ ( µ, ν ) = 0 . Here, OT ¯ h,λ ( µ, ν ) is the EOT problem with cost c ( x, y ) = ∑ d i =1 h ( y i -x i ) + h ( x i -y i ) . This cost function is induced by the function ¯ h ( z ) = h ( z ) + h ( -z ) and hence satisfies (SC d ) provided that h satisfies (SC d ) . Moreover, OT h,λ ( µ, ν ) ≤ OT ¯ h,λ ( µ, ν ) by construction so that ε h,λ ( µ, ν ) ∈ [0 , 1] yielding a normalized index. The corresponding notion of entropic multivariate almost stochastic dominance can thus be defined.

Definition 3 (Entropic Multivariate Almost FSD) . We define ( h, λ, ε 0 ) -FSD, the entropic multivariate almost FSD via the violation ratio as follows:

In light of Theorem 1, lim λ ↓ 0 ε h,λ ( µ, ν ) = OT h, 0 ( µ,ν ) OT ¯ h, 0 ( µ,ν ) with the convention that this latter quantity is zero when OT ¯ h, 0 ( µ, ν ) = 0 . In the case that h is the squared hinge function described in Example 1 and d = 1 , we recover the univariate index from Section 2.1. As aforementioned, the squared hinge function is not sufficiently smooth to enable us to characterize the asymptotic fluctuations of the empirical index in arbitrary dimensions. See Example 1 which lists other examples of costs satisfying (SC d ) ; this condition is sufficient for the following statistical developments. We underscore that the choice of cost influences the notion of stochastic dominance reflected by the violation ratio. In applications where a practitioner wishes to formulate a domain-specific notion of dominance, a data-driven approach can be employed by replacing the fixed cost h in the previous development by a collection of costs ( h i,θ i ) d i =1 for each dimension where θ i is the corresponding parameter (e.g. β in the logistic function) and optimizing over these parameters. The results presented herein readily adapt to this setting, but, for simplicity, we restrict our attention to the case of a fixed h throughout.

4.1 Statistical Properties

Wenowlay the groundwork for performing principled statistical inference with the empirical estimator of the entropic index ε h,λ . Namely, we establish the asymptotic properties of the plug-in estimator ε h,λ (ˆ µ n , ˆ ν n ) , where ˆ µ n = 1 n ∑ n i =1 δ X i , ˆ ν n = 1 n ∑ n j =1 δ Y j are the empirical distributions from n independent observations, ( X i ) n i =1 and ( Y j ) n j =1 of µ and ν respectively. Furthermore, we establish consistency of the bootstrap procedure. To this end, given sets of n iterations observations of µ and ν , ( X i ) n i =1 and ( Y j ) n j =1 as above and sets ( X B i ) n i =1 and ( Y B j ) n j =1 of n independent samples from

ˆ µ n and ˆ ν n , ˆ µ B n := 1 n ∑ n i =1 δ X i B and ˆ ν B n := 1 n ∑ n j =1 δ Y j B are the corresponding bootstrap empirical distributions. P B denotes the conditional probability given the data.

Theorem 2 (Limit distribution and bootstrapping) . Assume that µ, ν ∈ P ( R d ) are sub-Gaussian with a shared parameter τ 2 > 0 and that h satisfies (SC d ) . Let ( φ h , ψ h ) and ( φ ¯ h , ψ ¯ h ) be any pairs of optimal potentials for OT h,λ ( µ, ν ) and OT ¯ h,λ ( µ, ν ) respectively satisfying the Schrödinger system (4) on R d × R d . Then, if OT ¯ h,λ ( µ, ν ) > 0 ,

Observe that the limiting distribution in Theorem 2 is non-pivotal in the sense that the variance depends on the population distributions ( µ, ν ) rendering direct estimation of the limiting variance highly non-trivial. The bootstrap consistency result in the second point enables us to establish confidence intervals for ε h,λ ( µ, ν ) . Explicitly, if ζ β denotes the smallest value of t ∈ R for which P B ( ε h,λ (ˆ µ B n , ˆ ν B n ) ≤ t ) ≥ 1 -β for any β ∈ (0 , 1) , then ε h,λ ( µ, ν ) ∈ [0 , 2 ε h,λ (ˆ µ n , ˆ ν n ) -ζ α ] with probability approaching 1 -α for any α ∈ (0 , 1) due to Lemma 23.3 in Van der Vaart [2000].

The proof of Theorem 2 is based on the functional delta method [Römisch, 2006] which extends the standard delta method to functionals defined on normed vector spaces, following the framework of Goldfeld et al. [2024b]. Formally, this approach consists of showing that the functional mapping τ 2 -sub-Gaussian distributions ( η, ρ ) to ε h,λ ( η, ρ ) is directionally differentiable at ( µ, ν ) and Lipschitz continuous in a suitable sense and that the relevant potentials lie in a space of sufficiently smooth functions using the assumption (SC d ) . Smoothness of the potentials is crucial to ensure that the empirical processes √ n (ˆ µ n -µ ) , √ n (ˆ ν n -ν ) converge when treated as functionals on the aforementioned space of smooth functions. Complete details are included in Appendix C.3, and a primer on the functional delta method can be found in Appendix D.

Remark 1 (On Theorem 2) . 1) We note that the condition that OT ¯ h,λ ( µ, ν ) > 0 in Theorem 2 is satisfied except in certain degenerate settings. Indeed, for a general nonnegative cost c , OT c,λ ( µ, ν ) = 0 if and only if ∫ cdµ ⊗ ν = 0 as follows from the fact that D KL ( π ∥ µ ⊗ ν ) ≥ 0 with equality if and only if π = µ ⊗ ν . In particular, OT ¯ h,λ ( µ, ν ) = 0 if and only if h ( x i -y i ) = h ( y i -x i ) = 0 for every x ∈ spt( µ ) and y ∈ spt( ν ) and every i ∈ { 1 , . . . , d } . If h is chosen as to generate a cost function which is compatible with multivariate FSD (recall Definition 1), h -1 ( { 0 } ) = ( -∞ , 0] so that OT ¯ h,λ ( µ, ν ) = 0 if and only if µ and ν are point masses at some shared a ∈ R d .

̸

2) Theorem 2 is presented in the balanced case with empirical measures from n samples. In the case where ˆ µ n and ˆ ν m are empirical measures from n = m samples with n n + m → s ∈ (0 , 1) , the implications of Theorem 2 are easily seen to hold with √ nm n + m in place of √ n and σ 2 s = var µ ( 1 -s OT ¯ h,λ ( µ,ν ) φ h -s OT h,λ ( µ,ν ) OT ¯ h,λ ( µ,ν ) 2 φ ¯ h ) +var ν ( 1 -s OT ¯ h,λ ( µ,ν ) ψ h -s OT h,λ ( µ,ν ) OT ¯ h,λ ( µ,ν ) 2 ψ ¯ h ) in place of σ 2 .

4.2 Multivariate FSD Hypothesis Testing In ML Models Benchmarking

With the statistical properties of the violation ratio in hand, we now consider using the violation ratio in the context of statistical testing for ( h, λ, ε 0 ) -FSD. Consider two d -dimensional distributions µ , ν . In our application, these will correspond to the distributions of performance of two language models' responses evaluated on d metrics. Given n, m samples from µ , ν respectively, we can apply Theorem 2 to create statistically valid tests comparing µ and ν . Similarly to Nitsure et al. [2023], we consider both absolute and relative testing (see Nitsure et al., 2023 for a complete discussion).

Absolute testing The most straightforward application of Theorem 2 is to specify a desired threshold ε 0 and consider the following hypothesis test for ( h, λ, ε 0 ) -FSD: H 0 : µ glyph[a0] ≽ ( h,λ,ε 0 ) -FSD ν versus the alternative H 1 : µ ≽ ( h,λ,ε 0 ) -FSD ν . Note that ν dominating µ would be tested separately. Given a desired

confidence 1 -α , the central limit theorem and bootstrap results in Theorem 2 suggest rejecting H 0 if

where σ 2 B is the bootstrapped variance (See Algorithm 1) and Φ is the CDF of the standard normal distribution. By Theorem 2, this test will be asymptotically valid.

̸

Relative testing A downside of absolute testing is that it requires specifying a threshold ε 0 . This threshold can be meaningful in pairwise comparisons, but when multiple distributions (e.g. multiple language models evaluations) are being compared for ranking purposes, it is difficult to determine a priori what threshold to use to ensure that the distributions can be separated. As in Nitsure et al. [2023], we therefore also present a relative test that compares each of k random vectors with measures µ 1 , . . . , µ k using a one-versus-all violation ratio. First, consider all pairs of violations ratios between the k measures: ε ( h,λ ) ij = ε h,λ ( µ i , µ j ) for i, j ∈ { 1 . . . k } , i = j. Let M = ( µ 1 , . . . µ k ) , and define the one-versus-all violation ratio of the dominance of µ i on all other variables µ j , j = i : ε ( h,λ ) i ( M ) = 1 k -1 ∑ j = i ε ( h,λ ) ij . We can then define relative stochastic dominance ( h, λ ) -R-FSD as µ i 1 ≽ R -FSD µ i 2 . . . ≽ R -FSD µ i k ⇐⇒ ε ( h,λ ) i 1 ( M ) ≤ · · · ≤ ε ( h,λ ) i k ( M ) . Here the most dominating model is the one with the lowest one-versus-all violation ratio. Testing for relative dominance of µ i on µ j we can then compare their one-versus-all ratios via the following statistic: ∆ ε ( ℓ ) ij ( M ) = ε ( ℓ ) i ( M ) -ε ( ℓ ) j ( M ) . To test for ( h, λ ) -R-FSD of µ i versus µ j then, we have the null hypothesis H 0 : ∆ ε ij ( M ) ≥ 0 versus the alternative H 1 : ∆ ε ij ( M ) < 0 . It is possible to extend the central limit theorem and bootstrapping results in Theorem 2 to this relative statistic under an independence assumption (omitted for brevity). Let ˆ M n = (ˆ µ 1 ,n , . . . ˆ µ k,n ) be the empirical measures for n samples from each distribution. As in the absolute case, we then reject H 0 with a confidence 1 -α if: ∆ ε i 1 ,i 2 ( ˆ M n ) ≤ √ 1 n σ B, relative ( i 1 , i 2 )Φ -1 ( α ) where σ 2 B, relative ( i 1 , i 2 ) is the bootstrapped variance (see Algorithm 1 for the variance expression).

̸

Multitesting and Ranking To apply the above multivariate FSD violation ratio hypothesis tests to ranking of multiple distributions, we follow the approach of Nitsure et al. [2023]. We aggregate the set of all pairwise tests, ensure multitesting statistical validity via Family-Wise Error Rate (FWER) control, and, if needed, aggregate the pairwise test results into a numerical ranking. Our approach is described below and summarized in Algorithm 1. The overall complexity of performing one pairwise test is dominated by the cost of computing the EOT cost with h and ¯ h . To this end, the Sinkhorn algorithm is used [Cuturi, 2013], which computes EOT between distributions on N points with a complexity of O ( N 2 K ( d ) + N 2 ) , where K ( d ) denotes the cost of complexity of computing the cost c ( x, y ) between x, y ∈ R d .

̸

Ranking multiple distributions In our experiments below, we seek to use the pairwise tests above to obtain a statistically valid ranking of a set of k random vectors X ( i ) with measures µ ( i ) , e.g. samples of X ( i ) can be the per-sample evaluation metrics for a set of language models. To rank k models at a specified significance level α , we first test all k 2 -k pairs ( µ ( i ) , µ ( j ) ) , i = j , employing a FWER correction (see next paragraph) to guarantee a valid control on the overall false rejection rate. This yields a set of trinary outcomes for each ( i, j ) pair, with 1 if the null is rejected in the positive direction, -1 if the null is rejected in the negative direction, and 0 if the null is not rejected. These pairwise rankings are then combined into a single rank using a simple Borda count [de Borda, 1781] rank aggregation algorithm.

Multitesting FWER control When running a family of T tests each at a significance level 1 -α , the true probability that at least one test falsely rejects the null scales with T . If the output of all T tests needs to be trusted simultaneously, instead it is desirable that the probability of any test falsely rejects the null is less than or equal to the specified α . Achieving this requires adjusting, or 'correcting' the significance level of each of the T tests, in a process called Family-Wise Error Rate (FWER) control. In the present work, we use the Bonferroni correction, which sets the significance level of the i th test to 1 -α i with α i = α/T . Note that while the Bonferroni correction is known to be pessimistic, we choose it as it sets uniform significance levels for all tests, as opposed to other strategies such as the Holm correction [Holm, 1979] which is tighter but yields highly nonuniform statistical power across the family of tests. Exploring sensible ways to employ nonuniform FWER control approaches in the context of performance ranking is an interesting avenue for future work.

̸

5 Experiments

All experiments were run on NVIDIA A100 80GB GPUs using PyTorch [Ansel et al., 2024] (v.2.3.0, BSD-3 license) and the Python Optimal Transport package [Flamary et al., 2021] (v.0.9.3, MIT license) to compute optimal transport distances with and without regularization. Code for these experiments is available at https://github.com/IBM/stochastic-order-eval.

5.1 Synthetic Data Experiment

In this section we analyze our method on a synthetic toy dataset that enables us to parametrically control the level of the multivariate stochastic dominance between two random variables. Given a dimension d , a parameter p ∈ [0 , 1] , and mean and variance parameters µ , σ 2 , our synthetic dataset is generated by sampling from the multivariate random variables X,Y ∈ R d :

X i ∼ N ( µ, σ 2 ) for i = 1 , . . . , d

Y i = X i +(2 · B i ( p ) -1) U i with B i ( p ) = Bernoulli ( p ) ∈ { 0 , 1 } and U i = Uniform (0 , 1) .

These variables X and Y are designed in such a way that p parametrizes the dominance of Y over X . In particular, X ≽ ( h,λ ) -FSD Y if p < 0 . 5 , and Y ≽ ( h,λ ) -FSD X if p > 0 . 5 .

As a baseline for our synthetic experiments, we also compute the violation ratio for the standard FSD framework using unregularized OT (EMD), i.e. ε hinge , 0 , as a function of p for fixed d = 5 , µ = 0 , σ 2 = 1 . 0 and N = 100 samples from X and Y . We then investigate how well this baseline is approximated by ε log ,λ> 0 , the entropically regularized ratio with a logistic cost as in Example 1. Fig. 1 shows that as the entropic regularization parameter λ decreases towards 0 and as the gain of the logistic cost β increases, ε log ,λ> 0 converges towards ε hinge , 0 across all values of p ∈ [0 , 1] . In all cases, multivariate FSD violation ratio predicts linearly p , indicating that it is captures well the FSD violations. This experiment indicates that, to best approximate the standard FSD violation ratio, λ should be taken as small as possible (c.f. Theorem 1) and β should be taken as large as possible (c.f. Example 1). There is, in practice, a tradeoff that must be made when computing the regularized index, as Sinkhorn's algorithm requires the matrix e -C/λ , where C is the matrix of pairwise costs. As such, if the ratio β/λ is too large, numerical underflow will occur and the algorithm will fail. As a rule of thumb, it is recommended to set β first and increase the value of λ if instability occurs.

0.2

0.4

0.6

0.8

p

0.0

0.2

0.4

0.6

0.8

(

x

,

Y

)

logistic cost with

=8.0 (varying

)

OT

hinge

, 0

(

EMD

)

OT

log

,

,

=0.1

OT

log

,

,

=0.5

OT

log

,

,

=1.0

OT

log

,

,

=2.0

0.2

0.4

0.6

0.8

p

0.0

0.2

0.4

0.6

0.8

(

x

,

Y

)

logistic cost with varying

(

=0.1)

OT

hinge

, 0

(

EMD

)

OT

log

,

,

=8.0

OT

log

,

,

=4.0

OT

log

,

,

=2.0

OT

log

,

,

=1.0

We now assess the power of our proposed test with as a function of the number of samples and the dimension. We consider the same setup as the previous experiment and set p = 0 . 65 so that

Y ≽ ( h,λ, 0 . 5) -FSD X . We then estimate the type I and type II error of the relative test statistic by averaging across 100 repetitions, these results are compiled in Fig. 2.

100

500

1000

2000

3000

4000

5000

Sample Size

0

.

55

0

.

60

0

.

65

0

.

70

0

.

75

0

.

80

Eval Type

Relative Testing FSD w/ P(IC)

Relative Testing FSD w/ P(EC)

MultiVariate Relative Testing

5.2 LLMBenchmarking

To test our method on real world scenarios, we have chosen a current topic of significant interest to the community: LLM Benchmarking. We show through our experiments that our method can provide a more holistic ranking of LLMs evaluated on different metrics as opposed to present strategies which involve mean win rate. To demonstrate our method's application to LLM Benchmarking, we have conducted assessments on two different sets of data.

Mixinstruct For our first evaluation we use the dataset from Jiang et al. [2023] (MIT license) that consists of responses from 12 different instruction following LLMs, with each response evaluated on 9 metrics such as BLEU, ROUGE, BERTScore, BARTScore, etc. The data has a train (100K rows) and test (5k rows) split where each row consists of an instruction, input sentence, the expected output from users, as well as the responses of a set of different LLMs with their decoding parameters and evaluation scores on different metrics. However for the test set, Jiang et al. [2023] also did a

pairwise evaluation of the responses from the models by asking ChatGPT which response was better. We use this test set and generate a ranking of the LLMs using Entropic Multivariate FSD (Algorithm 1), where for each LLM we construct an empirical measure on R 9 using n samples varying from 100 to 5000. We then compute the pairwise ratios for these empirical distributions using the logistic loss with β = 0 . 2 , the regularization parameter λ = 0 . 1 , and utilize the relative testing procedure from Section 4.2 to rank the 12 LLMS (see Fig. 4 for the ranking obtained with n = 5000 in appendix Appendix B). The confidence intervals are then generated using 1000 bootstrap repetitions. Finally, we compare the resulting ranking with the (univariate ranking) provided by ChatGPT scoring (which serves as a human proxy) as a function of the sample size n using Kendall Tau similarity. These results are presented in Fig. 3.

We then compare different methods that reduce multivariate ordering to univariate FSD via aggregation. The first method is a portfolio aggregation with Independent Copula P(IC) [Nitsure et al., 2023], where a dimension is normalized with a global univariate CDF across all models and a geometric mean is performed across all dimensions. A univariate FSD is then applied on the resulting univariate random variables. The second method, referred to as portfolio aggregation with Empirical Copula P(EC) [Ruschendorf, 1976, Ulan et al., 2021], estimates a global multivariate CDF across all models, and then assigns to each evaluation vector the value of its CDF. Similarly, a univariate FSD is applied on this one dimensional data.

Results We see from Fig. 3 that the multivariate FSD, is sample efficient and has the highest Kendall tau rank similarity with GPT score. We hypothesize that this thanks to its ability to capture dependencies between the metrics. The independent copula P(IC), ignores the dependencies and hence lags a little behind but is still sample efficient. Whilst the empirical copula P(EC) captures the dependencies, it suffers from the curse of dimension and is not sample efficient.

6 Conclusion

In this paper, we proposed entropic multivariate FSD violation ratio as a statistic for assessing multivariate first order dominance. We addressed the convergence of these ratios as the entropic regularization goes to zero and established a central limit theorem and bootstrap consistency for this statistic. These statistical properties were leveraged in a framework for multivariate FSD testing which was applied to multi-metrics benchmarking machine learning models, showing its benefits in capturing the metric dependencies. Casting testing for stochastic order as an optimal transport problem with a smooth cost and devising an entropic regularization to ensure beneficial statistical and computational properties is an interesting framework that we envision to be useful and versatile for other stochastic orders. For instance the µ -first order dominance of Galichon and Henry [2012] uses optimal transport maps as multivariate quantiles [Carlier et al., 2014] and defines a µ -stochastic dominance; our entropic violation ratio framework can be extended to that case and, upon proving central limit theorems on the OT potentials, will lead to similar central limit theorems to the one presented in this work. Similarly, for the multivariate Lorenz order [Fan et al., 2024] that is of interest when the agent making the choice is risk averse. The Lorenz order can be expressed in terms of optimal transport maps and can be extended to our statistical testing framework using the tools introduced in this paper. We leave these developments for future work.

Acknowledgments and Disclosure of Funding

G. Rioux is partially supported by the NSERC postgraduate fellowship PGSD-567921-2022.

References

Ambrosio, N. Gigli, and G. Savaré. Gradient flows: in metric spaces and in the space of probability measures . Birkhäuser Basel, Basel, 2005. URL https://doi.org/10.1007/978-3-7643-8722-8 .

Bommasani, P. Liang, and T. Lee. Holistic evaluation of language models. Annals of the New York Academy of Sciences , 2023.

Carlier, V. Chernozhukov, and A. Galichon. Vector quantile regression: an optimal transport approach. arXiv preprint arXiv:1406.4643 , 2014.

Chang, X. Wang, J. Wang, Y. Wu, K. Zhu, H. Chen, L. Yang, X. Yi, C. Wang, Y. Wang, et al. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109 , 2023.

Constantine and T. Savits. A multivariate Faa di Bruno formula with applications. Transactions of the American Mathematical Society , 348(2):503-520, 1996.

Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013. URL https://proceedings.neurips.cc/paper_files/paper/2013/ file/af21d0c97db2e27e13572cbf59eb343d-Paper.pdf .

J.-C. de Borda. Mémoire sur les élections au scrutin. Histoire de l'Académie Royale des Sciences , 1781.

del Barrio, J. A. Cuesta-Albertos, and C. Matrán. An optimal transportation approach for assessing almost stochastic order. In E. Gil, E. Gil, J. Gil, and M. Á. Gil, editors, The Mathematics of the Uncertain: A Tribute to Pedro Gil , pages 33-44. Springer International Publishing, Cham, 2018. URL https://doi.org/10. 1007/978-3-319-73848-2_3 .

del Barrio, A. G. Sanz, J.-M. Loubes, and J. Niles-Weed. An improved central limit theorem and fast convergence rates for entropic transportation costs. SIAM Journal on Mathematics of Data Science , 5(3): 639-669, 2023. URL https://doi.org/10.1137/22M149260X .

Dror, G. Baumer, S. Shlomov, and R. Reichart. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers) , pages 1383-1392, 2018.

M. Dudley. Uniform central limit theorems , volume 142. Cambridge university press, 2014.

Eckstein and M. Nutz. Convergence rates for regularized optimal transport via quantization. Mathematics of Operations Research, Articles in Advance , pages 1-18, 2023. URL https://doi.org/10.1287/moor. 2022.0245 .

Fan, M. Henry, B. Pass, and J. A. Rivero. Lorenz map, inequality ordering and curves based on multidimensional rearrangements, 2024.

Flamary, N. Courty, A. Gramfort, M. Z. Alaya, A. Boisbunon, S. Chambon, L. Chapel, A. Corenflos, K. Fatras, N. Fournier, L. Gautheron, N. T. Gayraud, H. Janati, A. Rakotomamonjy, I. Redko, A. Rolet, A. Schutz, V. Seguy, D. J. Sutherland, R. Tavenard, A. Tong, and T. Vayer. Pot: Python optimal transport. Journal of Machine Learning Research , 22(78):1-8, 2021. URL http://jmlr.org/papers/v22/20-451.html .

B. Folland. Real analysis: modern techniques and their applications , volume 40. John Wiley & Sons, 1999.

O. Friedrichs. The identity of weak and strong extensions of differential operators. Transactions of the American Mathematical Society , 55:132-151, 1944.

Galichon and M. Henry. Dual theory of choice with multivariate risks. Journal of Economic Theory , 147(4): 1501-1516, 2012.

Genevay, L. Chizat, F. Bach, M. Cuturi, and G. Peyré. Sample complexity of sinkhorn divergences. In K. Chaudhuri and M. Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics , volume 89 of Proceedings of Machine Learning Research , pages 15741583. PMLR, 16-18 Apr 2019. URL https://proceedings.mlr.press/v89/genevay19a.html .

Goldfeld, K. Kato, G. Rioux, and R. Sadhu. Limit theorems for entropic optimal transport maps and sinkhorn divergence. Electronic Journal of Statistics , 18(1):980-1041, 2024a.

Goldfeld, K. Kato, G. Rioux, and R. Sadhu. Statistical inference with regularized optimal transport. Information and Inference: A Journal of the IMA , 13(1):iaad056, 2024b.

Graf and H. Luschgy. Foundations of Quantization for Probability Distributions . Springer Berlin Heidelberg, Berlin, Heidelberg, 2000. URL https://doi.org/10.1007/BFb0103945 .

Groppe and S. Hundrieser. Lower complexity adaptation for empirical entropic optimal transport. arXiv preprint arXiv:2306.13580 , 2023.

Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics , pages 65-70, 1979.

Huang, Q. Zhang, L. Sun, et al. Trustgpt: A benchmark for trustworthy and responsible large language models. arXiv preprint arXiv:2306.11507 , 2023.

Hundrieser, T. Staudt, and A. Munk. Empirical optimal transport between different measures adapts to lower complexity. arXiv preprint arXiv:2202.10434 , 2022.

Hundrieser, M. Klatt, and A. Munk. Limit distributions and sensitivity analysis for empirical entropic optimal transport on countable spaces. The Annals of Applied Probability , 34(1B):1403 - 1468, 2024. doi: 10.1214/23-AAP1995. URL https://doi.org/10.1214/23-AAP1995 .

Jiang, X. Ren, and B. Y. Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561 , 2023.

W. Keener. Theoretical Statistics: Topics for a Core Course . Springer New York, New York, NY, 2010. URL https://doi.org/10.1007/978-0-387-93839-4 .

Kouaissah. Using multivariate stochastic dominance to enhance portfolio selection and warn of financial crises. The Quarterly Review of Economics and Finance , 80:480-493, 2021.

Manole and J. Niles-Weed. Sharp convergence rates for empirical optimal transport with smooth costs. The Annals of Applied Probability , 34(1B):1108-1135, 2024.

D. Maso. An introduction to Γ -convergence . Birkhäuser Boston, Boston, MA, 1993. URL https: //doi.org/10.1007/978-1-4612-0327-8 .

Mena and J. Niles-Weed. Statistical bounds for entropic optimal transport: sample complexity and the central limit theorem. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/ 5acdc9ca5d99ae66afdfe1eea0e3b26b-Paper.pdf .

MosaicLM. Llm evaluation scores. https://www.mosaicml.com/llm-evaluation , 2023.

Nietert, Z. Goldfeld, and K. Kato. Smooth p -Wasserstein distance: structure, empirical approximation, and statistical applications. In International Conference on Machine Learning , pages 8172-8183. PMLR, 2021.

Nitsure, Y. Mroueh, M. Rigotti, K. Greenewald, B. Belgodere, M. Yurochkin, J. Navratil, I. Melnyk, and J. Ross. Risk assessment and statistical significance in the age of foundation models. arXiv preprint arXiv:2310.07132 , 2023.

Nutz. Introduction to entropic optimal transport. Lecture notes, Columbia University , 2021.

Ogryczak and A. Ruszczynski. Dual stochastic dominance and related mean-risk models. SIAM Journal on Optimization , 13(1):60-78, 2002.

Römisch. Delta method, infinite dimensional. In Encyclopedia of Statistical Sciences . John Wiley & Sons, Ltd, 2006. URL https://doi.org/10.1002/0471667196.ess3139 .

Ruschendorf. Asymptotic Distributions of Multivariate Rank Order Statistics. The Annals of Statistics , 4(5): 912 - 923, 1976.

Santambrogio. Optimal Transport for Applied Mathematicians: Calculus of Variations, PDEs, and Modeling . Springer International Publishing, Cham, 2015. URL https://doi.org/10.1007/978-3-319-20828-2 .

Shaked and J. G. Shanthikumar. Stochastic Orders . Springer New York, New York, NY, 2007. URL https://doi.org/10.1007/978-0-387-34675-5 .

Shapiro. On concepts of directional differentiability. Journal of optimization theory and applications , 66: 477-487, 1990.

Shapiro. Asymptotic analysis of stochastic programs. Annals of Operations Research , 30(1):169-186, Dec. 1991.

J. Stromme. Minimum intrinsic dimension scaling for entropic optimal transport. arXiv preprint arXiv:2306.03398 , 2023.

W. Tu. An Introduction to Manifolds . Springer New York, New York, NY, 2011. URL https://doi.org/ 10.1007/978-1-4419-7400-6 .

Ulan, W. Löwe, M. Ericsson, and A. Wingkvist. Copula-based software metrics aggregation. Software Quality Journal , 29(4):863-899, 2021. URL https://doi.org/10.1007/s11219-021-09568-9 .

Ulmer, C. Hardmeier, and J. Frellsen. deep-significance-easy and meaningful statistical significance testing in the age of neural networks. arXiv preprint arXiv:2204.06815 , 2022.

W. Van der Vaart. Asymptotic statistics , volume 3. Cambridge university press, 2000.

W. van der Vaart and J. A. Wellner. Weak Convergence and Empirical Processes: With Applications to Statistics . Springer New York, New York, NY, 1996. URL https://doi.org/10.1007/978-1-4757-2545-2 .

S. Varadarajan. On the convergence of sample probability distributions. Sankhy¯ a: The Indian Journal of Statistics (1933-1960) , 19(1/2):23-26, 1958.

Villani. Optimal Transport: Old and New . Springer Berlin Heidelberg, Berlin, Heidelberg, 2009. URL https://doi.org/10.1007/978-3-540-71050-9 .

B. . C. F. . N. H. S. H. N. L. N. R. O. S. L. T. T. Wolf. Open llm leaderboard. https://huggingface.co/ spaces/HuggingFaceH4/open_llm_leaderboard , 2023.

G. Zhang and M. Hardt. Inherent trade-offs between diversity and stability in multi-task benchmarks, 2024.

A Algorithm

Algorithm 1 describes our multitesting-based ranking procedure, for both the absolute and relative testing frameworks.

̸

̸

̸

̸

B Ranking from the Mix Instruct experiment

C Proofs of main results

C.1 Proof of Lemma 1

Observe that the set { ( x, y ) ∈ R d × R d : x ≤ y } is closed, as any limit point, ( x, y ) , of a sequence { ( x n , y n ) } n ∈ N satisfying x n ≤ y n is such that x ≤ y as the relevant inequalities are preserved in the limit. It follows that ✶ { x ≤ y } ( x, y ) is a lower semicontinuous function and hence there exists a coupling ¯ π ∈ Π( µ, ν ) for which inf π ∈ Π( µ,ν ) ∫ ✶ { x ≤ y } ( x, y ) dπ ( x, y ) = ∫ ✶ { x ≤ y } ( x, y ) d ¯ π ( x, y ) for any choice of µ, ν ∈ R d (cf. e.g. Theorem 4.1 in Villani, 2009).

Assume that inf π ∈ Π( µ,ν ) ∫ ✶ { x ≤ y } ( x, y ) dπ ( x, y ) = 0 such that there exists a coupling ¯ π for which 0 = ∫ ✶ { x ≤ y } ( x, y ) d ¯ π ( x, y ) = P ( ˆ X, ˆ Y ) ∼ ¯ π ( ˆ X ≤ ˆ Y ) = 1 -P ( ˆ X, ˆ Y ) ∼ ¯ π ( ˆ X > ˆ Y ) i.e. P ( ˆ X, ˆ Y ) ∼ ¯ π ( ˆ X ≥ ˆ Y ) ≥ P ( ˆ X, ˆ Y ) ∼ ¯ π ( ˆ X > ˆ Y ) = 1 which, in light of Theorem 6.B.1. in [Shaked and Shanthikumar, 2007], implies that X ≽ FSD Y .

C.2 Proof of Theorem 1

Throughout, we fix the metric d : (( x, y ) , ( x ′ , y ′ )) ∈ R 2 d × R 2 d ↦→ ∥ x -x ′ ∥ 1 + ∥ y -y ′ ∥ 1 on R 2 d . We first show that costs induced by functions h satisfying (SC d ) are Lipschitz continuous with respect to d with constant L .

Lemma 2. For any ( x, y ) , ( x ′ , y ′ ) ∈ R d × R d , | c ( x, y ) -c ( x ′ , y ′ ) | ≤ L d (( x, y ) , ( x ′ , y ′ )) .

Proof. As c ( x, y ) = ∑ d i =1 h ( y i -x i ) for ( x, y ) ∈ R d × R d and h is Lipschitz continuous with constant L ≥ 0 , we have that, for any x, y, x ′ , y ′ ∈ R d ,

Lemma 3. For any choice of ( µ, ν ) , ( µ ′ , ν ′ ) ∈ P ( R d ) ×P ( R d ) and π ∈ Π( µ, ν ) , π ′ ∈ Π( µ ′ , ν ′ ) , we have that

where W 1 ( π, π ′ ) = OT d ( π, π ′ ) is the 1 -Wasserstein distance for the metric d .

Proof. Let ( µ, ν ) , ( µ ′ , ν ′ ) ∈ P ( R d ) ×P ( R d ) be arbitrary and fix any π ∈ Π( µ, ν ) and π ′ ∈ Π( µ ′ , ν ′ ) . Let γ ∈ Π( π, π ′ ) be arbitrary and consider

As γ is arbitrary, it follows that ∣ ∣ ∫ cd ( π -π ′ ) ∣ ∣ ≤ OT d ( π, π ′ ) .

The proof of Theorem 1 is based on the framework developed in [Eckstein and Nutz, 2023] and, in particular, the proof of Theorem 3.1 therein. As a byproduct of their proof technique, it is demonstrated that, for any π ∈ Π( µ, ν ) , there exists a coupling π ′ ∈ Π( µ, ν ) which is independent of c satisfying

provided that the cost satisfies the condition (7) (see Definition 3.1 in [Eckstein and Nutz, 2023] for a weaker condition) for any choice of µ n , ν n ∈ P n ( R d ) , the set of all probability measures on R d supported on at most n points. We note that OT |·| , 0 is simply the 1 -Wasserstein distance for the distance induced by the 1 -norm and recall that | · | has Lipschitz constant 1 due to the reverse triangle inequality (recall the notation OT h,λ from Section 3). It is then shown in their proof that

provided that there exists µ n , ν n ∈ P n ( R d ) satisfying ( OT |·| , 0 ( µ n , µ ) ∧ OT |·| , 0 ( ν n , ν ) ) ≤ Cn -α for n = ⌊ λ -1 / α ⌋ for some α ∈ (0 , 1] and C ≥ 0 .

Lemma 4. Fix δ > 0 and assume that η ∈ P ( R d ) is not mutually singular with respect to the Lebesgue measure. Then, there exists a measure η n ∈ P n ( R d ) satisfying

for every n ≥ ( 20 / δ ) d .

Proof. First, note that if ∑ d i =1 E η | X i | 1+ δ = ∞ , the inequality holds vacuously. We hence assume that ∑ d i =1 E η | X i | 1+ δ < ∞ .

Fix δ > 0 . Then, by Lemmas 6.6 and 6.7 in [Graf and Luschgy, 2000], there exists constants C 1 , C 2 , C 3 > 0 depending on δ for which

for every n ≥ (2 C 3 ) d . Here C 1 = 5 2 δ and C 2 , C 3 > 0 can be chosen as any constants satisfying Γ(2)Γ( δn -1) Γ( δn ) = 1 δn -1 ≤ C 2 5 n for every n ≥ C 3 5 . Observe that C 2 = C 3 = 10 δ satisfy these conditions.

We conclude by noting that the infimum in (10) is achieved by Theorem 4.12 in [Graf and Luschgy, 2000].

Lemma 5. Fix δ > 0 and η ∈ P ( R d ) . Then, there exists a measure η n ∈ P n ( R d ) satisfying

for every n ≥ ( 20 / δ ) d .

Proof. Again, if ∑ d i =1 E η | X i | 1+ δ = ∞ , the inequality trivially holds.

Assume that ∑ d i =1 E η | X i | 1+ δ < ∞ and, for t ≥ 0 , let g t : R d → R denote the density of an isotropic mean-zero normal distribution with covariance t 2 Id . Define the probability measure η ∗ g t via

✶ for any Borel measurable set A ⊂ R d . From the above display, η ∗ g t has density z ∈ R d ↦→ ∫ g t ( z -x ) dη ( x ) with respect to the Lebesgue measure.

It follows from a minor modification of Lemma 7.1.10 in [Ambrosio et al., 2005], that, for any 1 ≤ p < ∞ , OT |·| p , 0 ( η ∗ g t , η ) ≤ t p ∫ ∥ z ∥ p 1 g t ( z ) dz < ∞ as normal distributions have finite absolute moments of all orders. Hence, lim t ↓ 0 OT |·| , 0 ( η ∗ g t , η ) = 0 and E η ∗ g t | X i | 1+ δ → E η | X i | 1+ δ as t ↓ 0 (see Theorem 6.9 in [Villani, 2009]).

Now, let η n ∈ P n ( R d ) be such that

for every n ≥ ( 20 / δ ) d as in Lemma 4. It follows for the triangle inequality for Wasserstein distances (see Chapter 6 in Villani, 2009) that

The claimed result follows by applying the upper bound from the penultimate display and taking the limit t ↓ 0 on both sides of the resulting inequality.

Lemma 5 provides the worst case scaling for OT |·| , 0 ( η n , η ) . As noted in the text, it is anticipated that d can be replaced by a suitable notion of intrinsic dimension for η .

Proof of Theorem 1. We begin with part 1 . The lower bound 0 ≤ OT h,λ ( µ, ν ) -OT h, 0 ( µ, ν ) is due to the fact that the Kulback-Leibler divergence is non-negative. As for the upper bound, letting π ∈ Π( µ, ν ) be an optimal plan for OT h, 0 ( µ, ν ) , such a plan always exists due to Theorem 4.1 in [Villani, 2009]. It follows from (9) and Lemma 5 that there exists a plan π ′ ∈ Π( µ, ν ) satisfying

provided that ⌊ λ -d ⌋ ≥ ( 20 / δ ) d . By minimizing both sides of the above display with respect to π ′ ∈ Π( µ, ν ) , Part 1 readily follows.

The proof of Part 2 will follow from the fact that the set of all couplings Π( µ, ν ) is tight and hence admits a limit point in the weak topology by Prokhorov's theorem (see Lemma 4.4 in Villani, 2009) and Corollary 7.20 in [Maso, 1993] once we establish that the functionals

are such that F λ Γ -converges to F when treated as functionals on the separable metric space ( P ( R d × R d ) , d ′ ) , where d ′ is the Lévy-Prokhorov metric which metrizes the weak convergence of probability distributions (cf. e.g. p.72 in [Billingsley, 2013]), we refer the reader to [Maso, 1993] as a standard reference on Γ -convergence. In light of Proposition 8.1 in [Maso, 1993] it suffices to show that:

for every π ∈ P ( R d × R d ) and any sequence P ( R d × R d ) ∋ π λ → π with respect to d ′ , F ( π ) ≤ lim inf λ ↓ 0 F λ ( π λ ) .

for every π ∈ P ( R d × R d ) there exists a sequence P ( R d × R d ) ∋ π λ → π with respect to d ′ satisfying F ( π ) = lim λ ↓ 0 F λ ( π λ ) .

Westart by proving the first statement. Fix π ∈ P ( R d × R d ) and any sequence ( π λ ) λ ↓ 0 ⊂ P ( R d × R d ) converging to π with respect to W 1 . By Lemma 4.4 in [Villani, 2009], Π( µ, ν ) is tight and hence precompact with respect to d ′ by Prokhorov's theorem. It is easy to see that Π( µ, ν ) is closed under the weak convergence such that it is in fact compact with respect to d ′ . It follows that if π ̸∈ Π( µ, ν ) , π λ ̸∈ Π( µ, ν ) for every λ sufficiently small, hence F λ ( π λ ) → + ∞ = F ( π ) as λ ↓ 0 . Now, if π ∈ Π( µ, ν ) , compactness of Π( µ, ν ) implies that π λ ∈ Π( µ, ν ) for every λ sufficiently small. As µ, ν have finite first moments and c is Lipschitz continuous with constant L , any γ ∈ Π( µ, ν ) satisfies

Conclude from Lemma 5.1.7 in [Ambrosio et al., 2005] that

proving the first condition.

As for the second condition, if π ̸∈ Π( µ, ν ) , the constant sequence π λ = π satisfies F λ ( π λ ) = F λ ( π ) = F ( π ) = + ∞ . If π ∈ Π( µ, ν ) , let µ n , ν n ∈ P n ( R d ) be such that OT |·| , 0 ( µ n , µ ) , OT |·| , 0 ( ν n , ν ) → 0 as n → ∞ (i.e. µ n and ν n converge weakly to µ and ν respectively and E µ n [ ∥ X ∥ 1 ] → E µ [ ∥ X ∥ 1 ] , E ν n [ ∥ Y ∥ 1 ] → E ν [ ∥ Y ∥ 1 ] ). For instance, the empirical versions of µ and ν constructed from independent samples satisfy this condition almost surely (see Theorem 3 in [Varadarajan, 1958] for weak convergence, convergence of the moments is due to the law of large numbers). By (8) and the surrounding discussion, there exists π λ ∈ Π( µ, ν ) satisfying

From the proof of Theorem 3.1 in [Eckstein and Nutz, 2023], λ D KL ( π λ || µ ⊗ ν ) ≤ λ log( ⌊ λ -1 ⌋ ) → 0 as λ ↓ 0 and

Conclude that

and, as W 1 ( π λ , π ) → 0 , π λ converges weakly to π (see [Villani, 2009]).This concludes the proof.

C.3 Proof of Theorem 2

We first establish some useful properties of optimal potentials for this problem, there exists a pair

Lemma 6. Fix λ > 0 and sub-Gaussian distributions µ, ν ∈ P ( R d ) with a shared constant τ 2 > 0 . Then, for any choice of h satisfying (SC d ) , there exists a unique pair of continuous optimal potentials ( φ, ψ ) for OT h,λ ( µ, ν ) for which the Schrödinger system (4) holds at all points ( x, y ) ∈ R d × R d and φ (0) = 0 . Moreover, this pair of potentials satisfies the estimates

for any multi-index α ∈ N d 0 of order | α | ≤ k = ⌊ d / 2 ⌋ +1 with the constants k, L, p k from the (SC d ) condition. The quantities in the subscripts of the constants indicate what parameters the constants depend on.

Proof. Let ( φ h , ψ h ) be an arbitrary pair of optimal potentials for OT h,λ ( µ, ν ) so that

and, setting C = ∫ ψ h dν -1 2 OT h,λ ( µ, ν ) , we have that ( φ ′ h , ψ ′ h ) = ( φ h + C , ψ h -C ) is another pair of optimal potentials, and ∫ φ ′ h dµ = ∫ ψ ′ h dν = 1 2 OT h,λ ( µ, ν ) .

We further consider the functions defined on R d by

We will show that ( φ , ψ ) are optimal potentials satisfying the claimed bounds.

′′ h ′′ h

It follows by Jensen's inequality and Lemma 2 that

observing that 2 ≥ E ν [exp( ∥ X ∥ 2 1 / 2 τ 2 )] ≥ E ν [ ∥ X ∥ 2 1 / 2 τ 2 ] such that ∫ ∥ · ∥ 1 dν ≤ √ 4 τ 2 (again by Jensen's inequality). It follows that φ ′′ h ( x ) ≤ C d,L,τ,h (0) (1 + ∥ x ∥ 1 ) where C d,L,τ,h (0) depends on d, L, τ, and the value of h (0) . The same bound evidently holds for ψ ′′ h on R d and for ψ ′ h on the support of ν . Applying this bound and Lemma 2, it holds that

where we have used the fact that E ν [ e t ∥ X ∥ 1 ] ≤ E ν [ e τ 2 t 2 2 + ∥ X ∥ 2 1 2 τ 2 ] ≤ 2 e τ 2 t 2 2 for any t ∈ R as follows from Young's inequality and the sub-Gaussian assumption. The same argument implies that ψ ′′ h satisfies an analogous bound.

To see that ( φ ′′ h , ψ ′′ h ) is a pair of optimal potentials, observe that Jensen's inequality yields

as follows from the fact that ( φ ′ h , ψ ′ h ) satisfy (4). Conclude that ∫ φ ′′ h dµ + ∫ ψ ′′ h dν ≥ OT h,λ ( µ, ν ) such that equality must hold (indeed ∫ e φ ′′ h ( x )+ ψ ′′ h ( y ) -c ( x,y ) λ dµ ⊗ ν ( x, y ) = 1 by construction, so the left hand side of the ineqalit is the objective value in the dual form of the EOT problem) and, by strict concavity of the logarithm, φ ′′ h = φ ′ h µ -a.e. and ψ ′′ h = ψ ′ h ν -a.e. such that ( φ ′′ h , ψ ′′ h ) are indeed optimal potentials for this problem.

Now, consider the potentials ( φ, ψ ) = ( φ ′′ h -φ ′′ h (0) , ψ ′′ h + φ ′′ h (0)) , which satisfy (4) on R d × R d φ (0) = 0 . The bounds for φ ′′ h and ψ ′′ h evidently carry over to φ and ψ so that there exists a constant C ′′ d,L,τ,h (0) < ∞ depending only on d, L, τ, and h (0) for which

for every ( x, y ) ∈ R d × R d . Now, suppose that ( ¯ φ, ¯ ψ ) is any other pair of potentials for which (4) holds on R d × R d and ¯ φ (0) = 0 . As discussed in Section 3, one has that ( φ, ψ ) and ( ¯ φ, ¯ ψ ) coincide µ - and ν -a.e. so that, for every x ∈ R d ,

which shows that ¯ φ = φ , the equality of ψ and ¯ ψ follows analogously.

We now establish the bounds on the derivatives. By the multivariate Faà di Bruno formula (cf. e.g. Constantine and Savits, 1996), for any multi-index α ∈ N d 0 of order | α | ≥ 1 , the derivative -D α φ ′′ h ( x ) (assuming it exists) can be expressed as a linear combination of products of derivatives of the form

where β j ∈ N d 0 and k j ∈ N 0 satisfy ∑ | α | j =1 β j k j = α . By the dominated convergence theorem, the derivative commutes with the integral sign and, applying the same formula to the derivative D β j x e -c ( x,y ) λ , we see that it can be expressed as a linear combination of products of the form

where γ m ∈ N d 0 , l m ∈ N 0 with ∑ | β j | m =1 γ m l m = β j , and we assume that all relevant derivatives exist. As h satisfies (SC d ) , it is k -times continuously differentiable for k = ⌊ d / 2 ⌋ + 1 with derivatives of order s ≤ k satisfying | h ( s ) ( x ) | ≤ C k (1 + | x | ) p k for some C k < ∞ and p k > 1 which may depend on k . From this assumption, (12) is well defined provided that | β j | ≤ k and we observe that ∣ ∣ ∣ D γ m x ∑ d i =1 h ( y i -x i ) ∣ ∣ ∣ ≤ ∑ d i =1 C k (1+ | y i -x i | ) p k ≤ ∑ d i =1 C k 2 p k -1 (1+ | y i -x i | p k ) as p k > 1 , so that ∣ ∣ ∣ D β j x e -c ( x,y ) λ ∣ ∣ ∣ can be bounded as e -c ( x,y ) λ C d,k,p k ,λ (1 + ∥ y -x ∥ 1 ) p k | β j | by appealing to (12).

Returning to (11), we infer that D α φ ′′ h is well-defined for any | α | ≤ k and that

We will split this integral into the regions ∥ y ∥ 1 < τ and ∥ y ∥ 1 ≥ τ for some τ > 0 which will be chosen later in the proof.

In this first region,

by the triangle inequality so that

For the second region, we have from Lemma 2 and the first part of the proof that

To bound this integral, note that

and that

by the Cauchy-Schwarz inequality. Similarly,

Now, ∫ e 4 L ∥ y ∥ 1 λ dν ( y ) ≤ 2 e 16 L 2 τ 2 2 λ 2 due to the bound E ν [ e t ∥ x ∥ 1 ] ≤ E ν [ e τ 2 t 2 2 + ∥ X ∥ 2 1 2 τ 2 ] ≤ 2 e τ 2 t 2 2 which holds for every t ≥ 0 . Further,

where we have applied the Cauchy-Schwarz inequality in the second line. By sub-Gaussianity, 2 ≥ E ν [ e ∥ X ∥ 2 1 / 2 τ 2 ] ≥ E ν [ ∥ X ∥ 2 p 1 (2 τ 2 ) p p ! ] for any p ∈ N so that √ E ν [ ∥ X ∥ 4 p k | β j | 1 ] ≤ √ 2(2 τ 2 ) p k | β j | √ (2 p k | β j | )! .

Combining all of these bounds,

The denominator can be bounded as ( ∫ e ψ ′′ h ( y ) -c ( x,y ) λ dν ( y ) ) -1 = e φ ′′ h ( x ) λ ≤ e h (0)+ L √ 4 τ 2 + L ∥ x ∥ 1 λ as follows from the first part of the proof. The ratio we wish to bound thus includes the exponential term e h (0)+2 L √ 4 τ 2 +2 L ∥ x ∥ 1 λ + 4 L 2 τ 2 λ 2 -τ 2 8 τ 2 , which we equate to 1 by setting τ 2 = 8 τ 2 λ ( h (0) + 2 L √ 4 τ 2 + 2 L ∥ x ∥ 1 ) + 32 L 2 τ 4 λ 2 = C L,τ,λ,h (0) + 16 τ 2 λ L ∥ x ∥ 1 . All in all,

the products in (11) can thus be bounded as C d,k,τ,λ,p k ,L,h (0) (1 + ∥ x ∥ 1 ) p k | α | , for any | α | ≤ k . The same argument establishes analogous bounds for ψ ′′ h . As ( φ, ψ ) coincides with ( φ ′′ h , ψ ′′ h ) up to additive constants, the derivative bounds evidently transfer over.

In what follows, we always choose the unique solution for any given EOT problem described in Lemma 6.

As aforementioned, our approach to proving limit distributions is based on the functional delta method (see Appendix D for a summary of the method). As ¯ h also satisfies (SC d ) with Lipschitz constant 2 L , C k replaced by 2 C k , and ¯ h (0) = 2 h (0) , there exists a constant C d,h,τ for which all bounds from Lemma 6 hold simultaneously for some choice of potentials ( φ h , ψ h ) for OT h,λ ( µ, ν ) and ( φ ¯ h , ψ ¯ h ) for OT ¯ h,λ ( µ, ν ) . We thus instantiate the function classes

and

where f ⊕ g is understood as the function ( x, y ) ∈ R d × R d ↦→ f ( x ) + g ( y ) . We further consider the class of probability distributions

for some τ > 0 . Throughout, we will treat P ⊗ τ as a subset of ℓ ∞ ( F ⊕ τ,h ) , the Banach space of bounded real functions on F ⊕ τ,h endowed with the supremum norm ∥ ℓ ∥ ∞ , F ⊕ τ,h = sup f ⊕ g ∈F ⊕ τ,h | ℓ ( f ⊕ g ) | ; the action of µ ⊗ ν ∈ P ⊗ τ on f ⊕ g is given by µ ⊗ ν ( f ⊕ g ) = ∫ fdµ + ∫ gdν . It is easy to see that µ ⊗ ν defines a bounded function on this function class due to the growth bounds inherent to F τ,h .

Our approach is similar to that of Proposition 1 in [Goldfeld et al., 2024b] and, in particular, Section 6 of that work. Namely, we prove a type of directional differentiability and Lipschitz continuity for the EOT cost.

Lemma 7. Fix λ > 0 and sub-Gaussian distributions µ, ν, ρ, η ∈ P ( R d ) with constant τ 2 > 0 . Then, for any choice of optimal potentials ( φ, ψ ) solving OT h,λ ( µ, ν ) and satisfying (4) on R d × R d ,

Proof. For t ∈ [0 , 1] , let µ t := µ + t ( ρ -µ ) and ν t := ν + t ( η -ν ) and, let ( φ ( µ t ,ν t ) , ψ ( µ t ,ν t ) ) denote the unique pair of optimal potentials for OT h,λ ( µ t , ν t ) satisfying φ ( µ t ,ν t ) (0) = 0 and solving the Schrödinger system on R d × R d (see Lemma 6) and likewise for ( φ ( µ t ,ν ) , ψ ( µ t ,ν ) ) and ( φ ( µ,ν ) , ψ ( µ,ν ) ) . Observe that

where we recall that the potentials satisfy the relevant Schrödinger systems (4) such that ∫ e φ ( µt ,ν t ) ( x )+ ψ ( µt ,ν t ) ( y ) -c ( x,y ) λ dµ t ( x ) ≡ 1 and ∫ e φ ( µt ,ν ) ( x )+ ψ ( µt ,ν ) ( y ) -c ( x,y ) λ dν ( y ) ≡ 1 on R d so that

and, analogously,

It suffices, therefore, to show pointwise convergence of all relevant potentials to ( φ ( µ,ν ) , ψ ( µ,ν ) ) in the limit t ↓ 0 . Here we will only show convergence of ( φ ( µ t ,ν t ) , ψ ( µ t ,ν t ) ) , convergence of the other set of potentials follows analogously. For convenience set ( φ t , ψ t ) = ( φ ( µ t ,ν t ) , ψ ( µ t ,ν t ) )

Let [0 , 1] ∋ t n ↓ 0 be arbitrary and fix a subsequence t n ′ . By Lemma 6 we can apply the ArzelàAscoli theorem (cf. e.g. Theorem 4.44 in Folland, 1999) to infer that ( φ t , ψ t ) converges to a pair of continuous functions ( φ, ψ ) uniformly on compact sets and, in particular, pointwise along a further subsequence t n ′′ . From Lemma 6,

where this final term is integrable with respect to any product of sub-Gaussian measures. By the dominated convergence theorem, for any ( x, y ) ∈ R d × R d ,

as t n ′′ ↓ 0 such that the pair ( φ, ψ ) satisfies the Schrödinger system (4) pointwise and hence is a pair of optimal potentials for OT h,λ ( µ, ν ) and φ (0) = lim t n ′′ ↓ 0 φ t n ′′ (0) = 0 , whence ( φ, ψ ) = ( φ ( µ,ν ) , ψ ( µ,ν ) ) . Combining (13) and (14), we conclude that

and, as the limit is independent of the choice of original subsequence it follows that

This limit is invariant under the transformation ( φ ( µ,ν ) , ψ ( µ,ν ) ) ↦→ ( φ ( µ,ν ) + C, ψ ( µ,ν ) -C ) , proving the claim.

Lemma 8. Fix λ > 0 and arbitrary sub-Gaussian distributions ρ, η, ρ ′ , η ′ ∈ P ( R d ) with a shared constant τ 2 > 0 . Then, we have that

Proof. Let ( φ ( ρ,η ) , ψ ( ρ,η ) ) , ( φ ( ρ ′ ,η ′ ) , ψ ( ρ ′ ,η ′ ) ) , and ( φ ( ρ ′ ,η ) , ψ ( ρ ′ ,η ) ) be the optimal potentials for OT h,λ ( ρ, η ) , OT h,λ ( ρ ′ , η ′ ) , and OT h,λ ( ρ ′ , η ) described in Lemma 6. Then, by analogy with (13) and (14)

such that

As the constants from Lemma 6 (and hence the constants defining F ⊕ τ,h ) are independent of the choice of sub-Gaussian distributions (rather they depend only on the sub-Gaussian constant), ( φ ( ρ ′ ,η ) , ψ ( ρ ′ ,η ′ ) ) and ( φ ( ρ,η ) , ψ ( ρ ′ ,η ) ) are elements of F ⊕ τ,h so that

Lemma 9. Fix λ > 0 and sub-Gaussian distributions µ, ν ∈ P ( R d ) with a shared constant τ 2 > 0 . Then, √ n (ˆ µ n ⊗ ˆ ν n -µ ⊗ ν ) d → G µ ⊗ ν in ℓ ∞ ( F ⊕ τ,h ) , where G µ ⊗ ν is a tight Gaussian process in ℓ ∞ ( F ⊕ τ,h ) for which G µ ⊗ ν ( f ⊕ g ) = N (0 , var µ ( f ) + var ν ( g )) .

Proof. From the proof of Lemma 27 in [Goldfeld et al., 2024b] (see also Lemma 8 in Nietert et al., 2021), we see that the function class F τ,h is µ -Donsker and ν -Donsker (i.e. the associated empirical processes √ n (ˆ µ n -µ ) and √ n (ˆ ν n -ν ) converge weakly to tight mean-zero Brownian bridge processes in ℓ ∞ ( F τ,h ) with respective covariance functions given by ( f, g ) ∈ F τ,h × F τ,h ↦→ cov µ ( f, g ) and ( f, g ) ∈ F τ,h × F τ,h ↦→ cov ν ( f, g ) respectively) provided that ∑ ∞ r =1 r d + kp k -1 P µ ( ∥ X ∥ ≥ r -1) 1 / 2 < ∞ for k = ⌊ d / 2 ⌋ +1 and likewise for ν . By the Chernoff bound, P µ ( ∥ X ∥ ≥ r -1) ≤ E µ [ e t ∥ X ∥ ] e -t ( r -1) for any t > 0 . The standard inequality ∥ z ∥ ≤ ∥ z ∥ 1 for any z ∈ R d yields E µ [ e t ∥ X ∥ ] ≤ E µ [ e t ∥ X ∥ 1 ] ≤ 2 e t 2 τ 2 2 (recall the proof of Lemma 6). It readily follows that the sum ∑ ∞ r =1 r d + kp k -1 P µ ( ∥ X ∥ ≥ r -1) 1 / 2 is finite, establishing Donskerness of the class with respect to µ and hence ν by the same argument.

By independence of the samples ( X i ) n i =1 and ( Y j ) n j =1 we have by Example 1.4.6 in [van der Vaart and Wellner, 1996], Lemma 3.2.4 in [Dudley, 2014], and Donskerness of the class that

where G µ and G ν are independent tight µ - and ν -Brownian bridge processes. As the map ( ℓ, ℓ ′ ) ∈ ℓ ∞ ( F τ,h ) × ℓ ∞ ( F τ,h ) ↦→ ℓ ⊗ ℓ ′ ∈ ℓ ∞ ( F ⊕ τ,h ) is continuous (indeed ∥ ℓ ⊗ ℓ ′ ∥ ∞ , F ⊕ τ,h ≤ ∥ ℓ ∥ F τ,h + ∥ ℓ ′ ∥ F τ,h ), we have by the continuous mapping theorem that

Proof of Theorem 2. Throughout, we fix some ¯ τ > τ and observe that if µ, ν are τ 2 -sub-Gaussian, then they are also ¯ τ 2 -sub-Gaussian. From the proof of Proposition 1 in [Goldfeld et al., 2024b] (see also Remark 4 of the same reference), Lemmas 7 and 8 together imply that the functional ρ ⊗ η ∈ P ⊗ ¯ τ ↦→ OT h,λ ( ρ, η ) is Hadamard directionally differentiable at µ ⊗ ν tangentially to P ⊗ ¯ τ (treated as a convex subset of ℓ ∞ ( F ⊕ ¯ τ,h ) ) with derivative γ ∈ T P ⊗ ¯ τ ( µ ⊗ ν ) ↦→ γ ( φ ⊕ ψ ) , where ( φ, ψ ) denote any pair of optimal potentials for OT h,λ ( µ, ν ) satisfying (4) on R d × R d , and the derivative is defined on the tangent cone to P ⊗ ¯ τ at µ ⊗ ν which is defined as

see Appendix D for precise definitions. Note that we have identified OT h,λ ( ρ, η ) with a functional on P ⊗ ¯ τ ; such an identification is well-defined in light of the discussion following Proposition 1 in [Goldfeld et al., 2024b].

The same implications hold for the functional ρ ⊗ η ∈ P ⊗ ¯ τ ↦→ OT ¯ h,λ ( ρ, η ) in light of the discussion preceding Lemma 7, with corresponding derivative γ ∈ T P ⊗ ¯ τ ( µ ⊗ ν ) ↦→ γ ( ¯ φ ⊕ ¯ ψ ) , where ( ¯ φ, ¯ ψ ) is any pair of optimal potentials for OT h,λ ( µ, ν ) satisfying (4) on R d × R d .

Note that ε h,λ ( µ, ν ) = f ◦ ( OT h,λ ( µ, ν ) , OT ¯ h,λ ( µ, ν ) ) for f : ( x, y ) ∈ R × R ↦→ x / y such that the chain rule (cf. e.g. Proposition 3.6 in Shapiro, 1990) guarantees that ρ ⊗ η ∈ P ⊗ ¯ τ ↦→ ε h,λ ( ρ, η ) is Hadamard directionally differentiable at µ ⊗ ν tangentially to P ⊗ ¯ τ with derivative

which is notably linear as a function of γ . It will be useful to rewrite this expression in terms of a single evaluation of γ . To this end, observe that if f 1 ⊕ f 2 , g 1 ⊕ g 2 ∈ F ⊕ ¯ τ,h , then so too is ( αf 1 -βg 1 ) ⊕ ( αf 2 -βg 2 ) for any α, β ∈ R with | α | + | β | ≤ 1 . Moreover, setting M = 1 OT ¯ h,λ ( µ,ν ) ∨ OT h,λ ( µ,ν ) OT ¯ h,λ ( µ,ν ) 2 (which is assumed to be nonzero by construction), we have that 0 ≤ 1 2 M 1 OT ¯ h,λ ( µ,ν ) ∨ 1 2 M OT h,λ ( µ,ν ) OT ¯ h,λ ( µ,ν ) 2 ≤ 1 2 so that we can write

where the final term in brackets is an element of F ⊕ ¯ τ,h . Further, for any γ ∈ T P ⊗ ¯ τ ( µ ⊗ ν ) , there exists a sequence ( ρ n ⊗ η n ) n ∈ N ⊂ P ⊗ ¯ τ which converges to µ ⊗ ν in ℓ ∞ ( F ⊕ ¯ τ,h ) and a sequence t n ↓ 0 for which γ = lim n →∞ ρ n ⊗ η n -µ ⊗ ν t n . Thus, if f 0 ⊕ f 1 , g 0 ⊕ g 1 ∈ F ⊕ ¯ τ,h are such that f 0 + g 0 ⊕ f 1 + g 1 ∈ F ⊕ ¯ τ,h , we have that

Likewise, if αf 0 ⊕ αf 1 ∈ F ⊕ ¯ τ,h for some α ∈ R , and f 0 ⊕ f 1 ∈ F ⊕ ¯ τ,h , then

With this, (15) can be written as

Given the above differentiability result and Lemma 9, part 1 of Theorem 2 will follow from the functional delta method (see Lemma 10 in Appendix D) upon showing that ˆ µ n ⊗ ˆ ν n ∈ P ⊗ ¯ τ with probability approaching one and noting that G µ ⊗ ν ∈ T P ⊗ ¯ τ ( µ ⊗ ν ) with probability one as follows from the portmanteau theorem. To this end, note that, by the law of large numbers,

almost surely such that ˆ µ n is ¯ τ 2 -sub-Gaussian with probability approaching one. The same deliberations imply that ˆ ν n share the same property.

By applying the delta method, we obtain that

and using the explicit expression for the derivative from (16), we see that ( ε h,λ ) ′ µ ⊗ ν ( G µ ⊗ ν ) is equal in distribution to 2 MN (0 , v 2 + w 2 ) , where v 2 = var µ ( 1 2 M OT ¯ h,λ ( µ,ν ) φ -OT h,λ ( µ,ν ) 2 M OT ¯ h,λ ( µ,ν ) 2 ¯ φ ) and w 2 = var ν ( 1 2 M OT ¯ h,λ ( µ,ν ) ψ -OT h,λ ( µ,ν ) 2 M OT ¯ h,λ ( µ,ν ) 2 ¯ ψ ) ; the 2 M terms in the variance and multiplying the normal distribution evidently cancel out to give the desired formula for the limiting variance.

As for the bootstrap consistency result, since (15) is linear, it follows from Corollary 1 in [Goldfeld et al., 2024b] that ( ρ, η ) ↦→ ε h,λ ( ρ, η ) is Hadamard directionally differentiable at µ ⊗ ν tangentially to spt( G µ ⊗ ν ) . As in the proof of Lemma 9, the class F ¯ τ is µ - and ν -Donsker such that the bootstrapped empirical processes √ n (ˆ µ B n -ˆ µ n ) and √ n (ˆ ν B n -ˆ ν n ) are asymptotically measurable and converge conditionally in distribution to the µ - and ν -Brownian bridge processes G µ and G ν respectively (see Chapter 3.6 in van der Vaart and Wellner, 1996). By Lemma 1.4.4 and Example 1.4.6 in [van der Vaart and Wellner, 1996], ( √ n (ˆ µ B n -ˆ µ n ) , √ n (ˆ ν B n -ˆ ν n )) is asymptotically measurable and converges conditionally in distribution to ( G µ , G ν ) as elements of ℓ ∞ ( F ¯ τ ) × ℓ ∞ ( F ¯ τ ) . As the map ( ℓ, ℓ ′ ) ∈ ℓ ∞ ( F ¯ τ ) × ℓ ∞ ( F ¯ τ ) ↦→ ℓ ⊗ ℓ ′ ∈ ℓ ∞ ( F ⊕ ¯ τ,h ) is continuous, ( √ n (ˆ µ B n -ˆ µ n ) ⊗ √ n (ˆ ν B n -ˆ ν n )) is asymptotically measurable and converges conditionally in distribution to G µ ⊗ ν as elements of ℓ ∞ ( F ⊕ ¯ τ,h ) where G µ ⊗ ν ( f 0 ⊕ f 1 ) = G µ ( f 0 ) + G ν ( f 1 ) for any f 0 ⊕ f 1 ∈ F ⊕ ¯ τ,h .

Bootstrap consistency then follows from Theorem 23.9 in [Van der Vaart, 2000] by applying the logic from the first half of the proof.

D The Functional Delta Method

Our strategy for deriving limit distributions and consistency of the bootstrap is based upon the functional delta method, which generalizes the standard delta method for functions of simple random variables. This section provides a brief introduction to the functional delta method following the exposition of [Römisch, 2006]. Throughout, convergence in distribution is understood in the sense of Hoffmann-Jørgensen when necessary (cf. e.g. Chapter 1 in [van der Vaart and Wellner, 1996]).

Much like the delta method which, identifies the distributional limit of √ n ( g ( X n ) -g ( µ )) as N (0 , σ 2 ( g ′ ( µ )) 2 ) n →∞ provided that √ n ( X n -µ ) d → N (0 , σ 2 ) and that g : R → R is differentiable at µ (see Proposition 8.14 in Keener, 2010), the functional delta method establishes the limit distribution of a functional f : Θ ⊂ D → R , where D is a normed vector space. In this setting, the surrogate for the derivative in the standard delta method is the Hadamard directional derivative.

Definition 4 (Hadamard directional derivative [Römisch, 2006, Shapiro, 1990]) . Let D be a normed vector space and fix a non-empty set Θ ⊂ D . The tangent cone to Θ at θ ∈ Θ is given by

A functional f : Θ → R is Hadamard directionally differentiable at θ ∈ Θ tangentially to Θ if there exists a map f ′ θ : T Θ ( θ ) → R satisfying

for any h ∈ T Θ ( θ ) , t n ↓ 0 , and h n → h in D with θ + t n h n ∈ Θ .

This notion of differentiability is compatible with distributional convergence of random elements of D in the sense that the following generalization of the delta method holds.

Lemma 10 (Functional delta method [Römisch, 2006, Shapiro, 1991]) . Fix a probability space (Ω , Σ , P ) and let D be a normed vector space and f : Θ ⊂ D → R be Hadamard directionally differentiable at θ ∈ Θ tangentially to T Θ ( θ ) with derivative f ′ θ : T Θ ( θ ) → R . Let T n : Ω → Θ be maps such that r n ( T n -θ ) d → T for some norming sequence r n →∞ and a measurable map T : Ω →T Θ ( θ ) ⊂ D . Then r n ( f ( T n ) -f ( θ ) ) d → f ′ θ ( T ) and, if Θ is convex, r n ( f ( T n ) -f ( θ ) ) -f ′ θ ( r n ( T n -θ ) ) → 0 in outer probability.

̸

Whilst Lemma 10 is sufficient to derive limit distributions, bootstrap consistency typically requires the following notion of full Hadamard differentiability (see e.g. Theorem 23.9 in [Van der Vaart, 2000] or Theorem 3.9.11 in van der Vaart and Wellner, 1996). A functional f : D → R is said to be Hadamard differentiable at θ tangentially to a vector subspace D 0 ⊂ D if there exists a continuous linear functional f ′ θ : D 0 ↦→ R satisfying (17) for any h ∈ T Θ ( θ ) , t n = 0 , t n → 0 , and h n → h in D with θ + t n h n ∈ Θ . The following lemma enables a connection between full and directional Hadamard differentiability.

Lemma 11 (Lemma 2 in [Goldfeld et al., 2024b]) . If f : Θ ⊂ D → R is Hadamard directionally differentiable at θ ∈ Θ tangentially to T Θ ( θ ) and f ′ θ is linear on a subspace D 0 ⊂ T Θ ( θ ) , then f is Hadamard differentiable at θ tangentially to D 0 .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction make the following claims: 1) we introduce a statistic for accessing a notion of multivariate almost stochastic dominance 2) we establish a CLT and bootstrapping for this statistic 3) we use this statistic to compare the relative performance of models. The first two points are addressed in Section 4 whereas Section 4.2 clarifies the testing framework and our approach is validated empirically in Section 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: As we mention in the discussion section, a limitation of our work is that does not treat the case where choices are risk averse. However, we indicated how these situations can be addressed by combining our statistical testing framework with multivariate Lorenz order.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The complete proofs of all results are provided in the appendices along with a short outline in the main text. All relevant assumptions are also clearly stated.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Algorithm 1, included in the appendix, provides the relevant pseudocode to implement the proposed statistical framework. The descriptions of the numerical experiments provide all required details to reproduce the results provided. A complete code package is also included in the submission.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: As highlighted in the previous point, the code is included in the submission, and all relevant experimental details are provided.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All relevant experimental details are provided in the main text.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experiments are provided with error bars and are computed using the bootstrap as justified by the main theoretical results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided details on the type of hardware and computation time needed in the experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This work provides a theoretical framework for assessing a statistical property of multivariate data. The application explored in this work is to benchmarking of different models according to a vector of metrics. Such a framework can be broadly useful for identifying ethical concerns in models, but in itself poses no such concerns.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The statistical framework proposed in this paper is used to compare the outputs of different models based on metrics of interest. On its own, this framework has no social impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No new data or models were released in conjunction with this paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All relevant details regarding existing assets are provided in the main text.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets were released in conjunction with this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects was included in this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects was included in this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the task of estimating the degree of stochastic dominance between two multivariate distributions. In the univariate setting, stochastic dominance is a useful tool for tasks such as benchmarking LLMs, where a practitioner may have estimates of some quality metric for responses. An efficient way to approximately estimate multivariate stochastic dominance could thus be helpful in setting where models may be evaluated on many metrics, rather than just a single value. The authors introduce a natural definition of near-stochastic dominance based on an existing definition in the univariate case and a connection to optimal transport, and show that while this notion may not allow for efficient estimation, adding an entropic regularization term allows for better guarantees. Through experiments, the authors demonstrate that their proposed metric converges to the true unregularized value in synthetic experiments, and for an LLM dataset, their approach outputs a ranking of models far more correlated with a \"ground-truth\" ranking.\n\nSTRENGTHS:\n- The paper is clearly written, and does a good job of explaining how their results rest on prior results in univariate stochastic domination and regularization methods. \n\n- The goal of obtaining good methods for estimating multivariate stochastic domination is very natural, and seems very relevant to many areas of practice, especially evaluating LLMs on multiple axes, as the authors mention.\n\nWEAKNESSES:\n- The experiments section is a bit confusing, in particular the explanations of the various approaches tested in the LLM experiment. The reasoning behind the choice of the ground-truth ranking is also a bit unclear to me.\n\nQUESTIONS:\n- Does your method also improve estimates of the degree of stochastic dominance (e.g. the first-ranked model is much better than the second-ranked model, but the second- and third-ranked models are comparable)? Is this something that can be demonstrated empirically?\n\n- If GPT-generated prompt comparisons are viewed as good enough to use as a ground-truth ranking, why not use this as the gold-standard benchmarking method? If not, is there a better baseline to compare the multivariate FSD approach to? \n\n- Moreover, is there a reason why we should expect a \"human\" ranking to match the true stochastic order induced by some set of metrics? It seems that in some cases we might actually want the order induced by a set of metrics to look somewhat different than what the average human might output as a ranking.\n\nLIMITATIONS:\nI think the authors adequately address the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInsipred by the uni-dimensional case of First order Stochastical Dominance (FSD), the authors indtroduce multidimensional FSD which does away making approximations using aggregations and reductions of multi dimensional metrics and reduce the orderings to unidimensional case. This is done vi Optimal Transport framework but this comes with a caveat since, using empirical Optimal Transport framework in multi dimensional case suffers from the curse of dimensionality. This is dealt with using  approximations in form of entropic regularization using ideas from Cuturi 2013, to get Entropic Optimal Transport framework. \nTheorem 1 in the paper shows that the approximation error has a computable upper bound. The framework is tested on a simulated data and LLM metric evaluation case study.\n\nSTRENGTHS:\n1. It is great to see a method which can help LLM research community regarding proper evaluation and benchmarking. There is clearly a need for such metrics and methods, since most LLM's responses are highly stochastic, even when using same prompt and same parameters. \n2. It is now high time for LLM research community to turn to rigorous statistical analysis of their research methodology and this paper contributes to that so it is both timely and useful. \n3. The authors have applied the framework to two different domains: financial domain with portfolio allocation study and LLM evaluation.\n4. Clear comparison between OT and EOT frameworks, absoulute and relative tests.\n5. Figures are clear and support the claims made.\n\nWEAKNESSES:\n- Unclear writing in many parts of the paper which makes such a difficult technical paper harder to read and understand. I strongly advise the authors to take couple of polishing passes.\n- Line 287-290 unclear writing\n- Line 314-319 unclear\n- predicts linearly p, indicating that it is captures well \n- Figure 1 caption and title don't match.\n- Maybe in experiments use multiple LLMs and not just Chat GPT\n\nQUESTIONS:\nSome typos and writing mistakes\n1. Some references are repeated, - Del Barrio 2018\n2.  How to tune the hyperparameters: $\\beta$ and $\\lambda$ in Sec 5.2 ?\n3.  It will be helpful to clearly state that for Kendall tau statistic, lower values are better Fig 2.\n4. How reasonable it is to take consensus with ChatGPT as the ground truth ?\n\nLIMITATIONS:\nI am not an expert in theory and this paper is heavy on OT theory, and the readibility of the paper is low in many important places. It would have been great if the authors would have listed clearly what limitations do these methods bring, one can ofcourse be related to computation when using multidimenional statistics over aggregation to uni-dimensional case, at what dimension could the cost become too prohibitive ?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
In this paper authors propose a testing framework for  First order Stochastic Dominance (FSD) for multivariate random variables. To achieve this, the authors use ideas from optimal transport using Entropic Regularization to derive a hypothesis testing procedure for testing multivariate FSD. The proposed methodology includes theoretical results showing the distributional convergence of the test statistic.

STRENGTHS:
- The paper considers an important problem, which is well motivated. 
- Theoretical results back the methodology
- The paper is well-presented

WEAKNESSES:
1. The experimental results do not present the Type I and II errors for the hypothesis testing methodology proposed. The authors should consider adding these results (with a synthetic setup if needed). 
2. The results should also investigate how these errors change with increasing $N$? And increasing $d$?

Overall, I think more extensive empirical investigation is needed to show how fast the convergence takes place, and how well the methodology scales with increasing $d$ and $N$ (in terms of computational complexity)

**Other comments:**

Lines 166-167:
- Notation should be $OT_{h, 0}$ instead of $OT_{0, \lambda}$
- Kendall tau rank has not been defined
- From the definition of $\mathcal{E}_{\mathcal{W}_2}$  below line 73, we should have that 

$\mathcal{E}_{\mathcal{W}_2}= 1$ 
when 

$F_Y^{-1}(t) \leq F_X^{-1}(t)$ for a.e. t. However, line 77 states that in this case $\mathcal{E}_{\mathcal{W}_2} = 0$. I'm not sure if this is correct.

QUESTIONS:
See above

LIMITATIONS:
The authors discuss the limitations in the discussion section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17

comments: SUMMARY:
The paper studies the testing of multivariate stochastic dominance, i.e., deciding an order between two multivariate random variables. The authors generalized the notion of index of almost stochastic dominance, which is for uni-variate rv. The new index is based on regularized value of optimla transport problems. Convergence of plug-in estimate of that index, and its bootstrap theory is developed. As a main application, it is applied to LLM Benchmarking, where an LLM is evaluated on many metrics.

STRENGTHS:
1. This generalized notion of stochastic order index for multivariate random variables is interesting. Its application to LLM benchmarking is also interesting. 

2. Many advanced tools from probability is used in the proof section. It is interesting to see that these theories find application in LLM evaluation.

WEAKNESSES:
1. There are two hyperparameters that need to be chosen to define the ratio statistics. The author should provide guidance on how to choose them.
2. Can the authors discuss the computation complexity of computing the ratio statistics?
3. The new notion of stochastic dominance is interesting. I think it would explain this new concept more if the authors could compile a list of examples that showcase this notion. For example the example in Sec 5.1 is intuitive. But it is too simplified. Since "dominance" is a complicated concept for multivariate distributions, explaining what kind of dominance is being captured and what not could be useful when practitioners apply the method.

QUESTIONS:
- What does $\hat \mu$ $\hat \nu$ mean in the context of LLM benchmarking? Do we divide the dataset into multiple smaller datasets and evaluate LLM on them?
- Typo at line 185. Should be "whenever $OT_{\bar h, \lambda } = 0$"

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-19