A Pseudo-Metric between Probability Distributions based on Depth-Trimmed Regions
Authors: Guillaume Staerman
Pavlo Mozharovskyi
Pierre Colombo
Stephan Clémençon
Florence d'Alché-Buc
Venue: Accepted by TMLR ()

Abstract: The design of a metric between probability distributions is a longstanding problem motivated by numerous applications in machine learning. Focusing on continuous probability distributions in the Euclidean space $\mathbb{R}^d$, we introduce a novel pseudo-metric between probability distributions by leveraging the extension of univariate quantiles to multivariate spaces. Data depth is a nonparametric statistical tool that measures the centrality of any element $x\in\mathbb{R}^d$ with respect to (w.r.t.) a probability distribution or a dataset. It is a natural median-oriented extension of the cumulative distribution function (cdf) to the multivariate case. Thus, its upper-level sets---the depth-trimmed regions---give rise to a definition of multivariate quantiles. The new pseudo-metric relies on the average of the  Hausdorff distance between the depth-based quantile regions for each distribution. Its good behavior regarding major transformation groups, as well as its ability to factor out translations, are depicted. Robustness, an appealing feature of this pseudo-metric, is studied through the finite sample breakdown point. Moreover, we propose an efficient approximation method with linear time complexity w.r.t. the size of the dataset and its dimension. The quality of this approximation and the performance of the proposed approach are illustrated in numerical experiments.

Published in Transactions on Machine Learning Research (MM/YYYY)
A Pseudo-Metric between Probability Distributions based on Depth-Trimmen Regions
Guillaume Staerman
guilaanee.tterrmane@hhriff
LTCI, Télécom Paris, Institut Polytechnique de Paris
Pavlo Mozharovskyi
LTCI, Télécom Paris, Institut Polytechnique de Paris
Pierre Colombo
pierre. hatrrr.oolommoeeeeennneeeeepppp
Equall.ai and MICS, CentraleSupélec, Université Paris-Saclay
Stéphan Clémençon
stephaan.cemmenccoolllccooomn..
LTCI, Télécom Paris, Institut Polytechanique de Paris
Florence d'Alché-Buc
forrenessdaichee@@tteccco...p
LTCI, Télécom Paris, Institut Polytechnique de Paris
Reviewed on OpenReview: https: openreview. net/ forum? id= QysSSS577PP
Abstract
The design of metric between probability distributions longstanding problem motivated by numerous applications in machine learning. Focusing on probability distributions the Euclidean space Rd, we introduce novel pseudo-metric between probability distributions by leveraging the extension of univariate quantiles to multivariate spaces. Data depth is nonparametric statistical tool that measures the centrality of any element x E Rd with respect to (w.rtt)) a probability distribution or a dataset. is a natural median-orientod extension of the cumulative distribution function (cdf) to the multivariate case. Thus, its upper-level scts-the depth-trimmed regions-give rise to definition of multivariate quantiles. The new pseudo-metric relies on the average of the Hausdorff distance between the depth-based quantile regions for each distribution. Its good behavior regarding major transformation groups, as well as its ability to factor out translations, are depicted. Robustness, appealing feature of this pscudo-metric, is studied through the finite sample brcakdown point. Morcover, we propose an efficient approximation method with linear time complexity w.r.t. the size of the dataset and its dimension. The quality of this approximation and the performance of the proposed approach are illustrated in numerical experiments.
1 Introduction
Metrics pseudo-metrics between probability distributions have attracted long-standing interest in information theory (Kullback, 1959; Rényi, 1961; Csiszår, 1963; Stummer & Vajda, 2012), probability theory and statistics (Billingslcy, 1999; Sriperumbudur et al., 2012; Panarctos & Zemel, 2019; Rachev, 1991). While they serve many purposes in machine learning (Cha & Srihari, 2002; MacKay, 2003), they are of crucial importance in automatic evaluation natural language generation (see e.g. Kusner et al., 2015; Zhang et al., 2019), especially when leveraging deep contextualized embeddings such as the popular BERT (Devlin al., 2018). Yet designing a measure to compare two probability distributions is a challenging research field. This is certainly due to the inherent difficulty in capturing in single measure typical desired properties such

---

Published in Transactions on Machine Learning Research (MM/YYY)
(i) metric pseudo metric properties, (fi) invariance under specific geometric transformations, (iii) efficient computation, and (iv) robustness to contamination.
One can find in the literature vast collection of discrepancies between probability distributions that rely on different principles. The f-divergences (Csiszår, 1963) are defined as the weighted average by a well-chosen function f of the odds ratio between the two distributions. They are widely used statistical inference but are, by design, ill-defined when the supports of both distributions do not overlap, which is significant limitation in many applications. Integral Probability Metrics (IPMs; Sriperumbudur et al., 2012) are based variational definition of the metric, i.e. the maximum difference in expectation for both distributions calculated over class of mcasurable functions and give rise to various metrics (Maximum Mcan Discrepancy (MMD), Dudley's metric, Lasaassertiin Distance) depending on the choice of this class. However, except in the case of MMD, which enjoys a dlsed-form solution, the variational definition raises issues in computation. From the side of Optimal transport (OT) (see Villani, 2003; Peyré & Cuturi, 2019), the Ln--asssrsttein distance is based on a ground metric able to take into account the goometry of the space on which the distributions are defined. Its ability to handle non-overlapping support and appealing theoretical properties make OT a powerful tool, mainly when applied to generative models (Arjovsky et al., 2017), domain adaptation (Courty et al., 2014; Courty et al., 2017), realign datascts in natural sciences (Janati et al., 2019; Schiebinger et al., 2019) or automatic text evaluation (Zhao et al., 2019; Colombo et al., 2021a).
In this work, we adopt another angle. Focusing on probability distributions in the Euclidean space R we propose to consider a new metric between probability distributions by leveraging the extension of univariate quantiles to multivariate spaces. The notion of quantile function an interesting ground to build a comparison between two probability measures as illustrated by the closed-form of the Wasserstein distance defined over R. However, given the lack of natural ordering on R as soon as d >>> 1, extending the concept of univariate quantiles to the multivariate case raises a real challenge. Many extensions have been proposed in the literature, such as minimum volume sets (Einhmahl & Mason, 1992), spatial quantiles (Koltchinskii & Dudley, 1996) or data depth (Tukey, 1975). The latter offers different ways of ordering multivariate data regarding a probability distribution. Precisely, data depths are non-parametrie statistics that determine the centrality of any element E w.r.t. a probability measure. They provide a multivariate ordering based on topological propertics of the distribution, allowing it to be characterized by its location, scale shape (sce, c.g. Mosler, 2013 Chapter 2 of Staerman, 2022 for a review). Several data depths were subsequently proposed, such convex hull peeling depth (Barnett, 1976), simplicial depth (Liu, 1990), Oja depth (Oja, 1983) or zonoid depth (Koshevoy& & Mosler, 1997) differing in their properties and applications. With a substantial body of litcrature devoted to its computation, recent advances allow for fast exact (Pokotylo et al., 2019) and approximate (Dyckerhoff el al., 2021) computation of several depth notions. The desirable properties of data depth, such as affine invariance, continuity w.r.t. its arguments, and robustness (Zuo Serfling, 2000) make it an important tool in many fields. Today, in its variety of notions and applications, data depth constitutes versatile methodology (Mosler & Mozharovskyi, 2021) that has been successfully employed in a variety of machine learning tasks such as regression (Rousseeuw & Hubert, 1999; Hallin et al., 2010), classification (Li et al., 2012; Lange et al., 2014), anomaly detection (Serfling, 2006; Roussceuw & Hubert, 2018; Stacrman et al., 2020) and clustering (Jörnsten, 2004).
This paper presents a new discrepancy measure between probability distributions, well-defined for non- overlapping supports, that leverages the interesting features of data depths. This mcasure is studiod through the lens of the previously stated properties, yielding the contributions listed below.
Contributions:
A new discrepancy measure between probability distributions involving the upper-level sets of data depth is introduced. We show that this measure defines pseudo-metric in general. Its good behavior regarding major transformation groups, as well as its ability to factor out translations, are depicted. Its robustness is investigated through the concept of finite sample breakdown point.
An efficient approximation of the depth-trimmed regions-based pscudo-metric proposed for convex depth functions such as halfspace and projection. This approximation relies on a nice feature of the Hausdorff distance when computed between convex bodies.
2

---

Published in Transactions on Machine Learning Research (MM/YYYY)
The behavior of this algorithm regarding its parameters is studied through numerical experiments, which also highlight the by-design robustness of the depth-trimmed regions based pseudo metric. Applications to robust clustering of images and automatic evaluation of natural language generation (NLG) show the benefits of this approach when benchmarkod with state-of-tho-art probability metrics.
2 Background on Data Depth
In this section, we recall the concept. of statistical dala depth function and its attractive theoretical propert for clarity. Here and throughout, the space of all probability measures on R with N is denoted by (R By 9 we denote the push-forward operator of the function g. Introduced by Tukey (1975), the concept of data depth initially extends the notion of median to the multivariate setting. In other words, it measures the centrality of any element E He regarding probability distribution (respectively, dataset). Formally, data depth is defined as follows:
D: R M1 (xr, p)
[0 D(.,
(1)
We denote by D(x,p) p) (or Dp(z) for brevity) the depth x E R wr... P MI The higher D(x, p). the deeper it is in p. The depth-induced median of p is then defined by the set attaining supzer4 D(x, p). Since data depth naturally and in a nonparametric way defines pro-order on R2 w.r.t. a probability distribution, it can be seen a centrality-based alternative to the cumulative distribution function (edf) for multivariate data. For any [0, 1], the associated a-depth region of a depth function is defined as its upper-level set:
Dr = fx (x(x) 2 a).
It follows that depth regions are nested, i.. Dø D for any d. These depth regions generalize the notion of quantiles to a multivariate distribution.
A depth function's relevance to capturing information about distribution relies on the statistical properties it satisfies. Such properties have been thoroughly investigated in Liu (1990); Zuo & Serfling (2000) and Dyckerhoff (2004) with slightly different sets of axioms (or postulates) to be satisfied by a proper depth function. In this paper, we restrict to convez depth functions (Dyckerhoff, 2004) mainly motivated by recent algorithmic developments including theoretical results (Nagy al., 2020) as well implementation guidelines (Dyckerhoff et al., 2021).
The general formulation (1) opens the door to various possible definitions. While these differ in thcorctical and practically related properties such as robustness or computational complexity (see Mosler & Mozharovskyi, 2021 for a detailed discussion), several postulates have been developed throughout the recent decades the "good" depth function should satisfy. Formally, a function is called a convex depth function if it satisfics the following postulates:
D1 (AFFINE INVARIANCE) D(g(z). 9:P) = D(z, p) holds for g: g:ze Rd + Ax + b with any nin-sing matrix A E Rdxd and any vector b E TRd.
D2 (VANISHING AT INFINITY) lim Do(x)= 0. ||+||-00
D3 (UPPER SEMICONTINUITY) ERERd Rdd Dr(x) < af is an open set for every a E (0, 1].
D4 (QUASICONCAVITY) For every X [0, and v.ye Rd, D2 + (1 (1-)))) 2 Din(D,(e),D,(u1))
While (D1) is useful in applications providing independence w.r.t. measurement units and coordinate system, (D2) and (D3) appear as natural propertics since data depth is (conter-outward) generalization of cdf. Limit values vanish due to median-oriented construction. (D4) allows to preserve the original center-outward ordering goal of data depth and induces convexity of the depth regions. Furthermore, is easy to see that.
3

---

Published in Transactions on Machine Learning Research (MM/YYYY)
(D1-D4) respectively yield properties of affine equivariance, boundedness, closedness and convexity of the central regions D (Dyckerhoff, 2004). Thanks to (D2-D4), if ax>x>. O, non-emply regions associated to convex depth functions are convex bodies (compact convex set in IR)..
Below we recall two convex depth functions satisfying (D1-D4) that will be used throughout the paper: the halfspace depth (Tukey, 1975) and the projection depth (Liu, 1992), which are probably the most studied in the literature. For this, let gd-1 be the unit sphere in R and X a random variable defined on a certain probability space (S2, A, that takes values in X' C Rd following distribution p. The halfspace depth of a given x E Rd w.r.t. p defined as the smallest probability mass that can be contained a closed halfspace containing x:
HD,(z)= inf P (()u,X) S (u, aES
Projection depth, being a monotone transform of the Stahel-Donoie outlyingness (Donoho & Gasko, 1992; Stahel, 1981), is defined as follows:
(u,x) me, -1
PD₂(*)= = 1+ sup wE84-1
MAD((u, X))
where med and MAD stand for the univariate median and median absolute deviation from the median, respectively.
Remark 2.1. Data depth functions have connections with the density function in particulor cases. Indeed, for elliptical distributions, the level sets of any data depth satisfying (D1-D4) are concentric ellipsoids with the same center, and orientation the densily level sels (Liu & Singh, 1993). The densily is local measure assigning the score of an element as the probability mass in an infinitesimal neighborhood. In contrast, data depths are global measures of ordering taking into account the whole distribution to assign a score to on element and are thus not equivalent to the density for general distributions. However, they provide interesting alternatives in many applications, such as anomaly delection (see e.g- Staerman el al., 2021b). For example, the density will assign zero score to every x Rd for from a concentrated group of observations regardless of the distance. At the same time, the projection depth described above will be able to rank these "outliers" " depending how il moves away from them.
3 A Pseudo-Metric based on Depth- Trimmed Regions
In this section, we introduce the depth-based pseudo-metric and study its properties. We consider depth regions possessing the same probability mass to compare those from different probability distributions fairly. Following Paindaveine & Bever (2013), we denote by a : (8, E [0, 1] x MR(R4) - a(3 p) E [0. the highest level such that the probability mass of the depth-trimmed trimmed region at this level is at least B. Precisely, for any pair (8, p) E (0, x (RR))
a(3, p) suply E [0.1]: P (D7) 8}.
(2)
In the remainder of this paper, when the quantity p(3,p) will be associated with depth regions of p, the second argument of the function a(:))) will be omitted, for notation simplicity. It is worth mentioning that Do(0") C (o( for any 3r, since B + a(3, p) is monotone decreasing function. Thus, (2(0) is the smallest depth region with probability larger than or equal to 3 and can be defined in an identical way as:
Da(0)
n D% 7ET,(9)
where (3) E 1]: p (DS) > 3). The strict inoqualitics (2) and in the definition of T (3) climinate cases where the supremum does not exist. Indeed, when 5 = 0, the depth region is then an infinitesimal set with probability higher than zero. To the best of our knowledge, the supremum exists (without necessarily
4

---

Published in Transactions on Machine Learning Research (MM/YYYY)
being unique) in the case of the halfspace depth (Roussceuw & Rutz, 1999) and the projection depth (Zuo, 2003) under mild assumptions. The set {DP()) E [0, 10,1-8j, -jj E], E E (0, 1]} where each region probability mass
is equal to then defines quantile regions of p.
Let u.v y be two probability measures X, C IRd respectively. Denote by dz(A,, B) the Hausdorff distance between the sets A and B. The pseudo-metric between probability distributions μ and v based the depth-trimmed regions is defined as follows.
Definition 3.1. Let & E (0, and E (0, x)) for all pairs (u,v) in Mi(2) x Mi()), the depth-trimmed regions (DRp.) discrepancy measure between and v is defined as
D -f0 pl=s f(- du (Da()),Do())
D d3.
(3)
Our discrepancy measure relies on the Hausdorff distance averaged over depth-trimmed regions with the same probability mass w.r.t. each distribution. Properties (D2 D3) ensure that for every 0 S 8 < 1, D is a non-empty compact subset of IR leading to well-defined discrepancy measure. Observe that the parameter can be considered as a robustness tuning parameter. Indeed, choosing higher & amounts to ignore the larger upper-level sets of data depth function, i.e. the tails of the distributions, see Sections 3.2 and 5.1.
Remark 3.2. Data depths provide robustness to (3) together with the E-trimming. Indeed, data depths such the three previously introduced in Section 2 eahibit attractive robustness properties. The asymptotic breakdown point of the halfspace median is higher than 1/(d + 1). In controst, the projection median is known to have a breakdown point equal to 1/2 (Donoho & Gasko, 1992; Ramsay et al,, 2019).
Remark 3.3. When d = 1, the Lpp Wasserstein distance enjoys an explicil expression involving quantile and distribution functions. Let X f1, Y ~ 1n1 two random variables where Pl, 11 are univariate probubility distributions. Denoting by Fx1 the quantile function of X', the Lp- Wasserstein distance can be wrillen as
Vi( f...:)))f/ - -1 dg.
(4)
Since data depth and its central regions are extensions of odf and quantiles to dimension d. > 1, DRp. is then possible (center-oulward) generalization of (4) to higher dimensions. When DRp.s is associaled with the halfspace depth, a simple calculus (see Lemma A.3 in the Appendic for mathematicnl details) leads to
1/2
DR p.s (1), =2 max (1 -Fr (1 -011-))-1 ( - -))p] dq.
Je/2
Thus, (1( < lim DR par +0+
(1), in general where the equality holds for symmetric distributions.
3.1 Metric Properties
We now investigate to which extent the proposed discrepancy measure satisfics the metric axioms. As a first go, we show that DRp,s fulfills most conditions. However, does not define distance in general. Proposition 3.4 (METRIC PROPERTIES). For any convez data depth, DRp is positive, symmetric and does not hold in generol.
satisfies triangular inequality but the entailment DRp.e(H,V) = -
Thus, DRp.e defines a pseudo-metric rather than a distance. Based on distance, the proposed discrepancy measure preserves isometry invariance, as stated the following proposition.
Proposition 3.5 (ISOMETRY INVARIANCE). Let A E Rdxd be non-singular matria and b RRd. Define the isometry mapping g z E Rd Ax + with AAT Las then it holds:
(9.. gru) DR,.e
where Ieft the push-forward of P by g. In particular, it ensures invariance of DRp, under translations and rotations.
5

---

DRp (Dn +00 OEN Published in Transactions on Machine Learning Research (MM/YYYY)
Although formulas (3) and (4) are based on the same spirit, there apparent reasons why the proposed pseudo-metric should have the same behavior the Wasserstein distance. It is the purpose of Proposition 3.6 to investigate the ability to factor out translations, for DR2,s associated with the halfspace depth, giving a positive answer for the case of two Gaussian distributions with equal covariance matricos.
Proposition 3.6 (TRANSLATION CHARACTERIZATION) Consider X, Y two random variables following IL E M1(X) and M1()) with ezpectations m1, m2 and variance-conariance matrices E1. En respectively. Denoting by n², v² the centered versions of p. v, it holds:
DR DR (v²" - 1|mm -m2l? $2 (")))(""""")(mm mall.
Now, let ~ (m1,, En) and v N(m,, Ez). Then it holds:
( u)---u--mmm m2ll S Ce sup
| VVTEnn E1u vu Ezu
where C. -0-00 1-- 1 1- -a(33) d3 with + the edf of the univariate slandord Gaussian distribulion.
Following Proposition 3.6: when E, E2, one has DR2,e(u,u) (R,,, = ||m m2ll for any p [(11. Σn) and v ~ M(m,, Σg) providing closed-form expression in the Gaussian case. This proposition shows that DR2,s can factor out translations in a similar way as Wasserstein distance if 1 (4",, ,,*) is zero. Furthermore, is clear that ( v*) = 0 then ( is zero too.
3.2 Robustness
In this part, we explore the robustness of the proposed distance, associated with the halfspace depth, given the finite sample breakdown point (BP; Donoho, 1982; Donoho & Hubert, 1983). This notion invest investigates the smallest contamination fraction under which the estimation breaks down the worst case. Considering a sample Sn (X,,...,XX Xn) composed of i.idd. observations drawn from a distribution μ with empirical measure in (1/n) Ei=1 8x,, the finite sample breakdlwn point of DRp. w.r.t. Sn, denoted by BP(DRp..,., is defined as:
0
min
sup n+ Zi.....
where Anto n4o Eh= 5x. + Ez==1 is the "concatenate" empirical measure between X1,.... .....,, Xn and the contamination sample 71.-.., %, with o E N It. is well known that the extremal regions of the halfspace depth are not robust while its central regions rather stable under contamination (Donoho & Gasko, 1992). Fortunately, by construction, the parameter allows to ignore these extremal depth regions and thus ensure the robustness of the depth-trimmed regions distance. Based on the results of Donoho & Gasko (1992) and Nagy & Dvořák (2021), the following proposition provides a lower bound on the finite sample breakdown point of DRp.es which highlights the robustness of the proposed distance as well as its dependence on E.
Proposition 3.7 (BREAKDOWN POINT). For the halfspace depth function, for any B E [0, E] such that aB,, (m)) O'max (fm), il holds:
E, a(1 -11--, e, pn ifa(1- - Pm)) n2 + [na([ - E, /11 a(1 e, (mn)) O (max((pn)
(pn) ++ ((m))
BP(DDp.e,5n)
otherwise,
1 + (maxx(()))
where O'max (max((p)) m1x HDp, (z).
DERd

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Thus, at least proportion a(1 - E, (.) a(1 - fa)) of outliers must be added to break down D.p. when considering larger regions, while central regions are robust independently of e. For two datasets, Dpp. breaks down if depth regions for at least one of the datasets do. The breakdown point is then the minimum between the breakdown points of cach dataset. However, the breakdown point considers the worst case, i.c. the supremum over all possible contaminations, and often pessimistic. Indeed the proposed pseudo-metric can handle more outliers in certain cases, as experimentally illustrated in Soction 5.1.
4 Efficient Approximate Computation
Exact computation of DRp.e can appear time-consuming due to the high time complexity of the algorithms that calculate depth-trimmed regions (c.f. Liu & Zuo, 2014 and Liu et al., 2019a for projection and halfspace depths, respectively) rapidly growing with dimension. However, we design a universal approximate algorithm that achieves (log-) linear time complexity in n. Since properties (D2 D4) ensure that depth regions are convex bodies in Rd they can be characterized by their support functions defined by lyc(u) sup (z, u), x e KC for any Sd-1 where IC is convex compact of Rd. Following Schneider (1993), for two (convex) regions (1) and Do(()) the Hausdorff distance between them can be calculated as:
(x(()) (2(())
sup l DDpgoo (u) - h Dgces (u)l-
WEB
As shall see in Section 5.1, mutual approximation of h p (u) by points from the sample and of sup by laking maximum over a finite set of directions allows for stable estimation quality. Recently, motivated by their numerous applications, many algorithms have been developed for the (exact and approximate) computation of data depths; sec, e.g., Section 5 of Mosler & Mozharovskyi (2021) for recent overview. Depths satisfying the projection property (which also include halfspacc and projection depth, see Dyckerhoff (2004)) can be approximated by taking minimum over univariate depths; see e.g. Rousseeuw & Struyf (1998); Chen et al. (2013); Liu & Zuo (2014), Nagy et al. (2020) for theoretical guarantees, and Dyckerhoff et al. (2021) for an experimental validation.
Empirical data. Let X, Y be two samples X (XI...., Xa) and Y (Yi,..,, Yn) from μr, u such that An (L/n) Ei= 8x, and Dn (1/m) Dimi ōy. When calculating approximated depth of sample points D A (D))) (respectively D ) a mairix MX E RaaK (respectively MY Rm=x) of projections of sample points on (a common) set of K E N directions (with its element M
t (ux, X,) for some ~ (((--1)), where 2((-) is the uniform probability distribution) can be obtained as a side product. More precisely, D* DY,M,, MY are used in Algorithm 1, which implements the MC-approximation of the integral in (3). See Figure for an illustration of the principle of this algorithm in practice.
(100 oo tooo (D, mppp
Coois D. teom (D, dp
fooe D. (ow (D, by
== 022
8= 0.5
0.9
Figure 1: Illustration of the principle of the depth trimmed-regions based pseudo-metric with n. 3 3 and
f = {0.2, 0.9).
7

---

Published in Transactions on Machine Learning Research (MM/YYY)
Particular cases of approximation algorithms for the halfspace depth and the projection depth are recalled in Section C in the Appendix. Time complexity of Algorithm 1 is O (n V m, d) V na (n V m))), where (:,-).) stands for the complete complexity of computing univariate depths in projections on u u-for all points of the sample. As a byproduct, projections on u can be saved to be reused after for the approximation of h "Daces (u). For the halfspace depth Rnsp(n, d) = (n(d V log n)) composed projection of the data onto u, ordering them, and passing to record the depths, see Mozharovskyi et al. (2015). For the projection depth, (prj(n, d) = (nd), where after projecting the data onto u, univariate median and MAD can be computed with complexity Onn), see Liu & Zuo (2014)). In comparison with popular distances, fixing = m, the Wasserstein distance is of order (d n)) with approximations in ((n2d) for Sinkhorn (Cuturi et al., 2013) and in O(K v log(n))) for the Sraeed-assrrsteii distance (Rabin el al., 2012); the MMD (Gretton et al., 2007) is of order dn" For example, the computational complexity of DRp.s with the projection depth is only of O(Kn(d na)) and thus competes with the fastest (max) sliced- Wasserstein distance.
Algorithm Approximation of DRp.c
Inilialization: X,Y Ka,
1: H O; compute D* DV M MY
for f= L...., Ra do
Draw Be~ 00 =])
4: Compute âe(-) == ()
5: Determine points inside oz rrggion:::
Z = (i:
Dx > ax)); D2 [j: Dj > a(())}
6: for k- = -,..., K do
7:
Compute approximation of support functions: 1 =mx MX end for Increase cumulative Hausdorff distance:
IX k
max MY N
I.
H += max hx hx KSK
10: end for
Output: DRp.. (H/na)
5 Numerical Experiments
In this section, we first investigate different properties of the proposed pseudo-metric such as the convergence rates of the pseudo-metric estimator w.r.t. the sample size, the quality of the approximation rntrodd Soction and its dependency on the number of projections. Further, we present two studies on the robustness of the proposed pseudo-metric DRp,s to outliers. Finally, we show the performance of this pseudo-mebric two machine learning tasks, clustering and automatic evaluation of neural language generation. Where applicable, we include statc-of-the-art methods for comparison.
5.1 Statistical Convergence, Approximation and Robustness
This part describes the behavior of the proposed pseudo-metric through different perspectives. On synthetic datasets, we investigate the statistical convergence rates of the empirical version of DR,, to the population one. We assess the Monte Carlo approximation proposod in Scction 4 and compare it to the Sliced Wasserstein distance. Finally, we highlight how DR,,, behaves under the presence of outliers using two different settings. Due to space limitations, experiments the influence of the parameters n and are deferred to the Appendix section.
Empirical analysis of statistical rates. Deriving theoretical finile-sample analysis may appear to be challenging for the proposed pseudo-metric. Thus, we numerically investigate the statistical convergence spoed of DR2g- To that end, we simulate two samples and Y from two standard Gaussian distributions in dimension two with varying sample sizes from n 10 to n 10000, see Section D.3 for additional experiments with d E (5, 10). We compute the DR2, between X and Y with na (5, 20, 100) using the halfspace and
8

---

Published in Transactions on Machine Learning Research (MM/YYYY)
the projection depths. Our proposed metric is computed with high number of directions K 25000 to isolate the statistical error. We report the estimation error averaged over ten runs in Figure 2 (log-log scale), that is, the value of the mettic itself, the true value of DR2,2 being equal to zero. When the Monte Carlo approximation error influenced by the parameter na is negligible (n = 100), Figure suggests that the statistical rates should be O(m Furthermore, Figure 7 indicates a rales of order O(n Onn-0.8/5 and Figure 8 of order O(n-0.6/4). These observations suggest a slow rate that depends on the dimension of the data. However, the approximation error being negligible due to the K 25000 sampled directions, the statistical rates seem to depend only linearly the dimension. Looking at the error values for n 10000 for d = 2, 5, 10, it increases by a factor of two, such as the dimension.
10a
10a
Error
n
na 20
na 100
10
10
102
10
10a
10a
102
10a
10
Number of samples
Number of samples
Figure 2: Empirical analysis of statistical convergence rates. Resulting error of the proposed psoudo-metric when increasing the sample size using the projection depth (left) and the halfspace depth (right) for various na parameters.
A Approximation error in terms of the number of projections. Proposition 3.6 allows to derive a closed form expression for DR2, (n, v) when μt, V are Gaussian distributions with the same variance-covariane matrix. In order to investigate the quality of the approximation we focus on computing DRp, with p 2, 0.3, no = 20 and using the halfspace depth for varying nunber of random projections K between a sample of 1000 points stemming from NO Ia) for d d= 5 and two different samples. These two samples are constructed from 1000 observations stemmning from Gaussian and symmetrical Cauchy distributions, both with a center equal to 7 Comparison with the approximation of max
light-tailed and heavy-tailed distributions,
(max-SW; see e.g. Kolouri et al., 2019), which shares the same closed-form as DR2,e, is also provided. Denoting by max-SW the Monte-Carlo approximation of the max-SW, the relative approximation errors, i.., (DRp.s / and (max-SW 7 are computed investigating both the quality of the approximation and the robustness of these discrepancy mcasures. Results that report the averaged approximation error and the 25-75% empirical quantile intervals are depicted in Figure 3. They show that DRp. possesses the same behavior as max-SW when considering Gaussians while it behaves advantageously for Cauchy distribution. Computation times are depicted in Figure 4, highlighting a constant- constant-multiple improvement compared to the max-SW, which is already computationally fast.
Robustness to outliers. We analyze the robustness of DRp.c by measuring ability overcome outliers (its robustness regarding the influence of the parameter & are given in the Section D.4 in the Appendix). In this benchmark, we naturally include existing robust extensions of the Wasserstein distance: Subspace Robust Wasserstein (SRW; Paty & Cuturi, 2019) searching for maximal distance lower-dimensionall subspaccs, ROBOT (Mukherjee et al., 2020) and RUOT (Balaji et al., 2020) being robust modifications of the unbalanced optimal transport (Chizat et al., 2018). Medians-of- 2021a) that replaces the empirical means in the Kantorivich duality formulae by the robust mean estimator MoM (sce c.g. Lecué & Lerasle, 2020; Laforgue et al., 2021), is not employed due high computational burden. Further, for completeness, we add the standard Wasserstein distance (W) and its approximation, the Sliced- Wasserstsnnn (Sliced-W; Rabin et al., 2012) distance, the trimmed Sliced- Wasserstein (TSW; Manole
Wasserstein (MoMW; Staerman el al.,
9

---

Published in Transactions on Machine Learning Research
DRp.
10
10-1
max-SW2
100
Error
10-2
10-2
300
600
900
300
600
900
Number of directions
Number of directions
Figure 3: Relative approximation error (averaged over 100 runs) of DRp. and the max Sliced-Wasssene Wassserstein for Gaussian (left) and Cauchy (right) sample with dimension d = 5 for differing numbers of approximating directions.
10-1
DR
Time (s)
Time (s)
max-SW2
10-1
300
600 Number of directions
900
300
600
900
Number of directions
Figure 4: Computation time (averaged over 100 runs) of DRp. and the max Sliced- Wassserstenr for Gaussian with n = 100, d = 5 (left) and n 1000, d 50 (right) for differing numbers of approximating directions.
et al., 2022) with the same number of projections (K 1000) as DRp.e- Since the scales of the compared methods differ, relative error is used as performance metric, i.e., the ratio of the absolute difference of the computed distance with and without anomalies divided by the latter. Two settings for pair of distributions are addressed: (a) Fragmented hypercube precedently studied Paty & Cuturi (2019), where the source distribution uniform in the hypercube [-1,1] 1)2 and the target distribution is transformed from the source via the map T + + 2sign(x) where sign(.) is taken clement-wiscly. Outliers are drawn uniformly from [-4, 4J2 (b) Two multivariate standard Gaussian distributions, one shifted by 10,, with outliers drawn uniformly from [-10, 20)2. Our analysis is conducted over 500 sampled points from the distributions described above.
To investigate the robustress of DRp.ss we consider the following value of e: 0.3 computed with the projection depth. We set the same trimming value for TSW. Thus, data depths are computed on source and target distributions such that 30% of data with lower depth values w.r.t. each distribution are not used computation of D.p...,, respectively. Figure 5, which plots the relative error depending on the portion of outliers varying up to 30%, illustrates advantageous behavior of DRp. for reasonable (starting with 2.5%) contamination.
5.2 Machine Learning Applications
This part presents two machine learning applications, clustering applied images and automatic evaluation of natural language generation. On real image dataset extracted from Mushion-MNIST where images are bags of pixels, evaluate the robustness of spectral clustering based DRpc Further, analyze
10

---

Published in Transactions on Machine Learning Research (MM/YYN
DR, & 0.3
ROBOT RUOT
W Sliced-W
SRW
TSW, 0.3
0.04
Relative error
0.1
0.1 Outliers proportion
0.2
0.0
a1
0.2
Outliers proportion
Figure 5: Relative error (averaged over 100 runs) of different distances for increasing outliers proportion on fragmented hypercube (left) and Gaussian (right) data.
the relevance of using DRp, evaluation metric in natural language generation to compare the empirical distributions of words of a pair of texts.
(Robust) Clustering on bags of pixels. We demonstrate the relevance of the proposed pseudo-metric through an application to (robust) clustering. that end, we perform spectral clustering (Shi & Malik, 2000) on two datasets derived from MaSSi (FM). Each grayscale image is seen as bag of pixels (Jebara, 2003), i.e. an empirical probability distribution over 3-dimensional space (the two first dimensions indicate the pixel position and the third onc, its intensity). The first dataset (FM) is constructed by taking the 100 first images in each class of the Fashion-MNIST dataset. The second dataset (Cont. FM), considered contaminated, is designed by introducing white patches on the left corner of 50 images drawn uniformly in the first dataset, which yields 5% of contamination. We benchmark DR,,s (using the projection depth) setting p == 2 and E = 0.1 with the Wasserstein (W), the Sliced- Wasserstein (Sliced-W) and the Maximum Mcan Discrepancy (MMD; Gretton et al., 2007) distances. DRp. and the by Monte-Carlo using 100 directions while the MMD distance is computed using Gaussian kernel with a bandwidth equal to 1. As baseline method, spectral clustering is also applied to images considered as vectors using Euclidean distance. Standard parameters of the scikit-leamd spectral clustering implementation are employed with number of clusters fixed to 10. Performances of the benchmarked metrics are assessed by measuring the normalized mutual information (NMI; Shannon, 1948) and the adjusted rank index (ARI; Hubert Arabie, 1985), which are standard clustering evaluation measures when the ground truth class labels are available. Results presented in Table show that for both cases, i.c. with or without contamination, spectral clustcring based on DRg. outperforms spectral clustcring basod on the other metrics.
are approximated
FM
Cont. FM
NMT
ART
NMT
ARI
DRpss
0.58
0.43
0.55
0.42
W
0.50
0.35
0.48
0.30
Sliced-W
0.55
0.39
0.47
0.33
MMD
0.54
0.37
0.50
0.36
Euclidean
0.50
0.32
0.48
0.30
Table 1: Spectral clustering performances.
Automatic evaluation of natural language generation (NLG). Collecting human annotations
evaluate NI.G systems is both expensive and time-consuming. Thus, automatically assessing the similarity
11

---

Published in Transactions on Machine Learning Research (MM/YYY)
Correctnoss
Data Coverage
Relevance
P
p
DRp,s
89.4
80.0
92.6
84.2
58.3
72.3
86.2
62.7
72.9
W
86.2
73.0
86.7
80.4
45.3
62.3
83.8
51.3
67.6
Sliced- W
86.1
73.0
85.8
80.9
45.5
60.0
82.0
51.3
68.2
MMD
25.4
71.7
8.3
19.1
45.3
10.0
26.1
51.3
15.0
BertS
85.5
73.3
83.4
74.7
53.3
68.2
83.3
65.0
79.4
MoverS
84.1
73.3
84.1
78.7
53.3
66.2
82.1
65.0
77.4
BLEU
77.6
60.0
66.3
55.7
36.6
50.2
63.0
51.6
65.2
ROUGE
80.6
65.0
65.0
76.5
60.3
76.3
64.3
56.7
69.2
MET.
86.5
70.0
66.3
77.3
46.6
50.2
82.1
58.6
65.2
TER
79.6
58.0
78.3
69.7
38.0
58.2
75.0
77.6
70.2
Table 2: Absolute correlation at the system level with three human judgment criteria. The best overall results are indicated in bold, best results their group are underlined.
between two texts highly interesting for the NLP community (Specia et al., 2010). This task aims to build an evaluation metric that achieves a high correlation with the score given by human annotator. String-based metrics (i.e. that compare the string representations of texts) such as BLEU (Papineni et al., 2002), METEOR (MET.; Banerjee & Lavie, 2005), ROUGE (Lin, 2004), TER (Snover et al., 2006), have been outperformed in many tasks by embedding- based metries, i.., that rely continuous representations (Devlin et al., 2019). Embeddingg-aasee (MoverS; Zhao et al., 2019) that are now the state-of-the-art domain, compare input and reference texts both represented as probability distributions and are both constructed similarly. The first step relies on a deep contextualized encoder (BERT in our case, see Devlin et al., 2019) that maps texts into elements of a collection of words, where each word represented by
metrics, c.g BertScore (BertS; Zhang et al., 2019) and MoverScore
finite-dimensional space. Each text corresponds element in Rd, where d is fixed by the encoder. The second step involves using a function that measures the similarity between the embedded texts.
We follow previous BERT-based metrics and evaluate performances of DRp.s (with p= 2, 2, = 0.01 and using the AI-IRW depth (Staerman el al., 2021b)) on two different NLG tasks namely: dala2lext generation (using the WebNLG 2020 dataset (Ferreira et al., 2020)) and summarization. For the sake of place, summarization results and additional experimental details are reported Section in the Appendix. For WebNLG, we follow standard methods assess the performance of NLG metrics (see e.g. Zhao et al., 2019). We compute the correlation with the following annotation scores: correctness, data coverage, and relevance. We report in Table 2 correlation results the WebNLG task using Pearson (r), Spearman (p) and Kendall (7) correlation coelficients. When performing a fair comparison between metrics, i.e. when DRps, W, Sliced-W, MMD are directly used on the output of BERT, we observe that DRp,s achieves the best results on all configurations. It is worth noting that DRp. also compares favorably against existing state-of-the-art NLG methods in many different scenarios and shows promising results.
6 Discussion
Leveraging the notion of statistical data depth function, a novel pseudo-metrin between multivariate probability distributions- meets the aforementioned requirements- was introduced. The developed framework exhibits inherent versatility due to numerous data depth variants. The linear approximation algorithm and the robustness property make DRp.c promising tool for large spectrum of applications beyond
12

---

Published in Transactions on Machine Learning Research (MM/YYYY)
clustering and NLG, e.g. in generative adversarial networks (GANs) or information retrieval. Moreover, recent, works extending the notion of data depth to further types of data such as functional and time-series data (Nieto-Reyes & Battey, 2016; Gijbels & Nagy, 2017), directional (or spherical) data (Ley et al., 2014), random matrices (Paindaveine & Van Bever, 2018), curves (or paths) data (Lafaye et al., 2020), and random sets (Cascos al., 2021) shall allow for the use of the proposed pseudo-metric for a wide range of applications.
Acknowledgments
The authors thank the Jean Zay supercomputer operated by GENCI IDRIS with the compute grant 2023- AD01 4668R1 and Adastra with the grant AD010614770 4770, where the NLP experiments have been done.
13

---

Published in Transactions on Machine Learning Research (MM/YYY)
References
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint ar Xrxi 1ix::1101000 701.07887, 2017.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for a web of open data. In The semantic web, pp. 722 735. Springer, 2007.
Yogesh Balaji, Rama Chellappa, and Soheil Feizi Robust optimal transport with applications in generative modeling and domain adaptation. Advances in Neural Information Processing Systems, 33:12934-12944, 2020.
Satanjeev Banerjec and Alon Lavic. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or or summarization, pp. 65-72, 2005.
Vic Barnett. The ordering of multivariate data. Journal of the Royal Statistical Society: Series A (General), 139(3):318-34) 344, 1976.
Manik Bhandari, Pranav Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig, Re-evaluating evaluation in text summarization. arXiv preprint arXiv:2010. 10X1:02000001000, 2020.
Patrick Billingsley. Convergence of probabilily measures (2nd ed.). John Wiley & Sons, 1999.
Victor-Emmanuel Brunel. Concentration of the empirical level sets of tukey's halfspace depth. Probability Theory and Related Fields, 173(3):1165-1199) 2019.
Ignacio Cascos, Qiyu Li, and Dya Molchanov. Depth and outliers for samples of sets and random sets distributions. Australian & New Zealand Journal of Statistics, 63(1):55 82, 2021.
Sung-Hyuk Cha and Sargur N. Srihari. On measuring the distance between histograms. Pollern Recognil., 35 00-1555-1370, 2002.
Emile Chapuis, Pierre Colombo, Matteo Manica, Matthieu Labeau, and Chloe Clavel. Hierarchical pre-training for sequence labelling in spoken dialog. arXie preprint or arXis:2009.11153 111009 2020.
Emile Chapuis, Pierre Colombo, Matthicu Labcau, and Chloe Clave. Code-switched inspired losses for generic spoken dialog representations. arXiv preprint arXiv:2108. 12465, 2021.
Eirini Chatzikoumi. How evaluate machine translation: review of automated and human metrics. Natural
Language Engineering, 26(2):137 2020.
Dan Chen, Pat Morin, and Uli Wagner. Absolute approximation of lukey depth: Theory and experiments. Computational Geometry, 46(5):566- 573, 2013.
Yen-Chun Chen and Mohit Bansal. Fast abstractive summarization with reinforce-selected sentence rewriting. arXiv preprint 10X.::1805.11000 2018
Lénaïc Chizat, Gabriel Peyré, Bernhard Schmitzer, and François-Xatin Xavier Vialard. Unbalanced optimal transport: dynamic and kantorovich formulations. Journal of Functional Analysis, 274(11):3090 3123, 2018.
Pierre Colombo, Wojeiech Witon, Ashutosh Modi, James Kennedy, and Mubbasir Kapadia. Affect-driven dialog generation. arXiv preprint arXiv:1904. 1XX401994,02793, 02799, 2019.
Pierre Colombo, Emile Chapuis, Matteo Manica, Emmanuel Vignon, Giovanna Varni, and Chloe Clavel. Guiding attention in sequenco-to--squeeee models for dialogue act prodiction. In Proceedings of the AAAI Conference on Artificial Indelligence, volume 34, pp. 7594 7601, 2020.
Pierre Colombo, Guillaume Staerman, Chlo6 Clavel, and Pablo Piantanida. Automatic text evaluation through the lens of Wasserstein barycenters. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10450-10448 Association for Computational Linguistics, 2021a.
14

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Pierre Colombo, Chouchang Yang. Giovanna Varni, and Chloé Clavel. Beam search with bidirectional strategies for neural response generation. aiviv preprint orXiu:21 0.03389. 2021b.
N. Courty, R. Flamary, D. Tuia, and A. Rakotomamonjy. Optimal transport for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(9):1853-188) 2017.
Nicolas Courty, Rémi Flamary, and Devis Tuia. Domain adaptation with regularized optimal transport. Tn Toon Calders, Floriana Esposito, Eyke Hüllermeier, and Rosa Meo (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 289, 2014.
Imre Csiszår. Eine informationsthoorttichhe unglcichung und ihre anwendung auf den bewis der crgodizität von markhoffsccen kette. Magyer Tud. Akud. Mol. Kulalo Inl. Koezl, 8:85-106 1963.
Marco Cuturi, Olivier Teboul, and Jean-Philippe Vert. Sinkhorn distances: Lightspeed computation of optimal transportation. In Advances in Newral Information Processing Systems, 2013.
Hoa Trang Dang and Karolina Owezarzak. Overview of the lac 2008 update summarization task. In Proceedings of the Text Analysis Conference (TAC), 2008.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1xxx:1810.04805, 2018.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Línguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, June 2019.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wure Hon. Unified language model pre-training for natural language understanding and generation. Xiviv preprint arXiv:190s 08197, 2019.
David L. Donoho. Breakdown properties of location estimators. P.h.D., qualifying paper, Dept. Statistics, Hardvard University, 1982.
David L. Donoho and Miriam Gasko. Breakdown properties of location estimates based on half space depth and projected outlyingness. The Annals of Statistics, 20:1803- 20:1803-1827, 1992.
David L. Donoho and Peter J. Hubert. The notion of breakdown point. A Festschrift for Erich Lehman, pp. 157-184, 1983.
Rainer DyckerhofT. Data depth satisfying the projection property. Allgemeines Slalistisches Archiu, 88(2): 163- 100-190,22 2004.
Rainer Dyckerhoff, Pavlo Mozharovskyi, and Stanislav Nagy. Approximate computation of projection depths. Computational Statistics and Data Analysis, 157:107166, 2021.
John H.J. Einhmahl and David M. Mason. Generalized quantile process. The annals of statistics, 20(2): 1062-1078, 1992.
Thiago Ferreira, Claire Gardent, Nikolai Tlinykh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. The 2020 bilingual, bi-directional webnlg+ shared task overview and evaluation results (webulg+ 2020). In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), 2020.
Thiago Castro Ferreira, Diego Moussallem, Emiel Krahmer, and Sander Wubben. Enriching the webnlg corpus. In Proceedings of the 11th International Conference on Natural Language Generation, pp. 171 176, 2018.
Alexandre Garcia, Pierre Colombo, Slim Essid, Florence d' Alché Buc, and Chloé Clavel. From the token the review: A hierarchical multinodal approach to opinion mining. arXiv preprint ar Xiv: 1909908877 11216, 2019.
15

---

Published in Transactions on Machine Learning Research (MM/YYYY
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. Creating training corpora for nlg micro-plamning. In 55th annual meeling of the Associalion, for Compulational Linguistics (ACT.), 2017.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. Bottom-up abstractive summarization. Tn Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4098-4094 -498-4109, 2018.
Irène Gijbels and Stanislav Nagy. On general definition of depth for functional data. Statistical Science, (4):630-68) 009, 2017.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. kernel method for the pro-sample-problem Advances in Neural Information Processing Systems, 2007.
Marc Hallin, Davy Paindaveine, and Miroslav Siman. Multivariate quantiles and multiple-output regression quantiles: From 11 optimization to halfspace depth. Ann. Statist., 38(2):635-669, 04 2010.
Abdelhamid Hassairi and Ons Regaieg. On the tukey depth of a continuous probability distribution. Statistics 8 Probability Letters, 78(15):2308 2313, 2008.
Lawrence Hubert and Phipps Arabic. Comparing partitions. Journal of Classification, 2 218, 1985.
Hicham Janati, Marco Cuturi, and Alexandre Gramfort. Wasserstein regularization for sparse multi-task regression. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1407-1416. PMLR, 2019.
Tony Jebara. Images bags of pixels. In Proceedings of the Ninth IEEE International Conference on Computer Vision, pp. 265-272, 2003.
Rebecka Jörnsten. Clustering and classification based on the 11 data depth. Journal of Multivariate Analysis, 90(1):67 89, 2004.
Chris Kedzie, Kathleen McKeown, and Hal Daume III. Content selection in deep learning models of summarization. arXiv preprint Xiv: 1ttt18810.223333 12943, 2018.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81199 1938.
Philipp Koehn. Statistical machine translation. Cambridge University Press, 2009.
Soheil Kolouri, Kimia Nadjahi, Simsekli Umut, Roland Badeau, and Gustavo Rohde K. Generalized sliced wasserstein distance. In Advances Newral Information Processing Systems, 2019.
Vladimir I. Koltchinskii and Robert M. Dudley. On spatial quantiles. Unpublished manuscript, 1996.
Gleb Koshevoy and Karl Mosler. Zonoid trimming for multivariate distributions. The Annals of Statistics, 25 (5):1998-200)) 10 1997.
Solomon Kullback. Information Theory and Statistics. John Wiley, 1959.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In International conference on machine learning, pp. 957-966. PMLR, 2015.
Pierre Lafaye, Pavlo Mozharovskyi, and Myriam Vimond. Depth for curve data and applications. Journal of the American Statistical Association, pp. 1-17, 2020. in press.
Pierre Laforgue, Guillaume Staerman, and Stephan Clémençon. Generalization bounds in the presence of outliers: median-of-means study. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 5937-5947, 2021.
Tatjana Lange, Karl Mosler, and Pavlo Mozharovskyi. Fast nonparametric classification based on data depth. Stolistical Papers, 55(1) (1):49 69, 2014.
16

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Guillaume Lecué and Matthien Lerasle. Robust machine learning by median-of-means:: Theory and practice. The Annals of Stalistics, 18(2))906-931) 04 2020.
Gregor Leusch, Nicola Ueffing, Hermann Ney, et al. A novel string-to-string distance measure with applications machine translation evaluation. In Proceedings of Mt Summit IX, pp- 240-247, 2003.
Gregor Leusch, Nicola Ueffing, and Hermann Ney. CDER: Efficient MT evaluation using block movements. In 11th Conference of the EACL, 2006.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequencc-to-soquennc pre-training for natural language generation, translation, and comprehension. arXiv preprint or urXiv:1910.138 13461, 2019.
Christophe Ley, Camillc Sabbah, and Thomas Verdebout. A new concept of quantiles for directional data and the angular Mahalanobis depth. Electronic Journal of Statistics, 8(1):795 816, 2014.
Jun Li, Juan A. Cuesta-Albertos, and Regina Y. Liu. Dd-classifier: Nonparametric classification procedure based on dd-plot. JASA, 107(498):7377-75) 2012.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74 81, 2004.
Regina Y. Liu. On a notion of data depth based on random simplices. The Annals of Statistics, 18(1):405-44 414, 1990.
Regina Y. Liu. Data Depth and Multivariate Rank Tests, pp. 279-294.
Amsterdam, 1992.
Regina Y. Liu and Kesar Singh. A quality index based on data depth and multivariate rank tests. Journal of the American Statisticol Association, 88(421):252 260, 1993.
Xiaohui Liu and Yijun Zuo. Computing projection depth and its associated estimators. Stolistics and Computing, 24(1):51 63, 2014.
Xiaohui Liu, Karl Mosler, and Pavlo Mozharovskyi. Fast computation of tukey trimmed regions and median in dimension p > 2. Journal of Computational and Graphical Statistics, 28(3):682 697, 2019a.
Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. arXie preprint ar Xiv: 10t::998.000 083458 2019.
Yinhan Lin, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint XrXio:197..1 11692, 2019b.
David J. C. MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003.
François Mairesse, Milica Gasic, Filip Jurcicek, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. Phraso-based statistical language generation using graphical models and active learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 1552 1561, 2010.
Tudor Manole, Sivaraman Balakrishnan, and Larry Wasserman. Minimax confidence intervals for the sliced wasserstein distance. Electronic Journal of Statistics, 16(1):2252 2345, 2022.
Paul McNamce and Hoa Trang Dang. Overview of the tac 2009 knowledge base population track. In Proceedings of the Terl Analysis Conference (TAC), volume 17, pp. 111 111-113, 2009.
Dan Melamed, Ryan Green, and Joseph Turian. Precision and recall of machine translation. In Companion Volume of the Proceedings of HLT-NAACL 2003-Short Papers, pp. 61 61-63, 2003.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. Xix preprint 1r11::1301.3788 2013.
17

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Karl Mosler. Multivariate Dispersion, Central Regions, and Depth. Springer, 2002.
Karl Mosler. Depth statistics. Robustness and compler data structures, 2013.
Karl Mosler and Pavlo Mozharovskyi. Choosing among notions of depth for multivariate data. Statistical Science, 2021. In press.
Karl Mosler and Pavlo Mozharovskyi. Choosing among notions of multivariate depth statistics. Slatistical Science, 37(3)3348-363 2022.
Pavlo Mozharovskyi, Karl Mosler, and Tatjana Lange. Classifying real-world data with the DDa-procedure Advances in Data Analysis and Classification, 9(3):287-314, 2015.
Debarghya Mukherjee, Aritra Guha, Justin Solomon, Yuekai Sun, and Mikhail Yurochkin. Outlier-robust optimal transport. arXiv preprint arXiv:2012.074 073663, 2020.
Debarghya Mukherjee, Aritra Guha, Justin M Solomon, Yuekai Sun, and Mikhail Yurochkin. Outlier-robust optimal transport. In International Conference on Machine Learning, pp. 7850-7860. PMLR, 2021.
Stanislav Nagy. Halfspace depth does not characterize probability distributions. Stalislical Papers, 26(3): 1135 1139, 2019.
Stanislav Nagy and Jifi Dvořák, Illumination depth. Journal of Computational and Graphical Statistics, 30 (1):78 90, 2021.
Stanislav Nagy, Carsten Schütt, and Elisabeth M. Werner. Halfspace depth and floating body. Statist, Surv., 13:52-118, 2019. doi: 10.1214/19-SSI1
Stanislav Nagy, Rainer Dyckerhoff, and Pavlo Mozharovskyi. Uniform convergence rates for the approximated halfspace and projection depth. Electronic Journal of Statistics, (4(2):3939-3917) 2020.
Shashi Narayan, Shay B Cohen, and Mirella, Lapata. Ranking sentences for extractive summarization with reinforcement learning. arXiv preprint ar 08636, 2018
Ani Nenkova. and Rebecca J Passonneau. Exaluating content selection summarization: The pyramid method. In Proceedings of the human language technology conference of the north american chapter of the association for computational linguistics: Hlt-neacl 2004, pp. 142-152, 2004.
Aui Nenkova, Rebecca Passonneau, and Kathleen McKeown. The pyramid method: Incorporating human content selection variation in summarization evaluation. ACM Transactions on Speech and Language Processing (TSLP), 4(2):4es,, 2007.
Alicia Nieto-Reyes and Heather Battey, A topologically valid definition of depth for functional data. Statistical Science, 31 (1):61177,, 2016.
Hanmu Oja. Descriptive statistics for multivariate distributions. Statistics & Probability Letters, 1(6))327-332 1983.
Davy Paindaveine and Germain Van Bever. From depth to local depth: A focus centrality. Journal of the American Statistical Associalion, 108(503):1105-1111 1119, 2013.
Davy Paindaveine and Germain Van Bever. Halfspace depths for scatter, concentration and shape matrices. The Annals of Statistics, 46(6B):3276-31) 3307, 12 2018.
Victor M. Panaretos and Yoav Zemel. Statistical aspects of wasserstein distances. Annuol Review of Statistics and Its Application, 6(1):405-433 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, July 2002.
18

---

Published in Transactions on Machine Learning Research (MM/YYYY)
François-Piens Pierre Paty and Marco Cuturi. Subspace robust Wasserstein distances. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 5072-5081 2019.
Jeffrey Pennington, Richard Socher, and Christopher Manning, GloVe: Global Vectors for Word Representa- tion. In Proceedings of the 2014 EMNLP (EMNLP), pp. 1532-1543. ACL, 2014.
Laura Perez- Berez-Bellra,hini, Rania Sayed, and Claire Gardent. Building rdf content for data-to-text generation. In The 26th International Conference on Computational Linguistics (COLING 2016). 2016.
Matthew E. Peters, Mark Neumann, Mohit Tyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL, 2018.
Gabricl Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends@ in Machine Learning, 11(5-6):355 607, 2019.
Oleksüi Pokotylo, Pavlo Mozharovskyi, and Rainer Dyckerhoff. Depth and depth- based classification with R-Package ddalpha. Journal of Statistical Software, Articles, 01(5::1-40) 2019.
Julien Rabin, Gabriel Peyré, Julie Delon, and Marc Bernot. Wasserstein barycenter and its application to texture mixing. In Alfred M. Bruckstein, Bart M. ter Haar Romeny, Alexander M. Bronstein, and Michael M. Bronstein (eds.), Scale Space and Variational Methods in Computer Vision, pp. 435-446, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg. ISBN 978-3-642--44885-9
S.T. Rachev. Probability Metrics and the Stability of Stochastic Models. Wiley Series in Probability and Statistics Applied Probability and Statistics Section. Wiley, 1991.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint 1910.1000.00000 2019.
Kelly Ramsay, Stéphane Durocher, and Alexandre Leblanc. Integrated rank-weighted depth. Journal of Multivariate Analysis, 173:51-69, 2019.
Peter A Rankel, John Conroy, Hoa Trang Dang, and Ani Nenkova. A decade of automatic content evaluat ovaluation news summaries: Reassessing the state the art In Association for Computational Lènguistics (ACL), pp. 131 136, 2013.
Peter J. Rousseeuw and Mia Hubert. Regression depth. Journal of the Americon Statistical Association, 94 (446):388 402, 1999.
Peter J. Rousseeuw and Mia Hubert. Anomaly detection by robust statistics. WIREs Dala Mining and Knowledge Discovery, 8(2):1236, 2018.
Peter J. Rousseeuw and Ida Rutz. The depth function of a population distribution. Metrika, 49(3):213-22 244, 1999.
Peter Rousseeuw and Anja Struyf. Computing location depth and regression depth in higher dimensions. Statistics and Computing, 8(3):193 203, 1998.
Alfréd Rényi. On measures of entropy and information. In Proceedings of the 4th Berkeley Symposium Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, pp. 547-561, Berkeley, Calif., 1961. University of California Press.
Geoffrey Schiebinger, Jian Shu, Marcin Tabaka, Brian Cleary, Vidya Subramanian, Aryeh Solomon, Joshua Gould, Siyan Liu, Stacie Lin, Peter Berube, Lia Lee, Jenny Chen, Justin Brumbaugh, Philippe Rigollet, Konrad Hochedlinger, Rudolf Jacnisch, Aviv Regev, and Eric Lander. Optimal-transport analysis of singlo-cell gene expression identifies developmental trajectories in reprogramming. Cell, 176(4):928-943) 943.e22, 2019.
19

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Rolf Schneider. Convez Bodies: The Brann-Minkouski Theory. Cambridge University Press, Cambridge, 1993.
Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. arXiv preprint orXim:1704.00 044404368 2017.
Robert Serfling. Depth functions in nonparametric multivariate inference. DIMACS Series in Discrete Mathematics and Theoretical Computer Science, 72, 2006.
C. E. Shamnon. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379 423, 1948.
J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis & Machine Intelligence, (2(08):888 905, 2000.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pp. 231, 2006.
Lucia Specia, Dhwaj Raj, and Marco Turchi. Machine translation evaluation versus quality estimation.
Machine translation, 24(1):39 50, 2010.
Bharath K. Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert R. G. Lanckriet. On the empirical estimation integral probability metrics. Electronic Journal of Stalisties, 6:1550 1599,
2012.
Guillaume Staerman. Functional anomaly detection and robust estimation. PhD thesis, Institut polytechnique de Paris, 2022.
Guillaume Staerman, Pavlo Mozharovskyi, and Stéphan Clémengon. The area of the convex hull of sampled curves: robust functional statistical depth measure. In Proceedings of the 23nd International Conference Artificial Intelligence and Stalislics, volume 108, pp. 570-579, 2020.
Guillaume Staerman, Pierre Laforgue, Pavlo Mozharovskyi, and Florence d'Alché Buc. When ol meets mom: Robust estimation of wasserstein distance. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130, pp. 136-144, 2021a.
Guillaume Staerman, Pavlo Mozharovskyi, and Stéphan Clémengon. Affine-invariant integrated rank-weighted depth: Definition, properties and finite sample analysis. arXiv preprint arXiv:2106.1106 11068, 2021b.
Werner. A. Stahel. Breakdown of covariance estimators. Technical report, Fachgruppe für Statistik, ETH, Zürich, 1981.
Peter Stanchev, Weiyue Wang, and Hermann Ney. Eod: Extended edit distance mcasure for machine translation. In Proceedings of the Fourth WMT (Volume 2: Shared Task Popers, Day 1), pp. 514-520, 2019.
Wolfgang Stummer and Igor Vajda. On bregman distances and divergences of probability measures. IEEE Transactions on Information Theory, 58(3):1277 1288, 2012.
John W. Tukey. Mathematics and the picturing of data. In R.D. James (ed.), Proceedings of the International Congress of Mathematicians, volume 2, pp. 523 531. Canadian Mathematical Congress, 1975.
Cedric Villani. Topics in Optimal Transportation. Graduate Studies in Mathematics Series. American Mathematical Society, New York, 2003.
Danqing Wang, Pengfei Liu, Yining Zhong, Xipong Qiu, and Xuanjing Huang. Heterogencous graph neural networks for extractive document summarization. arXiv preprint orXi::200,.12883 2020.
Weiyue Wang,
Peter, Hendrik Rosendahl, and Hermann Ney. Character: Translation edit rate character level. In Proceedings of the First WMT: Volume 2, Shared Task Papers, pp. 505-5 505-510, 2016.
20

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Isung-Hsien Wen, Milica Gaăié, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. Semantically conditioned LSTM-basee natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Notural Language Processing, pp. 1711-111111111 1111-122,, 2015.
Wojciech Witon, Pierre Colombo, Ashutosh Modi, and Mubbasir Kapadia, Disney at iest 2018: Predicting emotions using an ensemble. In Proceedings of the 9th Workshop on Compulational Approaches to Subjectivity, Sentiment and Social Media Analysis, pp. 248 253, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:910.008
2019.
Wonjin Yoon, Yoon Sun Yeo, Minbyul Jeong, Bong-Jun Yi, and Jaewoo Kang. Learning by semantic similarity makes abstractive summarization better. arXiv preprint arXiv:2002.07761 2020.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted gap- sentences for abstractive summarization. In Proceedings of the 37th International Conference Machine Learning, volume 119, pp. 11328 11339, 2020.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. Xiv preprint 1ix::199440990 2019.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. Moverscore: Text genera- tion evaluating with contextualizod embeddings and carth mover distance. arXiv preprint ar Xiv: 190.0000.0022,, 2019.
Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. Searching for effective neural extractive sunmarization: What works and what's 's next. arXiv preprint 19X.v:1907.3341, 2019.
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. Extractive summarization as text matching. arXiv preprint ar orXie:2004.0877 087995 2020.
Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. Neural document summarization by jointly learning to score and select sentences. ai preprint 1rXim:1807.02300 2018.
Zuo. Projected based depth functions and associated medians. The annals of statistics, 31(5):1460 1490, 2003.
B.Y. Zuo and R. Serfling. Goneral notions of statistical depth function. The Annals of Statistics, 28(2): 461-482, 2000.
Appendix
This Appendix is organized as follows:
Appendix A contains additional notations, preliminary results and additional information about the proposed pseudo-metric.
Appendix B contains the proofs of the propositions/theorrnss provided in the paper.
Appendix C contains approximation algorithuns to compute
depth.
Appendix contains additional synthetic experiments.
Appendix E contains details on experimental settings of NLP applications.
21

---

Published in Transactions on Machine Learning Research (MM/YW) MM/YYYY)
A
Preliminary Results
First, we introduce additional notations and recall some lemmas, used in the subsequent proofs.
A.1
Hausdorff Distance
The Hausdorff distance between two bounded subspaces K2, K2 of Rd is defined as:
(K1, = max sup inf || - vll. (EeR, yEk2
sup inf yeka #E
||x - wll y
Furthermore, if 1C and K2 convex bodics, i.c. non empty compact convex scts, the Hausdorff distance can be reformulated with support functions of K1,
(z)) = sup lix, (u) (x)( aES4-1
where by, (u) (u, a K.).
A.2 Quantile Regions
Let e gd-1 and X ~ μ where μ M1(X) with X c Rd. We define the (1 B) directional quantile of distribution j in the direction u
fe = inf {t E R: P((u, S 2 83
and the upper (1 3) quantile set of μr:
: (u, S shad Vu AuEst-1 Q1- = {x ER.:
A.3 Auxiliary Results
We now recall useful results, so as to characterize the halfspace depth regions.
Lemma A.1 (Brunel, 2019, Lemma 1). Let E Mi(X)), for any B E 1), it holds: Df. = =Q--" Lemma A.2 (Brunel, 2019, Proposition 1). Let μ E Mi(x) with a (1 - 3) directional quantile qpa for any E gd-1 Assume that u + q/-d are sublinear, i.e., ap I 9.-0 + X 9/.0 <X> 0. Then for any u E sd-1, holds MQ)-2(1) = 9/.00
Lemma A.3. Let d and X pi, Y1 in be two random variables where pl, V1 are univariale probability distributions. Denoting by Fxi the quantile function of X2, then the depth-trimmed region based meeudo-metric (associated with the halfspace depth) is defined as
1/2
DR (11, v1) =2
max
( FFH(P). 17 - -F (1)1p) dg.
Je/2
Proof. In dimension one, the halfspace depth of any t R w..tt. μr and vi boils down to
D(t, p1) min (f), (x) (xx and D(t, min ( (r)
and for any 7 E [0, its upper-level sets to intervals
D2, = (Fx(()) F 7) and DE, = (F))()), F ()]]
(5)
22

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Now, the quantile function a(),.) can be explicitly derived function of B E [0, 1]:
a(3, = sup {7 E [0, pi (1 (1 2
ssup {7E [0, 1- 2 2
1-8
2
Following the same reasoning, it holds a(3,v1) = 1-8 2- Further, by change of variables
1ee du (DL-9)/2 D d3 = l du (D), D2,) dq.
e/2
Combining (5) and the Hausdorff distance definition recalled in Section A.1 lead to the result.
A.4 Additional information
This part provides additional information and remarks about the proposed pseudo-metric. We also summarize propertios of DRp.c in Table 3 w.r.t. different depth functions used in the paper.
First, in some cases of convex data depth, the pseudo-metric could define a distance. DR is a distance if and only if the upper-level sets of the chosen data depth fully characterize probability distributions. To our knowledge, It has been proved only for the Halfspace depth under mild assumptions (Hassairi & Regaicg, 2008; Nagy al., 2019) and the Zonoid depth assuming the first moment on the distribution (Mosler, 2002).
Remark A.4 (FLEXIBILITY.). One of the main benefils of our pseudo-metric ils flexibilily. Our general definition allows the use of any depth function, see e.g. Mosler & Mozharouskyi (2022) for a review of the main depths, at the price of choosing one that is relevant for the underlying data.
Remark A.5 (ROBUSTNESS.). The trimming improves the robustness of DR and sliced- Wasserstein. While il is not the only source of robustness in DRp.s; the Wasserstein metric is generally known be non-robust (Mukherjee et ol,, 2021; Balaji et al., 2020). Indeed, the robustness in DRpp also comes from the robustness of the chosen depth and the trimming step may not be helpful in specific situations.
Remark A.6 (TRIMMING COMPARISON WITH TSW.). The trimming occurs in different spaces for the trimmed sliced Wasserstein and DRp... Our pseudo-metris trimming relies on peeling the larger quantile regions (depth regions) that directly consider the multivoriate data' 's structure. In controst, the trimming in SW occurs at the projection level. Since, in practice, the projections are chosen uniformly on the unit sphere, this trimming does not consider the correlation of the data, for example. It may remove non-outlier points of the data.
B Technical Proofs
We now prove the main results stated in the paper.
B.1 Proof of Proposition 3.4
For any 0 < 1 1-c with e (0, 1]. and any μ E Mi(X), Ve Mi()'), (o( (2(0) are non-empty compact subsets of Rd due to the properties (D2-D3). The Hausdorff distance dæ, recalled in Section A.1, is known to be a distance on the space non-empty compact sets which implies that DRp satisfies positivity, symmetry and
the triangle inequality (thanks to Minkowski inequality). Tf f then De( (z(1) A 7 E [0, 1 s- which
23

---

Published in Transactions on Machine Learning Research (MM/YYY)
Halfspace
Projection
IRW
A1-IRW
Pseudo-metric
Isometry invariance
X
Fast approximation with support vector
X
Depth computation
O(Kn(d v logn))
O(ndK)
O(ndK)
O(d + ndk)
Robust depth regions
X
Table Properties satisfied by DRpps associated with the halfspace, projection and affine-invarianl integrated renk-weighted depths.
leads to D (ni v) = O. The reverse is not true. DRp (1, v/) = implies D2 D2(()) v E [0, 10,1-5 that not leads μ = v. Indeod, convex depth regions do not characterize probability distributions in general (see Nagy, 2019 for the halfspace depth) that would be the first step order prove the previous entailment.
B.2 Proof of Proposition 3.5
Let A E gaxd be non-singular matrix and b Id such that g : x Ar b. Then, it holds:
( 1-c
dn((D(()) 9tH
_a(3))) ) d3
DR (9:M) 9/v)
J
(dn(ADz()) bb, (D) + +b] d8,
(6)
where (i) holds because any data depth satisfies (D1) by definition. Furthermore,
(n(AD + (D) +b = max
sup xED yED
inf
||Ax - Ayll,
sup vED
inf 2ED
||Ax - Aull ]
(8)
max
inf wED (E))(A)
sup
||x - all.
sup wED
inf ZED
||0 - y/l
dd(DD D)())),
where (i holds by virtue of hypothesis AAT = Ig- Replacing it in (6) yields the desired results.
B.3 Proof of Proposition 3.6
First assertion. Denote Z1, Z2 two random variables following n², v² respectively. Assume that X, Y, Zz, Zg are defined on the probability space (2, P). For any x E I2 and 3 E [0, - =),
E (a(0) + +H),(r) 2 a(8) - VuEs P(uu, - (u, 2 a(B)
VuE dd-1 P(uu, Zi mi) (u, 2 a(3)
YuES VuEgd-1 gd-1 P(uu, Zı) I (u, mi)) 2 a(3)
x mi E (8))
The same reasoning holds for and v²_ Following this, for any [0, 1 -]] and gd-1 it holds:
24

---

Published in Transactions on Machine Learning Research (MM/YYY)
h pocto (u) pocto (u) (u, m1)
and 1poto (u) prsso) (1) {u, m2)-
Thus it holds:
/-¹ sup h Minc00 (u) - (u, m,) h putoo (u) {u, my) dg
DR (1,
wEB4-1
sup ueS
(u, m m2)
sup | papa (u) - h paler (u) dg
MEB
+ 2 sup (u,, m, - mz /
sup
| [MDacos(u) (w) - MD (w)] dB
weSd
uESd
m+ m2|12 2 DR (ut v²))"""">+>"" m m2 -mllDRR ,²)
(7)
On the other side, we have:
DR3 2 sup
(u, m1 m2) m²))²+ + fo
sup wEBd
|| pocoo (u) - hDg. (w)) d3
wgg1-1
sup
(u,, m1 m2)1 f
sup
1 Dutcs (u) h pacos (u)l db
w
uggd
||m
m2 + DRG (u", 2||m: D2 (M" ,²).
(8)
Combining (7) and (8) lead to the desired result.
Second assertion. For any u E gd-1, the (1 - a(3)) quantiles of random variables (u, X) and (u, Y) such that (u, ~ ((u, mi). u U) and (u, Y) ~ N(uu, me), u Egu) are defined by
qr () (u, mi) x- (1- Vu Enu
L 2 (u, m2) + x + a1-1((3)) Ezu,
Ezu,
where + is the cumulative distribution function of the univariate standard Gaussian distribution. Now, apply Lemma A.2, it is sufficient to prove that directional quantiles are sublinear. It holds using subadditivity of the square root function. Indeed, for any u, e gd-1 and x > O, we have:
(u Au, mi) + + a33) / (u+ Ao) T Ei(u 2o) (u, mi) + mi) + t t- (1--((3)) a(3)) (u Xo) E Av)
S (u, mm) + mm)
+ a x a(3)) VuTEIu + V V
1-000) A qu.0 1-a00)
The same reasoning holds for v. Applying Lemma A. and Lemma A.2, for any E gd-1, we have h DDe(s) (w) = (3)) and h (u) = ai (0)) It follows:
DRI,(H, fr-e 1-c
dH (eg())))))))) d3 = f'r- 1-c
on u
sup
1 puto h 1g00 (n)
D
d8
1-c
sup
(u. - m2) + x ((1 a3)) /Vu VuT Eju Vu Σgu
d3
-
11-
< [m mzll
Jo
sup wE3W-1
T Φ-1(1 - a(3)) V Vu Vu Σzu
d3
=
[
mall
+ C.
sup
VVu T E1u - E2u
MEBd
25

---

Published in Transactions on Machine Learning Research (MM/YYY)
with C. -o' - @ (1-((3)) d3. The lower bound is obtained by means the same reasoning. Notice that
m m2|1
sup wE84
(u, mi m2)
sup wesd
(u, mi m2) dB.
Introducing h D (u), 1o (u) and using triangular inequality, subadditivity of the supremum and linearity of the integral, we obtain:
m --2m & DRI,e + C. sup
V EuuEEu Eiu Vu Σz
wese-1
which ends the proof.
B.4 Proof of Proposition 3.7
For D.p. to break down at Snz it needs to have at least trimmed-region that breaks down. Then the breakdown point of DRp. is higher than the minimum of the breakdown point of each region. Indeed, we have
o
DRp (pxto: = +00 Dallsin)) +00
B S.) = min
sup ZinnZo
n
sup du (Da(Paeetto) Dan Zr...
min
min nn+o
8, l-e)
min 0.1
BP(Da(B.iu)) An
Sn).
Now applying Lemma 3.1 Donoho Gasko (1992) and Theorem in Nagy Dvořák (2021), lower bound of the breakdown point of each halfspace region, for every [0, 1 e), is given by
(n) a(1 11 + [nox(1 - Pm) a(1 - E, (n)) Ommxx
if a<1 e, fn) 5
(max(Pa) I+ (fbn)
BP(Dal Ann) Sn) ,n))
otherwise,
1 + amax (pn)
where amax
max xER
HD (z).
C Approximation Algorithm
In this part, we display the approximation algorithms of the halfspace depth (see Algorithm 2), the projection depth (sce Algorithm 3) and the AI-IRW depth (see Algorithm 4, proposed in Staerman al., 2021b) used in the first step of the Algorithm
Algorithm 2 Approximation of the halfspacc depth
Initialization: E guxd K.
Construct U E Bd by sampling uniformly K vectors UI...., Un in gd-1
Compute M - XU
Compute the rank value ai. k), the rank of index i in M. for every i n and & < K
4: Set D; = min a(i, k) for every i < n KSK
Output: D.M
26

---

Published in Transactions on Machine Learning Research (M//YYYY)
Algorithm Approximation of the projection depth
Initialization: X E Ruxd, K.
Construct U gdxk by sampling uniformly vectors Ui,..., UK in ga-1
Compute M = XU
3: Find Mmod. the median value of M.,,, V k S K
4: Compute MAD, = median MI,A Mmeddk i is n for k S K
Compute V s.t. V, M Mmed, /MAD
6: Set Dj = min 1/(1 Vi,e) for every is
ks
Output: D,M
Algorithm 4 Approximation of the AI-IRW depth
Initialization: X E Ruxd, K.
Construct U Ra by sampling uniformly vectors UI...., UK in ga-1
Compute Σ using any estimator
3: Perform Cholesky or SVD on Σ to obtain Ê-1/2
4: Compute V = /=S-/////E-11/0
5: Compute M XV
6: Compute the rank value aii, k), the rank index i in M. for every I n and k S K
7: Set D; R EE (i, for every i S
Output: D,M
D Additional Experiments
D.1 Illustration of Data Depth Contours
Figure 6, which plots family of AI-IRW (using MCD estimator) depth induced trimmed-contours for a dataset contaminated with outliers, illustrates its robustness.
Figure 6: AI-IRW depth contours for a bivariate sample contaminated with outliers.
D.2 Illustration of the Depth Trimmed- Heimmed-Regions based Sseudo-Metree
Figure 1, which plots family of (approximated) AI-IRW depth induced trimmed-regions for two datasets contaminated with outliers, illustrates the key idea of proposed pseudo-metric.
27

---

Published in Transactions on Machine Learning Research (MM/YYN)
D.3 Empirical Analysis of Statistical Rates
This part presents complementary results of those obtained in the Section 5.1. Considering the same experiment as in the core paper, Figures 7 and 8 display the results of the same experiment but with dimension d = 5 and d = 10, respectively.
Error
10
100
na
na 20
na 100
10a
10a
10
104
10a
10
Number of samples
Number of samples
Figure 7: Empirical analysis of statistical convergence rates. Resulting error of the proposed pseudo- metric when increasing the sample size using the projection depth (left) and the halfspace depth (right) for various na parameters with 5.
Error
na
n₂ 20
10a
na 100
10a
10
102
10a
10a
101
102
10a
104
Number of samples
Number of samples
Figure 8: Empirical analysis of statistical convergence rates. Resulting error of the proposed pseudo- pseudo-metric metric when increasing the sample size using the projection depth (left) and the halfspace depth (right) for various no parameters with d = 10.
D.4 The Influence of the Parameter E
The parameter & plays the role of the robust tuning parameter of DR2,.- In this part, we complete our theoretical results provided in Section 3.2. We assess the robustness of our pseudo-metric making varying the parameter Precisely,
simulate two normal samples X and Y from two standard Gaussian distributions in dimension two with sample size of 10000. From that, construct abnormal samples with proportion of anomalies equal to (1%, 10%, 20%}. To that end, we choose a proportion of normal samples and replace their first (for X) and second (for Y) coordinates as follows: Xanom = 30 + 50Z and Yanom = -30 50Z where follows a uniform distribution on [0, 1]; leading to points far from the normal distributions. Thus, we compute DR2, with both robust and non-robust data depths, i.c. the projection and halfspace depths between X and Y being used as benchmark. Further, we compute DR2,, between abnormal samples and report mean error (comparing values obtained between normal samples and values obtained between
28

---

Published in Transactions on Machine Learning Research (MM/YYYY)
abnormal samples; averaged over ten runs) Figure 9. First, when computing with robust depth function, we can see that the robustness of the proposed pseudo-metric relies directly the parameter &. This is shown by the presence of an elbow when the parameter s reaches the level of the proportion of anomalies. In contrast, we can see that for a non-robust depth function such as the halfspace depth, our proposed pseudo-metric becomes non-robust once the abnormal proportion is higher than 1%, leading to a poorly robust. depth. This experiment then confirms our theoretical results on the Breakdown Point of DRp. highlighted in Propostion 3.7. The parameter e provides robustness our pseudo-metrie when combined with robust. depth function.
1%
104
20%
10
30%
Error
100
10a
10-1
0.1
0.2
0.3
0.4
0.0
0.2
0.4
E
Figure 9: Influence of the parameter on the robustness of the proposed psoudo-metric with a robust depth function (the projection depth, left) and non-robust one (the halfspace depth, right) for various proportion of anomalies.
D.5 The Choice of the Parameter na
Proposition 3.6 allows derive a closed form expression for D) (2, v) when μr, v are Gaussian distributions with the same variance-covariance matrix. In order investigate the quality of the approximation on light-tailed and heavy-tailed distributions, of na between sample of 1000 points stemming from μ >~((0,, Σ) for d E {2,3, 10), Σ drawn from the Wishart distribution (with parameters (d, Ia)) on the space of definite matrices and three different samples (which yields nine settings). These three samples are constructed from 1000 observations stemming from elliptically symmetric Cauchy, Student-tz and Gaussian distributions all centered at 7 Results that report the averaged approximation error and the 25-75% empirical quantile intervals are depicted Figure 10. They show that DRp, converges slowly for Cauchy with growing Na, while converges with small na for Gaussian and Studenl-la distributions.
focus on computing DR2 (with K 500) for varying number
Relative error
10
-0
d=2
d=3
d=10
10
10
4x10-1
3x
12
flg
fg
fg
Figure 10: Relative approximation error (averaged 100 repetitions, y-axis in log scale) of D for elliptically symmetric Cauchy (left), Student-lz (middle) and Gaussian (right) distributions for differing numbers of Bo.
29

---

Published in Transactions on Machine Learning Research (MM/YVV (M//YYY)
D.6 Robustness to Outliers
Datasets on which experiments regarding "Robustness to outlier" in Section 5 have been performed are displayed in Figure 11.
Source distribution distriuiitttbuutin distributian Outilers
20
6
2
-30
10
20
Figure 11: datasets related to robustness experiments depicted in Section with 20% of outliers for fragmented hypercube (left) and Gaussian (right).
D.7 Trimming approximation comparison between DRp. and TSW p.c
This part investigates the effect of the trimming parameter for DRg. and the trimmed sliced Wassorstein, T TSW,,s p.s, regarding the stability and elficiency of the trimming. We simulate two datasets of size 100 stemming from d-dimensional centred Gaussian distribution with d E {3, 103 and E drawn from Wishart distribution. First, We compute the two metrics with p 2 and ε O. Further, and TSW2, for & E [0.05, 0.45] and compute the absolute differences with the non-trimmed values, i.e. DR2,0 - DR2,el and (TSW20 TSW2.ll- The experiment is repeated 10 times, and the results are reported in Figure 12. While the trimming effect does not depend on the dimension the trimmed sliced Wasserstein, the trimming effect drastically deteriorates the given value of the sliced Wasserstein. contrast, the trimming effect on DR deteriorates much less the metric quality, even if this increases with the dimension.
compute DR2,-
D.8 Illustration different depths and outliers
This part provides intuition between three data depth, IRW, AI-IRW and the robust version of AI-IRW regarding how they discard abnormal data. The experiment conducted as follows. We simulate two dimensional Gaussian distribution and add some isolated anomalies (orange) and aggregated anomalies (red) at hand. We compute the three depth functions on this dataset and draw the several quantile regions, see Figure 13. Quantiles regions defined by each depth are of different shape, characterizing abnormal data in different manners. Regarding the pseudo-metric DRp.s discarding abnormal data will depends on (1) the quantiles regions defined by the chosen data depth and (2) the trimming parameter e. The quality of the DRp.e in discarding anomalies relies on the quality of the chosen depth functions that are gathererd, c.g. Mosler & Mozharovskyi (2022).
E Application to NLP
In this section, we gather details on experimental settings and additional results on the automatic evaluation of natural language generation (NLG)..
E.1 Extended related works Automatic Evaluation of NLG
Many metrics have been recently introduced for the automatic evaluation of text generation. In this work, rely untrained metrics. These metrics can be grouped into two categories: string-based metrics that
30

---

Published in Transactions on Machine Learning Research
DRp.r
d=3
TSWp.
d=10
Relative estimation error
10.2
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.60
0.45
Trimming value
Figure 12: Approximation error between the metric computed with e = 0 (without trimming) and e varying in [0.05, 0.45] for both DRp, and TSW p.s.
IRW
AI-IRW
Robust A1-IRW
20
16
Figure 13: Contours of different. depths computed on data with outliers, IRW (left), AI-IRW (middle) and the robust version of A1-IRW (right). Red crosses and orange triangles are two different type of outliers.
31

---

Published in Transactions on Machine Learning Research (MM/YYYY
depend the string representation of the imput texts to compute the similarity score and embedding-basee metrics that rely on continuous representation of the texts.
String matching metrics can be divided into two categories: N-gram matching and edit distance-based metrics. Perhaps the most used N-gram matching metrics are BLEU, ROUGE and METEOR. Edit distance-based metrics (e.g. TER; Snover et al., 2006) measure the distance as the number of basic operations such as "edit'/ aedit'/delete" /'insert'. Variants of TER include CHARACTERE (Wang al., 2016), CDER (Leusch et al., 2006), EED (Stanchev al., 2019). String- String-based metrics fail to produce meaningful scores in the case of paraphrases, especially if no common n-grams are found between the candidate and the reference text.
The second category of untrained metrics (namely embedding basedd metrics) achieves state-of-the-art performance in many NLG evaluation tasks and has been introduced to address the issues mentioned above. Originally introduced for the widely used words embedding (Garcia et al., 2019; Colombo et al., 2019; 2020; 2021b) such as Word2Vec (Mikolov et al., 2013) or Glove (Pennington et al., 2014), this class of metrics has leveraged recently introduced contextualized word representations (CWR). CWR such as BERT, ELMO (Peters et al., 2018), HILAMOD (Chapuis et al., 2020; 2021) or ROBERTA (Liu al., 2019b) popular in NLP (Witon et al., 2018) as they achieve SOTA performance many tasks. The two most popular metrics are MoverScore and BertScore.
E.2 Evaluation
For the task of evaluation of text generation, we assuine that we have access to dalaset (TR. (T,, (172)) yer where T2 represents the i-th generated text by the j-th natural genera- tion system, and h(Tz,) represents score assigned by the human annotator' to Té and TR, is the reference text. or is the number of available texts, and ns the number of different systems.
To assess the relevance of an evaluation metric M, the correlation with the human judgment is considered one of the most important criteria (Banerjee & Lavie, 2005; Koehn, 2009; Chatzikoumi, 2020). To measure this correlation, two evaluation strategies are commonly adopted and built on top of classical correlation measure, denoted C, e.g. Kendall (T; Kendall, 1938), Pearson (r; Leusch et al., 2003) or Spearman (p: Melamed et al., 2003).
The text level correlation (Cteat) measures the ability of the metric to distinguish between badly and well generated text. Formally, Crent is defined as follows:
Crert
C(M H
(9)
NT
F=1
M.ezt [TT(TRR, T
(T.R. To
H
( ... (TT.))
The system level correlation (Cays) assesses the ability of metric to distinguish between good and bad systems. Formally, Ceys is defined as follows:
C sys
=M Ho.*),
(10)
M( Te.)
M (T, Ims)
nT
1 Re
sys
(TT)))
h(TT:))
nT
nT
We refer the reader Bhandari al. (2020) for further details on the evaluation of text generation.
In practice averaged score is considered as each sentence is annotated by 3 different annotators. The considered datasets directly provide the aggregated score.
32

---

Published in Transactions on Machine Learning Research (MM/YYY)
Correctness
Data Coverage
Relevance
P
P
p
DRp.
89.4
80.0
92.6
84.2
58.3
72.3
86.2
62.7
72.9
Wasserstein
86.2
73.0
86.7
80.4
45.3
62.3
83.8
51.3
67.6
86.1
73.0
85.8
80.9
45.5
60.0
82.0
51.3
68.2
MMD
25.4
71.7
8.3
19.1
45.3
10.0
26.1
51.3
15.0
BertScore
85.5
73.3
83.4
74.7
53.3
68.2
83.3
65.0
79.4
MoverScore
84.1
73.3
84.1
78.7
53.3
66.2
82.1
65.0
77.4
BLEU
77.6
60.0
66.3
55.7
36.6
50.2
63.0
51.6
65.2
ROUGE-1
80.6
65.0
65.0
76.5
60.3
76.3
64.3
56.7
69.2
ROUGE-2
73.6
58.3
63.3
54.7
35.0
43.1
62.0
46.7
60.8
METEOR
86.5
70.0
66.3
77.3
46.6
50.2
82.1
58.6
65.2
TER
79.6
58.0
78.3
69.7
38.0
58.2
75.0
77.6
70.2
Table 4: WebNLG 2020 (full results): absolute correlation at the system level with three human judgment criteria. Best overall results are indicated in bold, best results in their group are underlined.
E.3 Results on Data2text
In this section, we gather further details and results on data2text automatic evaluation.
E.3.1 Task Description
In WebNLG 2020, the goal is to create new efficient generation algorithms that can verbalise knowledge-based fragments. These algorithms are called Knowledge Base Verbalizers (Gardent et al., 2017) and are used during the micro-planning phase of NLG systems (Ferreira el al., 2018). WebNLG has been gathered to be more representative of the progress of recent NLG systems than previously existing task-oriented dialogue datasets (see e.g. SFHOTEL (Wen et al., 2015) and BAGEL (Mairesse et al., 2010)). As previously mentioned for the data2text task work the W202 challenge (Gardent et al., 2017; Perez- Beeer-Beltrachii et al., 2016). Data and system performances can be found in https:///ebbl: htap::://bebblg-ccallenegg challenge loria. fr/. The task consists in mapping RDF triples to natural language (RDF format is used for many application For WebNLG 2020, the triplets are extracted from DBpedia (Auer et al., 2007). Data have been made freely available from the authors at https: //gitlab.
including FOAF (sce http://wwtt wwwff wwwpppot...o...... www.foof--rroecctttt
wwwpaaaoonnneeeennggaaaaaaeeee /ttpa//eeee//eeeeee/eeeeeeeeeeee wttpee//mmtteeeeeeeeeeeeeeeeeeee v3.0. To compose this dataset, systems (both symbolic and neural-based) have been used. The final dataset composed of over 3k samples of human annotations (see for more details).
htwp///bbnlghcahhlggggeeeee loria. wff.aaat/fillle//eeel-------- http:////eeeeeooo---o----reaaaaa pdf
Example: Given the following triplet (John_Bi (John_Blaha birthDate (942_08_26) (John_Bil Blaha birthPlace San uon_Antoooo) (John_B1ag Blaha job Pilot) the ground-truth reference is John Blaha, born in San Antonio on 1942-08-26, worked as pilot.
E.3.2 Results
We gather in Table 4 complete results on the WebNLG tasks including results on ROUGE-2. To compare DR,, (with 0.01, n, p 2) with the different metrics (i.e. Wasserstein, Sliced- Wasserstein, MMD), work on Roberta-based model from the HuggingFace hub (Wolf et al., 2019) and extract representation from the 11th layer. From Table 4, we observe similar behavior from BertScore and MoverScore. This similarity has also been reported in different setting in the previous work of Zhao el al. (2019). Overall, we observe that DRp. is always among its group's top-scoring metrics and achieves the best overall results on several configurations.
is worth noticing that DR,,s only relies on information available the candidate and the reference text. In contrast, BertScore and MoverScore use IDF information computed on every dataset.
33

---

Published in Transactions on Machine Learning Research
E.4 Results Summarization
In this section, we gather experimental details and results on the automatic evaluation of the text summia- rization task.
E.4.1 Task Description
Text summarization has attracted much attention in recent years (Zhang al., 2020). Two types of models exist: estractive and abstractive. In extractive summarization, the system copies chunks of informative fragments from the input texts, whereas, in abstractive summarization, the system generates novel words. In this section, we describe our experimental setting. We present the tasks and the baseline metrics used for the automatic evaluation of summarization. We work with the dataset from Bhandari et al. (2020) for this task. This dataset has been introduced to solve several flaws (Rankel et al., 2013) present existing summarization datasots such as TAC (Dang & Owezarzak, 2008; McNamee Dang, 2009). The datasct has been annotated using the pyramid score (Nenkova al., 2007; Nenkova & Passonneau, 2004) and automatically built from the CNN//Dall News (Bhandari et al., 2020). It gathers 11 490 summaries coming from 11 extractive systems (See et al., 2017; Chen & Bansal, 2018; Raffel et al., 2019; Gehrmann et al., 2018; Dong et al., 2019; Liu & Lapata, 2019; Lewis et al., 2019; Yoon et al., 2020) and 14 abstractive systems (Zhou et al., 2018; Narayan et al., 2018; Kedzie et al., 2018; Zhong et al., 2019; Liu & Tapata, 2019; Dong et al., 2019; Wang et al., 2020; Zhong et al., 2020).
Example: The goal is to assign similarity score between reference text: "Manchester United take Monchester City Sunday. Match will begin at pm local time at United's Old Trafford home. Police have objections to kick-off being so late in the afternoon. Last lale aflernoon weekend kick-off in the Munchester derby saw 34 fans arrested at Wembley in 2011 fa cup semi-final" and the text generated by a NLG system: Manchester Derby takes place at Old Trafford on Sunday afternoon police have no objections to the late afternoon kick-off both sides are challenging for lop-four spot in the Premier League the man in charge of patrolling the sell-out clash has no such fears".
E.4.2 Results
We gather in Table 5, the results on the summarization task. We use bert-based uncased model and rely on the representations extracted from the 9th layer (similarly to BertScore). For this experiment the following parameters are used: & 0.01, n, 5, p 2. For this task, we can reproduce results from Bhandari et al. (2020) where the different behavior regarding the extractive and the abstractive systems is also observed. In this experiment, we observe that DRp,c can achieve stronger results than other metrics based Wasserstein, Siced-Wasseeree and MMD. We also observe that DRp. outperforms MoverScore and BertScorre on extractive systems (on and T). We believe these results support our approach.
34

---

Published in Transactions on Machine Learning Research (MM/YYY)
Abstractive
Extractive
P
DRp
72.1
72.1
70.1
91.5
91.5
69.2
Wasserstein
71.0
70.4
71.1
74.2
74.2
40.0
Slicod-Wars Wasscrstein
70.1
68.7
71.0
72.4
73.9
69.2
MMD
68.2
67.5
67.9
75.6
75.6
56.1
BertScore
71.7
71.9
72.0
70.9
72.9
73.8
MoverScore
72.4
71.9
73.0
76.1
76.1
47.4
ROUGE-1
73.5
73.0
74.4
72.2
74.0
69.1
ROUGE-2
73.0
73.5
73.0
55.1
53.2
69.1
JS-2
68.9
6.8
69.8
92.9
5.5
19.0
Table 5: Summarization: absolute correlation coefficients (using Pearson (r), Spearman (p) and Kendall (r) coefficient) between different metrics on text summarization. Best overall results indicated in bold, best. results in their group are underlined.
35


Reviews:
[{"review_id": "puuJbvuOAD", "reviewer": "Reviewer_vcJR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The manuscript is, in my opinion, well-written and accessible to the audience of TMLR. The introduction is simple to read, and a thorough background is given on the notion of statistical depth, and the traditional properties of depth functions. The theoretical results and proofs are written with sufficient care, and the numerical results are presented with sufficient computational details. Furthermore, I have read several of the theoretical results, and I am not aware of any mistakes. To the best of my reading, I therefore believe the manuscript meets the presentation and correctness standards of TMLR, and warrants publication. My remaining comments below deal with the possible impact of the proposed divergences, which I kindly ask the authors to consider in their revision. \n\nThe idea of constructing a metric between probability distributions using the Hausdorff distance between depth regions is new, to the best of my knowledge. Nevertheless, I do not think the authors have adequately substantiated why this metric is more favorable than other popular metrics, in particular the Wasserstein distance. A long line of recent optimal transport literature (cf. [1]) has argued that optimal transport maps can be used to define quantile functions in general dimension, thus there is a heuristic sense in which the Wasserstein distance is connected to the comparison of multivariate quantiles, similarly to the divergence of the authors. Furthermore, there exist trimmed analogues of the Wasserstein distance [2-3], which have a trimming constant that plays a similar role as the parameter \\epsilon appearing in the current manuscript. These metrics share similar robustness properties as the divergence of the authors. There also exists a trimmed analogue of the Sliced Wasserstein distance [4], which is not only robust but can also be computed easily. It is reasonable to expect that these trimmed optimal transport metrics share similar qualitative benefits as the proposed divergence, thus I encourage the authors to expand their discussion on its possible advantages. \n\n[1] Hallin, M. (2022). Measure transportation and statistical decision theory. Annual Review of Statistics and Its Application, 9, 401-424.\n\n[2] Munk, A., & Czado, C. (1998). Nonparametric validation of similar distributions and assessment of goodness of fit. Journal of the Royal Statistical Society Series B: Statistical Methodology, 60(1), 223-241.\n\n[3] Alvarez-Esteban, P. C., Del Barrio, E., Cuesta-Albertos, J. A., & Matran, C. (2008). Trimmed comparison of distributions. Journal of the American Statistical Association, 103(482), 697-704.\n\n[4] Manole, T., Balakrishnan, S., & Wasserman, L. (2022). Minimax confidence intervals for the sliced wasserstein distance. Electronic Journal of Statistics, 16(1), 2252-2345.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript proposes a new family of divergences over the space of probability measures, which generalize the univariate Wasserstein distance. These divergences are defined by the Hausdorff metric between the depth regions of the two distributions, integrated over all scales. In this definition, extreme scales are omitted, which makes the divergences robust to outliers. \n\nThe authors verify certain elementary metric properties of their proposed divergences, and they derive several results connecting their divergences to the Wasserstein distance. They also quantify the robustness of their divergences via the traditional notion of breakdown point. The authors devise a clever algorithm for computing their divergence, by noting that the Hausdorff distance between convex depth regions can be efficiently computed. They then conduct a large range of simulations and numerical applications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- As per my previous comment, I encourage the authors to expand their discussion of the possible benefits of their metric as compared to (trimmed) Wasserstein-type metrics. \n - I also encourage the authors to consider using the trimmed Wasserstein metric (or its sliced, max-sliced, etc. versions) in their numerical comparisons. I believe this would provide a more fair comparison in Figure 5. \n- On page 8, the authors comment that \"deriving theoretical finite-sample analysis may appear to be challenging\" --- could the authors elaborate on this comment? What makes it challenging to analyze estimators of the DR metric? \n- On page 9, could the authors please elaborate on their comment that the sliced-Wasserstein Monte Carlo approximation degrades exponentially in the dimension? Reference [5, Theorem 6] would suggest otherwise. \n\nTypographical comments:\n - On page 4, replaced \"bever\" by \"Bever\". \n - On page 6, please put the condition o \\in \\mathbb{N}^* inside the curly braces of the minimum. \n - On page 8, n^{1/4} should be n^{-1/4}, and similarly for the subsequent rates.  \n\n\n[5] Nadjahi, K., Durmus, A., Chizat, L., Kolouri, S., Shahrampour, S., & Simsekli, U. (2020). Statistical and topological properties of sliced probability divergences. Advances in Neural Information Processing Systems, 33, 20802-20812.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hPREmBdSmE", "reviewer": "Reviewer_bLUo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The article is clear and the concepts of depth regions and data depth functions are well presented, as well as the motivation behind the proposed discrepancy. Indeed, for convex data depth functions and elliptical distributions, the depth-trimmed regions can be compared to the density level sets, allowing to better appreciate the definition of the discrepancy. In particular, the divergence, defined to be robust to outliers (a claim supported by a theoretical result), is relevant to both the statistical community and in machine learning tasks. \n- The detailed bibliography on related discrepancies for absolutely continuous measures is significant. The choices of the metrics to be compared with in the experiments (which may or may not handle outliers) is also relevant in my opinion.\n- An approximation method for calculating the discrepancy is proposed and shows good time complexity. The codes to reproduce the experiments in the paper are provided.\n\n\nWeaknesses: \n- The main weakness in my opinion is the lack of theoretical study of statistical rates of convergence when dealing with empirical probability measures (an analysis which must be quite difficult to develop, due to the Hausdorff distance). Nevertheless, an empirical analysis of statistical rates is proposed. Would it be possible to have some insights on this for particular cases (e.g. elliptical distributions and projection depth)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new pseudo-metric for comparing absolutely continuous probability measures, based on the Hausdorff distance between the depth regions of two measures, extending the notion of one-dimensional quantile to the multivariate framework. Depth regions are constructed from convex data depth functions, which are characterized by their support functions, enabling to rewrite the Hausdorff distance in a more convenient way. The proposed discrepancy is designed to enforce robustness, by discarding the tail of the distributions, as a function of a hyperparameter. The usefulness of this discrepancy is evaluated in a clustering task and an automatic evaluation of language generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Adding of a table (or equivalent) to summarize the properties of the discrepancy depending on the type of data depth function (or depth functions themselves), in terms of robustness, computation time of the approximation algorithm, breakpoint for example, could help clarifying the different contributions. In particular, the proposed discrepancy is not a metric in general as $DR_{p,\\varepsilon}(\\mu,\\nu)=0$ does not imply $\\mu=\\nu$. However, I suppose this could be the case for particular convex data depth functions?\n\n- I would have find interesting to analyze the discrepancy (in simple cases, e.g. elliptic distributions) depending on the convex data depth functions, to better understand which outliers are discarded for example.\n\n- I may have missed something, but the depth-trimmed regions discrepancy measure (Def 3.1) is defined for absolutely continuous measures, whereas in the robustness result (Prop 3.7), the discrepancy is considered between empirical measures. Could you clarify this point?\n\nIn my opinion, if manageable, these three points are important to strengthen the paper.", "broader_impact_concerns": "I don't think this paper requires a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "887m8WSZMC", "reviewer": "Reviewer_vUsG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n* The paper investigates a broadly applicable problem, namely computing similarities between distributions. As such, the paper should have a readership as required by the TMLR policy.\n* The experiments highlight the empirical statistical robustness of the proposed similarity measure.\n\n## Weakness\n* As a non-expert, I find that the main issue with the paper is that the paper is rather difficult to follow. I detail some of these issues below.\n* The paper could be much improved by including attempts to visualize the key ideas.\n* Equation 1 is very generic, which implies that the subsequent discussion is quite difficult for readers who are not already familiar with data depths (I am not).\n* It is difficult to determine what I am supposed to learn from the studied machine learning 'applications'. The first (clustering) seems to be very much a toy-solution, which is not representative of how one would solve such a problem. The second (NLP) is beyond my area of expertise. My key concern is that I do not know what I am supposed to learn from these experiments (what is the conclusion?)\n\n\n## Questions\n(these are listed in the order my questions appeared while reading the paper)\n* Page 1 (minor): what is an 'IPM'?\n* Page 3: What should I learn from the second line of Eq 1? Isn't it trivial that a function is a mapping?\n* Page 3: How do I derive from Eq 1 that \"data depth naturally and in a nonparametric way defines a pre-order on $\\mathbb{R}^d$\"? I do not doubt that the statement is true, but does it perhaps require more than Eq. 1?\n* Page 3 and 4: you consider convex depth functions, but I find it very unclear, which practical limitations this places on the associated distribution. I mean, when would I ever want a convex depth function alongside a non-convex probability density function? Eq. 2 seems to suggest that you indeed consider convex PDFs. \n* Page 3 (minor): why are the postulates \"D1 -- D4\" named \"D<number>\"? What is the \"D\" short for?\n* Page 4: \"To the best of our knowledge, the supremum exists...\" Does this statement imply that you do not have proof of the statement and have been unable to locate one in the literature?\n* Page 6: \"This proposition shows that $DR_{2, \\epsilon}$ is able to factor...\" So the proposition does not provide a guarantee? In this case, why is the proposition even interesting?\n* Page 10 (minor): What does \"$\\mathbf{10}_2$\" mean?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper develops a new pseudo-metric between probability distributions based on the notion of 'data depth', which provides an ordered series of subsets of the input space. The empirical behavior of the metric is studied in small case studies, and two demonstration applications are proposed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like for manuscript to be sufficiently clear that my above questions can be answered.", "broader_impact_concerns": "The paper is on the abstract-and-conceptual side of the ML research spectrum, so I have no concerns in this regard.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers and myself found the paper interesting and should be accepted.
Decision: Accept (poster)