Title: Outlier-Robust Wasserstein DRO
Venue: NeurIPS 2023
Accepted: yes
Keywords: distributionally robust optimization

robust statistics

optimal transport

Wasserstein distance
Abstract: Distributionally robust optimization (DRO) is an effective approach for data-driven decision-making in the presence of uncertainty. Geometric uncertainty due to~sampling or localized perturbations of data points is captured by Wasserstein DRO (WDRO), which seeks to learn a model that performs uniformly well over a Wasserstein ball centered around the observed data distribution. However, WDRO fails to account for non-geometric perturbations such as adversarial outliers, which can greatly distort the Wasserstein distance measurement and impede the learned model. We address this gap by proposing a novel outlier-robust WDRO framework for decision-making under both geometric (Wasserstein) perturbations and non-geometric (total variation (TV)) contamination that allows an $\varepsilon$-fraction of data to be arbitrarily corrupted. We design an uncertainty set using a certain robust Wasserstein ball that accounts for both perturbation types and derive minimax optimal excess risk bounds for this procedure that explicitly capture the Wasserstein and TV risks. We prove a strong duality result that enables tractable convex reformulations and efficient computation of our outlier-robust WDRO problem. When the loss function depends only on low-dimensional features of the data, we eliminate certain dimension dependencies from the risk bounds that are unavoidable in the general setting. Finally, we present experiments validating our theory on standard regression and classification tasks.

Outlier-Robust Wasserstein DRO

Abstract

Distributionally robust optimization (DRO) is an effective approach for data-driven decision-making in the presence of uncertainty. Geometric uncertainty due to sampling or localized perturbations of data points is captured by Wasserstein DRO (WDRO), which seeks to learn a model that performs uniformly well over a Wasserstein ball centered around the observed data distribution. However, WDRO fails to account for non-geometric perturbations such as adversarial outliers, which can greatly distort the Wasserstein distance measurement and impede the learned model. We address this gap by proposing a novel outlier-robust WDRO framework for decision-making under both geometric (Wasserstein) perturbations and nongeometric (total variation (TV)) contamination that allows an ε -fraction of data to be arbitrarily corrupted. We design an uncertainty set using a certain robust Wasserstein ball that accounts for both perturbation types and derive minimax optimal excess risk bounds for this procedure that explicitly capture the Wasserstein and TV risks. We prove a strong duality result that enables tractable convex reformulations and efficient computation of our outlier-robust WDRO problem. When the loss function depends only on low-dimensional features of the data, we eliminate certain dimension dependencies from the risk bounds that are unavoidable in the general setting. Finally, we present experiments validating our theory on standard regression and classification tasks.

1 Introduction

The safety and effectiveness of various operations rely on making informed, data-driven decisions in uncertain environments. Distributionally robust optimization (DRO) has emerged as a powerful framework for decision-making in the presence of uncertainties. In particular, Wasserstein DRO (WDRO) captures uncertainties of geometric nature, e.g., due to sampling or localized (adversarial) perturbations of the data points. The WDRO problem is a two-player zero-sum game between a learner (decision-maker), who chooses a decision θ ∈ Θ , and Nature (adversary), who chooses a distribution ν from an ambiguity set defined as the p -Wasserstein ball of a prescribed radius around the observed data distribution ˜ µ . Namely, WDRO is given by 1

whose solution ˆ θ ∈ Θ is chosen to minimize risk over the Wasserstein ball with respect to (w.r.t.) the loss function ℓ . WDROhas received considerable attention in many fields, including machine learning [6, 22, 45, 48, 59], estimation and filtering [36, 37, 46], and chance constraint programming [12, 55].

In many practical scenarios, the observed data may be contaminated by non-geometric perturbations, such as adversarial outliers. Unfortunately, the WDRO problem from (1) is not suited for handling this

issue, as even a small fraction of outliers can greatly distort the W p measurement and impede decisionmaking. In this work, we address this gap by proposing a novel outlier-robust WDRO framework that can learn well-performing decisions even in the presence of outliers. We couple it with a comprehensive theory of excess risk bounds, statistical guarantees, and computationally-tractable reformulations, as well as supporting numerical results.

1.1 Contributions

We consider a scenario where the observed data distribution ˜ µ is subject to both geometric (Wasserstein) perturbations and non-geometric (total variation (TV)) contamination, which allows an ε -fraction of data to be arbitrarily corrupted. Namely, if µ is the true (unknown) data distribution, then the Wasserstein perturbation maps it to some µ ′ with W p ( µ ′ , µ ) ≤ ρ , and the TV contamination step further produces ˜ µ with ∥ ˜ µ -µ ′ ∥ TV ≤ ε (e.g., in the special case of the Huber model [28], ˜ µ = (1 -ε ) µ ′ + εα where α is an arbitrary noise distribution). To enable robust decision-making under this model, we replace the Wasserstein ambiguity set in (1) with a ball w.r.t. the recently proposed outlier-robust Wasserstein distance W ε p [38, 39]. The W ε p distance (see (2) ahead) filters out the ε -fraction of mass from the contaminated distribution that contributed most to the transportation cost, and then measures the W p distance

µ

0

W

p

(

µ,µ

′

)

ρ

ρ

0

µ

′

W

ε

p

(˜

µ,µ

)

ρ

∥

µ

′

-

˜

µ

∥

TV

ε

ρ

0

˜

µ

= (1

-

ε

)

µ

′

+

εα

ρ

post-filtering. To obtain well-performing solutions for our WDRO problem, the W ε p ball is intersected with a set that encodes standard moment assumptions on the uncorrupted data distribution, which are necessary for meaningful outlier-robust estimation guarantees.

We establish minimax optimal excess risk bounds for the decision ˆ θ that solves the proposed outlierrobust WDRO problem. The bounds control the gap E [ ℓ ( ˆ θ, Z )] -E [ ℓ ( θ ⋆ , Z )] , where Z ∼ µ follows the true data distribution and θ ⋆ = argmin θ E [ ℓ ( θ, Z )] is the optimal decision, subject to regularity properties of ℓ ⋆ = ℓ ( θ ⋆ , · ) . In turn, our bounds imply that the learner can make effective decisions using outlier-robust WDRO based on the contaminated observation ˜ µ , so long as ℓ ⋆ has low variational complexity. The bounds capture this complexity using the Lipschitz or Sobolev seminorms of ℓ ⋆ and clarify the distinct effect of each perturbation (Wasserstein versus TV) on the quality of the learned ˆ θ solution. We further establish their minimax optimality when p = 1 , by providing a matching lower bound in the setting when an adversary picks a class of Lipschitz functions over which the learner must perform uniformly well. The excess risk bounds become looser as the data dimension d grows. We show that this degradation is alleviated when the loss function depends on the data only through k -dimensional affine features, by providing risk bounds that adapt to k instead of d .

We then move to study the computational side of the problem, which may initially appear intractable due to non-convexity of the constraint set. We resolve this via a cheap preprocessing step that computes a coarse robust estimate of the mean [34] and replaces the original constraint set (that involves the true mean) with a version centered around the estimate. We adapt our excess risk bounds to this formulation and then prove a strong duality theorem. The dual form is reminiscent of the one for classical WDRO with adaptations reflecting the constraint to the clean distribution family and the partial transportation under W ε p . Under additional convexity conditions on the loss, we further derive an efficiently-computable, finite-dimensional, convex reformulation. The optimization results are also adapted to the setting with low-dimensional features. Using the developed machinery, we present experiments that validate our theory on simple regression/classification tasks and demonstrate the superiority of the proposed approach over classical WRDO, when the observed data is contaminated.

1.2 Related Work

Distributionally robust optimization. The Wasserstein distance has emerged as a powerful tool for modeling uncertainty in the data generating distribution. It was first used to construct an ambiguity set around the empirical distribution in [40]. Recent advancements in convex reformulations and

approximations of the WDRO problem, as discussed in [8, 20, 35], have brought notable computational advantages. Additionally, WDRO is linked to various forms of variation [2, 9, 19, 43] and Lipschitz [7, 11, 44] regularization, which contribute to its success in practice. Robust generalization guarantees can also be provided by WDRO via measure concentration argument or transportation inequalities [18, 30, 31, 51, 53, 54]. Several works have raised concerns regarding the sensitivity of standard DRO to outliers [24, 27, 58]. An attempt to address this was proposed in [56] using a refined risk function based on a family of f -divergences. This formulation aims to prevent DRO from overfitting to potential outliers but is not robust to geometric perturbations. Further, their risk bounds require a moment condition to hold uniformly over Θ , in contrast to our bounds that depend only on θ ⋆ . We are able to address these limitations by setting a WDRO framework based on partial transportation. While partial OT has been previously used in the context of DRO problems, it was introduced to address stochastic programs with side information in [17] rather than to account for outlier robustness. Another closely related line of work is presented in [4, 5], where the ambiguity set is constructed using an f -divergence to mitigate statistical errors and the Prokhorov distance to handle outlier data. The proposed model is both computationally efficient and statistically reliable. However, they have not investigated its minimax optimality or robustness against the Huber contamination model, which we aim to do in this paper. Additionally, a best-case favorable analysis approach has been proposed in [29] to address outlier data. This approach is an alternative to the worst-case distributionally robust method. However, it requires solving a non-convex optimization problem, significantly impacting its scalability, and is not accompanied by any proof of minimax optimality.

Robust statistics. The problem of learning from data under TV ε -corruptions dates back to [28]. Over the years, various robust and sample-efficient estimators, particularly for mean and scale parameters, have been developed in the robust statistics community; see [41] for a comprehensive survey. The theoretical computer science community, on the other hand, has focused on developing computationally efficient estimators that achieve optimal estimation rates in high dimensions [13, 16]. Relatedly, the probably approximate correct (PAC) learning framework has been well-studied in similar models [1, 10]. Recently, [58] developed a unified robust estimation framework based on minimum distance estimation that gives sharp population-limit and promising finite-sample guarantees for mean and covariance estimation, as well linear regression. Their analysis centers on a generalized resilience quantity, which is essential to our work. We are unaware of any results in the settings above which extend to combined TV and W p corruptions. Finally, our analysis relies on the outlier-robust Wasserstein distance from [38, 39], which was shown to yield an optimal minimum distance estimate for robust distribution estimation under W p loss.

2 Preliminaries

Notation. Consider a closed, non-empty set Z ⊆ R d equipped with the Euclidean norm ∥ · ∥ . A continuously differentiable function f : Z → R is called α -smooth if ∥∇ f ( z ) -∇ f ( z ′ ) ∥ ≤ α ∥ z -z ′ ∥ , for all z, z ′ ∈ Z . The perspective function of a lower semi-continuous (l.s.c.) and convex function f is P f ( x, λ ) := λf ( x/λ ) for λ > 0 , with P f ( x, λ ) = lim λ → 0 λf ( x/λ ) when λ = 0 . The convex conjugate of f is f ∗ ( y ) := sup x ∈ R d y ⊤ x -f ( x ) . We denote by χ Z the indicator function of Z , that is, χ Z ( z ) = 0 if z ∈ Z and χ Z ( z ) = ∞ otherwise. The convex conjugate of χ Z , denoted by χ ∗ Z , is termed as the support function of Z .

We use M ( Z ) for the set of signed Radon measures on Z equipped with the TV norm ∥ µ ∥ TV := 1 2 | µ | ( Z ) , and write µ ≤ ν for set-wise inequality. The class of Borel probability measures on Z is denoted by P ( Z ) . We write E µ [ f ( Z )] for expectation of f ( Z ) with Z ∼ µ ; when clear from the context, the random variable is dropped and we write E µ [ f ] . Let Σ µ denote the covariance matrix of µ ∈ P 2 ( Z ) . Define P p ( Z ) := { µ ∈ P ( Z ) : inf z 0 ∈Z E µ [ ∥ Z -z 0 ∥ p ] < ∞} . The push-forward of f through µ ∈ P ( Z ) is f # µ ( · ) := µ ( f -1 ( · )) , and, for A ⊆ P ( Z ) , write f # A := { f # µ : µ ∈ A} . The p th order homogeneous Sobolev (semi)norm of continuously differentiable f : Z → R w.r.t. µ is ∥ f ∥ ˙ H 1 ,p ( µ ) := E µ [ ∥∇ f ∥ p ] 1 /p . The set of integers up to n ∈ N is denote by [ n ] ; we also use the shorthand [ x ] + = max { x, 0 } . We write ≲ , ≳ , ≍ for inequalities/equality up to absolute constants.

Classical and outlier-robust Wasserstein distances. For p ∈ [1 , ∞ ) , the p -Wasserstein distance between µ, ν ∈ P p ( Z ) is W p ( µ, ν ) := inf π ∈ Π( µ,ν ) ( E π [ ∥ X -Y ∥ p ]) 1 /p , where Π( µ, ν ) := { π ∈ P ( Z 2 ) : π ( · × Z ) = µ, π ( Z × · ) = ν } is the set of all their couplings. Some basic properties of

W p are (see, e.g., [42, 52]): (i) W p is a metric on P p ( Z ) ; (ii) the distance is monotone in the order, i.e., W p ≤ W q for p ≤ q ; and (iii) W p metrizes weak convergence plus convergence of p th moments: W p ( µ n , µ ) → 0 if and only if µ n w → µ and ∫ ∥ x ∥ p dµ n ( x ) → ∫ ∥ x ∥ p dµ ( x ) .

To handle corrupted data, we employ the ε -outlier-robust p -Wasserstein distance 2 , defined by

The second equality is a useful consequence of Lemma 4 in [39] (see Appendix A for details, along with an interpretation of W ε p as a partial OT distance).

Robust statistics. Resilience is a standard sufficient condition for population-limit robust statistics bounds [49, 58]. The p -Wasserstein resilience of a measure µ ∈ P ( Z ) is defined by

and that of a family G ⊆ P ( R ) by τ p ( G , ε ) := sup µ ∈G τ p ( µ, ε ) . The relation between W p resilience and robust estimation is formalized in the following proposition.

Proposition 1 (Robust estimation under W p resilience [39]) . Fix 0 ≤ ε ≤ 0 . 49 . For any clean distribution µ ∈ G ⊆ P ( Z ) and corrupted measure ˜ µ ∈ P ( Z ) such that W ε p (˜ µ, µ ) ≤ ρ , the minimum distance estimate ˆ µ = argmin ν ∈G W ε p ( ν, ˜ µ ) satisfies W p (ˆ µ, µ ) ≲ ρ + τ p ( G , 2 ε ) . 3

Throughout, we focus on the bounded covariance class G cov := { µ ∈ P ( Z ) : Σ µ ⪯ I d } .

Proposition 2 ( W p resilience bound for G cov [39]) . Fixing 0 ≤ ε ≤ 0 . 99 and 1 ≤ p ≤ 2 , we have τ p ( G cov , ε ) ≲ √ dε 1 /p -1 / 2 .

3 Outlier-robust WDRO

We perform stochastic optimization with respect to an unknown data distribution µ , given access only to a corrupted version ˜ µ . We allow both localized Wasserstein perturbations, that map µ to some µ ′ with W p ( µ, µ ′ ) ≤ ρ , and TV ε -contamination that takes µ ′ to ˜ µ with ∥ ˜ µ -µ ′ ∥ TV ≤ ε . Equivalently, both perturbations are captured by W ε p (˜ µ, µ ) ≤ ρ . 4 To simplify notation, we henceforth suppress the dependence of the loss function ℓ on the model parameters θ ∈ Θ , writing ℓ for ℓ ( θ, · ) for a specific function and L = { ℓ ( θ, · ) } θ ∈ Θ for the whole class. Our full model is as follows.

Setting A: Fix a p -Wasserstein radius ρ ≥ 0 and TV contamination level ε ∈ [0 , 0 . 49] . Let L ⊆ R Z be a family of real-valued loss functions on Z , such that each ℓ ∈ L is l.s.c. with sup z ∈Z ℓ ( z ) 1+ ∥ z ∥ p < ∞ , and fix a class G ⊆ P p ( Z ) encoding distributional assumptions. We consider the following model:

(i) Nature selects a distribution µ ∈ G , unknown to the learner;

(ii) The learner observes a corrupted measure ˜ µ ∈ P ( Z ) such that W ε p (˜ µ, µ ) ≤ ρ ;

(iii) The learner selects a decision ˆ ℓ ∈ L and suffers excess risk E µ [ ˆ ℓ ] -inf ℓ ∈L E µ [ ℓ ] .

We seek a decision-making procedure for the learner which provides strong excess risk guarantees when ℓ ⋆ = argmin ℓ ∈L E µ [ ℓ ] is appropriately 'simple.' To achieve this, we introduce the ε -outlierrobust p -Wasserstein DRO problem :

3.1 Excess Risk Bounds

We quantify the excess risk of decisions made using OR-WDRO for the two most popular choices of order, p = 1 , 2 . Proofs are provided in Supplement B.

Theorem 1 (OR-WDRO risk bound) . Under Setting A, let ˆ ℓ minimize (OR-WDRO) . Then, writing c = 2(1 -ε ) -1 /p , the excess risk is bounded by

Note that c = O (1) since ε ≤ 0 . 49 . These bounds imply that the learner can make effective decisions when ℓ ⋆ has low variational complexity 5 . In contrast, there are simple regression settings with TV corruption that drive the excess risk of standard WDRO to infinity. Our proof derives both results as a special case of a general bound in terms of the W p regularizer, defined by R µ,p ( ρ ; ℓ ) := sup ν ′ ∈P ( Z ): W p ( ν ′ ,ν ) ≤ ρ E ν ′ [ ℓ ] -E ν [ ℓ ] . Introduced in [18], this quantity appears implicitly throughout the WDRO literature. In particular, for each ℓ ∈ L , we derive the following bound:

whose radius reveals the effect of each perturbation (viz. Wasserstein versus TV) on the quality of the decision. The first bound of the theorem follows by plugging in p = 1 and controlling R µ, 1 via Kantorovich duality. The second bound uses p = 2 and controls R µ, 2 by replacing ℓ with its Taylor expansion about Z ∼ µ . We now instantiate Theorem 1 for the bounded covariance class G cov .

Corollary 1 (Risk bounds for G cov ) . Under the setting of Theorem 1 with G ⊆ G cov , we have

Since G cov encodes second moment constraints, τ 2 ( G cov , ε ) ≍ d is independent of ε . Therefore, the first bound is preferable as ε → 0 if ∥ ℓ ⋆ ∥ ˙ H 1 , 2 ( µ ) ≈ ∥ ℓ ⋆ ∥ Lip , while the second is better when ε = Ω(1) and ∥ ℓ ⋆ ∥ ˙ H 1 , 2 ( µ ) ≪∥ ℓ ⋆ ∥ Lip . 6 Distinct trade-offs are observed under stronger tail bounds like sub-Gaussianity, i.e., for G subG := { µ ∈ P ( Z ) : E µ [ e ( θ ⊤ ( Z -E [ Z ]) 2 ] ≤ 2 , ∀ θ ∈ S d -1 } .

Corollary 2 (Risk bounds for G subG ) . Under the setting of Theorem 1 with G ⊆ G subG , the excess risk E µ [ ˆ ℓ ] -E µ [ ℓ ⋆ ] is bounded up to constants by

Remark 1 (Comparison to MDE under W ε p ) . We note that the excess risk R µ,p ( cρ +2 τ p ( G , 2 ε ); ℓ ⋆ ) from (3) can alternatively be obtained by performing standard p -WDRO with an expanded radius cρ + 2 τ 1 ( G , 2 ε ) around the minimum distance estimate ˆ µ = argmin ν ∈G W ε 1 (˜ µ, ν ) . However, obtaining ˆ µ is an expensive preprocessing step, and we are unaware of any efficient algorithms for such MDE in the finite-sample setting. In Supplement D, we explore recentering WDRO around a tractable estimate obtained from iterative filtering [15], but find the resulting risk to be highly suboptimal. Furthermore, the improvements to our risk bounds under low-dimensional structure, which are derived in Section 4, do not extend to decisions obtained from these alternative procedures.

We now show that Theorem 1 cannot be improved in general. In particular, the first bound is minimax optimal over Lipschitz loss families when µ ∈ G cov .

Proposition 3 (Lower bound) . Fix Z = R d and ε ∈ [0 , 0 . 49] . For any L ≥ 0 , there exists a family L ⊆ Lip L ( R d ) , independent of ε , such that for any decision rule D : P ( Z ) →L there exists a pair ( µ, ˜ µ ) ∈ G cov ×P ( Z ) with W ε 1 (˜ µ, µ ) ≤ ρ satisfying E µ [ D (˜ µ )] -inf ℓ ∈L E µ [ ℓ ] ≳ L ( ρ + √ dε ) .

Each family L encodes a multivariate regression problem. Our proof combines a one-dimensional lower bound of [49] for linear regression with lower bounds of [39] for robust estimation under W 1 .

3.2 Statistical Guarantees

We next formalize a finite-sample model and adapt our excess risk bounds to it.

̸

Setting B: Fix ρ, ε, L , G as in Setting A, and let Z 1 , . . . , Z n be identically and independently distributed (i.i.d.) according to µ ∈ G , with empirical measure ˆ µ n = 1 n ∑ n i =1 δ Z i . Upon observing these clean samples, Nature applies a W p perturbation of size ρ 0 , producing { Z ′ i } n i =1 with empirical measure µ ′ n such that W p (ˆ µ n , µ ′ n ) ≤ ρ 0 . Finally, Nature corrupts up to ⌊ εn ⌋ samples to obtain { ˜ Z i } n i =1 with empirical measure ˜ µ n such that ∥ ˜ µ n -µ ′ ∥ TV = 1 n ∑ n i =1 1 { ˜ Z i = Z i } ≤ ε . Equivalently, the final dataset satisfies W ε p (˜ µ n , ˆ µ n ) ≤ ρ 0 . 7 The learner is now tasked with selecting ˆ ℓ ∈ L given ˜ µ n .

The results from Section 3 apply whenever ρ ≥ ρ 0 + W p ( µ, ˆ µ n ) . In particular, we obtain the following corollary as an immediate consequence of Theorem 1 and Theorem 3.1 of [32].

Corollary 3 (Finite-sample risk bounds) . Under Setting B, fix ˆ ℓ ∈ L minimizing (OR-WDRO) centered at ˜ µ = ˜ µ n with ρ ≥ ρ 0 +100 E [ W p ( µ, ˆ µ n )] . Then the excess risk bounds of Theorem 1 hold with probability at least 0.99. If G ∈ {G cov , G subG } , p = 1 , and d ≥ 3 , or if G = G subG , p = 2 , and d ≥ 5 , then E [ W p ( µ, ˆ µ n )] ≲ √ dn -1 /d .

Remark 2 (Smaller radius) . In the classic WDRO setting with ρ 0 = ε = 0 , the radius ρ can be taken significantly smaller than n -1 /d if L and µ are sufficiently well-behaved. For example, [18] proves that ρ = ˜ O ( n -1 / 2 ) gives meaningful risk bounds when µ satisfies a T 2 transportation inequality. 8 While this high-level condition may be hard to verify in practice, Supplement E shows that this improvement can be lifted to an instance of our outlier-robust WDRO problem.

3.3 Tractable Reformulations and Computation

For computation, we restrict to µ ∈ G cov . Initially, (OR-WDRO) may appear intractable, since G cov is non-convex when viewed as a subset of the cone M + ( Z ) . Moreover, enforcing membership to this class is non-trivial. To remedy these issues, we use a cheap preprocessing step to obtain a robust estimate z 0 ∈ Z of the mean E µ [ Z ] , and we optimize over the modified class G 2 ( σ, z 0 ) := { ν ∈ P ( Z ) : E ν [ ∥ Z -z 0 ∥ 2 ] ≤ σ 2 } , with σ ≳ ∥ z 0 -E µ [ Z ] ∥ + √ d taken so that µ ∈ G 2 ( σ, z 0 ) . Finally, for technical reasons, we switch to the one-sided robust distance W ε p ( µ ∥ ν ) := inf µ ′ ∈P ( R d ): µ ′ ≤ 1 1 -ε µ W p ( µ ′ , ν ) . Altogether, we arrive at the modified DRO problem

which, as stated next, admits risk bounds matching Corollary 1 up to empirical approximation error.

Proposition 4 (Risk bound for modified problem) . Consider Setting B with G ⊆ G cov . Fix z 0 ∈ Z such that ∥ z 0 -E µ [ Z ] ∥ ≤ E = O ( ρ 0 + √ d ) , and suppose that W p (ˆ µ n , µ ) ≤ δ . Take ˆ ℓ minimizing (4) with ρ = ( ρ 0 + δ )(1 -ε ) -1 /p + τ p ( G cov , ε ) and σ = √ d + E . We then have

Parameters ρ, σ are taken so that µ ∈ G 2 ( σ, z 0 ) and W ε p (˜ µ n ∥ µ ) ≤ ρ . Noting this, the proof mirrors that of Theorem 1, using a W p resilience bound for G 2 ( σ, z 0 ) . To ensure W p (ˆ µ n , µ ) ≤ δ with decent probability, one should take δ to be an upper bound on sup ν ∈G E [ W p (ˆ ν n , ν )] . When p = 2 , this quantity is only finite if Z is bounded or if G encodes stronger tail bounds than G cov (see, e.g., [32]).

For efficient computation, we must specify a robust mean estimation algorithm to obtain z 0 and a procedure for solving (4). The former is achieved by taking a coordinate-wise trimmed mean.

Proposition 5 (Coarse robust mean estimation) . Consider Setting B with G ⊆ G cov and ε ≤ 1 / 3 . For n = Ω(log( d )) , there is a trimmed mean procedure, which applied coordinate-wise to { ˜ Z i } n i =1 , returns z 0 ∈ R d with ∥ z 0 -E µ [ Z ] ∥ ≲ √ d + ρ 0 with probability at least 0 . 99 , in time ˜ O ( d ) .

More sophisticated methods, e.g., iterative filtering [15], achieve dimension-free estimation guarantees at the cost of additional sample and computational complexity. We will return to these techniques in Section 4, but overlook them for now since they do not impact worst-case excess risk bounds.

We next show that that the inner maximization problem of (4) can be simplified to a minimization problem involving only two scalars provided the following assumption holds.

Assumption 1 (Slater condition I) . Given the distribution ˜ µ n and the fixed point z 0 , there exists ν 0 ∈ P ( Z ) such that W ε p (˜ µ n ∥ ν 0 ) < ρ and E ν 0 [ ∥ Z -z 0 ∥ 2 ] < σ 2 . Additionally, we require ρ > 0 .

Notice that Assumption 1 indeed holds for ν 0 = µ as applied in Proposition 4.

Proposition 6 (Strong duality) . Under Assumption 1, for any ℓ ∈ L and z 0 ∈ R d , we have

The minimization problem over ( λ 1 , λ 2 , α ) is an instance of stochastic convex optimization, where the expectation of the implicit function ℓ is taken w.r.t. the contaminated empirical measure ˜ µ n . In contrast, the dual reformulation for classical WDRO only involves λ 2 and takes the expectation of the implicit function ℓ ( z ; λ 2 ) := sup ξ ∈Z ℓ ( ξ ) -λ 2 ∥ ξ -z ∥ p w.r.t. ˜ µ n . The additional λ 1 variable above is introduced to account for the clean family G 2 ( σ, z 0 ) , and the use of partial transportation under W ε p results in the introduction of the operator [ · ] + and the decision variable α .

Remark 3 (Connection to conditional value at risk (CVaR)) . The CVaR of a Borel measurable loss function ℓ acting on a random vector Z ∼ µ ∈ P ( Z ) with risk level ε ∈ (0 , 1) is defined as

CVaR is also known as expected shortfall and is equivalent to the conditional expectation of ℓ ( Z ) , given that it is above an ε threshold. This concept is often used in finance to evaluate the market risk of a portfolio. With this definition, the result of Proposition 6 can be written as

When ε → 0 and σ → ∞ , whence CVaR reduces to expected value and the constrained class G 2 ( σ, z 0 ) expands to P ( Z ) , the dual formulation above reduces to that of classical WDRO [8, 21].

Evaluating ℓ requires solving a maximization problem, which could be in itself challenging. To overcome this, we impose additional convexity assumptions, which are standard for WDRO [35, 43].

Assumption 2 (Convexity condition) . The loss ℓ is a pointwise maximum of finitely many concave functions, i.e., ℓ ( ξ ) = max j ∈ [ J ] ℓ j ( ξ ) , for some J ∈ N , where ℓ j is real-valued, l.s.c., and concave 9 . The set Z is closed and convex. The atoms of ˜ µ n are in the relative interior of Z .

Theorem 2 (Convex reformulation) . Under Assumption 1, for any ℓ ∈ L satisfying Assumption 2 and z 0 ∈ R d , we have

where P h is the perspective function (i.e., P h ( ζ, λ ) = λh ( ζ/λ ) ) of

The minimization problem in Theorem 2 is a finite-dimensional convex program. In Section 5, we use this result in conjunction with Proposition 5 to efficiently perform outlier-robust WDRO.

We conclude this section by characterizing the worst-case distribution, i.e., the optimal adversarial strategy, for our outlier-robust WDRO problem. To that end, we need the primal formulation below.

Theorem 3 (Worst-case distribution) . Under Assumption 1, for any ℓ ∈ L satisfying Assumption 2 and z 0 ∈ R d , we have

The discrete distribution ν ⋆ = ∑ ( i,j ) ∈Q q ⋆ ij δ ξ ⋆ ij /q ⋆ ij achieves the worst-case expectation on the lefthand side, where ( q ⋆ ij , ξ ⋆ ij ) ( i,j ) ∈ [ n ] × [ J ] are optimizers of the maximization problem on the right and Q := { ( i, j ) ∈ [ n ] × [ J ] : q ⋆ ij > 0 } .

The maximization problem from Theorem 3 is the conjugate dual of the minimization in Theorem 2. Subsequently, we propose a systematic approach for constructing a discrete distribution based on a solution derived from the maximization problem that achieves the worst-case expected loss.

Remark 4 (Comparison to WDRO worst-case distribution) . Recall that our robust WDRO approach reduces to the classic WDRO approach as ε = 0 and σ → ∞ . Consequently, this implies that the constraints ∑ j ∈ [ J ] q ij ≤ 1 / ( n (1 -ε )) and ∑ ( i,j ) ∈ [ n ] × [ J ] P ∥·∥ 2 ( ξ ij -q ij z 0 , q ij ) ≤ σ 2 can be dropped under this specific choice of ε and σ . As a result, our construction simplifies to the approach presented in [35, Theorem 4.4] for WDRO problems.

Remark 5 (Parameter tuning) . In practice, ε , ρ 0 , and the relevant tail bound may be unknown. Thus, in Appendix F, we consider learning under Setting B with G = G cov ( σ ) for potentially unknown ε , σ , and ρ 0 . First, we observe that knowledge of upper bounds on these parameters is sufficient to attain risk bounds scaling in terms of said upper bounds. This approach avoids meticulous parameter tuning but may result in suboptimal risk. To efficiently match our risk bounds with known parameters, we show that it is necessary and sufficient to know ρ 0 and at least one of ε or σ (up to constant factors).

4 Low-Dimensional Features

While Proposition 3 shows that the excess risk bounds from Theorem 1 cannot be improved in general, finer guarantees can be derived when the optimal loss function depends only on k -dimensional affine features of the data. Defining G ( k ) as the union of the projections { U # µ : µ ∈ G} over U ∈ R k × d with UU ⊤ = I k 10 , we improve the excess risk bound of Theorem 1 for this setting.

Theorem 4 (Excess risk bound) . Under Setting A, let ˆ ℓ minimize (OR-WDRO) , and assume that ℓ ⋆ = ℓ ◦ A for an affine map A : R d → R k and some ℓ : R k → R . Writing c =2(1 -ε ) -1 /p , we have

This dependence on G ( k ) rather than G = G ( d ) is a substantial improvement when k ≪ d .

Corollary 4 (Risk bounds for G cov ) . Under the setting of Theorem 4 with G ⊆ G cov , we have

We again have a matching lower bound for the Lipschitz setting, this time using k -variate regression. Proposition 7 (Lower bound) . Fix Z = R d and ε ∈ [0 , 0 . 49] . For any L ≥ 0 , there exists a family L ⊆ Lip L ( R d ) , independent of ε , such that each ℓ ∈ L decomposes as ℓ = ℓ ◦ A for A ∈ R k × d and ℓ : R k → R , and such that for any decision rule D : P ( Z ) →L there exists a pair ( µ, ˜ µ ) ∈ G cov ×P ( Z ) with W ε 1 (˜ µ, µ ) ≤ ρ satisfying E µ [ D (˜ µ )] -inf ℓ ∈L E µ [ ℓ ] ≳ L ( ρ + √ kε ) .

For computation, we turn to a slightly modified n -sample contamination model. Our analysis for the low-dimensional case only supports additive TV corruptions (sometimes called Huber contamination).

Setting B ′ : Fix ρ, ε, L , G as in Setting A, and fix m = ⌈ (1 -ε ) n ⌉ for some n ∈ N . Let Z 1 , . . . , Z m be drawn i.i.d. from µ ∈ G , with empirical measure ˆ µ m = 1 m ∑ m i =1 δ Z i . Upon observing these clean samples, Nature applies a W p perturbation of size ρ 0 , producing { Z ′ i } i m =1 with empirical measure µ ′ m such that W p (ˆ µ m , µ ′ m ) ≤ ρ 0 . Finally, Nature adds ⌊ εn ⌋ samples to obtain { ˜ Z i } n i =1 with empirical measure ˜ µ n such that µ ′ m ≤ 1 1 -ε ˜ µ n . Equivalently, the final dataset satisfies W ε p (˜ µ n ∥ ˆ µ m ) ≤ ρ 0 .

As before, we modify (OR-WDRO) using a centered alternative to G cov . Defining G cov ( σ, z 0 ) := { µ ∈ P ( Z ) : E µ [( Z -z 0 )( Z -z 0 ) ⊤ ] ⪯ σ 2 I d } , we consider the outlier-robust WDRO problem

To start, we provide a corresponding risk bound which matches Corollary 4 when k = O (1) .

Proposition 8 (Risk bound for modified problem) . Consider Setting B ′ with G ⊆ G cov , and assume ℓ ⋆ = ℓ ◦ A for affine A : R d → R k and ℓ : R k → R . Fix z 0 ∈ Z such that ∥ z 0 -E µ [ Z ] ∥ ≤ E = O ( ρ 0 +1) , and assume W p (ˆ µ m , µ ) ≤ δ . If ˆ ℓ minimizes (7) with ρ = ρ 0 + δ and σ = 1 + E , then

Here, the stronger requirement for the robust mean estimate, the restriction to additive contamination, and the need to optimize over the centered G cov class rather than G 2 all stem from the fact that the resilience term τ p (( G cov ) k , ε ) scales with √ k rather than √ d . Fortunately, efficient computation is still possible. First, we employ iterative filtering [15] for dimension-free robust mean estimation.

Proposition 9 (Refined robust mean estimation) . Consider Setting B or B ′ with G = G cov and ε ≤ 1 / 12 . For n = ˜ Ω( d ) , there exists an iterative filtering algorithm which takes ˜ µ n as input, runs in time ˜ O ( nd 2 ) , and outputs z 0 ∈ R d such that ∥ z 0 -E µ [ Z ] ∥ ≲ ρ 0 +1 with probability at least 0.99.

The analysis requires care when p = 1 , since W 1 perturbations can arbitrarily increase the initial covariance bound. Fortunately, this increase can be controlled by trimming out a few samples.

Next, we show that computing the inner worst-case expectation in (7) can be simplified into a minimization problem involving only a scalar and a positive semidefinite matrix provided the following assumption holds (which is indeed the case in the setting of Proposition 8).

Assumption 3 (Slater condition II) . Given the distribution ˜ µ n and fixed point z 0 , there exists ν 0 ∈ P ( Z ) such that W ε p (˜ µ n ∥ ν 0 ) < ρ and E ν 0 [( Z -z 0 )( Z -z 0 ) ⊤ ] ≺ σ 2 I d . Further, we require ρ> 0 .

Proposition 10 (Strong duality) . Under Assumption 3, for any ℓ ∈ L and z 0 ∈ R d , we have

The minimization problem over the variables (Λ 1 , λ 2 , α ) belongs to the class of stochastic convex optimization problems. As before, we show that under the convexity condition from Assumption 2 we obtain a tractable reformulation that does not involve an extra optimization problem for evaluating ℓ .

where P h is the perspective function of h defined in (6) .

5 Experiments

Lastly, we implement our tractable reformulations and validate their excess risk bounds. Fixing Z = X × Y = R d -1 × R , we focus on linear regression with the mean absolute deviation loss, i.e., L = { ℓ θ ( x, y ) = | θ ⊤ x -y | : θ ∈ R d } . The experiments below were run in 80 minutes on an M1 MacBook Air with 16GB RAM. See Supplement G for full details and additional experiments treating classification and multivariate regression. Code is available at https://github.com/sbnietert/ outlier-robust-WDRO .

Fix Z = R d for d ≥ 2 , ρ = ε = 0 . 1 , and θ ⋆ ∈ S d -2 . Taking X ∼ N (0 , I d -1 ) , we fix clean data ( X,θ ⊤ ⋆ X ) ∼ µ and corrupted data ( ˜ X, ˜ Y ) ∼ ˜ µ such that ( ˜ X, ˜ Y ) = ( X,θ ⊤ ⋆ X + ρ ) with probability 1 -ε and ( ˜ X, ˜ Y )=( CX, -C 2 θ ⊤ ⋆ X ) with probability ε , so that W ε 0 p (˜ µ ∥ µ ) ≤ ρ . In Figure 2 (top), we fix d = 10 , C = 8 and compare the excess risk E µ [ ℓ ˆ θ ] -E µ [ ℓ θ ⋆ ] of standard WDRO and outlierrobust WDRO with G = G 2 , as described by Proposition 4 and implemented via Theorem 2. The results are averaged over T = 20 runs for sample size n ∈{ 10 , 20 , 50 , 75 , 100 } . We run outlier-robust WDROwithcorruption fraction ˆ ε ∈{ 0 , ε, 2 ε } , achieving low excess risk when ˆ ε ≥ ε as predicted. In Figure 2 (bottom), to highlight the Section 4 improve-

10

20

30

40

50

60

70

80

90

100

# samples

0

0.5

1

1.5

2

2.5

3

3.5

excess risk (mean absolute deviation)

Excess Risk for Varied Sample Size and Method

standard WDRO

5

10

15

20

25

30

35

40

0

0.2

0.4

0.6

0.8

1

1.2

excess risk (mean absolute deviation)

Excess Risk for Varied Dimension and Method

OR-WDRO w/ G = G

2

OR-WDRO w/ G = G

cov

dimension

ments due to low-dimensional structure, we fix n = 20 , C = 100 and compare the excess risk of outlier-robust WDRO with G = G 2 to that with G = G cov , as described by Proposition 8 and implemented via Theorem 5. We average over T = 10 runs and present results for dimension d ∈{ 5 , 10 , 25 , 40 } . Confidence bands in both plots depict the top and bottom 10% quantiles among 100 bootstrapped means from the T runs. Implementations were performed in MATLAB using the YALMIP toolbox [33] and the Gurobi and SeDuMi solvers [23, 50].

6 Concluding Remarks

In this work, we have introduced a novel framework for outlier-robust WDRO that allows for both geometric and non-geometric perturbations of the data distribution, as captured by W p and TV, respectively. We provided minimax optimal excess risk bounds and strong duality results, with the latter enabling efficient computation via convex reformulations. There are numerous directions for future work, including refined statistical guarantees for ρ ≪ ρ 0 + n -1 /d and convex reformulations for distribution families beyond G cov . Overall, our approach enables principled, data-driven decisionmaking in realistic scenarios where observations may be subject to adversarial contamination.

References

D. Angluin and P. Laird. Learning from noisy examples. Machine Learning , 2:343-370, 1988.

D. Bartl, S. Drapeau, J. Obloj, and J. Wiesel. Robust uncertainty sensitivity analysis. arXiv preprint arXiv:2006.12022 , 2020.

A. Ben-Tal and M. Teboulle. An old-new concept of convex risk measures: The optimized certainty equivalent. Mathematical Finance , 17(3):449-476, 2007.

A. Bennouna, R. Lucas, and B. Van Parys. Certified robust neural networks: Generalization and corruption resistance. In International Conference on Machine Learning , 2023.

A. Bennouna and B. Van Parys. Holistic robust data-driven decisions. arXiv preprint arXiv:2207.09560 , 2022.

J. Blanchet, P. W. Glynn, J. Yan, and Z. Zhou. Multivariate distributionally robust convex regression under absolute error loss. In Advances in Neural Information Processing Systems , 2019.

J. Blanchet, Y. Kang, and K. Murthy. Robust Wasserstein profile inference and applications to machine learning. Journal of Applied Probability , 56(3):830-857, 2019.

J. Blanchet and K. Murthy. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research , 44(2):565-600, 2019.

J. Blanchet, K. Murthy, and N. Si. Confidence regions in Wasserstein distributionally robust estimation. Biometrika , 109(2):295-315, 2022.

N. H. Bshouty, N. Eiron, and E. Kushilevitz. PAC learning with nasty noise. Theoretical Computer Science , 288(2):255-275, 2002.

R. Chen and I. C. Paschalidis. A robust learning approach for regression models based on distributionally robust optimization. Journal of Machine Learning Research , 19(1):517-564, 2018.

Z. Chen, D. Kuhn, and W. Wiesemann. Data-driven chance constrained programs over Wasserstein balls. Operations Research (Forthcoming) , 2022.

Y. Cheng, I. Diakonikolas, and R. Ge. High-dimensional robust mean estimation in nearly-linear time. In ACM-SIAM Symposium on Discrete Algorithms , 2019.

L. Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

I. Diakonikolas, G. Kamath, D. M. Kane, J. Li, A. Moitra, and A. Stewart. Being robust (in high dimensions) can be practical. In International Conference on Machine Learning , 2017.

I. Diakonikolas and D. Kane. Algorithm High-Dimensional Robust Statistics . Cambridge University Press, 2022.

A. Esteban-Pérez and J. M. Morales. Distributionally robust stochastic programs with side information based on trimmings. Mathematical Programming , 195(1-2):1069-1105, 2022.

R. Gao. Finite-sample guarantees for Wasserstein distributionally robust optimization: Breaking the curse of dimensionality. Operations Research (Forthcoming) , 2022.

R. Gao, X. Chen, and A. J. Kleywegt. Wasserstein distributionally robust optimization and variation regularization. Operations Research (Forthcoming) , 2022.

R. Gao and A. Kleywegt. Distributionally robust stochastic optimization with Wasserstein distance. Mathematics of Operations Research , 48(2):603-655, 2023.

R. Gao and A. Kleywegt. Distributionally robust stochastic optimization with wasserstein distance. Mathematics of Operations Research , 48(2):603-655, 2023.

R. Gao, L. Xie, Y. Xie, and H. Xu. Robust hypothesis testing using Wasserstein uncertainty sets. In Advances in Neural Information Processing Systems , 2018.

Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023. URL https://www. gurobi.com .

T. Hashimoto, M. Srivastava, H. Namkoong, and P. Liang. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning , 2018.

J.-B. Hiriart-Urruty and C. Lemaréchal. Fundamentals of Convex Analysis . Springer, 2004.

S. Hopkins, J. Li, and F. Zhang. Robust and heavy-tailed mean estimation made simple, via regret minimization. In Advances in Neural Information Processing Systems , 2020.

W. Hu, G. Niu, I. Sato, and M. Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning , 2018.

P. J. Huber. Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics , 35(1):73-101, 1964.

N. Jiang and W. Xie. Distributionally favorable optimization: A framework for data-driven decision-making with endogenous outliers. Optimization Online , 2023.

Y. Kwon, W. Kim, J.-H. Won, and M. C. Paik. Principled learning method for Wasserstein distributionally robust optimization with local perturbations. In International Conference on Machine Learning , 2020.

J. Lee and M. Raginsky. Minimax statistical learning with Wasserstein distances. In Advances in Neural Information Processing Systems , 2018.

J. Lei. Convergence and concentration of empirical measures under Wasserstein distance in unbounded functional spaces. Bernoulli , 26(1):767 - 798, 2020.

J. Lofberg. YALMIP: A toolbox for modeling and optimization in MATLAB. In IEEE international conference on robotics and automation , pages 284-289. IEEE, 2004.

G. Lugosi and S. Mendelson. Robust multivariate mean estimation: the optimality of trimmed mean. The Annals of Statistics , 49(1):393-410, 2021.

P. Mohajerin Esfahani and D. Kuhn. Data-driven distributionally robust optimization using the Wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Programming , 171(1-2):115-166, 2018.

V. A. Nguyen, D. Kuhn, and P. Mohajerin Esfahani. Distributionally robust inverse covariance estimation: The Wasserstein shrinkage estimator. Operations Research , 70(1):490-515, 2022.

V. A. Nguyen, S. Shafieezadeh-Abadeh, D. Kuhn, and P. Mohajerin Esfahani. Bridging Bayesian and minimax mean square error estimation via Wasserstein distributionally robust optimization. Mathematics of Operations Research , 48(1):1-37, 2023.

S. Nietert, R. Cummings, and Z. Goldfeld. Outlier-robust optimal transport: duality, structure, and statistical analysis. In International Conference on Artificial Intelligence and Statistics , 2022.

S. Nietert, R. Cummings, and Z. Goldfeld. Robust estimation under the Wasserstein distance. arXiv preprint arXiv:2302.01237 , 2023.

G. Pflug and D. Wozabal. Ambiguity in portfolio selection. Quantitative Finance , 7(4):435-442, 2007.

E. M. Ronchetti and P. J. Huber. Robust Statistics . John Wiley & Sons Hoboken, 2009.

F. Santambrogio. Optimal Transport for Applied Mathematicians . Springer, 2015.

S. Shafieezadeh-Abadeh, L. Aolaritei, F. Dörfler, and D. Kuhn. New perspectives on regularization and computation in optimal transport-based distributionally robust optimization. arXiv preprint arXiv:2303.03900 , 2023.

S. Shafieezadeh-Abadeh, D. Kuhn, and P. Mohajerin Esfahani. Regularization via mass transportation. Journal of Machine Learning Research , 20(103):1-68, 2019.

S. Shafieezadeh-Abadeh, P. Mohajerin Esfahani, and D. Kuhn. Distributionally robust logistic regression. In Advances in Neural Information Processing Systems , 2015.

S. Shafieezadeh-Abadeh, V. A. Nguyen, D. Kuhn, and P. Mohajerin Esfahani. Wasserstein distributionally robust Kalman filtering. In Advances in Neural Information Processing Systems , 2018.

A. Shapiro. On duality theory of conic linear problems. In M. Á. Goberna and M. A. López, editors, Semi-Infinite Programming , pages 135-165. Kluwer Academic Publishers, 2001.

A. Sinha, H. Namkoong, and J. Duchi. Certifying some distributional robustness with principled adversarial training. In International Conference on Learning Representations , 2018.

J. Steinhardt, M. Charikar, and G. Valiant. Resilience: A criterion for learning in the presence of arbitrary outliers. In Innovations in Theoretical Computer Science Conference , 2018.

J. F. Sturm. Using SeDuMi 1.02, A MATLAB toolbox for optimization over symmetric cones. Optimization Methods and Software , 11(1-4):625-653, 1999.

Z. Tu, J. Zhang, and D. Tao. Theoretical analysis of adversarial learning: A minimax approach. In Advances in Neural Information Processing Systems , 2019.

C. Villani. Topics in Optimal Transportation . American Mathematical Society, 2003.

R. Volpi, H. Namkoong, O. Sener, J. Duchi, V. Murino, and S. Savarese. Generalizing to unseen domains via adversarial data augmentation. In Advances in Neural Information Processing Systems , pages 5339-5349, 2018.

Y. Wang, X. Ma, J. Bailey, J. Yi, B. Zhou, and Q. Gu. On the convergence and robustness of adversarial training. In International Conference on Machine Learning , 2019.

W. Xie. On distributionally robust chance constrained programs with Wasserstein distance. Mathematical Programming , 186(1):115-155, 2021.

R. Zhai, C. Dan, Z. Kolter, and P. Ravikumar. DORO: Distributional and outlier robust optimization. In International Conference on Machine Learning , 2021.

J. Zhen, D. Kuhn, and W. Wiesemann. A unified theory of robust and distributionally robust optimization via the primal-worst-equals-dual-best principle. Operations Research (Forthcoming) , 2023.

B. Zhu, J. Jiao, and J. Steinhardt. Generalized resilience and robust statistics. The Annals of Statistics , 50(4):2256 - 2283, 2022.

S. Zhu, L. Xie, M. Zhang, R. Gao, and Y. Xie. Distributionally robust weighted k-nearest neighbors. In Advances in Neural Information Processing Systems , 2022.

A Preliminary Results on Robust OT and Wasserstein DRO

We first recall properties of the robust Wasserstein distance W ε p which will be used throughout the supplement. To start, we show that our definition coincides with another based on partial OT, considered in [39]. In what follows, we fix p ≥ 1 , write c P ( Z ) := { cµ : µ ∈ P ( Z ) } , and, for µ, ν ∈ c P ( Z ) , we define Π( µ, ν ) := c Π( µ/c, ν/c ) and W p ( µ, ν ) p := c W p ( µ/c, ν/c ) .

Lemma 1 ( W ε p as partial OT) . Fix ε ∈ [0 , 1] . For any µ, ν ∈ P ( Z ) , we have

Proof. We write W ε p ( µ, ν ) for the rightmost expression; this is definition of W ε p considered in [39]. We first show that W ε p ( µ, ν ) ≤ W ε p ( µ, ν ) . Fix any µ ′ feasible for the W ε p problem. Then, by the approximate triangle inequality for W ε p (Proposition 3 of [39]), we have

Indeed, writing c := ( µ ∧ µ ′ )( Z ) ≥ 1 -ε , the last inequality uses that W ε p ( µ, µ ′ ) ≤ W p ( 1 -ε c µ ∧ µ ′ , 1 -ε c µ ∧ µ ′ ) = 0 . Infimizing over feasible µ ′ gives that W ε p ( µ, ν ) ≤ W ε p ( µ, ν ) .

For the other direction, take any µ ′ , ν ′ feasible for the W ε p problem. Let π ∈ Π( µ ′ , ν ′ ) be any optimal coupling for the W p ( µ ′ , ν ′ ) problem, and write µ ′′ = µ ′ +( ν -ν ′ ) ∈ P ( Z ) . Defining the coupling π ′ = π +(Id , Id) # ( ν -ν ′ ) ∈ Π( µ ′′ , ν ) , we compute

By construction, ∥ µ ′′ -µ ∥ TV ≤ ε , and so W ε p ( µ, ν ) ≤ W p ( µ ′ , ν ′ ) . Infimizing over feasible µ ′ , ν ′ gives that W ε p ( µ, ν ) ≤ W ε p ( µ, ν ) .

We thus inherit several results for W p given in [39].

Lemma 2 (Approximate triangle inequality [39]) . If µ, ν, κ ∈ P ( Z ) and ε 1 , ε 2 ∈ [0 , 1] , then

Lemma 3 ( W ε p modulus of continuity, [39], Lemma 3) . For any G ⊆ P ( Z ) , we have

Lemma 4 (One-sided vs. two-sided W ε p ) . For µ, ν ∈ P ( Z ) , we have

Proof. Fix any µ ′ , ν ′ ∈ (1 -ε ) P ( Z ) with µ ′ ≤ µ and ν ′ ≤ ν . By design, we have

′ ′

Infimizing over µ and ν and applying Lemma 1 gives the lemma.

Next, we specify explicit constants for the W p resilience of G cov . Our analysis goes through the related notion of mean resilience [49], defined by τ ( µ, ε ) = sup µ ′ ∈P ( Z ): µ ′ ≤ 1 1 -ε µ ∥ E µ ′ [ Z ] -E µ [ Z ] ∥ . We say that Z ∼ µ ∈ P ( Z ) is ( τ 0 , ε ) -resilient in mean or under W p if µ ∈ τ ( τ 0 , ε ) or µ ∈ τ p ( τ 0 , ε ) . Lemma 5 ( W p resilience for G 2 and G cov ) . Fix ε ∈ [0 , 1) and σ ≥ 0 . For 1 ≤ p ≤ 2 , we have τ p ( G 2 ( σ ) , ε ) ≤ 4 σε 1 /p -1 / 2 (1 -ε ) -1 /p . Moreover, we have τ p ( G cov ( σ ) , ε ) ≤ τ p ( G 2 ( √ dσ ) , ε ) .

Proof. Fix any Z ∼ µ ∈ G 2 ( σ, z 0 ) . By definition, we have E [( ∥ Z -z 0 ∥ p ) 2 /p ] = E [ ∥ Z -z 0 ∥ 2 ] ≤ σ 2 = ( σ p ) 2 /p . Thus, standard bounds (e.g., Lemma E.2 of [58]) give that ∥ Z -z 0 ∥ p is ( σ p ε 1 -p/ 2 (1 -ε ) -1 , ε ) -resilient in mean. By Lemma 7 of [39], we thus have that Z is (2 σε 1 /p -1 / 2 (1 -ε ) -1 /p + 2 ε 1 /p σ, ε ) -resilient under W p . This gives the first result. For the second, we observe that for Z ∼ µ ∈ G cov ( σ ) , we have E [ ∥ Z -E [ Z ] ∥ 2 ] = tr(Σ µ ) ≤ √ dσ .

Lastly, we turn to the W p regularizer.

Lemma 6 (Controlling W p regularizer, [18], Lemmas 1 and 2) . For any ℓ ∈ L , we have R ν, 1 ( ρ ; ℓ ) ≤ ρ ∥ ℓ ∥ Lip , with equality if ℓ is convex. For α -smooth ℓ , we have |R ν, 2 ( ρ ; ℓ ) -ρ ∥ ℓ ∥ ˙ H 1 , 2 ( ν ) | ≤ 1 2 αρ 2 .

The factor of 1/2 under smoothness was not present in Lemma 2 of [18], since the proof only used that | ℓ ( z ) -ℓ ( z 0 ) -∇ ( z 0 ) ⊤ ( z -z 0 ) | ≤ α ∥ z -z 0 ∥ 2 , instead of the tight upper bound of α 2 ∥ z -z 0 ∥ 2 .

This quantity naturally bounds the excess risk of standard WDRO.

Lemma 7 (WDRO excess risk bound) . Under Setting A with ε = 0 , the standard WDRO estimate ˆ ℓ = argmin ℓ ∈L sup ν ∈P ( Z ): W p (˜ µ,ν ) ≤ ρ E ν [ ℓ ] satisfies E µ [ ˆ ℓ ] -E µ [ ℓ ⋆ ] ≤ R µ,p (2 ρ ; ℓ ⋆ ) .

Proof. We bound

as desired.

Note that this bound does not incorporate the distributional assumptions encoded by G .

B Proofs for Section 3

B.1 Proof of Theorem 1

We compute

Combining this bound with Lemma 6 gives the theorem.

B.2 Proof of Corollary 1

The corollary follows as an immediate consequence of Theorem 1 and the resilience bounds of Proposition 2.

B.3

Proof of Corollary 2

The corollary follows as an immediate consequence of Theorem 1 and the resilience bound τ p ( G subG , ε ) ≲ √ d + p +log 1 ε ε 1 /p established in [39, Theorem 2].

B.4 Proof of Proposition 3

For ease of presentation, suppose d = 2 m is even. Consider R d as R m × R m , fix w ∈ R m with ∥ w ∥ = ρ , and let L consist of the following loss functions:

Fixing corrupted measure ˜ µ = δ 0 , we consider the following candidates for the clean measure µ :

where Id : x ↦→ x is the identity map. By design, W ε 1 (˜ µ ∥ µ + , 0 ) , W ε 1 (˜ µ ∥ µ -, 0 ) , W ε 1 (˜ µ ∥ µ + , 1 ) , and W ε 1 (˜ µ ∥ µ -, 1 ) are all at most ρ and µ + , 0 , µ -, 0 , µ + , 1 , µ -, 1 ∈ G cov . Moreover,

Thus, for any ˆ ℓ = D (˜ µ ) ∈ L , there exists µ ∈ { µ + , 0 , µ -, 0 , µ + , 1 , µ -, 1 } such that E µ [ ˆ ℓ ] -inf E µ [ ℓ ] = E µ [ ˆ ℓ ] ≳ L max { ρ, √ dε } ≍ L ( ρ + √ dε ) .

B.5 Proof of Proposition 4

Since µ ∈ G cov , we have

Consequently, we have µ ∈ G 2 ( σ, z 0 ) . Next, we bound

Writing G ′ = G 2 ( σ, z 0 ) and mirroring the proof of Theorem 1, we have for each ℓ ∈ L that

When p = 1 , we bound the regularizer radius by

using Lemma 5. Similarly, when p = 2 , we bound the radius by

Taking ℓ = ℓ ⋆ and applying Lemma 7 gives the proposition.

B.6 Proof of Proposition 5

To start, we fix d = 1 . Given 0 ≤ γ < 1 / 2 and ν ∈ P ( R ) with cumulative distribution function (CDF) F ν , define the γ -trimming T γ ( ν ) ∈ P ( R ) as the law of F -1 ν ( U ) , where U ∼ Unif([ γ, 1 -γ ]) , and let m γ ( ν ) := E T γ ( ν ) [ Z ] denote the γ -trimmed mean. If ν = 1 | A | ∑ a ∈ A δ a is uniform over a finite set A = { a 1 < a 2 < · · · < a n } and γn is an integer, we have m γ ( ν ) = 1 (1 -2 γ ) n ∑ (1 -γ ) n i = γn +1 a i .

Our robust mean estimate when d = 1 is z 0 = m γ (˜ µ n ) with γ = 1 / 3 . The smaller choice of γ = ε gives tighter guarantees at the cost of increased sample complexity; we keep the larger choice since we only require a coarse estimate.

Lemma 8. Consider Setting B with d = 1 , G = G cov , ρ 0 = 0 , ε ≤ 1 / 3 . Fix sample size n = Ω(log(1 /δ )) , for 0 < δ < 1 / 2 . Then, ∥ m γ (˜ µ n ) -E µ [ Z ] ∥ ≲ 1 with probability at least 1 -δ .

Proof. This follows by Proposition 1.18 of [16] applied to the distribution µ with corruption fraction γ , ε ′ = 4 γ/ 3 < 1 / 2 , and resilience bound τ ( G cov , 2 ε ′ ) ≲ √ γ ≲ 1 .

Now, since we are free to permute the order of the TV and W p corruptions (see Lemma 1), there exist { W i } n i =1 ⊆ R with empirical measure ν n ∈ P ( R ) such that ∥ ν n -ˆ µ n ∥ TV ≤ ε and W p ( ν n , ˜ µ n ) ≤ ρ 0 . By Lemma 8, we have | m γ ( ν n ) -E µ [ Z ] | ≲ 1 . Of course, we do not observe ν n , so this result is not immediately useful. To apply this fact, we use that T γ is an approximate Wasserstein contraction.

Proof. Writing F α and F β for the CDFs of α and β , respectively, we compute

as desired.

Applying Lemma 9, we bound

This matches the proposition statement when d = 1 .

For general d > 1 , we propose the coordinate-wise trimmed estimate z 0 ∈ R d given by ( z 0 ) i = m γ ( e ⊤ i # ˜ µ n ) . Plugging in δ ← 1 / (100 d ) into Lemma 8 and taking a union bound over coordinates, we condition on the 0 . 99 probability event that the one-dimensional bound holds for all coordinates. We can then bound

as desired. The penultimate inequality is a consequence of the reverse Minkowski inequality.

Lemma 10. Fix α, β ∈ P ( R d ) Write α i = e ⊤ i # α and β i = e ⊤ i # β , i ∈ [ d ] , for their coordinate-wise marginals. We then have

Proof. Take ( X,Y ) to be an optimal coupling for the W 1 ( α, β ) problem. Writing ∆ i = ∥ Y i -X i ∥ 2 for i ∈ [ d ] , the right hand side of (8) can be written as the L 1 / 2 norm ∥ ∑ d i =1 ∆ i ∥ 1 / 2 . We then bound

where the final inequality follows by the reverse Minkowski inequality for L 1 / 2 .

B.7 Proof of Proposition 6

We have

where the first equality follows from the definitions of G 2 ( σ, z 0 ) and W ε p (˜ µ n ∥ ν ) . The second equality holds because ˜ µ n = 1 n ∑ i ∈ [ n ] δ ˜ Z i , which implies that the distributions µ ′ , ν and π take the form

µ ′ = ∑ i ∈ [ n ] m i δ ˜ Z i , ν = ∑ i ∈ [ n ] m i ν i , and π = ∑ i ∈ [ n ] m i δ ˜ Z i ⊗ ν i , respectively. Note that the distribution ν i models the probability distribution of the random variable Z condition on the event that Z ′ = ˜ z i . Using the definition of the expectation operator and introducing the positive measure ν ′ i = m i ν i for every i ∈ [ n ] , we arrive at

where the second equality follows from strong duality, which holds because the Slater condition outlined in [47, Proposition 3.4] is satisfied thanks to Assumption 1. The proof concludes by removing the decision variables r and s and using the definition of ˜ µ n .

B.8 Proof of Theorem 2

̸

The proof requires the following preparatory lemma. We say that the function f is proper if f ( x ) > -∞ and dom( f ) = ∅ .

Lemma 11. The followings hold.

(i) Let f ( x ) = λg ( x -x 0 ) , where λ ≥ 0 and g : R d → R is l.s.c. and convex. Then, f ∗ ( y ) = x ⊤ 0 y + λg ∗ ( y/λ ) .

(ii) Let f ( x ) = ∥ x ∥ p for some p ≥ 1 . Then, f ∗ ( y ) = h ( y ) , where the function h is defined as in (6) .

Proof. The claims follows from [25, §E, Proposition 1.3.1 ], [57, Lemma B.8 (ii)] and [25, §E, Example 1.1.3], respectively.

Proof of Theorem 2. By Proposition 6 and exploiting the definition of ˜ µ n , we have

where the second equality follows form Assumption 2. For any fixed i ∈ [ n ] and j ∈ [ J ] , we have

where the equality is a result of strong duality due to [57, Theorem 2] and Lemma 11. The claim follows by substituting all resulting dual minimization problems into (9) and eliminating the corresponding minimization operators.

B.9 Proof of Theorem 3

Thanks to Remark 3, we have

where M ε := { m ∈ R n + : m i ≤ 1 / ( n (1 -ε )) , ∀ i ∈ [ n ] , ∑ i ∈ [ n ] m i = 1 } , and the equality follows from the primal representation of the CVaR as a coherent risk measure [3, Example 4.1] and the fact that ˜ µ n is discrete. By Assumption 2, we have

where Q ε := { q ∈ R n × J + : ∑ j ∈ [ J ] q ij ≤ 1 n (1 -ε ) , ∀ i ∈ [ n ] , ∑ ( i,j ) ∈ [ n ] × [ J ] q ij = 1 } , and the last equality easily follows by introducing the variables q ij as a means to merge the variables m i and the maximum operator. Note that the final supremum problem is nonconvex as we have bilinearity between q ij and ξ ij . Using the definition of the perspective function and the simple variable substitution ξ ij ← ξ ij /q ij , however, one can convexify this problem and arrive at

Note that strong duality holds similar to the proof of [57, Section 6]. This allows us to interchange the infimum and supremum without changing the optimal value of the problem. Then, infimizing over λ 1 and λ 2 , and noticing that the resulting supremum problem is solvable, since the feasible set is compact, conclude the first part of the proof. Following the discussion in [57, § 6], it is easy to show that the proposed discrete distribution ν ⋆ solves the worst-case expectation problem. The details are omitted for brevity. This concludes the proof.

C Proofs for Section 4

C.1 Proof of Theorem 4

We start with the following lemma.

Lemma 12. Under the setting of Theorem 4, we may decompose ℓ ⋆ = ˜ ℓ ◦ Q for Q ∈ R k × d with QQ ⊤ = I k and some ˜ ℓ : R k → R d . For any such decomposition, we have

Proof. To start, we decompose A ( z ) = RQz + z 0 , where Q ∈ R k × d with QQ ⊤ = I k , R ∈ R k × k , and z 0 ∈ R k . Note that the orthogonality condition ensures that Q ⊤ isometrically embeds R k into R d . We then choose ˜ ℓ ( w ) = ℓ ( Rw + z 0 ) .

Next, given ν ∈ G , we have Q # ν ∈ G ( k ) with W ε p ( Q # ν, Q # ˜ µ ) ≤ W ε p ( ν, ˜ µ ) , and E ν [ ℓ ] = E Q # ν [ ˜ ℓ ] . Thus, the RHS supremum is always at least as large as the LHS. It remains to show the reverse.

Fix ν ∈ G k with W ε p ( ν, Q # ˜ µ ) . Take any ν ′ ∈ P ( R k ) with W p ( ν, ν ′ ) ≤ ρ and ∥ ν ′ -Q # ˜ µ ∥ TV ≤ ε . Write κ = Q ⊤ # ν ∈ G and κ ′ = Q ⊤ # ν ′ . Since Q ⊤ is an isometric embedding, we have κ ∈ G , W p ( κ, κ ′ ) = W p ( ν, ν ′ ) ≤ ρ , and ∥ κ ′ -˜ µ ∥ TV = ∥ ν ′ -Q # ˜ µ ∥ TV ≤ ε . Finally, we have E ν [ ℓ ] = E κ [ ˜ ℓ ] . Thus, the RHS supremum is no greater than the LHS, and we have the desired equality.

Writing µ k = Q # µ , we mirror the proof of Theorem 1 and bound

To obtain the theorem, we apply Lemma 6 and observe that ∥ ˜ ℓ ∥ Lip = ∥ ℓ ⋆ ∥ Lip and ∥ ˜ ℓ ∥ ˙ H 1 , 2 ( µ k ) = ∥ ℓ ⋆ ∥ ˙ H 1 , 2 ( µ ) (since Q ⊤ is an isometric embedding from R k into R d ).

C.2 Proof of Corollary 4

This follows as an immediate consequence of Theorem 4 and Corollary 1.

C.3 Proof of Proposition 7

We simply instantiate the lower bound construction from Proposition 3 in R k , viewed as a subspace of R d . Extending each ℓ ∈ L to R d by ℓ ( z ) = ℓ ( z 1: k ) , the same lower bound applies with d ← k .

C.4 Proof of Proposition 8

For Z ∼ µ ∈ G cov and z 0 , w ∈ R d , we bound

Consequently, we have µ ∈ G cov (1 + ∥ z 0 -E [ Z ] ∥ , z 0 ) ⊆ G cov ( σ, z 0 ) . Moreover, we have W ε p (˜ µ n , µ ) ≤ W p ( µ ′ m , µ ) ≤ ρ 0 + δ = ρ . Decomposing ℓ ⋆ = ˜ ℓ ◦ Q as in Lemma 12 and writing µ k = Q # µ , the same approach applied in the proof of Theorem 4 gives

When p = 1 , we bound the regularizer radius by

using Lemma 5. Similarly, when p = 2 , we bound the radius by

We then conclude as in Theorem 4.

C.5 Proof of Proposition 9

Since iterative filtering works by identifying a subset of samples with bounded covariance and W p perturbations can arbitrarily increase second moments when p < 2 , it is not immediately clear how to apply this method. Fortunately, W 2 perturbations have a bounded affect on second moments, and, by trimming out a small fraction of samples, we can ensure that a W 1 step is bounded under W 2 .

Lemma 13. For any µ, ν ∈ P ( R d ) and 0 < γ ≤ 1 , there exists ν ′ ∈ P ( R d ) with ∥ ν ′ -ν ∥ TV ≤ τ such that W 1 ( µ, ν ′ ) ≤ W 1 ( µ, ν ) and W ∞ ( µ, ν ′ ) ≤ W 1 ( µ, ν ) /γ .

Proof. Let ( X,Y ) be a coupling of µ and ν with E [ ∥ X -Y ∥ ] = W 1 ( µ, ν ) . Writing ∆ = ∥ X -Y ∥ , the event E that ∆ ≤ W 1 ( µ, ν ) /γ has probability at least 1 -γ by Markov's inequality. We shall take ν ′ be the law of Y ′ = 1 E Y +(1 -1 E ) X . By design, ∥ ν ′ -ν ∥ TV ≤ γ , and

Finally, we bound W ∞ ( µ, ν ) ≤ ∥ 1 E ∆ ∥ ∞ ≤ W 1 ( µ, ν ) /γ .

For consistency between Settings B and B ′ , we let m = n if we are the former. Thus, in both cases, we have W p (ˆ µ m , µ ′ m ) ≤ ρ 0 and ∥ µ ′ m -˜ µ n ∥ TV ≤ ε ≤ ε 0 := 1 / 12 . It is well known that the empirical measure ˆ µ m will inherit the bounded covariance of µ for m sufficiently large, so long as a small fraction of samples are trimmed out. In particular, by Lemma A.18 of [15] and our sample complexity requirement, there exists a uniform discrete measure α k over a subset of k = (1 -ε 0 / 120) m points, such that ∥ E α k [ Z ] -E µ [ Z ] ∥ ≲ 1 and Σ α k ⪯ O (1) I d with probability at least 0.99.

Moreover, applying Lemma 13 with γ = ε 0 / 120 , there exists β ∈ P ( R d ) with ∥ β -µ ′ m ∥ TV ≤ ε 0 / 120 and W 2 ( β, ˆ µ n ) ≤ 240 ρ 0 /ε 0 . Combining, we have that W ε 0 / 120+ ε 0 / 120+ ε 0 2 ( α m , ˜ µ n ) = W 61 ε 0 / 60 2 ( α k , ˜ µ n ) ≤ 240 ρ 0 /ε 0 , and so there exists κ ∈ P ( R d ) such that W ∞ ( α k , κ ) ≤ 240 ρ 0 /ε 0 and ∥ κ -˜ µ n ∥ TV ≤ 61 ε 0 / 60 . The W ∞ bound implies that Σ κ ⪯ O (1 + ρ 2 0 ε -2 0 ) I d .

Thus, by the proof of Theorem 4.1 in [26] and our sample complexity requirement, the iterative filtering algorithm (Algorithm 1 therein, based on that of [15]) applied with an outlier fraction of 61 / 60 ε 0 ≤ 1 / 10 returns a reweighting of ˜ µ n whose mean z 0 ∈ R d is within O ( √ ε 0 + ρ 0 / √ ε 0 ) = O (1 + ρ 0 ) of that of κ . Thus, we obtain

as desired.

C.6 Proof of Proposition 10

We have

where the first equality follows from the definitions of G cov ( σ, z 0 ) and W ε p (˜ µ n ∥ ν ) . The second and the third equalities follow from the same variable substitution as in the proof of Proposition 6. The last optimization problem admits the dual form

Strong duality holds thanks to Assumption 3 and [47, Proposition 3.4]. The proof of the second claim concludes by removing the decision variables r and s and using the definition of ˜ µ n .

C.7 Proof of Theorem 5

By Proposition 10 and exploiting the definition of ˜ µ n , we have

where the second equality follows form Assumption 2. For any fixed i ∈ [ n ] and j ∈ [ J ] , we have

where the equality is a result of strong duality due to [57, Theorem 2] and Lemma 11. The claim follows by introducing the epigraph variable τ ij for the term 1 4 ( ζ G ij ) ⊤ Λ -1 1 ζ G ij , and substituting all resulting dual minimization problems into (10) and eliminating the corresponding minimization operators. Note that by the Schur complement argument, we have

which implies that the resulting reformulation is indeed convex.

D Comparison to WDRO with Expanded Radius around Minimum Distance Estimate (Remark 1)

First, we prove the claimed excess risk bound for WDRO with an expanded radius around the minimum distance estimate ˆ µ = ˆ µ (˜ µ, G , ε ) := argmin ν ∈G W ε p ( ν, ˜ µ ) . We write c = 2(1 -ε ) -1 /p as in Theorem 1.

Lemma 14. Under Setting A, let ˆ ℓ = argmin ℓ ∈L sup ν ∈P ( Z ): W p ( ν, ˆ µ ) ≤ ρ ′ E ν [ ℓ ] , for the expanded radius ρ ′ := cρ +2 τ p ( G , 2 ε ) . We then have E µ [ ˆ ℓ ] -E µ [ ℓ ⋆ ] ≤ R µ,p ( cρ +2 τ p ( G , 2 ε ); ℓ ⋆ ) .

Proof. Since W ε p ( µ, ˜ µ ) ≤ ρ and µ ∈ G , we have W ε p (ˆ µ, ˜ µ ) ≤ ρ . Thus, Lemma 2 gives that W 2 ε p (ˆ µ, µ ) ≤ 2 ρ . By Lemma 3, we then have W p (ˆ µ, µ ) ≤ cρ +2 τ p ( G , 2 ε ) , and so Lemma 7 gives the desired result.

In practice, we are unaware of efficient finite-sample algorithms to compute ˆ µ . For the class G cov , we instead propose the spectral reweighing estimate ˇ µ = ˇ µ (˜ µ, ε ) := argmin ν ∈P 2 ( Z ) , ν ≤ 1 1 -2 ε ˜ µ ∥ Σ ν ∥ op , where ∥ · ∥ op is the matrix operator norm (see [26] for varied applications of spectral reweighing). In practice, when ˜ µ = ˜ µ n is an n -sample empirical measure, one can efficiently obtain a feasible measure ν whose objective value is optimal up to constant factors for the problem with ε ← 3 ε , using the iterative filtering algorithm [15]. We work with the exact minimizer ˇ µ for convenience, but our results are robust to such approximate solutions.

Lemma 15. Under Setting A with G = G cov , p = 1 , and 0 < ε ≤ 0 . 2 , we have W 1 (ˇ µ, µ ) ≲ √ dρ + √ dε , and this bound is tight; that is, there exists an instance ( µ, ˜ µ ) ∈ G cov × P ( R d ) with W ε 1 ( µ, ˜ µ ) ≤ ρ such that W 1 (ˇ µ, µ ) ≳ √ dρ + √ dε . Consequently, the WDRO estimate ˇ ℓ = argmin ℓ ∈L sup ν ∈P ( Z ): W 1 ( ν, ˇ µ ) ≤ ˇ ρ E ν [ ℓ ] satisfies E µ [ ˇ ℓ ] -E µ [ ℓ ⋆ ] ≤ R µ, 1 ( O ( √ dρ + √ dε ); ℓ ⋆ ) .

Proof. Upper bound: For the upper bound on W 1 estimation error, fix any ˜ µ ∈ P ( R d ) with W ε 1 (˜ µ, µ ) ≤ ρ . Take any µ ′ ∈ P ( R d ) such that W 1 ( µ ′ , µ ) ≤ ρ and ∥ µ ′ -˜ µ ∥ TV ≤ ε . By Lemma 13, there exists α ∈ P ( R d ) with W 1 ( α, µ ) ≤ ρ , W 2 ( α, µ ) ≤ ρ √ 2 /ε , and ∥ α -˜ µ ∥ TV ≤ 2 ε . Fixing an optimal coupling π ∈ Π( α, µ ) for the W 2 ( α, µ ) problem and letting ( Z, W ) ∼ π , we bound

Thus, α ∈ G cov (1 + ρ √ 2 /ε ) . Write β := 1 ( α ∧ ˜ µ )( R d ) α ∧ ˜ µ , and note that this midpoint measure is feasible for the problem defining ˇ µ . Hence, we have

and so ˇ µ ∈ G cov ( (1 -2 ε ) -1 / 2 (1 + ρ √ 2 /ε ) ) . Moreover, we have ∥ ˇ µ -α ∥ TV ≤ 4 ε . Thus, using Lemma 5 and the fact that 4 ε is bounded away from 1, we bound

By the triangle inequality, we have W 1 (ˇ µ, µ ) ≲ √ dρ + √ dε . The risk bound follows by Lemma 7. Taking the final error measurement using distance between means instead of W 1 , we observe that ∥ E ˇ µ [ Z ] -E µ [ Z ] ∥ 2 ≲ ρ + √ ε .

Lower bound: To see that this guarantee cannot be improved, fix clean measure µ = δ 0 ∈ G cov , and consider the corrupted measure ˜ µ = (1 -3 ε ) δ 0 +2 ε ( 1 2 δ ρ 2 ε e 1 + 1 2 δ -ρ 2 ε e 1 ) + ε N ( 0 , ρ 2 100 ε 2 I d ) , constructed so that W ε 1 (˜ µ, µ ) ≤ ρ . Intuitively, iterative filtering seeks to drive down the operator norm of covariance matrix and will thus focus on removing mass from the second mixture component.

To formalize this, we decompose the output of filtering as (1 -2 ε )ˇ µ = (1 -3 ε -τ ) δ 0 + α + β , where 0 ≤ τ ≤ 2 ε and α, β ∈ M + ( R d ) such that α ≤ 2 ε ( 1 2 δ ρ 2 ε e 1 + 1 2 δ -ρ 2 ε e 1 ) and β ≤ ε N ( 0 , ρ 2 100 ε 2 I d ) . We have τ +(2 ε -α ( R d )) + ( ε -β ( R d )) = 2 ε by the definition of ˇ µ . Further note that ∥ E ˇ µ [ Z ] ∥ ≲

√ ε + ρ ≲ ρ , by the bounds above. Now suppose for sake of contradiction that β ( R d ) ≤ ε/ 2 . Then α ( R d ) ≥ ε/ 2 , and so we bound

On the other hand, another feasible outcome for spectral reweighing is µ ′ = 1 1 -ε (1 -3 ε ) δ 0 + ε N ( 0 , ρ 2 100 ε 2 I d ) , for which we have ∥ Σ µ ′ ∥ 1 / 2 op = ρ 10 √ ε . Since 10 > √ 8 , this contradicts optimality of ˇ µ if ε ≤ cρ 2 for a sufficiently small constant c . However, if ε > cρ 2 , then a lower bound of Ω( √ dε ) suffices. This bound holds even without Wasserstein perturbations; see the G cov lower risk bound of Ω( √ dε ) in Theorem 2 of [38].

We now suppose that β ( R d ) ≥ ε/ 2 . Let Z ∼ N ( 0 , ρ 2 100 ε 2 ) and write F for the CDF of ∥ Z ∥ 2 (which has a scaled χ 2 d distribution). We then have ∫ ∥ z ∥ d β ( z ) ≥ ε E [ ∥ Z ∥ | ∥ Z ∥ 2 ≤ F -1 (1 / 2) ] ≳ √ dρ , using concentration of χ 2 d about its mean. Thus, W 1 (ˇ µ, µ ) ≥ E ˇ µ [ ∥ Z ∥ ] -E µ [ ∥ Z ∥ ] ≳ √ dρ .

E Smaller Robustness Radius for Outlier-Robust WDRO (Remark 2)

In the classical WDRO setting with ρ 0 = ε = 0 , the radius ρ can often be taken significantly smaller than n -1 /d if L and µ are sufficiently well-behaved. In particular, when µ satisfies a T 2 transportation inequality, [18] proves that ρ = ˜ O ( n -1 / 2 ) gives meaningful risk bounds. Recall that µ ∈ T 2 ( τ ) if

where H ( ν ∥ µ ) := ∫ Z log( dν/dµ ) dν denotes relative entropy when ν ≪ µ (and is + ∞ otherwise). We note that T 2 is implied by the log-Sobolev inequality, which holds for example when µ has strongly log-concave density. Under T 2 , [18] shows the following.

Proposition 11 (Example 3 in [18]) . Fix Z = R d × R , τ, B > 0 , and an α -smooth and L -Lipschitz function f : R → R . Consider the parameterized family of loss functions L = { ( x, y ) ↦→ ℓ θ ( x, y ) = f ( θ ⊤ x -y ) : θ ∈ Θ } , where Θ ⊂ { θ ∈ R d : ∥ θ ∥ ≤ B } . Fix µ ∈ P ( Z ) whose first marginal µ X = µ ( · × R ) satisfies µ X ∈ T 2 ( τ ) and such that inf θ ∈ Θ E µ [ f ′ ( θ ⊤ X -Y ) 2 ] > 0 . Write

For t > 0 , define

Then, with probability at least 1 -2 /n -2 e -t , we have

We note that (11) is stated without the absolute value on the right hand side, but that this strengthened result holds due to the discussion after [18, Theorem 1]. This generalization bound immediately gives the following excess risk bound.

Corollary 5. Assume n ≥ 800 . Fix ρ n , δ n , η n , and L as in Proposition 11 with t = 7 , and take ℓ ˆ θ ∈ L minimizing (1) with p = 2 and radius ρ = ρ n . Then, with probability at least 0 . 99 , we have

Proof. By Proposition 11 and [18, Remark 1], we have

with probability at least 1 -2 /n -2 e -t ≥ 0 . 995 . Since ℓ θ is α -smooth, Lemma 6 gives that

with probability at least 0.995. By Markov's inequality, we can substitute ˆ µ n with µ at the cost of a constant factor blow-up in excess risk and a decrease in the confidence probability to, say, 0.99.

In the example above, excess risk is controlled by the 2-Wasserstein regularizer with radius ρ n = O ( n -1 / 2 ) , up to O ( n -1 ) correction terms, which is significantly smaller that the typical radius size of O ( n -1 /d ) . We shall now lift this improvement to the outlier-robust setting. Similar to Proposition 4, we perform outlier-robust DRO with a modified choice of A . This time, writing G 2 ( σ ) = ∪ z 0 ∈Z G 2 ( σ, z 0 ) , we have the following.

Proposition 12 (Outlier-robust WDRO under T 2 ) . Assume n ≥ 800 and µ ∈ G cov . Fix ρ n , δ n , η n , and L as in Proposition 11 with t = 8 , and take ℓ ˆ θ minimizing (OR-WDRO) with center ˜ µ n , radius ρ = ρ 0 +15 ρ n +200 √ d , and A = G 2 (15 √ d + ρ n ) . Then, with probability at least 0.99, we have

Proof. Noting that G cov ⊆ G 2 ( √ d ) , we have by Markov's inequality that ˆ µ n ∈ G 2 (15 √ d ) with probability at least 0.995. In other words, there exists z 0 ∈ Z such that W 2 (ˆ µ n , δ z 0 ) ≤ 15 √ d . Thus, for any ν ∈ P ( Z ) with W 2 (ˆ µ n , ν ) ≤ ρ n , we have W 2 ( ν, δ z 0 ) ≤ 15 √ d + ρ n , and so ν ∈ G 2 (15 √ d + ρ n ) . By Lemmas 4 and 5, this implies that

Next, by Proposition 11, with probability at least 1 -1 / 400 + 2 e -8 ≥ 0 . 9985 , we have for each θ ∈ Θ that

Let c = 2 ( 1 -ε 1 -2 ε ) 1 /p . Using optimality of ˆ ℓ and Lemma 2, we further bound

As in Corollary 5, we can substitute ˆ µ n with µ at the cost of a constant factor increase in excess risk along with a small decrease in the confidence probability (in this case, sufficiently small such that the total failure probability is at most 0 . 01 ).

The main goal of Proposition 12 was to demonstrate that one can expect improved excess risk bounds for outlier-robust WDRO in situations where such improvements hold for standard WDRO. We conjecture that similar guarantees hold for additional settings and under milder assumptions like the T 1 inequality, but leave such refinements for future work. In particular, for the class G cov , it would be desirable to prove such bounds when p = 1 , so that the TV contribution to the risk vanishes as ε → 0 .

F Parameter Tuning (Remark 5)

To clarify the parameter selection process, we consider Setting B with the class G = G cov ( σ ) , p = 1 , and ε ≤ 1 / 3 . We aim to efficiently achieve excess risk

matching Proposition 4, when

for some parameter guesses ˆ σ , ˆ ε , and ˆ ρ , and a robust mean estimate z 0 . First, we observe that the coordinate-wise trimmed mean estimate from Proposition 5 is computed without knowledge of the parameters, so it is safe to assume that ∥ z 0 -E µ [ Z ] ∥ ≲ √ d + ρ 0 . If the parameter guesses are conservative, i.e., ˆ σ ≥ σ , ˆ ε ≥ ε , and ˆ ρ ≥ ρ 0 + W 1 ( µ, ˆ µ n ) , then we may still employ Proposition 4. If they are not too large, i.e., ˆ σ ≲ σ , ˆ ε ≲ ε , and ˆ ρ ≤ ρ 0 + σ √ dn -1 /d , this gives the desired excess risk.

We now explore what prior knowledge of σ , ε , and ρ 0 is needed to obtain such guesses. First, we show that effective learning is impossible without knowledge of ρ 0 , even for standard WDRO (i.e., ε = 0 , σ = ∞ ). For ease of presentation, we present the following lower bound without sampling.

Lemma16. There exists a family of loss functions L over R such that, for any C > 0 and decision rule D : P ( R ) →L , there are µ, ˜ µ ∈ P ( R ) such that E µ [ D (˜ µ )] > C (inf ℓ ∈L E µ [ ℓ ] + W 1 ( µ, ˜ µ ) ∥ ℓ ∥ Lip ) .

Proof. Let L = { ℓ θ : θ > 0 } , where ℓ θ ( z ) := z/θ + θ . By design, we have ∥ ℓ θ ∥ Lip = 1 /θ . Let ˜ µ = δ 0 and write D (˜ µ ) = ℓ ˆ θ for some ˆ θ > 0 . We then set µ = δ ρ for ρ = 10 ˆ θ 2 C 2 . This gives

and

Thus, we assume in what follows that ˆ ρ = ρ 0 is known. Moreover, we require knowledge of at least one of ε and σ . If both are unknown, then is information theoretically impossible to meaningfully distinguish inliers from outliers (see Exercise 1.7b of [16] for a discussion of this issue in the setting of robust mean estimation). If ε is known, then we can choose ˆ σ as 2 i for the smallest i such that the supremum of Eq. (13) is feasible (or, equivalently, such that the associated dual is bounded for some fixed ℓ ∈ L ). We can overshoot by at most a factor of two and thus achieve the desired risk bound. Using binary search, this adds a multiplicative overhead logarithmic in the ratio of the initial guess for σ and its true value. The same approach can be employed if σ is known but not ε .

G Additional Experiments

We now provide several experiments in addition to those in the main body. Code is again provided at https://github.com/sbnietert/outlier-robust-WDRO .

First, in Fig. 3, we extend the experiment summarized Fig. 2 (top) to include runs with ˆ ε = ε and varied Wasserstein radius ˆ ρ ∈ { ρ/ 2 , ρ, 2 ρ } . For this simple learning problem and perturbation model, we find that the choice of ˆ ρ plays a minor role in the resulting excess risk. We emphasize that, in the worst case, selection of ˆ ρ can be critical, as demonstrated by Lemma 16.

̸

Next, we consider linear classification with the hinge loss, i.e. L = { ℓ θ ( x, y ) = max { 0 , 1 -y ( θ ⊤ x ) } : θ ∈ R d -1 } . This time (to ensure that the resulting optimization problem is convex), our approach supports Euclidean Wasserstein perturbations in the feature space, but no Wasserstein perturbations in the label space (this corresponds to using Z = R d -1 × R equipped with the (extended) norm ∥ ( x, y ) ∥ = ∥ x ∥ 2 + ∞· 1 { y = 0 } . We consider clean data ( X,Y = sign( θ ⊤ ⋆ X )) ∼ µ , where X ∼ N (0 , I d -1 ) . The corrupted data ( ˜ X, ˜ Y ) ∼ ˜ µ satisfies ( ˜ X, ˜ Y ) = ( X + ρe 1 , Y ) with probability 1 -ε and ( ˜ X, ˜ Y ) = (100 X, -Y ) with probability ε , so that W ε p (˜ µ ∥ µ ) ≤ ρ . In Figure 4 (left), we fix d = 10 and compare the excess risk E µ [ ℓ ˆ θ ] -E µ [ ℓ θ ⋆ ] of standard

10

20

30

40

50

60

70

80

90

100

# samples

0

0.5

1

1.5

2

2.5

3

3.5

4

excess risk (mean absolute deviation)

Excess Risk for Varied Sample Size and Method

standard WDRO

WDRO and outlier-robust WDRO with G = G 2 , as described by Proposition 4 and implemented via Theorem 2. The results are averaged over T = 20 runs for sample size n ∈ { 10 , 20 , 50 , 75 , 100 } . We note that this example cannot drive the excess risk of standard WDRO to infinity, so the separation between standard and outlier-robust WDRO is less striking than regression, though still present.

We further present results for multivariate regression. This time, we consider Z = R d × k equipped with the ℓ 2 norm, and use the loss family L = { ℓ M ( x, y ) = ∥ Mx -y ∥ 1 : M ∈ R k × d } . We consider clean data ( X,Y = M ⊤ ⋆ X ) ∼ µ , where M ⋆ ∈ R k × d and X have standard normal entries. The corrupted data ( ˜ X, ˜ Y ) ∼ ˜ µ satisfies ( ˜ X, ˜ Y ) = ( X + ρe 1 , Y ) with probability 1 -ε and ( ˜ X, ˜ Y ) = (10 X, -100 M ⋆ X ) with probability ε , so that W ε p (˜ µ ∥ µ ) ≤ ρ . In Figure 4 (right), we fix d = 10 and k = 3 , and compare the excess risk E µ [ ℓ ˆ M ] -E µ [ ℓ M ⋆ ] of standard WDRO and outlierrobust WDRO with G = G 2 , as described by Proposition 4 and implemented via Theorem 2. The results are averaged over T = 10 runs for sample size n ∈ { 10 , 20 , 50 , 75 , 100 } . We are restricted to low k since the ℓ 1 norm in the losses is expressed as the maximum of 2 k concave functions (specifically, we use that ℓ M ( x, y ) = max α ∈{-1 , 1 } k α ⊤ ( Mx -y ) ).

Finally, we turn to a classification task with image data. We train a robust linear classifier to distinguish

10

20

30

40

50

60

70

80

90

100

# samples

0.2

0.4

0.6

0.8

1

1.2

excess risk (hinge loss)

Excess Classification Risk with WDRO

standard WDRO

OR-WDRO

10

20

30

40

50

60

70

80

90

100

# samples

10 -4

10 -3

10 -2

10 -1

10 0

10 1

10 2

Excess Linear Regression Risk with WDRO

standard WDRO

OR-WDRO

between the MNIST [14] digits '3' and '8' when 10% of training labels are flipped uniformly at random, using outlier-robust WDRO with G = G 2 and the hinge loss, as above, applied with ε = 0 . 1 and ρ = 0 . 01 . To ensure tractability, we first pass the images through principal component analysis to reduce the input to 50 dimensions, and we again use Theorem 2 for implementation. In Fig. 5, we plot the classification accuracy of our robust classifier compared to one learned via standard WDRO for training set size n ∈ { 10 , 20 , 50 , 100 , 150 } , averaged over T = 10 runs. In this case, we do not witness a noticeable improvement over standard WDRO. We suspect that the relevant G 2 class is too large to be of significant use in this high-dimensional setting.

For all experiments, confidence bands are plotted representing the top and bottom 10% quantiles

0

50

100

150

sample size

0.75

0.8

0.85

0.9

0.95

classification accuracy (% correct)

MNIST Classification Accuracy

regular WDRO

OR-WDRO

among 100 bootstrapped means from the T runs. The additional experiments were performed on an M1 Macbook Air with 16GB RAM in roughly 30 minutes each.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper empowers WDRO with the ability to resist outliers, building upon the outlier-robust Wasserstein distance $W_p^\\epsilon$. The excess risk of the solution to both the outlier-robust WDRO and its empirical version is given. An improved bound of the excess risk is derived for the setting of low-dimensional features. The optimization algorithm for the outlier-robust WDRO is developed with the dual form of the minimax problem. Empirical study validates the effectiveness of the proposed method with a simple regression setting involving both Wasserstein and Total Variance contaminiation.\n\nSTRENGTHS:\n1. DRO is known to be over-pessimistic in the existence of outliers. Empowering Distributionally Robust Optimziation with robustness against noisy labels is important. The outlier-robust Wasserstein distance is a well-developed revision of the original Wasserstein distance to account for certain degree of contamination. Thus, constructing the uncertainty set of DRO with $W_p^\\epsilon$ is both significant and reasonable.\n2. The theoretical analysis of the excess risk of the proposed method is sound and comprehensive. The results quantify how the solution adapts to given geometric and TV contamination. The effectiveness of the empircial algorithm is also guaranteed by the empirical bound of excess risk.\n\nWEAKNESSES:\n1. The effectiveness of outlier-robust WDRO is theoretically guaranteed by the excess risk bound. My main concern is over the **advantage** of outlier-robust WDRO v.s. WDRO. As is stated by the authors, the excess risk of outlier-robust WDRO is upper bounded by the $W_p$ regularizer with a larger Wasserstein radius. The authors also recognize that the same upper bound could be achieved by a radius-expanded WDRO. Two advantages of WDRO given in Remark 1 are not convincing. \n\n- The authors claim that a heavy preprocessing step is required by WDRO to estimate the exact radius. However, in practical settings beyond simulated experiments, the exact parameters of contamination levels $\\rho, \\epsilon$  are also latent. Either an estimate or tuning of the radius is necessary for both WDRO and the outlier-robust version. \n- The authors prove an improved bound of the excess risk for outlier-robust WDRO in the setting of low-dimensional features and claim that the bound of WDRO could not be improved, which is unsupported. A tight bound of the excess risk of WDRO with low-dimensional features might be given to consolidate the author's claim. Otherwise, an inequality between the excess risk of WDRO and its outlier-robust version might be given similarly to Eq.4. Furthermore, the proposed low-dimensional feature setting is somewhat impractical because the exact dimension $k$ is typically unknown in real datasets. \n\nIn the experiment section, I suppose the radius for WDRO is selected to be the true contaminaton radius. The curve of performances of WDRO and outlier-robust version with increasing radius might demonstrate their gap more convincingly.\n\n2. I am concerned if the formulation of outlier-robust WDRO (eq.3) is well defined. Consider a simple case where all the samples of $\\tilde \\mu$ are discretely distributed on $2/\\epsilon$ points, one of which is denoted by $(x_0,y_0)$. By arbitarily modifying $y_0$ to $y'$ we get a new distribution $\\nu(y')$. According to Line 131 we have $W_p^\\epsilon(\\tilde \\mu, \\nu) =0$. Thus, all the $\\nu(y')$ is included in the uncertainty set of the correspondong $W_p^\\epsilon$ DRO. However, the risk of a given predictor on the group of $\\nu(y')$ could be unbounded since $y'$ is unconstrained. As a consequence, there might be no solution to eq.3. Intuitively, though eq.3 incorporates the clean distribution into the uncertainty set, it might also include more dirty distributions since the outlier-robust Wasserstein distance tolerates a small fraction of outliers, but the risk on these outliers could be unbounded. Therefore, I'm concerned if the porposed outlier-robust DRO would be biased towards more dirty distributions instead of recovering the clean one.\n\n3. I am also concerned with the selection of the radius $\\rho$ of outlier-robust WDRO. As is indicated by Theorem 1, larger $\\rho$ leads to higher excess risk while Line 203 states that the excess risk bound only stands for $\\rho \\geq \\rho_0 + W_p(\\mu, \\hat \\mu_n)$, implying that $\\rho$ shall not be too small. Since both $\\rho_0$ and $W_p(\\mu, \\hat \\mu_n)$ are unavailable in practical settings, the selection of the parameter seems tricky.\n\nQUESTIONS:\n1. Why doesn't the dual form of outlier-robust WDRO in Proposition 7 reduce to that of WDRO when $\\epsilon \\rightarrow 0$, even if neglecting $\\lambda_1$? Specifically, the operator $[\\cdot]_+$ claimed to be vital for outlier-robust WDRO does not disappear when outlier-robust WDRO reduce to WDRO. \n\nSome typos:\n\n- Line 158: $\\nu$ v.s. $\\mu$\n- Supplementary Line 565: In second equality, it might be $s_i\\geq r_i$ and $s_i \\geq 0$.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIt is well known that Wasserstein distances do not commute well with total variation distance - a slight perturbation in TV can change the Wasserstein distance by a lot. This means that models that are robust to corruptions in the data distribution in Wasserstein distance can still be vulnerable to outliers or corruptions in TV sense. This paper addresses this problem by considering robustness with respect to both TV and Wasserstein. The paper studies worst-case excess risk w.r.t. corruptions in a new distance termed the 'outlier robust Wasserstein distance'. The authors derive upper and lower bounds on the worst-case excess risk for families of distributions that are either sub-Gaussian or have bounded covariance. The upper bounds depend on a recently introduced term called the \"resilience\" of a probability measure. Intuitively, resilience of a measure quantifies the deviation in expectation of a function (in this case the loss function) when taken w.r.t. a measure in a probability ball around the original measure. The authors also propose a tractable reformulation of the min-max problem via strong duality. Finally, the authors tighten their results for the case when the true distribution lies on a low dimensional linear subspace, by extending their results to corruptions in outlier robust max-sliced Wasserstein distance. The authors also verify their excess risk bounds on a toy dataset for linear regression problem with mean absolute deviation loss.\n\nSTRENGTHS:\n- significance: the problem of incorporating outlier robustness into the framework of wasserstein distributionally robust optimization (WDRO) is of significance to both ML and optimization communities, and has already received some recent interest. This paper makes progress on this significant problem. \n- novelty: I think the upper and lower bounds on the min-max excess risk are novel. The strong duality result appears to be a generalization of Gao and Kleywegt's result on strong duality for WDRO, but I think it is is non-trivial.\n\nWEAKNESSES:\n- **Contextualizing the work within related literature**: Although the paper is overall well written, I wish it did a better job at contextualizing their results properly by comparing it with the two special cases of WDRO without TV corruption and of TV robustness without WDRO. For WDRO without TV corruption for example, a natural point of comparison is [this](https://optimization-online.org/wp-content/uploads/2016/04/5396.pdf) paper by Gao and Kleywegt. One point of comparison could be the existence and the form of the worst-case distribution in the min-max risk. This question is answered for the WDRO without TV corruption in Gao and Kleywegt's paper but this paper does not address it at all, which seems odd to me. For TV corruption without WDRO (aka Huber contamination), there are several works from the robust statistics community.\n- **Usefulness on top of stricter WDRO**: I think Remark 1 deserves much more discussion. From proposition 1 it is clear that the new compound model of outlier robust WDRO can be subsumed under plain vanilla DRO by increasing the budget of the Wasserstein contamination in proportion to the resilience of the true distribution. Then, how much additional utility do we gain by analyzing the min-max excess risk under the compound model in detail? I must admit I don't understand what the authors mean by the expensive pre-processing step for WDRO. Please explain this to me satisfactorily, and I am willing to change my opinion on this particular weakness.\n\nQUESTIONS:\n- Proposition 3: Is this lower bound valid for p = 2, or only for p = 1? The loss function family is only assumed to be Lipsclitz and not $\\alpha$-smooth, whereas for the upper bound, the loss is assumed to be $\\alpha$-smooth for the case of p = 2. \n- Proposition 4 could just be a remark. \n- Something is broken in reference [38]. Are you sure this is the correct reference? I did not find a Proposition 3.4 in it. Also, as I have stated previously, I would very much like a closer comparison of the strong duality result with that of Gao and Kleywegt. Do both the results become identical if one of the lambda's is zero?\n- Do the results of section 4 still go through if the max-sliced wasserstein distance is replaced with the sliced wasserstein distance that takes expectation over all slices instead of max?\n- Lemma 5, which seems crucial to the proof of Theorem 1, is taken from [12], supposedly a forthcoming paper, which I am unable to access anywhere. So, I am unable to check Theorem 1 fully.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an outlier-robust Wasserstein Distributionally Robust Optimization (DRO) framework that aims to capture both geometric uncertainties and non-geometric perturbations, such as adversarial outliers. By utilizing the outlier-robust Wasserstein distance, the proposed framework allows for the arbitrary corruption of a fraction of the data. The authors design an uncertainty set using a robust Wasserstein ball and derive minimax optimal excess risk bounds. They also establish a strong duality for efficient computation. The resulting problem involves tuning three parameters:\n\n- the bounded covariance parameter $\\sigma$,\n  \n- the radius of the ambiguity set $\\rho$,\n  \n- and the contamination parameter $\\varepsilon$.\n  \n\nMoreover, the authors address dimension dependencies in risk bounds for low-dimensional features by introducing the projection robust optimal transport. The paper concludes with experimental validation of the theory on regression and classification tasks.\n\nSTRENGTHS:\n- The paper replaces the Wasserstein distance with the outlier-robust Wasserstein distance to tackle the case that the observed distribution is contaminated with outliers.\n- The authors establish the excess risk bounds of decisions for the cases $p=1,2$.\n- The authors derive tractable reformulation by replacing $\\mathcal{G}_{\\text{cov}}$ and leveraging dual of the problem.\n- The low-dimensional features are considered to address the problem of dimension dependency.\n\nWEAKNESSES:\n- In Remark 3 and Appendix E, the paper briefly touches on the selection of the parameter $\\varepsilon$. However, the overall discussion on parameter selection is limited. The model introduced in the paper involves several parameters, such as $\\rho$, $\\varepsilon$, and $\\sigma$, which may not be fully independent. Therefore, a more comprehensive discussion about the tuning of these parameters is warranted.\n  \n- In the experiments, the paper exclusively uses the standard Wasserstein DRO (WDRO) as the baseline for comparison. However, considering that both models aim to address the outlier challenge, it would be valuable to include DFO [A], which is another model specifically designed for handling outliers, as a reasonable baseline for comparison.\n  \n- The presence of a few missing references should be addressed in order to enhance the completeness and accuracy of the paper. In Section 4, it would be beneficial to include references [B, C, D], which introduce the concepts of Wasserstein projection pursuit and projection robust Wasserstein, respectively.\n\n[A] Jiang, Nan, and Weijun Xie. \"DFO: A Framework for Data-driven Decision-making with Endogenous Outliers.\" (2022).\n\n[B] Huang, Minhui, Shiqian Ma, and Lifeng Lai. \"A riemannian block coordinate descent method for computing the projection robust wasserstein distance.\" *International Conference on Machine Learning*. PMLR, 2021.\n\n[C] Paty, François-Pierre, and Marco Cuturi. \"Subspace robust Wasserstein distances.\" *International conference on machine learning*. PMLR, 2019.\n\n[D] Niles-Weed, Jonathan, and Philippe Rigollet. \"Estimation of wasserstein distances in the spiked transport model.\" *Bernoulli* 28.4 (2022): 2663-2688.\n\nQUESTIONS:\n- Would it be possible to provide further elaboration on the process of determining suitable parameter values for $\\rho$, $\\varepsilon$, and $\\sigma$?\n- In the proof of Proposition 7 (Appendix B.8), is the constraint $\\sum_{i\\in [n]}m_i = 1$ included to ensure that $\\mu^\\prime$ is a valid probability distribution? It appears that in the problem mentioned above l566, the constraint should be $0 \\leq r_i \\leq s_i$ rather than the constraint $0 \\leq s_i \\leq r_i$.\n- In lines 48-49, it appears that the term $W_p^\\varepsilon$ is defined in reference to outlier-robust optimal transport, rather than the partial optimal transport as described in [D], which focuses on transporting partial mass from the source distribution to the target distribution.\n\nLIMITATIONS:\nUnless I missed it, I believe the authors do not expand on the limitations of their approach. It would help to add a short section discussing that. This work does not have a negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to make Wasserstein distributionally robust optimization problems robust to adversarial outliers including geometric perturbations and non-geometric contamination of the data. This goal is achieved by considering relevant Wasserstein ball which includes both type of adversarial attacks. Later, the authors provides strong duality results which allows them to improve the computation complexity of the proposed outlier-robust WDRO problem.\n\nSTRENGTHS:\nThis is a well written paper with novel contributions. The robustness of WDRO to the outliers is a very natural problem and the authors proposed a Wasserstein distance based constraints to provide robustness. This is particularly hard if we do not make any presumptions on the distribution of data; therefore, I find the contributions of this paper to be significant to the literature. There might be some concerns that I have mentioned in the weaknesses part, but, overall I am impressed by the work done here.\n\nWEAKNESSES:\nAs I have mentioned at the strengths part, I believe this paper has a novel contribution to the robust optimization literature. Some few weaknesses that I observed was that authors assumed the readers have prior knowledge on relation between adverserial attacks and Wasserstein and TV perturbations. These are not so obvious for me and might be useful to include some discussion/motivation about how these attacks can be associated with claimed perturbations in metric space. \n\nI believe it would strengthen the paper if authors can obtain some results on larger datasets. Considering the time constraint of the conference, I believe it would be sufficient to add one large dataset to experiments.\n\nQUESTIONS:\n- What is the definition for $\\theta$ performing uniformly well over Wasserstein ball (stated in Line 29)? \n- I could not see the definition of $\\Sigma_\\mu$ in the paper.\n\nLIMITATIONS:\nThe Assumption 2 can be challenging to maintain/check in practice. Moreover, this application might also be hard to implement on large datasets such as classification problems on MNIST or FMNIST. That being said, I don't see these possible limitations as a major problem considering the theoretical novelty of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}