Title: Exact Generalization Guarantees for (Regularized) Wasserstein Distributionally Robust Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: robust optimization

distributionally robust optimization

optimization under uncertainty

generalization

Wasserstein distance

optimal transport
Abstract: Wasserstein distributionally robust estimators have emerged as powerful models for prediction and decision-making under uncertainty. These estimators provide attractive generalization guarantees: the robust objective obtained from the training distribution is an exact upper bound on the true risk with high probability. However, existing guarantees either suffer from the curse of dimensionality, are restricted to specific settings, or lead to spurious error terms. In this paper, we show that these generalization guarantees actually hold on general classes of models, do not suffer from the curse of dimensionality, and can even cover distribution shifts at testing. We also prove that these results carry over to the newly-introduced regularized versions of Wasserstein distributionally robust problems.

Exact Generalization Guarantees for (Regularized) Wasserstein Distributionally Robust Models

Abstract

Wasserstein distributionally robust estimators have emerged as powerful models for prediction and decision-making under uncertainty. These estimators provide attractive generalization guarantees: the robust objective obtained from the training distribution is an exact upper bound on the true risk with high probability. However, existing guarantees either suffer from the curse of dimensionality, are restricted to specific settings, or lead to spurious error terms. In this paper, we show that these generalization guarantees actually hold on general classes of models, do not suffer from the curse of dimensionality, and can even cover distribution shifts at testing. We also prove that these results carry over to the newly-introduced regularized versions of Wasserstein distributionally robust problems.

1 Introduction

1.1 Generalization and (Wasserstein) Distributionally Robust Models

We consider the fundamental question of generalization of machine learning models. Let us denote by f θ the loss induced by a model parametrized by θ for some uncertain variable ξ (typically a data point). When ξ follows some distribution P , seeking the best parameter θ writes as minimizing the expected loss

We usually do not have a direct knowledge of P but rather we have access to samples ( ξ i ) n i =1 independently drawn from P . The empirical risk minimization approach then consists in minimizing the expected loss over the associated empirical distribution ̂ P n = 1 n ∑ n i =1 δ ξ i (as a proxy for the expected loss over P ), i.e.,

Classical statistical learning theory ensures that, with high probability, E P [ f θ ] is close to E ̂ P n [ f θ ] up to O (1 / √ n ) error terms, see e.g., the monographs Boucheron et al. (2013); Wainwright (2019).

A practical drawback of empirical risk minimization is that it can lead to over-confident decisions (when E ̂ P n [ f θ ] < E P [ f θ ] , the real loss can be higher that the empirical one (Esfahani and Kuhn, 2018)). In addition, this approach is also sensitive to distribution shifts between training and application. To overcome these drawbacks, an approach gaining momentum in machine learning is distributionally robust optimization, which consists in minimizing the worst expectation of the loss when the distribution lives in a neighborhood of P n :

where the inner sup is thus taken over Q in the neighborhood U ( ̂ P n ) of ̂ P n in the space of probability distributions. Popular choices of distribution neighborhoods are based on the Kullback-Leibler (KL) divergence (Laguel et al., 2020; Levy et al., 2020), kernel tools (Zhu et al., 2021a; Staib and Jegelka, 2019; Zhu et al., 2021b), moments (Delage and Ye, 2010; Goh and Sim, 2010), or Wasserstein distance (Shafieezadeh Abadeh et al., 2015; Esfahani and Kuhn, 2018). If P ∈ U ( ̂ P n ) , distributionally robust models can benefit from direct generalization guarantees as,

Thus, for well-chosen neighborhoods U ( ̂ P n ) , distributionally robust objectives are able to provide exact upper-bounds on the expected loss over distribution P , i.e., the true risk.

Wasserstein distributionally robust optimization (WDRO) problems correspond to (1) with

where W ( ̂ P n , Q) denotes the Wasserstein distance between ̂ P n and Q and ρ > 0 controls the required level of robustness around ̂ P n . As a natural metric to compare discrete and absolutely continuous probability distributions, the Wasserstein distance has attracted a lot of interest in both machine learning (Shafieezadeh Abadeh et al., 2015; Sinha et al., 2018; Shafieezadeh-Abadeh et al., 2019; Li et al., 2020; Kwon et al., 2020) and operation research (Zhao and Guan, 2018; Arrigo et al., 2022) communities; see e.g., the review articles Blanchet et al. (2021); Kuhn et al. (2019).

WDRObenefits from out-of-the-box generalization guarantees in the form of (2) since it inherits the concentration properties of the Wasserstein distance (Esfahani and Kuhn, 2018). More precisely, under mild assumptions on P , (Fournier and Guillin, 2015) establishes that W ( ̂ P n , P) ≤ ρ with high probability as soon as ρ ∼ 1 /n 1 /d where d denotes the dimension of the samples space. Thus, a major issue is the prescribed radius ρ suffers from the curse of the dimensionality: when d is large, ρ decreases slowly as the number of samples n increases. This constrasts with other distributionally robust optimization ambiguity sets, such as Maximum Mean Discrepancy (MMD) (Staib and Jegelka, 2019; Zeng and Lam, 2022), where the radius scales as 1 / √ n . Moreover, the existing scaling for WDRO is overly conservative for WDRO objectives since recent works (Blanchet et al., 2022a; Blanchet and Shapiro, 2023) prove that a radius behaving as 1 / √ n is asymptotically optimal. The main difference with (Esfahani and Kuhn, 2018) is that they - and us - consider the WDRO objective as a whole, instead of proceeding in two steps: first considering the Wasserstein distance independently and invoking concentration results on the Wasserstein distance and then plugging this result in the WDRO problem.

1.2 Contributions and related works

In this paper, we show that WDRO provides exact upper-bounds on the true risk with high probability. More precisely, we prove non-asymptotic generalization bounds of the form of (2), that hold for general classes of functions, and that only require ρ to scale as 1 / √ n and not 1 /n 1 /d . To do so, we construct an interval for the radius ρ for which it is both sufficiently large so that we can go from the empirical to the true estimator (i.e., at least of the order of 1 / √ n ) and sufficiently small so that the robust problem does not become degenerate (i.e., smaller than some critical radius, that we introduce as an explicit constant). Our results imply proving concentration results on Wasserstein Distributionally Robust objectives that are of independent interest.

This work is part of a rich and recent line of research about theoretical guarantees on WDRO for machine learning. One of this first results, Lee and Raginsky (2018), provides generalization guarantees, for a general class of models and a fixed ρ , that, however, become degenerate as the radius goes to zero. In the particular case of linear models, WDRO models admit an explicit form that allows Shafieezadeh-Abadeh et al. (2019); Chen and Paschalidis (2018) to provide generalization guarantees (2) with the radius scaling as 1 / √ n . The case of general classes of models, possibly non-linear, is more intricate. Sinha et al. (2018) showed that a modified version of (2) holds at the price of non-negligible error terms. Gao (2022); An and Gao (2021) made another step towards broad generalization guarantees for WDRO but with error terms that vanish only when ρ goes to zero.

In contrast, our analysis provides exact generalization guarantees in the form (2) without additional error terms, that hold for general classes of functions and allow for a non-vanishing uncertainty radius to cover for distribution shifts at testing. Moreover, our guarantees also carry over to the recently introduced regularized versions of WDRO (Wang et al., 2023; Azizian et al., 2023), whose statistical properties have not been studied yet.

This paper is organized as follows. In Section 2, we introduce notations and our blanket assumptions. In Section 3, we present our main results, an idea of proof, and discussions. The complete proofs are deferred to the appendix.

2 Setup and Assumptions

In this section, we formalize our setting and introduce Wasserstein Distributionally Robust risks.

2.1 Wasserstein Distributionally Robust risk functions

In this paper, we consider as a samples space Ξ a subset of R d equipped with the Euclidean norm ‖·‖ . We rely on Wasserstein distances of order 2 , in line with the seminal work Blanchet et al. (2022a) on generalization of WDRO. This distance is defined for two distributions Q , Q ′ in the set of probability distributions on Ξ , denoted by P (Ξ) , as

where P (Ξ × Ξ) is the set of probability distributions in the product space Ξ × Ξ , and π 1 (resp. π 2 ) denotes the first (resp. second) marginal of π .

We denote by f : Ξ → R the loss function of some model over the sample space. The model may depend on some parameter θ , that we drop for now to lighten the notations; instead, we consider a class of functions F encompassing our various models and losses of interest (we come back to classes of parametric models of the form F = { f θ : θ ∈ Θ } in Section 4).

We define the empirical Wasserstein Distributionally Robust risk ̂ R ρ 2 ( f ) centered on ̂ P n and similarly the true robust risk R ρ 2 ( f ) centered on P as

̂ Note that ̂ R ρ 2 ( f ) , which is based on the empirical distribution ̂ P n , is a computable proxy for the true robust risk R ρ 2 ( f ) . Note also that the true robust risk R ρ 2 ( f ) immediately upper-bounds the true (non-robust) risk E ξ ∼ P [ f ( ξ )] and also upper-bounds E ξ ∼ Q [ f ( ξ )] for neighboring distributions Q that correspond to distributions shifts of magnitude smaller than ρ in Wasserstein distance.

2.2 Regularized versions

Entropic regularization of WDRO problems was recently studied in Wang et al. (2023); Blanchet and Kang (2020); Piat et al. (2022); Azizian et al. (2023) and used in Dapogny et al. (2023); Song et al. (2023); Wang and Xie (2022); Wang et al. (2022). Inspired by the entropic regularization in optimal transport (OT) (Peyré and Cuturi, 2019, Chap. 4), the idea is to regularize the objective by adding a KL divergence, that is defined, for any transport plan π ∈ P (Ξ × Ξ) and a fixed reference π ∈ P (Ξ × Ξ) , by

Unlike in OT though, the choice of the reference measure in WDRO is not neutral and introduces a bias in the robust objective (Azizian et al., 2023). For their theoretical convenience, we take reference measures that have Gaussian conditional distributions

where σ > 0 controls the spread of the second marginals, following Wang et al. (2023); Azizian et al. (2023). Then, the regularized version of R ρ 2 ( f ) (WDRO empirical risk) is given by and similarly, the regularized version of R ρ 2 ( f ) is given by

These regularized risks have been studied in terms of computational or approximation properties, but their statistical properties have not been investigated yet. The analysis we develop for WDRO estimators is general enough to carry over to these settings.

In (5) and (6), note finally that the regularization is added as a penalization in the supremum, rather than in the constraint. As in Wang et al. (2023), penalizing in the constraint leads to an ambiguity set defined by the regularized Wasserstein distance, that we introduce in Section 3.2. We refer to Azizian et al. (2023) for a unified presentation of the two penalizations.

2.3 Blanket assumptions

Our analysis is carried under the following set of assumptions that will be in place throughout the paper. First, we assume that the sample space Ξ ⊂ R d is convex and compact, which is in line with previous work, e.g., (Lee and Raginsky, 2018; An and Gao, 2021).

Assumption 1 (On the set Ξ ) . The sample space Ξ is a compact convex subset of R d .

Second, we require the class of loss functions F to be sufficiently regular. In particular, we assume that they have Lipschitz continuous gradients.

Assumption 2 (On the function class) . The functions of F are twice differentiable, uniformly bounded, and their derivatives are uniformly bounded and uniformly Lipschitz.

Finally, we assume that ̂ P n is made of independent and identically distributed (i.i.d.) samples of P and that P is supported on the interior of Ξ (which can be done without loss of generality by slightly enlarging Ξ if needed).

Assumption 3 (On the distributions) . ̂ P n = 1 n ∑ n i =1 δ ξ i where ξ 1 , . . . , ξ n are i.i.d. samples of P . We further assume that there is some R > 0 such that P satisfies suppP+ B (0 , R ) ⊂ Ξ .

3 Main results and discussions

The main results of our paper establish that the empirical robust risk provide high probability bounds, of the form of (2), on the true risk. Since the results and assumptions slightly differ between the WDRO models and their regularized counterparts, we present them separately in Section 3.1 and Section 3.2. In Section 3.3, we provide the common outline for the proofs of these results, the proofs themselves being provided in the appendix. Finally, in Section 4, we detail some examples.

3.1 Exact generalization guarantees for WDRO models

In this section, we require the two following additional assumptions on the function class. The first assumption is common in the WDRO litterature, see e.g., Blanchet et al. (2022a); Blanchet and Shapiro (2023); Gao (2022); An and Gao (2021).

The second assumption we consider in this section makes use of the notation d ( ξ, A ) , for a set A ⊂ Ξ and a point ξ ∈ Ξ , to denote the distance between ξ and A , i.e., d ( ξ, A ) = inf ζ ∈ A ‖ ξ -ζ ‖ .

Assumption 4. The quantity inf f ∈F E P [ ‖∇ f ‖ 2 ] is positive.

Assumption 5.

The following growth condition holds: there exist µ > 0 and L > 0 such that, for all f ∈ F , ξ ∈ Ξ and ξ ∗ a projection of ξ on arg max f , i.e., ξ ∗ ∈ arg min arg max f ‖ ξ -·‖ ,

The first item of this assumption has a natural interpretation: we show in Lemma A.7, that it is equivalent to the relative compactness of the function space F w.r.t. to the distance

where D H denotes the (Hausdorff) distance between sets and ‖ f ‖ ∞ := sup ξ ∈ Ξ | f ( ξ ) | is the infinity norm. The last one is a structural assumption on the functions F that is new in our context but is actually very close the so-called parametric Morse-Bott condition, introduced in of bilevel optimization (Arbel and Mairal, 2022), see Section A.5.

We now state our main generalization result for WDRO risks.

Theorem 3.1. Under Assumptions 4 and 5, there is an explicit constant ρ c depending only on F and P such that for any δ ∈ (0 , 1) and n ≥ 1 , if

then, there is ρ n = O (√ 1+log 1 /δ n ) such that, with probability 1 -δ ,

In particular, with probability 1 -δ , we have

The range of admissible radiuses is described by (7). The lower-bound, roughly proportional to 1 / √ n , is optimal, following the results of Blanchet et al. (2022a). The upper-bound, almost independent of n , depends on a constant ρ c , that we call critical radius and that has an interesting interpretation, that we formalize in the following remark. Note, finally, that, the big-O notation in this theorem has a slightly stronger meaning 1 than the usual one, being non-asymptotic in n and δ .

The second part of the result, (9), is an exact generalization bound: it is an actual upper-bound on the true risk E ξ ∼ P [ f ( ξ )] , that we cannot access in general, through a quantity that we can actually compute with ̂ P n . The first part of the result, (8) gives us insight into the robustness guarantees offered by the WDRO risk. Indeed, it tells us that, when ρ is greater than the minimal radius ρ n ∝ 1 / √ n by some margin, the empirical robust risk ̂ R ρ 2 ( f ) is an upper-bound on the loss even with some perturbations of the true distribution. Hence, as long as ρ is large enough, the WDRO objective enables us to guarantee the performance of our model even in the event of a distribution shift at testing time. In other words, the empirical robust risk is an exact upper-bound on the true robust risk R ρ ( ρ -ρ n ) ( f ) with a reduced radius.

Remark 3.2 (Interpretation of critical radius) . The critical radius ρ c , appearing in (7), is defined by

It can be interpreted as the threshold at which the WDRO problem w.r.t. P starts becoming degenerate. Indeed, when ρ 2 ≥ E ξ ∼ P [ 1 2 d 2 ( ξ, arg max f ) ] for some f ∈ F that we fix, the distribution Q given by the second marginal of the transport plan π defined by, satisfies

As a consequence, the robust problem is equal to

Thus, when the radius exceeds ρ c , there is some f such that the robust problem becomes degenerate as it does not depend on P nor ρ anymore.

Finally, note that we can obtain the same generalization guarantee as Theorem 3.1 without Assumption 5 at the expense of losing the above interpration on the condition on the radius. More precisely, we have the following result.

Theorem 3.3. Let Assumption 4 hold. For any δ ∈ (0 , 1) and n ≥ 1 , if ρ satisfies (7), and if, in addition, it is smaller than a positive constant which depends only on P , F and Ξ , then both conclusions of Theorem 3.1 hold.

This theorem can be compared to existing results, and in particular with Gao (2022); An and Gao (2021). These two papers provide generalization bounds for WDRO under a similar assumption on F and a weakened version of Assumption 3. However, these generalization bounds involve extra error terms, that require ρ to be vanishing. In comparison, with a similar set of assumptions, Theorem 3.3 improves on these two issues, by allowing ρ not to vanish as n →∞ and by providing the exact upper-bound (9). Allowing non-vanishing radiuses is an attractive feature of our results that enables us to cover distribution shifts.

3.2 Regularized WDRO models

The analysis that we develop for the standard WDRO estimators is general enough to also cover the regularized versions presented in Section 2.2. We thus obtain the following Theorem 3.4 which is the first generalization guarantee for regularized WDRO. This theorem is very similar to Theorem 3.1 with still a couple of differences. First, the regularization leads to ambiguity sets defined in terms of W 2 ,τ (P , · ) , the regularized Wasserstein distance to the true distribution P , defined, for some regularization parameter τ > 0 , as where π σ appears in the definition of the regularized robust risk (6). Besides, the regularization allows us to avoid Assumptions 4 and 5 to show our generalization result.

Theorem 3.4. For σ = σ 0 ρ with σ 0 > 0 , ε = ε 0 ρ with ε 0 > 0 such that ε 0 /σ 2 0 is small enough depending on F , P , Ξ , there is an explicit constant ρ c depending only on F , P and Ξ such that for all δ ∈ (0 , 1) and n ≥ 1 , if then, there are τ = O ( ερ ) and ρ n = O (√ 1+log 1 /δ n ) such that, with probability at least 1 -δ ,

∀ f ∈ F , ̂ R ε ρ 2 ( f ) ≥ E ξ ∼ Q [ f ( ξ )] for all Q such that W 2 2 ,τ (P , Q) ≤ ρ ( ρ -ρ n ) . (10) Furthermore, when σ 0 and σ are small enough depending on P and Ξ , with probability 1 -δ ,

The first part of the theorem, (10), guarantees that the empirical robust risk is an upper-bound on the loss even with some perturbations of the true distribution. As in OT, the regularization added to the Wasserstein metric induces a bias that may prevent W 2 2 ,τ (P , P) from being null. As a result, the second part of the theorem involves a smoothed version of the true risk: the empirical robust risk provides an exact upper-bound the true expectation of a convolution of the loss with π σ .

A few additional comments are in order:

Our result prescribes the scaling of the regularization parameters: ε and σ should be taken proportional to ρ .

The critical radius ρ c has a slighlty more intricate definition, yet the same interpretation as in the standard WDRO case inRemark 3.2; see Section D.2.

The regularized OT distances do not suffer from the curse of dimensionality (Genevay et al., 2019). However this property does not directly carry over to regularized WDRO. Indeed, we cannot choose the same reference measure as in OT and we have to fix the measure π σ , introducing a bias. As a consequence, we have to extend the analysis of the previous section to obtain the claimed guarantees that avoid the curse of dimensionality.

3.3 Idea of the proofs

In this section, we present the main ideas of the proofs of Theorem 3.1, Theorem 3.3, and Theorem 3.4. The full proofs are detailed in appendix; we point to relevant sections along the discussion. First, we recall the duality results for WDRO that play a crucial role in our analysis. Second, we present a rough sketch of proofs that is common to both the standard and the regularized cases. Finally, we provide a refinement of our results that is a by-product of our analysis.

Duality in WDRO. Duality has been a central tool in both the theoretical analyses and computational schemes of WDRO from the onset (Shafieezadeh Abadeh et al., 2015; Esfahani and Kuhn, 2018). The expressions of the dual of WDRO problems for both the standard case (Gao and Kleywegt, 2016; Blanchet and Murthy, 2019) and the regularized case (Wang et al., 2023; Azizian et al., 2023) can be written with the following dual generator function φ defined as

where λ is the dual variable associated to the Wasserstein constraint in (3), (5) and (6). The effect of regularization appears here clearly as a smoothing of the supremum. Note also that this function depends on the conditional reference measures π σ ( ·| ξ ) but not on other probability distributions. Then, under some general assumptions (specified in Section 2.3 in appendix), the existing strong duality results yield that the (regularized) empirical robust risk writes

and, similarly, the (regularized) true robust risk writes

These expressions for the risks are the bedrock of our analysis.

Sketch of proof. In both the standard case and the regularized case, our proof is built on two main parts: the first part is to obtain a concentration bound on the dual problems that crucially relies on a lower bound of the dual multiplier; the second part then consists in establishing such a lower bound. All the bounds are valid with high probability, and we drop the dependency on the confidence level δ of the theorems for simplicity.

For the first part of the proof (Section B), we assume that there is a deterministic lower-bound λ > 0 on the optimal dual multiplier in (12) that holds with high-probability. As a consequence, we can

restrict the range of λ in (12) to obtain:

In the above, we used that the inner supremum, which is random, can be bounded by a deterministic and explicit quantity that we call ρ 2 n , i.e.,

Hence, we obtain an upper-bound on the robust risk w.r.t. the true distribution with radius ρ 2 -ρ 2 n . Moreover, we show that ρ 2 n = O (1 / ( λ √ n )) which highlights the need for a precise lower bound λ to control the decrease in radius.

The second part of the proof thus consists in showing that the dual variable is indeed bounded away from 0 , which means that the Wasserstein constraint is sufficiently active . We have to handle two cases differently:

when ρ is small, i.e., close to ρ n (Section C),

when ρ is large, i.e., close to the critical radius ρ c (Section D). Note that the additional Assumption 5 is required here: where we need to control the behaviors of f ∈ F close to their maxima (see (11) for ε = 0 and small λ ).

In both cases we obtain that λ scales as 1 /ρ for the respective ranges of admissible radiuses. As a consequence ρ 2 n is bounded by ρρ n with ρ n = O (1 / √ n ) and (14) becomes which leads to our main results.

Extension: upper and lower bounds on the empirical robust risk. The proof that we sketched above actually shows that R ε ρ ( ρ -ρ n ) ( f ) is a lower bound of ̂ R ε ρ 2 ( f ) . This proof technique also yields an upper bound by exchanging the roles of P and P n .

̂ Theorem 3.5. In the setting of either Theorem 3.1, Theorem 3.3 or Theorem 3.4 (with ε = 0 or ε > 0 ), with probability at least 1 -δ , it holds that

This result shows how two robust objectives w.r.t. P provide upper and lower bounds on the empirical robust risk, with only slight variations in the radius. Furthermore, when the number of data points n grows, both sides of the bound converge to the same quantity R ε ρ 2 ( f ) . Hence our generalization bounds of the form (15) are asymptotically tight.

As a final remark, we underline that the proofs of this theorem and of the previous ones rely on the cost being the squared Euclidean norm and the extension to more general cost functions is left as future work. In particular, the Laplace approximation of Section A.3 in the regularized case and the analysis of Section D.1 in the standard WDRO case would need further work to accomodate general cost functions.

4 Examples: parametric models

Our main theorems Theorems 3.1, 3.3 and 3.4 involve a general class F of loss functions. We explain in this section how to instantiate our results in the important class of parametric models. We then illustrate this setting with logistic regression and linear regression in Examples 4.1 and 4.2.

Let us consider the class of functions of the form

where Θ , the parameter space, is a subset of R p and Ξ , the sample space, is a subset of R d .

For instance, this covers the case of linear models of the form f ( θ, ξ ) = /lscript ( 〈 ξ, θ 〉 ) with /lscript a convex loss. This class of models is studied by Shafieezadeh-Abadeh et al. (2019); Chen and Paschalidis (2018) in a slightly different setting, where they obtain a closed form for the robust objective and then establish a generalization bound similar to (9).

Let us show how to instantiate our theorems in the case of (16).

If f is twice continuously differentiable on a neighborhood of Θ × Ξ with Θ and Ξ both compact, then Assumption 2 is immediately satisfied. Therefore, Theorem 3.4 can be readily applied and its generalization guarantee hold.

/negationslash

As for Assumption 4, it is equivalent to, for all θ ∈ Θ , P( ∇ ξ f ( θ, ξ ) = 0) > 0 . Thus disregarding the degenerate case of ∇ ξ f ( θ, ξ ) being null for P -almost every ξ (e.g., when the loss does not depend on ξ ), we are in the setting of Theorem 3.3.

Satisfying Assumption 5, needed for Theorem 3.1, requires some problem-dependent developments, see the examples below. Note though that the second item of Assumption 5 is implied by the parametric Morse-Bott property (Arbel and Mairal, 2022); see Section A.5.

We discuss linear and non-linear examples of this framework. In light of the above, we focus our discussion on Assumption 5. We first present the examples of linear models, Examples 4.1 and 4.2, where the latter assumption is satisfied. We then consider several examples of nonlinear models: kernel regression (Example 4.3), smooth neural networks (Example 4.4) and families of invertible mappings (Example 4.5). In Section H, we also provide numerical illustrations for linear models.

Example 4.1 (Logistic Regression) . For a training sample ( x, y ) ∈ R p × {-1 , +1 } , the logistic loss for a parameter θ ∈ R p is given by log ( 1 + e -y 〈 x,θ 〉 ) . It fits into our framework by defining f ( θ, ξ ) = log ( 1 + e 〈 ξ,θ 〉 ) with ξ playing the role of -y × x . We assume that Θ is a compact set that does not include the origin, and, for the sake of simplicity, we take Ξ as a closed Euclidean ball, i.e., Ξ = B (0 , r ) . We are going to show that Assumption 5 is satisfied, and, for this, we need the following elements. For any θ , the maximizer of f ( θ, · ) over Ξ = B (0 , r ) is reached at ξ ∗ := rθ ‖ θ ‖ . Besides, for any ξ ∈ Ξ , it holds that so that, since ‖ ξ ∗ ‖ = r , we have

We can now turn to the verification of Assumption 5.

Take some R > 0 and some ξ ∈ Ξ such that ‖ ξ -ξ ∗ ‖ ≥ R . Then, (17) yields

Since u ↦→ log(1 + e u ) is increasing, this yields that f ( θ, ξ ) -f ( θ, ξ ∗ ) is bounded away from 0 by a negative constant uniformly in θ . The first item of Assumption 5 is thus satisfied.

Fix θ ∈ Θ ; by Taylor expanding u ↦→ log(1 + e u ) around 〈 θ, ξ ∗ 〉 we get

where the big-O remainder is uniform over θ ∈ Θ . Using the first inequality in (18) , we get for ξ close enough to ξ ∗

This shows that the second item of Assumption 5 is satisfied locally around ξ ∗ . It can be made global by using the uniform Lipschitz-continuity of f , which introduces a term of the form L 6 ‖ ξ -ξ ∗ ‖ 3 .

Example 4.3 (Kernel Ridge Regression) . Using a kernel k : X × X → R with X compact and k smooth, for instance Gaussian or polynomial, we consider the following class of loss functions:

Example 4.2 (Linear Regression) . With samples of the form ξ = ( x, y ) ∈ R p × R and parameters θ ∈ R p , the loss is given by f ( θ, ξ ) = 1 2 ( 〈 θ, x 〉 -y ) 2 . Similarly to the previous example, we take Θ as a compact set of R d that does not include the origin and Ξ of the form B (0 , r ) × [ -r ′ , r ′ ] . The maximizers of f ( θ, · ) on Ξ are ξ ∗ 1 = ( rθ/ ‖ θ ‖ , -r ′ ) and ξ ∗ 2 = ( -rθ/ ‖ θ ‖ , r ′ ) . By symmetry, one can restrict to the case of ξ ∗ 1 and 〈 θ, x 〉 -y ≥ 0 ; the same rationale as above can then be applied.

where ξ = ( x, y ) , Ξ is some compact subset of X × R , θ = ( α 1 , . . . , α m , x 1 , . . . , x m ) , Θ = A m ×X m , m is a fixed integer, A is a compact subset of R m , X m can be any closed subset of X m and µ ≥ 0 is the regularization parameter. A typical choice for X m would be the datapoints of the training set. This class then fits into our framework of parametric models above. Finally, further information on the kernel would be needed to ensure that Assumption 5 is satisfied.

Example 4.4 (Smooth Neural Networks) . Denote by NN ( x, θ, σ ) a multi-linear perceptron that takes x as input, has weights θ and a smooth activation function σ , for instance the hyperbolic tangent or the Gaussian Error Linear Units (GELU). We choose /lscript (ˆ y, y ) a smooth loss function and we consider the loss f ( θ, ( x, y )) = /lscript ( NN ( x, θ, σ ) , y ) with θ ∈ Θ some compact set. Provided that the inputs ( x, y ) lie in a compact set Ξ , this class fits the parametric framework above. Note that we require σ to be smooth, further work would be required for non-smooth activation functions.

Example 4.5 (Family of diffeomorphisms) . Consider maps h : Ξ → Ξ and ( θ, ξ ) ∈ Θ × Ξ ↦→ g θ ( ξ ) ∈ Ξ and define the parametric loss f ( θ, ξ ) = h ( g θ ( ξ )) . Assume that these functions are twice differentiable, that h satisfies the second item of Assumption 5 and that, for every θ ∈ Θ , g θ has a inverse g -1 θ which is also continuously differentiable in a neighborhood of Θ × Ξ .

As before, this setting fits into the framework above. We now show that Assumption 5 is satisfied.

Since h is continuous, h satisfies the first item of Assumption 5. It is satisfied by F as well thanks to g -1 θ being Lipschitz-continuous in ξ uniformly in θ by compactness of Θ × Ξ .

Take C such that both g θ and g -1 θ are C -Lipschitz in ξ uniformly in θ . Since arg max f ( θ, · ) = g -1 θ (arg max h ) , it holds that min ζ /star ∈ arg max h ‖ g θ ( ξ ) -ζ /star ‖ = min ξ /star ∈ arg max f ( θ, · ) ‖ g θ ( ξ ) -g θ ( ξ /star ) ‖ which lies between C -1 min ξ /star ∈ arg max f ( θ, · ) ‖ ξ -ξ /star ‖ and C min ξ /star ∈ arg max f ( θ, · ) ‖ ξ -ξ /star ‖ . Combined with h satisfying the second item of Assumption 5, this shows that f satisfies this condition as well.

5 Conclusion and perspectives

In this work, we provide generalization guarantees for WDRO models that improve over existing literature in the following aspects: our results avoid the curse of dimensionality, provide exact upper bounds without spurious error terms, and allow for distribution shifts during testing. We obtained these bounds through the development of an original concentration result on the dual of WDRO. Our framework is general enough to cover regularized versions of the WDRO problem: they enjoy similar generalization guarantees as standard WDRO, with less restrictive assumptions.

Our work could be naturally extended in several ways. For instance, it might be possible to relax any of the assumptions (on the sample space, the sampling process, the Wasserstein metric, and the class of functions) at the expense of additional technical work. Moreover, the crucial role played by the radius of the Wasserstein ball calls for a principled and efficient procedure to select it.

Acknowledgments and Disclosure of Funding

This work has been supported by MIAI Grenoble Alpes (ANR-19-P3IA-0003).

References

An and R. Gao. Generalization bounds for (wasserstein) robust optimization. Advances in Neural Information Processing Systems , 34:10382-10392, 2021.

Arbel and J. Mairal. Non-Convex Bilevel Games with Critical Point Selection Maps. Advances in Neural Information Processing Systems , 36:1-34, 2022.

Arrigo, C. Ordoudis, J. Kazempour, Z. De Grève, J.-F. Toubeau, and F. Vallée. Wasserstein distributionally robust chance-constrained optimization for energy and reserve dispatch: An exact and physically-bounded formulation. European Journal of Operational Research , 296(1):304322, 2022.

Azizian, F. Iutzeler, and J. Malick. Regularization for Wasserstein distributionally robust optimization. ESAIM: COCV , 29:33, 2023.

Blanchet and Y. Kang. Semi-supervised learning based on distributionally robust optimization. Data Analysis and Applications 3: Computational, Classification, Financial, Statistical and Stochastic Methods , 5:1-33, 2020.

Blanchet and K. Murthy. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research , 44(2):565-600, 2019.

Blanchet and A. Shapiro. Statistical limit theorems in distributionally robust optimization. arXiv preprint arXiv:2303.14867 , 2023.

Blanchet, K. Murthy, and V. A. Nguyen. Statistical analysis of wasserstein distributionally robust estimators. In Tutorials in Operations Research: Emerging Optimization Methods and Modeling Techniques with Applications , pages 227-254. INFORMS, 2021.

Blanchet, K. Murthy, and N. Si. Confidence regions in wasserstein distributionally robust estimation. Biometrika , 109(2):295-315, 2022a.

Blanchet, K. Murthy, and F. Zhang. Optimal transport-based distributionally robust optimization: Structural properties and iterative schemes. Mathematics of Operations Research , 47(2):15001529, 2022b.

Boucheron, G. Lugosi, and P. Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence . Oxford University Press, 2013.

Boumal. An introduction to optimization on smooth manifolds . Cambridge University Press, 2023.

Chen and I. C. Paschalidis. A robust learning approach for regression models based on distributionally robust optimization. Journal of Machine Learning Research , 19(13), 2018.

Dapogny, F. Iutzeler, A. Meda, and B. Thibert. Entropy-regularized wasserstein distributionally robust shape and topology optimization. Structural and Multidisciplinary Optimization , 66(3): 42, 2023.

Delage and Y. Ye. Distributionally Robust Optimization Under Moment Uncertainty with Application to Data-Driven Problems. Operations Research , 58:595-612, 2010.

M. Esfahani and D. Kuhn. Data-driven distributionally robust optimization using the wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Programming , 171 (1):115-166, 2018.

Feydy, T. Séjourné, F.-X. Vialard, S.-i. Amari, A. Trouvé, and G. Peyré. Interpolating between optimal transport and MMD using Sinkhorn divergences. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 2681-2690. PMLR, 2019.

Fournier and A. Guillin. On the rate of convergence in wasserstein distance of the empirical measure. Probability Theory and Related Fields , 162(3):707-738, 2015.

Gao. Finite-sample guarantees for wasserstein distributionally robust optimization: Breaking the curse of dimensionality. Operations Research , 2022.

Gao and A. J. Kleywegt. Distributionally robust stochastic optimization with wasserstein distance. Mathematics of Operations Research , 2016.

Genevay, M. Cuturi, G. Peyré, and F. Bach. Stochastic Optimization for Large-scale Optimal Transport. In NIPS 2016 - Thirtieth Annual Conference on Neural Information Processing System , Proc. NIPS 2016, 2016.

Genevay, L. Chizat, F. Bach, M. Cuturi, and G. Peyré. Sample complexity of sinkhorn divergences. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1574-1583. PMLR, 2019.

Goh and M. Sim. Distributionally Robust Optimization and Its Tractable Approximations. Operations Research , 58:902-917, 2010.

Kuhn, P. M. Esfahani, V. A. Nguyen, and S. Shafieezadeh-Abadeh. Wasserstein distributionally robust optimization: Theory and applications in machine learning. In Operations Research & Management Science in the Age of Analytics . INFORMS, 2019.

Kwon, W. Kim, J.-H. Won, and M. C. Paik. Principled learning method for wasserstein distributionally robust optimization with local perturbations. In International Conference on Machine Learning , pages 5567-5576. PMLR, 2020.

Laguel, J. Malick, and Z. Harchaoui. First-order optimization for superquantile-based supervised learning. In 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP) , pages 1-6. IEEE, 2020.

Lee and M. Raginsky. Minimax statistical learning with wasserstein distances. Advances in Neural Information Processing Systems , 31, 2018.

M. Lee. Introduction to Riemannian manifolds , volume 2. Springer, 2018.

Levy, Y. Carmon, J. C. Duchi, and A. Sidford. Large-scale methods for distributionally robust optimization. Advances in Neural Information Processing Systems , 33, 2020.

Li, C. Chen, and A. M.-C. So. Fast Epigraphical Projection-based Incremental Algorithms for Wasserstein Distributionally Robust Support Vector Machine. In Advances in Neural Information Processing Systems , volume 33, pages 4029-4039. Curran Associates, Inc., 2020.

F.-P. Paty and M. Cuturi. Regularized Optimal Transport is Ground Cost Adversarial. In ICML , 2020.

Peyré and M. Cuturi. Computational optimal transport: With applications to data science. Foundations and Trends® in Machine Learning , 11(5-6):355-607, 2019.

Piat, J. Fadili, F. Jurie, and S. da Veiga. Regularized robust optimization with application to robust learning. 2022.

T. Rockafellar and R. J.-B. Wets. Variational Analysis . Grundlehren Der Mathematischen Wissenschaften. Springer-Verlag, 1998.

Rudin. Real and Complex Analysis . McGraw-Hill, 1987.

Shafieezadeh Abadeh, P. M. Mohajerin Esfahani, and D. Kuhn. Distributionally Robust Logistic Regression. In Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

Shafieezadeh-Abadeh, D. Kuhn, and P. M. Esfahani. Regularization via Mass Transportation. Journal of Machine Learning Research , 20:1-68, 2019.

Sinha, H. Namkoong, and J. Duchi. Certifying some distributional robustness with principled adversarial training. In International Conference on Learning Representations , 2018.

Song, N. He, L. Ding, and C. Zhao. Provably convergent policy optimization via metric-aware trust region methods. arXiv preprint arXiv:2306.14133 , 2023.

Staib and S. Jegelka. Distributionally robust optimization and generalization in kernel methods. Advances in Neural Information Processing Systems , 32, 2019.

J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

Wang and Y. Xie. A data-driven approach to robust hypothesis testing using sinkhorn uncertainty sets. In 2022 IEEE International Symposium on Information Theory (ISIT) , pages 3315-3320. IEEE, 2022.

Wang, R. Moore, Y. Xie, and R. Kamaleswaran. Improving sepsis prediction model generalization with optimal transport. In Machine Learning for Health , pages 474-488. PMLR, 2022.

Wang, R. Gao, and Y. Xie. Sinkhorn distributionally robust optimization. arXiv preprint arXiv:2109.11926 , 2023.

Zeng and H. Lam. Generalization bounds with minimal dependency on hypothesis class via distributionally robust optimization. Advances in Neural Information Processing Systems , 35: 27576-27590, 2022.

Zhao and Y. Guan. Data-driven risk-averse stochastic optimization with wasserstein metric. Operations Research Letters , 46(2):262-267, 2018.

J.-J. Zhu, W. Jitkrittum, M. Diehl, and B. Schölkopf. Kernel distributionally robust optimization: Generalized duality theorem and stochastic approximation. In International Conference on Artificial Intelligence and Statistics , pages 280-288. PMLR, 2021a.

J.-J. Zhu, C. Kouridi, Y . Nemmour, and B. Schölkopf. Adversarially robust kernel smoothing. arXiv preprint arXiv:2102.08474 , 2021b.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides theoretical results characterizing the generalization capabilities of methods based on Wasserstein distributionally robust approaches. In particular, the results presented extend the conditions under which the performance guarantees are not affected by the curse of dimensionality and are applicable for general classes of models.\n\nSTRENGTHS:\nThe paper shows that the usage of Wasserstein radius of the order 1/sqrt(n) can provide generalization bounds in situations more general than those considered in existing works (linear models). In addition, the results presented also cover regularized versions of WDRO. The more general results are obtained using a novel type of proof based on a concentration bound for the dual problem, which is of independent interest.\n\nWEAKNESSES:\nThe paper contribution with respect to the state of the art needs to be better described. In particular, the extension to non-linear models of the scaling 1/sqrt(n). \n\nThe problematic dimension-dependent scaling arises in Wasserstein methods while other techniques based on robust risk minimization have been shown to provide performance guarantees with the scaling 1/sqrt(n). It would be good if the authors describe this fact and the related work. \n\nIf I am not mistaken, examples 3.6 and 3.7 correspond to cases for which the right scaling was already proven in previous works. In order to better assess the paper's contribution, it would be good if the authors discuss interesting examples for which the paper provides the right scaling while existing results cannot.\n\nQUESTIONS:\nWould it be possible to include numerical results describing the theoretical results presented?. The choice of the radius in practice is often problematic in Wasserstein methods. The theoretical results provide the scaling of such radius but not a concrete recipe to choose it. It would be useful to explore choices of such radius with the right scaling that result in small error and provide performance guarantees.\n\nLIMITATIONS:\nThe paper adequately describes the limitations of the methods proposed, mostly in terms of the specific assumptions needed for the results to hold.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents generalization bound for Wasserstein DRO and entropic regularized Wasserstein DRO (or called Sinkhorn DRO in Wang et al.) formulations. Those generalization bounds do not suffer from the curse of dimensionality. The theoreical analysis is also supported by two examples in Section 3.4.\n\nSTRENGTHS:\n- The theoretical analysis is interesting from two aspects. First, the authors reveal that the radius selection of WDRO to make the empirical robust loss  dominate the true loss does not suffer from the curse of dimensionality. The analysis follows different techniques from existing literature such as Gao et al, Blanchet et. al, etc. Second, the technique is general enough so that it also applies to entropic regularized Wasserstein DRO (or called Sinkhorn DRO in Wang et al.) formulations. This is the first work that investigates the statistical properties of such formulations.\n- The authors also present two examples in machine learning to demonstrate the technique assumption holds and the proposed theoretical analysis applies.\n\nWEAKNESSES:\n- The writing of this paper could be potentially improved:\n 1. There should be a comma in Eq.(1), or equation between line 83-84, or Eq. (4), or equation between line 194-195, or equation between line 301-302. \n 2. There should be a period in Eq. (10).\n 3. The contribution and related work part in the introduction section should be separated.\n 4. It would be a little bit confusing to first introduce KL-divergence regularized WDRO risk in Eq.(5-6) and then introduce it corresponds to the Sinkhorn ambiguity set in line 194-195. The authors should put them together in Section 2.2\n 5. The notation could be potentially improved. For example, in Eq. (7) the authors use $\\hat{\\mathcal{R}}$ to refer to the risk based on empirical distribution $P_n$. I would suggest replace the notation $P_n$ with $\\hat{P}_n$ for consistency. Further, in Eq. (7) I think the authors are meaning $\\rho$ should at least scale in the order of $\\sqrt{(1+\\log(1/\\delta))/n}$, then why not write $\\Omega(\\sqrt{(1+\\log(1/\\delta))/n})\\le \\rho$ instead of $O(\\sqrt{(1+\\log(1/\\delta))/n})\\le \\rho$? The same applies for equation between line 199-200.\n\n- It is great that the authors present statistical analysis for entropic regularized Wasserstein DRO. I would suggest the authors add some explanation or numerical example to demonstrate the benefit of introducing entropic regularization. Will it bring extra benefits than standard WDRO?\n\n- The analysis is limited to quadratic cost function, which could be restrictive. From my own trial and reading, I think the major difficulty for generalization is that, it is difficult to apply Laplace approximation technique for general p-th power of norm function. In other words, it is difficult to obtain the p-th power of norm counterpart of Lemma A.3 and Lemma G.1.If so, I suggest the authors add explanation for the difficulty of extension.\n\n- Some literature is missing. For example, readers may wonder why consider adding entropic regularization to WDRO problem and what is the applications? I suggest the authors make the following revisions:\n   1. update reference [J. Wang, R. Gao, and Y. Xie. Sinkhorn distributionally robust optimization. arXiv preprint arXiv:2109.11926, 2021] as [J. Wang, R. Gao, and Y. Xie. Sinkhorn distributionally robust optimization. arXiv preprint arXiv:2109.11926, 2023]. In the updated version, the authors demonstrate that people can find $\\delta$-optimal solution to general entropic regularization WDRO problem with complexity $\\tilde{O}(1/\\delta^2)$. So one major benefit of adding entropic regularization is the computational tractability;\n  2. add several application papers brought by entropic regularization WDRO in literature review:\n     (i) Dapogny, Charles, et al. \"Entropy-regularized Wasserstein distributionally robust shape and topology optimization.\" Structural and Multidisciplinary Optimization 66.3 (2023): 42.\n     (ii) Song, Jun, et al. \"Provably Convergent Policy Optimization via Metric-aware Trust Region Methods.\" arXiv preprint arXiv:2306.14133 (2023).\n     (iii) Wang, Jie, and Yao Xie. \"A data-driven approach to robust hypothesis testing using sinkhorn uncertainty sets.\" 2022 IEEE International Symposium on Information Theory (ISIT). IEEE, 2022.\n     (iv) Wang, Jie, et al. \"Improving sepsis prediction model generalization with optimal transport.\" Machine Learning for Health. PMLR, 2022.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This work proves generalization guarantees for Wasserstein DRO models that only require the radius of order $O(n^{-1/2})$ under mild assumptions for general classes of models. This provides concentration results that do not suffer from the curse of dimensionality.

STRENGTHS:
The theoretical contribution is the main strength. The empirical concentration of Wasserstein distance suffers from the curse of dimensionality, and this paper is able to prove the results (under some assumptions) that do not have this curse of dimensionality issue and provide statistical guarantees on the performance of WDRO solutions.

WEAKNESSES:
There is no significant weakness in this paper. Nevertheless, I think adding some discussion or examples for which the assumptions and thus the results in this paper do not hold can be beneficial; it can show failure cases and may also motivate future directions for the extension.

QUESTIONS:
It might be better to split Section 3 into two shorter sections for better readability. And the discussion may also be extended by adding examples of failure cases with potential methods of relaxation.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides generalization guarantees of Wasserstein DRO for a general class of functions, in which the radius scales as $1/\\sqrt{n}$ and does not suffer from the curse of dimensionality. Moreover, these guarantees hold for any distribution in the neigbourhood of the true distribution, so that they still apply when the distribution shifts at testing time. The results in this paper hold for both constrained and regularized version of Wasserstein DRO. The authors also provide a proof sketch that explains the main ideas and techniques used in the proof, and apply their results to logistic and linear regression.\n\nSTRENGTHS:\n1. This paper provides novel generalization guarantees such that the robustness radius does not suffer from the curse of dimensionality. To the best of my knowledge, the results in this paper are novel and make a non-trivial contribution to the DRO community. Moreover, the authors consider the regularized version of Wasserstein DRO and provide similar guarantees as well.\n\n2. Most parts of the paper are well-written. The necessary backgrounds are clearly explained, and theorems are accompanied with detailed explanations of related definitions and concepts.\n\nWEAKNESSES:\nIn Section 3.4, the authors considers logistic and linear regression as applications of their theorems. It would be better if more complicated and popular parametric models can be included in this section to justify the main assumptions.\n\nQUESTIONS:\n1. How does the setting considered in this paper compared with other works? Could you give a brief and high-level discussion of why the generalization guarantee is dimension-independent in your setting?\n\n2. Is it possible to obtain similar generalization guarantee for DRO with $\\phi$-divergence?\n\nLIMITATIONS:\nThis paper does not have potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}