Generalization Bounds for (Wasserstein) Robust Optimization

1 Introduction . Consider the following stochastic optimization problem of seeking a function f : X ! R from the hypothesis space F so as to minimize the expected loss : min f2F Ltrue ( f ) : = EX⇠Ptrue [ f ( X ) ] . Here the data X follows some underlying distribution Ptrue on X . In practice , Ptrue is often replaced with its empirical estimate Pn = 1n Pn i=1 x̂i , where is the Dirac measure : min f2F { Lermn ( f ) : = EX⇠Pn [ f ( X ) ] } . The discrepancy between Ptrue and Pn may be resulted from ( i ) The sampling error of Pn when data are independently drew from Ptrue ; ( ii ) Non-i.i.d . data , e.g. , xi ’ s are samples from a Markov chain with a stationary distribution Ptrue ; ( iii ) The deploying environment is ( adversarially ) different from the data-collecting environment . Such discrepancy largely contributes to the poor generalization behavior of empirical risk minimization . To tackle this issue , robust optimization [ 4 , 37 , 38 ] studies min f2F ⇢ Lron , p ( f ; % ) : = max { xi } ni=12X n 1 n nP i=1 f ( xi ) : 1 n nP i=1 kxi x̂ikp 1 p % o . ( RO ) Here p 2 [ 1,1 ] ; the uncertainty set , denoted by Xp ( % ) , describes the data perturbations to hedge against ; and the radius % > 0 specifies the level of robustness . In a similar spirit , Wasserstein robust optimization [ 15 , 7 , 19 ] studies min f2F n Lwon , p ( f ; % ) : = maxP Ex⇠P [ f ( x ) ] : Wp ( P , Pn ) % o . ( WO ) Here the distributional uncertainty set , denoted by Pp ( % ) , consists of all probability distributions that are close to Pn in p-Wasserstein distance . When p = 1 , ( WO ) and ( RO ) coincide . Both problems have received increasing attentions in machine learning ; see [ 25 ] for a survey . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . When data is generated stochastically , such as i.i.d . or Markovian ( see ( i ) ( ii ) above ) , we would like to choose the radius % = % n adaptive to the sample size , so that the robust loss can balance between performance and robustness . More precisely , % n should be large enough so that the robust loss dominates the true loss with high probability , but not be too large to produce an inferior solution . When the testing environment is adversarially different from the training environment ( see ( iii ) above ) , consider the adversarial loss minimization min f2F ⇢ Ladvp ( f ; % ) : = maxP2Pp ( % ) Ex⇠P [ f ( x ) ] : Wp ( P , Ptrue ) % , ( Adv ) where the radius % captures the difference between the training and testing environments . When Ptrue is not available , a surrogate problem ( WO ) based on Pn is often considered . In this case , % is often fixed and we hope that the robust solution yielding from ( WO ) generalizes well . The goal of this paper is to provide a comprehensive analysis of the generalization capability of the above robust learning paradigms . Specifically , in the stochastic setting , under proper choice of the radius % n , we derive finite-sample guarantees for ( RO ) and ( WO ) . In the adversarial setting , we derive generalization bounds for ( Adv ) with a small fixed radius % . Related Literature Stochastic setting . The most related work to ours is [ 17 ] , but we differ in several important ways . First , [ 17 ] studies only ( WO ) while we also consider ( RO ) . Second , [ 17 ] studies i.i.d . data while we also consider non-i.i.d . by exploiting a different notion of transport inequality . Third , even in the i.i.d . setting , [ 17 ] studies only smooth losses and Wasserstein order p = 1 , 2 . We consider non-smooth losses and p 2 [ 1 , 2 ] , which includes important cases like |✓ > x y|p . Moreover , for p = 1 , we tighten the order of the remainder ; for p = 2 , we obtain a cleaner expression using a different set of assumptions . All these extensions require in-depth analysis on worst-case scenarios . For ( WO ) with i.i.d . data , [ 15 ] originally proposed to choose the radius % n such that Ptrue is contained in the uncertainty set with high probability , but the resulting performance bound is overly conservative and suffers from the curse of dimensionality . This issue has been mitigated asymptotically for smooth loss functions [ 6 , 8 , 33 ] , and non-asymptotically for linear models [ 32 , 12 ] and for smooth loss functions [ 17 ] . As mentioned above , our results apply to non-i.i.d . data and non-smooth losses . For ( RO ) in the stochastic setting , its generalization properties were first studied in the pioneer work [ 39 ] , which established algorithmic generalization generalization bounds for various learning tasks with i.i.d . and Markovian data . Nonetheless , many bounds developed therein suffer from the curse of dimensionality . This framework is then extended to metric learning [ 3 ] and ensemble models [ 41 ] . Our results circumvent the curse of dimensionality of the bounds developed in [ 39 ] . Adversarial setting . For p 2 [ 1,1 ) , generalization properties of ( WO ) with fixed radius was considered first in [ 27 ] . The main difference is that we consider different magnitudes of data perturbation . We mainly focus on adversarial robustness where the radius is often a tiny number , and our results in Section 5 suggest that the generalization gap is small when the radius is small . While their motivation was from domain adaptation , and their bound is mostly useful when the Wasserstein radius is not extremely small , which suggests that the generalization gap becomes smaller as the radius grows to infinity ( see their Remark 4 ) . The bounds developed in [ 34 , 30 ] does not depend on the Wasserstein radius , while our result indicates that when the radius is small , the remainder actually demonstrates its linear scaling and therefore tighter . For p = 1 , [ 24 , 40 , 1 ] focus on deriving generalization bounds for specific loss function classes such as linear hypotheses and neural networks . Our results apply to generic function classes , and most importantly , the bound suggests the the generalization gap is controlled by not only the complexity of the loss function class , but also the complexity of the gradient norm functions of the loss functions . The intuition is that the gradient norm controls the robustness of the model subject to adversarial perturbations . As far as we know , the bound in such form is new in the literature . There are also some earlier works focusing on investigating the fundamental limit of adversarial learning and its difference from empirical risk minimization , such as [ 31 , 35 , 10 , 11 ] . They consider somewhat different settings than ours , which usually involves a stylized data generating model that facilitates theoretical findings . Regularization . Part of our analysis is based on the connection between robustness and regularization . The connection between ( RO ) ( WO ) and norm regularization was established for various machine learning tasks [ 38 , 37 , 5 , 6 , 12 ] , and the connection between ( WO ) and Lipschitz/gradient regularization has been established for smooth loss functions [ 32 , 34 , 16 , 13 , 8 , 2 ] and non-smooth loss functions [ 18 ] . Our results improve the sandwich bounds in [ 18 ] for p = 1 , and generalizes the results therein to piecewise Hölder smooth loss functions for p 2 ( 1,1 ] . Finally , there are also works connecting regularization and distributional robustness under other choices of distance [ 26 , 21 , 14 , 23 ] . 2 Preliminaries . In this section , we prepare several technical tools that are useful in deriving our main results . Notations For p 2 [ 1,1 ] , we denote by q its Hölder conjugate number , i.e . 1p + 1 q = 1 . Let P ( X ) be the set of Borel probability distributions on X = ( Rd , k·k ) , and let Pp ( X ) = { P 2 P ( X ) : EP [ kxkp ] < 1 } . For a given probability measure Q , denote by ( LpQ ( X ) , k·kp , Q ) the set of Lp functions on X with respect to Q . Unless stated otherwise , we will not assume the samples are i.i.d . Instead , we use Sn = { x̂i } ni=1 to represent a sample of size n , and denote by µSn the sampling distribution and use ESn to indicate the expectation with respect to the µSn , and Ltrue ( f ) = ESn [ 1n Pn i=1 f ( x̂ i ) ] . For a fixed set of n points , we denote Pn as the empirical uniform distribution on these n points . For two real numbers , a and b , we denote a ^ b = min ( a , b ) , and a _ b = max ( a , b ) . Transport inequalities In the stochastic setting , we will work with distributions that satisfy the following condition . Let p 2 [ 1,1 ) and ⌧ > 0 . A distribution Q 2 Pp ( X ) satisfies a transport inequality , denoted as Tp ( ⌧ ) , if for all P 2 Pp ( X ) it holds that W2p ( P , Q ) ⌧ Z X log ( dP/dQ ) dP , where dP/dQ denotes the Radon-Nikodym derivative . The integral on the right-hand side of the inequality above represents the relative entropy between P and Q . The transportation-information inequality Tp ( ⌧ ) covers a variety of useful classes of distributions . For example , if X is bounded by B , then any distribution on X satisfies T1 ( 8B2 ) . Gaussian distributions with covariance ⌃ satisfies T2 ( 2 max ( ⌃ ) ) . Interested readers are referred to [ 22 ] for a recent survey . Variation Regularization Following [ 18 ] , we define the local and global slope of a continuous function f : X ! R at x 2 X , denoted by | @ f | ( x ) and G ( f ) ( x ) respectively , as | @ f | ( x ) = lim sup kx̃ xk ! 0 ( f ( x̃ ) f ( x ) ) + kx̃ xk , G ( f ) ( x ) = supx̃2X ( f ( x̃ ) f ( x ) ) + kx̃ xk . When f is differentiable at x , we have | @ f | ( x ) = krf ( x ) k⇤ . When f is L-Lipschitz , we have G ( f ) ( x ) L for all x 2 X and kG ( f ) k1 , Ptrue = kfkLip . For a function f , denote by kfkLip its Lipschitz norm whenever it exists , and by k| @ f |kq the Lq ( Ptrue ) -norm of its slope function . We define variation regularization as min f2F Lvrn , q ( f ; % ) : = ⇢ 1 n Pn i=1 f ( x̂ i ) + % k| @ f |kq , Pn , q < 1 , 1 n Pn i=1 f ( x̂ i ) + % kfkLip , q = 1 . ( VR ) We introduce the following class of functions consisting of slope functions of loss functions . @ Fq : = | @ f |q : f 2 F , Nq : = n | @ f |q k| @ f |kqq : f 2 F o . ( 1 ) Throughout we utilize the following assumptions , which covers most loss functions that are amenable to optimize efficiently using first-order methods . Assumption 1 . Every f 2 F is L-Lipschitz . Assumption 2 . Every f 2 F is piecewise differentiable . There exists h , ↵ > 0 such that for every f 2 F and every x̃ , x in the same piece of f , it holds that krf ( x̃ ) rf ( x ) k⇤ hkx̃ xk↵ .

Summary: This paper derives finite-sample generalization bounds for robust optimization and Wasserstein distributionally robust optimization. The authors consider a broad class of piecewise Hölder smooth loss functions, under both stochastic setting—i.i.d. or weakly dependent data—and adversarial environment, assuming that the underlying data-generating distribution satisfies transportation-information inequalities. The proofs are built on a general connection between robustness and variation regularization (including Lipschitz and gradient regularization, among others) and new local Rademacher complexity results for variation regularization.