Generalization Bounds for (Wasserstein) Robust Optimization

1 Introduction . Consider the following stochastic optimization problem of seeking a function f : X ! R from the hypothesis space F so as to minimize the expected loss : min f2F Ltrue ( f ) : = EX⇠Ptrue [ f ( X ) ] . Here the data X follows some underlying distribution Ptrue on X . In practice , Ptrue is often replaced with its empirical estimate Pn = 1n Pn i=1 x̂i , where is the Dirac measure : min f2F { Lermn ( f ) : = EX⇠Pn [ f ( X ) ] } . The discrepancy between Ptrue and Pn may be resulted from ( i ) The sampling error of Pn when data are independently drew from Ptrue ; ( ii ) Non-i.i.d . data , e.g. , xi ’ s are samples from a Markov chain with a stationary distribution Ptrue ; ( iii ) The deploying environment is ( adversarially ) different from the data-collecting environment . Such discrepancy largely contributes to the poor generalization behavior of empirical risk minimization . To tackle this issue , robust optimization [ 4 , 37 , 38 ] studies min f2F ⇢ Lron , p ( f ; % ) : = max { xi } ni=12X n 1 n nP i=1 f ( xi ) : 1 n nP i=1 kxi x̂ikp 1 p % o . ( RO ) Here p 2 [ 1,1 ] ; the uncertainty set , denoted by Xp ( % ) , describes the data perturbations to hedge against ; and the radius % > 0 specifies the level of robustness . In a similar spirit , Wasserstein robust optimization [ 15 , 7 , 19 ] studies min f2F n Lwon , p ( f ; % ) : = maxP Ex⇠P [ f ( x ) ] : Wp ( P , Pn ) % o . ( WO ) Here the distributional uncertainty set , denoted by Pp ( % ) , consists of all probability distributions that are close to Pn in p-Wasserstein distance . When p = 1 , ( WO ) and ( RO ) coincide . Both problems have received increasing attentions in machine learning ; see [ 25 ] for a survey . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . When data is generated stochastically , such as i.i.d . or Markovian ( see ( i ) ( ii ) above ) , we would like to choose the radius % = % n adaptive to the sample size , so that the robust loss can balance between performance and robustness . More precisely , % n should be large enough so that the robust loss dominates the true loss with high probability , but not be too large to produce an inferior solution . When the testing environment is adversarially different from the training environment ( see ( iii ) above ) , consider the adversarial loss minimization min f2F ⇢ Ladvp ( f ; % ) : = maxP2Pp ( % ) Ex⇠P [ f ( x ) ] : Wp ( P , Ptrue ) % , ( Adv ) where the radius % captures the difference between the training and testing environments . When Ptrue is not available , a surrogate problem ( WO ) based on Pn is often considered . In this case , % is often fixed and we hope that the robust solution yielding from ( WO ) generalizes well . The goal of this paper is to provide a comprehensive analysis of the generalization capability of the above robust learning paradigms . Specifically , in the stochastic setting , under proper choice of the radius % n , we derive finite-sample guarantees for ( RO ) and ( WO ) . In the adversarial setting , we derive generalization bounds for ( Adv ) with a small fixed radius % . Related Literature Stochastic setting . The most related work to ours is [ 17 ] , but we differ in several important ways . First , [ 17 ] studies only ( WO ) while we also consider ( RO ) . Second , [ 17 ] studies i.i.d . data while we also consider non-i.i.d . by exploiting a different notion of transport inequality . Third , even in the i.i.d . setting , [ 17 ] studies only smooth losses and Wasserstein order p = 1 , 2 . We consider non-smooth losses and p 2 [ 1 , 2 ] , which includes important cases like |✓ > x y|p . Moreover , for p = 1 , we tighten the order of the remainder ; for p = 2 , we obtain a cleaner expression using a different set of assumptions . All these extensions require in-depth analysis on worst-case scenarios . For ( WO ) with i.i.d . data , [ 15 ] originally proposed to choose the radius % n such that Ptrue is contained in the uncertainty set with high probability , but the resulting performance bound is overly conservative and suffers from the curse of dimensionality . This issue has been mitigated asymptotically for smooth loss functions [ 6 , 8 , 33 ] , and non-asymptotically for linear models [ 32 , 12 ] and for smooth loss functions [ 17 ] . As mentioned above , our results apply to non-i.i.d . data and non-smooth losses . For ( RO ) in the stochastic setting , its generalization properties were first studied in the pioneer work [ 39 ] , which established algorithmic generalization generalization bounds for various learning tasks with i.i.d . and Markovian data . Nonetheless , many bounds developed therein suffer from the curse of dimensionality . This framework is then extended to metric learning [ 3 ] and ensemble models [ 41 ] . Our results circumvent the curse of dimensionality of the bounds developed in [ 39 ] . Adversarial setting . For p 2 [ 1,1 ) , generalization properties of ( WO ) with fixed radius was considered first in [ 27 ] . The main difference is that we consider different magnitudes of data perturbation . We mainly focus on adversarial robustness where the radius is often a tiny number , and our results in Section 5 suggest that the generalization gap is small when the radius is small . While their motivation was from domain adaptation , and their bound is mostly useful when the Wasserstein radius is not extremely small , which suggests that the generalization gap becomes smaller as the radius grows to infinity ( see their Remark 4 ) . The bounds developed in [ 34 , 30 ] does not depend on the Wasserstein radius , while our result indicates that when the radius is small , the remainder actually demonstrates its linear scaling and therefore tighter . For p = 1 , [ 24 , 40 , 1 ] focus on deriving generalization bounds for specific loss function classes such as linear hypotheses and neural networks . Our results apply to generic function classes , and most importantly , the bound suggests the the generalization gap is controlled by not only the complexity of the loss function class , but also the complexity of the gradient norm functions of the loss functions . The intuition is that the gradient norm controls the robustness of the model subject to adversarial perturbations . As far as we know , the bound in such form is new in the literature . There are also some earlier works focusing on investigating the fundamental limit of adversarial learning and its difference from empirical risk minimization , such as [ 31 , 35 , 10 , 11 ] . They consider somewhat different settings than ours , which usually involves a stylized data generating model that facilitates theoretical findings . Regularization . Part of our analysis is based on the connection between robustness and regularization . The connection between ( RO ) ( WO ) and norm regularization was established for various machine learning tasks [ 38 , 37 , 5 , 6 , 12 ] , and the connection between ( WO ) and Lipschitz/gradient regularization has been established for smooth loss functions [ 32 , 34 , 16 , 13 , 8 , 2 ] and non-smooth loss functions [ 18 ] . Our results improve the sandwich bounds in [ 18 ] for p = 1 , and generalizes the results therein to piecewise Hölder smooth loss functions for p 2 ( 1,1 ] . Finally , there are also works connecting regularization and distributional robustness under other choices of distance [ 26 , 21 , 14 , 23 ] . 2 Preliminaries . In this section , we prepare several technical tools that are useful in deriving our main results . Notations For p 2 [ 1,1 ] , we denote by q its Hölder conjugate number , i.e . 1p + 1 q = 1 . Let P ( X ) be the set of Borel probability distributions on X = ( Rd , k·k ) , and let Pp ( X ) = { P 2 P ( X ) : EP [ kxkp ] < 1 } . For a given probability measure Q , denote by ( LpQ ( X ) , k·kp , Q ) the set of Lp functions on X with respect to Q . Unless stated otherwise , we will not assume the samples are i.i.d . Instead , we use Sn = { x̂i } ni=1 to represent a sample of size n , and denote by µSn the sampling distribution and use ESn to indicate the expectation with respect to the µSn , and Ltrue ( f ) = ESn [ 1n Pn i=1 f ( x̂ i ) ] . For a fixed set of n points , we denote Pn as the empirical uniform distribution on these n points . For two real numbers , a and b , we denote a ^ b = min ( a , b ) , and a _ b = max ( a , b ) . Transport inequalities In the stochastic setting , we will work with distributions that satisfy the following condition . Let p 2 [ 1,1 ) and ⌧ > 0 . A distribution Q 2 Pp ( X ) satisfies a transport inequality , denoted as Tp ( ⌧ ) , if for all P 2 Pp ( X ) it holds that W2p ( P , Q ) ⌧ Z X log ( dP/dQ ) dP , where dP/dQ denotes the Radon-Nikodym derivative . The integral on the right-hand side of the inequality above represents the relative entropy between P and Q . The transportation-information inequality Tp ( ⌧ ) covers a variety of useful classes of distributions . For example , if X is bounded by B , then any distribution on X satisfies T1 ( 8B2 ) . Gaussian distributions with covariance ⌃ satisfies T2 ( 2 max ( ⌃ ) ) . Interested readers are referred to [ 22 ] for a recent survey . Variation Regularization Following [ 18 ] , we define the local and global slope of a continuous function f : X ! R at x 2 X , denoted by | @ f | ( x ) and G ( f ) ( x ) respectively , as | @ f | ( x ) = lim sup kx̃ xk ! 0 ( f ( x̃ ) f ( x ) ) + kx̃ xk , G ( f ) ( x ) = supx̃2X ( f ( x̃ ) f ( x ) ) + kx̃ xk . When f is differentiable at x , we have | @ f | ( x ) = krf ( x ) k⇤ . When f is L-Lipschitz , we have G ( f ) ( x ) L for all x 2 X and kG ( f ) k1 , Ptrue = kfkLip . For a function f , denote by kfkLip its Lipschitz norm whenever it exists , and by k| @ f |kq the Lq ( Ptrue ) -norm of its slope function . We define variation regularization as min f2F Lvrn , q ( f ; % ) : = ⇢ 1 n Pn i=1 f ( x̂ i ) + % k| @ f |kq , Pn , q < 1 , 1 n Pn i=1 f ( x̂ i ) + % kfkLip , q = 1 . ( VR ) We introduce the following class of functions consisting of slope functions of loss functions . @ Fq : = | @ f |q : f 2 F , Nq : = n | @ f |q k| @ f |kqq : f 2 F o . ( 1 ) Throughout we utilize the following assumptions , which covers most loss functions that are amenable to optimize efficiently using first-order methods . Assumption 1 . Every f 2 F is L-Lipschitz . Assumption 2 . Every f 2 F is piecewise differentiable . There exists h , ↵ > 0 such that for every f 2 F and every x̃ , x in the same piece of f , it holds that krf ( x̃ ) rf ( x ) k⇤ hkx̃ xk↵ .

Summary: This paper aims to theoretically analyze (and compare) a number of different robust learning paradigms. In particular, it has been shown that three different notions of robustness, namely 1) robust optimization 2) Wasserstein distributionally robust optimization, and 3) variation regularization, would result in close loss functions (based on some non-asymptotic bounds) whenever the strength of attacks becomes infinitesimally small. While connections between such methods have been conjectured (or even analyzed) in some recent works, I still found the results in this paper interesting.  Also, author(s) have tried to achieve non-asymptotic generalization bounds for cases where data samples are weakly-dependent (instead of i.i.d. generated) and/or the space of loss functions are only piece-wise smooth. I have not checked the proofs for this part, and results have not been clearly discussed nor compared with existing ones.  Overall, a well-motivated paper which tries to add a number of nice contributions to the area. However, paper suffers from lack of clarifications and intuitive discussions in order to shed light on the technical significance of its many theorems. Also, some notations in parts of the paper (including theorem 2) are not properly framed. Another problem with this work is that author(s) might have missed many existing works that have already tried to analyze the same (or at least slightly different) problem sets. As a result, paper is not well-positioned w.r.t. the literature and cannot be easily compared with other researches.  I cannot recommend the paper for acceptance in its current shape, thus my vote at this stage is weak reject.