Certifying Distributional Robustness using Lipschitz Regularisation

1 INTRODUCTION . Regularised risk minimisation has been the workhorse of learning nonlinear hypotheses such as deep neural networks and kernel machines . Recently , distributional robust risk ( DRR ) minimization has emerged as a promising instance with marked efficacy and flexibility . Instead of perturbing the observed data points , DRRs consider perturbations to the empirical distribution , constituting an ambiguity set P that lives in the space of data distributions . Let Ω be an outcome space with ( true ) distribution µ , e.g. , the joint space of input and output . Given a loss function ` , a model f suffers a loss value ` f ( ω ) over an outcome ω , and the risk of f under µ is risk ` ( f , µ ) : = Eµ [ ` f ] . In DRR minimisation , a model f is sought that minimises the expectation of loss ` over an ambiguity set P , i.e. , that minimises supν∈P risk ` ( f , ν ) ( Delage & Ye , 2010 ; Goh & Sim , 2010 ; Wiesemann et al. , 2014 ) . The ambiguity sets can be constructed by moment matching ( Bhattacharyya et al. , 2005 ; Farnia & Tse , 2016 ) , divergence balls ( Ben-Tal et al. , 2013 ; Duchi et al. , 2016 ; Hu & Hong , 2016 ) , or Wasserstein distance balls ( Kantorovitch , 1958 ) . In this work we focus on the last due to its favorable properties in statistics and computation , along with extensive applications in DRR ( Esfahani & Kuhn , 2018 ; Gao & Kleywegt , 2016 ; Zhao & Guan , 2018 ) . Despite the generality of DRR , its computational efficiency remains a challenge , since the supremum is over a ( typically ) uncountably infinite dimension space . Tractable equivalent convex programs can be derived only for a limited range of loss functions along with linear hypothesis spaces ( Blanchet et al. , 2016 ; El Ghaoui & Lebret , 1997 ; Shafieezadeh-Abadeh et al. , 2015 ; 2017 ; Xu et al. , 2009a ; b ) . Although Shafieezadeh-Abadeh et al . ( 2017 ) developed lifted variants for reproducing kernel Hilbert spaces ( RKHS ) to accommodate nonlinear hypotheses , the perturbation was applied to Φ ( ω ) , where Φ is the implicit feature map . This still falls short of robustness with respect to distributions over Ω . A more promising technique for optimizing DRRs over nonlinear hypothesis spaces—including deep neural networks and kernel machines—is by dualising it to a form that is amenable to ( approximate ) optimisation ( Esfahani & Kuhn , 2018 ) . The fundamental strong duality result was established independently by Blanchet & Murthy ( 2019 ) and Gao & Kleywegt ( 2016 ) , and has been applied to various tasks such as specification of regularisation parameter ( Blanchet et al. , 2016 ) , design of transport cost ( Blanchet et al. , 2017 ) , and selection of ambiguity region size for optimal confidence interval ( Duchi et al. , 2016 ) . In particular , Sinha et al . ( 2018 ) used it to construct an efficiently computable certificate on the level of robustness for the worst-case population loss . However , these methods are still subject to marked restrictions when applied to smooth nonlinear models . First , Sinha et al . ( 2018 ) restricts the perturbation to be small , which despite the common interest of imperceptible perturbations , leaves unaddressed the equally interesting regimes of medium to large perturbations ; see discussions in Openreview ( 2018 ) . Moreover , although the global solvability of the inner Lagrangian penalty problem ( robust surrogate loss ) can be ensured by small enough perturbation , there is no practical procedure to compute the threshold . Finally , the overall optimisation of the nonlinear model is still subject to nonconvexity , precluding tractable global solutions for restricted but still general classes of nonlinear models such as kernel methods . The first goal of this work , therefore , is to develop a novel certificate on distributional robustness that dispenses with these restrictions ( §3 ) . Specifically , we will leverage the McShane-Whitney extension theorem ( McShane , 1934 ; Whitney , 1934 ) to upper bound DRRs by the empirical risk regularized with the Lipschitz constant of the model f , while additionally accounting for the underlying transport cost and the loss ` . The result vastly generalises the vector norm regularisation in linear binary classification ( Shafieezadeh-Abadeh et al. , 2017 , Thm . 3.11 ) to nonlinear models and extended real-valued cost functions that encode constraints , along with an arbitrary metric space of labels that is general enough for multiclass problems . Appealing to any magnitude of perturbation , it also enjoys improved computational efficiency compared with the robust surrogate loss in Sinha et al . ( 2018 ) . A particularly effective domain to apply this new certificate is adversarial learning ( Szegedy et al. , 2014 ) , where models are trained to be resilient to malicious distortions on the data . Although Lipschitz regularisation has been a popular recipe for robustness ( e.g. , Anil et al. , 2019 ; Cisse et al. , 2017 ; Farnia et al. , 2019 ; Gouk et al. , 2018 ; Huster et al. , 2018 ; Scaman & Virmaux , 2018 ) and generalisation accuracy ( Miyato et al. , 2018 ; Yoshida & Miyato , 2017 ) , it remains a heuristic and therefore our second major contribution is to reveal in §3.1 that adversarial risks ( Goodfellow et al. , 2015 ; Madry et al. , 2018 ; Shaham et al. , 2018 ) can be bounded by a DRR . Such a rigorous justification has hitherto been restricted to logistic loss ( Suggala et al. , 2019 , Thm . 9 ) , and a similar tightness result has been established only for linear models ( Shafieezadeh-Abadeh et al. , 2017 , Thm . 3.20 ) . As a result , our new certificate amounts to a new bound on the worst-case risk under attacks , complementing the existing certificates ( Raghunathan et al. , 2018 ; Tsuzuku et al. , 2018 ; Weng et al. , 2018 ; Wong & Kolter , 2018 ; Wong et al. , 2018 ) with a more computationally efficient approach . It further achieved state-of-the-art accuracy under a range of attacks on standard benchmark datasets ( §5 ) . In practice , however , the evaluation of Lipschitz constant L is NP-hard for neural networks ( Scaman & Virmaux , 2018 ) , compelling approximations of it , or explicit engineering of layers to respect Lipschitz while analyzing the expressiveness in specific cases ( e.g. , ` ∞ norm in Anil et al . ( 2019 ) ) . We , instead , pursue a new path and explore the following question : does there exist a hypothesis space which : a ) is expressive enough in modeling ; b ) allows the exact value of L to be computed efficiently ; c ) enforcing the Lipschitz constant leads to a convex constraint that renders efficient optimisation . Interestingly , kernel machines satisfy all these requirements for some kernels . For example , Gaussian kernels are universal , whose RKHS can approximate any continuous function on a compact set in a uniform sense ( Micchelli et al. , 2006 ) . The RKHS of multi-layer inverse kernels compactly encompasses ` 1-regularized neural networks ( Shalev-Shwartz et al. , 2011 ) , degrading the generalisation performance by only a polynomial constant ( Zhang et al. , 2016 ; 2017 ) . Similar results have been conjectured for Gaussian kernels ( Shalev-Shwartz et al. , 2011 ) . Our third contribution proves that b ) can be achieved for product kernels such as Gaussian kernels with high probability by using the Nyström approximation ( Drineas & Mahoney , 2005 ; Williams & Seeger , 2000 ) , and ε approximation error of L requires only O ( 1/ε2 ) samples ( §4 ) . Empirically this approximation is also effective for non-product kernels like inverse kernels . Such a sampling based approach also leads to a single convex constraint , making it scalable to 60k examples with even an interior-point solver ( §5 ) . The convenience in evaluating L renders our certificate of DRR even more favorable than those based on robust surrogate losses ( Blanchet & Murthy , 2019 ; Gao & Kleywegt , 2016 ; Sinha et al. , 2018 ) . 2 PRELIMINARIES . The vast majority of our technical results and proofs are deferred to Appendix A , for which theoremlike statements are numbered to be consistent . The extended real line is R̄ : = [ −∞ , +∞ ] , R̄≥0 : = [ 0 , ∞ ] , and [ n ] : = { 1 , 2 , . . . , n } . For topological spaces X , Y , the Borel subsets are B ( X ) , and the Borel probability measures are P ( X ) . The universal sigma algebra is U ( X ) : = ⋂ µ∈P ( X ) Bµ ( X ) where Bµ ( X ) is the completion of the Borel sets with respect to µ ∈ P ( X ) . Let L0 ( X , Y ) denote the Borel measurable mappings X → Y , and L1 ( X , µ ) denote the Borel functions f ∈ L0 ( X , R ) with ∫ |f |dµ < ∞ for µ ∈ P ( X ) . For two measures µ , ν ∈ P ( Ω ) the set of ( µ , ν ) -couplings is Π ( µ , ν ) : = { π ∈ P ( Ω ×Ω ) | µ = ∫ π ( · , dω ) , ν = ∫ π ( dω , · ) } . Let c ∈ L0 ( Ω ×Ω , R̄ ) . The c-transportation cost of µ , ν ∈ P ( Ω ) , and c-transportation cost ball of radius r ≥ 0 , centred at µ ∈ P ( Ω ) are respectively costc ( µ , ν ) : = inf { ∫ cdπ : π ∈ Π ( µ , ν ) } and Bc ( µ , r ) : = { ν ∈ P ( Ω ) | costc ( µ , ν ) ≤ r } . ( 1 ) A function f : Ω → R̄ is c-Lipschitz if there exists L ≥ 0 such that ∀ω1 , ω2 ∈ dom f : |f ( ω1 ) − f ( ω2 ) | ≤ Lc ( ω1 , ω2 ) . ( 2 ) The least c-Lipschitz constant of f ( cf . Cranko et al. , 2019 ) is the infimum over L ≥ 0 satisfying ( 2 ) , and is denoted by lipc ( f ) , so that when ( X , d ) is a metric space lipd ( f ) agrees with the usual Lipschitz notion . When c : X → R̄ ( e.g. , when c is a norm ) , we take c ( x , y ) : = c ( x − y ) for all x , y ∈ X in ( 1 ) and ( 2 ) . 3 CERTIFICATE FOR DISTRIBUTIONAL ROBUSTNESS . While an elegant concept , the DRR suffers from a lack of tractability . That is , in order to effectively minimise it , we first need to be able to compute or estimate it . When the loss function is convex with respect to the input space this is straight-forward , however in general approximations are necessary . Our first contribution is an upper bound for it . For a function f : X → R̄ there is another function co f : X → R̄ , called the convex envelope of f . It is the greatest closed convex function that minorises f . The quantity ρ ( f ) : = supx∈X ( f ( x ) − co f ( x ) ) was first suggested by Aubin & Ekeland ( 1976 ) to quantify the lack of convexity of a function , and has since shown to be of considerable interest for a variety of nonconvex applications ( Askari et al. , 2019 ; Kerdreux et al. , 2019 ; Lemaréchal & Renaud , 2001 ; Udell & Boyd , 2016 ) . When X = Rn there are well-known ways to compute both co f and ρ ( f ) , and a brief discussion on these appears in the appendix ( Remark 2 on p. 18 ) . ρ ( f ) = 0 when f is closed convex . Theorem 1 . Assume X is a separable Fréchet space and fix µ ∈ P ( X ) . Assume c : X → R̄≥0 is sublinear and continuous , and f ∈ L1 ( X , µ ) is upper semicontinuous . Then for all r ≥ 0 , DRR : = sup ν∈Bc ( µ , r ) risk ` ( f , ν ) ≤ r lipc ( ` f ) + risk ` ( f , µ ) . ( 3 ) The tightness of the bound can be quantified as follows . Let ∆ ( µ ) : = r lipc ( ` f ) + risk ` ( f , µ ) − supν∈Bc ( µ , r ) risk ` ( f , ν ) . If lipc ( f ) < ∞ then ∆ ( µ ) ≤ r ( lipc ( ` f ) − [ lipc ( co ` f ) − 1 r ∫ ( ` f − co ` f ) dµ ] + ) , ( 4 ) where [ · ] + : = max { · , 0 } and 1/0 : =∞ , so that when ` f is closed convex there is equality in ( 3 ) . Clearly ( 4 ) is tight for convex ` f . Furthermore , Proposition 1 shows that ( 4 ) is also tight for a large family of nonconvex functions and distributions — particularly the upper-semicontinuous loss functions on a compact set X0 ⊆ X , with the collection of probability distributions supported on X0 . Proposition 1 . Assume X is a separable Fréchet space with X0 ⊆ X . Assume c : X → R̄≥0 is sublinear and continuous , and ` f ∈ ⋂ µ∈P ( X0 ) L1 ( X , µ ) is upper semicontinuous , has lipc ( ` f ) < ∞ , and attains its maximum on X0 . Then for all r ≥ 0 with 1/0 : =∞ , sup µ∈P ( X0 ) ∆ ( µ ) = r ( lipc ( ` f ) − [ lipc ( co ` f ) − 1 r ρ ( ` f ) ] + ) . Theorem 1 subsumes many existing results ( viz . Gao & Kleywegt , 2016 , Cor . 2 ( iv ) , Cisse et al. , 2017 , §3.2 , Sinha et al. , 2018 , Shafieezadeh-Abadeh et al. , 2017 , Thm . 3.20 ) with a great deal more generality , applying to a very broad family of models , loss functions , and outcome spaces . It is the first time to our knowledge that the slackness in ( 3 ) has been characterised tightly . The extension of Theorem 1 for robust classification in the absence of label noise is straight-forward . Corollary 1 . Assume X is a separable Fréchet space and Y is a topological space . Fix µ ∈ P ( X × Y ) . Assume c : ( X × Y ) × ( X × Y ) → R̄ satisfies c ( x , y , x′ , y′ ) = cX ( x− x′ ) whenever y = y′ and c ( x , y , x′ , y′ ) =∞ whenever y 6= y′ , where cX : X → R̄ is symmetric , sublinear , and continuous , and f ∈ L1 ( X × Y , µ ) is upper semicontinuous . Then for all r ≥ 0 there is ( 3 ) . To see the tightness of the bound , if lipc ( ` f ) < ∞ there is ( 4 ) , where the closed convex hull is interpreted as co ( ` f ) ( x , y ) : = co ( ` f ( · , y ) ) ( x ) . If additionally ` f ( · , y ) is closed convex for all y ∈ Y , there is equality in ( 3 ) .

Summary: Through the lens of Distributional Robust Risk (DRR), this work draws a link between adversarial robustness and Lipschitz constant regularisation. The authors first provide an upper bound of the DRR (with a Wasserstein ball as the ambiguity set) in terms of the true risk and the Lipschitz constant of the loss function under the current model. They show that the standard adversarial risk can be upper bounded by the DRR, emphasizing that the Lipschitz constant regularised loss can be used as a proxy for adversarially robust training.