Certifying Distributional Robustness using Lipschitz Regularisation

1 INTRODUCTION . Regularised risk minimisation has been the workhorse of learning nonlinear hypotheses such as deep neural networks and kernel machines . Recently , distributional robust risk ( DRR ) minimization has emerged as a promising instance with marked efficacy and flexibility . Instead of perturbing the observed data points , DRRs consider perturbations to the empirical distribution , constituting an ambiguity set P that lives in the space of data distributions . Let Ω be an outcome space with ( true ) distribution µ , e.g. , the joint space of input and output . Given a loss function ` , a model f suffers a loss value ` f ( ω ) over an outcome ω , and the risk of f under µ is risk ` ( f , µ ) : = Eµ [ ` f ] . In DRR minimisation , a model f is sought that minimises the expectation of loss ` over an ambiguity set P , i.e. , that minimises supν∈P risk ` ( f , ν ) ( Delage & Ye , 2010 ; Goh & Sim , 2010 ; Wiesemann et al. , 2014 ) . The ambiguity sets can be constructed by moment matching ( Bhattacharyya et al. , 2005 ; Farnia & Tse , 2016 ) , divergence balls ( Ben-Tal et al. , 2013 ; Duchi et al. , 2016 ; Hu & Hong , 2016 ) , or Wasserstein distance balls ( Kantorovitch , 1958 ) . In this work we focus on the last due to its favorable properties in statistics and computation , along with extensive applications in DRR ( Esfahani & Kuhn , 2018 ; Gao & Kleywegt , 2016 ; Zhao & Guan , 2018 ) . Despite the generality of DRR , its computational efficiency remains a challenge , since the supremum is over a ( typically ) uncountably infinite dimension space . Tractable equivalent convex programs can be derived only for a limited range of loss functions along with linear hypothesis spaces ( Blanchet et al. , 2016 ; El Ghaoui & Lebret , 1997 ; Shafieezadeh-Abadeh et al. , 2015 ; 2017 ; Xu et al. , 2009a ; b ) . Although Shafieezadeh-Abadeh et al . ( 2017 ) developed lifted variants for reproducing kernel Hilbert spaces ( RKHS ) to accommodate nonlinear hypotheses , the perturbation was applied to Φ ( ω ) , where Φ is the implicit feature map . This still falls short of robustness with respect to distributions over Ω . A more promising technique for optimizing DRRs over nonlinear hypothesis spaces—including deep neural networks and kernel machines—is by dualising it to a form that is amenable to ( approximate ) optimisation ( Esfahani & Kuhn , 2018 ) . The fundamental strong duality result was established independently by Blanchet & Murthy ( 2019 ) and Gao & Kleywegt ( 2016 ) , and has been applied to various tasks such as specification of regularisation parameter ( Blanchet et al. , 2016 ) , design of transport cost ( Blanchet et al. , 2017 ) , and selection of ambiguity region size for optimal confidence interval ( Duchi et al. , 2016 ) . In particular , Sinha et al . ( 2018 ) used it to construct an efficiently computable certificate on the level of robustness for the worst-case population loss . However , these methods are still subject to marked restrictions when applied to smooth nonlinear models . First , Sinha et al . ( 2018 ) restricts the perturbation to be small , which despite the common interest of imperceptible perturbations , leaves unaddressed the equally interesting regimes of medium to large perturbations ; see discussions in Openreview ( 2018 ) . Moreover , although the global solvability of the inner Lagrangian penalty problem ( robust surrogate loss ) can be ensured by small enough perturbation , there is no practical procedure to compute the threshold . Finally , the overall optimisation of the nonlinear model is still subject to nonconvexity , precluding tractable global solutions for restricted but still general classes of nonlinear models such as kernel methods . The first goal of this work , therefore , is to develop a novel certificate on distributional robustness that dispenses with these restrictions ( §3 ) . Specifically , we will leverage the McShane-Whitney extension theorem ( McShane , 1934 ; Whitney , 1934 ) to upper bound DRRs by the empirical risk regularized with the Lipschitz constant of the model f , while additionally accounting for the underlying transport cost and the loss ` . The result vastly generalises the vector norm regularisation in linear binary classification ( Shafieezadeh-Abadeh et al. , 2017 , Thm . 3.11 ) to nonlinear models and extended real-valued cost functions that encode constraints , along with an arbitrary metric space of labels that is general enough for multiclass problems . Appealing to any magnitude of perturbation , it also enjoys improved computational efficiency compared with the robust surrogate loss in Sinha et al . ( 2018 ) . A particularly effective domain to apply this new certificate is adversarial learning ( Szegedy et al. , 2014 ) , where models are trained to be resilient to malicious distortions on the data . Although Lipschitz regularisation has been a popular recipe for robustness ( e.g. , Anil et al. , 2019 ; Cisse et al. , 2017 ; Farnia et al. , 2019 ; Gouk et al. , 2018 ; Huster et al. , 2018 ; Scaman & Virmaux , 2018 ) and generalisation accuracy ( Miyato et al. , 2018 ; Yoshida & Miyato , 2017 ) , it remains a heuristic and therefore our second major contribution is to reveal in §3.1 that adversarial risks ( Goodfellow et al. , 2015 ; Madry et al. , 2018 ; Shaham et al. , 2018 ) can be bounded by a DRR . Such a rigorous justification has hitherto been restricted to logistic loss ( Suggala et al. , 2019 , Thm . 9 ) , and a similar tightness result has been established only for linear models ( Shafieezadeh-Abadeh et al. , 2017 , Thm . 3.20 ) . As a result , our new certificate amounts to a new bound on the worst-case risk under attacks , complementing the existing certificates ( Raghunathan et al. , 2018 ; Tsuzuku et al. , 2018 ; Weng et al. , 2018 ; Wong & Kolter , 2018 ; Wong et al. , 2018 ) with a more computationally efficient approach . It further achieved state-of-the-art accuracy under a range of attacks on standard benchmark datasets ( §5 ) . In practice , however , the evaluation of Lipschitz constant L is NP-hard for neural networks ( Scaman & Virmaux , 2018 ) , compelling approximations of it , or explicit engineering of layers to respect Lipschitz while analyzing the expressiveness in specific cases ( e.g. , ` ∞ norm in Anil et al . ( 2019 ) ) . We , instead , pursue a new path and explore the following question : does there exist a hypothesis space which : a ) is expressive enough in modeling ; b ) allows the exact value of L to be computed efficiently ; c ) enforcing the Lipschitz constant leads to a convex constraint that renders efficient optimisation . Interestingly , kernel machines satisfy all these requirements for some kernels . For example , Gaussian kernels are universal , whose RKHS can approximate any continuous function on a compact set in a uniform sense ( Micchelli et al. , 2006 ) . The RKHS of multi-layer inverse kernels compactly encompasses ` 1-regularized neural networks ( Shalev-Shwartz et al. , 2011 ) , degrading the generalisation performance by only a polynomial constant ( Zhang et al. , 2016 ; 2017 ) . Similar results have been conjectured for Gaussian kernels ( Shalev-Shwartz et al. , 2011 ) . Our third contribution proves that b ) can be achieved for product kernels such as Gaussian kernels with high probability by using the Nyström approximation ( Drineas & Mahoney , 2005 ; Williams & Seeger , 2000 ) , and ε approximation error of L requires only O ( 1/ε2 ) samples ( §4 ) . Empirically this approximation is also effective for non-product kernels like inverse kernels . Such a sampling based approach also leads to a single convex constraint , making it scalable to 60k examples with even an interior-point solver ( §5 ) . The convenience in evaluating L renders our certificate of DRR even more favorable than those based on robust surrogate losses ( Blanchet & Murthy , 2019 ; Gao & Kleywegt , 2016 ; Sinha et al. , 2018 ) . 2 PRELIMINARIES . The vast majority of our technical results and proofs are deferred to Appendix A , for which theoremlike statements are numbered to be consistent . The extended real line is R̄ : = [ −∞ , +∞ ] , R̄≥0 : = [ 0 , ∞ ] , and [ n ] : = { 1 , 2 , . . . , n } . For topological spaces X , Y , the Borel subsets are B ( X ) , and the Borel probability measures are P ( X ) . The universal sigma algebra is U ( X ) : = ⋂ µ∈P ( X ) Bµ ( X ) where Bµ ( X ) is the completion of the Borel sets with respect to µ ∈ P ( X ) . Let L0 ( X , Y ) denote the Borel measurable mappings X → Y , and L1 ( X , µ ) denote the Borel functions f ∈ L0 ( X , R ) with ∫ |f |dµ < ∞ for µ ∈ P ( X ) . For two measures µ , ν ∈ P ( Ω ) the set of ( µ , ν ) -couplings is Π ( µ , ν ) : = { π ∈ P ( Ω ×Ω ) | µ = ∫ π ( · , dω ) , ν = ∫ π ( dω , · ) } . Let c ∈ L0 ( Ω ×Ω , R̄ ) . The c-transportation cost of µ , ν ∈ P ( Ω ) , and c-transportation cost ball of radius r ≥ 0 , centred at µ ∈ P ( Ω ) are respectively costc ( µ , ν ) : = inf { ∫ cdπ : π ∈ Π ( µ , ν ) } and Bc ( µ , r ) : = { ν ∈ P ( Ω ) | costc ( µ , ν ) ≤ r } . ( 1 ) A function f : Ω → R̄ is c-Lipschitz if there exists L ≥ 0 such that ∀ω1 , ω2 ∈ dom f : |f ( ω1 ) − f ( ω2 ) | ≤ Lc ( ω1 , ω2 ) . ( 2 ) The least c-Lipschitz constant of f ( cf . Cranko et al. , 2019 ) is the infimum over L ≥ 0 satisfying ( 2 ) , and is denoted by lipc ( f ) , so that when ( X , d ) is a metric space lipd ( f ) agrees with the usual Lipschitz notion . When c : X → R̄ ( e.g. , when c is a norm ) , we take c ( x , y ) : = c ( x − y ) for all x , y ∈ X in ( 1 ) and ( 2 ) . 3 CERTIFICATE FOR DISTRIBUTIONAL ROBUSTNESS . While an elegant concept , the DRR suffers from a lack of tractability . That is , in order to effectively minimise it , we first need to be able to compute or estimate it . When the loss function is convex with respect to the input space this is straight-forward , however in general approximations are necessary . Our first contribution is an upper bound for it . For a function f : X → R̄ there is another function co f : X → R̄ , called the convex envelope of f . It is the greatest closed convex function that minorises f . The quantity ρ ( f ) : = supx∈X ( f ( x ) − co f ( x ) ) was first suggested by Aubin & Ekeland ( 1976 ) to quantify the lack of convexity of a function , and has since shown to be of considerable interest for a variety of nonconvex applications ( Askari et al. , 2019 ; Kerdreux et al. , 2019 ; Lemaréchal & Renaud , 2001 ; Udell & Boyd , 2016 ) . When X = Rn there are well-known ways to compute both co f and ρ ( f ) , and a brief discussion on these appears in the appendix ( Remark 2 on p. 18 ) . ρ ( f ) = 0 when f is closed convex . Theorem 1 . Assume X is a separable Fréchet space and fix µ ∈ P ( X ) . Assume c : X → R̄≥0 is sublinear and continuous , and f ∈ L1 ( X , µ ) is upper semicontinuous . Then for all r ≥ 0 , DRR : = sup ν∈Bc ( µ , r ) risk ` ( f , ν ) ≤ r lipc ( ` f ) + risk ` ( f , µ ) . ( 3 ) The tightness of the bound can be quantified as follows . Let ∆ ( µ ) : = r lipc ( ` f ) + risk ` ( f , µ ) − supν∈Bc ( µ , r ) risk ` ( f , ν ) . If lipc ( f ) < ∞ then ∆ ( µ ) ≤ r ( lipc ( ` f ) − [ lipc ( co ` f ) − 1 r ∫ ( ` f − co ` f ) dµ ] + ) , ( 4 ) where [ · ] + : = max { · , 0 } and 1/0 : =∞ , so that when ` f is closed convex there is equality in ( 3 ) . Clearly ( 4 ) is tight for convex ` f . Furthermore , Proposition 1 shows that ( 4 ) is also tight for a large family of nonconvex functions and distributions — particularly the upper-semicontinuous loss functions on a compact set X0 ⊆ X , with the collection of probability distributions supported on X0 . Proposition 1 . Assume X is a separable Fréchet space with X0 ⊆ X . Assume c : X → R̄≥0 is sublinear and continuous , and ` f ∈ ⋂ µ∈P ( X0 ) L1 ( X , µ ) is upper semicontinuous , has lipc ( ` f ) < ∞ , and attains its maximum on X0 . Then for all r ≥ 0 with 1/0 : =∞ , sup µ∈P ( X0 ) ∆ ( µ ) = r ( lipc ( ` f ) − [ lipc ( co ` f ) − 1 r ρ ( ` f ) ] + ) . Theorem 1 subsumes many existing results ( viz . Gao & Kleywegt , 2016 , Cor . 2 ( iv ) , Cisse et al. , 2017 , §3.2 , Sinha et al. , 2018 , Shafieezadeh-Abadeh et al. , 2017 , Thm . 3.20 ) with a great deal more generality , applying to a very broad family of models , loss functions , and outcome spaces . It is the first time to our knowledge that the slackness in ( 3 ) has been characterised tightly . The extension of Theorem 1 for robust classification in the absence of label noise is straight-forward . Corollary 1 . Assume X is a separable Fréchet space and Y is a topological space . Fix µ ∈ P ( X × Y ) . Assume c : ( X × Y ) × ( X × Y ) → R̄ satisfies c ( x , y , x′ , y′ ) = cX ( x− x′ ) whenever y = y′ and c ( x , y , x′ , y′ ) =∞ whenever y 6= y′ , where cX : X → R̄ is symmetric , sublinear , and continuous , and f ∈ L1 ( X × Y , µ ) is upper semicontinuous . Then for all r ≥ 0 there is ( 3 ) . To see the tightness of the bound , if lipc ( ` f ) < ∞ there is ( 4 ) , where the closed convex hull is interpreted as co ( ` f ) ( x , y ) : = co ( ` f ( · , y ) ) ( x ) . If additionally ` f ( · , y ) is closed convex for all y ∈ Y , there is equality in ( 3 ) .

Summary: This paper uses results from distributional robustness to provide bounds of p-norm-constrained adversarial risk which depend on the Lipschitz constant of the underlying classifier. The bulk of the paper focuses on sample-efficient mechanisms to approximate the Lipschitz constants of kernel methods so that a constraint on this Lip constant can be enforced during training. Empirically, the kernel methods are compared to existing deep learning approaches and are shown to be competitive at this scale.