Boosting the Certified Robustness of L-infinity Distance Nets

1 INTRODUCTION . Modern neural networks , while achieving high accuracy on various tasks , are found to be vulnerable to small , adversarially-chosen perturbations of the inputs ( Szegedy et al. , 2013 ; Biggio et al. , 2013 ) . Given an image x correctly classified by a neural network , there often exists a small adversarial perturbation δ , such that the perturbed image x+ δ looks indistinguishable to x , but fools the network to predict an incorrect class with high confidence . Such vulnerability creates security concerns in many real-world applications . A large body of works has been developed to obtain robust classifiers . One line of works proposed heuristic approaches that are empirically robust to particular attack methods , among which adversarial training is the most successful approach ( Goodfellow et al. , 2014 ; Madry et al. , 2017 ; Zhang et al. , 2019a ) . However , a variety of these heuristics have been subsequently broken by stronger and adaptive attacks ( Carlini & Wagner , 2017 ; Athalye et al. , 2018 ; Uesato et al. , 2018 ; Tramer et al. , 2020 ; Croce & Hein , 2020 ) , and there are no formal guarantees whether the resulting model is truly robust . This motivates another line of works that seeks certifiably robust classifiers whose prediction is guaranteed to remain the same under all allowed perturbations . Representatives of this field use convex relaxation ( Wong & Kolter , 2018 ; Mirman et al. , 2018 ; Gowal et al. , 2018 ; Zhang et al. , 2020b ) or randomized smoothing ( Cohen et al. , 2019 ; Salman et al. , 2019a ; Zhai et al. , 2020 ; Yang et al. , 2020a ) . However , these approaches typically suffer from high computational cost , yet still can not achieve satisfactory results for commonly used ` ∞-norm perturbation scenario . Recently , Zhang et al . ( 2021 ) proposed a fundamentally different approach by designing a new network architecture called ` ∞-distance net , a name coming from its construction that the basic neuron is defined as the ` ∞-distance function . Using the fact that any ` ∞-distance net is inherently a 1- Lipschitz mapping , one can easily check whether the prediction is certifiably robust for a given data point according to the output margin . The whole procedure only requires a forward pass without any additional computation . The authors further showed that the model family has strong expressive power , e.g. , a large enough ` ∞-distance net can approximate any 1-Lipschitz function on a bounded domain . Unfortunately , however , the empirical model performance did not well reflect the theoretical advantages . As shown in Zhang et al . ( 2021 ) , it is necessary to use a conventional multi-layer perception ( MLP ) 1 on top of an ` ∞-distance net backbone to achieve better performance compared to the baseline methods . It makes both the training and the certification procedure complicated . More importantly , it calls into question whether the ` ∞-distance net is really a better model configuration than conventional architectures in the regime of certified robustness . In this paper , we give an affirmative answer by showing that ` ∞-distance net itself suffices for good performance and can be well learned using an improved training strategy . We first mathematically prove that under mild assumptions of the dataset , there exists an ` ∞-distance net with reasonable size by construction that achieves perfect certified robustness . This result indicates the strong expressive power of ` ∞-distance nets in robustness certification , and shows a fundamental advantage over conventional networks with typical certification approaches ( Mirman et al. , 2021 ) . However , it seems to contradict the previous empirical observations , suggesting that the model may fail to find an optimal solution and further motivating us to revisit the optimization process designed in Zhang et al . ( 2021 ) . Due to the non-smoothness of the ` ∞-distance function , Zhang et al . ( 2021 ) developed several training tricks to overcome the optimization difficulty . A notable trick is called the ` p-relaxation , in which ` p-distance neurons are used during optimization to give a smooth approximation of ` ∞-distance . However , we find that the relaxation on neurons unexpectedly relaxes the Lipschitz constant of the network to an exponentially large value , making the objective function no longer maximize the robust accuracy and leading to sub-optimal solutions . We develop a novel modification of the objective function to bypass the problem mentioned above . The objective function is a linear combination of a scaled cross-entropy term and a modified clipped hinge term . The cross-entropy loss maximizes the output margin despite the model ’ s Lipschitzness and makes optimization sufficient at the early training stage when p is small . The clipped hinge loss then focuses on robustness for correctly classified samples at the late training phase when p approaches infinity . The switch from cross-entropy loss to clipped hinge loss is reflected in the mixing coefficient , which decays to zero as p grows to infinity throughout the training procedure . Despite its simplicity , our experimental results show significant performance gains on various datasets . In particular , an ` ∞-distance net backbone can achieve 40.06 % certified robust accuracy on CIFAR-10 ( = 8/255 ) . This goes far beyond the previous results , which achieved 33.30 % certified accuracy on CIFAR-10 using the same architecture ( Zhang et al. , 2021 ) . Besides , it surpasses the relaxation-based certification approaches by at least 5 points ( Shi et al. , 2021 ; Lyu et al. , 2021 ) , establishing a new state-of-the-art result . To summarize , both the theoretical finding and empirical results in this paper demonstrate the merit of ` ∞-distance net for certified robustness . Considering the simplicity of the architecture and training strategy used in this paper , we believe there are still many potentials for future research of ` ∞-distance nets , and more generally , the class of Lipschitz architectures . 2 PRELIMINARY . In this section , we briefly introduce the ` ∞-distance net and its training strategy . An ` ∞-distance net is constructed using ` ∞-distance neurons as the basic component . The ` ∞-distance neuron u takes vector x as the input and calculates the ` ∞-norm distance between x and parameter w with a bias term b . The neuron can be written as u ( x , { w , b } ) = ‖x−w‖∞ + b . ( 1 ) Based on the neuron definition , a fully connected ` ∞-distance net can then be constructed . Formally , an L layer network g takes x ( 0 ) = x as the input , and the lth layer x ( l ) is calculated by x ( l ) i = u ( x ( l−1 ) , { w ( l , i ) , b ( l ) i } ) = ‖x ( l−1 ) −w ( l , i ) ‖∞ + b ( l ) i , l ∈ [ L ] , i ∈ [ nl ] . ( 2 ) 1Without any confusion , in this paper , a conventional neural network model is referred to as a network composed of linear transformations with non-linear activations . Here nl is the number of neurons in the lth layer . For K-class classification problems , nL = K. The network outputs g ( x ) = x ( L ) as logits and predicts the class arg maxi∈ [ K ] [ g ( x ) ] i . An important property of ` ∞-distance net is its Lipschitz continuity , as is stated below . Definition 2.1 . A mapping f ( z ) : Rm → Rn is called λ-Lipschitz with respect to ` p-norm ‖ · ‖p , if for any z1 , z2 , the following holds : ‖f ( z1 ) − f ( z2 ) ‖p ≤ λ‖z1 − z2‖p . ( 3 ) Proposition 2.2 . The mapping of an ` ∞-distance layer is 1-Lipschitz with respect to ` ∞-norm . Thus by composition , any ` ∞-distance net g ( · ) is 1-Lipschitz with respect to ` ∞-norm . ` ∞-distance nets naturally possess certified robustness using the Lipschitz property . In detail , for any data point x with label y , denote the output margin of network g as margin ( x , y ; g ) = [ g ( x ) ] y −max j 6=y [ g ( x ) ] j . ( 4 ) If x is correctly classified by g , then the prediction of a perturbed input x + δ will remain the same as x if ‖δ‖∞ < margin ( x , y ; g ) /2 . In other words , we can obtain the certified robustness for a given perturbation level according to I ( margin ( x , y ; g ) /2 > ) , where I ( · ) is the indicator function . We call this margin-based certification . Given this certification approach , a corresponding training approach can then be developed , where one simply learns a large margin classifier using standard loss functions , e.g. , hinge loss , without using adversarial training . Therefore the whole training procedure is as efficient as training standard networks with no additional cost . Zhang et al . ( 2021 ) further show that ` ∞-distance nets are Lipschitz-universal approximators . In detail , a large enough ` ∞-distance net can approximate any 1-Lipschitz function with respect to ` ∞-norm on a bounded domain arbitrarily well . Training ` ∞-distance nets . One major challenge in training ` ∞-distance net is that the ` ∞-distance operation is highly non-smooth , and the gradients ( i.e . ∇x‖x−w‖∞ and∇w‖x−w‖∞ ) are sparse . To mitigate the problem , Zhang et al . ( 2021 ) used ` p-distance neurons instead of ` ∞-distance ones during training , resulting in approximate and non-sparse gradients . Typically p is set to a small value ( e.g. , 8 ) in the beginning and increases throughout training until it reaches a large number ( e.g. , 1000 ) . The authors also designed several other tricks to further address the optimization difficulty . However , even with the help of tricks , ` ∞-distance nets only perform competitively to previous works . The authors thus considered using a hybrid model architecture , in which the ` ∞-distance net serves as a robust feature extractor , and an additional conventional multi-layer perceptron is used as the prediction head . This architecture achieves the best performance , but both the training and the certification approach become complicated again due to the presence of non-Lipschitz MLP layers . 3 EXPRESSIVE POWER OF ` ∞-DISTANCE NETS IN ROBUST CLASSIFICATION . In this section , we challenge the conclusion of previous work by proving that simple ` ∞-distance nets ( without the top MLP ) suffice for achieving perfect certified robustness in classification . Recall that Zhang et al . ( 2021 ) already provides a universal approximation theorem , showing the expressive power of ` ∞-distance nets to represent Lipschitz functions . However , their result focuses on realvalued function approximation and is not directly helpful for certified robustness in classification . One may ask : Does a certifiably robust ` ∞-distance net exist given a dataset ? If so , how large does the network need to be ? We will answer these questions and show that one can explicitly construct an ` ∞-distance net that achieves perfect certified robustness as long as the dataset satisfies the following ( weak ) condition called r-separation ( Yang et al. , 2020b ) . Definition 3.1 . ( r-separation ) Consider a labeled datasetD = { ( xi , yi ) } where yi ∈ [ K ] is the label of xi . We say D is r-separated with respect to ` p-norm if for any pair of samples ( xi , yi ) , ( xj , yj ) , as long as yi 6= yj , we have ‖xi − xj‖p > 2r . It is easy to see that r-separation is a necessary condition for robustness under ` p-norm perturbation = r. In fact , the condition holds for all commonly used datasets ( e.g. , MNIST , CIFAR-10 ) : the value of r in each dataset is much greater than the allowed perturbation level as is demonstrated in Yang et al . ( 2020b ) ( see Table 1 above ) . The authors took a further step and showed there always exists a classifier that achieves perfect robust accuracy if the condition holds . We now prove that even if we restrict the classifier to be the network function class represented by ` ∞-distance nets , the conclusion is still correct : a simple two-layer ` ∞-distance net with hidden sizeO ( n ) can already achieve perfect robustness for r-separated datasets . Theorem 3.2 . LetD be a dataset with n elements satisfying the r-separation condition with respect to ` ∞-norm . Then there exists a two-layer ` ∞-distance net with hidden size n , such that when using margin-based certification , the certified ` ∞ robust accuracy is 100 % onD under perturbation = r. Proof sketch . Consider a two layer ` ∞-distance net g defined in Equation ( 2 ) . Let its parameters be assigned by w ( 1 , i ) = xi , b ( 1 ) i = 0 for i ∈ [ n ] w ( 2 , j ) i = C · I ( yi = j ) , b ( 2 ) j = −C for i ∈ [ n ] , j ∈ [ K ] where C = 4 maxi∈ [ n ] ‖xi‖∞ is a constant , and I ( · ) is the indicator function . For the above assignment , it can be proved that the network outputs [ g ( x ) ] j = x ( 2 ) j = − min i∈ [ n ] , yi=j ‖x− xi‖∞ . ( 5 ) From Equation ( 5 ) the network g can represent a nearest neighbor classifier , in that it outputs the negative of the nearest neighbor distance between input x and the samples of each class . Therefore , given data x = xi in dataset D , the output margin of g ( x ) is at least 2r due to the r-separation condition . In other words , g achieves 100 % certified robust accuracy on D. Remark 3.3 . The above result can be extended to multi-layer networks . In general , we can prove the existence of such networks with L layers and no more than O ( n/L + K + d ) hidden neurons for each hidden layer where d is the input dimension . See Appendix A for details of the proof . The significance of Theorem 3.2 can be reflected in the following two aspects . Firstly , our result explicitly shows the strong expressive power of ` ∞-distance nets in robust classification , which complements the universal approximation theorem in Zhang et al . ( 2021 ) . Moreover , Theorem 3.2 gives an upper bound of O ( n ) on the required network size which is close to practical applications . It is much smaller than the size needed for function approximation ( O ( 1/εd ) under approximation error ε , proved in Zhang et al . ( 2021 ) ) , which scales exponentially in the input dimension d. Secondly , our result justifies that for well-designed architectures , using only the global Lipschitz property is sufficient for robustness certification . It contrasts to the prior view that suggests leveraging the local Lipschitz constant is necessary ( Huster et al. , 2018 ) , which typically needs sophisticated calculations ( Wong et al. , 2018 ; Zhang et al. , 2018 ; 2020b ) . More importantly , as a comparison , Mirman et al . ( 2021 ) very recently proved that for any conventional network , the commonlyused interval bound propagation ( IBP ) ( Mirman et al. , 2018 ; Gowal et al. , 2018 ) intrinsically can not achieve perfect certified robustness on a simple r-separation dataset containing only three data points ( under = r ) . In other words , ` ∞-distance nets certified using the global Lipschitz property have a fundamental advantage over conventional networks certified using interval bound propagation .

Summary: This paper is a follow-up paper of Zhang et al. (2021). In Zhang et al. (2021), the authors proposed a new network architecture, l_infty distance net. By construction, the network is 1-Lipschitz w.r.t. l_infty distance. However, the training procedure therein is problematic. This paper resolves the issue by a new loss design of scaled cross entropy loss + clipped hinge loss. Without using MLP on top of the l_infty distance net backbone, the proposed new training method outperforms the original one in Zhang et al. (2021) and improves over the state-of-the-art by more than 5% for 8/255 and other radiuses. Theoretically, the paper shows the expressive power of l_infty distance net for well-separated data.