Tactics on Refining Decision Boundary for Improving Certification-based Robust Training

In certification-based robust training , existing methods utilize relaxation based methods to bound the worst case performance of neural networks given certain perturbation . However , these certification based methods treat all the examples equally regardless of their vulnerability and true adversarial distribution , limiting the model ’ s potential in achieving optimal verifiable accuracy . In the paper , we propose new methods to include the customized weight distribution and automatic schedule tuning methods on the perturbation schedule . These methods are generally applicable to all the certifiable robust training with almost no additional computational cost . Our results show improvement on MNIST with = 0.3 and CIFAR on = 8/255 for both IBP and CROWN-IBP based methods . 1 INTRODUCTION . Deep neural networks ( DNNs ) have been shown to be highly vulnerable to adversarial attacks , carefully-crafted inputs that are nearly indistinguishable from naturally-occurring data but are misclassified by the network ( Goodfellow et al. , 2014 ; Szegedy et al. , 2014 ) . There exist many algorithms for both crafting adversarial attacks ( Papernot et al. , 2016 ) and building neural networks that are robust against such attacks . Fast gradient sign method ( FGSM ) ( Goodfellow et al. , 2014 ) was the very first approach to generate strong adversary . Projected gradient descent ( PGD ) ( Madry et al. , 2018 ) is one of the most successful and widely-used defense methods available . Adversarial training seeks to minimize the worst-case loss under adversarial perturbations within a pre-defined perturbation level , where multi-step PGD is used to estimate the worst-case attack during training . Compared to standard training , the adversarial term introduces risk of over-fitting ( Moosavi-Dezfooli , 2021 ; Rice et al. , 2020 ; Wang et al. , 2019 ) and training instability ( Tsipras et al. , 2019 ; Zhang et al. , 2020b ) for adversarial training . There exist many related works on improving the model performance by additional regulation ( Zhang et al. , 2020b ; Cisse et al. , 2017 ) and customizing training curriculum ( Zhang et al. , 2019 ; Wang et al. , 2020 ; Cai et al. , 2018 ) for attack-based scenario . While adversarial training has been shown to be empirically effective against many types of attacks , it can not be proven that the resultant models are robust against all adversaries . In fact , it has been shown that many defense methods dependent on heuristic techniques , including adversarial training , can be bypassed by stronger adversaries ( Athalye et al. , 2018 ) . This has motivated a separate branch of research focused on robustness certification/verification : computing provable guarantees on the robustness performance of neural networks against inputs with arbitrary perturbations within some ` p norm-bounded ball . There are two main types of verification methods : complete and incomplete verification . The former computes exact robustness bounds using computationally-expensive methods such as mixed-integer programming ( MIP ) ( Tjeng et al. , 2019 ; Bunel et al. , 2018 ) , whereas the latter provides looser robustness bounds with different branches of methods like randomized smoothing ( Cohen et al. , 2019 ; Salman et al. , 2019 ; Lecuyer et al. , 2019 ) , Lipschitz-based robustness ( Trockman & Kolter , 2021 ; Tsuzuku et al. , 2018 ) and convex adversarial polytype ( Weng et al. , 2018 ; Zhang et al. , 2018 ; Dvijotham et al. , 2018b ; Gowal et al. , 2019 ) on which in this paper we mainly focus . A sparsely related branch of research , called certified robust training ( Dvijotham et al. , 2018a ; Gowal et al. , 2019 ; Zhang et al. , 2020a ) , aims to train a certifiably robust model . These methods compute verified robustness bounds and incorporate them into the training process , such that the resultant models can then be proven to be robust using a verification method . Currently , the most efficient certified training method is interval bound propagation ( IBP ) ( Gowal et al. , 2019 ) , which requires only two additional forward passes during training . It is important to note that IBP is significantly more efficient than adversarial training , which often require many PGD steps to achieve high robustness ( Madry et al. , 2018 ) . In standard certified training , a uniform and loss function weight are usually used across all the training examples , this is not ideal for optimal verified accuracy since the examples are not necessarily equivalent in terms of vulnerability . 1.1 OUR CONTRIBUTIONS . In this paper , we propose two novel methods to improve robustness via refining certified decision boundary for verifiable adversarial training methods such as IBP and CROWN-IBP . In particular , our algorithms are generally applicable to all the verifiable adversarial training methods at almost no additional computational cost . We sum up the key contributions as following : 1 . Zeng et al . ( 2020 ) pointed out the adversarial example distribution deviates from the clean data . We further analyze the importance weight to correct empirical distribution from a novel perspective and theoretically prove more weight is needed for the examples closer to decision boundary if sampled from clean data distribution . 2 . Building upon previous analysis , we come up with a symmetrical re-weighting function based on worst case margin of the correct labels , emphasizing the examples around the decision boundary . 3 . For verifiable adversarial perturbation , empirically a slightly larger train achieves optimal evaluation robustness accuracy on eval , while this uniform setup is not ideal for examples around decision boundary . To address the issue of large perturbation , we developed an auto-tuning algorithm to customize the train for each individual example . 2 BACKGROUND . 2.1 ADVERSARIAL ATTACKS AND TRAINING . Let D = { ( xi , yi ) } ni=1 represent the dataset , where xi ∈ X and yi ∈ Y = { 0 , 1 , ... , C − 1 } . Let B ( xi , ) denote the set of points in the ` p-norm ball with radius around xi . The objective function is min θ 1 n n∑ i=1 l ( fθ ( x ′ i ) , yi ) , where x ′ i = arg max x′∈B ( xi , ) l ( fθ ( x ′ ) , yi ) ( 1 ) fθ : X −→ RC is a score function and l : RC × Y −→ R is the loss function . Adversarial training tackles this min-max problem by alternating between solving inner loop by attack method to estimate the worst-case adversarial scenario in B ( xi , ) and outer loop to update the model parameters and minimize loss function . 2.2 CERTIFIED TRAINING . Robustness Verification . Different from attack-based training , certified training provides a guarantee for worst case scenario via bounding the neural network outputs . The certified accuracy is the lower bound of robustness accuracy under any attack method , thus improving the certified accuracy helps in understanding the potential of the neural network in defending against adversarial attacks . Interval Bound Propagation . There have been many proposed neural network verification algorithms that can be used to compute worst case output . IBP ( Gowal et al. , 2019 ) utilizes a simple , interval arithmetic approach to propagate bounds through a network . Let zk−1 denote the input to a layer zk . IBP bounds zk by computing lower and upper bounds zk , zk such that zk ≤ zk ≤ zk holds element-wise . For affine layers represented by hk ( zk−1 ) = Wzk−1 + b , IBP computes : zk = W zk−1+zk−1 2 − |W | zk−1−zk−1 2 + b and zk = W zk−1+zk−1 2 + |W | zk−1−zk−1 2 + b. Propagating the bounds through the network allows us to compute the upper and lower bound of last layer logits zK , zK and evaluate if an input x is verifiably robust . The logit of the true class equals the lower bound and the logits of other classes equal to the upper bound : ẑK , m = { zK , m if m is the true class zK , m otherwise ( 2 ) IBP training uses a hyperparameter schedule on ( starting from 0 and increasing to train , typically set at train which is slightly larger than eval ) and a mixed loss function that combines natural and robust cross-entropy loss : minθ E ( x , y ) ∼P [ κl ( zK , y ) + ( 1 − κ ) l ( ẑK , y ) ] , where P is the data distribution , l is the cross entropy loss , κ is a hyperparameter that balances the weight between natural and robust loss , and ẑK represents the worst case logits computed using IBP . CROWN-IBP . CROWN was introduced by Zhang et al . ( 2018 ) and achieves a tight bound by adaptively selecting the linear approximation . Zhang et al . ( 2020a ) proposed CROWN-IBP combining IBP forward bounding pass and CROWN style backward bounding pass . CROWN-IBP trained models has a tighter bound compared with IBP models under the IBP metric , with incurring cost on computational efficiency from CROWN backward propagation and generally requiring more epochs for training stability . 2.3 ATTACK-BASED RE-WEIGHTING . Motivated by the idea that all data points are not equally vulnerable to adversarial attack , researchers proposed methods to re-weight the minimax risk by adding a re-weighting term ω ( xi , yi ) ahead of individual example loss . For instance , Zeng et al . ( 2020 ) noted the adversarial distribution deviation from the clean examples and assigned weights that monotonically decreases with the examples ’ confidence margin . The “ confidence margin ” is calculated by attack methods such as PGD , then the risk is re-weighted by a parameterized exponential family as : minθ 1 n ∑n i=1 ω ( xi , yi ) l ( fθ ( x ′ i ) , yi ) , s.t . ω ( xi , yi ) = exp ( −α margin ( fθ , xi + δi , yi ) ) , where α is a positive hyper-parameter and the new risk biases larger weights towards the mis-classified examples . Zhang et al . ( 2020c ) propose GAIRAT ( Geometry-Aware Adversarial Training ) , a method to reweight adversarial examples based on how close they are to the decision boundary . During the training process , GAIRAT explicitly assigns larger/smaller weights to data points closer/farther to the decision boundary respectively : ω ( xi , yi ) = ( 1 + tanh ( λ + 5 × ( 1 − 2 × κ ( xi , yi ) /K ) /2 , where λ is a hyper-parameter , K is the maximal allowed attack iteration and κ is the least iteration number that the attack method requires to fool the classifier . Similar to re-weighting , Wang et al . ( 2021 ) improves clean image performance by prioritizing the robustness between the most dissimilar groups 2.4 CUSTOMIZED ADVERSARIAL TRAINING . In most adversarial training methods , the adversarial attack strength usually follows a pre-defined scheduler throughout the training process . For instance , the perturbation is a uniform number for all examples and usually gradually increases . Cheng et al . ( 2020 ) argued that this assumption may be problematic given the fact that adversarial examples are not equally vulnerable and proposed an auto-tuning method by assigning individual i to each data and increasing it if the current attack is successful . In Zhang et al . ( 2020b ) , they proposed friendly adversarial training ( FAT ) by progressively increasing the attack perturbation with early-stop PGD and alleviating the issue of strong adversarial attack and cross-over mixture .

Summary: This paper proposes bound-based weighted loss and epsilon auto-tuning to improve the performance of certifiable training. The insights of the improvements are mainly borrowed from well-developed adversarial training while they are customized for certifiable training considering bound margins provided bound propagation methods. The experimental results clearly show the improvements from individuals and the combination of the two methods.