SPROUT: Self-Progressing Robust Training

1 INTRODUCTION . While deep neural networks ( DNNs ) have achieved unprecedented performance on a variety of datasets and across domains , developing better training algorithms that are capable of strengthening model robustness is the next crucial milestone toward trustworthy and reliable machine learning systems . In recent years , DNNs trained by standard algorithms ( i.e. , the natural models ) are shown to be vulnerable to adversarial input perturbations ( Biggio et al. , 2013 ; Szegedy et al. , 2014 ) . Adversarial examples crafted by designed input perturbations can easily cause erroneous decision making of natural models ( Goodfellow et al. , 2015 ) and thus intensify the demand for robust training methods . State-of-the-art robust training algorithms are primarily based on the methodology of adversarial training ( Goodfellow et al. , 2015 ; Madry et al. , 2018 ) , which calls specific attacking algorithms to generate adversarial examples during model training in order to learn robust models . Albeit effective , adversarial training based methods have the following limitations : ( i ) poor scalability – the process of generating adversarial examples incurs considerable computation overhead . For instance , our own experiments show that , with the same computation resources , standard adversarial training ( with 7 attack iterations per sample in every minibatch ) of Wide ResNet on CIFAR-10 consumes 10 times more clock time per training epoch when compared with standard training ; ( ii ) attack specificity – adversarially trained models are usually most effective against the same attack they trained on , and the robustness may not generalize well to other types of attacks ( Tramèr & Boneh , 2019 ; Kang et al. , 2019 ) ; ( iii ) preference toward wider network – adversarial training are more effective when the networks have sufficient capacity ( e.g. , having more neurons in network layers ) ( Madry et al. , 2018 ) . To address the aforementioned limitations , in this paper we propose a new robust training method named SPROUT , which is short for self-progressing robust training . We motivate SPROUT by introducing a general framework that formulates robust training objectives via vicinity risk minimization ( VRM ) , which includes many robust training methods as special cases . It is worth noting that the robust training methodology of SPROUT is fundamentally different from adversarial training , as SPROUT features self-adjusted label distribution during training instead of attack generation . In addition to our proposed parametrized label smoothing technique for progressive adjustification of training label distribution , SPROUT also adopts Gaussian augmentation and Mixup ( Zhang et al. , 2018 ) to further enhance robustness . In contrast to adversarial training , SPROUT spares the need for attack generation and thus makes its training scalable by a significant factor , while attaining better or comparable robustness performance on a variety of experiments . We also show that SPROUT can find robust models from either randomly initialized models or pretrained models , and its robustness performance is less sensitive to network width . 1.1 CONTRIBUTIONS . Multi-dimensional performance enhancement . To illustrate the advantage of SPROUT over adversarial training and its variations , Figure 1 compares the model performance of different training methods with the following five dimensions summarized from our experimental results : ( i ) Clean Acc – standard test accuracy , ( ii ) L inf Acc – accuracy under ` ∞-norm projected gradient descent ( PGD ) attack ( Madry et al. , 2018 ) , ( iii ) C & W Acc – accuracy under ` 2-norm Carlini-Wagner ( C & W ) attack , ( iv ) scalability – per epoch clock run-time , and ( v ) invariance – invariant transformation tests including rotation , brightness , contrast and gray images . Comparing to PGD- ` ∞ based adversarial training ( Madry et al. , 2018 ) and TRADES ( Zhang et al. , 2019 ) , SPROUT attains at least 20 % better L inf Acc , 2 % better Clean Acc , 5× faster run-time ( scalability ) , 2 % better invariance , while maintaining C & W Acc , suggesting a new robust training paradigm that is scalable and comprehensive . We further summarize the main contributions of this paper as follows : •We propose SPROUT , a self-progressing robust training method composed of three modules that are efficient and free of attack generation : parametrized label smoothing , Gaussian augmentation and Mixup ( Section 3 ) . They altogether attain the state-of-the-art robustness performance and are scalable to large-scale networks . In Section 4.6 we will show that these modules are complimentary to enhancing robustness . We also perform an ablation study to demonstrate that our proposed parametrized label smoothing technique contributes to the major gain in boosting robustness . • To provide technical explanations for SPROUT , in Section 2 we motivate its training methodology based on the framework of vicinity risk minimization ( VRM ) . We show that many robust training methods , including attack-specific and attack-independent approaches , can be characterized as a specific form of VRM . The superior empirical results of SPROUT provide new insights on developing efficient robust training methods and theoretical justification based on VRM . •We evaluate the multi-dimensional performance of different training methods on ( wide ) ResNet and VGG networks using CIFAR-10 and ImageNet datasets . Notably , although SPROUT is attackindependent during training , we find that SPROUT significantly outperforms two major adversarial training methods , PGD- ` ∞ adversarial training ( Madry et al. , 2018 ) and TRADES ( Zhang et al. , 2019 ) , against the same type of attacks they used during training ( Section 4.2 ) . Moreover , SPROUT is more scalable and runs at least 5× faster than adversarial training methods ( Section 4.5 ) . It also attains higher clean accuracy and generalizes better to various invariance tests ( Section 4.4 ) and is less sensitive to network width ( Section 4.6 ) . 1.2 RELATED WORK . Attack-specific robust training . The seminal work of adversarial training with a first-order attack algorithm for generating adversarial examples ( Madry et al. , 2018 ) has greatly improved adversarial robustness under the same threat model ( e.g. , ` ∞-norm bounded perturbations ) as the attack algorithm . It has since inspired many advanced adversarial training algorithms with improved robustness . For instance , TRADES ( Zhang et al. , 2019 ) is designed to minimize a theoretically-driven upper bound on prediction error in adversarial examples , which led to the first-ranked defense in the NeurIPS 2018 Adversarial Vision Challenge . Bilateral adversarial training ( Wang , 2019 ) finds robust models by adversarially perturbing the data samples as well as the associated data labels using attack algorithms . A feature-scattering based adversarial training method is proposed in ( Zhang & Wang , 2019 ) . Different from attack-specific robust training methods , our proposed SPROUT is free of attack generation , yet it can outperform attack-specific methods . Another line of recent works uses an adversarially trained model along with additional unlabeled data ( Carmon et al. , 2019 ; Stanforth et al. , 2019 ) or self-supervised learning with adversarial examples ( Hendrycks et al. , 2019 ) to improve robustness , which in principle can also be used in SPROUT but is beyond the scope of this paper . Attack-independent robust training . Here we discuss related works on Gaussian data augmentation , Mixup and label smoothing . Gaussian data augmentation during training is a commonly used baseline method to improve model robustness ( Zantedeschi et al. , 2017 ) . It is revisited in ( Cohen et al. , 2019 ) as a scalable and certifiable defense method called random smoothing . Mixup ( Zhang et al. , 2018 ) and its variants ( Verma et al. , 2018 ; Thulasidasan et al. , 2019 ) are a recently proposed approach to improve model robustness and generalization by training a model on convex combinations of data sample pairs and their labels . Label smoothing was originally proposed in ( Szegedy et al. , 2016 ) as a regularizer to stabilize model training . The main idea is to replace one-hot encoded labels by assigning non-zero ( e.g. , uniform ) weights to every label other than the original training label . Although label smoothing is also shown to benefit model robustness ( Shafahi et al. , 2018 ; Goibert & Dohmatob , 2019 ) , its robustness gain is relatively marginal when compared to adversarial training . In contrast to current static ( i.e. , pre-defined ) label smoothing function , in SPROUT we propose a novel parametrized label smoothing scheme , which enables adaptive sampling of training labels from a parameterized distribution on the label simplex . The parameters of the label distribution are progressively adjusted according to the updates of model weights . 2 GENERAL FRAMEWORK FOR FORMULATING ROBUST TRAINING . In supervised learning , the task is essentially learning a K-class classification function f ∈ F that has a desirable mapping between a data sample x ∈ X and the corresponding label y ∈ Y . Consider a loss function L that penalizes the difference between the prediction f ( x ) and the true label y from an unknown data distribution P , ( x , y ) ∼ P . The population risk can be expressed as R ( f ) = ∫ L ( f ( x ) , y ) P ( x , y ) dxdy ( 1 ) However , as the distribution P is unknown , in practice machine learning uses empirical risk minimization ( ERM ) with the empirical data distribution of n training data { xi , yi } ni=1 Pδ ( x , y ) = 1 n n∑ i=1 δ ( x = xi , y = yi ) ( 2 ) to approximate P ( x , y ) , where δ is a Dirac mass . However , a more principled approach is to use Vicinity Risk Minimization ( VRM ) ( Chapelle et al. , 2001 ) , which is Pν ( x , y ) = 1 n n∑ i=1 ν ( x̃ , ỹ|xi , yi ) ( 3 ) where ν is a vicinity distribution that measures the probability of finding the virtual sample-label pair ( x̃ , ỹ ) in the vicinity of the training sample-label pair ( xi , yi ) . Therefore , ERM can be viewed as a special case of VRM when ν = δ. VRM has also been used to motivate Mixup training ( Zhang et al. , 2018 ) . Based on VRM , we propose a general framework that encompasses the objectives of many robust training methods as the following generalized cross entropy loss : H ( x̃ , ỹ , f ) = − K∑ k=1 [ log g ( f ( x̃ ) k ) ] h ( ỹk ) ( 4 ) where f ( x̃ ) k is the model ’ s k-th class probability output of the input x̃ , g ( · ) : R→ R is a mapping adjusting the output probability , and h ( · ) : R → R is a mapping adjusting the training label distribution . When x̃ = x , ỹ = y and g = h = I , where I denotes the identity mapping function , the loss in ( 4 ) degenerates to the conventional cross entropy loss used in ERM . Based on the general VRM loss formulation in ( 4 ) , in Table 1 we summarize a large body of robust training methods in terms of different expressions of g ( · ) , h ( · ) and ( x̃ , ỹ ) . For example , the vanilla adversarial training in ( Madry et al. , 2018 ) aims to minimize the loss of adversarial examples generated by the ( multi-step ) PGD attack with perturbation budget , denoted by PGD ( · ) . Its training objective can be rewritten as x̃ = PGD ( x ) , ỹ = y and g = h = I . In addition to adversarial training only on perturbed samples of x , Wang ( 2019 ) designs adversarial label perturbation where it uses x̃ = PGD ( x ) , ỹ = ( 1 − α ) y + αPGD ( y ) , and α ∈ [ 0 , 1 ] is a mixing parameter . TRADES ( Zhang et al. , 2019 ) improves adversarial training with an additional regularization on the clean examples , which is equivalent to replacing the label mapping function h ( · ) from identity to ( 1 − α ) y + αf ( x̃ ) . Label smoothing ( LS ) alone is equivalent to the setup that g = I , x̃ = x , ỹ = y and h ( · ) = ( 1 − α ) y + αu , where u is often set as a uniform vector with value 1/K for a K-class supervised learning task . Joint training with Gaussian augmentation ( GA ) and label smoothing ( LS ) as studied in ( Shafahi et al. , 2018 ) is equivalent to the case when x̃ = N ( x , ∆2 ) , ỹ = y , g = I and h ( y ) = ( 1 − α ) y + α/K . We defer the connection between SPROUT and VRM to the next section .

Summary: This paper proposes a novel training method to build robust models. A new framework SPROUT is introduced to adjust label distribution during training. It also integrates mixup and Gaussian augmentation to further improve the robustness. The proposed method is built upon the Vicinity Risk Minimization (VRM) framework. Experiments show that the proposed method significantly outperforms the existing best methods in terms of robustness against attacks.