Perturbation Deterioration: The Other Side of Catastrophic Overfitting

1 INTRODUCTION . While neural networks keep advancing the state of the arts , their vulnerability to adversarial attacks ( Szegedy et al. , 2013 ) casts a shadow over their applications—subtle , human-imperceptible input shifts can fool these models and alter their predictions . Adversarial training adds perturbations to model inputs during training , and is one of the most successful approaches to establish model robustness ( Goodfellow et al. , 2014 ; Madry et al. , 2017 ; Kurakin et al. , 2017 ; Tramèr et al. , 2017 ; Zhang et al. , 2019b ; Liu et al. , 2018 ) . One crucial and common challenge in adversarial training is the significant computation overhead , e.g. , it may take 3-30 times longer to conduct adversarial training than the vanilla training . In response to this challenge , there has been a recent surge in work aiming to reduce the computation overhead ( Goodfellow et al. , 2014 ; Shafahi et al. , 2019a ; Zhang et al. , 2019a ; Wong et al. , 2020 ) . Although these methods successfully accelerates adversarial training , they lead to an unexpected phenomenon—when conducting FGSM-style adversarial training , the model robustness would abruptly drop to zero ( Rice et al. , 2020 ) at certain epoch . This phenomenon is referred as catastrophic overfitting , and the robustness drop is usually viewed as model overfitting—the model overfits to one specific type of perturbation ( Rice et al. , 2020 ; Wong et al. , 2020 ; Kim et al. , 2021 ) . Here , we show that the robustness drop is a twin process . Besides the model overfitting to one type of perturbation , we observe perturbations becoming too weak to establish model robustness . For example , as visualized in Figure 1 , both the model and the perturbation change dramatically between Epoch 15 and Epoch 16—the robust accuracy of the model drops from 45 to almost zero , the perturbation strength deteriorates dramatically . As implied by this phenomenon , we suggest that there exists strong correlations between the catastrophic overfitting and perturbation deterioration . Intuitively , without perturbation deterioration , even if the model overfits to one type of perturbation with a reasonable strength , it would lead to a sub-optimal robustness instead of entirely diminished robust accuracy . Meanwhile , once the perturbation deteriorates into random noise , overfitting to that random noise-like perturbation could cause the model robustness drop to zero . 1Code will be released under the Apache-2.0 license for future studies . From this perspective , the key to prevent catastropic overfitting falls upon shielding models from perturbation deterioration . Correspondingly , we design an adaptive adversarial training method , APART , which parameterizes perturbation generation and updates its parameters to progressively strengthen the perturbation generator with gradient ascent . Specifically , we first treat perturbation initialization as parameters—instead of starting from scratch every time , APART improve the initialization with gradient ascent . Then , APART factorize the input perturbation as a series of perturbations , which are integrated with learnable step sizes and can self-adapt to different scenarios . In our experiments , APART leads to consistent performance improvements over FGSM-style algorithms while maintaining roughly the same efficiency . 2 PRELIMINARIES AND NOTATIONS . Given a neural network with n convolution blocks , we denote the input of the i-th block as xi , the input and output of the entire network as x and y . Note that , x and x1 are the same in conventional residual networks , such as ResNet ( He et al. , 2016a ) , Wide ResNet ( Zagoruyko & Komodakis , 2016 ) , and Pre-Act ResNet ( He et al. , 2016b ) . Adversarial training aims to establish the model robustness by solving the optimization problem as below ( θ is the network parameter , δ is the perturbation , ( x , y ) is a data-label pair , and f ( · ) is the perturbation generation function ) . min θ L ( θ , x + δ , y ) s.t . δ = f ( θ , x , y ) . ( 1 ) Different adversarial training methods generate perturbations differently . Ideally , adversarial training should use the most effective ( i.e. , strongest ) perturbation within the same norm constraint , i.e. , f∗ ( · ) = argmax||δ||≤ L ( θ , x+δ , y ) . In practice , as an approximation , f ( · ) is typically implemented as gradient ascent with fixed iteration and step size . For example , the FGSM algorithm directly calculates perturbations as fFGSM ( θFGSM , x , y ) = · sign ( ∆x ) . Also , we use θ ( i ) method to refer to model parameters that are trained by a specific adversarial training algorithm for i epochs . We use Acc ( θ ( i1 ) A , fB ( θ ( i2 ) C , · ) ) to indicate the performance of model θ ( i1 ) A , under fB ( θ ( i2 ) C , · ) ) , i.e. , the perturbation generated by method fB ( · ) to attack θ ( i2 ) C . These notations are summarized in Table 1 . 3 PERTURBATION DETERIORATION AND CATASTROPHIC OVERFITTING . Typically , the robustness drop is viewed as model overfitting ( Rice et al. , 2020 ; Wong et al. , 2020 ; Kim et al. , 2021 ) and this phenomenon is referred as catastrophic overfitting . Meanwhile , it has been observed that catastrophic overfitting only happens to simple methods like FGSM ( Goodfellow et al. , 2014 ) 2 , indicating this phenomenon is not only about model overfitting , but also the strength of perturbation . Inspired by this observation , we aim to explore the other side of catastophic overfitting , i.e. , perturbation deterioration . 3.1 PERTURBATION STRENGTH . To verify our intuition , we try to empirically estimate the perturbation strength and analyze its dynamics during the adversarial training . First , we try to estimate the perturbation strength of fA as Acc ( θ30PGD-10 , fA ( θ i A , · ) ) , i.e. , using fA ( θiA ) to attack a model trained separately with PGD-10 for 30 epochs . For comparisons , we also list the performance of θ30PGD-10 under random noise . As visualized in Figure 2 ( a ) , in the first 15 epochs , the perturbation strength of FGSM keeps increasing , while at the 16 epoch , its strength dramatically drops to the random noise level . This shows that , after the perturbation deterioration , the perturbations not only look like random noise , but also behaves like random noise . Alternatively , we estimate the perturbation strength of fA as its gap to a more powerful method fB . Specifically , we calculate the strength for fA ( θ ( i ) A , · ) as GB , i ( A ) in Equation 2 . Intuitively , the weaker the perturbation fA is , the larger GB , i ( A ) would be . GB , i ( A ) = L ( δA ) − L ( δB ) where L ( δmethod ) = L ( θ ( i ) A ; x + fmethod ( θ ( i ) A , x , y ) , y ) . ( 2 ) We conduct experiments with Pre-Act ResNet18 on the CIFAR-10 dataset and visualize GPGD-10 , i ( FGSM ) in Figure 2 ( b ) . It shows that the strength gap between PGD-10 and FGSM is small in the early stage , dramatically explodes at the 16th epoch , and keeps a large value since then ( i.e. , the perturbation strength mostly vanishes at the 16th epoch ) . Perturbation Deterioration . In both cases , the timing of perturbation strength deterioration coincides with the timing of the robustness drop , thus supporting our intuition that the other side of 2For a FGSM trained Pre-ResNet18 ( on CIFAR-10 ) , its accuracy under PGD-20 attacks drops from 45 % ( epoch 15 ) to 0 % ( epoch 16 ) ; For a PGD-10 trained model , it drops from 49 % ( epoch 30 ) to 38 % ( epoch 200 ) . Algorithm 1 : APART ( the first and the second round propagations are marked with red and blue ; is the perturbation bound ; µθ , µω , and µα are learning rates for θ , ω , and α ; Table 1 summarizes others notations ) . 1 while not converged do 2 for x , y in the training set do 3 δ1 ← αω · ωx //initialize the perturbation for the model input . 4 δ1 ← max ( min ( δ1 + α1 · sign ( ∂L ( θ ; x+δ1 , y ) ∂x1 ) , − ) , + ) //calculate δ1 . 5 for each residual block with index i > 1 do 6 δi ← αi · sign ( ∂L ( θ ; x+δ1 , y ) ∂xi ) //calculate perturbations for block i . 7 θ = θ − µθ · ∂L ( θ ; { xi } n i=1+ { δi } n i=1 , y ) ∂θ //update parameters . 8 ωx = max ( min ( ωx + µω · sign ( ∂L ( θ ; { xi } n i=1+ { δi } n i=1 , y ) ∂ωx ) , −1 ) , 1 ) //update ωx . 9 αi = αi + µα · ( ∂L ( θ ; { xi } n i=1+ { δi } n i=1 , y ) ∂αi − λ · ∂|αi| 2 2 ∂αi ) //update step sizes . 10 return θ catastrophic overfitting is perturbation deterioration . Intuitively , adversarial training can not establish satisfying model robustness without strong enough perturbations , and strong perturbations require a small gap G ( · ) . However , one local optima of Equation 1 is the parameter θ that deteriorates f ( θ , · ) into random noise , which advances the optimization of Equation 1 at the cost of deteriorated perturbations . Also , since all parameter updates are made to decrease L , most existing methods have no regularization to keep the perturbation strength . 3.2 ADAPTIVITY HELPS PREVENT CATASTROPHIC OVERFITTING . Here , we further verify our intuition by showing that it prevents the catastrophic overfitting by alleviating the perturbation deterioration . Intuitively , one straightforward way to strengthen generators is to parameterize them and update them together with model parameters . Specifically , we treat the perturbation initialization for the input ( denoted as ωx ) and the step size ( referred as α ) as parameters of FGSM , and change the objective from Equation 1 to : max α , ω min θ L ( θ ; x + fFGSM ( α , ωx ; θ , x , y ) , y ) . ( 3 ) During model training , we update θ with gradient descent and update α and ωx with gradient ascent . We refer this variant as APART-Simple . Note that its only difference to FGSM is that APARTSimple is parameterized and can be enhanced during training , thus suffering less from perturbation deterioration . We conduct experiments with Pre-Act ResNet18 on the CIFAR-10 dataset , and visualize Acc ( θ ( 30 ) PGD-10 , fFGSM ( θ ( i ) APART-Simple , · ) ) in Figure 2 ( a ) and GPGD-10 , i ( APART-Simple ) in Figure 2 ( b ) . It shows that APART-Simple does not suffer from the catastrophic overfitting . This not only supports our intuition that the perturbation deterioration is one cause of the catastrophic overfitting , but motivates us to add more adaptivity to the perturbation generator .

Summary: The paper made two contributions: 1. it identified the co-occurrence of abrupt robustness drop and perturbation degrading to (potentially) random noise; 2. it proposed a computationally faster adversarial training algorithm while attaining a good accuracy. Point 1. is interesting: it suggests previously identified "catastrophic overfitting" may be due to the perturbation in training time fails to mimic those in test time in distribution, while the adversarial training is being performed. This is different from classical empirical risk optimizations, where the training data do not depend on the training processes.