Imbalanced Adversarial Training with Reweighting

1 INTRODUCTION . The existence of adversarial samples ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ) has risen huge concerns on applying deep neural network ( DNN ) models into security-critical applications , such as autonomous driving ( Chen et al. , 2015 ) and video surveillance systems ( Kurakin et al. , 2016 ) . As countermeasures against adversarial attacks , adversarial training ( Madry et al. , 2017 ; Zhang et al. , 2019 ; Wang et al. , 2019 ) has been empirically proven to be one of the most effective and reliable defense methods . In general , it can be formulated to minimize the model ’ s average error on adversarially perturbed input examples ( Madry et al. , 2017 ) . Although promising to improve the model ’ s robustness , most existing adversarial training methods assume that the number of training examples from each class is equally distributed . However , datasets collected from real-world applications typically have imbalanced distribution ( Everingham et al. , 2010 ; Lin et al. , 2014 ) . Hence , it is natural to ask : What is the behavior of adversarial training under imbalanced scenarios ? Can we directly apply existing imbalanced learning strategies in natural training to tackle the imbalance issue for adversarial training ? Recent studies find that adversarial training usually presents distinct properties from natural training . For example , compared to natural training , adversarially trained models suffer more from the overfitting issue ( Schmidt et al. , 2018 ) , and they tend to present strong class-wise performance disparities , even if the training examples are uniformly distributed over different classes ( Xu et al. , 2020a ) . Imagine that if the training data distribution is highly imbalanced , these properties of adversarial training can be greatly exaggerated and make it extremely difficult to be applied in practice . Therefore , it is necessary but challenging to answer aforementioned questions . As the initial effort to study the imbalanced problem in adversarial training , in this work , we first investigate the performance of existing adversarial training under imbalanced settings . As a preliminary study shown in Section 2.1 , we apply both natural training and PGD adversarial training ( Madry et al. , 2017 ) on multiple imbalanced training datasets constructed from CIFAR10 training dataset ( Krizhevsky et al. , 2009 ) and evaluate trained models ’ performance on class-balanced test dataset . From the preliminary results , we observe that , compared to naturally trained models , adversarially trained models always present very low standard & robust accuracy1 on under-represented classes . This observation suggests that adversarial training is more sensitive to imbalanced data distribution than natural training . Thus , when applying adversarial training in practice , imbalance learning strategies should always be considered for help . As a result , we explore potential solutions which can handle the imbalance issue for adversarial training . In this work , we focus on studying the behavior of the reweighting strategy ( He & Ma , 2013 ) and leave other strategies such as resampling ( Estabrooks et al. , 2004 ) for one future work . In Section 2.2 , we apply the reweighting strategy to adversarial training with varied weights assigning to one under-represented class and evaluate trained models ’ performance . From the results , we observe that , in adversarial training , increasing weights for an under-represented class can substantially improve the standard & robust accuracy on this class , but drastically hurt the model ’ s performance on the well-represented class . This finding indicates that the performance of adversarially trained models is very sensitive to the reweighting manipulations and it could be very hard to figure out an eligible reweighting strategy which is optimal for all classes . It is also worth noting that , in natural training , we find that upweighting the under-represented class increases model ’ s standard accuracy on this class but only slightly hurts the accuracy on the well-represented class , even when adopting a large weight for the under-represent class . To further investigate the possible reasons leading to different behaviors of the reweighing strategy in natural and adversarial training , we visualize their learned features ( in Figure 3 ) , and observe that features learned by the adversarially trained model of different classes tend to mix together while they are well separated for the naturally trained model . This observation motivates us to theoretically show that when the given data distribution has poor data separability , upweighting under-represented classes will hurt the model ’ s performance on well-represented classes . Motivated by our theoretical understanding , we propose a novel framework Separable Reweighted Adversarial Training ( SRAT ) to facilitate the reweighting strategy in imbalanced adversarial training by enhancing the separability of learned features . Through extensive experiments , we validate the effectiveness of SRAT . 2 PRELIMINARY STUDY . 2.1 THE BEHAVIOR OF ADVERSARIAL TRAINING . In this subsection , we conduct preliminary studies to examine the performance of PGD adversarial training ( Madry et al. , 2017 ) . Following previous works ( Cui et al. , 2019 ; Cao et al. , 2019 ) , we construct an imbalanced CIFAR10 ( Krizhevsky et al. , 2009 ) training dataset , where each of the first 5 classes ( a.k.a . well-represented classes ) has 5,000 training examples and each of the last 5 classes ( a.k.a . under-represented classes ) has 50 training examples . Figure 1 shows the performance of naturally and adversarially trained models using a ResNet18 ( He et al. , 2016 ) architecture . From the figure , we can observe that , compared with natural training , PGD adversarial training will result in a larger performance gap between well-represented classes and under-represented classes . For example , in natural training , the ratio between the average standard 1In this work , we denote standard accuracy as model ’ s accuracy on the input samples without perturbations and robust accuracy as model ’ s accuracy on the input samples which are adversarially perturbed . Without clear clarification , we consider the perturbation is constrained by l∞-norm 8/255 . accuracy of well-represented classes ( brown ) and under-represented classes ( violet ) is about 2:1 , while in adversarial training , this ratio expands to 16:1 . Moreover , for adversarial training , it has extremely poor performance on under-represented classes . There are 3 out of the 5 under-represented classes with 0 % standard & robust accuracy . As a conclusion , the performance of adversarial training is easier to be affected by imbalanced distribution than natural training and suffers more on under-represented classes . More results are reported in Appendix A.1 , which further support our findings . 2.2 THE REWEIGHTING STRATEGY IN NATURAL TRAINING V.S . IN ADVERSARIAL TRAINING . The preliminary study in Section 2.1 demonstrates that it is highly demanding to adjust the original adversarial training methods to accommodate imbalanced data distribution . Next , we investigate the effectiveness of adopting the reweighting strategy ( He & Ma , 2013 ) in adversarial training . Our experiments are conducted under a binary classification setting , where the training dataset contains two classes that are randomly selected from CIFAR10 dataset , with each class having 5,000 and 50 training examples respectively . Based on this training dataset , we arrange multiple trails of ( reweighted ) natural training and ( reweighted ) adversarial training , with the weight ratio between the under-represented class and well-represented class ranging from 1:1 to 200:1 . Figure 2 shows the experimental results with training data sampled from the classes “ cat ” and “ horse ” . As demonstrated in Figure 2 , increasing the weight for the under-represented class ( horse ) will drastically increase the model ’ s performance on this class , while also immensely decreasing the performance on the well-represented class ( cat ) . For example , when increasing the weight ratio from 1:1 to 150:1 , the standard accuracy of the under-represented class is improved from 0 % to∼ 60 % and its robust accuracy from 0 % to ∼ 50 % . However , the standard accuracy on the well-represented class drops from 100 % to 60 % , and its robust accuracy drops from 100 % to 50 % . These results illustrate that adversarial training ’ s performance can be significantly affected by the reweighting strategy . As a result , the reweighting strategy in this setting can hardly help improve the overall performance no matter which weight ratio is chosen , because the model ’ s performance always presents a strong tension between these two classes . More experiments using different binary imbalanced datasets are reported in Appendix A.2 , where we have similar observations . 3 THEORETICAL ANALYSIS . In Section 2.2 , we observe that in natural training , the reweighting strategy can only make a small impact on the two classes ’ performance . This phenomenon has been extensively studied by recent works ( Byrd & Lipton , 2019 ; Xu et al. , 2021 ) , where they find that a linear classifier optimized by SGD on a linearly separable data will converge to the solution of the hard-margin support vector machine ( Noble , 2006 ) . In other words , as long as the data can be well separated , reweighting will not make huge influence on the finally trained models . Inspired by their conclusions , we hypothesize that , as the adversarially trained models separate the data poorly , their performance is highly sensitive to the reweighting strategy . As a direct validation of our hypothesis , in Figure 3 , we visualize the learned ( penultimate layer ) features of the imbalanced training examples used in the binary classification problem in Section 2.2 . We find that adversarially trained models do present obviously poorer separability on the learned features . Next , we theoretically analyze the impact of reweighting on linear models which are optimized under poorly separable data and provide all detailed proof in Appendix A.3 . Binary Classification Problem . To construct the theoretical study , we focus on a binary classification problem , with a Gaussian mixture distribution D which is defined as : y ∼ { −1 , +1 } , x ∼ { N ( µ , σ2I ) , if y = +1 N ( −µ , σ2I ) , if y = −1 and µ = ( dim=d︷ ︸︸ ︷ η , ... , η ) , ( 1 ) where the two classes ’ centers ( ±µ ∈ Rd ) with each dimension have mean value ±η ( η > 0 ) and variance σ2 . Formally , we define the data separability as S = η/σ2 . Intuitively , when S is larger , it suggests that two classes are well separated . Previous work ( Byrd & Lipton , 2019 ) also closely studied this term to describe data separability . Besides , we assume the imbalanced training dataset satisfying the condition Pr . ( y = +1 ) = K · Pr . ( y = −1 ) and K > 1 , which indicates the imbalance ratio between two classes . During test , we assume two classes have the equal probability to appear . Under the data distribution D , we will discuss the performance of linear classifiers f ( x ) = sign ( wTx− b ) where w and b are the weight and bias terms of the model f . If a reweighting strategy is involved , we define the model upweights the under-represented class “ -1 ” by ρ. Lemma 3.1 Under the data distribution D as defined in Eq . ( 1 ) , with an imbalanced ratio K and a reweight ratio ρ , the optimal classifier which minimizes the ( reweighted ) empirical risk : f∗ = argmin f ( Pr . ( f ( x ) ̸=y|y=−1 ) · Pr . ( y=−1 ) · ρ+ Pr . ( f ( x ) ̸=y|y=+1 ) · Pr . ( y=+1 ) ) ( 2 ) has the solution : w = 1 and b = 12 log ( ρ K ) dσ2 η = 1 2 log ( ρ K ) d S . Lemma 3.1 indicates that the final optimized classifier has a weight vector equal to 1 and its bias term b only depends on K , ρ and the data separability S. In the following , we first focus on one special setting when ρ = 1 , which is the original ERM model without reweighting . Specifically , we aim to compare the behavior of linear models when they can poorly separate data ( like adversarial trained models ) or they can well separate data ( like naturally trained models ) . Theorem 3.1 Under two data distributions ( x ( 1 ) , y ( 1 ) ) ∈ D1 and ( x ( 2 ) , y ( 2 ) ) ∈ D2 with different separabilities S1 > S2 , let f∗1 and f ∗ 2 be the optimal non-reweighted classifiers ( ρ = 1 ) under D1 and D2 , respectively . Given the imbalance ratio K is large enough , we have : Pr . ( f∗1 ( x ( 1 ) ) ̸= y ( 1 ) |y ( 1 ) = −1 ) − Pr . ( f∗1 ( x ( 1 ) ) ̸= y ( 1 ) |y ( 1 ) = +1 ) < Pr . ( f∗2 ( x ( 2 ) ) ̸= y ( 2 ) |y ( 2 ) = −1 ) − Pr . ( f∗2 ( x ( 2 ) ) ̸= y ( 2 ) |y ( 2 ) = +1 ) . ( 3 ) Intuitively , Theorem 3.1 suggests that when the data separability S is low ( such as D2 ) , the optimized classifier ( without reweighting ) can intrinsically have a larger error difference between the underrepresented class “ -1 ” and the well-represented class “ +1 ” . Similar to the observation in Section 2.1 and Figure 3 , adversarially trained models present a weak ability to separate data , and they also present a strong performance gap between the well-represented class and under-represented class . Conclusively , Theorem 3.1 indicates that the poor ability to separate the training data can be one important reason which leads to the strong performance gap of adversarially trained models . Next , we consider the case when the reweighting strategy is applied . In particular , we compare the impact of upweighting the under-represented class on the performance of well-represented class . Theorem 3.2 Under two data distributions ( x ( 1 ) , y ( 1 ) ) ∈ D1 and ( x ( 2 ) , y ( 2 ) ) ∈ D2 with different separabilities S1 > S2 , let f∗1 and f ∗ 2 be the optimal non-reweighted classifiers ( ρ = 1 ) under D1 and D2 , respectively , and let f ′1 ∗ and f ′2 ∗ be the optimal reweighted classifiers under D1 and D2 given the optimal reweighting ratio ( ρ = K ) . Given the imbalance ratio K is large enough , we have : Pr . ( f ′1 ∗ ( x ( 1 ) ) ̸= y ( 1 ) |y ( 1 ) = +1 ) − Pr . ( f∗1 ( x ( 1 ) ) ̸= y ( 1 ) |y ( 1 ) = +1 ) < Pr . ( f ′2 ∗ ( x ( 2 ) ) ̸= y ( 2 ) |y ( 2 ) = +1 ) − Pr . ( f∗2 ( x ( 2 ) ) ̸= y ( 2 ) |y ( 2 ) = +1 ) . ( 4 ) As Theorem 3.2 shows , when the data distribution has poorer data separability ( such asD2 ) , upweighting the under-represented class can cause greater hurt on the performance of the well-represented class . It is also consistent with our empirical findings about adversarial training models . Since the adversarially trained models poorly separate the data ( Figure 3 ) , upweighting the under-represented class always drastically decreases the performance of the well-represented class ( Section 2.2 ) . Through the discussions in both Theorem 3.1 and Theorem 3.2 , we conclude that the poor separability can be one important reason which makes adversarial training and its reweighted variants extremely difficult to achieve good performance under imbalance data distribution . Therefore , in the next section , we will explore potential solutions which can facilitate the reweighting strategy in adversarial training .

Summary: This paper studies the effect of adversarial training (AT) in the context of imbalanced datasets. In particular, the authors demonstrate empirically that adversarially trained models tend to have a larger gap in accuracy between well and under represented classes, as compared to their standard counterparts. Moreover, they show that standard reweighting strategies do not considerably alleviate this issue. Motivated by their findings, they propose adding a regularization function to adversarial training to encourage greater feature separation between classes (SRAT), and study its effectiveness on unbalanced datasets.