Year: 2021

Abstract:
Adversarial training has been empirically proven to be one of the most effective and reliable defense methods against adversarial attacks. However, the majority of existing studies are focused on balanced datasets, where each class has a similar amount of training examples. Research on adversarial training with imbalanced training datasets is rather limited. As the initial effort to investigate this problem, we reveal the facts that adversarially trained models present two distinguished behaviors from naturally trained models in imbalanced datasets: (1) Compared to natural training, adversarially trained models can suffer much worse performance on under-represented classes, when the training dataset is extremely imbalanced. (2) Traditional reweighting strategies may lose efficacy to deal with the imbalance issue for adversarial training. For example, upweighting under-represented classes will drastically hurt the model’s performance on well-represented classes, and as a result, finding an optimal reweighting value can be tremendously challenging. In this paper, to further understand our observations, we theoretically show that the poor data separability is one key reason causing this strong tension between underrepresented and well-represented classes. Motivated by this finding, we propose Separable Reweighted Adversarial Training (SRAT) to facilitate adversarial training under imbalanced scenarios, by learning more separable features for different classes. Extensive experiments on various datasets verify the effectiveness of the proposed framework.

heading: 1 INTRODUCTION
text: The existence of adversarial samples (Szegedy et al., 2013; Goodfellow et al., 2014) has risen huge concerns on applying deep neural network (DNN) models into security-critical applications, such as autonomous driving (Chen et al., 2015) and video surveillance systems (Kurakin et al., 2016). As countermeasures against adversarial attacks, adversarial training (Madry et al., 2017; Zhang et al., 2019; Wang et al., 2019) has been empirically proven to be one of the most effective and reliable defense methods. In general, it can be formulated to minimize the model’s average error on adversarially perturbed input examples (Madry et al., 2017). Although promising to improve the model’s robustness, most existing adversarial training methods assume that the number of training examples from each class is equally distributed. However, datasets collected from real-world applications typically have imbalanced distribution (Everingham et al., 2010; Lin et al., 2014). Hence, it is natural to ask: What is the behavior of adversarial training under imbalanced scenarios? Can we directly apply existing imbalanced learning strategies in natural training to tackle the imbalance issue for adversarial training? Recent studies find that adversarial training usually presents distinct properties from natural training. For example, compared to natural training, adversarially trained models suffer more from the overfitting issue (Schmidt et al., 2018), and they tend to present strong class-wise performance disparities, even if the training examples are uniformly distributed over different classes (Xu et al., 2020a). Imagine that if the training data distribution is highly imbalanced, these properties of adversarial training can be greatly exaggerated and make it extremely difficult to be applied in practice. Therefore, it is necessary but challenging to answer aforementioned questions.
As the initial effort to study the imbalanced problem in adversarial training, in this work, we first investigate the performance of existing adversarial training under imbalanced settings. As a preliminary study shown in Section 2.1, we apply both natural training and PGD adversarial training (Madry et al., 2017) on multiple imbalanced training datasets constructed from CIFAR10 training
dataset (Krizhevsky et al., 2009) and evaluate trained models’ performance on class-balanced test dataset. From the preliminary results, we observe that, compared to naturally trained models, adversarially trained models always present very low standard & robust accuracy1 on under-represented classes. This observation suggests that adversarial training is more sensitive to imbalanced data distribution than natural training. Thus, when applying adversarial training in practice, imbalance learning strategies should always be considered for help.
As a result, we explore potential solutions which can handle the imbalance issue for adversarial training. In this work, we focus on studying the behavior of the reweighting strategy (He & Ma, 2013) and leave other strategies such as resampling (Estabrooks et al., 2004) for one future work. In Section 2.2, we apply the reweighting strategy to adversarial training with varied weights assigning to one under-represented class and evaluate trained models’ performance. From the results, we observe that, in adversarial training, increasing weights for an under-represented class can substantially improve the standard & robust accuracy on this class, but drastically hurt the model’s performance on the well-represented class. This finding indicates that the performance of adversarially trained models is very sensitive to the reweighting manipulations and it could be very hard to figure out an eligible reweighting strategy which is optimal for all classes.
It is also worth noting that, in natural training, we find that upweighting the under-represented class increases model’s standard accuracy on this class but only slightly hurts the accuracy on the well-represented class, even when adopting a large weight for the under-represent class. To further investigate the possible reasons leading to different behaviors of the reweighing strategy in natural and adversarial training, we visualize their learned features (in Figure 3), and observe that features learned by the adversarially trained model of different classes tend to mix together while they are well separated for the naturally trained model. This observation motivates us to theoretically show that when the given data distribution has poor data separability, upweighting under-represented classes will hurt the model’s performance on well-represented classes. Motivated by our theoretical understanding, we propose a novel framework Separable Reweighted Adversarial Training (SRAT) to facilitate the reweighting strategy in imbalanced adversarial training by enhancing the separability of learned features. Through extensive experiments, we validate the effectiveness of SRAT.

heading: 2 PRELIMINARY STUDY

heading: 2.1 THE BEHAVIOR OF ADVERSARIAL TRAINING
text: In this subsection, we conduct preliminary studies to examine the performance of PGD adversarial training (Madry et al., 2017). Following previous works (Cui et al., 2019; Cao et al., 2019), we construct an imbalanced CIFAR10 (Krizhevsky et al., 2009) training dataset, where each of the first 5 classes (a.k.a. well-represented classes) has 5,000 training examples and each of the last 5 classes (a.k.a. under-represented classes) has 50 training examples.
Figure 1 shows the performance of naturally and adversarially trained models using a ResNet18 (He et al., 2016) architecture. From the figure, we can observe that, compared with natural training, PGD adversarial training will result in a larger performance gap between well-represented classes and under-represented classes. For example, in natural training, the ratio between the average standard
1In this work, we denote standard accuracy as model’s accuracy on the input samples without perturbations and robust accuracy as model’s accuracy on the input samples which are adversarially perturbed. Without clear clarification, we consider the perturbation is constrained by l∞-norm 8/255.
accuracy of well-represented classes (brown) and under-represented classes (violet) is about 2:1, while in adversarial training, this ratio expands to 16:1. Moreover, for adversarial training, it has extremely poor performance on under-represented classes. There are 3 out of the 5 under-represented classes with 0% standard & robust accuracy. As a conclusion, the performance of adversarial training is easier to be affected by imbalanced distribution than natural training and suffers more on under-represented classes. More results are reported in Appendix A.1, which further support our findings.

heading: 2.2 THE REWEIGHTING STRATEGY IN NATURAL TRAINING V.S. IN ADVERSARIAL TRAINING
text: The preliminary study in Section 2.1 demonstrates that it is highly demanding to adjust the original adversarial training methods to accommodate imbalanced data distribution. Next, we investigate the effectiveness of adopting the reweighting strategy (He & Ma, 2013) in adversarial training. Our experiments are conducted under a binary classification setting, where the training dataset contains two classes that are randomly selected from CIFAR10 dataset, with each class having 5,000 and 50 training examples respectively. Based on this training dataset, we arrange multiple trails of (reweighted) natural training and (reweighted) adversarial training, with the weight ratio between the under-represented class and well-represented class ranging from 1:1 to 200:1.
Figure 2 shows the experimental results with training data sampled from the classes “cat” and “horse”. As demonstrated in Figure 2, increasing the weight for the under-represented class (horse) will drastically increase the model’s performance on this class, while also immensely decreasing the performance on the well-represented class (cat). For example, when increasing the weight ratio from 1:1 to 150:1, the standard accuracy of the under-represented class is improved from 0% to∼ 60% and its robust accuracy from 0% to ∼ 50%. However, the standard accuracy on the well-represented class drops from 100% to 60%, and its robust accuracy drops from 100% to 50%. These results illustrate that adversarial training’s performance can be significantly affected by the reweighting strategy. As a result, the reweighting strategy in this setting can hardly help improve the overall performance no matter which weight ratio is chosen, because the model’s performance always presents a strong tension between these two classes. More experiments using different binary imbalanced datasets are reported in Appendix A.2, where we have similar observations.

heading: 3 THEORETICAL ANALYSIS
text: In Section 2.2, we observe that in natural training, the reweighting strategy can only make a small impact on the two classes’ performance. This phenomenon has been extensively studied by recent works (Byrd & Lipton, 2019; Xu et al., 2021), where they find that a linear classifier optimized by SGD on a linearly separable data will converge to the solution of the hard-margin support vector machine (Noble, 2006). In other words, as long as the data can be well separated, reweighting will not make huge influence on the finally trained models.
Inspired by their conclusions, we hypothesize that, as the adversarially trained models separate the data poorly, their performance is highly sensitive to the reweighting strategy. As a direct validation of our hypothesis, in Figure 3, we visualize the learned (penultimate layer) features of the imbalanced training examples used in the binary classification problem in Section 2.2. We find that adversarially trained models do present obviously poorer separability on the learned features. Next, we theoretically analyze the impact of reweighting on linear models which are optimized under poorly separable data and provide all detailed proof in Appendix A.3.
Binary Classification Problem. To construct the theoretical study, we focus on a binary classification problem, with a Gaussian mixture distribution D which is defined as:
y ∼ {−1,+1}, x ∼ { N (µ, σ2I), if y = +1 N (−µ, σ2I), if y = −1 and µ = ( dim=d︷ ︸︸ ︷ η, ..., η), (1)
where the two classes’ centers (±µ ∈ Rd) with each dimension have mean value ±η (η > 0) and variance σ2. Formally, we define the data separability as S = η/σ2. Intuitively, when S is larger, it suggests that two classes are well separated. Previous work (Byrd & Lipton, 2019) also closely studied this term to describe data separability.
Besides, we assume the imbalanced training dataset satisfying the condition Pr.(y = +1) = K · Pr.(y = −1) and K > 1, which indicates the imbalance ratio between two classes. During test, we assume two classes have the equal probability to appear. Under the data distribution D, we will discuss the performance of linear classifiers f(x) = sign(wTx− b) where w and b are the weight and bias terms of the model f . If a reweighting strategy is involved, we define the model upweights the under-represented class “-1” by ρ.
Lemma 3.1 Under the data distribution D as defined in Eq. (1), with an imbalanced ratio K and a reweight ratio ρ, the optimal classifier which minimizes the (reweighted) empirical risk:
f∗ = argmin f
( Pr.(f(x) ̸=y|y=−1) · Pr.(y=−1) · ρ+ Pr.(f(x) ̸=y|y=+1) · Pr.(y=+1) ) (2)
has the solution: w = 1 and b = 12 log( ρ K )
dσ2
η = 1 2 log( ρ K ) d S .
Lemma 3.1 indicates that the final optimized classifier has a weight vector equal to 1 and its bias term b only depends on K, ρ and the data separability S. In the following, we first focus on one special setting when ρ = 1, which is the original ERM model without reweighting. Specifically, we aim to compare the behavior of linear models when they can poorly separate data (like adversarial trained models) or they can well separate data (like naturally trained models).
Theorem 3.1 Under two data distributions (x(1), y(1)) ∈ D1 and (x(2), y(2)) ∈ D2 with different separabilities S1 > S2, let f∗1 and f ∗ 2 be the optimal non-reweighted classifiers (ρ = 1) under D1 and D2, respectively. Given the imbalance ratio K is large enough, we have:
Pr.(f∗1 (x (1)) ̸= y(1)|y(1) = −1)− Pr.(f∗1 (x(1)) ̸= y(1)|y(1) = +1)
< Pr.(f∗2 (x (2)) ̸= y(2)|y(2) = −1)− Pr.(f∗2 (x(2)) ̸= y(2)|y(2) = +1).
(3)
Intuitively, Theorem 3.1 suggests that when the data separability S is low (such as D2), the optimized classifier (without reweighting) can intrinsically have a larger error difference between the underrepresented class “-1” and the well-represented class “+1”. Similar to the observation in Section 2.1 and Figure 3, adversarially trained models present a weak ability to separate data, and they also present a strong performance gap between the well-represented class and under-represented class. Conclusively, Theorem 3.1 indicates that the poor ability to separate the training data can be one important reason which leads to the strong performance gap of adversarially trained models.
Next, we consider the case when the reweighting strategy is applied. In particular, we compare the impact of upweighting the under-represented class on the performance of well-represented class.
Theorem 3.2 Under two data distributions (x(1), y(1)) ∈ D1 and (x(2), y(2)) ∈ D2 with different separabilities S1 > S2, let f∗1 and f ∗ 2 be the optimal non-reweighted classifiers (ρ = 1) under D1
and D2, respectively, and let f ′1 ∗ and f ′2 ∗ be the optimal reweighted classifiers under D1 and D2 given the optimal reweighting ratio (ρ = K). Given the imbalance ratio K is large enough, we have:
Pr.(f ′1 ∗ (x(1)) ̸= y(1)|y(1) = +1)− Pr.(f∗1 (x(1)) ̸= y(1)|y(1) = +1)
< Pr.(f ′2 ∗ (x(2)) ̸= y(2)|y(2) = +1)− Pr.(f∗2 (x(2)) ̸= y(2)|y(2) = +1).
(4)
As Theorem 3.2 shows, when the data distribution has poorer data separability (such asD2), upweighting the under-represented class can cause greater hurt on the performance of the well-represented class. It is also consistent with our empirical findings about adversarial training models. Since the adversarially trained models poorly separate the data (Figure 3), upweighting the under-represented class always drastically decreases the performance of the well-represented class (Section 2.2). Through the discussions in both Theorem 3.1 and Theorem 3.2, we conclude that the poor separability can be one important reason which makes adversarial training and its reweighted variants extremely difficult to achieve good performance under imbalance data distribution. Therefore, in the next section, we will explore potential solutions which can facilitate the reweighting strategy in adversarial training.

heading: 4 SEPARABLE REWEIGHTED ADVERSARIAL TRAINING (SRAT)
text: The observations from both preliminary study and theoretical understandings indicate that more separable data will advance the reweighting strategy in adversarial training under imbalanced scenarios. Thus, in this section, we present a framework, Separable Reweighted Adversarial Training (SRAT), which enables the effectiveness of the reweighting strategy in adversarial training under imbalanced scenarios by increasing the separability in the learned latent feature space.

heading: 4.1 REWEIGHTED ADVERSARIAL TRAINING
text: Given an input example (x, y), adversarial training (Madry et al., 2017) aims to obtain a robust model fθ that can make the same prediction y for an adversarial example x′, generated by applying an adversarially perturbation on x. The adversarial perturbations are typically bounded by a small value ϵ under Lp-norm, i.e., ∥x′ − x∥p ≤ ϵ. As indicated in Section 2.1, adversarial training cannot be applied in imbalanced scenarios directly, as it presents very low performance on under-represented classes. To tackle this problem, a natural idea is to integrate existing imbalanced learning strategies proposed in natural training, such as reweighting, into adversarial training to improve the trained model’s performance on those under-represented classes. Hence, the reweighted adversarial training can be defined as
min θ
1
n n∑ i=1 max ∥x′i−xi∥p≤ϵ wiL(fθ(x′i), yi), (5)
where wi is a weight value assigned for each input sample (xi, yi) based on the example size of the class (xi, yi) belongs to or some properties of (xi, yi). In most existing adversarial training methods (Madry et al., 2017; Zhang et al., 2019; Wang et al., 2019), the cross entropy (CE) loss is adopted as the loss function L(·, ·). However, the CE loss could be suboptimal in imbalanced settings and some new loss functions designed for imbalanced learning specifically, such as Focal loss (Lin et al., 2017) and LDAM loss (Cao et al., 2019), have been proven their superiority in natural training. Hence, besides CE loss, Focal loss and LDAM loss can also be adopted as the loss function L(·, ·) in Eq. (5).

heading: 4.2 INCREASING FEATURE SEPARABILITY
text: Our preliminary study indicates that only reweighted adversarial training cannot work well under imbalanced scenarios, and the reweighting strategy in adversarial training behaves very differently from natural training. Hence, in order to facilitate the reweighting strategy in adversarial training under imbalanced scenarios, inspired by our theoretical analysis, we equip a feature separation loss with our SRAT method to enforce the learned feature space as separable as possible. More specifically, the goal of our feature separation loss is to make the learned features of examples from the same class well clustered while from different classes well separated. Hence, adjusting the decision boundary via
the reweighting strategy to fit under-represented classes’ examples more will not hurt well-represented classes drastically. The feature separation loss is formally defined as:
Lsep(x′i) = − 1 |P (i)| ∑
p∈P (i)
log exp(z′i · z′p/τ)∑
a∈A(i) exp(z ′ i · z′a/τ)
, (6)
where z′i is the feature representation of the adversarial example x ′ i of xi, τ ∈ R+ is a scalar temperature parameter, P (i) denotes the set of input examples belonging to the same class with xi and A(i) indicates the set of all input examples except x′i. Our proposed feature separation loss Lsep(·) is inspired by the supervised contrastive loss proposed in (Khosla et al., 2020). The main difference is, instead of applying data augmentation techniques to generate two different views of each data example and feeding the model with augmented data examples, our feature separation loss directly takes the adversarial example x′i of each data example xi as input.

heading: 4.3 TRAINING SCHEDULE
text: By combining the feature separation loss with the reweighted adversarial training, the final object function for Separable Reweighted Adversarial Training (SRAT) is defined as:
min θ
1
n n∑ i=1 max ∥x′i−xi∥p≤ϵ wiL(fθ(x′i), yi) + λLsep(x′i), (7)
where λ is a hyper-parameter to balance the contributions of two terms.
In practice, in order to better take advantage of the reweighting strategy in our SRAT method, we adopt a deferred reweighting training schedule (Cao et al., 2019). Specifically, before annealing the learning rate, SRAT first trains a model without introducing the reweighting strategy and then applies reweighting into model training process with a smaller learning rate. Since SRAT enables to learn more separable feature space, comparing with applying the reweighting strategy from the beginning of training, this deferred reweighting training schedule enables the reweighting strategy to obtain more benefits from our SRAT method. The training algorithm for SRAT is shown in Appendix A.4.

heading: 5 EXPERIMENT
text: In this section, we perform experiments to validate the effectiveness of our SRAT method. We first compare SRAT with several representative imbalanced learning methods in adversarial training under various imbalanced scenarios and then conduct ablation study to understand SRAT more deeply.

heading: 5.1 EXPERIMENTAL SETTINGS
text: Datasets. We conduct experiments on multiple imbalanced training datasets artificially created from three benchmark image datasets CIFAR10, CIFAR100 (Krizhevsky et al., 2009) and SVHN (Netzer et al., 2011) with diverse imbalanced distributions. Specifically, we consider two different imbalance types: Exponential (Exp) imbalance (Cui et al., 2019) and Step imbalance (Buda et al., 2018). For Exp imbalance, the number of training examples of each class will be reduced according to an exponential function n = niτ i, where i is the class index, ni is the number of training examples in the original training dataset for class i and τ ∈ (0, 1). We categorize half classes with most frequent example sizes in the imbalanced training dataset as well-represented classes and the remaining half classes as under-represented classes. For Step imbalance, we follow the similar process adopted in Section 2.1. Moreover, we denote imbalance ratio K as the ratio between training example sizes of the most frequent and least frequent class. We construct different imbalanced datasets “Step-10”, “Step100”, “Exp-10” and “Exp-100”, by adopting different imbalanced types (Step or Exp) with different imbalanced ratios (K = 10 or K = 100) to train models, and evaluate model’s performance on the original uniformly distributed test datasets of CIFAR10, CIFAR100 and SVHN correspondingly. More detailed information about imbalanced training sets can be found in Appendix A.5.
Baseline methods. We implement several representative and state-of-the-art imbalanced learning methods (or their combinations) into adversarial training as baseline methods. These methods include: (1) Focal loss (Focal); (2) LDAM loss (LDAM); (3) Class-balanced reweighting (CBReweight) (Cui et al., 2019), where each example is reweighted proportionally by the inverse of
the effective number2 of its class; (4) Class-balanced Focal loss (CB-Focal) (Cui et al., 2019), a combination of Class-balanced method and Focal loss, where well-classified examples will be downweighted while hard-classified examples will be upweighted controlled by their corresponding effective number; (5) deferred reweighted CE loss (DRCB-CE), where a deferred reweighting training schedule is applied based on the CE loss; (6) deferred reweighted Class-balanced Focal loss (DRCBFocal), where a deferred reweighting training schedule is applied based on the CB-Focal loss; (7) deferred reweighted Class-balanced LDAM loss (DRCB-LDAM) (Cao et al., 2019), where a deferred reweighting training schedule is applied based on the CB-LDAM loss. In addition, we also include the original PGD adversarial training method using cross entropy loss (CE) in our experiments.
Our proposed methods. We evaluate three variants of our proposed SRAT method3 with different implementations of the prediction loss L(·, ·) in Eq. (5), i.e., CE loss, Focal loss and LDAM loss. The variant utilizing CE loss is denoted as SRAT-CE, and, similarly, other two variants are denoted as SRAT-Focal and SRAT-LDAM, respectively. For all these three variants, Class-balanced method (Cui et al., 2019) is adopted to set weight values within the deferred reweighting training schedule.
Implementation details. All aforementioned methods are implemented using a Pytorch library DeepRobust (Li et al., 2020). For CIFAR10/CIFAR100 based datasets, the adversarial examples used in training are calculated by PGD-10, with a perturbation budget ϵ = 8/255 and step size γ = 2/255; in evaluation, we report robust accuracy under l∞-norm 8/255 attacks generated by PGD-20 on Resnet-18 (He et al., 2016) models. For SVHN based datasets, the settings are similar, excepts we set step size γ to 1/255 in both training and evaluation, as suggested in (Wu et al., 2020). We set the total training epochs to 200 and the initial learning rate to 0.1, and decay the learning rate at epoch 160 and 180 with the ratio 0.01. The deferred reweighting strategy will be applied starting from epoch 160.

heading: 5.2 PERFORMANCE COMPARISON
text: Table 1 and Table 2 show the performance comparison on two different imbalanced CIFAR10 datasets. In these two tables, we use bold values to denote the highest accuracy among all methods and use the underline values to indicate our SRAT variants which achieve the highest accuracy among their corresponding baseline methods utilizing the same loss function for making predictions. Due to the limited space, we report the performance comparison on other imbalanced datasets created from CIFAR10, CIFAR100 and SVHN datasets in Appendix A.6.
From Table 1 and Table 2, we can make the following observations. First, compared to baseline methods, our SRAT method obtains improved performance in terms of both overall standard & robust accuracy under almost all imbalanced settings. More importantly, SRAT makes significant improvements on those under-represented classes, especially under the extremely imbalanced settings. For the CIFAR10 Step-100 dataset, our SRAT-Focal method improves the standard accuracy on under-represented classes from 21.81% achieved by the best baseline method utilizing Focal loss to 51.83% and robust accuracy from 3.24% to 15.89%. These results demonstrate that our SRAT method is able to obtain more robustness under imbalanced settings. Second, the performance gap among
2The effective number is defined as the volume of examples and can be calculated by (1− βni)/(1− β), where β ∈ [0, 1) is a hyperparameter and ni denotes the number of examples of class i.
3The implementations can be found via https://github.com/anonymous2share/SRAT.
three SRAT variants are mainly caused by the gap between the loss functions in these methods. As shown in these two tables, DRCB-LDAM typically performs better than DRCE-CE and DRCB-Focal, and similarly, SRAT-LDAM outperforms SRAT-CE and SRAT-Focal under the same settings.

heading: 5.3 ABLATION STUDY
text: In this subsection, we provide ablation study to understand our SRAT method more comprehensively.
Feature space visualization. In order to facilitate the reweighting strategy in adversarial training under the imbalanced setting, we present a feature separation loss in our SRAT method. The main goal of the feature separation loss is to enforce the learned feature space as much separated as possible. For checking whether the feature sepa-
ration loss can work as expected, we apply t-SNE (Van der Maaten & Hinton, 2008) to visualize the latent feature space learned by our SRAT-LDAM method as well as by original PGD adversarial training method (CE) and DRCB-LDAM method in Figure 4.
As shown in Figure 4, the feature space learned by our SRAT-LDAM method is more separable than two baseline methods, which demonstrates that, with our feature separation loss, the adversarially trained model is able to learn much better features and thus SRAT can achieve superiority performance.
From Figure 5, we can obverse that, for all SRAT variants, the model’s standard accuracy is increased with the increasing of the weights for under-represented classes. However, the robust accuracy for these three methods do not synchronize with the change of their standard accuracy. When increasing the weights for under-represented classes, robust accuracy of SRAT-LDAM is almost unchanged and of SRAT-CE and SRAT-Focal even has slight decrease. As a trade-off, using a relative large weight, such as 78 or 100, in SRAT can obtain satisfactory performance on both standard & robust accuracy.

heading: Impact of hyper-parameter λ.
text: In our SRAT method, the contributions of feature separation loss and prediction loss are controlled by a hyper-parameter λ. In this part, we study how this hyper-parameter affects the performance of SRAT. In experiments, we evaluate the models’ performance of all SRAT variants with different values of λ used in
training process on both “Step-100” and “Exp-100” datasets.
As shown in Figure 6, the performance of all SRAT variants are not very sensitive with the choice of λ. However, a large value of λ, such as 8, may hurt the model’s performance.

heading: 6 RELATED WORK
text: Adversarial Robustness. The vulnerability of DNN models to adversarial examples has been verified by many existing successful attack methods (Goodfellow et al., 2014; Carlini & Wagner, 2017). To improve model robustness against adversarial attacks, various defense methods have been proposed (Madry et al., 2017; Raghunathan et al., 2018; Cohen et al., 2019). Among them, adversarial training has been proven to be one of the most effective defense methods (Athalye et al., 2018). Adversarial training can be formulated as solving a min-max optimization problem where the outer minimization process enforces the model to be robust to adversarial examples, generated by the inner maximization process via some existing attacking methods like PGD (Madry et al., 2017). Based on adversarial training, several variants, such as TRADES (Zhang et al., 2019), MART (Wang et al., 2019), have been presented to improve the model’s performance further. More details about adversarial robustness can be found in recent surveys (Chakraborty et al., 2018; Xu et al., 2020b). Since almost all studies of adversarial training are focused on balanced datasets, it’s worthwhile to investigate the performance of adversarial training methods on imbalanced training datasets.
Imbalanced Learning. Most existing works of imbalanced training can be roughly classified into two categories, i.e., re-sampling and reweighting. Re-sampling methods aim to reduce imbalance level through either over-sampling examples from under-represented classes (Buda et al., 2018; Byrd & Lipton, 2019) or under-sampling examples from well-represented classes (Japkowicz & Stephen, 2002; Drummond et al., 2003; He & Garcia, 2009). reweighting methods allocate different weights for different classes or even different examples. For example, Focal loss (Lin et al., 2017) enlarges the weights of wrongly-classified examples while reducing the weights of well-classified examples in the standard cross entropy loss; and LDAM loss (Cao et al., 2019) regularizes the underrepresented classes more strongly than the well-represented classes to attain good generalization on under-represented classes. More details about imbalanced learning can be found in recent surveys (He & Ma, 2013; Johnson & Khoshgoftaar, 2019). The majority of existing methods focused on the nature training scenario and their trained models will be crashed when facing adversarial attacks (Szegedy et al., 2013; Goodfellow et al., 2014). Hence, in this paper, we develop a novel method that can defend adversarial attacks and achieve well-pleasing performance under imbalance settings.

heading: 7 CONCLUSION
text: In this work, we first empirically investigate the behavior of adversarial training under imbalanced settings and explore potential solutions to assist adversarial training in tackling the imbalanced issue. As neither adversarial training itself nor adversarial training with reweighting can work well under imbalanced settings, we further theoretically verify that the poor data separability is one key reason causing the failure of adversarial training based methods. Based on our findings, we propose the Separable Reweighted Adversarial Training (SRAT) framework to facilitate the reweighting strategy in imbalanced adversarial training. We validate the effectiveness of SRAT via extensive experiments. In the future, we plan to examine how other types of defense methods perform under imbalanced scenarios and how other types of balanced learning methods behave under adversarial training.

heading: A APPENDIX
text: In this section, we provide more details about the proposed SRAT framework, as well as the full experimental results of the preliminary study and evaluation.

heading: A.1 THE BEHAVIOR OF ADVERSARIAL TRAINING
text: In order to examine the performance of PGD adversarial training under imbalanced scenarios, we adversarially train ResNet18 (He et al., 2016) models on multiple imbalanced training datasets based on CIFAR10 dataset (Krizhevsky et al., 2009). Similar with observations we discussed in Section 2.1, as shown in Figure 7, Figure 8 and Figure 9, adversarial training produces larger performance gap between well-represented classes and under-represented classes than natural training. Especially, in all imbalanced scenarios, adversarially trained models obtain very low robust accuracy on underrepresented classes, which proves again that adversarial training cannot be applied in practical imbalanced scenarios directly.

heading: A.2 REWEIGHTING STRATEGY IN NATURAL TRAINING V.S. IN ADVERSARIAL TRAINING
text: For exploring whether the reweighting strategy can help adversarial training deal with imbalanced issues, we evaluate performance of adversarial trained models using diverse binary imbalanced training datasets with different weights assigning to under-represented class. As shown in Figure 10, Figure 11, Figure 12, for adversarially trained models, increasing the weights assigning to underrepresented class will improve models’ performance on under-represented class. However, as the same time, the models’ performance on well-represented class will be drastically decreased. As a comparison, adopting larger weights in naturally trained models will also improve models’ performance on under-represented class but only result in slight drop in performance on wellrepresented class. In other words, the reweighting strategy proposed in natural training to handle imbalanced problem may only provide limited help in adversarial training, and, hence, new techniques are needed for adversarial training under imbalanced scenarios.

heading: A.3 PROOFS OF THE THEOREMS IN SECTION 3

heading: A.3.1 PROOF OF LEMMA 3.1
text: Lemma 3.1 Under the data distribution D as defined in Eq. (1), with an imbalanced ratio K and a reweight ratio ρ, the optimal classifier which minimizes the (reweighted) empirical risk:
f∗ = argmin f
( Pr.(f(x) ̸=y|y=−1) · Pr.(y=−1) · ρ+ Pr.(f(x) ̸=y|y=+1) · Pr.(y=+1) ) (2)
has the solution: w = 1 and b = 12 log( ρ K )
dσ2
η = 1 2 log( ρ K ) d S .
Proof 1 (Proof of Lemma 3.1) We will first prove that the optimal model f∗ has parameters w1 = w2 = · · · = wd (or w = 1) by contradiction. We define G = {1, 2, . . . , d} and make the following assumption: for the optimal w and b, we assume if there exist wi < wj for i ̸= j and i, j ∈ G. Then we obtain the following standard errors for the class “-1” and the class “+1” of this classifier f with weight w:
Pr.(f∗(x) ̸= y|y = −1) = Pr.(wTN (−η, σ2)− b > 0) = Pr.{ ∑
k ̸=i,k ̸=j
wkN (−η, σ2) + wiN (−η, σ2) + wjN (−η, σ2)− b > 0},
Pr.(f∗(x) ̸= y|y = +1) = Pr.(wTN (+η, σ2)− b < 0) = Pr.{ ∑
k ̸=i,k ̸=j
wkN (+η, σ2) + wiN (+η, σ2) + wjN (+η, σ2)− b < 0}.
(8)
However, if we define a new classier f̃ whose weight w̃ uses wj to replace wi, we obtain the errors for the new classifier:
Pr.(f̃(x) ̸= y|y = −1) = Pr.{ ∑
k ̸=i,k ̸=j
wkN (−η, σ2) + wjN (−η, σ2) + wjN (−η, σ2)− b > 0},
Pr.(f̃(x) ̸= y|y = +1) = Pr.{ ∑
k ̸=i,k ̸=j
wkN (+η, σ2) + wjN (+η, σ2) + wjN (+η, σ2)− b < 0}. (9)
Comparing the errors in Eq. (8) and Eq. (9), as wi < wj , then the classifier f̃ has smaller standard error in each class. Therefore, it contradicts with the assumption that f is the optimal classifier with smallest error. Thus, we conclude for an optimal linear classifier in natural training, it must satisfies w1 = w2 = · · · = wd (or w = 1) if we do not consider the scale of w. Next, we calculate the optimal bias term b given w = 1, where we find an optimal b can minimize the (reweighted) empirical risk:
Errortrain(f∗) = Pr.(f∗(x) ̸= y|y = −1) · Pr.(y = −1) · ρ+ Pr.(f∗(x) ̸= y|y = +1) · Pr.(y = +1) ∝ Pr.(f∗(x) ̸= y|y = −1) · ρ+ Pr.(f∗(x) ̸= y|y = +1) ·K
= ρ · Pr.( d∑
i=1
N (−η, σ2)− b > 0) +K · Pr.( d∑
i=1
N (η, σ2)− b < 0)
= ρ · Pr.(N (0, 1) < − b+ dη dσ ) +K · Pr.(N (0, 1) < b− dη dσ ),
and we take the derivative with respect to b:
∂Errortrain ∂b = ρ√ 2π · (− 1 dσ ) exp(−1 2 (−b+ dη dσ )2) + K√ 2π · ( 1 dσ ) exp(−1 2 ( b− dη dσ )2).
When ∂Errortrain/∂b = 0, we can calculate the optimal b which gives the minimum value of the empirical error, and we have:
b = 1
2 log(
ρ K ) dσ2 η = 1 2 log( ρ K ) d S .

heading: A.3.2 PROOF OF THEOREM 3.1
text: Theorem 3.1 Under two data distributions (x(1), y(1)) ∈ D1 and (x(2), y(2)) ∈ D2 with different separabilities S1 > S2, let f∗1 and f ∗ 2 be the optimal non-reweighted classifiers (ρ = 1) under D1 and D2, respectively. Given the imbalance ratio K is large enough, we have:
Pr.(f∗1 (x (1)) ̸= y(1)|y(1) = −1)− Pr.(f∗1 (x(1)) ̸= y(1)|y(1) = +1)
< Pr.(f∗2 (x (2)) ̸= y(2)|y(2) = −1)− Pr.(f∗2 (x(2)) ̸= y(2)|y(2) = +1).
(3)
Proof 2 (Proof of Theorem 3.1) Without loss of generality, for distribution D1, D2 with different mean-variance pairs (±η1, σ21) and (±η2, σ22), we can only consider the case η1 = η2 and σ21 < σ22 . Otherwise, we can simply rescale one of them to match the mean vector of the other and will not impact the results. Under this definition, the optimal classifier f∗1 and f ∗ 2 has weight vector w1 = w2 = 1 and bias term b1, b2, with the value as demonstrated in Lemma 3.1. Next, we will prove the Theorem 3.1 by 2 steps.
Step 1. For the error of class “-1”, we have:
Pr.(f∗1 (x (1)) ̸= y(1)|y(1) = −1) = Pr.( d∑ i=1 N (−η, σ21)− b1 > 0)
< Pr.( d∑
i=1
N (−η, σ21)− b2 > 0) (because S1 > S2)
< Pr.( d∑
i=1
N (−η, σ22)− b2 > 0) (because σ21 < σ22)
= Pr.(f∗2 (x (2)) ̸= y(2)|y(2) = −1).
Step 2. For the error of class “+1”, we have:
Pr.(f∗1 (x (1)) ̸= y(1)|y(1) = +1) = Pr.( d∑ i=1 N (η, σ21)− b1 < 0)
= Pr.(N (0, 1) < b1 − dη dσ1 ) = Pr.(N (0, 1) < − log(K) · σ1 2η − η σ1 ),
(10)
and similarly,
Pr.(f∗2 (x (2)) ̸= y(2)|y(2) = +1) = Pr.(N (0, 1) < − log(K) · σ2 2η − η σ2 ). (11)
Note that when K is large enough, i.e., log(K) > 2·η 2
σ1·σ2 , we can get the Z-score in Eq. (10) is larger than Eq. (11). As a result, we have:
Pr.(f∗1 (x (1)) ̸= y(1)|y(1) = +1) > Pr.(f∗2 (x(2)) ̸= y(2)|y(2) = +1). (12)
By combining Step 1 and Step 2, we can get the inequality in Theorem 3.1.

heading: A.3.3 PROOF OF THEOREM 3.2
text: Theorem 3.2 Under two data distributions (x(1), y(1)) ∈ D1 and (x(2), y(2)) ∈ D2 with different separabilities S1 > S2, let f∗1 and f ∗ 2 be the optimal non-reweighted classifiers (ρ = 1) under D1 and D2, respectively, and let f ′1 ∗ and f ′2
∗ be the optimal reweighted classifiers under D1 and D2 given the optimal reweighting ratio (ρ = K). Given the imbalance ratio K is large enough, we have:
Pr.(f ′1 ∗ (x(1)) ̸= y(1)|y(1) = +1)− Pr.(f∗1 (x(1)) ̸= y(1)|y(1) = +1)
< Pr.(f ′2 ∗ (x(2)) ̸= y(2)|y(2) = +1)− Pr.(f∗2 (x(2)) ̸= y(2)|y(2) = +1).
(4)
Proof 3 (Proof of Theorem 3.2) We first show that under both distribution D1 and D2, the optimal reweighting ratio ρ is equal to the imbalance ratio K. Based on the results in Eq. (8) and calculated model parameters w and b, we have the test error (given the model trained by reweight value ρ):
Errortest(f∗)
= Pr.(f∗(x) ̸= y|y = −1) · Pr.(y = −1) + Pr.(f∗(x) ̸= y|y = +1) · Pr.(y = +1)
∝ Pr.(N (0, 1) < − b+ dη dσ ) + Pr.(N (0, 1) < b− dη dσ ) = Pr.(N (0, 1) < −1 2 log( ρ K )− σ η ) + Pr.(N (0, 1) < 1 2 log( ρ K )− σ η ).
The value of taking the minimum when its derivative with respect to ρ is equal to 0, where we can get ρ = K and the bias term b = 0. Note that the variance values have the relation: σ21 < σ 2 2 . Therefore, it is easy to get that:
Pr.(f ′1 ∗ (x(1)) ̸= y(1)|y(1) = +1) = Pr.( d∑ i=1 N (η, σ21) < 0)
< Pr.( d∑
i=1
N (η, σ22) < 0) = Pr.(f ′2 ∗ (x(2)) ̸= y(2)|y(2) = +1).
(13)
Combining the results in Eq. (12) and (13), we have proved the inequality in Theorem 3.2.

heading: A.4 ALGORITHM OF SRAT
text: The algorithm of our proposed SRAT framework is shown in Algorithm 1. Specifically, in each training iteration, we first generate adversarial examples using PGD for examples in the current batch (Line 5). If the current training iteration does not reach a predefined starting reweighting epoch Td, we will assign same weights, i.e., wi = 1 for all adversarial examples xi in the current batch (Line 6). Otherwise, the reweighting strategy will be adopted in the final loss function (Line 15), where a specific weight wi will be assigned for each adversarial example xi if its corresponding clean example xi comes from an under-represented class.
Algorithm 1 Separable Reweighted Adversarial Training (SRAT). Input: imbalanced training dataset D = {(xi, yi)}ni=1, number of total training epochs T , starting
reweighting epoch Td, batch size N , number of batches M , learning rate γ Output: An adversarially robust model fθ
1: Initialize the model parameters θ randomly; 2: for epoch = 1, . . . , Td − 1 do 3: for mini-batch = 1, . . . ,M do 4: Sample a mini-batch B = {(xi, yi)}Ni=1 from D; 5: Generate adversarial example x′i for each x ′ i ∈ B;
6: L(fθ) = 1N ∑N i=1 max∥x′i−xi∥p≤ϵ L(fθ(x ′ i), yi) + λLsep(x′i) 7: θ ← θ − γ∇θL(fθ) 8: end for 9: Optional: γ ← γ/κ
10: end for 11: for epoch = Td, . . . , T do 12: for mini-batch = 1, . . . ,M do 13: Sample a mini-batch B = {(xi, yi)}Ni=1 from D; 14: Generate adversarial example x′i for each x ′ i ∈ B;
15: L(fθ) = 1N ∑N i=1 max∥x′i−xi∥p≤ϵ wiL(fθ(x ′ i), yi) + λLsep(x′i) 16: θ ← θ − γ∇θL(fθ) 17: end for 18: Optional: γ ← γ/κ 19: end for

heading: A.5 DATA DISTRIBUTION OF IMBALANCED TRAINING DATASETS
text: In our experiments, we construct multiple imbalanced training datasets to simulate various kinds of imbalanced scenarios by combining different imbalance types (i.e., Exp and Step) with different imbalanced ratios (i.e., K = 10 and K = 100). Figure 13 and Figure 14 show the data distribution of all ten-classes imbalanced training datasets used in our preliminary studies and experiments based on CIFAR10 (Krizhevsky et al., 2009) and SVHN (Netzer et al., 2011) datasets, respectively.

heading: A.6 PERFORMANCE COMPARISON

heading: A.6.1 PERFORMANCE COMPARISON ON IMBALANCED CIFAR10 DATASETS
text: Table 3 and Table 4 show the performance comparison on two imbalanced CIFAR10 datasets. We use bold values to denote the highest accuracy among all methods and use the underline values to indicate our SRAT variants which achieve the highest accuracy among their corresponding baseline methods utilizing the same loss function for making predictions.
From Table 3 and Table 4, we get similar observation that, comparing with baseline methods, our proposed SRAT method can produce a robust model which can achieve improved overall performance when the training dataset is imbalanced. In addition, based on the experimental results in Table 3 to Table 4, we find that, compared with the performance improvement between DRCB-LDAM and SRAT-LDAM, the improvement between DRCB-CE and SRAT-CE and the improvement between DRCB-Focal and SRAT-Focal are more obviously. The possible reason behind this phenomenon is, the LDAM loss can also implicitly produce a more separable feature space (Cao et al., 2019) while CE loss and Focal loss do not conduct any specific operations on the latent feature space. Hence, the feature separation loss contained in SRAT-CE and SRAT-Focal could be more effective on learning separable feature space and facilitate the Focal loss on prediction. However, in SRAT-LDAM, the feature separation loss and LDAM loss may affect each other on learning feature representations and, hence, the effectiveness of the feature separation loss may be counteracted or weakened.

heading: A.6.2 PERFORMANCE COMPARISON ON IMBALANCED SVHN DATASETS
text: We report the performance comparison on various imbalanced SVHN datasets with different imbalance types and imbalance ratios from Table 5 to Table 8.
Based on experimental results shown in Table 5 and Table 8, we find that the robust model trained by our SRAT method can achieve superior overall performance under various imbalanced scenarios, when comparing with their corresponding baselines methods which utilize the same prediction loss function. Hence, experiments conducted on various imbalanced SVHN datasets further verify the effectiveness of our SRAT method under diverse imbalanced scenarios.

heading: A.6.3 PERFORMANCE COMPARISON ON IMBALANCED CIFAR100 DTASETS
text: Table 9 and Table 10 show the performance comparison between our SRAT variants with baseline methods on two imbalanced CIFAR100 datasets.
As shown in Table 9 and Table 10, although the increasing of total number of classes in the dataset brings more challenges on the model training process, our SRAT variants are able to achieve notable improvements on both standard accuracy and robust accuracy.
Therefore, in conclusion, experiments conducted on multiple imbalanced datasets created from three benchmark image datasets CIFAR10, CIFAR100 (Krizhevsky et al., 2009) and SVHN (Netzer et al., 2011) with diverse imbalanced distributions verify the effectiveness of our proposed SRAT method under various imbalanced scenarios.
A.7 ABLATION STUDY
Impact of weight values. We study the impact of weight values in Section 5.3 via evaluating the overall standard accuracy and robust accuracy of SRAT trained models with different weights assigning to under-represented classes. Here, we further investigate the impact of weight values in our SART method from the class-wise perspective. In Figure 15, we report the standard & robust accuracy of models trained by our SRAT-LDAM variant with
different weight values on all ten classes, five well-represented classes and five under-represented classes of the Step-100 CIFAR10 dataset separately. Same as before, we assign well-represented classes with weight 1 and change the weight for under-represented classes from 10 to 200. In addition, we use an approximation integer 78 to denote the weight calculated by the Class-balanced method (Cui et al., 2019).
As shown in Figure 15, when increasing of the weights for under-represented classes, the model’s performance on under-represented classes can be increased greatly, with the cost of relatively small performance dropping on well-represented classes, and the overall performance of the model is also increased. This phenomenon can be observed much clear in the standard accuracy case, as shown in Figure 15a. Recall that in our preliminary study (Section 2), we find the reweighting strategy cannot work well in adversarial training, as it causes a strong tension of the model’s performance between well-represented and under-represented class. As this strong tension does not exist in the model trained by our SRAT method, we believe our SRAT method indeed facilitate the reweighting strategy in adversarial training under imbalanced scenarios.
Impact of imbalance ratio K. In previous experiments, we evaluate the effectiveness of our SRAT method using various imbalanced datasets with imbalance ratio K = 10 or K = 100. To investigate the performance of our SRAT method more comprehensively, in this part, we test our SRAT method on more imbalanced datasets with diverse imbalance ratios. Specifically, we construct a series of ”Step” imbalanced CIFAR10 datasets by setting the value of the imbalance ratio K from 5 to 100. For comparison, we apply both DRCB-Focal method and our SRATFocal variant to train models on those imbalanced datasets and test the trained models’ performance on the original uniformly distributed CIFAR10 test dataset. The experimental results are shown in Figure 16.
From Figure 16, we can obverse that, under different imbalanced scenarios, the model trained by our SRAT-Focal can always achieve better performance than the one trained by DRCB-Focal method. In other words, the effectiveness of our SRAT method will not be affected by the imbalanced ratio K, which determines the data distribution of the imbalanced training dataset.

Review summaries:
This paper studies the effect of adversarial training (AT) in the context of imbalanced datasets. In particular, the authors demonstrate empirically that adversarially trained models tend to have a larger gap in accuracy between well and under represented classes, as compared to their standard counterparts. Moreover, they show that standard reweighting strategies do not considerably alleviate this issue. Motivated by their findings, they propose adding a regularization function to adversarial training to encourage greater feature separation between classes (SRAT), and study its effectiveness on unbalanced datasets.

The paper focuses on adversarial training for imbalanced datasets. Imbalanced datasets emerge frequently in real-world applications, while as the paper demonstrates the adversarial training on the under-represented classes results in weak performance. The paper showcases that when the classes are not separable, then a linear model in under-represented classes is weak. Two modifications in adversarial training are proposed to ameliorate that: a) a weighted average of the loss per sample (depending on the class this sample belongs into), b) an additional loss that aims in maximizing the class separation.  

This paper provides two critical observations. The first is that adversarial training has worse performance on under-represented classes than natural training on imbalanced datasets. The second is that conventional reweighting methods that upweight under-represented classes largely hurt the performance on well-represented classes for adversarial training. Further, the authors theoretically analyze the reason for the above observations on a binary-classification case. Motivated by the theoretical analysis, the authors incorporate standard adversarial training with feature separation loss, namely SRAT, which is empirically validated SRAT can improve the performance on imbalanced datasets.