Title: RETHINKING SOFTMAX CROSS-ENTROPY LOSS FOR ADVERSARIAL ROBUSTNESS

Abstract: Previous work shows that adversarially robust generalization requires larger sample complexity, and the same dataset, e.g., CIFAR-10, which enables good standard accuracy may not suffice to train robust models. Since collecting new training data could be costly, we focus on better utilizing the given data by inducing the regions with high sample density in the feature space, which could lead to locally sufficient samples for robust learning. We first formally show that the softmax cross-entropy (SCE) loss and its variants convey inappropriate supervisory signals, which encourage the learned feature points to spread over the space sparsely in training. This inspires us to propose the Max-Mahalanobis center (MMC) loss to explicitly induce dense feature regions in order to benefit robustness. Namely, the MMC loss encourages the model to concentrate on learning ordered and compact representations, which gather around the preset optimal centers for different classes. We empirically demonstrate that applying the MMC loss can significantly improve robustness even under strong adaptive attacks, while keeping high accuracy on clean inputs comparable to the SCE loss with little extra computation.

1. INTRODUCTION:

The deep neural networks (DNNs) trained by the softmax cross-entropy (SCE) loss have achieved state-of-the-art performance on various tasks (Goodfellow et al., 2016) . However, in terms of robustness, the SCE loss is not sufficient to lead to satisfactory performance of the trained models. It has been widely recognized that the DNNs trained by the SCE loss are vulnerable to adversarial attacks (Carlini & Wagner, 2017a; Goodfellow et al., 2015; Kurakin et al., 2017; Moosavi-Dezfooli et al., 2016; Papernot et al., 2016) , where human imperceptible perturbations can be crafted to fool a high-performance network. To improve adversarial robustness of classifiers, various kinds of defenses have been proposed, but many of them are quickly shown to be ineffective to the adaptive attacks, which are adapted to the specific details of the proposed defenses (Athalye et al., 2018) .

Besides, the methods on verification and training provably robust networks have been proposed (Dvijotham et al., 2018a; b; Hein & Andriushchenko, 2017; Wong & Kolter, 2018) . While these methods are exciting, the verification process is often slow and not scalable. Among the previously proposed defenses, the adversarial training (AT) methods can achieve state-of-the-art robustness under different adversarial settings (Madry et al., 2018; Zhang et al., 2019b) . These methods either directly impose the AT mechanism on the SCE loss or add additional regularizers. Although the AT methods are relatively strong, they could sacrifice accuracy on clean inputs and are computationally expensive (Xie et al., 2019) . Due to the computational obstruction, many recent efforts have been devoted to proposing faster verification methods (Wong et al., 2018; Xiao et al., 2019) and accelerating AT procedures (Shafahi et al., 2019; Zhang et al., 2019a) . However, the problem still remains. Schmidt et al. (2018) show that the sample complexity of robust learning can be significantly larger than that of standard learning. Given the difficulty of training robust classifiers in practice, they further postulate that the difficulty could stem from the insufficiency of training samples in the commonly used datasets, e.g., CIFAR-10 (Krizhevsky & Hinton, 2009) . Recent work intends to solve this problem by utilizing extra unlabeled data (Carmon et al., 2019; Stanforth et al., 2019) , while we focus on the complementary strategy to exploit the labeled data in hand more efficiently. Note that although the samples in the input space are unchangeable, we could instead manipulate the local sample distribution, i.e., sample density in the feature space via appropriate training objectives. Intuitively, by inducing high-density feature regions, there would be locally sufficient samples to train robust classifiers and return reliable predictions (Schmidt et al., 2018) . Similar to our attempt to induce high-density regions in the feature space, previous work has been proposed to improve intra-class compactness. Contrastive loss (Sun et al., 2014) and triplet loss (Schroff et al., 2015) are two classical objectives for this purpose, but the training iterations will dramatically grow to construct image pairs or triplets, which results in slow convergence and instability. The center loss (Wen et al., 2016) avoids the pair-wise or triplet-wise computation by minimizing the squared distance between the features and the corresponding class centers. However, since the class centers are updated w.r.t. the learned features during training, the center loss has to be jointly used with the SCE loss to seek for a trade-off between inter-class dispersion and intra-class compactness. Therefore, the center loss cannot concentrate on inducing strong intra-class compactness to construct high-density regions and consequently could not lead to reliable robustness, as shown in our experiments.

In this paper, we first formally analyze the sample density distribution induced by the SCE loss and its other variants (Pang et al., 2018; Wan et al., 2018) in Sec. 3.2, which demonstrates that these previously proposed objectives convey unexpected supervisory signals on the training points, which make the learned features tend to spread over the space sparsely. This undesirable behavior mainly roots from applying the softmax function in training, which makes the loss function only depend on the relative relation among logits and cannot directly supervise on the learned representations.

We further propose a novel training objective which can explicitly induce high-density regions in the feature space and learn more structured representations. To achieve this, we propose the Max-Mahalanobis center (MMC) loss (detailed in Eq. ( 8)) as the substitute of the SCE loss. Specifically, in the MMC loss, we first preset untrainable class centers with optimal inter-class dispersion in the feature space according to Pang et al. (2018) , then we encourage the features to gather around the centers by minimizing the squared distance similar with the center loss. The MMC loss can explicitly control the inter-class dispersion by a single hyperparameter, and further concentrate on improving intra-class compactness in the training procedure to induce high-density regions, as intuitively shown in Fig. 1 . Behind the simple formula, the MMC loss elegantly combines the favorable merits of the previous methods, which leads to a considerable improvement on the adversarial robustness.

In experiments, we follow the suggestion by Carlini et al. (2019) that we test under different threat models and attacks, including the adaptive attacks (Athalye et al., 2018) on MNIST, CIFAR-10, and CIFAR-100 (Krizhevsky & Hinton, 2009; LeCun et al., 1998) . The results demonstrate that our method can lead to reliable robustness of the trained models with little extra computation, while maintaining high clean accuracy with faster convergence rates compared to the SCE loss and its variants. When combined with the existing defense mechanisms, e.g., the AT methods (Madry et al., 2018) , the trained models can be further enhanced under the attacks different from the one used to craft adversarial examples for training.

2. PRELIMINARIES:

This section first provides the notations, then introduces the adversarial attacks and threat models.

2.1. NOTATIONS:

In this paper, we use the lowercases to denote variables and the uppercases to denote mappings. Let L be the number of classes, we define the softmax function softmax(h) :

R L → R L as softmax(h) i = exp(hi) / L l=1 exp(h l ), i ∈ [L]

, where [L] := {1, • • • , L} and h is termed as logit.

A deep neural network (DNN) learns a non-linear mapping from the input x ∈ R p to the feature z = Z(x) ∈ R d . One common training objective for DNNs is the softmax cross-entropy (SCE) loss:

EQUATION

for a single input-label pair (x, y), where 1 y is the one-hot encoding of y and the logarithm is defined as element-wise. Here W and b are the weight matrix and bias vector of the SCE loss, respectively.

2.2. ADVERSARIAL ATTACKS AND THREAT MODELS:

Previous work has shown that adversarial examples can be easily crafted to fool DNNs (Biggio et al., 2013; Nguyen et al., 2015; Szegedy et al., 2014) . A large amount of attacking methods on generating adversarial examples have been introduced in recent years (Carlini & Wagner, 2017a; Chen et al., 2017; Dong et al., 2018; Goodfellow et al., 2015; Ilyas et al., 2018; Kurakin et al., 2017; Madry et al., 2018; Moosavi-Dezfooli et al., 2016; Papernot et al., 2016; Uesato et al., 2018) . Given the space limit, we try to perform a comprehensive evaluation by considering five different threat models and choosing representative attacks for each threat model following Carlini et al. (2019) :

White-box l ∞ distortion attack: We apply the projected gradient descent (PGD) (Madry et al., 2018) method, which is efficient and widely studied in previous work (Pang et al., 2019) .

White-box l 2 distortion attack: We apply the C&W (Carlini & Wagner, 2017a) method, which has a binary search mechanism on its parameters to find the minimal l 2 distortion for a successful attack.

Black-box transfer-based attack: We use the momentum iterative method (MIM) (Dong et al., 2018) that is effective on boosting adversarial transferability (Kurakin et al., 2018) .

Black-box gradient-free attack: We choose SPSA (Uesato et al., 2018) since it has broken many previously proposed defenses. It can still perform well even when the loss is difficult to optimize.

General-purpose attack::

We also evaluate the general robustness of models when adding Gaussian noise (Gilmer et al., 2019) or random rotation (Engstrom et al., 2019) on the input images.

Furthermore, to exclude the false robustness caused by, e.g., gradient mask (Athalye et al., 2018) , we modify the above attacking methods to be adaptive attacks (Carlini & Wagner, 2017b; Carlini et al., 2019; Herley & Van Oorschot, 2017) when evaluating on the robustness of our method. The adaptive attacks are much more powerful than the non-adaptive ones, as detailed in Sec. 4.2.

3. METHODOLOGY:

Various theoretical explanations have been developed for adversarial examples (Fawzi et al., 2016; 2018; Ilyas et al., 2019; Papernot et al., 2018) . In particular, Schmidt et al. (2018) show that training robust classifiers requires significantly larger sample complexity compared to that of training standard ones, and they further postulate that the difficulty of training robust classifiers stems from, at least partly, the insufficiency of training samples in the common datasets. Recent efforts propose alternatives to benefit training with extra unlabeled data (Carmon et al., 2019; Stanforth et al., 2019) , while we explore the complementary way to better use the labeled training samples for robust learning.

Although a given sample is fixed in the input space, we can instead manipulate the local sample distribution, i.e., sample density in the feature space, via designing appropriate training objectives. Intuitively, by inducing high-density regions in the feature space, it can be expected to have locally sufficient samples to train robust models that are able to return reliable predictions. In this section, we first formally define the notion of sample density in the feature space. Then we provide theoretical analyses of the sample density induced by the SCE loss and its variants. Finally, we propose our new Max-Mahalanobis center (MMC) loss and demonstrate its superiority compared to previous losses.

3.1. SAMPLE DENSITY IN THE FEATURE SPACE:

Given a training dataset D with N input-label pairs, and the feature mapping Z trained by the objective L(Z(x), y) on this dataset, we define the sample density nearby the feature point z = Z(x)

following the similar definition in physics (Jackson, 1999) as

SD(z) = ∆N Vol(∆B)

.

(2)

Here Vol(•) denotes the volume of the input set, ∆B is a small neighbourhood containing the feature point z, and ∆N = |Z(D) ∩ ∆B| is the number of training points in ∆B, where Z(D) is the set of all mapped features for the inputs in D. Note that the mapped feature z is still of the label y.

In the training procedure, the feature distribution is directly induced by the training loss L, where minimizing the loss value is the only supervisory signal for the feature points to move (Goodfellow et al., 2016) . This means that the sample density varies mainly along the orthogonal direction w.r.t. the loss contours, while the density along a certain contour could be approximately considered as the same. For example, in the right panel of Fig. 1 , the sample density induced by our MMC loss (detailed in Sec. 3.3) changes mainly along the radial direction, i.e., the directions of red arrows, where the loss contours are dashed concentric circles. Therefore, supposing L(z, y) = C, we choose

∆B = {z ∈ R d |L(z, y) ∈ [C, C + ∆C]},

where ∆C > 0 is a small value. Then Vol(∆B) is the volume between the loss contours of C and C + ∆C for label y in the feature space.

3.2. THE SAMPLE DENSITY INDUCED BY THE GENERALIZED SCE LOSS:

Generalized SCE loss. To better understand how the SCE loss and its variants (Pang et al., 2018; Wan et al., 2018) affect the sample density of features, we first generalize the definition in Eq. ( 1) as:

EQUATION

where the logit h = H(z) ∈ R L is a general transformation of the feature z, for example, h = W z +b in the SCE loss. We call this family of losses as the generalized SCE (g-SCE) loss. Wan et al. (2018) propose the large-margin Gaussian Mixture (L-GM) loss, where

h i = -(z -µ i ) Σ i (z -µ i ) -mδ i,y

under the assumption that the learned features z distribute as a mixture of Gaussian. Here µ i and Σ i are extra trainable means and covariance matrices respectively, m is the margin, and δ i,y is the indicator function. Pang et al. (2018) propose the Max-Mahalanobis linear discriminant analysis (MMLDA) loss, where

h i = -z -µ * i 2 2

under the similar mixture of Gaussian assumption, but the main difference is that µ * i are not trainable, but calculated before training with optimal inter-class dispersion. These two losses both fall into the family of the g-SCE loss with quadratic logits:

EQUATION

where B i are the bias variables. Besides, note that for the SCE loss, there is

softmax(W z + b) i = exp(W i z + b i ) l∈[L] exp(W l z + b l ) = exp(-z -1 2 W i 2 2 + b i + 1 4 W i 2 2 ) l∈[L] exp(-z -1 2 W l 2 2 + b l + 1 4 W l 2 2 )

.

According to Eq. ( 4), the SCE loss can also be regraded as a special case of the g-SCE loss with quadratic logits, where

EQUATION

2 and Σ i = I are identity matrices. Therefore, later when we refer to the g-SCE loss, we assume that the logits are quadratic as in Eq. ( 4) by default.

The contours of the g-SCE loss. To provide a formal representation of the sample density induced by the g-SCE loss, we first derive the formula of the contours, i.e., the closed-form solution of 3), we can represent the contours as the solution of

L g-SCE (Z(x), y) = C in the space of z, where C ∈ (0, +∞) is a given constant. Let C e = exp(C) ∈ (1, +∞), from Eq. (

EQUATION

The function in Eq. ( 5) does not provide an intuitive closed-form solution for the contours, since the existence of the term log l =y exp(h l ) . However, note that this term belongs to the family of Log-Sum-Exp (LSE) function, which is a smooth approximation to the maximum function (Nesterov, 2005; Nielsen & Sun, 2016) . Therefore, we can locally approximate the function in Eq. ( 5) with

EQUATION

∞, loss value → ∞

The case:

# $ > #& $ ' ( ∞, loss value → ∞ ' ) (a) g-SCE ∞, loss value → 0

The case: # $ < #& $ (Preferred by models since lower loss values)

' ( ∞, loss value → 0 ' ) , -,/ -, loss value=0 * , -,/ -, loss value=0 * (b) MMC 2 - * , loss value=0 ∞, loss value → ∞ ' ( ∞, loss value → ∞ ' )

Feature points Moving directions in training:

Figure 2 : Intuitive illustration on the inherent limitations of the g-SCE loss. Reasonably learned features for a classification task should distribute in clusters, so it is counter-intuitive that the feature points tend to move to infinity to pursue lower loss values when applying the g-SCE loss. In contrast, MMC induces models to learn more structured and orderly features.

where ỹ = arg max l =y h l . In the following text, we apply colored characters with tilde like ỹ to better visually distinguish them. According to Eq. ( 6), we can define L y,ỹ (z) = log[exp(h ỹ -h y ) + 1] as the local approximation of the g-SCE loss nearby the feature point z, and substitute the neighborhood

∆B by ∆B y,ỹ = {z ∈ R d |L y,ỹ (z) ∈ [C, C + ∆C]}.

For simplicity, we assume scaled identity covariance matrix in Eq. ( 4), i.e., Σ i = σ i I, where σ i > 0 are scalars. Through simple derivations (detailed in Appendix A.1), we show that if σ y = σ ỹ , the solution of L y,ỹ (z) = C is a (d -1)dimensional hypersphere with the center M y,ỹ = (σ y -σ ỹ ) -1 (σ y µ y -σ ỹ µ ỹ ); otherwise if σ y = σ ỹ , the hypersphere-shape contour will degenerate to a hyperplane.

The induced sample density. Since the approximation in Eq. ( 6) depends on the specific y and ỹ, we define the training subset 6) is

D k, k = {(x, y) ∈ D|y = k, ỹ = k} and N k, k = |D k, k|.

EQUATION

where for the input-label pair in

D k, k, there is L g-SCE ∼ p k, k(c).

Limitations of the g-SCE loss. Based on Theorem 1 and the approximation in Eq. ( 6), let

C * = log(1 + exp(B k, k(σ k -σ k ))) and C * e = exp(C * ), such that B k, k + log(C * e -1) σ k -σk = 0. According to Appendix A.1, if σ k > σ k,

then C * will act as a tight lower bound for C, i.e., the solution set of C < C * is empty. This will make the training procedure tend to avoid this case since the loss C cannot be further minimized to zero, which will introduce unnecessary biases on the returned predictions. On the other hand, if σ k < σ k, C could be minimized to zero. However, when C → 0, the sample density will also tend to zero since there is B k, k + log(Ce-1) σ k -σk → ∞, which means the feature point will be encouraged to go further and further from the hypersphere center M k, k only to make the loss value C be lower, as intuitively illustrated in Fig. 2(a) . This counter-intuitive behavior mainly roots from applying the softmax function in training. Namely, the softmax normalization makes the loss value only depend on the relative relation among logits. This causes indirect and unexpected supervisory signals on the learned features, such that the points with low loss values tend to spread over the space sparsely. Fortunately, in practice, the feature point will not really move to infinity, since the existence of batch normalization layers (Ioffe & Szegedy, 2015) , and the squared radius from the center M k, k increases as O(| log C|) when minimizing the loss C. These theoretical conclusions are consistent with the empirical observations on the two-dimensional features in previous work (cf. Fig. 1 in Wan et al. (2018) ).

Another limitation of the g-SCE loss is that the sample density is proportional to N k, k, which is on average N/L 2 . For example, there are around 1.3 million training data in ImageNet (Deng et al., 2009) , but with a large number of classes L = 1, 000, there are averagely less than two samples in each D k, k. These limitations inspire us to design the new training loss as in Sec 3.3.

Remark 1. If σ k = σ k (e.g., as in the SCE loss), the features with loss values in [C, C + ∆C] will be encouraged to locate between two hyperplane contours without further supervision, and consequently there will not be explicit supervision on the sample density as shown in the left panel of Fig. 1 .

Remark 2. Except for the g-SCE loss, Wen et al. (2016) propose the center loss in order to improve the intra-class compactness of learned features, formulated as L Center (Z(x), y) = 1 2 z -µ y 2 2 . Here the center µ y is updated based on a mini-batch of learned features with label y in each training iteration. The center loss has to be jointly used with the SCE loss as L SCE + λL Center , since simply supervise the DNNs with the center loss term will cause the learned features and centers to degrade to zeros (Wen et al., 2016) . This makes it difficult to derive a closed-form formula for the induced sample density. Besides, the center loss method cannot concentrate on improving intra-class compactness, since it has to seek for a trade-off between inter-class dispersion and intra-class compactness.

3.3. MAX-MAHALANOBIS CENTER LOSS:

Inspired by the above analyses, we propose the Max-Mahalanobis center (MMC) loss to explicitly learn more structured representations and induce high-density regions in the feature space. The MMC loss is defined in a regression form without the softmax function as

EQUATION

)

Here

µ * = {µ * l } l∈[L]

are the centers of the Max-Mahalanobis distribution (MMD) (Pang et al., 2018) . The MMD is a mixture of Gaussian distribution with identity covariance matrix and preset centers µ * , where µ * l 2 = C MM for any l ∈ [L], and C MM is a hyperparameter. These MMD centers are invariable during training, which are crafted according to the criterion: µ * = arg min µ max i =j µ i , µ j . Intuitively, this criterion is to maximize the minimal angle between any two centers, which can provide optimal inter-class dispersion as shown in Pang et al. (2018) . In Appendix B.1, we provide the generation algorithm for µ * in MMC. We derive the sample density induced by the MMC loss in the feature space, as stated in Theorem 2. Similar to the previously introduced notations, here we define the subset D k = {(x, y) ∈ D|y = k} and

EQUATION

)

where for the input-label pair in D k , there is L MMC ∼ p k (c).

According to Theorem 2, there are several attractive merits of the MMC loss, as described below.

Inducing higher sample density. Compared to Theorem 1, the sample density induced by MMC is proportional to N k rather than N k, k, where N k is on average N/L. It facilitates producing higher sample density. Furthermore, when the loss value C is minimized to zero, the sample density will exponentially increase according to Eq. ( 9), as illustrated in Fig. 2(b ). The right panel of Fig. 1 also provides an intuitive insight on this property of the MMC loss: since the loss value C is proportional to the squared distance from the preset center µ * y , the feature points with lower loss values are certain to locate in a smaller volume around the center. Consequently, the feature points of the same class are encouraged to gather around the corresponding center, such that for each sample, there will be locally enough data in its neighborhood for robust learning (Schmidt et al., 2018) . The MMC loss value also becomes a reliable metric of the uncertainty on returned predictions.

Better exploiting model capacity. Behind the simple formula, the MMC loss can explicitly monitor inter-class dispersion by the hyperparameter C MM , while enabling the network to concentrate on minimizing intra-class compactness in training. Instead of repeatedly searching for an internal tradeoff in training as the center loss, the monotonicity of the supervisory signals induced by MMC can better exploit model capacity and also lead to faster convergence, as empirically shown in Fig. 3(a) . Avoiding the degradation problem. The MMC loss can naturally avoid the degradation problem encountered in Wen et al. (2016) when the center loss is not jointly used with the SCE loss, since the preset centers µ * for MMC are untrainable. In the test phase, the network trained by MMC can still return a normalized prediction with the softmax function. More details about the empirical superiorities of the MMC loss over other previous losses are demonstrated in Sec. 4.

Remark 3. In Appendix B.2, we discuss on why the squared-error form in Eq. ( 8) is preferred compared to, e.g., the absolute form or the Huber form in the adversarial setting. We further introduce flexible variants of the MMC loss in Appendix B.3, which can better adapt to various tasks.

Remark 4. Pang et al. (2018) propose a Max-Mahalanobis linear discriminant analysis (MMLDA) method, which assumes the features to distribute as an MMD. Due to the Gaussian mixture assumption, the training loss for the MMLDA method is obtained by the Bayes' theorem as

L MMLDA (Z(x), y) = -log   exp(- z-µ * y 2 2 2 ) l∈[L] exp(- z-µ * l 2 2 2 )   = -log exp(z µ * y ) l∈[L] exp(z µ * l )

. ( 10) Note that there is Σ i = 1 2 I in Eq. ( 4) for the MMLDA loss, similar with the SCE loss. Thus the MMLDA method cannot explicitly supervise on the sample density and induce high-density regions in the feature space, as analyzed in Sec. 3.2. Compared to the MMLDA method, the MMC loss introduces extra supervision on intra-class compactness, which facilitates better robustness.

4. EXPERIMENTS:

In this section, we empirically demonstrate several attractive merits of applying the MMC loss. We experiment on the widely used MNIST, CIFAR-10, and CIFAR-100 datasets (Krizhevsky & Hinton, 2009; LeCun et al., 1998) . The main baselines for the MMC loss are SCE (He et al., 2016) , Center loss (Wen et al., 2016) , MMLDA (Pang et al., 2018) , and L-GM (Wan et al., 2018) . The codes are provided in https://github.com/P2333/Max-Mahalanobis-Training.

4.1. PERFORMANCE ON THE CLEAN INPUTS:

The network architecture applied is ResNet-32 with five core layer blocks (He et al., 2016) . Here we use MMC-10 to indicate the MMC loss with C MM = 10, where C MM is assigned based on the cross-validation results in Pang et al. (2018) . The hyperparameters for the center loss, L-GM loss and the MMLDA method all follow the settings in the original papers (Pang et al., 2018; Wan et al., 2018; Wen et al., 2016) . The pixel values are scaled to the interval [0, 1]. For each training loss with or without the AT mechanism, we apply the momentum SGD (Qian, 1999) optimizer with the initial learning rate of 0.01, and train for 40 epochs on MNIST, 200 epochs on CIFAR-10 and CIFAR-100. The learning rate decays with a factor of 0.1 at 100 and 150 epochs, respectively. 

4.2. ADAPTIVE ATTACKS FOR THE MMC LOSS:

As stated in Athalye et al. (2018) , only applying the existing attacks with default hyperparameters is not sufficient to claim reliable robustness. Thus, we apply the adaptive versions of existing attacks when evading the networks trained by the MMC loss (detailed in Appendix B.4). For instance, the non-adaptive objectives for PGD are variants of the SCE loss (Madry et al., 2018) , while the adaptive objectives are -L MMC (z, y) and L MMC (z, y t ) in the untargeted and targeted modes for PGD, respectively. Here y t is the target label. To verify that the adaptive attacks are more effective than the non-adaptive ones, we modify the network architecture with a two-dimensional feature layer and visualize the PGD attacking procedure in Fig. 3(b ). The two panels separately correspond to two randomly selected clean inputs indicated by black stars. The ten colored clusters in each panel consist of the features of all the 10,000 test samples in MNIST, where each color corresponds to one class. We can see that the adaptive attacks are indeed much more efficient than the non-adaptive one.

4.3. PERFORMANCE UNDER THE WHITE-BOX ATTACKS:

We first investigate the white-box l ∞ distortion setting using the PGD attack, and report the results in Table 1 . According to Carlini et al. (2019) , we evaluate under different combinations of the attacking parameters: the perturbation , iteration steps, and the attack mode, i.e., targeted or untargeted. Following the setting in Madry et al. (2018) , we choose the perturbation = 8/255 and 16/255, with the step size be 2/255. We have also run PGD-100 and PGD-200 attacks, and find that the accuracy converges compared to PGD-50. In each PGD experiment, we ran several times with different random restarts to guarantee the reliability of the reported results.

Ablation study. To investigate the effect on robustness induced by high sample density in MMC, we substitute uniformly sampled center set (Liu et al., 2018; Duan et al., 2019) , i.e., µ r = {µ r l } l∈[L] for the MM center set µ * , and name the resulted method as "MMC-10 (rand)" as shown in Table 1 . There is also µ r l 2 = C MM , but µ r is no longer the solution of the min-max problem in Sec. 3.3. From the results in Table 1 , we can see that higher sample density alone in "MMC-10 (rand)" can already lead to much better robustness than other baseline methods even under the adaptive attacks, while using the optimal center set µ * as in "MMC-10" can further improve performance. When combining with the AT mechanism, the trained models have better performance under the attacks different from the one used to craft adversarial examples for training, e.g, PGD un 50 with = 16/255. Then we investigate the white-box l 2 distortion setting. We apply the C&W attack, where it has a binary search mechanism to find the minimal distortion to successfully mislead the classifier under the untargeted mode, or lead the classifier to predict the target label in the targeted mode. Following the suggestion in Carlini & Wagner (2017a) , we set the binary search steps to be 9 with the initial constant c = 0.01. The iteration steps for each value of c are set to be 1,000 with the learning rate of 0.005. In the Part I of Table 2 , we report the minimal distortions found by the C&W attack. As expected, it requires much larger distortions to successfully evade the networks trained by MMC. As suggested in Carlini et al. (2019) , providing evidence of being robust against the black-box attacks is critical to claim reliable robustness. We first perform the transfer-based attacks using PGD and MIM. Since the targeted attacks usually have poor transferability (Kurakin et al., 2018) , we only focus on the untargeted mode in this case, and the results are shown in Fig. 4 . We further perform the gradient-free attacks using the SPSA method and report the results in the Part II of Table 2 . To perform numerical approximations on gradients in SPSA, we set the batch size to be 128, the learning rate is 0.01, and the step size of the finite difference is δ = 0.01, as suggested by Uesato et al. (2018) . We also evaluate under stronger SPSA attacks with batch size to be 4096 and 8192 in Table 3 , where the = 8/255. With larger batch sizes, we can find that the accuracy under the black-box SPSA attacks converges to it under the white-box PGD attacks. These results indicate that training with the MMC loss also leads to robustness under the black-box attacks, which verifies that our method can induce reliable robustness, rather than the false one caused by, e.g., gradient mask (Athalye et al., 2018) .

4.4. PERFORMANCE UNDER THE BLACK-BOX ATTACKS:



4.5. PERFORMANCE UNDER THE GENERAL-PURPOSE ATTACKS:

To show that our method is generally robust, we further test under the general-purpose attacks (Carlini et al., 2019) . We apply the Gaussian noise (Fawzi et al., 2016; Gilmer et al., 2019) and rotation transformation (Engstrom et al., 2019) , which are not included in the data augmentation for training.

The results are given in the Part III of Table 2 . Note that the AT methods are less robust to simple transformations like rotation, as also observed in previous work (Engstrom et al., 2019) . In comparison, the models trained by the MMC loss are still robust to these easy-to-apply attacks.

4.6. EXPERIMENTS ON CIFAR-100:

In Table 4 and Table 5 , we provide the results on CIFAR-100 under the white-box PGD and C&W attacks, and the black-box gradient-free SPSA attack. The hyperparameter setting for each attack is the same as it on CIFAR-10. Compared to previous defense strategies which also evaluate on CIFAR-100 (Pang et al., 2019; Mustafa et al., 2019) , MMC can improve robustness more significantly, while keeping better performance on the clean inputs. Compared to the results on CIFAR-10, the averaged distortion of C&W on CIFAR-100 is larger for a successful targeted attack and is much smaller for a successful untargeted attack. This is because when only the number of classes increases, e.g., from 10 to 100, it is easier to achieve a coarse untargeted attack, but harder to make a subtle targeted attack. Note that in Table 5 , we also train on the ResNet-110 model with eighteen core block layers except for the ResNet-32 model. The results show that MMC can further benefit from deep network architectures and better exploit model capacity to improve robustness. Similar properties are also observed in previous work when applying the AT methods (Madry et al., 2018) . In contrast, as shown in Table 5 , the models trained by SCE are comparably sensitive to adversarial perturbations for different architectures, which demonstrates that SCE cannot take full advantage of the model capacity to improve robustness. This verifies that MMC provides effective robustness promoting mechanism like the AT methods, with much less computational cost. According to the formula of the hypersphere surface area (Loskot & Beaulieu, 2007) , we have

5. CONCLUSION:

EQUATION

where Γ(•) is the gamma function. Finally we can obtain the sample density as

EQUATION

)

B TECHNICAL DETAILS In this section, we provide more technical details we applied in our paper. Most of our experiments are conducted on the NVIDIA DGX-1 server with eight Tesla P100 GPUs.

B.1 GENERATION ALGORITHM FOR THE MAX-MAHALANOBIS CENTERS:

We give the generation algorithm for crafting the Max-Mahalanobis Centers in Algorithm 1, proposed by Pang et al. (2018) . Note that there are two minor differences from the originally proposed algorithm. First is that in Pang et al. (2018) they use C = µ i 2 2 , while we use C MM = µ i 2 . Second is that we denote the feature z ∈ R d , while they denote z ∈ R p . The Max-Mahalanobis centers generated in the low-dimensional cases are quite intuitive and comprehensible as shown in Fig. 5 . For examples, when L = 2, the Max-Mahalanobis centers are the two vertexes of a line segment; when L = 3, they are the three vertexes of an equilateral triangle; when L = 4, they are the four vertexes of a regular tetrahedron. Difference one: The inter-class margin

• The AMS losses induce the inter-class margins mainly by encouraging the intra-class compactness, while the weights are not explicitly forced to have large margins (Qi & Zhang, 2018 ). • The MMC loss simultaneously fixes the class centers to be optimally dispersed and encourages the intra-class distribution to be compact. Note that both of the two mechanisms can induce inter-class margins, which can finally lead to larger inter-class margins compared to the AMS losses.

Difference two: The normalization

• The AMS losses use both WN and FN to exploit the angular metric, which makes the normalized features distribute on hyperspheres. The good properties of the AMS losses are at the cost of abandoning the radial degree of freedom, which may reduce the capability of models. • In the MMC loss, there is only WN on the class centers, i.e., µ * y = C MM , and we leave the degree of freedom in the radial direction for the features to keep model capacity. However, note that the MMC loss z -µ * y 2 2 ≥ ( z 2 -C MM ) 2 is a natural penalty term on the feature norm, which encourage z 2 to not be far from C MM . This prevents models from increasing feature norms for easy examples and ignoring hard examples, just similar to the effect caused by FN but more flexible.

:



Figures and Tables:

Figure 1: Intuitive illusion of how training data moves and how sample density varies in a two-dimensional feature space during the training procedure.

Intuitively, D k, k includes the data with the true label of class k, while the highest prediction returned by the classifier is class k among other classes. Then we can derive the approximated sample density in the feature space induced by the g-SCE loss, as stated in the following theorem: Theorem 1. (Proof in Appendix A.1) Given (x, y) ∈ D k, k, z = Z(x) and L g-SCE (z, y) = C, if there are Σ k = σ k I, Σ k = σ kI , and σ k = σ k, then the sample density nearby the feature point z based on the approximation in Eq. (

Figure 3: (a) Test error rates on clean images w.r.t training time on CIFAR-10. Here AT refers to 10-steps targeted PGD adversarial training, i.e., AT tar 10 . (b) Two-dimensional visualization of the attacks on trained MMC networks in the feature space of MNIST. For each attack there is = 0.3 with step size of 0.01. The total number of iteration steps is 50, where Iter-n indicates the perturbed features at n-th iteration step.

Figure 4: Classification accuracy under the black-box transfer-based attacks on the test set of CIFAR-10. The substitute model refers to the one used to craft adversarial examples, and the target model is the one that an adversary actually intends to fool. Here AT refers to AT tar 10 .

Figure 5: Intuitive illustration of the Max-Mahalanobis centers in the cases of L = 2, 3, 4.

The constant C MM , the dimension of vectors d and the number of classes L. (L ≤ d + 1) Initialization: Let the L mean vectors be µ * 1 = e 1 and µ *i = 0 d , i = 1.Here e 1 and 0 d separately denote the first unit basis vector and the zero vector in R d .for i = 2 to L do for j = 1 to i -1 do µ * i (j) = -[1 + µ * i , µ * j • (L -1)]/[µ * j (j) • (Lk = 1 to L do µ * k = C MM • µ * k end for Return:The optimal mean vectors µ * i , i ∈ [L].

Classification accuracy (%) on the white-box adversarial examples crafted on the test set of CIFAR-10. The superscript tar indicates targeted attacks, while un indicates untargeted attacks. The subscripts indicate the number of iteration steps when performing attacks. The results w.r.t the MMC loss are reported under the adaptive versions of different attacks. The notation ≤ 1 represents accuracy less than 1%. The MMC-10 (rand) is an ablation study where the class centers are uniformly sampled on the hypersphere.

Table content: <table><tr><td/><td/><td colspan="4">Perturbation = 8/255</td><td colspan="4">Perturbation = 16/255</td></tr><tr><td>Methods</td><td colspan="2">Clean PGD tar 10</td><td>PGD un 10</td><td>PGD tar 50</td><td>PGD un 50</td><td>PGD tar 10</td><td>PGD un 10</td><td>PGD tar 50</td><td>PGD un 50</td></tr><tr><td>SCE</td><td>92.9</td><td>≤ 1</td><td>3.7</td><td>≤ 1</td><td>3.6</td><td>≤ 1</td><td>2.9</td><td>≤ 1</td><td>2.6</td></tr><tr><td>Center loss</td><td>92.8</td><td>≤ 1</td><td>4.4</td><td>≤ 1</td><td>4.3</td><td>≤ 1</td><td>3.1</td><td>≤ 1</td><td>2.9</td></tr><tr><td>MMLDA</td><td>92.4</td><td>≤ 1</td><td>16.5</td><td>≤ 1</td><td>9.7</td><td>≤ 1</td><td>6.7</td><td>≤ 1</td><td>5.5</td></tr><tr><td>L-GM</td><td>92.5</td><td>37.6</td><td>19.8</td><td>8.9</td><td>4.9</td><td>26.0</td><td>11.0</td><td>2.5</td><td>2.8</td></tr><tr><td>MMC-10 (rand)</td><td>92.3</td><td>43.5</td><td>29.2</td><td>20.9</td><td>18.4</td><td>31.3</td><td>17.9</td><td>8.6</td><td>11.6</td></tr><tr><td>MMC-10</td><td>92.7</td><td>48.7</td><td>36.0</td><td>26.6</td><td>24.8</td><td>36.1</td><td>25.2</td><td>13.4</td><td>17.5</td></tr><tr><td>AT tar 10 (SCE)</td><td>83.7</td><td>70.6</td><td>49.7</td><td>69.8</td><td>47.8</td><td>48.4</td><td>26.7</td><td>31.2</td><td>16.0</td></tr><tr><td>AT tar 10 (MMC-10)</td><td>83.0</td><td>69.2</td><td>54.8</td><td>67.0</td><td>53.5</td><td>58.6</td><td>47.3</td><td>44.7</td><td>45.1</td></tr><tr><td>AT un 10 (SCE)</td><td>80.9</td><td>69.8</td><td>55.4</td><td>69.4</td><td>53.9</td><td>53.3</td><td>34.1</td><td>38.5</td><td>21.5</td></tr><tr><td>AT un 10 (MMC-10)</td><td>81.8</td><td>70.8</td><td>56.3</td><td>70.1</td><td>55.0</td><td>54.7</td><td>37.4</td><td>39.9</td><td>27.7</td></tr><tr><td colspan="2">PGD 10 𝐮𝐧 , 𝜀 = 8/256</td><td/><td colspan="2">MIM 10 𝐮𝐧 , 𝜀 = 8/256</td><td colspan="2">PGD 10 𝐮𝐧 , 𝜀 = 16/256</td><td colspan="2">MIM 10 𝐮𝐧 , 𝜀 = 16/256</td><td/></tr></table>

Experiments on CIFAR-10. Part I: Averaged l2 distortion of the white-box adversarial examples crafted by C&W with 1,000 iteration steps. Part II: Classification accuracy (%) under the block-box SPSA attack. Part III: Classification accuracy (%) under general transformations. The standard deviation σ for the Gaussian noise is 0.05, the degree range is ±30 • for random rotation.

Table content: <table><tr><td/><td>Part I</td><td/><td colspan="4">Part II ( = 8/255) Part II ( = 16/255)</td><td colspan="2">Part III</td></tr><tr><td>Methods</td><td colspan="3">C&amp;W tar C&amp;W un SPSA tar 10</td><td>SPSA un 10</td><td>SPSA tar 10</td><td>SPSA un 10</td><td colspan="2">Noise Rotation</td></tr><tr><td>SCE</td><td>0.12</td><td>0.07</td><td>12.3</td><td>1.2</td><td>5.1</td><td>≤ 1</td><td>52.0</td><td>83.5</td></tr><tr><td>Center loss</td><td>0.13</td><td>0.07</td><td>21.2</td><td>6.0</td><td>10.6</td><td>2.0</td><td>55.4</td><td>84.9</td></tr><tr><td>MMLDA</td><td>0.17</td><td>0.10</td><td>25.6</td><td>13.2</td><td>11.3</td><td>5.7</td><td>57.9</td><td>84.8</td></tr><tr><td>L-GM</td><td>0.23</td><td>0.12</td><td>61.9</td><td>45.9</td><td>46.1</td><td>28.2</td><td>59.2</td><td>82.4</td></tr><tr><td>MMC-10</td><td>0.34</td><td>0.17</td><td>69.5</td><td>56.9</td><td>57.2</td><td>41.5</td><td>69.3</td><td>87.2</td></tr><tr><td>AT tar 10 (SCE)</td><td>1.19</td><td>0.63</td><td>81.1</td><td>67.8</td><td>77.9</td><td>59.4</td><td>82.2</td><td>76.0</td></tr><tr><td>AT tar 10 (MMC-10)</td><td>1.91</td><td>0.85</td><td>79.1</td><td>69.2</td><td>74.5</td><td>62.7</td><td>83.5</td><td>75.2</td></tr><tr><td>AT un 10 (SCE)</td><td>1.26</td><td>0.68</td><td>78.8</td><td>67.0</td><td>73.7</td><td>60.3</td><td>78.9</td><td>73.7</td></tr><tr><td>AT un 10 (MMC-10)</td><td>1.55</td><td>0.73</td><td>80.4</td><td>69.6</td><td>74.6</td><td>62.4</td><td>80.3</td><td>75.8</td></tr></table>

Accuracy (%) of MMC-10 under SPSA with different batch sizes.

Table content: <table><tr><td/><td>CIFAR-10</td><td/></tr><tr><td colspan="2">Batch SPSA un 10</td><td>SPSA tar 10</td></tr><tr><td>128</td><td>57.0</td><td>69.0</td></tr><tr><td>4096</td><td>41.0</td><td>52.0</td></tr><tr><td>8192</td><td>37.0</td><td>49.0</td></tr></table>

Experiments on CIFAR-100. Part I: Classification accuracy (%) on the clean test samples. Part II: Classification accuracy (%) under the white-box PGD attacks and the block-box SPSA attack. The attacks are adaptive for MMC. Here the batch size for SPSA is 128. Part III: Averaged l2 distortion of the white-box adversarial examples crafted by C&W with 1,000 iteration steps and 9 binary search epochs.

Table content: <table><tr><td/><td>Part I</td><td/><td colspan="2">Part II ( = 8/255)</td><td/><td/><td>Part I</td></tr><tr><td>Methods</td><td colspan="2">Clean PGD tar 10</td><td>PGD un 10</td><td>SPSA tar 10</td><td>SPSA un 10</td><td colspan="2">C&amp;W tar C&amp;W un</td></tr><tr><td>SCE</td><td>72.9</td><td>≤ 1</td><td>8.0</td><td>14.0</td><td>1.9</td><td>0.16</td><td>0.047</td></tr><tr><td>Center</td><td>72.8</td><td>≤ 1</td><td>10.2</td><td>14.7</td><td>2.3</td><td>0.18</td><td>0.048</td></tr><tr><td>MMLDA</td><td>72.2</td><td>≤ 1</td><td>13.9</td><td>18.5</td><td>5.6</td><td>0.21</td><td>0.050</td></tr><tr><td>L-GM</td><td>71.3</td><td>15.8</td><td>15.3</td><td>22.8</td><td>7.6</td><td>0.31</td><td>0.063</td></tr><tr><td>MMC-10</td><td>71.9</td><td>23.9</td><td>23.4</td><td>33.4</td><td>15.8</td><td>0.37</td><td>0.085</td></tr></table>

In this paper, we formally demonstrate that applying the softmax function in training could potentially lead to unexpected supervisory signals. To solve this problem, we propose the MMC loss to learn more structured representations and induce high-density regions in the feature space. In our experiments, we empirically demonstrate several favorable merits of our method: (i) Lead to reliable robustness even under strong adaptive attacks in different threat models; (ii) Keep high performance on clean inputs comparable to SCE; (iii) Introduce little extra computation compared to the SCE loss; (iv) Compatible with the existing defense mechanisms, e.g., the AT methods. Our analyses in this paper also provide useful insights for future work on designing new objectives beyond the SCE framework.

Table content: <table/>

Classification accuracy (%) on the white-box adversarial examples crafted on the test set of CIFAR-10 and CIFAR-100. The results w.r.t the MMC loss are reported under the adaptive versions of different attacks. MMC can better exploit deep architectures, while SCE cannot.

Table content: <table><tr><td/><td/><td colspan="4">Perturbation = 8/255</td><td colspan="4">Perturbation = 16/255</td></tr><tr><td>Methods</td><td colspan="2">Cle. PGD tar 10</td><td>PGD un 10</td><td>PGD tar 50</td><td>PGD un 50</td><td>PGD tar 10</td><td>PGD un 10</td><td>PGD tar 50</td><td>PGD un 50</td></tr><tr><td/><td/><td/><td/><td colspan="2">CIFAR-10</td><td/><td/><td/><td/></tr><tr><td>SCE (Res.32)</td><td>93.6</td><td>≤ 1</td><td>3.7</td><td>≤ 1</td><td>3.6</td><td>≤ 1</td><td>2.7</td><td>≤ 1</td><td>2.9</td></tr><tr><td>MMC (Res.32)</td><td>92.7</td><td>48.7</td><td>36.0</td><td>26.6</td><td>24.8</td><td>36.1</td><td>25.2</td><td>13.4</td><td>17.5</td></tr><tr><td>SCE (Res.110)</td><td>94.7</td><td>≤ 1</td><td>3.0</td><td>≤ 1</td><td>2.9</td><td>≤ 1</td><td>2.1</td><td>≤ 1</td><td>2.0</td></tr><tr><td colspan="2">MMC (Res.110) 93.6</td><td>54.7</td><td>46.0</td><td>34.4</td><td>31.4</td><td>41.0</td><td>30.7</td><td>16.2</td><td>21.6</td></tr><tr><td/><td/><td/><td/><td colspan="2">CIFAR-100</td><td/><td/><td/><td/></tr><tr><td>SCE (Res.32)</td><td>72.3</td><td>≤ 1</td><td>7.8</td><td>≤ 1</td><td>7.4</td><td>≤ 1</td><td>4.8</td><td>≤ 1</td><td>4.7</td></tr><tr><td>MMC (Res.32)</td><td>71.9</td><td>23.9</td><td>23.4</td><td>15.1</td><td>21.9</td><td>16.4</td><td>16.7</td><td>8.0</td><td>15.7</td></tr><tr><td>SCE (Res.110)</td><td>74.8</td><td>≤ 1</td><td>7.5</td><td>≤ 1</td><td>7.3</td><td>≤ 1</td><td>4.7</td><td>≤ 1</td><td>4.5</td></tr><tr><td colspan="2">MMC (Res.110) 73.2</td><td>34.6</td><td>22.4</td><td>23.7</td><td>16.5</td><td>24.1</td><td>14.9</td><td>13.9</td><td>10.5</td></tr></table>

Peer review (iclr2020):
[Post rebuttal: The rebuttal addresses most of my concerns. I have updated my score.] This paper points out some limitation of the traditional softmax cross-entropy (SCE) loss, e.g., causing indirect and unexpected supervisory signals on the learned features so that the points with low loss values tend to spread over the space sparsely. To remedy this, the authors then proposes the max-mahalanobis center (MMC) loss. By analyzing the sample density, it is proved that the MMC loss has a sample density proportional to the number of data samples in each class. This means MMC loss induces higher feature densities than the SCE loss, thus is expected to be more robust under adverssarial attacks. I find the paper fairly well written in general, and the arguments are well supported by the development of the theoretical results. There are, however, several questions raised by the reviewer. One question is that by looking at the definition (8), it seems the MMC loss essentially tries to move data from different classes apart. This seems to be quite similar to the max margin loss. I would expect the authors differentiate the differences between these two. Consequently, I would also like to see empirical comparisons between these two losses. Another question is even if you have Theorem 2, the transfer from Theorem 2 to robustness does not seem direct, e.g., how do you guarantee when the density form in eq.9 is better than that in eq.7 for robustness? Then when looking back at the sample density definition in eq. (2). I wonder how the Vol is defined to guarantee eq.2 is a valid density function? e.g., to guarantee the integration equals to 1. In terms of experiments, since it is claimed that the proposed loss adds little computation cost compared to the SCE loss, I think it is better to include running time comparison in the results. The last two lines in page 8, what are unseen attacks? From the experiments, it seems that it means the PGD with different steps. I think the authors should be careful to use the term, because some literature use the term *different attacks* to refer to attacks with different *norms*. Second line below Table 5: it is said the MMC loss keeps state-of-the-art performance on clean data. However, by looking at Table 4, the state-of-the-art is obvious SCE. Why did you say that? Others: 1. S_k,	ilde{k}^2 is undefined. 2. How do you solve the minmax problem below eq.8? I guess you use the algorithm in Appendix B.1? Please clarify.