Title: LOGIT REGULARIZATION METHODS FOR ADVERSAR-IAL ROBUSTNESS

Abstract: While great progress has been made at making neural networks effective across a wide range of tasks, many are surprisingly vulnerable to small, carefully chosen perturbations of their input, known as adversarial examples. In this paper, we advocate for and experimentally investigate the use of logit regularization techniques as an adversarial defense, which can be used in conjunction with other methods for creating adversarial robustness at little to no cost. We demonstrate that much of the effectiveness of one recent adversarial defense mechanism can be attributed to logit regularization and show how to improve its defense against both white-box and black-box attacks, in the process creating a stronger black-box attacks against PGD-based models.
Neural networks, despite their high performance on a variety of tasks, can be brittle. Given data intentionally chosen to trick them, many deep learning models suffer extremely low performance. This type of data, commonly referred to as adversarial examples, represent a security threat to any machine learning system where an attacker has the ability to choose data input to a model, potentially allowing the attacker to control a model's behavior.
Today, adversarial examples are typically created by small, but carefully chosen transformations of data that models are otherwise high-performant on. This is primarily due to the ease of experimentation with existing datasets (Gilmer et al., 2018) , though the full threat of adversarial examples is only limited by the ability and creativity of an attacker's example generation process.
Even with the limited threat models considered in current research, performance on adversarially chosen examples can be dramatically worse than unperturbed data -for example, white-box accuracy on adversarially chosen examples for the CIFAR-10 image classification task (Krizhevsky & Hinton, 2009) is lower than 50%, even for the most robust defenses known today (Madry et al., 2018; Kannan et al., 2018) , while unperturbed accuracy can be as high as 98.5% Cubuk et al. (2018) .
Current defenses against adversarial examples generally come in one of a few flavors. Perhaps the most common approach is to generate adversarial examples as part of the training procedure and explicitly train on them ("adversarial training"). Another approach is to transform the model's input representation in a way that thwarts an attacker's adversarial example construction mechanism. While these methods can be effective, care must be taken to make sure that they are not merely obfuscating gradients (Athalye et al., 2018) . Last, generative models can be built to model the original data distribution, recognizing when the input data is out of sample and potentially correcting it (Song et al., 2018; Samangouei et al., 2018) . Of these, perhaps the most robust today is adversarial logit pairing (Kannan et al., 2018) , which extends the adversarial training work of Madry et al. (2018) by incorporating an additional term to make the logits (pre-softmax values) of an unperturbed and adversarial example more similar.

:

In this work, we show that adversarial logit pairing derives a large fraction of its benefits from regularizing the model's logits toward zero, which we demonstrate through simple and easy to understand theoretical arguments in addition to empirical demonstration. Investigating this phenomenon further, we examine two alternatives for logit regularization, finding that both result in improved robustness to adversarial examples, sometimes surprisingly so -for example, using the right amount of label smoothing (Szegedy et al., 2016) can result in greater than 40% robustness to a projected gradient descent (PGD) attack (Madry et al., 2018) on CIFAR-10 while training only on the original, unperturbed training examples, and is also a compelling black-box defense. We then present an alternative formulation of adversarial logit pairing that separates the logit pairing and logit regularization effects, improving the defense. The end result of these investigations is a defense that sets a new state-of-the-art for PGD-based adversaries on CIFAR-10 for both white box and black box attacks, while requiring little to no computational overhead on top of adversarial training.

2. OVERVIEW OF ADVERSARIAL TRAINING:

Before proceeding with our analysis, it is prudent to review existing work on adversarial training for context. While adversarial examples have been examined in the machine learning community in some capacity for many years (Dalvi et al., 2004) , their study has drawn a sharp focus in the current renaissance of deep learning, starting with Szegedy et al. (2014) and Goodfellow et al. (2015) . In Goodfellow et al. (2015) , adversarial training is presented as training with a weighted loss between an original and adversarial example, i.e. with a loss of

J(θ, x, y) = 1 m m i=1

αJ(θ, x (i) , y (i) ) + (1 -α)J(θ, g(x (i) ), y (i) )

(1)

where g(x) is a function representing the adversarial example generation process, originally presented as g(x) = x + • sign(∇ x J(θ, x, y)), α is a weighting term between the original and adversarial examples typically set to 0.5, and as usual θ are the model parameters to learn, J is a cross-entropy loss, m is the dataset size, x (i) is the ith input example, and y (i) is its label. Due to the use of a single signed gradient with respect to the input example, this method was termed the "fast gradient sign method" (FGSM), requiring a single additional forward and backward pass of the network to create. Kurakin et al. (2016) extended FGSM into a multi-step attack, iteratively adjusting the perturbation applied to the input example through several rounds of FGSM. This was also the first attack that could be described as a variant of projected gradient descent (PGD). Both of these approaches primarily target an L ∞ threat model, where the L ∞ norm between the original and adversarial example is constrained to a small value. Madry et al. (2018) built upon these works by initializing the search process for the adversarial perturbation randomly, and is the strongest attack currently available to the best of our knowledge.

Through extensive experiments, they showed that even performing PGD with a single random initialization is able to approximate the strongest adversary found with current first-order methods. However, as with multi-step FGSM, performing adversarial training with this approach can be rather expensive, taking an order of magnitude longer than standard training due to requiring N +1 forward and backward passes of the model, where N is the number of PGD iterations. 

J(θ, x, y) = 1 m m i=1 αJ(θ, x (i) , y (i) ) + (1 -α)J(θ, g(x (i) ), y (i) ) + λL(f (x (i) ; θ), f (g(x (i)) ; θ)).

(

EQUATION

where L was set to an L 2 loss in experiments and f (x, θ) returns the logits of the model corresponding to example x. Adversarial logit pairing has the motivation of increasing the amount of structure in the learning process, by encouraging the model to have similar prediction patterns on the original and adversarial examples, a process reminiscent of distillation (Hinton et al., 2015) . 

c as the logit of the model for class c on example i in its original, unperturbed form, and ˜ (i) c as the logit for the corresponding adversarial example. The logit pairing term in adversarial logit pairing is a simple L 2 loss:

L = 1 2 ( (i) c -˜ (i) c ) 2 (3)

While it is obvious that minimizing this term will have the effect of making the original and adversarial logits more similar in some capacity, what precise effect does it have on the model during training? To examine this, we look at the gradient of this loss term with respect to the logits themselves:

∂L ∂ (i) c = (i) c -˜ (i) c (4) ∂L ∂ ˜ (i) c = ˜ (i) c -(i) c (5)

Under the assumption that the adversarial example moves the model's predictions away from the correct label (as should be the case with any reasonable adversarial example, such as an untargeted PGD-based attack), we have that

(i) c > ˜ (i) c when c = y (i)

is the correct category, and

(i) c < ˜ (i) c otherwise.

Keeping in mind that model updates move in the direction opposite of the gradient, then the update to the model's weights will attempt to make the original logits smaller and the adversarial logits larger when c = y (i) and will otherwise attempt to make the original logits larger and the adversarial logits smaller.

However, this must be considered in the context of the adversarial training loss J -in particular, the standard cross-entropy loss used in J for the adversarial example g(x (i) ) already encourages the adversarial logits to be higher for the correct category and smaller for all incorrect categories, and furthermore the scale of the loss J typically is an order of magnitude larger than the adversarial pairing loss. Thus, we argue that the main effect of adversarial logit pairing is actually in the remaining two types of updates, encouraging the logits of the original example to be smaller for the correct category and larger for all incorrect categories. These two updates have very similar effects to simply regularizing model logits, e.g. in a manner similar to "logit squeezing" Kannan et al. (2018) or label smoothing Szegedy et al. (2016) .

We can also view this from a different perspective by explicitly incorporating the scale of the logits in the logit pairing term. If we factor out a shared scale factor γ from each logit, the logit pairing term becomes

EQUATION

meaning that the model would always attempt to update the scale of the logits in the opposite direction of the sign of γ, i.e. toward zero, so long as the logits were not identical. In practice, this affect is counterbalanced by the adversarial training term, which requires non-identical logits to minimize its cross-entropy loss.

Given this interpretation, in this work we now explore 1) whether this can be verified experimentally, 2) if other forms of logit regularization have similar effects, 3) if decoupling adversarial logit pairing explicitly into a form where the effect of logit regularization and pairing can be disentangled, and 4) whether the above analysis can yield insights to making more adversarially robust methods.

3.1. EXPERIMENTAL EVIDENCE:

Perhaps the most straightforward way to test our hypothesis is to examine the logits of a model trained with ALP vs one trained with standard adversarial training. If true, the model trained with ALP will have logits that are generally smaller in magnitude. We do this in Figure 1 (left), using a ResNet (He et al., 2016) classifier on CIFAR-10 ( Krizhevsky & Hinton, 2009) as our testbed.

We see that it is indeed the case that the logits for a model trained with ALP are of smaller magnitude than those of a model trained with PGD, with a variance reduction of the logits from 8.31 to 4.02 on clean test data (distributions on a set of PGD adversarial examples are very similar). This provides evidence that ALP does have the effect of regularizing logits, though this data alone is not sufficient to determine if this is a key mechanism in improved performance.

To answer this, we can examine if standard adversarial training can be improved by explicitly regularizing the logits. If adversarial robustness can be improved, but similar improvements can not be made to ALP, then at least some of the benefits of ALP can be attributed to logit regularization. We present the results of this experiments in Figure 1 (right), implemented using the "logit squeezing" form of regularization (L 2 -regularization on the logits).

These results show that incorporating regularization on model logits is able to recover slightly more than half of the improvement from logit pairing, with too little regularization having only a small effect, and too much regularization approaching the point of being harmful. However, when added to a model already trained with ALP, regularizing the logits does not lead to any improvement, and in fact hurts performance, likely due to putting too much strength into logit regularization. This evidence makes clear that one of the key improvements from logit pairing is due to a logit regularization effect.

We would like to emphasize that these results are not meant to diminish ALP in any sense -adversarial logit pairing works, and our goals are to investigate the mechanism by which it works and explore if it can be generalized or improved. Given these results, it is worth examining other methods that have an effect of regularizing logits in order to tell whether this is a more general phenomenon.

4. OTHER FORMS OF LOGIT REGULARIZATION:

Label Smoothing. Label smoothing is the process of replacing the one-hot training distribution of labels with a softer distribution, where the probability of the correct class has been smoothed out onto the incorrect classes (Szegedy et al., 2016) . Concretely, label smoothing uses the target distribution:

EQUATION

where p

(i)

c is the target probability for class c for example i, C is the number of categories and s ∈ [0, 1 -1 C ] is the smoothing strength. Label smoothing was originally introduced as a form of regularization, designed to prevent models from being too confident about training examples, with the goal of improved generalization. It can be easily implemented as a preprocessing step on the labels, and does not affect model training time in any significant way. Interestingly, Kurakin et al. (2016) found that disabling the small amount of label smoothing present in a model trained on ImageNet actually improved adversarial robustness roughly by 1%. Here we find a different effect, with the caveat of relatively different experimental setups from Kurakin et al. (2016) .

In Figure 2 (left) we show the effect label smoothing has on the performance of a model trained only on clean (i.e. non-adversarial) training data. Very surprisingly, using only label smoothing can result in a model that is nearly as robust as models trained with PGD-based adversarial training or adversarial logit pairing and take an order of magnitude less time to train -though we note that when PGD and ALP-based models are trained only on adversarial examples rather than a mixture of clean and adversarial data, their robustness exceeds this performance by around 5%. Furthermore, this benefit of label smoothing comes at no significant loss in accuracy on unperturbed test data, while generally, adversarial training tends to trade off original vs adversarial performance. Another curiosity is that adding in any label smoothing at all dramatically improves robustness to FGSMbased adversaries (adding label smoothing of s = .01 brought accuracy up from 6.1% to 38.3%), while PGD-based attacks saw much more gradual improvement.

Examining the logits (Figure 2 , right), we see a striking difference between the models -the model trained with label smoothing both has a dramatically smaller dynamic range of logits and also presents a much more bimodal logit distribution than the model trained without label smoothing. In other words, it has learned to predict extremely consistent values for logits, which is what we attribute its adversarial robustness to. Anecdotally, we observed that this behavior held for all positive values of s, with a stronger effect the higher s was. Additional experiments with label smoothing are given in Section 5.3. 2018), these types of data augmentation typically have the form of element-wise weighted averaging of two input examples (typically images), with the training label also determined as a weighted average of the original two training labels (represented as one-hot vectors). Besides making target labels soft (i.e. not 1-of-K) during training time, these methods also encourage models to behave linearly between examples, which may improve robustness to out of sample data. Interestingly, Zhang et al. (2018) found that this type of data augmentation improved robustness to FGSM attacks on ImageNet (Russakovsky et al., 2015) , but Kannan et al. (2018) found that the method did not improve robustness against a targeted attack with a stronger PGD-based adversary.

Paired-Example Data Augmentation:

In our experiments we found evidence agreeing with both conclusions -when applying mixup (Zhang et al., 2018) , we found a sizeable increase in robustness to FGSM adversaries, going from 6.1% on CIFAR-10 by training without mixup to 30.8% with mixup, but did not observe a significant change when evaluated against a PGD-based adversary. While robustness to a PGD adversary with only 5 steps increased by a tiny amount (from 0.0% to 0.5%), robustness to a 10step PGD adversary remained at 0%. In our experiments, we use VH-mixup, the slightly improved version of mixup from Summers & Dinneen (2018) .

4.1. DECOUPLING ADVERSARIAL LOGIT PAIRING:

While we have now considered alternate methods by which logits can be regularized, at this point it is still not clear exactly how they might be used with or interact with the logit regularization effect of adversarial logit pairing. Doing so requires separating out the logit pairing and logit regularization effects of ALP.

In adversarial logit pairing Kannan et al. (2018) , the logit pairing term is implemented as an L 2 loss:

EQUATION

though other losses such as an L 1 or Huber loss are also possible. We would like to break L into separate pairing and regularization terms:

EQUATION

)

where the purpose of the first term is explicitly for making the logits more similar (with as little logit regularization as possible), and the second term is explicitly for regularizing the logits toward zero.

There are several natural choices for h, such as the the Jensen-Shannon divergence, a cosine similarity, or any similarity metric that does not have a significant regularization effect. We have found that simply taking the cross entropy between the distributions induced by the logits was effective -depending on the actual values of the logits, this can either still have a mild squeezing effect (if the logits are very different), a mild expanding effect (if the logits are very similar), or something in between.

One implementation detail worth noting is that it can be difficult to reason about and set the relative strengths of the pairing loss and adversarial training loss. To that end, we set the strength of the pairing loss h as a constant fraction of the adversarial loss, implemented by setting the coefficient of the loss as a constant multiplied by a non-differentiable version of the ratio between the losses.

By decomposing adversarial logit pairing explicitly into logit pairing and logit regularization terms in this way, adversarial robustness to a 10-step PGD attack improves by an absolute 1.9% over ALP, or 5.6% over standard PGD-based adversarial training .

5. ADDITIONAL EXPERIMENTS:



5.1. IMPLEMENTATION DETAILS:

In the experiments on CIFAR-10 throughout this paper, we used a ResNet (He et al., 2016) , equivalent to the "simple" model of Madry et al. (2018) , with a weight decay of 2 • 10 -4 and a momentum optimizer with strength of 0.9. Standard data augmentation of random crops and horizontal flips was used. After a warm up period of 5 epochs, the learning rate peaked at 0.1 and decayed by a factor of 10 at 100 and 150 epochs, training for a total of 200 epochs for models not trained on adversarial examples and 101 epochs for models using adversarial training -adversarial accuracy tends to increase for a brief period of time after a learning rate decay, then quickly drop by a small amount, an empirical finding also echoed by Schmidt et al. (2018) . The minibatch size was 128.

Adversarial examples were constrained to a maximum L ∞ norm of .03, and all PGD-based attacks used a step size of 0.0078. Adversarial attacks were constructed using the CleverHans library (Papernot et al., 2018) , implemented in TensorFlow (Abadi et al., 2016) . All experiments were done on two Nvidia Geforce GTX 1080 Ti GPUs.

5.2. TOWARDS A MORE ROBUST MODEL:

Given these forms of logit regularization, perhaps the most natural question is whether they can be combined to create an even more robust model. Thus, in this section we focus exclusively on making a model (and comparable baselines) as robust as possible to PGD-based attacks. In particular, for baseline methods (PGD-based adversarial training (Madry et al., 2018) To combine the logit regularization methods together, we use a small amount of label smoothing (s = 0.1), use VH-mixup (Summers & Dinneen, 2018) on the input examples, use the logit pairing formulation of Section 4.1 with β = 10 -3 , and set the ratio between the adversarial training loss and the pairing loss to 0.125, which focuses the loss on keeping adversarial and original examples similar. These parameters were not tuned much due to resource constraints. We refer to this combination simply as LRM ("Logit Regularization Methods").

White-box performance is shown in Table 1 . LRM achieves the highest level of adversarial robustness of the methods considered for all PGD-based attacks, and to the best of our knowledge represents the most robust method on CIFAR-10 to date. However, like other adversarial defenses, this comes at the cost of performance on the original test set, which makes sense -from the perspective of adversarial training, a clean test image is simply the center of the set of feasible adversarial examples. Nonetheless, it is interesting that the tradeoff between adversarial and non-adversarial performance can continue to be pushed further, with the optimal value of that tradeoff dependent on application (e.g. whether worst-case performance is more important than performance on unperturbed examples).

Next, black-box performance is shown in 

5.3. LABEL SMOOTHING INVESTIGATIONS:

In both the white-box and black-box analyses, we found that label smoothing was surprisingly effective given its near-zero cost. For black-box attacks in particular (Table 2 ), label smoothing was generally among the most robust models across all different sources, with the label smoothing target network having the highest minimum performance across sources, a fact which was not shared even by any of the adversarially-trained models. Inspired by this, we conducted further investigation where we have found similarly surprising mixed behavior.

In this experiment, we investigated the robustness of a model trained with label smoothing to stronger white-box attacks, noting that performance on PGD-based attacks from Table 1 dropped considerably when moving from 5 iterations of PGD to 10 and 20 iterations. We found that this trend continued, with accuracy dropping all the way to 13.6% when performing a 200-iteration PGD attack, a trend that was not observed with any of the three models trained on PGD attacks. This suggests that label smoothing, while providing only a mild amount of worst-case adversarial robustness, can actually make the adversarial optimization problem much more challenging, which we believe is also the underlying reason for its effectiveness against black-box attacks. The exact mechanism by which it does this, however, remains elusive, which we think is an interesting avenue for further research in black-box defenses.

6. DISCUSSION:

In this work, we have shown the usefulness of logit regularization for improving the robustness of neural networks to adversarial examples. We first presented an analysis of adversarial logit pairing, the current state-of-the-art in adversarial defense, showing that roughly half of its improvement over adversarial training can be attributed to a non-obvious logit regularization effect. Based on this, we investigated two other forms of logit regularization, demonstrating the benefits of both, and then presented an alternative method for adversarial logit pairing that more cleanly decouples the logit pairing and logit regularization effects while also improving performance.

By combining these logit regularization techniques together, we were able to create both a stronger defense against white-box PGD-based attacks and also a stronger attack against PGD-based defenses, both of which come at almost no additional cost to PGD-based adversarial training. We also showed the surprising strength of label smoothing as a black-box defense and its corresponding weakness to only highly-optimized white-box attacks.

We anticipate that future work will push the limits of logit regularization even further to improve defenses against adversarial examples, possibly using more techniques originally devised for other purposes (Pereyra et al., 2017) . We also hope that these investigations will yield insights into training adversarially-robust models without the overhead of multi-step adversarial training, an obstacle that has made it challenge to scale up adversarial defenses to larger datasets without a sizable computational budget.



Figures and Tables:

Figure 1: Left: Distribution of logits on clean test data for models trained with and without logit pairing. Right: Performance against a PGD-based attack for models trained with varying amounts of logit regularization, with and without logit pairing.

Figure 2: Left: Clean and adversarial accuracy on CIFAR-10 as a function of the label smoothing strength. Note that models for this figure were trained exclusively on the original training data -no adversarial examples were involved in the training procedure. Right: Logit distribution of model trained with no label smoothing (top) and with label smoothing of s = .75 (bottom), evaluated on the original test images (left) and PGD-based adversarial examples (right).

Recently, a new form of data augmentation was found that, in contrast to standard label-preserving data augmentation, combined different training examples together, dramatically altering both the appearance of the training examples and their labels. Introduced concurrently by Zhang et al. (2018); Inoue (2018); Tokozume et al. (

and adversarial logit pairing (Kannan et al., 2018)), we opt to train exclusively on adversarial examples, effectively setting α = 0 in Equation 1, which roughly trades off accuracy of 4 -5% for clean test examples for a similar gain in adversarial performance.

White-box accuracy of models on CIFAR-10.

Table content: <table><tr><td>Adversary</td></tr></table>

As is standard in most black-box evaluations of adversarial defenses, this is performed by generating adversarial examples with one model (the "Source") and evaluating them on a separate independently trained model (the "Target"). As found in other works (e.

Table content: <table/>

Black-box accuracy of models on CIFAR-10 with a 10-step PGD adversary.

Table content: <table><tr><td>Source</td></tr></table>

Peer review (iclr2019):
 The paper studies how logit regularization techniques can affect adversarial robustness. The authors study the recently proposed technique of adversarial logit pairing (ALP) and attempt to understand the different effects it has on the resulting model and how each of them can affect robustness. They argue that a significant part of ALP*s success stems from regularizing the logits and that alternative logit regularization methods can lead to similar robustness. Finally, they propose an alternative to ALP that explicitly disentangles logit similarity and logit regularization to train models that are claimed to be more robust than state-of-the-art models. Before continuing with my review, I would like to note that the effectiveness of ALP in increasing robustness is still being debated. The robustness of ALP models has not been verified by a third party. The models released by the authors were shortly found to vulnerable to simply running a standard attack for more steps (https://arxiv.org/abs/1807.10272). The ALP authors claim that this is due to releasing a smaller model for a competition but they still have not released another model that is claimed to be robust. Nevertheless, I believe that this situation should not affect how the current paper is reviewed. The idea of regularizing the logits is natural and precisely understanding its effect on the model is a valuable research direction. Overall I like the idea of the paper. I agree that the effect of logit regularization methods on the model is unclear and attempting to dissect these techniques is a very important research direction. On the other hand, this is a very tricky area to perform such experiments. Accurately measuring the robustness of a model is a challenging task. It is therefore hard to understand if an increase in adversarial accuracy is due to a true increase in the robustness of a model or due to the attack being used performing sub-optimally when the setting is modified. Most of the results of the paper revolve around rather fine-grained differences in the adversarial accuracy of <5% (e.g. Figure 1, Table 1). Such small differences might in fact be artifacts of the PGD attack used performing sub-optimally. Moreover, for some of the results, it is clear that they are indeed an artifact of the attacks performing sub-optimally. For instance the left of Figure 2, label smoothing is reported to achieve high accuracy again PGD yet in Section 5.3 it is reported that more steps can reduce the accuracy to <15%. It is thus clear that the claim about label smoothing increasing robustness is misleading. I believe it is essential that the authors validate theirs results, at the very least Table 1, by evaluating the models on different attacks. Given that the models regularizes the logits, evaluating an attack optimizing the logits directly (e.g. the CW attack, https://arxiv.org/abs/1608.04644) is essential. Moreover, given that the gradients of the loss appear to not be informative, an attack based on finite differences is also crucial here (see SPSA attack of https://arxiv.org/abs/1802.05666). Moreover, I find the reasoning behind why should logit regularization techniques increase robustness very limited. The change in logit distributions (Figure 1,2) are rather marginal and do not suggest fundamental differences of the underlying models. I believe that additional experiments and justifications of the results are needed before the paper is considered for acceptance. I thus recommend rejection at this time. [UPDATE after reading the public discussion]: It appears the someone already suggested running SPSA as an attack. I really appreciate the authors evaluating the attack and reporting their findings. Indeed it seems that the reported robustness from logit regularization does not stand up to an SPSA attack. As a result, I think it is fairly clear that logit regularization has a limited effect on the robustness of the model and is only making PGD attacks less effective. I update my score to a clear reject. Minor comments to authors: -- I would suggest remove the sentence close to the end of the intro about ALP being perhaps the most robust defense. As I outlined above, the robustness of ALP has not been convincingly verified yet. Similarly for the text above equation 2. Perhaps most importantly, the last paragraph before Section 4 where you state that "...ALP _works_..." in a very definitive tone. -- Paragraph after (5): the logits of the other classes are not guaranteed to increase. Some of them might decrease in the process of making the loss as large as possible. -- I like equation (10) where you attempt to decompose that effect of similarity and regularization. It would be interesting to understand the interplay between lambda and beta.