Label Noise in Adversarial Training: A Novel Perspective to Study Robust Overfitting

Abstract: We show that label noise exists in adversarial training. Such label noise is due to the mismatch between the true label distribution of adversarial examples and the label inherited from clean examples – the true label distribution is distorted by the adversarial perturbation, but is neglected by the common practice that inherits labels from clean examples. Recognizing label noise sheds insights on the prevalence of robust overfitting in adversarial training, and explains its intriguing dependence on perturbation radius and data quality. Also, our label noise perspective aligns well with our observations of the epoch-wise double descent in adversarial training. Guided by our analyses, we proposed a method to automatically calibrate the label to address the label noise and robust overfitting. Our method achieves consistent performance improvements across various models and datasets without introducing new hyper-parameters or additional tuning.

Reviews and author responses:

 We would like to clarify a little more about the arguments in section 4.2.

First, we wish to note that in adversarial training, since the adversarial perturbation is generated by the probabilistic classifier being trained itself ($f_\theta$), the amount of label noise also changes according to the approximation quality of $f_\theta$ to the true label distribution (Eq. 6). 

Now in our theory, the approximation quality of $f_\theta$ to the true label distribution is constrained by the Lipschitz constant of the probabilistic classifier ($L_\theta$ in Eq. 5, Lemma 4.4), which is the specific condition we referred to in the reply. As in model training the Lipschitz constant generally increases [1] (especially when the learning rate decays), the last checkpoint may indeed not approximate the true label distribution well. However, the early checkpoints can still approximate the true label distribution well with appropriate Lipschitz conditions. In fact, based on our empirical measure of the approximation quality (NLL loss on a validation set), we found that the checkpoint that best approximates the true label distribution usually appears around the first learning rate decay. Therefore at this point in adversarial training, the label noise is the strongest, which coincides with the timing when robust overfitting usually appears in adversarial training. 

We provide additional results to demonstrate this. Following the setting of our previous static adversarial perturbation experiment, we use the model at different checkpoints in adversarial training to generate adversarial examples, and then employ standard training on them. One can find that the checkpoint that best approximates the true label distribution (quantified by NLL on a validation set) can induce strong label noise (quantified by the standard accuracy on the static dataset), while the last checkpoint may not.

| Noise type | SA | NLL (Val) |
|-|-|-|
| Adversarial perturbation (Best, Epoch=82) | 76.22 | 1.28 |
| Adversarial perturbation (Epoch=100) | 82.06 | 1.49 |
| Adversarial perturbation (Epoch=120) | 91.14 | 1.73 |
| Adversarial perturbation (Last, Epoch=160) | 94.32 | 2.86 |


Finally, we agree with the reviewer that “sufficient adversarial training“ at Line 222 may indeed not be accurate. We will revise it to better interpret our theoretical result. We deeply appreciate the reviewer to help clarify and improve this statement. We would be happy to discuss more if you have any other confusion or suggestions.

[1] Spectrally-normalized margin bounds for neural networks. Bartlett et al., 2017.



 Thank you for your detailed reply.

The idea of demonstrating the label noise in AT by comparing it with the performance on manually generated datasets with random flip noise is smart. 

I am still confused by the arguments in section 4.2:

According to my understanding, this section is devoted to quantifying the lower bound of label noise in the case when the adversarial perturbation is generated by a probabilistic classifier.\
You first show that "after sufficient adversarial training, the predictive label distribution of $f_\theta$ can approximate the true label distribution with high probability." (lemma 4.4)\
And the main conclusion of Theorem 4.5 is based on this result (lemma 4.4).

On the other hand, in your reply, you mentioned "Our theory shows that an adversarially trained model **has to satisfy specific conditions** to be able to approximate true label distribution. The model trained to the end **may not match such conditions well**"\
It seems to me that this statement would have weakened the arguments in section 4.2 since if $f_\theta$ does not approximate the true label distribution well in practice, the bound in Theorem 4.5 would also not reflect the amount of label noise in real situation.

BTW, according to the reply the statement  "after sufficient adversarial training, the predictive label distribution of $f_\theta$ can approximate the true label distribution with high probability." (line 222) seems to be kind of overstated. 

 I deeply appreciate the authors' effort in providing the feedback. It's good to see that the effectiveness of the alternative labeling is maintained for different perturbations radii and data qualities. Thus, I would like to increase my rating to 6.

 ## __
**Q. Case study of alternative labeling**


We note that our alternative labeling assigns soft labels to adversarial examples. Therefore it is hard to measure if such soft labels are reasonable without knowing the true label distribution.  


## __
**Q. Extraction of the validation set**


We reserve 10% of the training set as a validation set, which follows the standard experiment settings in [1]. Such a validation set is sufficient to determine the hyperparameters automatically.


## __
**Q. Local Lipschitz assumption of $f$.**


We believe that assuming that the network function to be local Lipschitz is commonly seen in the literature, especially those related to adversarial training [3, 4, 5], as also indicated by the other reviewers. In fact, adversarial training can be viewed as one way to regularize local Lipschitz since it tries to ensure output stability under a worst-case input perturbation.


[1] Robust Overfitting Maybe Mitigated by Properly Learned Smoothening. Chen et al., 2021. 

[2] On Calibration of Modern Neural Networks. Guo et al., 2017.

[3] Evaluating The Robustness Of Neural NetWorks: An Extreme Value Theory Approach. Weng et al., 2018.

[4] Adversarial Robustness Through Local Lipschitzness. Yang et al., 2020.

[5] Do Wider Neural Networks Really Help Adversarial Robustness? Wu et al., 2021.



 Thanks for your valuable comments. Please check our response below.

## __
**Q. Analyses of temperature scaling/interpolation and their connection to the proposed method**


The most important theoretical grounding of our method is that the calibration measure on a validation set (such as NLL loss) can reflect the distribution mismatch between predictive label distribution and the true label distribution, which provides a practical guide to reducing the label noise in adversarial training since it originates from such a distribution mismatch. To improve confidence calibration, temperature scaling and interpolation are probably the most well-known methods [2]. Meanwhile, they are among the simplest and most effective techniques introduced in recent works to mitigate robust overfitting [1]. Therefore, following our theory, we are able to draw the connection between confidence calibration and potential techniques for mitigating robust overfitting, and have a better understanding of their success, and further improve them strategically. Our analysis of the origin of label noise in adversarial can certainly provide opportunities to revisit more existing techniques for mitigating robust overfitting, but we believe these could be interesting future works.


Our observations that temperature scaling and interpolation can improve the predictive label distribution of individual examples thus serve as additional evidence of why they may help with confidence calibration. Since tuning hyperparameters for individual examples leads to a mode collapse, practically it is only feasible to tune a universarial hyperparameter and it should still be able to improve calibration, for instance, for the majority of data examples. We note that such reasoning maybe more relevant to the effectiveness of confidence calibration and be less relevant here. We thank the reviewer for bringing this to our attention and we will reorganize them in the revision. 


## __
**Q. Different perturbations radii and data qualities**


To validate the effectiveness of the alternative labeling, we conducted additional experiments on the perturbation radii and data qualities with CIFAR-10 and pre-activation ResNet as an example. Other settings are consistent with those in Table 1. We show that our alternative labeling can better mitigate robust overfitting for different perturbation radii and data qualities. Here for data quality we follow the setting in our Figure 5 to select high-quality data. 


||$\varepsilon$ (/255)| $T$| $\lambda$ | RA (Best)|RA (Last)| RA (Diff)|SA (Best)| SA (Last)| SA (Diff)|
|-|:-:|:-:|:-:|-|-|-|-|-|-|
| KD-AT |  $8$ | $2$| $0.5$ | $48.76$ | $46.33$ | $ 2.43$ | $82.89$ | $85.49$ | $-2.60$ | 
| KD-AT-Auto | $8$ |$1.47$ | $0.8$ | $49.05$ | $48.80$ | $ 0.25$ | $84.26$ | $84.47$ | $-0.21$ | 
| KD-AT | $12$ |$2$ | $0.5$ | $48.90$ | $47.70$ | $ 1.20$ | $76.73$ | $79.48$ | $-2.75$ | 
| KD-AT-Auto | $12$ | $1.47$| $0.82$ | $49.57$ | $49.29$ | $ 0.28$ | $77.55$ | $77.83$ | $-0.28$ | 
| KD-AT | $16$ | $2$ | $0.5$ | $48.28$ | $47.23$ | $ 1.05$ | $74.46$ | $75.66$ | $-1.20$ | 
| KD-AT-Auto | $16$ |$1.28$ |$0.81$ | $48.76$ | $48.78$ | $-0.02$ | $73.79$ | $74.00$ | $-0.21$ | 


||Max quality rank| $T$| $\lambda$| RA (Best)|RA (Last)| RA (Diff)|SA (Best)| SA (Last)| SA (Diff)|
|-|:-:|:-:|:-:|-|-|-|-|-|-|
| KD-AT | 40k | $2$ | $0.5$ |  $49.52$ | $49.38$ | $ 0.14$ | $82.54$ | $82.47$ | $ 0.07$ | 
| KD-AT-Auto | 40k | $2.49$| $0.9$ | $49.33$ | $49.24$ | $ 0.09$ | $81.63$ | $81.85$ | $-0.22$ | 
| KD-AT | 45k | $2$ | $0.5$ | $48.57$ | $48.06$ | $ 0.51$ | $82.93$ | $84.24$ | $-1.31$ | 
| KD-AT-Auto | 45k | $2.09$ | $0.86$ | $49.17$ | $48.91$ | $ 0.26$ | $83.74$ | $83.73$ | $ 0.01$ | 
| KD-AT | 50k | $2$ | $0.5$ | $48.76$ | $46.33$ | $ 2.43$ | $82.89$ | $85.49$ | $-2.60$ | 
| KD-AT-Auto | 50k | $1.47$ | $0.8$ | $49.05$ | $48.80$ | $ 0.25$ | $84.26$ | $84.47$ | $-0.21$ | 




 Thanks for your valuable comments. Please check our response below.

## __
**Q. Connection between label noise and robust overfitting**


We note that in this paper we focus on the first part of the problem, namely how the label noise originates in the adversarial training. The second part of the problem, namely how the label noise cause overfitting, has been studied in existing works [1, 2]. They’ve rigorously proven that label noise increases the variance of the model and thus makes the double descent more evident.

## __
**Q. Significance of label noise in realistic datasets**


The label noise defined in our paper describes the mismatch between the true label distribution and the assigned label distribution caused by adversarial perturbation. It is not simply flipping the true labels, thus is difficult to quantify directly in realistic datasets. 


However, we note that the static adversarial perturbation setting, namely employing standard training on a fixed set of adversarial examples, can be a playground to demonstrate the significance of such label noise. On the dataset of adversarial examples generated by adversarial perturbation of radius of 8/255 on CIFAR-10, a standard training can achieve test accuracy about 76%, while it can achieve test accuracy about 95% on a clean dataset. To match the same level of accuracy reduction, we have to inject up to 20% label flipping noise in the dataset (see the following table). This demonstrates the label noise caused by adversarial perturbation may be more severe than one’s impression. We will include more discussion on this in the revision. 
| Noise type | SA (Last) |
|-|-|
| No noise | $95.16$ |
| Adversarial perturbation ($\varepsilon=8/255$) | $76.22$ | 
|Label flipping noise ($10\\\%$)|  $88.57$ |
|Label flipping noise ($20\\\%$)| $80.48$ |
|Label flipping noise ($30\\\%$)| $72.47 $|
|Label flipping noise ($40\\\%$)| $60.87$ |


## __
**Q. Approximation of true label distribution by AT**

We note that an adversarially trained model approximates true label distribution doesn’t imply robust overfitting won’t happen. Our theory shows that an adversarially trained model has to satisfy specific conditions to be able to approximate true label distribution. The model trained to the end may not match such conditions well. Based on our empirical measure of the quality of the approximation (NLL loss on a validation set), the model at the last checkpoint in fact approximates the true label distribution poorly compared to early checkpoints. Therefore, the model in adversarial training also overfits in terms of learning true label distribution. Subsequently, the model at the last checkpoint for alternative labelling (KD-AT (Last)) also cannot alleviate robust overfitting well (see the following table using CIFAR-10 as an example), in contrast to the model at the best checkpoint (KD-AT (best)).

| |NLL (Val)|RA (Best)|RA (Last)| RA (Diff) |
|-|-|-|-|-|
|AT| - | $47.35$  | $41.42$ | $5.93 $ |
|KD-AT (Best)| $1.28$ | $48.76$    |  $46.33$ | $2.43 $|
|KD-AT (Last)| $2.86$ | $47.64$   |  $43.71$ | $3.93$ |

In practice, we use our approximation measure to select a model throughout adversarial training for alternative labelling, which often coincides with the best checkpoint in terms of the robust test accuracy. We thank the reviewer for bringing this to our attention and will include this discussion in the revision.


[1] Rethinking Bias-Variance Trade-off for Generalization of Neural Networks. Yang et al., 2020.

[2] Double Trouble in Double Descent: Bias and Variance(s) in the Lazy Regime. d’Ascoli., 2020.


 Thanks for your valuable comments. Please check our response below.

#### **______**
**Q. Assumption of external covering**


We believe this is a natural assumption as it is commonly seen in the analysis of generalization theory [1, 2]. 

#### **______**
**Q. Novelty of our method**


We believe our analyses of label noise in adversarial training naturally suggests the method that modifies the labeling of adversarial examples (adversarial knowledge distillation) to mitigate robust overfitting, and thus provides a principled understanding of such a method. Built upon such a baseline, our method that furthers calibrate the alternative labeling is novel as it is among the first to use calibration measure as an indicator to guide the mitigation of label noise in adversarial training. Starting from here, more techniques can be revisited and developed to address robust overfitting, which might be important future work.


#### **______**
**Q. Y and Y-hat as distributions**


We first wish to clarify that Y (true label) and Y-hat (assigned label) are random variables whose sample space is a set of integers, and the labels of images in a dataset are sampled from the distributions of Y and Y-hat. 

A case where the adversarial perturbation tweaks “dog” images to contain more “cat” semantics can in fact be described as a change of the true label distribution (Distribution of Y) in our theoretical framework. While the assigned labels are unchanged, it implies the existence of label noise in the dataset after adversarial perturbation and is indeed the problem we wish to address. 


#### **______**
**Q. Longer training**


Due to computational constraints, we choose to extend the training to 400 epochs for CIFAR-10, CIFAR-100 and Tiny-ImageNet. As shown by **Figure 9 in the updated draft**, our method (KD-AT-Auto) largely maintains the robust test accuracy for while the baseline (KD-AT) degrades constantly. 

#### **______**
**Q. Factor of data quantity**

We wish to clarify that both data quantity and quality are important in order to achieve better performance. From a bias-variance understanding of model generalization [3], the noise corrupting the dataset (data quality) and the sampling of the training set (data quantity) both contribute to the model variance. Our theory focuses on the former part and shows that the label noise is significant in adversarial training, thus hurting the robust generalization. Other works [4] may focus on the later part and show that the sufficient training data matters. These two are orthogonal as the model cannot generalize well even with infinite data if the distribution to sample labels is biased.

#### **______**
**Q. Iterative training**

We note that iterative training is certainly possible and worth trying. In standard training, this is known as multi-generation distillation [5], and have been shown to improve performance. In adversarial training, this might still be effective but may raise the concern of training efficiency.

[1] Robustness and Generalization. Xu and Mannor, 2010.

[2] Spectrally-normalized margin bounds for neural networks. Bartlett et al., 2017.

[3] Double Trouble in Double Descent: Bias and Variance(s) in the Lazy Regime. d’Ascoli et al., 2020.

[4] Unlabeled Data Improves Adversarial Robustness. Carmon et al., 2019.

[5] Born-Again Neural Networks. Furlanello et al., 2018.


 The paper tackles the problem of adversarial overfitting, that is a critical breakdown of robust accuracy that commonly happens in adversarial training and makes early stopping a necessity for AT. They give an interpretation of robust overfitting from the standpoint of label noise, ie they argue that the adversarial attacks change the true target for the image but AT does not adapt to the change in label. 

The paper first gives a theoretical analysis, in which they first show that for locally Lipschitz and optimal (P(Y|x) = f(x)) classifiers, the adversarially perturbed data distribution has non-negative label noise (if the labels remain unchanged) that depends, among others, on the size of the perturbation radius. A similar theorem is then given for the case of non-optimal classifiers. 

In the practical section, they show various phenomena, for example, that robust overfitting follows a double descent curve and that robust overfitting correlates with radius and data quality. 

Finally, they introduce a new adversarial training formulation, that uses temperature rescaling, interpolation, and the output of an external adversarially trained classifier to relabel the perturbed samples during adversarial training. In the evaluation, they show that this scheme is less likely to overfit and can yield better robust accuracies.  Strengths:

- The problem of robust overfitting is relevant and while the intuitive idea that AT can create a sample-label mismatch is quite obvious, the theoretical and practical connection to robust overfitting is interesting. 

- The paper is well written and clear in its explanations.

- The robustness evaluation uses a strong and reliable attack. 

- The proposed training method improves over AT, Trades and KD-AT in terms of robust accuracy on all 3 datasets. 

Weakenesses: 

- Like every theoretical paper, the authors make certain assumptions. This is fine and I think that the local Lipschitz assumption is quite common and established in the AT literature. However for the others, such as external covering in 4.4, the authors do not comment on whether or not this is a realistic assumption or if it is highly likely to be violated in practice. 

- The proposed training scheme is not very novel and resembles for example the cited [Robust overfitting may be mitigated by properly
learned smoothening] with a heuristic for hyperparameter optimization.  - Section 3 assumes that Y and Y-hat are integers, ie this means that a true label exists. Now for certain images, an adversarial attack might introduce a new class without completely deleting the original one, eg we could have an image with both a dog and a cat. Is it possible to extend your framework such that Y is a distribution to handle those circumstances? 

- Figure 7 shows that the proposed method does not overfit in 150 epochs. Is this also true if we continue training for 1k epochs? 

- In the past, papers like "Unlabeled Data Improves Adversarial Robustness" have shown that quantity matters. In light of your theorem, it seems like data quality is more important. Do you think this behavior is simply not explained by your theory or is there another explanation? 

- In a way, this scheme resembles self-training. Is it possible to iterate the method, ie use the first classifier trained with your new loss to train a second classifier with the first one as "teacher" (to label the adv samples during training)?  The limitations in the appendix address the largest issues. The training of the additional classifier can be expensive and the entire scheme can only work if one can actually find a model that can properly classify the adv samples for the new classifier during training. 

 This paper demonstrates that adversarial training will introduce label noise through both theoretical and empirical investigations. This noise helps to interpret the robust overfitting phenomenon. The paper further proposes to use a rectified model probability in adversarial training to mitigate the robust overfitting. Experiments on CIFAR10/100 and Tiny-ImageNet are conducted to illustrate the effectiveness of the proposed method. Strengths:
1. The observation that adversarial training introduces label noise is insightful and novel, and sheds light on the understanding of robust overfitting.
2. The paper provides a clear and in-depth theoretical investigation of the label noise caused by adversarial perturbations. The example and relevant experiments are helpful for readers to grasp the key idea of this novel label noise.
3. Based on the label noise understanding, the paper further proposes to use a rectified model probability to mitigate the robust overfitting. Comparisons with regular AT and KD-AT illustrate the effectiveness of the proposed method.

Weaknesses:
1. Though the experiments successfully demonstrate the connection between the label noise and robust overfitting, it would be overstated to say "robust overfitting can be adequately explained by such label noise" (line 76). A rigorous theoretical analysis is needed to establish that strong conclusion.
2. while the example shows that the label noise introduced by adversarial perturbation can be significant in synthetic datasets, it is not clear how severe would the label noise be in realistic cases. 

1. Can the authors estimate the significance of such AT-caused label noise in realistic cases, e.g., considering the classification task on CIFAR100?
2. The arguments in section 4.2 seem artificial.
if an AT-trained model can approximate the true label distribution, then the robust overfitting won't be a problem. That is it doesn't matter whether the AT will distort the true label distribution. 
3. A possible typo in table 1, in the last column for CIFAR100, -0.25 is in bold instead of -0.09. The paper addresses the limitations in the supplementary material. As far as I am concerned, there is an additional limitation: 
- the paper does not discuss how severe would the label noise be in realistic applications.

 The authors show that label noise exists implicitly in adversarial training due to the mismatch between the true label distribution and the assigned label distribution of adversarial examples.  Based on a label noise perspective, the authors formalize robust overfitting and show that it is the early part of an epoch-wise double descent in adversarial training.  Also, the authors propose an alternative labeling of adversarial examples by rectifying model probability, which uses temperature scaling and interpolation together. The authors nicely presents that adversarial perturbation implicitly generates label noise.  This phenomenon is explained intuitively and proven formally.  The observation is correct in that there is a mismatch between the true label distribution and the assigned label distribution of adversarial examples.  To solve this issue, the authors propose an alternate labeling scheme based on temperature scaling and interpolation.  As a result, the proposed training method achieves higher accuracy than conventional adversarial training methods.  Overall, the observation is interesting, and the theoretical analysis is well presented.

At the same time, I have the following concerns.

The authors prove that temperature scaling and interpolation can be effective for reducing the distribution mismatch.  However, it is not clear why these two methods are specifically selected.  Are there any other techniques applicable for the same purpose?  To me, introducing these two techniques is somewhat abrupt.

There is a gap between the theory and the actual method.  In Theorem 5.1 and Theorem 5.2, there exists satisfying $T$ or $\lambda$ for each example.  However, a single value is used for the hyper-parameters in the actual method (Equation (9)).  Thus, the connection between the theory and the method is not very rigorous.  In Section E. the optimal hyper-parameter values for each example are not highly concentrated, as opposed to the authors' claim.

Different perturbations and dataset qualities need to be considered for validating the effectiveness of the alternate labeling.  Currently, one a single perturbation radius is used.

It would be better to show how many adversarial examples are assigned labels different from those of the original examples.  The authors can discuss whether the alternating labeling schemes chooses a reasonable label when a different label is generated.  Such case study will be very interesting to the reader.

It is not clear how the validation set is extracted for determining the optimal parameter values in Table 1.  What if a sufficiently large validation set does not exist?

p. 2 is not shown using Acrobat on Windows.  (Okay on iPad.)

=== Post-rebuttal update ===

I have increased my rating from 5 to 6, mainly due to the authors' additional experiments. Why do you assume that $f$ is a L-Lipschitz function?  Is this assumption reasonable?  What if this assumption does not hold? It would be better to discuss the possibility that the alternative labeling even increases the risk of robust overfitting.

Meta-review: This paper introduces a new insight on the relationships between noisy labels and adversarial training. 

All reviewers agree the contriubtions of this paper in terms of interesting observation, novel perspective,s theoretical-practical anlaysis, and promising experimental results and gave all acceptance scores.

AC also agrees with reviewers' opinions, and thus recommends accepting this paper. 
Please more clarify the limitations that the reviewers referred to in the final version.




Decision: Accept