On Intriguing Layer-Wise Properties of Robust Overfitting in Adversarial Training
Authors: Duke Nguyen
Chaojian Yu
Vinoth Nandakumar
Young Choon Lee
Tongliang Liu
Venue: Accepted by TMLR ()

Abstract: Adversarial training has proven to be one of the most effective methods to defend against adversarial attacks. Nevertheless, robust overfitting is a common obstacle in adversarial training of deep networks. There is a common belief that the features learned by different network layers have different properties, however, existing works generally investigate robust overfitting by considering a DNN as a single unit and hence the impact of different network layers on robust overfitting remains unclear. In this work, we divide a DNN into a series of layers and investigate the effect of different network layers on robust overfitting. We find that different layers exhibit distinct properties towards robust overfitting, and in particular, robust overfitting is mostly related to the optimization of latter parts of the network. Based upon the observed effect, we propose a \emph{robust adversarial training} (RAT) prototype: in a minibatch, we optimize the front parts of the network as usual, and adopt additional measures to regularize the optimization of the latter parts. Based on the prototype, we designed two realizations of RAT, and extensive experiments demonstrate that RAT can eliminate robust overfitting and boost adversarial robustness over the standard adversarial training.

Published in Transactions on Machine Learning Research (10/2024)
On Intriguing Layer-Wise Properties of Robust Overfitting in Adversarial Training
Duke Nguyen Sydney AT Center
dngu31222 edw.eeeeeedd....
The University of Sydney
Chaojian Yu
Sumiisyyeye, edu.an
Sydney AI Center
The University of Sydney
Vinoth Nandakumar
vinoth. 90 90w.00000mmllllmm com
Sydney Al Center
The University of Sydney
Young Choon Lee
young. www@mq..eemmmmmmmmmmmmmm eda.oo
School of Computing Macquaric Universily
Tongliang Liu
tongliang Iiu Wwsydnnyyyy....mmmm edu.
Sydney AI Center
The University of Sydney
Reviewed OpenReview: https: o/penrvve/eenrvveeeee net/ forum? id=phk5CcKTcc
Abstract
Adversarial training has proven to be one of the most effective methods to defend against adversarial attacks. Nevertheless, robust overfitting is a common obstacle adversarial training of deep networks. There common beliel that the features learned by different network layers have different properties, however, existing works generally investigate robust overfitting by considering DNN single unit and hence the impact of different network layers on robust overfitting remains unclear. In this work, we divide DNN inlo series of layers and investigate the effect of different network layers on robust overfitting We find that different layers exhibit distinct properties towards robust overfitting, and in particular, robust overfitting is mostly related to the optimization of latter parts of the network. Based upon the observed effect, we propose robust. adversarial training (RAT) prototype: in minibatch, we optimize the front parts of the network as usual, and adopt additional measures to regularize the optimization of the latter parts. Based the protolype, designed two realizations of RAT, and extensive experiments demonstrate that RAT can climinate robust overfitting and boost adversarial robustnoss over the standard adversarial training.
1 Introduction
Deep neural networks (DNNs) have been widely applied in multiple fields, such as computer vision HHe
et al. 2016) and natural language processing (Devlin et al., 2018). Despite its achieved success, recent studies show that DNNs are vulnerable to adversarial examples. Well-constructed perturbations on the input images that are imperceptible human's eyes can make DNNs lead to a completely different prediction
*Corresponding Author

---

Published in Transactions on Machine Learning Research (10/2024)
(Szegedy et al. 2013 The security concern due to this weakness of DNNs has led to various works in the study of improving DNNs robustness against adversarial examples. Across existing defense techniques thus far, Adversarial Training (AT) (Goodfellow et al. 2014 Madry et al. 2017), which optimizes DNNs with adversarially perturbed data instead of natural data, is the most effective approach (Athalye et al.; 2018). However, it has been shown that networks trained by AT technique do not. generalize well (Rice et al, 2020). After a certain point AT, immediately after the first learning rate decay, the robust test accuracy continues decrease with further training. Typical regularization practices to mitigate overfitting such 11 & 12 regularization (weight decay), data augmentation, etc. are reported to be as inefficient in adversarial robustness compared to simple early stopping (Rice et al. 2020).
Many studies have attempted to improve the robust generalization gap in AT, and most have generally investigated robust overfitting by considering DNNs as whole. However, DNNs trained on natural images exhibit a common phenomenon: features obtained the first layers appear to be general and applicable widespread, while features computed by the last layers are dependent on a particular dataset and task Yosinski et al. 2014). Such behavior of DNNs sparks question: Do different. layers contribute differently to robust overfitting? Intuitively, robust overfitting acts as unexpected optimization state in adversarial training, and its occurrence may be closely related to the entire network. Nevertheless, the unique effect of different network layers on robust overfitting is still unclear. Without a detailed understanding of the layer-wise mechanism of robust overfitting, it is difficult to completely demystify the exact underlying cause of the robust overfitting phenomenon.
In this paper, we provide the first layer-wise diagnosis of robust overfitting. Specifically, instead of considering the network as whole, treat the network composition of layers and systematically investigate the impact of robust overfitting phenomenon different layers. To do this, we first fix the parameters for the selected layers, leaving them unoptimized during AT, and then normally optimize other layer parameters. We discovered that robust overfitting is always mitigated in the case where the latter layers are left unoptimized, and applying the same effect to other layers is futile for robust overfitting, suggesting a strong connection between the oplimization of the latter layers and the overfiuting phenomenon.
Based upon the observed effect, we propose robust adversarial training (RAT) prototype to relieve the issue of robust overfitting. Specifically, RAT works in cach mini-batch: it optimizes the front layers as usual, and for the latter layers, it implements additional measures on these parameters to regularize their optimization. It is a general adversarial training prototype, where the front and latter network layers can be separated by some simple test experiments, and the implementation of additional measures to regularize network layer optimization can be versatile. For instance, we designed two representative methods for the realizations of RAT: RATLR and wAT They adopt different strategies to hinder weight update, e.g. enlarging the learning rate and weight perturbation, respectively. Extensive experiments show that the proposed RAT prototype effectively eliminates robust overfitting. The contributions of this work are summarized as follows:
We provide the first diagnosis of robust overfitting on different network layers, and find that there strong connection between the optimization of the latter layers and the robust overfitting phe- nomenon.
Based on the observed propertics of robust overfitting, we propose the RAT prototype, which adopts additional measures to regularize the optimization of the latter layers and is tailored to prevent robust overfitting.
We design two different realizations of RAT, with extensive experiments on a number of standard benchmarks, verifying its effectiveness.
2
Related Work
2.1
Adversarial Training
Since the discovery of adversarial examples (Szegedy et al. 2013 Wen et al., 2020b there have been many defensive methods attempted to improve the DNN's robustness against such ad adversaries, such adversarial
2

---

Published in Transactions on Machine Learning Research (10/2024)
training (Madry et al. 2017). defense distillation Papernot al. 2016], imput denoising (Liao et al. 2018
regularization (Tramer et al. 2018: Wen et al. 2020a). So far, adversarial training (Madry et al. 2017))
has proven to be the most effective method. Adversarial training comprises two optimization problems: the inner maximization and outer minimization. The first one constructs adversarial examples by maximizing the loss and the second updates the weight by minimizing the loss on adversarial data:
EAT (w) mmnn) Σ
max (1)) a(
where for the DNN classifier with weight w, and E(.) is the loss function. d(-,-) -) specify the distance bet ween original imput data xy and adversarial data zy, which usually an b n--or ball such as the la and lo- -orm balls and is the maximum perturbation allowed.
A different type of AT variation that is commonly used is referred to TRADES Zhang et al. 2019 which involves optimizing surrogate loss that is a tradeoff between the natural accuracy and adversarial robustness:
(TRADES (w) min Σ (CE(Su(()))))) +B- max (Z((f(())))] d(z,,())
(2)
The surrogate loss consists of two parts: cross-entropy (CE) loss, which encourages the network to maxi- mize natural accuracy, and Kullback-Leibler (KT.) divergence, which encourages the improvement of robust accuracy. The hyperparameter 8 is used to control the tradeoff between natural accuracy and adversarial robustness.
Another line of work involves utilizing semi-supervised learning (SSL.) technique. Methods based on SSL Carmon al. 2019 Zhai et al., 2019: Najafi et al.) 2019 Uesato al. 2019) use additional unlabeled data to improve the nosustnness of the trained model. In these methods, a natural model is first trained labeled data to generate pseudo-labels for the unlabeled data. Then, robust model is trained using an adversarial loss function ((w) on both labeled and unlabeled data:
(SL) = glabeled + A gunlabeled(
(3)
where Ả control the weight on the unlabeled data.
2.2 Robust generalization
An interesting characteristic of deep neutral networks (DNNs) their ability to generalize well practice (Belkin al.] 2019). For the standard training setting, it is observed that test loss continues decrease for long periods of training (Nakkiran et al. 2020), thus the common practice is to train DNNs for long possible. However, in the case of adversarial training, further training past a certain point leads to significant decrease in the robust training loss of the classifier, while increasing the robust test loss. Figure [ depicts this phenomenon for adversarial training on CIFAR-10, where the robust test accuracy initially increases but then drops after the first learning rate decay. This phenomenon is called "robust overfitting", which has shown strong resistance to standard regularization techniques such as 4, le regularization and data augmentation methods, and can be observed on various datasots, including SVHN, CIFAR-100, and ImageNet (Rice et al. 2020
Schmidt et al. (2018) theorizes that robust generalization have large sample complexity, which requires substantially larger datasct. Many subsequent works have empirically validated such claim, such as AT with semi-supervised learning (Carmon el al.) 2019 Zhai et al.) 2019; Najafi et al.] 2019 Uesato el al. 2019), robust local feature (Song et al., 2020) and data interpolation (Lee et al. 2020 Chen et al. 2021). Chen al.] (2020) proposes to combine smoothing the logits via self-training and smoothing the weight via stochastic weight averaging mitigate robust overfitting. Ww et al. (2020) emphasizes the connection of weight loss landscape and robust generalization gap, and suggests injecting the adversarial perturbations
3

---

Published in Transactions on Machine Learning Research (10/2024)
0.9
Accuracy
0.3
Train standard
Train robust
0.2
0.1
Test standard
Test robust
50
100
150
200
Epoch
The learning curves of adversarial training using PreAct ResNet-18 on the CIFAR-10 dataset. The depicted curves reveal "robust overfitting", wherein the adversarially trained model bricfly achieves 52.01% test robust accuracy shortly after the first learning rate decay. Surprisingly, at this point, the adversarially trained model is actually more robust than it is at the end of training, where it only attains 43.95% test robust accuracy against a 20-step PGD adversary with an loo radius of e 8/255. The learning rate is decayed at 100 and 150 epochs.
Figure 1:
into both inputs and weights during AT to regularize the flatness weight loss landscape. The intriguing property of robust overfitting has motivated great amount of study and investigation (Wang al. 2019
Zhang et al. 2020; Liu et al. 2021 Rebuffi et al.) 2021 Dong et al. 2022b Yu et al. 2022 Dong et al. 2022a Li & Spratling 2023 Wang et al. 2023). but current works typically approach the phenomenon considering DNN as whole. In contrast, our work treats DNN as series of layers and reveals a strong connection between robust overfitting and the optimization of the latter layers, providing novel perspective into better understanding the phenomenon.
3 Intriguing Properties of Robust Overfitting
this section, we first investigate the layer-wise properties of robust overfitting by fixing model parameters in AT (Section 3.1). Based on our observations, we further propose a robust adversarial training (RAT) prototype to mitigate robust overfitting (Section 3.2 Finally, design two different. realizations for RAT to verify the effectiveness of the proposed method (Section 3.3
3.1 Layer-wise Analysis of Robust Overfitting
Current works usually study the robust overfitting phenomenon considering the network single unit [Rice et al. 2020 Wu et al. 2020 Yu et al. 2022: Dong et al. 2022a Li & Spratling) 2023 Wang et al, 2023). However, features computed by different layers exhibit different properties, such as first-layer features are general and last-layer features are specific (Yosinski et al.) 2014). We hypothesize that different network layers have different effects on robust overfitting. To empirically verify the above hypothesis, we deliberately fix the parameters of the selected network layers, leaving them unoptimized during AT and observe the behavior of robust overfitting accordingly. Specifically, we considered PreAct ResNet-18 architecture as a composition of 4 main layers, corresponding to 4 Residual blocks. We then train multiple PreAct ResNet-18 networks CIFAR-10 for 200 epochs using AT, each time selecting set of network layers to have their parameter fixed.
The robust lest performance in Figure a(a) shows a consistent pattern. Robust overfitting is miligated whenever we fix the parameters for layer 4 during AT, while any settings that do not fix the parameters for layer result in more severe gap between the best accuracy and the accuracy at the last epoch. For fii.x-pprram-1.44 AT AT-ii-parram-12,1 and AT-fx--arra--
example, for settings such as AT-fix-param-14, [3,4), robust overfitting is significantly reduced. On the other hand, for settings such AT-fix-pan (axxxpaaaa-m112)
4

---

Published in Transactions on Machine Learning Research (10/2024)
0.5
0.5
0.5
0.5
Test Rebustt Arcorey
0.4
0.4
033
0.3
0.3
condars
0.2
AT. parsm 12]
0.2
BC pamam. I3.
0.2
AT param 12.3
Ar.landard
0.2
AIJ.nsseemn 2.30
17.1% param [1. 2.40
A1. (L
Riganmnt 31
A_JX 3.4)
AT Dr pammntt
41
AT pwaannn 41
A1.1% w.t._rane_jp J2: 1,4)
0.1g
100
150
0.1g
150
0.1g
100
0.10
150
Epochs
(a) Robust test performance
0.8
2:4.
0.7
0.7
0.7
0.7
P.B.gamm. (1.4.))
Tran Rotsot Accurnry
AT Ar.ecpanm_D 2.1.
0.6
0.6
0.6
0.5
0.4
0.4
pamm
A7. cantard
0.3
1T parsm [3)
0.3
AT Bc Rc.amm.. [1.
D1.aomm 21
0.3
0.3
AT Ri.annntt [t.4 4]
0.2
50
100
150
200
0.2g
50
100
150
200
0.2g
50
300
150
200
0.2g
50
100
150
200
Epochs
(b) Robust train performance
Figure 2: The robust train/test performance of adversarial training with different sets of network layers fixed. AT-fix-param[1,2] corresponds to fixing the parameters of layers 1 & 2 during AT.
ATT-fx--arammi and AT-fix-param--22 when we fix the parameters of various set of layers but allow for the optimization of layer 4, robust overfitting still widely exists. For extreme case like AT-fi--param- [1,2,3], where we fix the first three front layers and only allow for the optimization of that last layer 4, the gap between the best accuracy and the last accuracy is still obvious. This clearly indicates that the optimization of the latter layers present a strong correlation to the robust overfitting phenomenon. Note that this relationship can be observed across variety of datasets, network architectures, AT methods, and threat models (shown in Appendix indicating that it is a general property in adversarial training.
In many of these settings, robust overfitting mitigated at the cost of robust test accuracy. For example in AT-fix-param-13, if we leave both layer & 4 unoptimized, robust overfitting will practically disappear, but the peak performance much worse compared to standard AT. When carefully examining the training performance in these settings shown in Figure bb, we generally observe that the network capacity to fit adversarial data is strong when we fix the parameters for the front layers, but it gradually gets weaker as we try to fix the latter layers. For instance, ATT-fx--aram-[1] has the highest train robust accuracy, then comes AT-fix-parmm|2 (2], AT-fix-parmm{3] and XT-fix-param{4] accuracy than AT- fffx-xaramm22.m [2,3,4]. This suggests fixing the latter layers' parameters can regularize the network stronger compared to fixing the front layers's parameters. This stronger regularization may be due to the latter layers containing more parameters than the front layers, or because fixing the parameters for the latter layers leads to more severe underfitting. However, these factors do not adequately explain the layer-wise property of robust overfitting. For example, the front layers also contain certain number of parameters. Yet, when we fix the parameters of the front layers, the degree of robust overfitting remains almost unchanged, as shown in Figure aa... On the other hand, we observe that fixing paramelers in front layers also leads to some degree underfitting, as shown in Figure ( However, these measures have almost no effect on the robust overfitting phenomenon. Nevertheless, introduce methods that specifically regularize the optimization of the latter layers, so as to mitigate robust overfitting without tradeoffs in robustness. We will compare the impact on robust overfitting when applied
(TT-fx-paramm[1,22 has higher training
the subsequent sections, we will
5

---

Published in Transactions on Machine Learning Research (10/2024)
such methods the front layers vs the latter layers, further highlighting the importance of the latter layers in relation to robust overfitting.
3.2 A Prototype of RAT
the optimization of AT in the latter layers is highly correlated to the existence of robust overfitting. To address this, we propose to train the network on adversarial data with some restrictions put onto the optimization of the latter layers, dubbed as Robust Adcersarial Training (RAT). RAT adopts additional measures to regularize the optimization of the latter layers and mitigate robust overfitting.
As witnesssd in Section 3.1
The RAT prototype given Algorithm It runs as follows. We start with a base adversarial training algorithm A. In Line 1-3, The inner maximization pass aims generate adversarial examples via maximizing the loss, and then the outer minimization pass updates the weight to minimize the loss adversarial data. Line 4 initiates a loop through all parts of the weight w from the front layers to the latter layers. Line 5-9 then manipulate different parts of the weight based on its layer conditions. If the parts of the weight belong to the front. layers (Ctrant), their gradients will be kept intact. Otherwise, a weight update scheme G is put onto the parts of the weight corresponding to the latter layers (Clatter). Finally, the optimizer D updates the model fw in Line 11.
Note that RAT is general prototype where layer conditions Erront Clatter and weight adjustment strategy 6 can be versatile. Based on the setting in Section 3.1, the ResNet architecture is treated as composition of 4 main layers, corresponding to 4 residual blocks. In our subsequent experiments, except for the case of including all layers, when we count from the first layer backward, we regard them as front layers. When we count from the fourth layer forward, we regard them as latter layers. For example, layer 1 & 2 is Efrent and layer & 4 is Clatter- G can also represent various strategies that serves to regularize the optimization of the latter layers. In the section below, we will propose two different strategies 6 in the implementations of RAT to demonstrate RAT's effectiveness.
Algorithm 1 HLTT-potooyyp (in mini-batch).
Require: base adversarial training algorithm A, optimizer D, network Swo model parameter w W2,..., ..., w,), training data {(..,))) mini-batch B, front and latter layer conditions Etrout and Clatter for fw, gradient adjustment strategy G.
{ui, W2,
1: Sample a mini-batch B = ([,,,, from D
2: B' = A.inner --.inner_mmaiineeeee maximizatiinn(f...b
3: V. 5 A.outer minimization
for j= do
if (W;) then
Vw, + Vw, 7: else if (w;) then
8: Vw, + G(fo, B', Vwy)
# adjust gradient
9: end
10: end for
11: (.stcp(Va.)
3.3
Two Realizations of RAT
In this section, we will propose two different methods to put certain restrictions on the optimization of selected parts of the network, and then investigate the robust overfitting behavior upon applying such method to the front layers vs the latter layers. These methods showcase a clear relation between the optimization of the latter layers and robust generalization gap.
RAT through enlarging learning rate. In standard AT, the sudden increases in robust test performance appears to be closely related to the drops the scheduled learning rate decay. We hypothesize that training AT without learning rate decays is sub-optimal, which can regularize the learning process of adversarial training. Comparison of the train/test performance between standard AT and AT without learning rate
6

---

Published in Transactions on Machine Learning Research (10/2024)
decay (1T--fix-I--11,,3344) are shown in Figure 3 b). Training performance of standard AT accelerates quickly right after the first learning rate drop, expanding the generalization gap with further training, whereas for AT without learning rate decay, training performance increases slowly and maintain a stable generalization gap. This suggests that AT optimized without learning rate decay has less capacity to fit adversarial data, and thus provides the regularization needed to relieve robust overfitting. As our previous analysis suggests that the optimization of the latter layers more important in mitigating robust overfitting, we propose using& a fixed learning rate of 0.1 for optimizing the latter parts of the network while applying the piecewise decay learning rate for the former parts to close the robust generalization gap. We refer to this approach as realization of RAT through enlarging learning rate, namely RATLR- Note that there is no difference between enlarging learning rate and enlarging gradients, since the amplification coeffcient has the same effects in increasing gradients and increasing the learning rate inside D. Therefore, compared to standard AT, RATLR essentially enlarge the weight gradient Vwy along the latter layers by 10 at the first learning rate decay and 100 at the second decay:
Vey = Vwyy wy:
(4)
where 1) is the amplification coefficient.
To demonstrate the effectiveness of RATLR, we train multiple PreAct ResNet-18 networks on CIFAR-10
for 200 epochs using AT, each time selecting a set of network layers to have their learning rate fixed to 0.1 while maintaining the piece-wise learning rate schedule for other layers. Figure B(a) validate our proposition. Robust overfitting is relieved for all settings that target layers that include layer 4 (AT-fix-Ir-44) [4], AT-fix-Ir- [14], AT-fix-lr- [2,4], etc.) while any settings that fix the learning rate of layers that exclude layer 4 do not roduce robust overfitting. Furthermore, all scttings that fix the learning rate for both layer 3 & 4, inchuding AT-fix-lr-(3,4), AT- AT-fix-Ir-22,3.44 & AT-fix-Ir[11,,3,44] completely eliminate robust overfitting, The observations indicate that regularizing the optimization of the latter layers by optimizing those layers without learning rate decays can prevent robust overfitting from occurring. An important observation is that RATLR (AT-fix-Ir-(3.4) can both overcome robust overfitting and achieve better robust test performance compared to the network using a fixed learning rate for all layers (AT,,,,-I--11,233,41) Examining the training performance between these two settings in Figure Ee., we find that RATLR exhibits a rapid rise in both robust and standard training performance immediately after the first learning rate decay similar to standard AT. The training performance of RATLR is able to benefit from the learning rate decay occurring at layer & 2, making a notable improvement compared to AT-fix-Ir-11,2,3, learning rate decays, we specifically put some restrictions on the optimization of only the latter parts of the network heavily responsible for robust overfitting, which can relieve robust overfitting without sacrificing too much performance. The experiment results provide another indication that the latter layers have strngr connections to robust overfitting than the front layers do, and regularizing the optimization of the latter layers from the perspective of learning rate can effectively mitigate robust. overfitting.
By training layers 3 & 4 without
RAT through adversarial weight pertubation. We continue to study the impact of different network layers to robust overfitting phenomenon from the perspective of adversarial weight perturbation (AWP). Wu et al. 2020) proposes AWP as a method to explicitly flatien weight loss landscape, by introdug adversarial perturbations into both inputs and weights during AT:
min max 30 VEV VVV
Σ max ( (l
(5)
where V is a feasible region for the perturbation v, and v is the adversarial weight perturbation generated by maximizing the classification loss:
v = V Σ ln
(6)
where Eg is the adversarial loss of x/. Then the norm of weight perturbation vy is restricted by its relative size to the norm of model weight wy:
||ل||| = =>>1=-/ >|wwy|
(7)

---

Published in Transactions on Machine Learning Research (10/2024)
0.550
0.550
0.530
0.525
0.525
0.525
0.525
0.500
0.500
0.500
0.500
Test Robost Accuncy
0.475
0.475
0.475
0.475
0.425
0.425
canders
0.400
AUJ.U
0.400
0.400
Al. alaaadd
0.400
ALTRUULL 4.30
6T.5% (1.2k 21
A7 de. I 12.3
AT PO DL. 3.40
0.375
ALT (1
0.375
ALS 31
0.375
0.375
ALTRUUU 3.4)
AT. FT. 1.43 3.4)
0.250
100
0.350
0.250
100
0.350
50
Epochs
(a) Robust test performance of all settings
bar, AT Jtadars maL A...A
0.225
0..8
box, At..
0.500
Nattuall Accureery
0.475
0.7
0.25
mir AT. coanders
0.400
1nah AT. B. D1
toi S.ndderd
45
M. 2.3.4
mair AT.5 3.3.4)
0.375
2an 2.2.
0.4
Iran BK
100
Epocts
Epochs
(b) Standard AT' vs AT' using fixed learning rate
(c) AT' using fixed learning rate for all layers vs the latter layers
Figure 3: The train/test performance of adversarial training using fixed learning rate for different sets of network layers. AT-fix-Ir(12) corresponds to using fixed learning rate for layers 1 & 2 during AT.
where 7 is the constraint on weight perturbation size.
As AWP keeps injecting the worst-case perturbations on weight during training, could also be viewed as a means to regularize the optimization of AT. Tn fact, the training of AWP exhibits a negative robust generalization gap, where robust training accuracy in short of robust testing accuracy by large margin, shown in Figure 4 cc.. This indicates AWP put significant restrictions to the optimization of AT, introducing huge trade-offs to training performance. As our previous analysis suggests strong correlation bet wetween robust overfitting and the optimization of the latter layers, we argue that the capacity to mitigate robust overfitting from AWP is mostly thanks to the perturbations occurring at latter layers' weight. As such, we propose to specifically apply AWP to the latter half of the network, and refer to this method as RATwP. In essence, RAT compute the adversarial weight perturbation vy under the layer condition Clatter (w;), so that only the parts of the weight along the latter half of the network are perturbed:
max
Σ
max ((fwoo(z)) d(su)()se
(8)
W Mo My
w.. w= 10.....
DjEV
V wy Σ C..
(9)
|By|| - >
(10)
To prove the effectiveness of RATwP, we train multiple PreAct ResNet-18 networks on CIFAR-10 for 200 epochs using AT, each time selecting a set of network layers to have their weight locally perturbed using AWP. As seen from Figure 4aa), there are only 3 settings that can overcome robust overfitting, namely (T-4wp-[3,4], AT-awp-(1.3 [1,44] and AT-3wp-[2.3,4). These scttings share one key similarity: both layer 3 & have their weight adversarially perturbed during AT. Simply applying AWP to any set of layers that exclude layers & 4 is not sufficient to mitigate robust. overfitting. This shows that AWP is effective in solving robust
8

---

Published in Transactions on Machine Learning Research (10/2024)
6.575
0.575
6,550
0.550
0,500
0.525
0.525
0.525
0.525
Tet Robost Accurcy
0.475
0.475
0.475
0.475
0.450
0.450
0.450
0.450
0.425
0.425
0.425
ctancard
0.400
sep [3]
0.400
sup (1,
0.400
AT swo 12, 3]
0.400
awp D. 2.24
6.375
0,375
0.375
A1 [i_.e_ J2,
0.375
ang.
pup
awo 13,
awp 3.4
0.250
100
150
0.350
0.350
100
150
0.350
50
Epochs
(a) Robust test performance of all settings
0.9
mii WWwLAT,WAB.11 (1.2 2
bax, AT ano (2.2.
min 2Ap.1
0.9
0.8
rebust
0.5
10bust
0.0
tox, At_.no__3344
0.8
0.6
Rohurt Accurncy
0.5
6.7
0.7
0.4
0.4
0.5
0.5
0.4
0.4
Guan Al, wtg.....
tett sup 11, 2,31
0.3
0.3
tran all [l__wp__
Bett aup 13
100
0.2
Epocts
Epochs
(b) AWP applied to front layers vs latter layers
(c) Robust generalization gap
Figure 4: The train/tes performance of adversarial training when applying AWP for different sets of network layers. [1122 means only layer 1 & 2 have their weight perturbed using AWP.
overfitting only when applied to both layer 3 and layer 4. Even when AWP is applied to the first 3 former layers out of 4 layers (AT-awp-[1,2,33) robust overfitting still widely exists. In another word, it is essential for the adversarial weight perturbations to occur at the latter part of the network in order to mitigate robust overfitting. To examine this phenomenon in detail, we compare the training performance of AWP applied to front layers (represented by (1,2,3) vs AWP applied to latter layers (represented by AT-awp- (3,4]), shown in Figure 4(b). AWP applied in the front layers have a much better training performance than AWP applied in the latter layers. Furthermore, AWP applied to front layers reveals a positive robust generalization gap (training accuracy >> testing accuracy) shortly after the first drop in learning rate, which continues to widen with further training. Conversely, AWP applied in the latter layers exhibits negative robust generalization gap throughout most of the training, only converging to 0 after the second drop in learning rate. These differences demonstrate that worst-case perturbations, when injected into the latter layers' weights, have a more powerful impact in regularizing the optimization of AT. Consistent with our previous findings, AWP applied to the latter layers can be considered as an approach to regularize the optimization of AT in those layers, which successfully mitigates robust overfitting. This finding supports our analysis thus far, further demonstrating that regularizing the optimization of the latter layers is key to improving the robust generalization.
4 Experiment
In this section, we conduct extensive experiments to verify the effectiveness of RATLR and RAT Details of the experiment settings and performance evaluation are introduced below.
4.1 Experimental Setup
We conduct extensive experiments on two realizations of RAT under two threat models (Lxo and L2) across three benchmark datasets:
9

---

Published in Transactions on Machine Learning Research (10/2024)
Table 1: Test robustness (%) on CIFARIO. We omit the standard deviations of 5 runs as they are very small (< 0.6%).
PGD-20
AA
Network
Threat Model
Method
Best
Last
Diff
Best
Last
Diff
52.31
44.45
7.86
47.95
42.05
5.90
Loo
RATLR
51.57
49.07
2.50
46.89
45.35
1.54
RATwp
54.85
53.98
0.87
49.19
48.24
0.95
PreAct ResNet-18
AT
69.27
65.86
3.41
67.65
64.64
3.01
L2
RATLE
68.97
68.21
0.76
64.26
63.44
0.82
RATwP
70.77
69.49
1.28
68.29
67.11
1.18
AT 55.57
47.37
8.20
52.13
46.09
6.04
Loo
RATLR
55.50
47.32
8.18
52.05
45.89
6.16
RATwP
58.92
58.23
0.69
54.46
53.98
0.48
Wide ResNet-33-10
AT
70.57
68.99
1.58
69.44
66.92
2.52
L2
RATER
71.91
68.94
2.96
70.53
67.90
2.63
RATwP
71.31
69.19
2.12
70.12
67.35
2.77
Table 2: Test robustness (%) on CIFARI00. We omit the standard deviations of 5 runs as they are very
small (..%).
PGD-20
AA
Network
Threat Model
Method
Best
Last
Diff
Best
Last
Diff
AT
28.07
21.24
6.83
23.61
18.41
5.20
Loo
RATLR
26.57
26.18
0.39
21.77
21.22
0.55
RATwP
30.91
30.42
0.49
25.52
24.57
1.05
PreAct ResNet-18
AT
41.38
35.34
6.04
37.94
33.58
4.36
La
RATLR
38.31
37.76
0.55
35.16
34.49
0.77
RATwP
45.23
44.93
0.3
41.32
39.47
1.85
AT 30.74
24.89
5.85
26.98
23.07
3.91
Too
RATLR
30.57
23.53
7.04
26.72
22.53
1.19
RATwr
30.81
25.46
5.35
27.11
23.56
3.55
Wide Nes-44-34-10
AT
44.05
41.22
2.83
41.39
39.34
2.05
L
RATLR
44.43
40.42
4.01
41.47
39.42
2.05
RATwP
46.12
44.64
1.48
41.94
40.38
1.56
CIFAR-10 (Krizhevsky et al. 2009 The CIFAR-10 datasct (Canadian Institute for Advanced Research, 10 classes) is a subset. of the Tiny Images dataset and consists of 60000 32x32 color images. The images are labelled with one of 10 mutually exclusive classes: airplane, automobile (but not truck or pickup truck), bird, cat, deer, dog, frog, horse, ship, and truck (but not pickup truck). There are 6000 images per class with 5000 training and 1000 testing images per class.
CIFAR-100 (Krizhevsky et al. 2009 The CIFAR-100 dataset (Canadian Institute for Advanced Research, 100 classes) a subset of the Tiny Images dataset and consists 60000 32x32 color images. The 100 classes the CIFAR-100 100 are grouped into 20 superclasses. There are 600 images per class. Each image comes with "fine" label (the class to which it belongs) and "coarse" label (the superclass to which it belongs). There 500 training images and 100 testing images per class.
SVHN (Ntzerr el al. 2011). Street View House Numbers (SVHN) is a digit classification benchmark dataset that contains 600,000 32 x2 RGB images of printed digits (from 0 to 9) cropped from pictures of house number plates. The cropped images are centered in the digit of interest, but ncarby digits and other distractors are kept in the image. SVHN has three sets: training, testing sets and an extra set with 530,000 images that are less difficult and can be used for helping with the training process.
10

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3:
Test robustness (%) on SVHN. We omit the standard deviations of runs as they are very small
(< 0.6%).
PGD-20
AA
Network
Threat Model
Method
Best
Last
Diff
Best
Last
Diff
AT
53.10
44.12
8.98
45.09
40.36
4.73
L
RATLR
53.32
43.41
9.92
45.98
39.61
6.37
RATwP
57.91
54.32
3.58
50.32
44.82
5.49
PreAct ResNet-18
AT
66.29
64.73
1.55
63.55
60.14
3.41
L2
RATLR
66.47
62.10
4.36
62.44
58.72
3.72
RATwP
71.66
65.68
5.98
65.17
59.64
5.53
AT
55.57
47.11
8.46
48.05
42.46
5.59
Leu
RATLR
55.34
46.81
8.53
47.94
42.12
5.82
RATwP
58.48
54.92
3.56
54.65
50.46
3.99
Wide ResNet-34-10
AT
67.19
65.08
2.11
62.58
60.86
1.72
L2
RATLR
67.50
64.24
3.27
62.79
59.94
2.85
RATwP
69.07
64.76
4.31
63.12
59.57
3.55
Table 4: The comparison between RAT we and AWP on the CIFAR-10 dataset under the Loo threat model with different adversarial Training methods and network architectures.
Natural
PGD-20
AA
Method
Network
Best
Last
Diff
Best
Last
Diff
Best
Last
Diff
AT
81.16
84.44
-3.28
52.31
44.45
7.86
47.95
42.05
5.90
AWP
PreAct R8sNet-18
81.11
82.00
-0.89
55.39
54.73
0.66
50.12
49.85
0.27
RATwP
82.24
83.36
-1.12
54.85
53.98
0.87
49.19
48.24
0.95
AT
85.49
86.50
-1.01
55.57
47.37
8,20
52.13
46.09
6.04
AWP
Wide ResNet-34110
85.30
85.39
-0.09
58.35
57.16
1.19
54.07
53.49
0.58
RATwP
86.08
86.23
-0.15
58.92
58.23
0.69
54.46
53.98
0.48
TRADES
82.77
82.94
-0.17
52.67
49.65
3.02
49.28
46.80
2.48
TRADES-AW
PreAct ResNet-18
82.05
82.57
-0.52
55.44
54.81
0.63
51.42
50.73
0.69
TRADES- ARA
82.61
83.08
-0.47
55.33
54.85
0.48
51.34
51.04
0.30
We use PreAct 18 (He et al. 2016) and Wide ResNet-34-10 (Zaoruyko & Komodakis 2016 following the same hyperparameter settings for AT in Rice al. (2020): for Loo threat model, 8//255, step size is 1/255 for SVHN, and 2/255 for CIFAR-10 10 and CIFAR- 100; for Le threat model, 128/255 /255, step size is 15/255 for all datasets. For training, all models are trained under 10-step PGD (PGD-10) attack for 200 epochs using SGD with momentum 0.9, weight decay 5 x 10-4, and a piecewise learning rate schedule with an initial learning rate 0.1. Standard data augmentation techniques, including random cropping with 4 pixels padding and random horizontal flips, are applied. The models are decomposed into a series of 4 main layers, corresponding 4 residual blocks of the ResNet architccture. For RATLR, lcarning rate for layer 3 & are set a fixed value of 0.1. For RATwP, weight perturbation is applied in layer & 4, and other hyperparameters are configured per the original paper. For testing, the robustness accuracy is evaluated under two different adversarial attacks, including 20-step PGD (PGD-20) and Auto Attack (AA) (Croce & Hein 2020b). Auto Attack is considered the most reliable robustness evaluation to date, which is an ensembble of complementary attacks, consisting of three white-box attacks (APGD-CE (Croce & Hein 2020b), APGD-DLR (Croce Hein 2020b), and FAB (Croce & Hein 2020a)) and a black-bos attack (Square Attack (Andriushchenko al. 2020))
128
4.2 Performance Evaluation
In this section, we present the experimental results of RATER and RAT wP across three benchmark datasets.
11

---

Published in Transactions on Machine Learning Research (10/2024)
Table 5: The ablation study with TRADES on the CIFAR-10 dataset using PreAct ResNet-18 under the
Loo threat model.
Natural
PGD-20
AA
Method
Layer
Best
Last
Diff
Best
Last
Diff
Best
Last
Diff
TRADES-RATwP
H
82.77
82.94
-0.17
52.67
49.65
3.02
49.28
46.80
2.48
TRADES-RATWE
[4]
82.98
83.11
-0.13
52.35
49.76
2.59
48.48
46.66
1.82
TRADES-RATWI
3.4]
82.61
83.08
-0.47
55.33
54.85
0.48
51.34
51.04
0.30
TRADES-RATwP
[23,4]
81.89
82.37
-0.48
55.44
54.91
0.53
51.24
51.20
0.04
TRADES-RATwP
[1,2,3.4]
82.05
82.57
-0.52
55.44
54.81
0.63
51.42
50.73
0.69
CIFAR-10 Results. The evaluation results CTFAR10 dataset are summarized in Table where "Best." is the highost test robustness achieved during training; "Last" is the test robustness at the last epoch chockpoint; "Difl" denoles the robust accuracy gap ween the "Best." & "Last". It is observed that RATwp generally achieves the best robust performance compared to RATLR & standard AT. Regardless, both RATLR and RAT tighten the robustness gaps by a significant margin, indicating they can effectively suppress robust overfitting.
CIFAR-100 Results. We also show the results on CIFAR100 dataset in Table We observe similar performance like CIFAR1O, where both RATLR and wP is able significantly reduce the robustness gaps. For robustness improvement, RATwp stands out to be the leading method. The results further verify the effectiveness of the proposed approach.
SVHN Results. The results on the SVHN dataset are shown in Table 3 where robustness gap are also narrowed down to a small margin by RATwP. SVHN dataset is a special case where RAT strategy does not improve robust overfitting- Unlike CTFARTO and CIFART00, learning rate decay SVHN's training does not have much connection to the sudden increases in robust test performance the prevalence of robust overfitling, and hence makes RATLR ineffective. Other than this, The improvement in robust generalization gaps can be witnessed in all cases, demonstrating the proposed approachs are generic and can be applied widely.
Comparision with AWP. We further provide a comparison between RAT and AWP (wu et al., 2020) on the CIFAR-10 dataset under the Los threat model, using different adversarial training methods and network architectures. The results are summarized Table 4 For natural accuracy, RAT enforces less regularization compared to AWP, thus achieving higher natural accuracy. For adversarial robustness, is observed that RATwr slightly degrades the model's performance in some cases. However, RAT generally maintains comparable adversarial robustness to AWP, which can be attributed to the strong correlation between the latter layers and robust overfitting.
Ablation Study. Finally, we provide an ablation study to illustrate the selection of layers in our ex- periments. Specifically, we apply regularizations to different layers in the RATwp method and conduct experiments with TRADES on the CIFAR-10 dataset using PreAct ResNet-18 under the Loc threat model. The experimental results are summarized in Table 5 is observed that when regularizations are applied only to layer 4, the model can mitigate robust overfitting to certain extent, but its robustness performance is poor. When regularizations are applied to layer 3&4, the model achieves better adversarial robustness while effectively mitigating robust overfitting. Therefore, in our experiments, we consistently chose to apply rogularizations to layer 3&4 in considcration of both robust overfitting and adversarial robustness.
5 Conclusion
In this paper, we investigate the effects of different network layers on robust overfitting and identify that robust overfitting is mainly driven by the optimization occurred the latter layers. Following this, we propose a robust adversarial training (RAT) prototype to specifically hinder the optimization of the latter layers in the process of training adversarial network. The approach prevents the model from overfitting
12

---

Published in Transactions on Machine Learning Research (10/2024)
the latter parts of the network, which effectively mitigate robust overfitting of the network whole. We then further demonstrate two implementations of RAT: one locally uses fixed learning rate for the latter layers and the other utilize adversarial weight perturbation for the latter layers. Extensive experiments show the effectiveness of both approaches, suggesting RAT is generic and can be applied across different network architectures, threat models and benchmark datasets to mitigate robust overfilting.
Acknowledgments
Tongliang Liu is partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031.
References
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-effciient black-box box adversarial attack via random search. In European Conference on Computer Vision, pp. 484-501. Springer, 2020.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning, pp. 274 283. PMLR, 2018.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias variance 15854, 2019.
bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849-
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C Duchi. Unlabeled data improves adversarial robustness. arXiv preprint 1rXiv:1905.13776, 2019.
Chen Chen, Jingfeng Zhang, Xilie Xu, Tianlei Hu, Gang Niu, Gang Chen, and Masashi Sugiyama. Guided interpolation for adversarial training. arXiv preprint arxiv:2 02. 07327, 2021.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening. In International Conference on Learning Representations, 2020.
Yumpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, and Jiashi Feng. Dual path networks. Advances in neural information processing systems, 30, 2017.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary allack. In International Conference on Machine Learning, pp. 2196 2205. PMLR, 2020a.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206 2216. PMIR, 2020b.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0448 2018.
Chengyu Dong, Liyuan Liu, and Jingbo Shang. Label noise in adversarial training: novel perspective to 2022a.
study robust overfitting. Advances in Neural Information Processing Systems,
Yinpeng Dong, Ke Xu, Xiao Yang, Tianyu Pang, Zhijie Deng, Hang Su, and Jun Zhu. Exploring memoriza- tion in adversarial training. In International Conference on Learning Representations, 2022b.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint orXii:1412.663 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In
Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
13

---

Published in Transactions on Machine Learning Research (10/2024)
Sachyung Lee, Hyungyu Lee, and Sungroh Yoon. Adversarial vertex mixup: Toward better adversarially robust generalization. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 272 281, 2020.
Lin Li and Michael Spratling. Data augmentation alone can improve adversarial training. arxio preprinl W:Xiv2231.098777 2023.
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1778-1787, 2018.
Feng Liu, Bo Han, Tongliang Liu, Chen Gong, Gang Niu, Mingyuan Zhou, Masashi Sugiyama, et al. Proba- bilistic margins for instance reweighting in adversarial training. Advances in Neural Information Processing Systems, 34:23258- 222699, 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arxiv: (eee:1766.00000 2017.
Amir Najafi, Shin-ichi Maeda, Masanori Koyama, and Takeru Miyato. Robustness to adversarial perturba- tions in learning from incomplete data. Advances in Newral Information Processing Systems, 32, 2019.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference on Learning Representations, 2020. URL https:://ppenreeienn http:://wppppaooooee.aee/rrrrrrp
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation a defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium security and privacy (SP), pp. 582-597. IEEE, 2016.
Sylvestro-Alvis Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann. Data augmentation can improve robustness. Advances in Neural Information Processing Systems, 04:29935-29990 2021.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pp. 8093-8 8104. PMI.R, 2020.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adversarially robust generalization requires more data. Advances in newral information processing systems, 31, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint (rXii:1449.15550 2014.
Chubiao Song, Kun He, Jiadong Lin, Liwei Wang, and John E Hoperoft. Robust local features for improving the generalization of adversarial training. In International Conference on Learning Representations, 2020.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXio preprint arXi::1312.611 61999999.1999.1 2013.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Tan Goodfellow, Dan Boneh, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Represen- tations, 2018
Jonathan Uesato, Jean-Baptiste Alayrac, Po-Sen Huang, Robert Stanforth, Alhussoin Fawzi, and Pushmoct Kohli. Are labels required for improving adversarial robustness? ar preprint arXio:1905. 100..9955.775, 2019.
Yifei Wang, Liangchen Li, Jiansheng Yang, Zhouchen Lin, and Yisen Wang. Balance, imbalance, and rebal- ance: Understanding robust overfitting from a minimax game perspective. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International Conference on Learning Represen- tations, 2019.
Yuxin Wen, Shuai Li, and Kui Jia. Towards understanding the regularization of advensarial robustness on neural networks. In International Conference on Machine Learning, pp. 10225-10235. PMLR, 2020a.
Yuxin Wen, Jiehong Lin, Ke Chen, CL Philip Chen, and Kui Jia. Geometry-aware generation of adversarial point clouds. TEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6):2984 2299,, 2020b.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. arXiv preprinl arXiv:2004.0588 2020.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
Chaojian Yu, Bo Han, Li Shen, Jun Yu, Chen Gong, Mingming Gong, and Tongliang Liu. Understanding robust overfitting of adversarial training and beyond. In International Conference on Machine Learning, pp. 25595-2556 25610. PMLR, 2022.
Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2403 2412, 2018.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He, John Hopcroft, and Liwei Wang. Adversarially robust generalization just requires more unlabeled data. arXiv preprint ar 1r::: 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theo- retically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pp.
PMLR, 2019.
Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry- aware instance-rewwiggtted adversarial training. orXiv preprint arXi::2010.0119 2020.
A More Evidences on the Layer-wise Properties of Robust Overfitting
In this section, we first show that robust overfitting is a phenomenon uniquely associated with adversarial training and does not occur in natural training. We then provide more empirical experiments to showcase the layer-wise properties of robust overfitting across different datasets, network architectures, threat models and adversarial training methods. Specifically, we use the proposed strategies mentioned in Section 3.3 to put restriction on the optimization of different network layers. We can always observe that there is significant robust overfitting relief when we regularize the optimization of the latter layers, while robust overfitting is prevalent for other settings. These evidences further highlight the strong relation between robust overfitting and the optimization of the latter layers.
A.1 Robust overfitting: A phenomenon unique to adversarial training
In Figure we have shown the phenomenon of robust overfitting within adversarial training. In this part, we show that robust overfitting is uniquely associated with adversarial training and does not occur in natural training. Specifically, we train a PreAct ResNet-18 using natural training on the CIFAR-10 10 dataset and evaluate the model's performance under different perturbation budgets, as shown observe that across various perturbation budgets, the model's test accuracy does not show a decrease similar to that in robust overfitting, indicating that robust overfitting phenomenon does not occur in natural training.
Figure 5 We can
15

---

Published in Transactions on Machine Learning Research (10/2024)
0.25
0.12
0.016
Towning (h-2/255))
Training (1/4/200))
0.014
Training 16-8/255)
0.2
sst (bx2/255)
Test (8:4/255)
Teat (b+6/153))
0.012
0.8
0.28
0.01
0.6
0.1
0.04
0.005
0.004
Training (1-d)
0.05
Test (2-0)
0.007
Epachs
Figure 5: The evaluation results of natural training using PreAct ResNet-18 on the CIFAR-10 dalaset.
A.2 Evidences across datasets
We show that the layer-wise properties of robust overfitting are universal across datasets CIFAR-100 and SVHN. We adversarially train PreAct ResNet-18 using AT under lo threat model on different datasets with the same settings as Section 3.3 The results are shown in Figure and 7 Note that for SVHN, regularization strategy utilizing a fixed learning rate (RATLR) does not improve robust overfitting (Figure G). Unlike CIFARIO and CIFAR100, SVHN's robust overfitting appears before the first learning rate decay. Also, learming rate decay in SVHN's training does not have any relation to the sudden increases in robust test performance the appearance of robust overfitting. Hence, SVHN dataset is special case where RÁT L.R does not apply. For all other cases, robust overfitting is effectively mitigated by regularizing the optimization of the latter layers.
A.3 Evidences across threat models
We further demonstrate that the generality of layer-wise properties of robust. overfitting by conducting experiments under l2 threat model. The settings are the same Section 3.3 The results are shown in Figure 8 and D Under l2 threat model, except for SVHN dataset where regularization strategy utilizing a fixed learning rate (RATLR) does not apply, robust overfitting is effectively mitigated by regularizing the optimization of the latter layers.
A.4 Evidences across network architectures
In this part, we show that the generality of layer-wise properties of robust overfitting by conducting exper- iments across different network architectures (PreAct ResNet-18 (He et al. 2016). PreAct ResNet-34 (He eet al.
2016), VGG-16 (Simonyan & Zisserman 2014), DPN-26 (Chen et al. 2017) and DLA Yuu el al. For PreAct ResNet-18 and PreAct Net-34,,,, the division of the network layer are the same as Scction 3.3 For DPN-26 and DLA, similarly, we consider the four main network blocks separately for layerl, layer2, layer3 and layer4. For VGG-16, "reeture,.0", "features. "features.3", "features.4", "features. 7", "features.8" regarded layer1, "features. 10", "features. 11", "features.144 14", "features. "features 117". "features. 18" are regarded layer2, "features.20 20"" "features.21" "features.22 24", "features.25 " "features.27" "features.22 28", "features.30 330" are regarded as layer3, and "features.31", "features.34" "features.35", "fea- tures.37", "features.33 33"", "features.40" "features.411 are regarded as layer4. The results are shown in Figure These results clearly indicate that the optimization of the latter layers present a strong correlation to the robust overfitting phenomenon, and the layer-wise properties of robust overfitting are common in different network architectures.
2018)
10
A.5 Evidences across adversarial training methods
We further demonstrate that the generality of layer-wise properties robust overfitting by conducting experiments using different adversarial training methods (Standard AT (Madry al. (017)) and TRADES (Zhang al.) ).19)). The settings are the same as Section 3.3 The results are shown Figure I1 The experiments results indicate that the latter layers have stronger connections to robust. overfitting than the
16

---

Published in Transactions on Machine Learning Research (10/2024)
front layers do, and the layer-wise properties of robust overfitting is generally hold regardless of the chosen adversarial training methods.
0.30
0.30
0.30
0.30
0.28
0.28
0.28
0.28
0.20
0.26
0.28
0.20
024
0.24
0.24
0.24
0.2
0.22
0.22
0.22
0.0
ca
0.20
0.20
0.20
Atolanderd
ALTRULL
0.18
0.18
0.18
0.18
0.18
0.16
0.16
0.36
1.4)
50
100
150
200
100
150
200
5.0
150
200
100
150
200
Epochs
(a) AT using PreAct ResNot-18 on CIFARI00 dataset under Loc. threat model
0.55
0.55
0.55
0.55
6.50
0.50
0.50
0.50
0.45
0.45
0.45
0.45
0.40
0.40
0.40
0.40
That
Sundare <7.x
A1 Saanddd
0.35
0.35
0.35
0.35
0.30
25
50
1000
130
125 150
0.30
25
Epochs
(b) AT using PreAct ResNet-18 on SVHN dataset under loo threat model
Figure 6: Robust lest performance of AT using fixed learning rate for different sets of network layers in PreAct ResNet-18 18, across datasets (CIFAR-100 100 and SVHN) under Lo threat model.
17

---

Published in Transactions on Machine Learning Research (10/2024)
6.35
0.35
0.35
0.35
0.30
/
0.30
0.30
0.30
Yom /
Test Accurecy
6.25
0.25
0.25
0.25
0.20
0.20
0.20
0.20
2w0 coustand
ung.12)
alundard
alanddrd
awo 12,
0.15
0.15
0.15
0.15
awp
0.10
50
150
200
0.10
100
150
200
9.10
50
150
200
0.10
100
150
200
Epochs
(a) using PreAct ResNet-18 on CIFAR100 dataset under loo threat model
0.60
0.00
0.60
0.60
0.55
Ronoo Accumey
6.45
0.45
0.45
0.45
0.40
0.40
0.40
Fi_wwn.o Standor
2w0 ctantand
unp. (T.
anp alundard
alandurd
awo 12,
6.30
0.90
0.90
0.30
0.25
125
150
25
50
100
125
150
0.25
100 125 150
25
50
100
125 150
Epochs
(b) AT using PreAct ResNet-18 on SVHN dataset under boo threat model
Figure 7: Robust test performance of AT applying AWP for different sets of network layers in PreAct ResNet-16 18, across datasets (CIFAR-100 and SVHN) under loo threat model.
18

---

Published in Transactions on Machine Learning Research (10/2024)
0.72
0.70
0.70
0.68
0.68
0.68
(mA))
0.68
0.66
0.00
0.00
0.64
0.64
0.04
Text
0.60
A1.%
0.60
0.60
AT. (t.otandrr3
0.60
ALTRULUL 2.30
21.x x12]
(e. 21
AT 1...i-11 12.3
0.58
AT PE 11. 2.44
41.1% x19]
AT Sx. 12.41
0.56
0.56
41
0.56
0.56
AR.. 1.4)
Epocts
(a) AT using PreAct ResNet-18 CIFARIO dataset under l2 threat model
0:450
6.425
0.425
6.400
0.400
0.400
0.400
0.375
0.375
0.375
0.375
0.350
0.325
8.325
Text
0.300
0.300
0.300
standard
0.300
A1.7
A1.3..e (.T. 21
K1. AT Sx. In 12.41
ATTRU DDL 2.43
6.275
21.1x x13]
0.275
0.275
0.275
AT Po 3.4)
AT3x (34)
AT 41
RUU. 1.4)
0.250
50
3.0
150
200
0.250
50
150
200
0.250
50
100
350
200
0.250
50
1.00
(b) AT using PreAct ResNet-18 on CIFAR100 dataset under la threat model
0.675
0.675
0.650
0.650
0.650
0.650
Text Robust Accuncy
0.625
0.625
0.625
0.625
0.600
0.575
0.550
0.500
standard
0.550
PXP 3.30
A13 U2)
01.3./1/1 21
ALTRUU 2.4
6.525
2T.1x x13]
0.525
AT. (1.11.31
0.525
5r 12.
0.525
AT 11. 3.40
AT.1x x34)
AT.5x wtw.t/i........
AT Sx. k 12.4
AT.
0.500
50
75
100
125
150
0.500
25
100
325
0.500
50
75
100
125
150 0.500
25
100 125
(c) AT using PreAct ResNet-18 on SVHN dataset under threat model
Figure 8: Robust test performance of AT using a fixed learning rate for different sets of network layers in PreAct ResNet-18 18, across datasets (CIFAR-10, CIFAR-100 and SVHN) under lz threat model.
19

---

Published in Transactions on Machine Learning Research (10/2024)
Test Rebust Arcumcy
mandord A[_0wp_]]]
A1 A1,300 ptaaddr
sep 12]
aap (1.
AT. owo 12, 22.31
awp 13 2.4
ang.
awp
0.50
150
0.50
100
0.50
9.50
Epocts
(a) AT using PreAct ResNet-18 CIFARIO dataset under l2 threat model
6.45
0.45
0.45
0.45
0.40
0.40
0.40
0.40
0.35
Text Rrb
standard 37,340_1
standard
awp
A1 (0_.wo_1 12, AT awo 12 41
smp [3]
pap
awp 1. 3.4
A7 p..w....
50
1.00
350
200
300
150
50
150
200
100
150
(b) AT using PreAct ResNet-18 on CIFAR100 dataset under la threat model
Text Robust Accurccy
0.60
0.60
sbandars
standard
awo
0.55
[.o__])) swp (3]
0.55
0.55
Al Al.
0.55
(1..). 13.22
axp. (1.33
XT avo 12, 22.41
awp 3.40
AT smp.
AT pwp.ppp....
AT. avo 13.41
awp 3.4)
0.50
300
125
150
0.50
25
50
75
1000
125
0.50
1000
125
150
0.50
25
50
75
100
125
150
(c) AT using PreAct ResNet-18 SVHN dataset under threat model
Figure Robust test performance of AT applying AWP for different sets of network layers in PreAct ResNet-18, across datascts (CIFARIO, CIFAR-100 and SVHN) under threat model.
20

---

Published in Transactions on Machine Learning Research (10/2024)
0.6
0.6
066
0.6
0.575
0.575
0.575
0.575
0.55
Tes: Pobat, Jobaat Actunay
0.5
05
1N
0.5
0.475
0.475
0.475
0.45
0.45
/W
0.45
0.425
0.4
AT. AlLengpalarderr (LLyn (1.ppp112:3
240,
(12241
0.375
AT AL.
0.375
0.575
200
[1.341
3.35
0.35
Jws 13.41
0.35
0.35
150
200
50
100
150
200
50
100
150
200
50
100
250
Epodas
(a) AT using PreAct ResNet-34 on CIFAR10 dataset under too threat model
0.535
0.525
0.535
0.535
Test Radket/ouus
0.035
0.425
0.475
0.475
0.435
0.425
0.425
0.425
AT
AT Jwn soadart
0.375
He
0.375
0.375
AT.amxp_1..0
0.575
PAD (1.2.41
0.35
0.35
AT. [44]
200
100
150
Fpncs
(b) AT using VGG-16 16 on CIFARIO dataset under 100 threat model
0.525
0.535
8.535
0.575
0.525
0.525
0.525
0.525
Text Todtaat//oooun Accursy
0.455
0.475
0.475
0.475
0.45
0.255
0.425
0,425
0.575
0.375
AT standad
0.375
WAp, connard
0.375
2WN condard
Anaa 11.21
0:35
Jop 12.01
0.35
AT Jwn 11224
0.325
0:325
0:325
0.1
0
50
100
150
200
50
100
150
200
50
100
150
200
50
100
150
200
Epocts
(c) AT using DPN-26 CIFARIO dataset under to0 threat model
0.525
0.55
0.55
0.55
0.55
Toxt Toloostmmm
0.5
0.475
0:475
0.475
D.475
0.4
Al Lowa Jaandad
0.4
0.4
0.8
Al LLLg standard
0.575
AT JMO standard
0.375
Sepp standard
0.37%
Al
0.325
47
Al O0ow [4]
0.325
Jep 11.47
0.325
0.325
AT AT.o (1.34)
0.3
0.3
0.3
(d) AT using DLA on CIFAR10 dataset under Loo threat model
Figure 10: Robust test performance of AT applying AWP for different sets of network layers across network architectures (PreAct ResNet-34, VGG-16, DPN-26 and DLA), CIFAR-10 datasets under loo threat model.
21

---

Published in Transactions on Machine Learning Research (10/2024)
0.575
0.975
0.575
0.575
0.55
0.55
0.55
0.55
0.525
0.525
0.525
0.525
Tea Rooket Aocuma
0.5
0.5
0.6
0.5
0.475
0.475
0.475
0.455
0.45
0.45
0.45
0.425
JAp standard
0.425
AT. LWW. stancard
Al.zwp Scaddard
AT Lang stannard
ATL.w (A.2]
0.4
12.31
[1.7.4
ATLANg [1.3]
12.4]
[t [L.54)
XT. pap (1:41
50
100
150
200
50
100
150
200
50
100
150
200
50
100
150
200
Epocas
(a) AT using PreAct ResNet-18 CIFAR10 dataset under too threat model
0.6
0.6
0.6
0.6
0.575
0.575
0.575
0.575
0.55
0.35
0.525
Text Ro..oott,oosss
0.5
0.5
05
0.5
0,47%
0.475
0.475
0475
0.45
0.45
/W
0.45
0.425
0.4
swg. standar
AT.
AT. 21..11
T1.241
0.375
0.375
0.575
(00) 1L3.41
3.35
0.35
owe 11,41
0.35
0.35
AT,000
100
150
200
50
100
150
200
50
200
150
200
50
100
250
200
Epodhs
(b) AT using PreAct ReaNett-4 on CIFARIO clataset under Ino threat model
0.575
0.575
0.55
0.55
3.56
0.525
0.525
0,525
0.525
Thet Actoo
0.5
8.5
0.5
0.475
0475
0.475
0.475
0.45
0.425
0425
0.425
0.425
0.4
0.4
0.375
TRADES,
TRADES,
0.375
TRADSS 2wn dbweled
ano 11.2
0.575
TRADES, anp standard
awn 12.1
0.375
TRACKS, zwo standard
0.325
TRAOFS, sap.
TRADE1 m4
0.315
TRADES [14)
0.325
0.325
TRADES, (23.41
100
150
200
Egocbs
(c) TRADES using PreAct ResNet-18 CIFAR10 datasct under too threat model
0.6
0.6
0.6
0.6
0.575
0.575
0.575
0.575
0.525
Test Rdbaut....mm
0.5
0.5
05
0.5
0.475
0,7%%
0.775
0.455
0.45
0.45
0.45
0.45
0.425
TRADES, sup, rep_standard
0.4
TRADES, .op_04
rup, [2]
04
TRASCS awo.
TRADES axp 12.31
TRAGES xwp, convarr
0.4
TRADES
TR6OES mxp (1.24)
0.755
0.375
1980E5 Jwo T1.
0.375
TRADES 1241
0.755
TRADES mop 11p.11
0.35
sep.
0.35
0.35
0.35
0
50
100
150
200
50
100
150
200
0
50
100
150
200
0
50
100
150
200
Fiodhs
(d) TRADES using PreAct ResNet-34 on CIFAR1O dataset under loc threat model
Figure 11: Robust test performance of AT and TRADES applying AWP for different sets of network layers in Prt ResNet-18 and PreAct ResNet-34, on CIFAR-10 datasets under loc threat model.
22


Reviews:
[{"review_id": "EtAd8Clt2y", "reviewer": "Reviewer_JDM1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n1. I am not familiar with the adversarial training literature, so my review should be taken with a grain of salt. But the authors clearly state how their focus differs from existing works on robust generalization, which convinced me that their investigation of layer-wise contribution to robust overfitting is unique. \n2. The two proposed solutions to mitigate robust overfitting are straightforward and should be scalable. \n\n**Weakness**:\n1. My main concern is that the latter blocks of ResNets contain more parameters than the former blocks. However, the current paper does not provide insights or analysis on whether the latter blocks’ larger effect on robust overfitting is due to the larger number of parameters or something else perhaps more inherent to the latter layers. Therefore, it is hard to conclude the cause of the empirical observations on the layer-wise effects presented in the paper. \n\n2. The effectiveness of the proposed RAT (e.g. RAT-WP) seems limited, when evidence in Table 4 shows that AWP (which is AWP applied to all layers, if I understood correctly), achieves better performance in both test robustness and robust generalization gap. I understand that in comparison to standard AT, the experiments suggest that regularizations on latter layers have a larger contribution to the reduction of the robust generalization gap. However, again, without an analysis on whether this is simply due to the larger number of parameters in the latter layers, it is hard to see how interesting the findings are. \n\n3. There are no quantitative results of RAT applied with other adversarial training methods like TRADES and with other networks architectures. The authors show visualizations of robust test performance of AWP under TRADES instead of standard AT in Figure 10 and performance of AWP for different network architectures in Figure 9, but it would be good to see some quantitative comparisons like in Table 1 through Table 4. That will help verify the effectiveness of RAT. It might also help to answer my second question in Requested Changes, please see below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission’s main contributions are (i) analyzing robust overfitting in neural networks in a layer-wise fashion and identifying that optimization in latter layers has more effect on robust overfitting; (ii) proposed a general prototype (RAT), where additional regularization is imposed on latter layers of a network, to mitigate robust overfitting. The proposed prototype is instantiated with two types of methods: fixed non-decaying learning rate and adversarial weight perturbation (AWP).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical**:\n1. It appears that Table 1 and Table 4 share the same experiment setup. If this is true, then why does the first row (AT) of Table 4 not correspond to the first row of Table 1, but to the first row of Table 3 instead? \n\n2. In Section 3.1, it is mentioned that “robust overfitting is mitigated at the cost of robust accuracy. For example in AT-fix-param-[3,4], if we leave both layer 3 & 4 unoptimized, robust overfitting will practically disappear, but the peak performance is much worse compared to standard AT”. Are there any motivations for selecting Layer 3&4 instead of Layer 4 only to apply regularizations in Section 4? For other types of network architectures, how does the selection of “latter layers” transfer? \n\n3. In Figure 4(b), why are the front layers “Layer 1&2&3” while the latter layers are “Layer 3&4”? I am confused with the definition of former vs latter layers here.\n\n**Nice to have**:\n1. How are the semi-supervised learning methods in Section 2.2 relevant to the methods and observations in this paper? Is there any reason for mentioning them in the Related Work section? \n\n2. In Section 4.2, it is mentioned that “learning rate decay in SVHN’s training does not have much connection to the sudden increases in robust test performance or the prevalence of robust overfitting”. Where is this evidence for the SVHN dataset, is it in Figure 5? \n\n3. In Figure 4(c), it is shown that RAT_WP has a negative robust generalization gap throughout most of the training. Can the authors offer some explanations for this unusual phenomenon? What about AWP applied to all layers?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0yHclz7y00", "reviewer": "Reviewer_bT7r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The paper is mostly well-organized and written clearly.\n\nWeaknesses:\n- The logical argument for why regularization should be preferentially applied to the later layers is flawed.\n- The experiments have many weaknesses.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the problem of overfitting in adversarial training of deep neural networks. It is observed that regularizing the final layers of a deep neural network has a larger effect than regularizing the earlier layers. Based on this observation, it is proposed that regularization be applied to the final layers of the network. In experiments, the authors compare different versions of their regularization scheme to unregularized models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The authors claim that weight decay is a regularization method that mitigates overfitting (Section 1 p1). This is false.\n- In Section 3.1, the authors claim Figure 2 shows that \"any settings that do not fix the parameters for layer 4 result in a more severe gap between the best accuracy and the accuracy at the last epoch.\" This is technically accurate, but the authors should acknowledge the more obvious effect is that fixing the Layer 4 parameters decreases performance! This underfitting is what causes there to be no overfitting. The way it is phrased in the paper is misleading and needs to be revised.\n- It is misleading to claim this phenomena as \"a general property of adversarial learning\" (Section 3.1, second paragraph).  The authors need to clarify their claim as being either: (1) this is a property particular to adversarial training, in which case they need to give evidence that it doesn't affect non-adversarial training, or (2) clarify to the reader that this is a general property of deep learning. I believe the latter is true. It is well known that weight updates in the last layer are particularly important in deep learning in general (not just in adversarial training), so for the experiments in Figure 2, I would expect similar patterns with non-adversarial training.\n- In the experiments comparing the proposed regularization methods, the authors report the \"best\" and \"final\" accuracy of the model. I think the final accuracy is irrelevant because in practice one would use early stopping. The best accuracy is problematic because taking the maximum of the test set loss over training gives an overestimate of the true generalization performance. The correct way to do this experiment is to have a train/valid/test split where early stopping is performed using the validation set and then the test set performance is reported. I think this is critical for fair experiments. \n- The experiments show that some regularization helps performance compared to non-regularized adversarial training. This is no surprise. But are there any experiments showing that applying regularization using the proposed RAT method is an improvement over the existing regularization methods mentioned? To be convincing, these experiments need to be done carefully with hyperparameter optimization on a validation set.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VgahC4MKCp", "reviewer": "Reviewer_CrXW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. Main claims are supported by most of the experiments\n2. Experiments are well documented\n3. Main ideas are clearly explained\n\n**Weaknesses**\n\n1. Some unclear parts\n2. Some experiments provide conflicting evidence\n3. The overall significance of the main claim is unclear", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study the problem of robust overfitting, that is, the fact that the test's robust accuracy learning curve has a U-shape. More specifically, in the present contribution, the authors focus on how much different parts of neural networks contribute to robust overfitting. It is demonstrated with many careful experiments that the last layers contribute more to the robust overfitting problem.\n\nBased on this observation, authors modify two methods, weight perturbation and learning rate (related to loss scaling), to act differently on the first and last layers. Both techniques perform reasonably well but do not appear to improve over state-of-the-art methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to stress that I am not an expert in adversarial machine learning, so the present review is a detached perspective from the outside of the field. I suggest Action Editor rely more on reviewers more experienced in the matter.\n\n**Unclear parts**\n\n1. Can authors please include a formal definition of robust accuracy? How robust accuracy is measured in training time? Is it done with the same adversarial?\n2. It is not entirely clear what constitutes the last layers. As I understand authors consider classification problems, so the network's output should correspond to probabilities or log probabilities of classes. Is this linear/softmax layer included in the last layer?\n3. Does the initialization of the network affect the outcomes of the experiments when some layers are frozen?\n\n**Conflicting evidence**\n\nAll the experiments with frozen layers suggest that the last layers' regularisation has more effect on robust overfitting (Figure 3a and Figure 4a). These experiments can be considered a special type of intervention/ablation when one deliberately fixes part of the parameters.\n\nHowever, in Table 4 we see that AWP performs better than RAT-WP. This case also can be considered a type of intervention to AWP: in AWP adversarial is allowed to perturb all weights, whereas in RAT-WP adversarial perturbs only weights in the last two layers. The result of this intervention suggests that it is better to perturb all weights, and later layers have no special significance.\n\nI would like to ask the authors to:\n1. Comment on this possible interpretation of presented data\n2. Add results with AWP to tables 1, 2, and 3 -- this will allow one to judge whether perturbation of all layers leads to better robustness\n\n**Overall significance**\n\nGiven that AWP performs better than RAT-WP, and RAT-WP does not show state-of-the-art results, it is unclear to what extent the observation made by the authors is significant.\n\nI suggest the authors to:\n1. Extend the discussion of comparison between RAT-WP and AWP.\n2. Provide additional data that shows differences between RAT-WP and AWP. For example, one may suspect that RAT-WP can lead to higher natural accuracy because it enforces less regularisation. Can the authors provide natural accuracy to all methods in Table 4?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission works on analyzing the overfitting issues existing in adverstrail training (e.g., robust overfitting) for addressing adversarial attacks. Previous analyses in such a context do not give a finer analysis per network layer. The proposed research first finds that different layers have varying effects on robust overfitting, and then proposes a robust adversarial training prototype that imposes additional regularization on the latter parts of networks.Two instatiations of RAT verify the efficacy of the proposed research.

Reviewer bT7r tends to think that some of the claims made in the submission are flawed, e.g., "weight decay is not a regularization method that mitigates overfitting", and "it is misleading to claim this phenomena as "a general property of adversarial learning"". While some of these disagreements have been resolved after the rebuttal phase, a few others remain. The key disagreement lies in whether the phenomena identified by the authors are unique to the context of robust overfitting, or they are just general phenomena for training NNs, and the reviewer also expects additional, ideally theoretical, evidence. AE concludes the debates by agreeing that although the identifications and contributions made in the submission are not completely resolving the robust overfitting issues, they show progress along the line and the arguments made in the submission are empirically supported by the experiments, thus satisfying the acceptance criteria of TMLR.  

Reviewers CrXW and JDM1 are getting positive on the submission after the rebuttal phase; their decisions are also based on the TMLR acceptance criteria. 

For the final version of the paper, please the authors also include experiments for the last few Q%A with the reviewers.
Decision: Accept (poster)