On Intriguing Layer-Wise Properties of Robust Overfitting in Adversarial Training
Authors: Duke Nguyen
Chaojian Yu
Vinoth Nandakumar
Young Lee
Tongliang Liu
Venue: Submitted to ICLR 2023 (2023)

Abstract: Adversarial training has proven to be one of the most effective methods to defend against adversarial attacks. Nevertheless, robust overfitting is a common obstacle in adversarial training of deep networks. There is a common belief that the features learned by different network layers have different properties, however, existing works generally investigate robust overfitting by considering a DNN as a single unit and hence the impact of different network layers on robust overfitting remains unclear. In this work, we divide a DNN into a series of layers and investigate the effect of different network layers on robust overfitting. We find that different layers exhibit distinct properties towards robust overfitting, and in particular, robust overfitting is mostly related to the optimization of latter parts of the network. Based upon the observed effect, we propose a robust adversarial training (RAT) prototype: in a mini-batch, we optimize the front parts of the network as usual, and adopt additional measures to regularize the optimization of the latter parts. Based on the prototype, we designed two realizations of RAT, and extensive experiments demonstrate that RAT can eliminate robust overfitting and boost adversarial robustness over the standard adversarial training.

Under review as a conference paper at ICLR 2023
ON INTRIGUING LAYER-WIS PROPERTIES OF RO- BUST OVERFITTING IN ADVERSARIAL TRAINING
Anonymous authors
Paper under double-blind review
ABSTRACT
Adversarial training has proven to be one of the most effective methods to defend against adversarial attacks. Nevertheless, robust overfitting is a common obsta- cle in adversarial training of deep networks. There is common belief that the features learned by different network layers have different properties, however, existing works generally investigate robust overfitting by considering DNN a single unit and hence the impact of different network layers on robust overfit- ting remains unclear. In this work, divide DNN into series of layers and investigate the effect of different network layers on robust overfitting. We find that different layers exhibit distinct properties towards robust overfitting, and in particular, robust overfitting is mostly related to the optimization of latter parts of the network. Based upon the observed effect, we propose robust adversar- ial training (RAT) prototype: in a minibatch, we optimize the front parts of the network as usual, and adopt additional measures regularize the optimization of the latter parts. Based on the prototype, we designed two realizations of RAT, and extensive experiments demonstrate that RAT can eliminate robust overfitting and boost adversarial robustness over the standard adversarial training.
INTRODUCTION
Deep neural networks (DNNs) have been widely applied in multiple fields, such as computer vi- sion (He et al.) 2016) and natural language processing (Devlin et al. 2018). Despite its achieved success, recent studies show that DNNs are vulnerable to adversarial examples. Well-constructed perturbations on the input images that are imperceptible to human's 's eyes can make DNNs lead to a completely different prediction (Szegedy et al., 2013). The security concern due to this weakness of DNNs has led to various works in the study of improving DNNs robustness against adversar- ial examples. Across existing defense techniques thus far, Adversarial Training (AT) (Goodfellow) et al. 2014 Madry et al. 2017), which optimizes DNNs with adversarially perturbed data instead of natural data, is the most effective approach (Athalye et al. 2018). However, it has been shown that networks trained by AT technique do not generalize well (Rice et al. 2020). After a certain point in AT, immediately after the first learning rate decay, the robust test accuracy continues to decrease with further training. Typical regularization practices to mitigate overfitting such as 11 & 12 regular- ization, weight decay, data augmentation, etc. are reported to be inefficient compared simple early stopping (Rice et al., 2020).
Many studies have attempted to improve the robust generalization gap in AT, and most have gen- erally investigated robust overfitting by considering DNNs as whole. However, DNNs trained on natural images exhibit common phenomenon: features obtained in the first layers appear to be general and applicable widespread, while features computed by the last layers are dependent on particular dataset and task (Yosinski et al. 2014). Such behavior of DNNs sparks question: Do different layers contribute differently to robust overlitting? Intuitively, robust overfitting acts as an unexpected optimization state in adversarial training, and its occurrence may be closely related to the entire network. Nevertheless, the unique effect of different network layers robust overfitting is still unclear. Without a detailed understanding of the lyserwise mechanism of robust overfitting, is difficult completely demystify the exact underlying cause of the robust overfitting phenomenon.
In this paper, we provide the first layer-wise diagnosis of robust overfitting. Specifically, instead of considering the network whole, we treat the network as composition of layers and sys-

---

Under review as a conference paper at ICLR 2023
tematically investigate the impact of robust overfitting phenomenon different layers. To do this, we first fix the parameters for the selected layers, leaving them unoptimized during AT, and then normally optimize other layer parameters. We discovered that robust overfitting is always mitigated in the case where the latter layers are left unoptimized, and applying the same effect to other layers is futile for robust overfitting, suggesting strong connection between the optimization of the latter layers and the overfitting phenomenon.
Based upon the observed effect, we propose a robust adversarial training (RAT) prototype to relieve the issue of robust overfitting. Specifically, RAT works in each mini-batch: it optimizes the front layers as usual, and for the latter layers. implements additional measures on these parameters to regularize their optimization. It is a general adversarial training prototype, where the front and latter network layers can be separated by some simple test experiments, and the implementation of additional measures to regularize network layer optimization can be versatile. For instance, we designed two representative methods for the realizations RAT: RATLR and RATwp- They adopt different strategies to hinder weight update, e.g., enlarging the learning rate and weight perturbation, respectively. Extensive experiments show that the proposed RAT prototype effectively eliminates robust overfitting. The contributions of this work are summarized as follows:
We provide the first diagnosis of robust overfitting different network layers, and find that there is a strong connection hetween the optimization of the latter layers and the robust overfitting phenomenon.
Based on the observed properties of robust overfitting, we propose the RAT prototype, which adopts additional measures to regularize the optimization of the latter layers and is tailored to prevent robust overfitting.
We design two different realizations of RAT, with extensive experiments on number of standard benchmarks, verifying its effectiveness.
2
RELATED WORK
2.1 ADVERSARIAL TRAINING
Since the discovery of adversarial examples, there have been many defensive methods attempted to improve the DNN's robustness against such adversaries, such adversarial training (Madry et al.) 2017], defense distillation (Papernot et al., 2016), input denoising (Liao et al. 2018), gradient reg- ularization (Tramer et al. 2018). So far, adversarial training (Madry al. 2017) has proven to be the most effective method. Adversarial training comprises two optimization problems: the inner maxinization and outer minimization. The first constructs adversarial examples by maximizing the loss and the second updates the weight by minimizing the loss on adversarial data. Here, fw is the DNN classifier with weight w, and f(-) is the loss function. d(...) specify the distance between original input data I; and adversarial data xi, which is usually an ln-norm ball such as the l2 and loe -oom balls and is the maximum perturbation allowed.
gAT ((T(W) (w)
min) Σ
max ((u(())) d(a, d(ainn)) Se
2.2 ROBUST GENERALIZATION
An interesting characteristic of deep neutral networks (DNNs) is their ability to generalize well in practice (Belkin et al., 2019). For the standard training setting, is observed that test loss continues to decrease for long periods of training (Nakkiran et al. 2020), thus the common practice is to train DNNs for as long as possible. However, this is no longer the case in adversarial training, which exhibits overfitting behavior the longer the training process (Rice et al. 2020). This phenomenon has been referred to as "robust overfitting" and has shown strong resistance to standard regularization techniques such as 1, la regularization and data augmentation methods. (Rice et al. (2020)
Schmidt et al. 2018) theorizes that robust generalization have a large sample complexity, which requires substantially larger dataset. Many subsequent works have empirically validated such claim,
2

---

Under review as a conference paper at ICLR 2023
such AT with semi-supervised learning (Carmon et al. 2019 Zhai al. 2019), robust local fea- ture (Song et al. 2020) and data interpolation (Lee et al. |020; Chen et al. 2021). (Chen et al. 2020) proposes to combine smoothing the logits via self-training and smoothing the weight via stochas- tic weight averaging to mitigate robust overfitring. [Wu et al. (2020) emphasizes the connection of weight loss landscape and robust generalization gap, and suggests injecting the adversarial perturba- tions into both inputs and weights during AT to regularize the flatness of weight loss landscape. The intriguing property of robust overfitting has motivated great amount of study and investigation, but current works typically approach the phenomenon considering a DNN as a whole. In contrast, our work treats DNN as series of layers and reveals a strong connection between robust overfitting and the optimization of the latter layers, providing a novel perspective into better understanding the phenomenon.
3 INTRIGUING PROPERTIES OF ROBUST OVERFITTING
In this section, we first investigate the layer-wise properties of robust overfitting by fixing model parameters in AT (Section [.T) Based on our observations, we further propose a robust adversar- ial training (RAT) prototype to eliminate robust overfitting (Section (.2). Finally, we design two
different realizations for RAT to verify the effectiveness of the proposed method (Section B.3
3.1 LAYER-WISK WISE ANALYSIS OF ROBUST OVERFITTING
Current works usually study the robust overfitting phenomenon considering the network as a single unit. However, features computed by different layers exhibit different properties, such as first-layer features are general and last-layer features are specific (Yosinski et al.) 2014). We hypothesize that different network layers have different effects robust overfitting. To empirically verify the above hypothesis, we deliberately fix the parameters of the selected network layers, leaving them unoptimized during AT and observe the behavior of robust overfitting accordingly. Specifically, considered ResNet-18 8 architecture as a composition of 4 main layers, corresponding to 4 Residual blocks. We then train multiple PreAct ResNet-18 88 networks on CIFAR-10 for 200 epochs using AT, each time selecting a set of network layers to have their parameter fixed.
The robust test performance in figure a shows a consistent pattern. Robust overfitting is mitigated whenever we fix the parameters for layer 4 during AT, while any settings that do not fix the param- eters for layer 4 result in more severe gap between the best accuracy and the accuracy at the last epoch. For example, for settings such as AT- AT-fix-param- param-[4], AT- AT--ix-parmm-- param-[1,4], AT-tix-param-Be and (atix-parammm3,4p1 robust overfitting is significantly reduced. On the other hand, for settings such -i-fix-param-1 various set of layers but allow for the optimization of layer 4, robust overfitting still widely exists. For extreme case like AT-fix-param-[1. [1,2,3, where we fix the first three front layers and only allow for the optimization of that last layer 4, the gap between the best accuracy and the last accuracy is still obvious. This clearly indicates that the optimization of the latter layers present strong cor- relation to the robust overfitting phenomenon. Note that this relationship can be observed across a variety of datasets, model architectures, and threat models (shown in Appendix A), indicating that it is a general property in adversarial training.
-arii--arrmm11.3 and (1-fix-paramm(2233 when we fix the parameters of
In many of these settings, robust overfitting is mitigated at the cost of robust accuracy. For example in (3-fix-param--3.44, if we leave both layer 3 & 4 unoptimized, robust overfitting will practically disappear, but the peak performance is much worse compared standard AT. When carefully ex- amining the training performance in these settings shown in figure [b), we generally observe that the network capacity to fit adversarial data is strong when we fix the parameters for the front layers, but it gradually gets weaker as we try to fix the latter layers. For instance, AT-fix fix-param--l [1] has the highest train robust accuracy, then comes AT-fix-param[2], AT-fix [arrmm]) and AT-fix- fix-paamm[[); Xl-fix-param[! [,233] has higher training accuracy than AT-fix- paraamm2..m) This suggests fixing the latter layers' parameters can regularize the network better compared to fixing the front layers's 's parameters. In the subsequent sections, we will introduce methods that specifically regularize the optimization of the latter layers, so as to mitigate robust overfitting without tradeoffs in robustness. We will compare the impact on robust overfitting when applied such methods on the front layers vs the latter layers, further highlighting the importance of the latter layers in relation to robust overfit- ting.
3

---

Under review as a conference paper at ICLR 2023
0.4
0.2
pewm..e (22)
0.2
Pic 22..0
0.2
# 2.
5.2
parc
(1.0 37.41
0.10
120
130
100
Spocts
(a) Robust Test Performance
Jore 12.43
47 parn 12.40
0.6
pem.erm. 12]
Sr 13.23
pacce
0.20
9.2
Spocte
(b) Robust Train Performance
Figure 1: The robust train/test performance of adversarial training with different sets of network layers fixed. (1-xi--param1122 corresponds to fixing the parameters of layers & during AT
3.2 A PROTOTYPE OF RAT
As witnessed in Section 3.1 the optimization of AT in the latter layers is highly correlated to the existence of robust overfitting. To address this, we propose to train the network adversarial data with some restrictions put onto the optimization of the latter layers, dubbed as Robust Adversarial Training (RAT). RAT adopts additional measures regularize the optimization of the latter layers, and ensures that robust overfitting will not occur.
The RAT prototype is given Algorithm [ It runs as follows. We start with base adversarial training algorithm A. In Line 1-3, The inner maximization pass aims to maximize the loss via cre- ating adversarial examples, and then the outer minimization pass updates the weight by minimizing the loss on adversarial data. Line 4 initiates loop through all parts of the weight w from the front layers to the latter layers. Line 5-9 then manipulate different parts of the weight based on its layer conditions. If the parts of the weight belong to the front layers (Erront). they will be kept intact. Otherwise, a weight update scheme G is put onto the parts of the weight corresponding to the latter layers (Clatter). The role of & is to apply the optimizer updates the model fu in Line 11.
regularization on the latter layers' weight. Finally,
Note that RAT is general prototype where layer conditions Crontt Clatter and weight adjustment strategy 6 can be versatile. For example, based on the observations in Section [3.] we treat the composition of 4 main layers, corresponding Crron indicates layer 1 & 2 and Clatter indicates layer 3 & 4. 6 can also represent various strategies that serves regularize the optimization of the latter layers. In the section below, we will propose two different strategies & in the implementations of RAT to demonstrate RAT's effectiveness.
Res-Net architecture
residual blocks, where
3.3 TWo REALIZATIONS OF RAT
In this section, we will propose two different methods to put certain restrictions on the optimization of selected parts of the network, and then investigate the robust overfitting behavior upon applying such method to the front layers vs the latter layers. These methods showcase a clear relation between the optimization of the latter layers and robust generalization gap.
RAT through enlarging learning rate. In standard AT, the sudden increases in robust test perfor- mance appears to be closely related to the drops in the scheduled learning rate decay. We hypothesize
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1 RAT- (in a mini-batch).
Require: base adversarial training algorithm A, optimizer D, network fw. model parameter {Wi, w2,,..., w, training data D = {(r,, 16)), mini-batch B, front and latter layer conditions Cirout and Clatter for fw. gradient adjustment strategy G
1: Sample a mini-batch = [ y.)) from D
2: B' A.inner maximization B)
3: V + A.outer minimization (fw, Lg)
4: for i =1=1,... -,..., n. do
5:
if Efront then
6:
Va:; V o/;
7: else if Clatter (W)) then
8:
Va. 6 6f Vw,)
# adjust gradient
9:
end if
end for
11:
D.step(V...
that training AT without learning rate decays is sub-optimal, which can regularize the learning pro- cess of adversarial training. Comparison of the train/test performance between standard AT and AT without learning rate decay (AT--ix-Ir-11,23344
are shown in figure 2b.. Training performance of standard AT accelerates quickly right after the first learning rate drop. expanding the generalization gap with further training, whereas for AT without learning rate decay, training performance increases slowly and maintain a stable generalization gap. This suggests that AT optimized without learning rate decay has less capacity to fit adversarial data, and thus provides the regularization needed to relieve robust overfitting. As our previous analysis suggests that the optimization the latter layers is more important in mitigating robust overfitting, we propose using fixed learning rate 0.1 for optimizing the latter parts of the network while applying the piecewise decay learning rate for the former parts to close the robust generalization gap. We refer this approach as realization of RAT, namely RATLR- Compared to standard AT, RATLR essentially enlarge the weight update step V., along the latter parts of the gradients by 10 the first learning rate decay and 100 at the second decay.
V w/ V/ we
where 77 is the amplification coefficient.
To demonstrate the effectiveness of RATLR. we train multiple PreAct ResNet- 18 networks on CIFAR- 10 for 200 epochs using AT, each time selecting a set of network layers to have their learning rate fixed to 0.1 while maintaining the piece-wise learning rate schedule for other layers. Figure 2[a) validate our proposition. Robust overfitting relieved for all settings that target layers that include layer 4 (AT-fix-lr-(4),, AT-fix-lr-11, [1,,4] AT-fi--r-12,44] etc.) while any settings that fix the learning rate of layers that exclude layer 4 do not reduce robust overfitting. Furthermore, all settings that fix the learning rate for both layer 3 & 4, including AT-fix-Ir-3.44,, AT-fix-1r-11,,44,, AT-fix-Ir-(2.3.4) AT-fix-Ir[11,233, completely eliminate robust overfitting. The observations verify that regulariz- ing the optimization of the latter layers by optimizing those layers without learning rate decays can prevent robust overfitting from occurring. An important observation is that RATLR (AT-fi--Ir-(344) can both overcome robust overfitting and achieve better robust test performance compared to the network using a fixed learning rate for all layers (AT-fix-lr-1) (1-ix- 1,2,3,4)). Examining the training per- formance between these two settings in Egureeec) we find that RAT exhibits a rapid rise both robust and standard training performance immediately after the first learning rate decay similar to standard AT. The training performance of RAT Ln is able benefit from the learning rate decay oc- curring at layer 1 & 2, making notable improvement compared to AT-fix- [1,,-lrr11,,,,,44 [1,4,3.4]. By training layers 3 & without learning rate decays, we specifically put some restrictions on the optimization of only the latter parts of the network heavily responsible for robust overfitting, which can relieve robust overfitting without sacrificing too much performance. The experiment results provide another indication that the latter layers have stronger connections to robust overfitting than the front layers do, and regularizing the optimization of the latter layers from the perspective of learning rate can effectively solve robust overfitting.
5

---

Under review as a conference paper at ICLR 2023
0.150
0.150
2,450
2,450
0.42
0.42
2.425
2,425
33.41
0.350
200
250
0.350
2.250
2.250
130
Spocts
(a) Robust test performance of all settings
1.5
0.593
yanded
0.2
67
0.7
0.403
0422
0.402
0.50
sports
(b) Standard AT vs AT using a fixed learning rate
(c) AT using a fixed learning rate for all layers vs the latter layers
Figure 2: The train/test performance of adversarial training using fixed learning rate for different
sets of network layers. AT-fix-Ir1,22] corresponds to using a fixed learning rate for layers 1 & 2 during AT
RAT through adversarial weight pertubation. We continue to study the impact of different net- work layers to robust overfitting phenomenon from the perspective of adversarial weight perturba- tion (AWP). Wu al. (2020) proposes AWP method to explicitly flatten weight loss landscape, by introducing adversarial perturbations into both inputs and weights during AT:
min max V
Σ
max ((fw+o(z)) y:), ((
(3)
where e is the adversarial weight perturbation generated by maximizing the classification loss:
V Σ
As AWP keeps injecting the worst-case perturbations weight during training, it could also be viewed as means to regularize the optimization of AT. In fact, the training of AWP exhibits negative robust generalization gap, where robust training accuracy racy by large margin, shown in figure 3).. This indicates AWP significant restrictions to the optimization of AT, introducing huge trade-offs training performance. As our previous analysis suggests strong correlation between robust overfitting and the optimization of the latter layers, argue that the capacity to mitigate robust overfitting from AWP is mostly thanks to the perturbations occurring at latter layers' weight. As such, we propose to specifically apply AWP to the latter half of the network, and refer to this method as RAT In essence, RATwP compute the adversarial weight perturbation v; under the layer condition (W;), so that only the parts of the weight along the latter half of the network are perturbed.
short of robust testing accu-
min
max I OJEV
Σ max ((fw+o(z)). d(xna/)se
(5)
U;
V Vwi
Σe.
4.
To prove the effectiveness of RATwP. we train multiple PreAct ResNet-18 networks on CIFAR-10 for 200 epochs using AT, each time selecting of network layers have their weight locally

---

Under review as a conference paper at ICLR 2023
0.315
2.575
0.350
0.550
3.550
5.550
0.35
0.25
2.525
0.525
2.425
2.425
0.00
121
0.00
5.400
svre 12
5.400
0.75
0.75
0.5/5
0.5/5
PL
0.350
200
250
0.350
2.350
2.250
130
Spocts
(a) Robust test performance of all settings
1.5
0.0
Bator NOWETT
20
0.
27 21
0.0
0.0
Spocts
Epcdu
(b) AWP applied to front layers vs latter layers
(c) Robust generalization gap
Figure 3: The train/test performance of adversarial training when applying AWP for different sets of network layers. AT-AWP-[1.2] means only layer 1 & 2 have their weight perturbed using AWP
perturbed using AWP. As seen from figure 3a), There are only settings that can overcome robust overfitting, namely AT-AWP-(3,44 AT-AWP-11,3,41 and AT-AWP-12.3.44, These settings share one key similarity: both layer 3&4 have their weight adversarially perturbed during AT. Simply applying AWP to any set of layers that exclude layers 3&4 is not sufficient to eliminate robust overfitting. This shows that AWP is effective in solving robust overfitting only when applied to both layer 3 and layer 4. Even when AWP is applied to the first 3 former layers out of 4 layers (AT-awp- [1,2,3]), robust overfitting still widely exists. weight perturbations to occur at the latter part of the network in order to mitigate robust overfitting. To examine this phenomenon front layers (represented by AT-AWP-[1,2,3] vs AWP applied to latter layers (represented by AT- AWP-[3,4]), shown in figure b). AWP applied in the front layers have a much better training performance than AWP applied in the latter layers. Furthermore, AWP applied to front layers reveals a positive robust generalization gap (training accuracy > testing accuracy) shortly after the first widen with further training. Conversely, AWP applied
another word,
essential for the adversarial
detail, we compare the training performance of AWP applied to
drop in learning rate, which continues in the latter layers exhibits a negative robust generalization gap throughout most of the training, only converging to after the second drop in learning rate. These differences demonstrate that worst-case perturbations, when injected into the latter layers' weights, have more powerful impact regularizing the optimization of AT. Consistent with our previous findings, AWP applied to the latter layers can be considered as an approach to regularize the optimization of AT in those layers, which successfully mitigates robust overfitting. This finding supports our analysis thus far, further demonstrating that regularizing the optimization of the latter layers is key to improving the robust generalization.
EXPERIMENT
In this section, we conduct extensive experiments to verify the effectiveness of RATI. i. and RAT Details of the experiment settings and performance evaluation are introduced below.
7

---

Under review as a conference paper at ICLR 2023
Table 1: Test robustness (%) CIFARIO. We omit the standard deviations of 5 runs they very small (< 0.6%)
PGD-20
AA
Network
Threat Model
Method
Best
Last
Diff
Best
Last
Diff
AT
52.31
44,45
7.86
47.95
42.05
5.90
Loo
RATLR
51.57
49.07
2.50
46.89
45.35
1.54
RATwe
54.85
53.98
0.87
49.19
48.24
0.95
PreAct ResNet-18
AT
69.27
65.86
3.41
67.65
64.64
3.01
L2
RATLR
68.97
68.21
0.76
64.26
63,44
0.82
RATwP
70.77
69.49
1.28
68.29
67.11
1.18
AT
55.57
47.37
8.20
52.13
46.09
Loo
RATLR
55.50
47.32
8.18
52.05
45,89
6.16
RATwP
58.92
58.23
0.69
54.46
53.98
0.48
Wide ResNet-34-10
AT
70.57
98.99
1.58
69.44
66.92
2.52
L
RATLR
71.91
68.94
2.96
70.53
67.90
2.63
RATwP
71.31
69.19
2.12
70.12
67.35
2.77
Table 2: Test robustness (%) on CIFAR100. We omit the standard deviations of runs as they are very small (< 0.6%).
PGD-20
AA
Network
Threat Model
Method
Best
Last
Diff
Best
Last
Diff
AT
28.07
21.24
6.83
23.61
18.41
5.20
Loo
RATLR
26.57
26.18
0.39
21.77
21.22
0.55
RATwr
30.91
30.42
0.49
25.52
24.57
1.05
PreAct ResNet-18
41.38
35.34
6.04
37.94
33.58
4.36
L
RAT
38.31
37.76
35.16
34.49
0.77
RATwr
45.23
44.93
0.3
41.32
39.47
1.85
AT
30.74
24.89
5,85
26.98
23.07
3,91
Loo
RATLR
30.57
23.53
7.04
26.72
22.53
4.19
RATwP
30.81
25.46
5.35
27.11
23.56
3.55
Wide ResNet-34-10
AT
44.05
41.22
2.83
41.39
39.34
2.05
L2
RATLR
44.43
40.42
4.01
41.47
39.42
2.05
RATwP
46.12
44.64
1.48
41.94
40.38
1.56
4.1 EXPERIMENTAL SETUP
We conduct extensive experiments two realizations of RAT across three benchmark datasets (CIFARI0 (Krizhessky et al. 2009). SVHN (Netzer et al. 2011) and CIFAR 100 (Krizhevsky et al., )009h) and two threat models (L and Lz). We use PreAct R8 He al. (2016) and Wide ResNet-34-10 following the same hyperparameter settings for AT in Rice et al. (020)) for Loo threat model, e 8/255, step size 1/255 for SVHN, and 2/ 255 for CIFAR-I0 and CIFAR-100: for La threat model, e = 1 128 /255 step size is 15/255 for all datasets. For training, all models are trained under 10-step PGD (PGD-10) attack for 200 epochs using SGD with momentum 0.9, weight decay x 10-4, and a piecewise learning rate schedule with an initial learning rate of 0.1. RAT models are decomposed into series of main layers, corresponding to residual blocks the ResNet architecture. For RATLR. learning rate for layer 3&4 are set to fixed value of 0.1. For RAT leveraging AWP in layer 3&4, 7 1 x 10-2 For testing, the robustness accuracy is evaluated under two different adversarial attacks, including 20-step PGD (PGD-20) and Auto Attack (AA) Croce & Hem (2020b). Auto Attack is considered the most reliable robustness evaluation to date, which is ensemble of complementary attacks, consisting of three white-box attacks (APGD-CE (Croce & Hein| (020b). APGD-DLL (Croce Hemn 20206),, and FAB (Croce & Hein 2020a)) and
black-box attack (Square Attack (Andriushchenko al. 2020
8

---

Under review as a conference paper at ICLR 2023
Table 3: Test robustness (%) on SVHN. We omit the standard deviations of 5 runs they very
small (< 0.6%)
PGD-20
AA
Network
Threat Model
Method
Best
Last
Diff
Best
Last
Diff
AT
53.10
44.12
8.98
45.09
40.36
4.73
L
RATLR
53.32
43,41
9.92
45.98
39.61
6.37
RATwr
57.91
54.32
3.58
50.32
44.82
5.49
PreAct ResNet-18
AT
66.29
64.73
1.55
63.55
60.14
3.41
L2
RATLR
66.47
62.10
4.36
62.44
58.72
3.72
RATwP
71.66
65.68
5.98
65.17
59.64
5.53
AT
55.57
47.11
8.46
48.05
42.46
5.59
Loo
RATLR
55.34
46.81
8.53
47.94
42.12
5.82
RATwP
58.48
54.92
3.56
54.65
50.46
3.99
Wide ResNet-34-10
AT
67.19
65.08
62.58
60.86
1.72
L2
RATLR
67.50
64.24
3.27
62.79
59.94
2.85
RATwP
69.07
64.76
4.31
63.12
59.57
3.55
4.2 PERFORMANCE EVALUATION
In this section, we present the experimental results of RATLR and RATwP across three benchmark datasets
The evaluation results on CIFARIO dataset are summarized in Table ! where "Best" is the highest test robustness achieved during training; "Last" is the test robustness at the last epoch checkpoint; "Diff" denotes the robust accuracy gap between the "Best" "Last". is observed that RATwP generally achieves the best robust performance compared to RATLR & standard AT. Regardless, both RATLR and RATwp tighten the robustness gaps by a significant margin, indicating they effectively suppress robust overfitting.
CIFARIO Results.
CIFAR100 Results. We also show the results on CIFARION dataset in Table We observe similar performance like CIFARIO. where both RATLR and RATwp is able to significantly reduce the robustness gaps. For robustness improvement, RATwF stands out to be the leading method. The results further verify the effectiveness of the proposed approach.
SVHN Results. Finally, we summarize the results on the SVHN dataset in Table B where robustness gap are also narrowed down to small margin by RAT SVHN dataset is special case where RATER strategy does not improve robust overfitting. Unlike CIFAR10 10 and CIFAR100, learning rate decay in SVHN's training does not have much connection the sudden increases in robust test performance or the prevalence of robust overfitting, and hence makes RATLR ineffective. Other than this, The improvement in robust generalization gaps can be witnessed in all cases, demonstrating the proposed approachs are generic and can be applied widely.
CONCLUSION
In this paper, we investigate the effects different network layers on robust overfitting and identify that robust overfitting is mainly driven by the optimization occurred at the latter layers. Following this, we propose robust adversarial training (RAT) prototype to specifically hinder the optimiza- tion of the latter layers in the process of training adversarial network. The approach prevents the model from overfitting the latter parts of the network, which effectively eliminate robust overfitting of the network as whole. We then further demonstrate two implementations of RAT: one locally uses a fixed learning rate for the latter layers and the other utilize adversarial weight perturbation for the latter layers. Extensive experiments show the effectiveness of both approaches, suggesting RAT generic and can be applied across different network architectures, threat models and benchmark datasets solve robust overfitting.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: a query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision, pp. 484-501. Springer, 2020.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense curity: Circumventing defenses to adversarial examples. In International conference on machine learning, pp. 274-283, PMLR, 2018.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical blas-variane trade-off. Proceedings of the National Academy of Sciences, 116632):55899-15588 2019.
Yair Carmon, Aditi Raghumathan, Ludwig Schmidt, Percy Liang, and John Duchi. Unlabeled data improves adversarial robustness. arXiv preprint arXiv:/905. 1et::9905.133333 /3736, 2019.
Chen Chen, Jingfeng Zhang, Xilie Xu, Tianlei Hu, Gang Niu, Gang Chen, and Masashi Sugiyama. Guided interpolation for adversarial training. arXiv preprint arXiv:2102.07327, 2021.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening. In International Conference on Learning Representations, 2020.
Francesco Croce and Matthias Hein, Minimally distorted adversarial examples with fast adaptive boundary attack. International Conference on Machine Learning, pp. 2196-2205. PMLR, 2020a.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference machine learning, pp. 2206- 2216. PMLR, 2020b.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 1rXiv:1412.6577 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Saehyung Lee, Hyungyu Lee, and Sungroh Yoon. Adversarial vertex mixup: Toward better adver- sarially robust generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 272-281, 2020.
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1778-1787, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks, arXiv preprint 2017.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang. Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference on Leamn- ing Representations, 2020. URL Int http///ppeeeeiiieeeeee wete:/oorumidddllgsaae
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 201 1.
10

---

Under review as a conference paper at ICLR 2023
Nicolas Paperno, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP), pp. 582-597. IEEE, 2016.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pp. 8093-8104, PMLR, 2020.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Ad- versarially robust generalization requires more data. Advances in Neural Information Processing Systems, 11:5014-5022 2018.
Chubiao Song, Kun He, Jiadong Lin, Liwei Wang, and John E Hopcroft. Robust local features for improving the generalization of adversarial training. In International Conference on Learning Representations, 2020.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 13X.v::3112.1/// 2013.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In International Conference Learning Representations, 2018.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust gener- alization. arXiv preprint arXiv:2004.05888 2020.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He, John Hopcroft, and Liwei Wang. Adversarially robust generalization just requires more unlabeled data, arXiv preprint 1rXiv:1906.0055: 2019.
0.30
0.30
2.30
0.20
0.20
0.20
0.20
0.20
0.22
0.22
0.22
0.20
0.20
0.20
0.20
0.10
6.30
13.44
0.30
3.30
Spoct'
(a) CIFAR100
6.55
0.75
3.55
0.55
0.30
0.30
2.30
2.30
0.15
0.15
3.15
5.45
0.30
2.30
5p007s
(b) SVHN
Figure 4: Robust test performance of adversarial training using fixed learning rate for different sets of network layers, across datasets (CIFAR- 100 and SVHN) under Loo threat
11

---

Under review as a conference paper at ICLR 2023
0.35
0.35
0.35
0.30
0.30
0.30
0.23
121
0.23
2.25
1irv 12
0.13
0.00
2.20
0.30
Spocta
(a) CIFAR100
0.40
0.40
3.40
0.40
6.55
0.55
3.55
0.55
0.45
0.45
3.45
5.45
0.40
0.40
2.40
2.40
0:35
2.
0:35
0.35
0.36
0.30
2.re 1X]
0:30
3.30
5.30
0.25
0.25
0.05
9.45
Spocts
(b) SVHN
Figure 5: Robust test performance of adversarial training applying AWP for different sets of network layers, across datasets (CIFAR-10) 100 and SVHN) under (o0 threat
A MORE EVIDENCES FOR THE LAYER-WISE PROPERTIES OF ROBUST OVERFITTING
In this section, provide empirical experiments showcase the layer-wise properties of robust overfitting across different datasets, model architectures and threat models. Specifically, we two strategies mentioned in Section 3.3 to put restriction the optimization of different network layers. We can always observe that there is no robust overfitting when we regularize the optimization of layers and 4 (the latter layers), while robust overfitting is prevalent for other settings. These evidences further highlight the strong relation between robust overfitting and the optimization of the latter layers.
A. EVIDENCES ACROSS DATASETS
We show that the layer-wiss properties of robust overfitting is universal across datasets CIFAR- 100 and SVHN. We adversarially train PreAct ResNet-18 18 under lo threat model on different datasets with the same settings as Section 3.3 The results shown in Figure and Note that for SVHN, regularization strategy utilizing a fixed learning rate (RAT LA) for does not improve robust overfitting (Figure 4p. Unlike CIFAR10 and CIFAR 100, SVHN's training overfits way before the first learning rate decay. Also, learning rate decay in SVHN'S training does not have any relation to the sudden increases in robust test performance
the appearance of robust overfitting. Hence, SVHN dataset is special case where RATLR does not apply. For all other cases, robust overfitting is effectively eliminated by regularizing the optimization of layers and 4.
A.2 EVIDENCES ACROSS THREEA MODELS
We further demonstrate that the generality of wise properties of robust overfitting by conduct- ing experiments under la threat model across datasets. The settings are the same as Section 3.3 The results are shown in Figure 6 and 7 Under l2 threat model, except for SVHN dataset where regularization strategy utilizing a fixed learning rate (RATTR) does not apply, robust overfitting is effectively eliminated by regularizing the optimization of layers 3 and 4.
12

---

Under review as a conference paper at ICLR 2023
0.72
0.22
0.22
5.72
0.00
0.00
0.00
5.10
0.00
0.80
0.00
0.00
0.60
0.66
5.66
0.6
0.64
5.64
0,22
0.82
0.2
5.82
0.00
0.00
5.00
0.50
0.50
0.50
0.50
1:34
0.30
0.50
2.30
201
200
200
CIFARIO
0.400
0.400
2,400
2,400
0.375
0.275
P3 231
0.275
2.275
0.275
50
300
350
200
50
300
350
200
50
302
150
300
50
102
130
300
(b) CIFARI00
0.050
0.050
2.050
0.050
0.625
0.625
5.425
6.425
Hinded
0.22
0,25
2.525
5,92
25
50
15
200
:15
150
25
50
75
225
150
25
30
75
120
325
250
43
30
75
320
445
250
Tpeete
(c) SVHN
Figure 6: Robust test performance of adversarial training using a fixed learning rate for di different sets of network layers, across datasets (CIFAR-10 10, CIFAR- 100 and SVHN) under ly threat
13

---

Under review as a conference paper at ICLR 2023
0.15
5.15
0.70
0.70
2.70
2.70
BomAda Wep Prinay
0.45
0.65
5.45
5.45
0.00
0.00
2.50
5.00
0.55
0.55
5.55
0.55
21
(31
22
100
150
200
0.50
22
155
200
22
1.00
153
300
53
100
150
200
CIFARIO
0.45
2.45
2.45
0.40
0.40
2.40
2.40
0.35
0.25
2.35
0.35
D1
(1
32
1.00
150
200
32
2000
150
200
32
300
150
300
50
100
150
300
(b) CIFARI00
0.00
0.80
2.50
2.00
0.55
0.25
3.55
3.55
FLows
0.50
25
30
73
120
122
350
0.5
23
10
75
100
122
150
23
30
100
12
230
23
35
73
303
250
Tpeete
(c) SVHN
Figure 7: Robust test performance of adversarial training applying AWP for different sets of network layers, across datasets (CIFAR1O, CIFAR- 100 and SVHN) under ly threat
14


Reviews:
[{"review_id": "xS7lN_7OiAo", "reviewer": "Reviewer_VqAU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates robust overfitting, an important phenomenon that occurs in the adversarial training. In contrast to previous approach, the paper does not regard the neural network as a black-box model but divide a DNN into a series of layers and investigate the effect of different network layers on robust overfitting. It finds that the latter layer has larger impact on the robust overfitting. To this end, the paper proposes two regularization methods on the latter layers of the networks, namely $RAT_{LR}$ and $RAT_{AWP}$, and shows that two these methods help to mitigate robust overfitting.", "summary_of_the_review": "Although the findings and directions of paper is novel, I think the main claim of the paper is not well supported. ", "strength_and_weaknesses": "Strength\n\n1. The paper studies an important problem of robust overfitting in the adversarial training. It is highly motivated to divide the DNN into several layers instead of regarding it as a black-box model.\n\n2. The experiments are intensive, showing that the proposed methods, RAT, help to mitigate robust overfitting.\n\nWeakness\n\n1. My major concern is about the number of parameters that the paper fix in the experiments of Section 3. Different blocks have different number of parameters. Therefore, fixing different blocks will result in the models with different number of effective parameters so that they will have different capacity. If the model that fixs the latter part has smaller number of parameters than the model that fixs the former part, it cannot prove the latter part has larger impact on the robust overfitting. Instead, the model capacity would be more important. In addition, adding more analyses beyond the architecture of ResNet would help to strengthen the conclusion.\n\n2. As mentioned by the paper, AWP helps to mitigate robust overfitting. What is the performance of applying AWP to all layers? If this one has better performance than $RAT_{AWP}$, then the proposed method seems to have small novelty.\n\n3. In the experiments of Section 4, as AWP is shown to improve the performance. $RAT_{AWP}$ should not only be compared with AT with AWP as well.\n\n4. In the experiment of Section 4, the final accuracy of $RAT_{LR}$ is lower than AT in many cases. Therefore, the empircal application of this method seems very limited.\n\nMinors\n\nHow are the last fully connected layers treated in the experiments? Are they fixed?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "sOU8nbd7R7", "reviewer": "Reviewer_2pex", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors study the phenomenon of robust overfitting during adversarial training (AT) of deep neural networks — specifically about the deviations that arise due to optimization at different layers of these networks. The paper demonstrates that if the deeper layers of the network are frozen or optimized with a lower learning rate during AT, robust overfitting is significantly reduced, albeit the accompaniment of reduction in test robustness as well, in contrast to that seen if the same is performed with the earlier layers. A similar phenomenon is seen with the Adversarial Weight Perturbation (AWP) regularizer applied to different layers, where it is seen that latter layers contribute more to robust overfitting. ", "summary_of_the_review": "As mentioned in the weaknesses section, the contributions of the paper are fairly limited, and the proposed method thus does not present significant benefits over the prior work AWP, which this paper seeks to incorporate. While the overall idea is well presented, the empirical evaluations could be improved to better understand the phenomenon of robust overfitting. ", "strength_and_weaknesses": "Strengths:\n1) The paper is well-written, and presents a sequence of empirical evaluations in a clear, concise manner to help understand the effect of optimization of different layers on robust overfitting. \n2) The proposed methods RAT-LR and RAT-AWP are fairly simple: RAT-LR applies a fixed learning rate to the deeper layers, while RAT-AWP in a similar vein applies AWP solely to the deeper layers, to impose additional regularization for the latter part of the network alone.\n\n\nWeaknesses:\n1) The contributions and novelty of the techniques presented are fairly limited, given that the two primary methods proposed are to either keep learning rate fixed or apply AWP for latter layers, instead of applying the same to the entire network as a whole. Furthermore, the deeper layers/blocks are known to have many more parameters (especially for ResNet based models), thus it is not entirely surprising that the latter layers contribute more to overfitting. This aspect could have been analyzed in much more detail in the paper, to study of depth vs parameter count on robust overfitting. \n2) The analysis presented is also fairly focussed to the very specific case of step-decay of learning rate as well, when in practice several recent works on robust defenses utilize other learning rate schedules such as cyclic, cosine, annealed cosine etc. Furthermore, the effect of the magnitude of the learning rate drop, the spacing of drop points, effect on deeper layers, only FC layer etc. could have been studied. Since the paper does not present theoretical viewpoints of the problem (which is certainly fine on its own), the extent of empirical study could have been significantly broadened to better understand robust overfitting in slightly more generalized settings.   \n3) This is further reinforced given the fact that AWP based regularization on its own is seen to drastically reduce robust overfitting, and appears to be the best performing method as seen from Figure 3 (a) and 3(c). Thus the contributions of the paper become slightly unclear in this setting, over and above AWP based training alone.\n4) Given the excellent performance of AWP-AT and its effect in mitigating robust overfitting, the baseline AWP method needs to be included in the main empirical evaluations as presented in Tables-1,2, since it is well known that far out-performs the standard AT baseline presented. From the original AWP paper, WideResNet-34-10 models trained on CIFAR-10 with AWP-AT achieves 54.04% and AWP-TRADES achieves 56.17% AutoAttack accuracy, while the proposed RAT-AWP-AT achieves 54.46%. Thus, the improvements in robust performance, or noteworthy contributions over the AWP method  are not clearly seen.\n5) Further, the clean accuracy of models need to be reported alongside the robust accuracies in the same tables, due to the well-known robustness-accuracy tradeoff. Thus, it is difficult to judge if a given method with slightly higher robust accuracy is inherently better, if it is accompanied by a large decrease in clean accuracy.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "X-8s5ul_Yr", "reviewer": "Reviewer_jxMW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the intriguing layerwise properties of robustness overfitting and reveals that adversarial overfitting typically happens in the deep layers of networks. Based on the observations, two regularizations are utilized for the latter layers to tackle the overfitting. Extensive experiments with different datasets, attack methods, and model architectures verify the proposed method.\n", "summary_of_the_review": "The analysis of this paper is rigorous and well-supported. I would lean to accept. If the authors can offer more comparisons with SOTA methods on addressing adversarial overfitting, I would like to increase my score further. Releasing code can also be a bonus.", "strength_and_weaknesses": "Pros:\n\n1. This paper is well-written and easy to follow.\n\n2. An in-depth analysis is performed to support the hypothesis.\n\n3. Significant performance improvement in many settings.\n\nCons:\n\n1. It would be better if the authors could offer some comparison with SOTA methods.\n\nQuestion:\n\n1. In Section 3.1, for AT-fix-param-[XXX], does fixing mean the parameters are fixed as random initialization? And do you still update the fc layer?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "NJxN92vUlM", "reviewer": "Reviewer_8P5a", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the impact of different DNN layers impact the performance of adversarial training (AT). The ablation experiments reveal that the latter (deeper) layers are more influential to the robust generalization gap and the final performance. Two techniques were then proposed to improve the performance of standard AT method via adaptive learning rate or adaptive weight perturbation.", "summary_of_the_review": "An interesting finding for adversarial training, and is confirmed from different perspectives by different techniques. However, the findings need deeper analysis and the robustness of the proposed mitigation methods needs a complete comparison. ", "strength_and_weaknesses": "Strengths:\n1. The student of an interesting phenomenon, robust overfitting, in adversarial training.\n2. The key findings are interesting and appear quite new to me.\n3. The two proposed defenses are validated on multiple datasets.\n\nWeaknesses:\n1. Missing important analysis and discussion to existing work [1], where a detailed grid search was applied to WideResNet to find that the deeper layers are also more impactful to the final performance of AT. The conclusion is the same as in this paper. And they find that this is because the deep layers are overparameterized and can be simply reduced to mitigate the robust overfitting. This work did a similar analysis as in [1] but using different techniques, i.e., training or not training, hyperparameter ablation, adaptive lr etc. An in-depth analysis should be conducted to connect this work to [1].\n\n2. Incomplete robustness evaluation with different AT methods. In table 1/2/3, the standard AT is compared with the proposed method. It is thus not clear to me whether other AT methods like TRADES, MART, and AWP are also suffering from robust overfitting, and how the proposed method performs with these AT methods. How the proposed techniques work with data augmentation strategies in [2]. \n\n3. Why latter layers are so special? Any theoretical insights or an in-depth analysis? How can we solve this issue completely? Is it guaranteed to improve the robust accuracy if the latter layers are treated differently, can it improve the current SOTA robustness shown in https://robustbench.github.io/\n\n\n[1] Huang, Hanxun, et al. \"Exploring architectural ingredients of adversarially robust deep neural networks.\" Advances in Neural Information Processing Systems 34 (2021): 5545-5559.\n[2] Rebuffi, Sylvestre-Alvise, et al. \"Fixing data augmentation to improve adversarial robustness.\" arXiv preprint arXiv:2103.01946 (2021).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject