Fantastic Robustness Measures: The Secrets of Robust Generalization
Authors: Hoki Kim
Jinseong Park
Yujin Choi
Jaewook Lee
Venue: NeurIPS 2023 poster (2023)
Keywords: Adversarial Robustness
Generalization
Measures
TL;DR: This study conducts a large-scale analysis over 1,300 models to investigate when and how existing measures (e.g., margin, smoothness, flatness) are effective in estimating robust generalization.

Abstract: Adversarial training has become the de-facto standard method for improving the robustness of models against adversarial examples. However, robust overfitting remains a significant challenge, leading to a large gap between the robustness on the training and test datasets. To understand and improve robust generalization, various measures have been developed, including margin, smoothness, and flatness-based measures. In this study, we present a large-scale analysis of robust generalization to empirically verify whether the relationship between these measures and robust generalization remains valid in diverse settings. We demonstrate when and how these measures effectively capture the robust generalization gap by comparing over 1,300 models trained on CIFAR-10 under the $L_\infty$ norm and further validate our findings through an evaluation of more than 100 models from RobustBench across CIFAR-10, CIFAR-100, and ImageNet. We hope this work can help the community better understand adversarial robustness and motivate the development of more robust defense methods against adversarial attacks.

Fantastic Robustness Measures: The Secrets of Robust Generalization
Hoki Kim Scoul National University ghr196130smu. kr
Jinseong Park Scoul National University iinsonggenu..at ac..r
Yujin Choi Seoul National University uznhigh@smu.c ac.kr
Jaewook Lee Seoul National University j jaewookdemu. ac kr
Abstract
Adversarial training has become the de-facto standard method for improving the robustness of models against adversarial examples. However, robust overfitting remains significant challenge, leading to a large gap between the robustness on the training and test datasets. To understand and improve robust generalization, various measures have been developed, including margin, smoothness, and flatness based measures. In this study, we present large-scale analysis of robust generalization to empirically verify whether the relationship between these measures and robust generalization remains valid in diverse settings. We demonstrate when and how these measures effectively capture the robust generalization gap by comparing over 1,300 models trained on CIFAR-10 under the Teo norm and further validate findings through an evaluation of more than 100 models from RobustBench [12] across CIFAR-10, CIFAR-100, and ImageNet. We hope this work can help the community better understand adversarial robustness and motivate the development of more robust defense methods against adversarial attacks.
1 Introduction
Deep neural networks have achieved tremendous success in various domains, but their vulnerability to subtle perturbations has been revealed through the existence of adversarial examples, which are not generally perceptible to human beings [57] 200. To obtain robustness against these adversarial examples, numerous defense methods have been proposed, and among them, adversarial training has become a common algorithm because of its effectiveness and ease implementation 42
71 61.
However, researchers have recently found that adversarial training methods also suffer from the problem of overfitting [63 52), where an adversarially trained model shows high robustness on training examples, yet significantly reduced robustness on test examples. As this robust overfitting progresses, the robust generalization gap increases, resulting in poor robustness for unseen examples.
To prevent robust overfitting and achieve high robust generalization, researchers have analyzed the properties of adversarial training and demonstrated the usefulness of some measures, such as margin- based measures, flatness- based measures, and gradient-based measures [53] 67 66 [56]. Researchers have used them to estimate the robust generalization gap of given models and further developed new training schemes for improving its robustness 44 66). While these measures offer significant insights into robust generalization, we find that the evaluation of some measures is often limited due to the lack of models or training setups. These limitations can potentially lead misleading conclusions, which may include inaccurate estimations of the robust generalization gap and misguided directions for the further development of adversarial training methods.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Therefore, to gain a more precise understanding of when and how these measures correlate with robust generalization, we train over 1,300 models on CIFAR-10 10 under the L norm across various training settings. We then investigate the relationships between a wide range of measures and their robust generalization gap. To further validate our findings, we also analyze over 100 models provided in RobustBench [12] across CIFAR-10, 10, CIFAR-100, and ImageNet. Based on our large-scale study, we summarize our key findings as follows:
Key findings.
1. Due the high sensitivity of the robust generalization gap to different training setups, the expectation of rank correlation across a wide range of training setups leads to high variance and may not capture the underlying trend.
2. Margin and smoothness exhibit significant negative correlations with the robust generaliza- tion gap. This suggests that, beyond a certain threshold, maximizing margin and minimizing smoothness can lead to degradation in robust generalization performance.
3. Flatness- measures, such as estimated sharpness, tend to exhibit poor correlations with the robust generalization gap. Rather, contrary to conventional assumptions, models with sharper minima can actually result in a lower robust generalization gap.
4. The norm of the input gradients consistently and effectively captures the robust generalization gap, even across diverse conditions, including fixed training methods and when conditioned on average cross-entropy.
To promote reproducibility and transparency in the field of deep learning, we have integrated the code used in this paper, along with pre-trained models, accessible to the public at https We hope that our findings and codes can help the community
//github. //githuub.cmm/aarryga.... hwt./oarry2///AAaA.a.. better understand adversarial robustness and motivate the development of more robust defense methods against adversarial attacks.
Related Work
The primary distinction between standard and adversarial training is that adversarial training aims to correctly classify not only benign examples but also adversarial examples as follows:
min Ia =allse
max
(x ,n).
where (xr y) is drawn from the training dataset and represents the model with trainable parameters w. Among the distance metrics || ||. in this paper, we focus robustness with respect to the Loo. norm. Note that the loss function L can also include (ax, w) to minimize the loss with respect to æ. By optimizing I we hope to minimize the robust error on the true distribution D, defined as:
E(un; e, D) ()
max [agadd -2e 10
(f w) v)
where 16 * y) is an indicator function that outputs 0 if the prediction I is same as the true label y. and otherwise.
The majority of researches have focused on optimizing | through the development of new loss
functions or training attacks. For instance, vanilla adversarial training (AT) 42 minimizes the loss of adversarial examples generated by projected gradient descent (PGD)
with multiple
iterations. Following AT, several variations, such as TRADES 771 and MART [61] have achieved significant reductions in robust errors on various datasets through the adoption of KL divergence and the regularization probability margins, based theoretical and empirical analyses.
However, recent works have revealed that the adversarial training framework has a challenging generalization problem, characterized by higher Rademacher complexity (68) and larger sample complexity [54]. The overfitting problem in adversarial training has been observed as a common phenomenon across various settings [52] and it can even occur in a more catastrophic manner during single-step adversarial training 63 As robust overfitting progresses, the following robust generalization gap g( ((w) increases,
((w) E(w; e, D) e(w,e,))
(3)

---

Thus, in order to minimize (w:; e, D), we should not only focus on minimizing the training objective function E(w; e, S) but also on reducing the robust generalization gap (
To gain a deeper understanding of the robust overfitting and further reduce the robust generalization gap g( ()) line of work has theoretically and empirically investigated measures, such as boundary thickness [67]. local Lipschitzness [66], and flatness [56]. While these studies claim that these mea- sures are reliable indicators of the robust generalization gap, the lack of consistency in experimental settings hinders us to identify their validity in practical scenarios. Therefore, in this work, we aim to investigate the clear relationship between these measures and the robust generalization gap by conducting comprehensive analysis using large set of models.
2.1 Comparison to Jiang et al. 22
The pioneering study [27 explored the empirical correlations between complexity measures and generalization with primary focus on the standard training framework. Our main contribution is delving into the realm of measures within the adversarial training framework-a context having different generalization tendencies from those the standard training framework [53 66). Indeed,
we observe that the metric th proposed in 2 has limitations in capturing the effectiveness of measures due to the high sensitivity of the robust generalization gap with respect to training setups. By introducing ness measures correlate with robust generalization. Furthermore, while Jiang et al. [27] employed customized parameter-ethe -effccceet neural networks, adopt widely-used model architectures such as ResNets, thereby providing insights that are not only relevant to recent research but also offer more practical implications.
metric Th, our work enhances the understanding of when and how robust-
Experimental Methodology
In the adversarial training framework, measures have played a crucial role by providing either theoretical upper bounds empirical correlations with robust generalization. Previous works have leveraged these measures to propose new training schemes [6] 67 64] and suggested directions to achieve high robustness [66 (6). However, discover that certain limitations and confusions exist when extending the findings of prior works to practical scenarios due to the use of a restricted set of models and training setups [67] 64 66). In order to gain comprehensive understanding of the true efficacy of these measures, it crucial to validate whether the effectiveness of measures remains valid in practical settings.
To address these challenges, our work aims to provide a comprehensive and accurate assessment of the effecttiveness of measures for robust generalization in practical settings. Our objective is to address the fundamental question:
Do measures remain effective in correlating with robust generalization in practical settings? If so, how and when are measures correlated with robust generalization?
To this end, in Sections 3.1 and and 3.2 we carefully construct the training space that considers practical scenarios within the adversarial training framework and gather a wide range of measures from previous works. In Section (3.3 we define the evaluation metrics and introduce specific variations to accurately capture the correlation between measures and robust generalization in practical settings.
3.1 Training Space
In the realm of adversarial training, various training procedures have been extensively explored to enhance adversarial robustness based on the development of AT, TRADES, and MART. Recently, to resolve the issue of robust overfitting, researchers have begun combining additional techniques, including commonly employed in the standard training framework, such as early-stopping [52], using additional data [9 (22,, manipulating training tricks (48 [1]. and adopting sharpness-aware minimization [64]. By integrating these techniques into AT, TRADES, and MART, high adversarial robustness have been achieved, outperforming other variants of adversarial training methods (21).
Based on these prior works, to mimic practical scenarios, we have carefully selected eight training parameters widely used for improving robust generalization: (1) Model architecture (ResNet18 23
3

---

Robust Genera wwl.zaattoonn Gap (%)
1e
60
86
WRN28-10
Test Robustness (%)
WRN34-10
Training Setup
Standard
Comerallzaddo
AT
40
TRADES
MART
30
SGD
20
20
AWP
FageFalse
10
Aug=True
Extra-data-False Extra-data=
40
80
100
0
10
20
30
40
50
60
70
Train Robustness (%)
Robust Generalization Gap (%)
Figure I: (Left) Scatter plot of train robustness and test robustness. Bright color corresponds high robust generalization gap, i.c., poor generalization. (Right) Boxplot of robust generalization gap for some training setups. All adversarial examples during training and testing are generated by PGDIO on CIFAR-10.
WRN28-10 70), WRN34-10 (70).. (2) Training methods |Standard,, AT 42 TRADES 7 MART
(1 (3) Inner maximization steps (1, 10), (4) Optimizer (SGD, AWP 64 ), (5) Batch-size (32, 64,
128, 256), (6) Data augmentation (No Augmentation, Use crop and flip). (7) Extra-data (No extra
data, Use extra data [)) and (8) Early-stopping (No early-stopping, Use early-stopping). Additional
training details can be found in Appendix C providing comprehensive overview of the training procedures.
In total, 1,344 models were trained using the CIFAR-I0 dataset with = 8/255. Given these models, evaluate their train/test robustness against PGD with 10 iterations (denoted as PGD10). While we acknowledge the existence of stronger adversarial attacks, such as AutoAttack [T]. we primarily use PGDI0 10 due to its prevalent in adversarial training and the high computational demands of Attact Additionally, considering the usage of PGD in calculating specific measures, such boundary thi ckness and local 1ip, ensures consistency in analysis. For detailed discussion, please refer to the Appendix A.44
The statistics of the trained models are summarized in Figure 1. In the left plot, we can observe that the selected range of training parameters generates a diverse set of models, exhibiting robust generalization gaps ranging from 0% to 70%. Notably, certain models achieve 100% robustness on training data against PGD10, but their maximum robustness on the test set is only 65%, highlighting the importance of robust generalization gap. The right plot is boxplot shows the distribution of robust generalization gaps for some training setups. As described in prior works 9 52 64), each
training setup has significant impact robust generalization.
3.2 Measures
Beyond the measures proposed under the adversarial training frameworks 67 66]. previous works [7 have demonstrated that certain measures can effectively capture the generalization gap under standard training framework. Therefore, this paper, we have gathered diverse measures from both the standard and adversarial training frameworks and categorized them into five different types based on their origins and formulas: (i) weight- norm 7
9 10 (ii) margin I2
(5). 23 We denote
I3
44
(ii)) smoothness (16) 17). (iv) flatness [8
19
20 21
and (v) gradient-norm 22
the chosen measures in teletype font (c.g., path norm). While we briefly introduce the concepts
of measures in each paragraph in Section we refer the readers Appendix B for the details of measures including their mathematical definitions due to the page limit.
Given these measures, we calculate their value on whole training examples for each trained model. This choice aligns with prior works, which argue that the most direct approach for studying general- ization is to prove generalization bound that can be calculated on the training set [27] and offer caution against the oversimplified notion that maximizing (or minimizing) measure value inherently leads to improved generalization (5).

---

3.3 Evaluation Metries
To uncover the relationship between measures and robust generalization performance, we adopt the Kendall rank correlation coefficient following prior works [27] 36]. We begin by defining search space θ 1 x 2 X ..... x 2 Each θ, corresponds a search space for each training parameter defined in Section 3.1 Given the search space O, we obtain the trained models (ww for A E e For each model fo (n). we measure the robust generalization gap ((f((w) and the corresponding measure value ju( fo (w). For simplicity, we denote g(0) == ((fo(1r) (ar)) and (u(0) == ((fo(w)). We
then calculate Kendall's rank coefficient [29]
as follows:
-(+) = θ [ 1)
Σ Σ
sgn(g(0) - g(0')) sgn(n(0) m)))))
8 0'E0.000'
where |6) is the number of elements in O and sgn(-) is a sign function. The value of r becomes 1 when the pairs have the same rankings and -1 when the pairs have reversed order rankings. Therefore, a higher value of 7 implies that as the value of a measure increases, the robust generalization gap also increases.
As noted by [27]. the measure may strongly correlate with the robust generalization gap only when a specific training setting is varied. Therefore, Jiang et al. (27] introduced the following metric:
(5) which captures the robust generalization gap when only the hyper-paramete θ₂ changes. However, we demonstrate that 4/2 may not perform well in cases where Simpson's 's paradox exists. Simpson's paradox refers to a situation where each of the individual groups exhibits a specific trend, but it disappears (or reverses) when the groups are combined. Thus, when a parameter O heavily affects the robust generalization gap, thk becomes not effective as it captures the overall trends by taking expectation across all parameters including ( In fact, within the adversarial training framework, the inner in 55 shows extremely high variance due to the high sensitivity of the robust generalization gap with respect to training setups, which will be discussed in Table
(+)) = Es,.... Pe...-..--....11...... 8,, [T( = (01, .0,)),,))) 0x E Ox}]l.
Therefore, we propose a metric for capturing the robust generalization performance by fixing the hyper-parameter O₃ as follows:
T (e) Es. [T( = (01.... ,,,), 0, E O Vi +
(6) measure i within a specific fixed training setup. This
Here, Tx( ()) represents the effectiveness enables us to discover that some measures only work for specific training settings, c.g., uncovering the strong correlation between is used as the training method. Furthermore, for measures exhibiting meaningful value of иh. we additionally provide their scatter plots mitigate the limitations of correlation analysis.
sonndarrttiikkees and the robust generalization gap when AT
4 Experimental Results
Based on the measures and trained models defined in Section 3, we calculate the measures for each model and their robust generalization gap. Notably, in the realm of adversarial training, models encounter two types of examples: benign examples and adversarial examples. Therefore, to gain deeper understanding of the relationship between robust generalization and measures, also calculate the values of cxample-depender measures for both benign examples and PGD10 examples.
Table I summarizes the results of t/a. for each measure. Further, in order to consider the distributional correlation 117 and quantify the precision of v/a, we also report the corresponding standard deviation of the inner in Eq. (5). A higher value of tha indicates stronger positive rank correlation, implying that as the measure value increases, the robust generalization gap increases. First of all, it is important perfect. While certain measures show high t/x, all measures
to note that none of the measures show high standard deviations. This observation indicates that no measure can provide perfect estimation of the model's 's robust generalization gap. Moreover, with such high variances, it becomes challenging to clearly identify the underlying correlation of the measures.
To address this limitation, we conduct further analyses with our proposed metric, T& in Eq. (6). The benefit of म. is that it reveals the potential hidden relationship between the measures and the robust generalization gap by fixing training settings, which th cannot captures. Thus, from now we will report क. of each measure and provide correlation analyses with the robust generalization gap, extending the connections beyond those presented in Table [
5

---

Table 1: Numerical results of the for each measure, along with its corresponding standard deviation. The total T indicates the Kendall's 's rank coefficient for the entire pairs (g. ft). * (PGD) indicates the same measure calculated on PGDIO examples for example-based measures.
Architecture
Model
Methods
Training
Steps
Optimizer
Batch-size
Aug
Extra-data
Stopping
Early
Total
nun, parans pats porn
0.05=0047
0.03
0.22=0.59
0,47:0.399
0.3820.92
0.4120.69 0.20:10.77
0.32-0073
0.29:0.72
0.28:20.3
0.35
Log-prod_of-
0.13::0.9 0.15:0.40
0.11
0.1610.75 0,1310.47
0.02-00.3
0.114
0.17.0.66
0.13
log_prod_ ofrr
0.07=0.44
0.48:0.39
0.62:0.79
0,5610.63 0.45:00466
0.38:0.65
0.35.10.70
0.37.:0.69
0.18
euclid, inrt
0.07=0:47
0.32=00.4
1.00=1.00
0.08#0.74
-4002000000
0.20=0069
0.16+0..8
-0.06=00.70
0.18
average_ce
4.16=00.22
10=1 11001.00-00000 -.00000000444
40:20=0.73
anverse_narg"
0.19:0:57
0.25==0337
0.45:00.8
+0,73
0.11:00.49
0.15:=0:70
0.08+:0.1
0.03-40.72
0.07
prob.nargin
0.19=0.61 0.23=00.52
0.09=1.00
0.26+0,74
0,30=0.53
0.25=0.72 0 0.30+0.20 0.20=00.65
0.24
houndary. thi cknaze
-0.06-0.61 0.05+0..5 1077+1.0000.1544-
0.07++1.00
-0.17-0.53
-.00--0.71
16--0.20-0.11--..66
kl_divergence
-0.42.0.53 3.33.....6.6666
0.71
0.22610.70 0,3610..6
0.56
0.38.10.69 0.44.10.74
4.45
1ip
-0.25-0.566 0:200-00:0001111100000000001 0.
0.13440.71
0.38-0069
2.200-0.99 -0.15++0,72
-.233
pacbayes flat
0.22:0.00 0.20:10.22 0.33.10.94
0.17.00.75
0.17.
0.05 1077
0.06.10.72 0,20.10.69
0.05
estimated_ shargnest
0.07-0.60 0.07:0.41
0.19
8.
0.05
00
003.10.4400.03-100
4.07
estimated inn sharpaedz
0.16:0.60 0.20::0355 0.42:00911
0.27
0.14
0.00=0.71
0.07++0.73 0.06::0.78
0.04
awerage_flat
0.30:0..3:00311:4.:4 8.58.1..2 0.2110..6
0.22100.4
4.43
0..2....68
0.330.0,71
40.36
x x_gred_nod norm
-0.36=0.56 000.-000000006066233000
0.33
0,
0.56
0..00
-044
v. "_grad_norm
0.17-0.56 0.23.10.36 0.50.10.86
0.31
0.1610.52
10.70
0.10.10.22
0.1010.78
0.07
avarage_ca (PDD)
0.39- 000-000--000015511 06
0.59
0.63
0.6
0000
0.59+
0..58
inverse parg iin (PG
0.56=0.49 0.40=0.31
0.75=0.66
0.33+0067
0.44
0.49=0.63
0.39=00.7
0.27=0.76
0.35
prob_nargie (PCD)
0.59-0:53 0.80+00.14
5.84+0.55
0.61+0.69
0.64+0.45
0.62-0.65
0.61+0688
0.59+0.71
0.79
pacbayen flat (PCD)
0.35..0.77 0.64.0221 0.82.1..88
0,61
0.6210.29
0.57.1..4
0.5010.46 0.68.10.57
0.59
est satinated_aha aharapesa (FCDT [9]
-0.05=0.64 -0. 09 -0 10=0.99
0,08+0,77
-0.03+0.555
15
-0
40222
estimated_ inv sharppasa (PQD)
0.08 0.50 0.199
0.08 0.06 4021 -0
12:40.75
0.25
x_grad_nera norn (PCD)
0..23
0.21 0.51.10.5500.1 0,111.0.7000 0.1.....70000........... -..2610..45
0..8 0.29..0.70
0.w.....000003...... 0.33..0..3 4.34
"-grad_norn(PPn
0..4-0..3 -...--0..-----00 19 0.01+0.78
00w.0.88-00009-00000 -.009-0.0770- 19
4.32
Table 2: (Norm-based measures) Numerical results of Te and its corresponding standard deviation.
Model
Training
Early
Architecture
Methods
Steps
Optimizer
Batch-size
Aug
Extra-data
Stopping
num_ pam_pprrams
-0.02:1.00
0000
-0.030.01
-0.02:0.03
-0.02::002
-0.03:0.000
-0.02:10.01
path_norm
0.35+0005
0,27:10.13 0.24=0.26
0.34-0.03
0.37=0003 0.35++0.05
0.35:10.03
0.46:10.14
log_prod_ spec.
0.03110.13
-0.0710.07 -0.00-0.01
-0.12-0.33
-0.11
-0.1310001
-0.13::0011
-0.13:10.06
pog.ppro__of.ooo ofr
0.37+-0.03
0.15:1..11
0.09-0.10 0.18-0.11
0,18+0.04
0,19:00.00
0.19-00.2
0.19-10.04
wellid_nnit__oxe
0,3210003
0.06:10.03
0.0910.10
0.17=0.00
0.20
0.1710001
0.17.00.33
0.20:10.11
Norm-based measures requires fixed model architecture. In many prior works, researchers have demonstrated the effectiveness of norm-based measures in estimating the generalization gap Among them, the weight norm-based measures, e.g., the product of Frobenius norm (fog-prod_of _ffo___f__fr _fr) [46). the product of spectral norm _log_prod_of_spee
27 17].
77. and the distance to the initial weight (euclid_ints (_rmm [27 40 are built on theoretical frameworks such as PAC- Bayes 43 33). path_ aath_normin is also often used to estimate the complexity of neural network, which calculates the sum of outputs for all-ones input after squaring all parameters [27]
In Table I the most of norm-based measures exhibit low correlation with the robust generaliza- tion gap for the metric 1/2 However, by using the proposed metric Ty in Table 2 we discover that Loo_prod_os of froo and maulidd_nit... inii_nnrr exhibit strong correlations with low standard de- viation when the model architecture is fixed. Intuitively, when the model architecture varies, the number of parameters and their corresponding values exhibit different ranges. Indeed, the range of log_prod_oo of _fr roughly shows [50, 100] for ResNet18 8, but shows [140, 200] for WRN28- 10. Thus, comparing models with different architectures degrades the precision of weight norm-based measures. Under the fixation of the model architecture, Log-prod_oor _f is positively correlated with the robust generalization gap, which consistents to the prior theoretical observations in PAC- Bayesian framework [50] or Lipschitz analysis [58] For log_prod_ of spec, we do not observe a strong correlation under any conditions.
Furthermore, we observe that path norm shows some extent of correlation for all the and mi- Upon conducting a more in-depth analysis, we find that the log of path_nost _oom yields almost linear relationship with the robust generalization gap when conditioned with average_co (PGD), resulting the total 7 0.68 (detailed in Appendix (..)). This finding suggests that path_not norm can be effectively utilized for estimating the robust generalization gap, with consistent to the prior works under the standard training framework [7.

---

Table (Margin- measures) Numerical results of .Ty and its corresponding standard deviation.
Model
Training
Farly
Architecture
Methods
Steps
Optimizer
Batch-size
Aug
Extra-data
Stopping
average_co
-0.24:00.03
-028=0.17
-030::027
-0:3400:00
-0.23=0004
-0.24:0:06
-0:23:0:06
-0.34=03T
aeverse_margan
0.07:+0.05
0.05=0.23 -0.18:0.32 0.09+0.06
0.08=0009
0.08:0.113
0.09::0.12
0.09=0.12
prob_nargin
0.2410002
0.28=0.16
0.31+0277
0.23::001
0.23=0.04
0.24:0.06
0.231:0.66
0.34=0.31
boundary_thicoe thicknss
l5
-0.02:+0.01
-0.1700.15
00.19:0.23
0..02+0.01
-..02=0.02
0..04:0.06
-0.02.10.05
-0.04=0.27
@verage__cePO0)
-0.78+0.01
-0.58+0.39
-0.47+0.40
-0,79+0.004
-0.79+0.01
0.800+0.03
40.79+0.00
40.78 0.01
inverse_nne waw.eenaaaiinnccmm (PCD)
0.34+0.02
0.26=0.22
0.05=:019
0.37++0.03
0.35=0.03
0.34:0.05
0.36:10.13
0.38=0.08
(rob_marggnnPGDD)
0.79:0.00
0.59
0.48:0.39 0.79:10.04
0.79:0001
0.80
0.79:10.00
0.78-0.02
Maximizing margin beyond a certain point harms ro- bust generalization. Traditionally, the maximizing margin considered as an ultimate goal in the adversarial train- ing framework [42) [1]. Indeed, average cross- entropy loss
60
average celPGD)
1.5
100000 stees
-55
42 and margin-related losses [6T are frequently used ad- versarial training methods. However, in both Table [ and Table 3 average_co ce (PGD) exhibit high negative corre- lation with the robust generalization gap across all varia- tions of training parameters. Similarly, TT and the total 7 of prob_margin (PGD) are extremely high (0.79). This suggests that lower cross-entropy on PGD examples (and higher margin in the probability space) leads to worse robust gen- eralization. Indeed, Fig. 2|shows a clear negative correlation between average_ (PGD) and the robust generalization gap. Notably, high test robustness is observed within the range of average_ (PGD) E [0.5, 1.0]. indicating that minimizing average ce (PGD) beyond a certain point may harm the gen- eralization performance as Ishida al. 225 observed in the standard training frameworks.
1.0
50
45
40
35
0.0
20
40
60
Robust Generalization Gap
Figure 2: Scatter plot for average_ _e (PGD) and the gap. Bright color indicates higher test robustness. For better visualization, we cutoff average_co ce (PGD) 2.
Given this observation, we argue that the margin maximization in adversarial training methods should be carefully revisited. Recent studies have highlighted that maximizing the margin might not necessarily be the optimal objective in adversarial training due to intricate gradient flow dynamics [59] and the non-cognitive concept of using predicted probabilities [ Additionally, in recent work (32), despite the similar robustness of TRADES and AT, their margin distributions on benign and adversarial examples are extremely different. This implies that the margin cannot be the sole determinant of adversarial robustness. Considering these findings and other recent studies [41 60), the margin maximization should be accompanied by a consideration of other factors such as weight regularization or gradient information.
The cross entropy and margin on benign examples, denoted as average_co _e and prob_ mmariinnnn also show some degree of correlation with the generalization gap. This correlation becomes particu- larly significant when using carly stopping, where their correlations reach up to 0.65. Note that inverse margin behaves differently because it uses the 10th-percentile of margins over the training dataset rather than the expectation.
Boundary thickness works well for models trained by AT. Yang et al. [67 introduced the concept of boundary thickness, which is an extended version of margin based on adversarial examples. They argue that thin decision boundary leads to both poor adversarial robustness and the gap. Therefore, boundary_.. thi ckness should be negatively correlated with the robust generalization gap. However, as shown in Table 3 it does not correlate well with the robust generalization gap. The main difference between our experiments and those in [67] is that we also considered TRADES and MART, whereas Yang et al. [67] sorely compared models trained with AT. Thus, in Fig. 3 we plot the inner 7 in T for each training method. It evident that the boundary thickness demonstrates a strong correlation with the robust generalization gap when the training method is fixed to AT. This suggests that boundary_ tii_thiiaa ckness is more effective for comparing models trained with AT. Furthermore, in Appendix A.2, we also discover that boundary_ thi ckness becomes more highly correlated with the robust generalization gap when the models are conditioned on average __teaaee_eeae _ee (PGD). Thus, when Satrrytthhhee
using boundary- thi thicks as the sole determinant of robust generalization, we should carefully revisit the choice of training methods and the robust accuracy of models train datasets.
7

---

inner in for Training Melhod
1.00
AT
0.50
0.25
ST ST
0.00
4ST
-0.25
ITRADES
AT
AT
MT
-0.75
-1.00
num, params
paih, norm log, prod, of spep stespe tng. prod of fro aucld _nit, norm average,
inverse, margin
brundary, thickmass
prob, margin
divergennce
pachayes, estimated sharpwwas
local, lip
eatimated inv shampees
average flat g.grad_ harm
av prarage cwjPGD)
grad norm
inverse, maryinPGGD)
prob, merginit mergin(PGD)
estimated, charpnessiPe GO) GD)
estimated, Inv wwwrphhssssppp.. (PD)
x x_grad, norm(PGD)
w..rrddd (arm(PGG)
parbayes flatff
Measure
Figure 3: Box plot of the inner T in TT. Eq. (6 where O₂ is training method. The text corresponds to the method for each outlier, e.g., boundary ckness performs well when AT is used.
Table 4:
(Smoothness-baat based and Flatness-basee measures) Numerical results of n and its corre-
sponding standard deviation.
Model
Training Methods
Early
Anchitecture
Steps
Optimizer
Batch-size
Aug
Extra-data
Stopping
xi_divergence
0.4510011
0.35 0.1310.19 100.0.99..... 0.46
0.10 0.46
0.43 10.12
0.45 1002
0.37:0.25
local_lip
-0.24-+0.06 00.15-0.19 19-00-00000177-7 10..0.17--.24-00000
0..00-0..05
0.23-00022
-0.21
pacbayes_flss _lat
0.04:10.9
0.04 -.14-.0..00014
0.06-0.000 0.04:0.08
0.10:0.12
0.88 0.07:0.16
estimated e
19
-0.07+0.022
10.0000--00013
-0.04-0.16
eatimated_inv__hhae
0.04:10.02
0.33 0..4:00.22001159 0.119...111 0.05
0.04:0.06
0.10:10.13
0.07:0.10 0.03:1.112
average_flat
-0.36+0.00 -0.24-0.21 1224-00110000001111 0.10=0015--088-000
010-00001-024-00..
1.37--006-0032---11
-00-0.000-0033--23
pacbayes_flat (PcD) 19
0.59=0.04 0.45=0.16 0.30=0.21 0.60=0.00
0.59=0.02 0.62=0.02 0.63=0.03 0.50=0.07
est astimated_eharpaaat
-0,22:1.01 000-000010000000.2
-0.17:0.07 00.19 119-0.06002112..0 0022120.32
estinated, iecpslivvv____eeae wttpeiaaeeeeeeeeeeeee (PGD)
0.24+0.02-0112-0114444
-0.23=0.06 -00-0.300.002
Smoothness does not guarantee low robust generalization gap. In the pursuit of achieving adversarial robustness, the smoothness between benign and adversarial examples is often considered as an indicative measure. For instance, TRADES (71] minimizes the x1_divergence between benign and adversarial logits. However, kl_ _i_divereence shows negative correlation for both and
Table and Table 4 This implies that, similar average_ce (PGD), kl kl_divergence cannot serve as an indicator for robust generalization.
While Xu al. 665 demonstrated that imposing local Lipschitzness (local_ _ical_lip) leads to better generalization in linear classification, recent research [66] argued an opposing perspective, suggesting that within neural networks, local Lipschitzness might hurt robust generalization. However, this conclusion was built on fewer than 20 models and evaluated solely on test examples. our large experiment, cannot observe that local Lipschitzness itself negatively affects robust generalization. Rather, it is more efficient in predicting robust accuracy (detailed in Appendix(A.I These findings are consistent with 45 39], which highlighted the importance of model architecture or weight norms when evaluating models with local Lipshitzness.
Flatness- based measures are not correlated well with the robust generalization gap. Flatness- based measures have recently regarded as powerful indicators generalization performance both standard and adversarial training frameworks [T8 64 This includes the maximum perturbation size in the weight space that do not dramatically changes the accuracy spacbayes_fles flat) ) 43 27). the loss increment by adversarial weight perturbation secrimated_sharppeeeee
[18), and its scale-ine ieale-invariant version (estimated _i__iin____ snv_sharpness) [B6]. However, our analysis reveals that flatness- based measures tend exhibit poor correlations with the robust generalization gap. In both Table T and T the majority of flatness-based measures exhibit near-zero correlations or even negative values. Only pacbayes_ flat (PGD) demonstrates strong correlation with robust generalization because effectively distinguishes between robust and non-robust models (refer to Appendix (A.2).
Recently, Stutz al. (56) demonstrated the importance of early stopping in the analysis of flatness. Similarly, we observe that, when early stopping is employed, the correlation of estimated sharpppssssa approaches zero. However, without early-stopping, we discover

---

Table 5: (Gradient-basse based measures) Numerical results тą and its corresponding standard deviation.
Model
Training
Early
Architecture
Methods
Steps
Optimizer
Batch-size
Aug
Extra-data
Stopping
x-ggrad norn
22
-0,422:10.03
40,4540.22 +0.31:0..33 -0,43 0.09
+0.42:00.2
-0.40-10.002 -0.41=:0.05
+0.3%
v_grad_nore norn
25
0.07++0.04 -0.02+0.33 -0.17=0.18 0.08=0.13
0.07=00088
0.11+:0.13
0.09:+0.09
0.05:+0.08
-_grad_norr _ord_dooomn (PCD)
-0.3410.02
-0.2410.21 -0.09=0.15
-0.36=0.05
-0.34:0.09
-0,32:0.12
-0.34:1..02
-0.25:0..55
v..radd (PCD)
+0.32.40.03 -0.16440.17 -0.15:1..8 -0.32.0.1 -0.33:+0.09
40.28 -0.31:0004 -0.28:0.31
Table Numerical results of Ty for each measure when θ₁ is given by average_ (PGD) values with bin size of 0.1, along with its corresponding standard deviation.
Measures
Fi:
Measures
mun parana
00.23=0.17
estimated sharpneaa
19
-0124:0.11 19
path norm
0.25=0.14 14
estimated, inv_ _erhaarrreee naargees
-0.1::0.17
oog_prod_of__oo swrooddof__ppp_ _ppcc
0.10=00.3
average_flst
0..12:0.17
log. prgd_rod_oot _ro
0.11 10
x_grad_ _orn
1..35
euclid ieiiiniitttgg
-0.11=0.14
x_grad_nora
-0.06:10.17
average_ce
0.03
inverse_matee margin (PUD
(PUD)
13
0.05:10.19
inverse_margi
-0.09+0.19
prob_margan
0.13+0.11
prob_nargin
-0.02:1.116
pacbayes_ flat (PGD
-0.19:0..00
boundary_thickh [6
0.06+00.19
aatimated__harpneee
(Pd1)
19
0.05+0.19
dl_divergeece
-0.09=0.14
eaticated_11o__aaarpees
(PCD)
0.01+0.19
local_
-0.11=0.16
1_grrad_nor(PPDD)
22
0.12-40.14
pacbayes_flss
-0.25
0.13+00.16
that estimated_ ssaaatte_shappeess exhibits significant negative correlation for models have low average_ (PGD) < 1.5. As shown in Fig. models with low astimated_sharppess show high robust generalization gaps. This finding aligns with the concurrent work of (5), which demonstrates that flatter solutions generalize worse out-of-distributien data. The additional results can be found in Appendix A.6
estimater sharpness
Test Robustrres (%)
0.6
50
40
0.4
30
20
In the case of average_ ffaat [56], which is calculated with random weight perturbations and their worst-case losses, demonstrates some degree of correlation. However, it is more efficient in predicting robust accuracy rather than the gap (refer to Appendix A.I). This result suggests that, the concurrent work 5 observed in the standard training framework, flatness measures may not serve reliable indicators of correlation in the adversarial training framework even they can be effectively used to achieving better performance.
10
20
40
60
Robust Generalization Gap (%)
Figure 4: Scatter plot for the robust generalization gap and aetimated_sharppiee ditioned on average_ce (etraage_ce(PG)) < 1.5 without early-stopping.
Con-
The norm of gradient of inputs robustly captures the gap even for models with similar eross- entropy losses. Although some prior works [53] 24] have argued that regularizing the input gradient norm might improve adversarial robustness, we observe that this cannot be argued as lower input gradient norm is better. Table summarizes of the gradient norm of input
and the gradient norm of weight (u_grad_nore -orddddoonm.-moo Among these, x x_grad_nov _nom show a strong correlation with the robust generalization gap. The negative correlation of x_ x_grad_ _oor_d_oommmmm indicates that models with larger input gradient norm are more likely to show lower robust generalization gap.
Furthermore, even when comparing models having similar (verage_ce(PGD) x_grad_ _orr is the most robust indicator of the robust generalization gap. Previous works in the standard training framework 27 17 have argued that the cross-entropy loss is strongly correlated with the robust generalization gap, and thus, they used early stopping based on certain cross-entropy thresholds during training to remove the influence of varying cross-entropy loss. However, within the adversarial training framework, employing the same early stopping based on loss becomes challenging as TRADES and MART minimize different loss functions from AT. Therefore, we categorize the trained models into groups based on average_ ce (PCD) values using bin size of 0.1. This grouping reduces TA of average_aac ce (PGD) to 10. The results are summarized in Table Compared to all other measures, x x_grad___p_ parmm norm exhibits the highest rank correlation with the robust generalization gap even when conditioned on average_ ce (PGD). We believe this finding highlights the importance of the norm of input gradients as valuable regularizer for improving model robustness and its generalization in practical settings.

---

1.00
0.75
CIFAR10
0.50
CIFAR100
0.25
ImageNel
Tocal
0.00
-0.50
-0.7 75
num, parama
path porm log prod of spec log. plod of fro
average
inverse mergin prob margin boundary, ttyckness li divergenca
lncal, pactay yes fiat estimated sharpnisss estimated, inv sharpness av averege fal
grait narm
w. narm
av ererag ce(PGD)
inverse margin (PG)
proh, margin( margin(PGGD) (PGD)
estimaled sharpness, (eppnss((PDD)) (PGD)
pactayes, baae,, fial(PG)
esimaled in sashshhrpppa (ppppssss/PG))
K_spad, rored(
"_Grad nomm
Figure 5: Experiment RobustBench [2]. For each dataset, plot the total of each measure and highlight the robust measures with | 2 0.25 for all datasets with yellow background.
Table 7: Numerical results of тTę for norm-based measures when O is the model structure along with its corresponding standard deviation. The total is the same as in Fig. 5 For ImageNet, is not applicable due to the limited number of pre-trained models in RobustBench [1]
CIFAR-10
Total r
CIFAR-100
Total r
path_nora log_prod__ of fro 0.12:10.40
0.24
0.02
path_porm
0.55=0005 0.30
0.09
0.10
Log_prod. of fro
10
0.17
Broader Impact with Benchmarks
RobustBench []2] provides set of pre-trained models that achieve high robust accuracy across various datasets, including CIFAR-10, CIFAR-100, and ImageNet. Leveraging this benchmark, we extend our observations to diverse models including transformer-based architectures 13 [3] or trained on diffusion-generatie of our findings in Section 4 also be applied to these models. Margin- measures such as average_cco _e, prob_ prob_marge margin, average_ce (PGD), and prob_margine (PGD) consistently exhibit strong correlations with the robust generalization gap. Additionally, observe that x_grad_mmg norm consistently shows reliable performance in predicting the robust generalization gap, even when applied to models in the RobustBench across various datasets.
datasets [5T] 62]. As shown Figure 5] we identify that some
Though a deeper analysis limited by the absence of training setting details, such as the use of early stopping. we additionally conduct an analysis with the model architecture by analyzing the pre-trained models. Table 7 summarizes Fy. for models with the same architecture. As we demonstrated in Section
4 norm-based measures exhibit higher correlation when comparing models with identical architectures. Notably, for CIFAR- 100, we find that path_ norm shows a strong correlation with the robust generalization gap. Regarding the low correlation and high standard deviation of norm-bart based measures, we hypothesize that other training settings, such as the choice of activation functions (e.g., Swish and SiLU instead of ReLU) and training datasets, may affect the values of norm-based measures. Further exploration of these aspects is left to future work, as additional research and experiments can provide more comprehensive understanding of these relationships.
Limitations and Future Work
While our study unveils the correlation between various measures and the robust generalization gap over 1,300 models, due our computational constraints, we focused on ResNet models, CIFAR-10. and PGD with the Loo norm. Thus, further investigations on a broader range of the use of stronger attacks may uncover new relationships beyond our analysis. We hope that future work would address these limitations.
and
7 Conclusion
Through large-scale experiments, we verified the underlying relationships between various measures and the robust generalization gap on CIFAR-10 under the L. norm. Our findings offer valuable insights into robust generalization and emphasize the need for caution when making statements such as, 'model A is superior to model B because model A exhibits better measure value than model B,' a frequently employed phrase in recent literature. We hope that our discoveries regarding diverse measures contribute to further advancement in the field of adversarial robustness.
10

---

Acknowledgements
This work was supported by the National Research Founda- tion of Korea (NRF) grant funded by the Korean government (MSIT) (No. 2019RI and the Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00984).
References
[1] Hiroki Adachi, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Revisiting restahee adversarial training. 2022.
[2] Jean- lean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? Advances in Neural Information Processing Systems, 32:12214-1222, 2019.
[3] Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et Xeit: Cross- covariance image transformers. Advances in neural information processing systems, 34:20014- 20027, 2021.
[4] Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adver- sarial training. Advances Neural Information Processing Systems, 33, 2020.
[5] Maksym Andriushchenko, Francesco Croce, Maximilian Müller, Matthias Hein, and Nicolas Flammarion. modern look at the relationship between sharpness and generalization. arXiv preprint arXiv:2302.07011, 2023.
[6] Peter L Bartlett, Michael Jordan, and Jon D McAuliffe, Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-156, 2006.
[7] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally- normalized margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
[8] Samuel Rota Bulo, Lorenzo Porzi, and Peter Kontschieder. In-place activated batchnorm for memory- opptiiie training of dnns. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5639-5647, 2018.
[9] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems, pages 11 11192-11203, 2019.
[10] Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks:
Improving robustness to adversarial examples. In International Conference on Machine Learning, pages 854-863. PMLR, 2017.
[11] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning. pages 2206-2216 PMLR, 2020.
[12] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateck Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. arXiv preprint S..i:200100096777 2020.
[13] Edoardo Debenedetti, Vikash Sehwag, and Prateek Mittal. light recipe to train robust vision transformers. In First IEEE Conference Secure and Trustworthy Machine Leurning, 2023.
htt.hhtttp::/ppeervvieeee wtt.://wwwweeeee.oooooon/iiaaaae
[14] Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Di- rect input space margin maximization through adversarial training. International Conference on Learning Representations, 2019.

---

[15] Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pages 1019-1028. PMLR, 2017.
[16] Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation-invariee invariant attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4312-4321, 2019.
[17] Gintare Karolina Dziugaite, Alexandre Drouin, Brady Neal, Nitarshan Rajkumar, Ethan Ca- ballero, Linbo Wang, Ioannis Mitliagkas, and Daniel M Roy. In search of robust measures of generalization. Advances in Neural Information Processing Systems, 33:11723-1111 11733, 2020.
[18) Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware min- imization for efficiently improving generalization. In International Conference on Learning Representations, 2020.
[19] Jiexian Ge, Xiaoxin Cui, Kanglin Xiao, Chenglong Zou, YiHsiang Chen, and Rongshan Wei. Barelu: combine batch normalization and rectified linear unit to reduce hardware overhead. In 2019 IEFE 13th International Conference on ASIC (ASICON), pages 1-4, IEFF, 2019.
[20] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint 1eXiv.1442.666 6572, 2014.
[21] Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010003593, 2020.
[22] Sven Gowal, Sylvestre- Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. Improving robustness using generated data. Advances in Neural Information Processing Systems, 01:44218-42333 2021.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[24] Zhichao Huang, Yanbo Fan, Chen Liu, Weizhong Zhang, Yong Zhang, Mathieu Salzmann, Sabine Süsstrrnnk, and Jue Wang. Fast adversarial training with adaptive step size, arXiv preprint arXiv:2206.02417; 2022.
[25] Takashi Ishida, Ikko Yamane, Tomoya Sakai, Gang Niu, and Masashi Sugiyama. Do we need zero training loss after achieving zero training emror? arXiv preprint arXiv:2002.08709 2020.
[26] Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. In International Conference on Learning Representations, 2019. URL
[27] Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In International Conference on Learning Representations, 2020. URL URL httpttt:/opperreiiiiiet
SJgIPJBFVH net:/foummmaia Sd-
[28] Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing, arXiv preprint 1003.v1803.06. 2018.
[29] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-93, 1938.
[30] Nitish Shirish Keskar, Jorge Nocedal, Ping Tak Peter Tang, Dheevatsa Mudigere, and Mikhail Smelyanskiy. On large-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representations, ICLR 20/7, 2017.
[31] Hoki Kim, Woojin Lee, and Jaewook Lee. Understanding catastrophic overfitting in single-step adversarial training. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8118-8127, 2021.
[32] Hoki Kim, Woojin Lee, Sungyoon Lee, and Jaewook Lee. Bridged adversarial training. Neural Networks, 06:226-2822. 2023.
12

---

[33] Hoki Kim, Jinseong Park, Yujin Choi, and Jaewook Lee. Stability analysis of sharpness aware minimization. arXiv preprint arXi::2301.06.08, 2023.
[34] Hoki Kim, Jinseong Park, Yujin Choi, Woojin Lee, and Jaewook Lee. Exploring the effect of multi-step ascent in sharpness-awar minimization. arXiv preprint vrXiv:2302.10181,, 2023.
[35] Hoki Kim, Jinseong Park, and Jaewook Lee. Generating transferable adversarial examples for speech classification. Pattern Recognition, 137:109286, 2023.
[36] Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pages 5905-5914 14. PMLR. 2021.
[37] Yuhang Li, Mingzhu Shen, Jian Ma, Yan Ren, Mingxin Zhao, Qi Zhang, Ruihao Gong, Fengwei Yu, and Junjie Yan. MQBench: Towards reproducible and deployable model quantization bench- mark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Bench- marks Track (Round I), 2021. URL https: pestts://opeee /ttp//eeeeeeeeeeeeeeeeeeeee
[38] Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geometry, and complexity of neural networks. In The 22nd international conference on artificial intelligence and statistics, pages 888-896. PMLR, 2019.
[39] Youwei Liang and Dong Huang. Large norms of enn layers do not hurt adversarial robustness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8565-8573, 2021.
[40] Chen Liu, Mathieu Salzmann, Tao Lin, Ryota Tomioka, and Sabine Süsstrunkk. On the loss landscape of adversarial training: Identifying challenges and how to overcome them, Advances in Neural Information Processing Systems, 33, 2020.
[41] Ziquan Liu and Antoni B Chan. Boosting adversarial robustness from the perspective of effective margin regularization. arXiv preprint arXiv:2210.05118, 2022.
[42] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706. 06083 2017.
[43] David A McAllester. Pac-bayesian model averaging. In Proceedings of the twelfth annual conference on Computational learning theory, pages 164-170, 1999.
[44] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robustness via curvature regularization, and vice versa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9078-9086, 2019.
[45] Laura Fee Nern and Yash Sharma. How adversarial robustness transfers from pre-training to downstream tasks. arXiv preprint arXiv:2208. 03383, 2022.
[46] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm- based capacity control in neural networks. In Conference on learning theory, pages 1376-14011 PMLR, 2015.
[47] Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A PAC-bayesian approach to spectrally-normaaizzedd margin bounds for neural networks. In International Conference on Learning Representations, 2018. URL httpp::://opeeeeeeee.eeeeeeeee http//frrrmmiisss__eeeeeeee
[48] Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Bag of tricks for adversarial training. arXiv preprint arXiv:20/0.00467, 2020.
[49] Adam Paszke. Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Ginelshein, Luca Antiga, et al. Pytorch: An imperative style, high--perfrrmance deep learning library. In Advances in neural information processing systems, pages 8026-8028 8037- 2019.
[50] Konstantinos Pitas, Mike Davies, and Pierre Vandergheynst. Pac-bayesian margin bounds for convolutional neural networks. arXiv preprint axXiv11801.0117,, 2017.
13

---

[51] Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann. Data augmentation can improve robustness. In A. Beygelzimer, Dauphin. P. Liang, J.d Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL htttps://peenreviee
[52] Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pages 1093-8104, PMLR, 2020.
[53] Andrew Ross and Finale Doshi- Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
[54] Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Ad- versarially robust generalization requires more data. Advances in neural information processing systems, 31:5014-5026, 2018.
[55] Chawin Sitawarin, Supriyo Chakraborty, and David Wagner. Improving adversarial robustness through progressive hardening. arXiv preprint arXiv:2003.0934, 2020.
[56] David Stutz, Matthias Hein, and Bernt Schiele. Relating adversarially robust generalization to flat minima. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7807-7817, 2021.
[57] Christian Szegedy, Wojeiech Zaremba. Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties neural networks. arXiv preprint 1rXiv:1312.6199, 2013.
[58] Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. posscitzz-arrghn training: Scalable certification of perturbation invariance for deep neural networks. In Advances in neural information processing systems, pages 6541-6550, 2018
[59] Gal Vardi, Gilad Yehudai, and Ohad Shamir. Gradient methods provably converge to non-robust networks. Advances in Neural Information Processing Systems, 35:20921-20932. 2022.
[60] Haotao Wang, Aston Zhang, Shuai Zheng, Xingjian Shi, Mu Li, and Zhangyang Wang. Remov- ing batch normalization boosts adversarial training. In International Conference on Machine Learning, pages 23433-23445. PMLR, 2022.
[61] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International Conference on Learning Representations, 2019.
[62] Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. arXiv preprint arXiv:2302.04663 2023.
[63] Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001. 03994, 2020.
[64] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust. generalization. Advances in Neural Information Processing Systems, 33, 2020.
[65] Huan Xu, Constantine Caramanis, and Shie Mannor. Robustness and regularization of support vector machines. Journal of machine learning research, 10(7). 2009.
[66] Yuo-Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Ruslan Salakhutdinov, and Kamalika Chaudhuri. A closer look at accuracy vs. robustness. Advances in Neural Information Processing Systems, 2020.
[67] Yaoqing Yang, Rajiv Khanna, Yaodong Yu, Amir Gholami, Kurt Keutzer, Joseph E Gonzalez, Kannan Ramchandran, and Michael W Mahoney. Boundary thickness and robustness in learning models. arXiv preprint arXiv:2007.05080 2020.
[68] Dong Yin, Ramchandran Kannan, and Peter Bartlett. Rademacher complexity for adversarially robust generalization. In International conference on machine learning, pages 7085-7094 PMLR, 2019.
14

---

[69] Stone Yun and Alexander Wong. Do all mobilenets quantize poorly? gaining insights into the effect of quantization on depthwise separable convolutional networks through the eyes of multi-scale distributional dynamics. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 2447-2456, 2021.
[70] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 87, 37.1-87.12. BMVA Press, September 2016. ISBN 1-901725-59-6.
doi: 10.5244/C.30.87. URL https://dxx doi org/10.g/.ooo 52440/...2244.0..0
[71] Hongyang Zhang, Yaodong Yu, Jiantao Jíao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pages 7472-7482. PMLR, 2019.
[72] Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference on Machine Learning, pages 26982-26992. PMLR, 2022.
[73] Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvornek, James Duncan, Ting Liu, et al. Surrogate gap minimization improves sharpness-aware training. In International Conference on Learning Representations, 2021.
15

---

A Additional Experiments
A.1 Estimating Test Robust Accuracy
Instead of estimating the robust generalization gap, might expect the analysis on the relationship between test robust accuracy and the measures. In this regard, we investigate the correlation between the measures and the test robust accuracy - E(w;; c, (D) the test dataset DD instead of the robust generalization gap (.)).
Table 8: Numerical results of 4/2 between each measure and test robust accuracy, along with its corresponding standard deviation.
Model
Training Methods
Early
Total
Architscture
Steps
Optimizer
Batch-size
Aug
Extra-data
Stopping
mun panapa path_porr
0.14-0.44
0.02
0.22.:0.57
0.3610.41
0.3310.94
0.0810,72 0,22.10.50
0.20-10.70
0.17.0.73
0.43.0.51
0.02
leg-prod_or_an
-0.22-0.43
0.06+00.2
0.05++1.00
0,0440.73
0.03++0.49
0.07-00733
0.07+0.23
07-+0.23-0.16++0.6
-0.16+0.68
0.15
1og_prod_pd of
0.12:0041
0.36
0.64
0,3011,,0
0.44:10.41
0.39 0.39-0.66
0.40.10.65 10.66
0.07
euclid cellddinnt__at init_porm
0.16=0.45
0.28:0.31
0.11420,99
0,01110.73
00144:00.48 40.13:0.69
0,11:10.68
00233.0.622
average_ce
-0.0900.22 0:03=0.53
1.08=1.00
4117:0.7--0.13-56
0.01=0.76
0.55=20.47
0.14
inwerse.narg.
0.24=0:56 0,44:0036 36 0.48=0088
0,44+0.60
0,18+0.50
0.28-0069
0.240068
0.71+00.37
0.44
prob_nargin
0.05=0.62 40.03=0.54 1..0--00000000000000000
0.11=0.57-00000
-0,01
00003:0.75
-0.56:20.46600
boundary. thickneez
15
0.07=0.00 0:24=00.9
0.26=0.97
0,29+0.71
0.06=0.53
0.17=00.55
0.21 0.62=0.43
0.32
k), <l_divargance
-0.55-0.49 0.557.0.36----- 0:94-00.5555---- 0.65-0..554--- -..66-0660040
00.55+066111 -0.63+0.477
-0.45
local_ Lip
-0.46=0.49 100-0049-0044420-00 -0:44000404000000000000000000000 0.700000000002222
05
-0.33=0..99
pachayea.
0,31-0:52 0.21+0.44 0.315
0.17+0.77 0,25+0.51
0.28-00.8
0.26+0067 -0.17+0.77
0.15
astimated_ eatimated_charpaeta
19
0.09 0.25 0.36.00.3
0.1610.75
0..50
0.30--088
0.32.10.77
0.20.10.66
0.18
eat Inv ahhrrpness
20
0,24=0.56 0.39=0.33 0356=0083
0,29+0,74
0.29=0.50
0:38-005
0.41+064
0.5500588
0.33
average_flas
-.44 0.60:0.41 0.65
10.51
0.40
0.
0.666
0.42
x_grad_nos norm
-0.40-0.48 0.344.0..34....
0.74
0.4910.559
0.37.10.41
4.26 0065
00.00.1..65
0.079
40.17
v_prad_norm
0.20-0:52 0.42-00.3
0.66+0.75
0.33++0.75
0,29+0.50
0.42-0.66
0.42+0655
0.60+0054
0.37
average,cm ce (PDD)
0.890
46-00
08
20.35
0.51=
40.58
inveree_ marrginn (POD)
13
0.58=0.47 0.64220.0 0.86:0.50
0.690045
0.51
0.49:0.22
0.59:0.56 0.7820.33
0.61
prob_nargi (PCD)
0.74=0.43 0.73=0.18 0.92=0.38
0.66+0433
0.60=0.39
0.54=0.58
0.52=00.0
0.58=0.50
0.60
pachayea_ flat (PGD)
0.62-0.44 0:59+0024 0
0.60+0.56
0,52+0.45
0.60-0448
0.72+0.38
0.11+0..7
0.61
estinsted wea.ppeeeseeeeeeeeeeee (PGU)
-0.17=0.6 411==0065-022820.544 90 25
inttnntedd_on sharpnese (PQD)
20
-0.20=0.65 -0:22:0 0 02 19 0..05
0.01
0.04+0.71
-048++0.488
4:21
x_grad_norn (PCCC
-0.37=0.51 0. 0.77 5.55 0.3440.44
0.21 0.666.6.66666-
0.63-0..45 -..5--00..0-..
v.grad_noru (PGD)
-0.30=0.61 -022::0.5000000000 080-:000000002:00---6-0-- 0.000000.700000..7 0.0550072 2...5.00000.....3........... -0.7000.00000000--0..000
Table 9: Numerical results of me between each measure and test robust accuracy, along with its corresponding standard deviation.
Model
Training Methods
Early
Total
Architecture
Steps
Optimizer
Batch-size
Aug
Extra-data
Stopping
nun, _un_parame parane path_porn
0.0420.04 0.05:0.12
0.03=0.00 0.02200:3
0.02=0:00
0.03-20:02
0.02:00022
0:02
0.01=0.12 0. 19-00.3.00114 0.0400000 0.0220.03
0.04:0.01
0.02-00.55
0.2710.18
0.02
Log_prod_ _f_dd_____pe__ spec
-0.06=0.11 12 -0.08=0.111 10000000000000000000000000000000 10-0.00=00.00
0.115+0.0000.4::00000
-0.15=0001
-0.15+0.03
-0.16+0.04
40.15
tog_prod__ frr
0.22-0008 0.06++0.05 -0.00=0.114
0.07+0.01
0.07+0.01
0.07=0002
0.06+00:11
0.12+0.05
0.07
euclid, init_porr
0.08=0.02 -0. 0..1020..16000 16.00000010.....1.
0.11:0000
0.11=0.04
0.11:0004
0.12:00.1
0.1622011
average_co ce
0.16:0006 0.26:1021
0332:0,21
0.13100:2
0.16100.2
0.15:0:02
0.15:00:3
0008.10.40
0.14
inverse_nare paaraenaargaai
0.45-0008 0.47-50.13
0.43--0.14
0.44--0.02
0,44
0.44-0004
0.44+00.3
0.34--0.22
0.44
prob proginargin
.0.16. 0.27:10.21
0.32..0.22
0.11....02
001.000220111..0000 0.15510.02
40. 1..01 0..15:1..33
0.08.10.41
40.14
boundary thi thicknses
15
0,33=0.02 0.36:20.22
0.1::0222
0.302:004
0.33=0002
0.32=00022
0.3220003
0.1920.38
0.32
.0.44.0001
0..4..0.15 0.29.0..66
0.45....0
00.45.0.033
10.0
0.40..0.20
0..5
local, Lip
-0.39=0.03 0.:4520.155 0.40000227
-0.38.0.00
-0.1010.03
00.43=0.011
0.40::0.03
-0.31
40.40
pacbayes
0.16=0.04 0.07=0030
-0.00.00..0.
0.1620.06
0.14=0.066
0.14=0008
0.14=0.10 10
0.26:20116
estimated charpnese eetimatod_eaarpees
0.1920.05 0.18:00111 1.05
0.1910.10
0.171:004
0.17:00.0
0.16:10.03
0.20:10111
0.18
est
estinsted iny, sharpaers
20
0,35=0.06 0.35=0020 0.20
0.35#0005
0.33=20.33
0.33=0000
0.32=0.01
0.28=20.77
0.33
average_[ls fltt
-042=00022 -0.42-0. 17 00
0.000.077
-0.42:0.022
00.45=00022
0..4++0.05 -0.36+0.244
x_grad_nor norm 22
-0.17-0.22 -0.1=20.11 0.0420.22-00
-0. 0.17:0.01
-0.1710000
40.20=00.16
111.0.1661111111100 16.1000110001111 18 -0.20:20.05
-0.17
*_gred norm
0.41=0.08 0.39=0022 0.24:0088
0,38+0.06
0.38=00055
0.36=0002
0.37=0011
0.29=0.01
0.37
avarage_ca ca (PaD)
4.56-00.0 000-0.06-0.52--00 0.52200.18 00.1 -1.4--04---00
0.55++0.05 0.557+0.01
0.59--0.000
-0588-0..8
-0.1100.07
-0.58
inverse iarreennarinnnee (PiD)
0.61:0.11 0.6410.04
0.5210.16
0.61
0.6110.01
0000
0.6010.02
0.56.10.15
0.61
prob_nargic, (PGD)
0.58=0.06
0.5420.20
0445:0.44
0.6110.05
0.59200.22
0.01
0.00:0099
0.6420.05
0.60
pachayes_ flat (PCD)
T6
0.60=0.02
0.29
0.05
0.61
0.60
0.61 0.58
0.62:0,10
0.61
estimated weapeeddeeapeeeeeeeeeeee (PCD)
estinated_
4.200 4..21.0.23
0.2220
0,201...8
40.24.0.02
-0.15.0.32
40.2
estinated, iny sharpness (POD) 20
-0.18=0.4
-0.23=20.24 -0 00.24=0 41 -0. -0.19 10000000000.2---000
-0.21
4..23
-021
(Prod_norr(PGD))
4.39-00.00
1. 0.2911..4 -0.370..04 00.3910.03 4..1 -0.3910.055 10.27.0..16 4..3
v_grod__oor(PPD)
-0.25=0.03 -0.28:0.24 -0.30:00 -0.27-0.00077077 0.23:0..0-00331000000 -0,30=0.03 -0.1100.28 -028
Tables and | present the values of th and TT for each measure. Fig. Biillustrates the difference in total T when the robust generalization gap and the test robust accuracy are used as the target variable for corrllatin analysis. Certain measures exhibit stronger rank correlations with the test robust accuracy than the robust generalization gap. w_ v_grad_ _od_m, inverse saverse_margin, and inverse gaverse_marginn (PGD) exhibit the total values exceeding 0.4 with respect to the test robust accuracy, whereas they show
16

---

Robust Generalization Gap
Rank Correlation
0.5
Test Robust Accuracy
0.0
0.5
num params path, norm log prod of spec tra euclid init, norm average ce .rrad norm grad, nerm
inverse margin prob. margin boundary, thickn thickness ees
divergenc gergeeee
local lip pactay yes flat estimated herppeess estimated, Inv Sharpness average, flat average Ce(PGD)
inverse marginn(Pnn prob, margin margin(PGD)
estimated sharp CharpnessiPGr (PGD)
estimated, Inv charpnesspoo (PGD)
parbayes flat fattPGD))
x. g.ard nomiPGD) w. grad norm((PD)
log prod
pacbayes
Measures
Figure 6: Comparison of the total 7 when the robust generalization ((w) (yellow) and the test robust accuracy Elw;e, e, D) (blue) used the target variables for correlation analysis.
near-7ero total values for the robust generalization gap. Similarly, local_ __pp and average flat show stronger correlations. Although we observe some different behavior measures, we find that estimating the test robust accuracy can be more challenging. When we perform a linear regression analysis, predicting the test robust accuracy yields poor R22 values. To ease comparison, we refer the readers detailed analysis to Table 15 in Appendix A.3
A.2 Focusing on Adversarially Robust models
In the main paper, we use all models regardless of their robust accuracy, to ensure the generality of our analyses. However, as we discussed in Table 4 adversarially robust and non-robust models may exhibit different behaviors with some measures. For instance, as shown Fig. non-robust models have extremely low values of pacbayes_ssa.c fla (PGD) less 5. constrast, robust models show higher values of pacbayes_ _aasssfflattt_ (PGD) over 5. Thus, investigating only ro-
60
pachayes Mat(/PD))
15
40
10
30
20
10
bust models might potentially reveal hidden behaviors of mea- sures in predicting the robust generalization gap.
20
40
60
Robust Generalization Gap (%)
In Tables 10 and we summarize 4ik and T& with condi- tioned on average ce (PCD) < 1.3. To ease the comparison between the previous results, we also illustrate the total r of those in Fig. 8 First, overall and th of path_nort norm increases, and the total t increases 0.35 to 0.63. As shown in Fig. 9a the log of path_notoe _orm shows almost linear rela- tionships between the robust generalization gap. Similarly, overall T& and th of prop_masaeee margin, and the total 7 decreased -0.23 to -0.62. Similar to average_ce, show more negative correlations. Thus, similar to probability margins adversarial examples, maximizing the margins on benign examples may harm to the robust generalization with high probability. Lastly, boundary thickness shows strong correlation the robust generaliza- tion gap for average_cce (PGD) < 1.5. As shown in Fig. 9c high boundary_ thi ckness thicsssssss low robust generalization gap. Thus, condition for boundary_thie thecsssss
Figure 7: Scatter plot of pacbayes_ flat (PGD). Adversar- ially robust models (bright colors) and non-robust models (darker colors) show distinct range of values for pacbayes_ flat (PGD).
the margin-basee measures, i.c., inverse __rrssinanniinn and
shows
conclude that average_co (PGD) is also effective ckness as well as the training method.
Notably, some flatness-based measures have weaker corrllation, average_flat (-0.36 to -0.15), @stimated_sharppeee (PGD) (-0.22 to -0.02), and aestimated_iv__sharaeee 0.03). Their overall values of t/a and πyç in Tables 10 and II also close to 0, which supports our claim that flatness measures cannot serve as reliable indicators of correlation in the adversarial training framework.
(PGD) (-0.25 to-
17

---

Table 10: Numerical results of th for each measure, along with its corresponding standard deviation. Conditioned on average_ce (PGD)
1.5.
Architseture
Model
Methods
Training
Steps
Optimizer
Batch-size
Aug
Extra-data
Early Stopping
Tocal
nun_parana
0.23-0064
0.04
path_pors
0.54
0.45.0.50
0.54110.4
0.77.10.66 0.32:10.52
0.71.10.49
0.7110.511
0.88.1..30
0.63
Log_prod_oo__ of__d_
-0,14=0.56
0.25=00.7
0.2110988
0,4400733
0.3420.50
0.16::0.79
0.2900.78
0.55
0.09
1og_prod_ of
0.14::0.88
0.56
0.55
0.6610..59 0.68--039
0.36
0.45:0.73
0.16
ouclid init_port
0.23=0062
0..02.0.56
0.04
0,3910.72 0.16..0.5
0,54,0.60
0.53-10.22
00.40.10.33
0.13
average_ce
0.30=0.55
0.740011
inverae_nary tn
-0.12=0 00:-0280000
0.23=0097- 0.333+077777
0.35
-0.48
preb_nargin
0.50-0.53 0.45+00.6
0.29-0.96 0.76+-038
0,74+0.34
0.70-0.48
0.75+0.41
0.71+40.53
0.63
boundary, thi ckneee
15
-0.38=0.57 00.55=0050000066....
kl <l_divergenge
16
-0.36-0.60 1.36-00.000.033---0003 0.500 - 0555-00..85 -.055-0..000
0.31-
0.700-0
0.36--0.73 -0.53+00.68
-0.31
_ecal_1ip/
0.16_.0.6 0.1510.55 0.33 0,2410.76
0..59
pachayea_ rlaa
0.32-0.65 0. 10=0.54 0.30=0.95
0.10+0&3
0.15+0.54
0.35
+0.31
0.77+072
0.00
aatinated__harpan
19
-0.02:0.67 0. 0.26:0097
0.161082
0.02
0
- 1.078
0.13:10.1
40.26
estinated inv sharppess
20
0.066 0.1410..56 0.32.10.95
0.191082
0.0210.63
0
0.22-11.77
0.03.18.8
40.29
average_flas
-0.12-0.56 0. 02: - 0.48--0.88
0.14 0009 0.48
0.277 0.7--077-00.2---00
0.22+-0,74
4.
x. g_grrd_nns norm
4.39..0.3 0.22..0.59 0.52.1..85 0.5210.68
0.65
0.
0.75.14.47 0..00.0.59
4.63
v_grad norn
-0.1300.5 00.14 0.41=0091
0.03
0.14
0.65
-0 00,34
-0.03
40,34
average_ccm ce (PGD)
-0.64=0.35 40, 4: -::00:00-000000:777
0.80
0.844
34
inverse_ nargin (PC
0.57-0.48 -0. 13+0061
0.55+0084
0.13+00.2
0.39+0.63
0.28-0.81
0. 0.26++0811
0.10++0.00-00
prob_nargin (PCD)
0.6110.58 0.57.00.2
0.66.10.75
0.8810.27
0.85.0.27 0.79 0.377 0.8410.34 0.7910.50 0.76
pacbayea_ flat (PCD)
0.45=0062 0.29=00.55
0.63++078
0.33++078
0.45+0.48
0.
0.00+0.83 0.65+0.59
0.26
pettae ted_ inv iiv__iivrrrea.ao sharppessaa (PQT)
ast astinated_aaaa sharpness (PCD)
[9]
0.10:00.6 0.03::0.5
0.12:10.9
0,4011.71
0.17100.1
4..1 7... 1..
0.29.10.6 0.02
-0.03=0.66 0.
0.
0.39+0.73
0,17+0.59
-..3
20+0.79
0.20+0.788
7.0000.78-0003
0.08-0064
0.09 0.09:00.1
0.48--0.88
0.1 0.11 00.5 0.61 +0.51 0.440 0.33240.75 4.19
v_grad_porasl (PCD))
0.09 0.11..0.57
0.06.1.000
0,2510.78
0.07
0.77 0.177..000
0.08.0.79
4.12
Table 11: Numerical results of m.. for each measure, along with its corresponding standard deviation. Conditioned on average_ce (PGD) 1.5. Total 7 is same as in Table TO
Model
Traning
Early
Total
Architecture
Methods
Steps
Optimizer
Batch-size
Aug
Extra-data
Stopping
run parane path_pcrm
203=000-00010000
0.03+0001
0.0820003
0.05=0:07
0.04::0022
0.04:20000
0.04
0.6420.03
0.65 0.62:1.02 0.6410.05 0.6810000
0.65
0.63110011
0.52.1.00
0.63
Log_prod_off of_ spec
0.19=0:25
00.09
-0.06=0.000 100.000-0000 00.00.0000222
-0.0000.000---2
0.07=000--000000000
0.0000000002
-000000000000000000
-0008-0006
-0.00000000.09
log_prod_ of__
0.47-0.04
0.1610.05
0.1610.01
0.1510.04
0.1910.02
0.17:00.1
0.1110.01
euclid, init_port
0.21=0.07
0.15=00.77
0.10:0.11
0.11+0.01
0.16220.03
0.11=0003
0.11200.2
0.1720.06
0.13
average_ca
0.63_0.04
0..0110:05
0..1100.05
40.6610.06
00.6.10:055
-0.56.10.13
inverse_nargth margtt
-0.8-00099
-0.4440.22
2.47-0.12-0474-000
-0.48
prob_margin
0.6310.03 0.71.0004 0.6510.03 0.6110.04
0.6210.03
0.06 0.63110.5
0.57.0..16
0.63
boundary thi cinces
15
-0.51=0.33 40.6100.33 -0.7:20.01 00.9910.07
-0.0020.04
00.53
40.50
kl_divergence
16
-0.31 0..40:1..14
D. 0.1.....1 0.33
1.
10.1 10..8.0.30..00 0.30.1..08
00.2710.133
Lecal_lip
0.03=0.12 0.01:0023 0.17
0.0110.04
0.03
0.13
pacbayes_ flst
18
-0.03=0.19 0:0000000000000000000000000 -00000000000
0.02:0.01
0.00 0. 0.007 0.07=0.00-0
0.00
estinateen siitmated_charppeee
19
-0.26_0.02 13 0.37.00.01
0.251.0.5
-.266..
4. 15 16 22-10.1
-0.3210.27
40.26
estinated inv__iivrrrr sharpgers
20
-0.28=0.02 40.28:0.14 00.02:0.011 00.9900.04 0..29
40, 05 15-1555--6666662
2.26220111
-0.32:10.77
-0.29
average_ flat. x. grad norm
-0.17=0.05 0..222 0.02= 18 -0 200000000 -0.16=0.05
-0.0= 19 -0. 14+0.04 -0.17+0.01
-0.15
-0.64=0.04 0..6920.13 00.55 0.655...0 +0.6210.08
40 0..22.0.04 -0,57.0.14
40.63
v.grad norm
-0.34=0.03 -0.3 -0.34420.111 11 -0 00.48
-0. 0..3 -0.33 08 -0, 0. -0.30 -0.35-00.27
-0.34
average_ (PaD)
-0.77-0.03 -0.77=0.01 -0.73+00.03
00766+0.02 -0.77+0.01
8.800-002
-..77+0.000000 007--00000..2:--..0
-0.76
inverse, margin (PC
...22.0.03 0.13:.0.11
.D. 0.147
1.011 0.02
0.01 10 0.02 .0.02.0.31
4.01
prob_nargin (PGD)
0.77=0.03 0.77.00.1
0.73
0.7610.11
0,77.0.02
0.80=0001
0.77:0.00
0.72:0.08
0.76
eat pacbayes__i flat (PCD)
astinated_eaa sharpnezs (PCUJ
abarpnesee
[9]
0.01=0.04 0.09
0.26=0008 0.07
0.25 0.15 0.29 10,03
0.06
0.01
0.01 10.13
0.26=-0.7
0. 0.14
0.34
0.07.10.11
0.36-10.04
0..1910.17
0.02
0.26
x. x_grad_norra (
inv charpnese (PQD)
-0.02=0.05 0..02=0.08
0.0020.20
-0.0620004
00.04
0.
0.02:0.10-0.00
10..0.0000.144..20
0.200.0.04
0.24.00.6
0.04.10.2
0,1910..13 0.2511..10
4. 18 0.11.00022 00.17.0.088
0.19
v_grad v_grad_porr[ (PCD)
-0.12=0.04 -0.08:0.08
0.06:00.21 -0.1310.01 -0.11=0.11 40.0 0.08.0.00000 -0.1920.18
4.11
18

---

Rank Correlation (t)
No condition CE(PGD) = 1.5
0.5
0.0
-0.5
num, params
log pooo spec log prod tro euclid init norm average, ce grad norm grad, nerm inverse grrgin
path, norm
prob. margin boundary thickn ess
divergence geene
local lip pachayes flat estimated sharpn hharneess estimated, sharphess
average. flat average Ce(PGD) inverse, Thargin(PGa
prob margini (PGD)
estimated pharpnessioe (PGD)
estimated Inv www.shaapppssse Pharpness (PGD)
pactay es flat
g.radd norm(PGD)
w_grad, norm(PGD)
Measures
Figure 8: Comparison of total 7 between no condition and average_ce < 1.5. Follow- ing measures show strong correlation on the condition: path_norp average_ ce, x_grad_norn inverse _mrrgin, prop_margiiiiii marginn, and boundary. thickness.
60
60
86
RobUST aad 150
bmundary thicknees
50
40
40
1.5
40
30
prob
1.0
201
20
0.2
Gap
Generalization Gap
Generalization Gap
(a) log(path__ooreeee _orm))
(b) grob_margin
(c) boundary_thicee chicsssss
Figure 9: Scatter plot the measures showing the most increased the total when conditioned on average_co ce (PCD) < 1.5.
19

---

Table 12: Regression analysis in Fig. 10 We summarize their R for each measure.
for the target variable, the robust generalization gap g(w)
Measures
R²
Measures
R-
num num_params
0.87
0,88
path_norm
0.87
estinated _t_
Log-prod_of__et _ppc
0.88
average_
Log-prod_of_
0.88
x_grad_ --grad_norm
0.91
foclid_init_nov
0.87
k_grad_norz
0.88
average_ce 12
0.86
inverse_nant careeeemrginnnn
0.86
inverse_margi
0.86
prob_ grob_margin(PODD
0.91
prob_margin
0.87
Macbayes_flattPPA
0.89
senddarytthickeess.
15
0.86
19
0.87
k1
x1_divergence
16
0.87
aetimated__nv__harppmee (PGD)
0.86
local_lip
0.87
x_grad_morm
(PCD)
0.88
pacbayes_flat
0,90
w_grad_morai porma_oommmeee (PCD)
0.87
1.00
Absolute Emror
0.75
4.025
0.50
0.25
num, params
path norm log prod of spec log prod of fra eucldd init norm inverse margin prob margin boundary, thickness
dilvergence
local lip paabasss, flat estimated sharpness estimated. Invariant.co sharpness average flat graad norm
Inverse marginiPGO)
w L.gaal norm
prob marg (Pg)))(PDD)
estimated Invariant, sharpness(PGD)
pacbayes flat(PGD)
x g_radd norm(PGD)
grad normiPGD)
estimated SharpnessiPoo
Q1 (baseline)
mean (baseline)
Q3 (baseline)
Q1
mean
Figure 10: Cumulative distribution of absolute errors for the robust generalization gap estimation is shown. A simple linear regression model constructed to estimate the robust generalization gap, using average_ (PGD) as the baseline measure, and each measure considered as an additional independent variable. The mean (solid line), Q1 (dotted line), and Q3 (dashed line) are plotted. lower absolute error indicates that the measure is more effective in estimating the robust generalization gap.
A.3 Robust Measures with Regression Analysis
In the seminar work of 177,, the concept of an affine oracle was proposed, which utilizes linear regression assess the performance of measures. Motivated by this experiment, conduct simple linear regression analysis. Specifically, for each measure, we calculate the optimal coefficients and bias predict the robust generalization gap g(w). Considering that average_co (PGD) consistently exhibits the highest correlation across all settings, we use it as a baseline for the regression, i.e., independent variable, resulting
31 x Xaverage ce (PGD) Bo- We then consider each measure new form of measure 31 x xaverage_.... ce (PCD) + x Xmeasure+ 3n. To ensure high predictability, we perform this regression analysis with robust models with average_co (PGD) < 1.5.
In Table 12,
we calculate the coefficient of determination (R²²) for the regression analysis to evaluate the extent to which each measure explained the generalization gap. Consistent with the findings in the main paper, x_grad__ooss _orm and Lroo_margginCC (PGD) exhibit the highest R² values.
Additionally, to examine the distributional information of each measure, we plot the cumulative distribution absolute errors for each model with the mean (solid line) and the interval denoted by the first quartile (Q1, dotted line) and the third quartile (Q3, dashed line). The results are illustrated Figure TO We observe that only a few measures, such as x x_grad_moaa norm and prob_marging (PGD), provided meaningful information about the generalization gap, while the effects of other measures
20

---

Table 13: Total of generated measures from the regression analysis. Both new measures show improved performance in predicting the robust generalization gap.
Generated Mcasures
Total
-51.1111x_ppp Beiixx___rad_dde norm- 23.87 xaveraa ce (PGD) +57.32
0.81 (+0.18)
$1.17 httpk/oooommmanPoa.1 13.52 average_ (PGD) 20.60
0.78 (+0.02)
Table 14; Total of generated predictors from the regression analysis with forward selection.
#Measures
Selected Mensures
5-fold (Avg. 4 (Sd.))
average_ (PCD)) (PCDD
0.7229=0. 1301
x *-grad_norma average_ ce (PGD)
0.768 1040
x-grad_norm, average_ (PCD), pacbayes_ flat (PCD)
0.8145=0.0725
x-grad_norm, average_ ce (PGD), __grad_norte parad_norm[a]) (PGD) pacbayes_mag. flatt
0.8219=0.0730
All
0.8165=0.0866
Table 15: Regression analysis for the target variable, the test robust accuracy E(w; c, D). We summarize their Rº for each measure.
Measures
R
Measures
R
nam_params
0.10
19
0.33
path_nore log-prod_of__e _spc
0,07
estinated_ lacpeeddii___shs_aaae
20
0.41
0.14
average_
0.12
Log.prod_off
fro frr
0.14
x_grad_nore _nom
0.09
suclid_init_.d norm
0.12
w_grad_nort norm
23
0.38
average_ce [2
0.12
inverse inverse_margin __w.arriinnnnn (PCD)
13
0,04
inverse_margin
0.05
arob_margin
0.24
prob_margin
0.20
pacbayes_flee laarees__aaatte (PGD)
0.48
boundary_thickner
0.20
estimated Shaimatteeddsappeesssssee (PGD)
0.04
x1 xl_divergence
0.13
decimated__iv_aahrrppe (PGD)
0.05
local_lip
0.16
x_grad__orm
0.16
pacbayes_flat
0.40
k_grad_norm (PGD)
0.05
negligible. Specifically, x_grad grrdd_oormmmm_______ norm and prob_margit (PCD) achieve the lowest mean absolute error across all trained models.
we evaluate the effectiveness of predictors generated from the regression analysis. The predictor using x_grad_nord achieves an exceptionally strong correlation 0.81, an increase of 0.18 compared to the previous value in Table 10, The predictor using prob_marghe (PGD) also exhibits improved performance. These results suggest the potential effectively predict the robust generalization gap by combining existing measures.
In Table [3]
To push further, conduct 5-fold evaluation strategy with linear regression model to predict the robust generalization gap. Specifically, we use the forward selection to identify the most effective set of measures. Table [4] we present the results of our 5-fold evaluation, reporting the average along with its standard deviation. average_ce (PGD) is selected as a prominent predictor, followed by the selection of x__rrd__aag norm. Furthermore, our exploration identifies pacbayes_mms _maa_flaaa (PGD) and s_grad_mmg ler.grdddddmm_e (PQD) as additional effective measures, resulting in higher average T compared to using the entire feature set.
gormm
In Table T5] we conduct a linear regression analysis to predict the test robust accuracy, rather than the robust generalization gap. Compared to Table 12, overall values of R for each measure are lower. This implies that directly predicting the test robust accuracy might more difficult than the robust generalization gap.
21

---

A.4 Robust Generalization Gap with AutoAttack [1
In the main paper, we estimate the robust generalization gap using PGD10. While we acknowledge the potential benefits of stronger attacks, such as AutoAttack [TT, we mainly use PGDIO due the following reasons. Firstly, the computa- tional cost of AutoAttack is substantial. Our experimental design involves training models across diverse adversarial settings and requires adversarial examples for both training and test datasets to estimate the robust generalization gap.
70
g(w) by AutoAttack
20
60
50
AutoAttack takes 10 min per batch for WRN-34-10 on our resources. Since we have 1300 models, we need at least year to obtain all adversarial examples even with 6 GPUs. Secondly, the prevalent usage of PGD among various meth- ods. AT, TRADES, and MART use PGD as baseline during training and further adopt carly-stopping by using PGD on training or validation sets. Lastly, some measures, namely boundary ttttttioe ckness and local_lip. rely on PGD adver- sarial examples for their calculation. As these robustness mea- sures are often computed using PGD, the choice to use PGD for evaluation contributes to consistency across experiments.
15 20 g(w) PGD
Figure Scatter plot of the ro- bust generalization gap calculated by PGD and AutoAttack. The color of each dot implies the test robust accu- racy on CIFAR-10.
However, we here highlight the potential benefits of utilizing AutoAttack future work. Fig.
we calculate the gap g(w) using AutoAttack for 30 models on CIFAR-10 in RobustBench [12] While the gaps calculated using PGDI0 and AutoAttack exhibit an almost linear relationship, there
a few exceptions (2 out of 30): 'Ding2020MMA'' [44] and "Sitawarin2220mproving [55]. We
leave this question open for further exploration.
A.5 Importance of Model Architecture in Norm-based Measures
150
400
ResNet18
100
125
log prod of fro
300
WRN28-10
WRN34-10
log_ee prod
200
50
100
Robust Generalization Gap (%)
Robust Generalization Gap (%)
(a) fog_prod_of_spet
(b) log_prod_oo_fre
Figure Distributions of norm-based measures for model architectures. For each model architecture, the range of measure extremely varies.
In Fig. 12 we plot log
proodooo_d_ooo ___eecc__pc_c__eee_____pp____ and srs_prod_of fro for each model architecture. We can observe that the range of each measure extremely varies with respect to the used model architecture. When the model architecture is fixed, they show some correlation with the robust generalization gap as described in the main paper.
A.6 Early Stopping and Estimated Sharpness
In the main paper, we discussed that estimated_ samaattddsharppesssssssess exhibits a significant negative correla- tion when early stopping not used. Fig. 13 shows the importance early-stopping for estimated sharpness measures. Compared to Figures and 13b] when early stopping is employed, the correlation approaches zero as shown in Figures 13a and 13e This results supports the observation of prior study [56) that the importance of carly stopping in the analysis of flatness.
22

---

endimated jnwaarrr sharpmens
essimatecd Inverian: sharpmess
0.20
15 85
estimattdd sharpress
Text Robuttncis (1)
Test Row.Ruuuuttnn (N)
Robun (50)
-50
0.15
0.15
0.10
0.10
0.05
0.05
10
10
20
30
40
20
40
60
20
40
Robust Generalization Gap (96)
Robust Generalization Caap (%)
Robust Gereraliaation Gap (S0
(a) estimated astimateed_ahrppnes w/ early-stopping
(b) estimated _il__iiisaaaaaa sharpneas
(c) estimated, inv si____ii__a____ sharpneas intasppppeensssseeeeseeee w/ early-stopping
w/o early-stopping
Figure 13: Scatter plot of the estimated sharpness measures for the use of early-stopping. The same condition, average_com ce (PGD) < 2, used as Fig.
B
Measures
In this section, we introduce the concept of each category of measures in the main paper, then explain the details of each measure and their mathematical definitions. Here, denote W; the weight tensor of i-th layer, following [17]. Given the number of layers d, the whole trainable parameters are vec(W, W2."",
denoted
weight norm-based measures are expected to be correlated with generalization performance. Among them, the product Frobenius norm [46] LLog-prod_oss of _f__fro, the product of spectral norm and path norm (path_nore) norm) have been considered as important mea-
Weight-norm. Based on some theoretical frameworks such as PAC-Bayes 43 47 38
7 (log_prod_onl borod_of_ppeeppeepeeepeepppee__e sures [27 [7]. Furthermore, the distance to the converged weight from the initial weight (euclid init (ant_norm]) is also used to estimate the generalization gap [27]. Liu et al. 40) also argues that this distance can be used to judge the difficulty of optimization in adversarial training. num_ auu__params.
Σ
ka-i1a+ 1),
where ej is the number of channels and h; is the kernel size at layer In the experiments, calculated aum_paramm by adding the number of parameters of all convolutional and linear layers. num_params is fixed value when model architecture is given. >> path_ahaa norm.
Efer
fur (1)[i).
(8)
where "2 is the element-wise square operation and f(-][( is the i-th logit output of the network. By setting all input variables 1, this measure captures geometric properties of optimization under scale-invariant characteristics. log_prod_oog spec.
d
log |
(9)
where || 112 is matrix Lo-norm, i.e., the largest singular value of each layer. log_prod_off of spec.
log II
(10)
where || || a Frobenius norm, i.e., the square root of the sum of the squares of the weight matrix. >> euclid, leclldd_iin init_nnonnntt norm.
d
Σ
w
w I|
(11)
where w, vec(W.) and the initial weight of i-th layer w.
23

---

Margin. Margins are also actively researched measures to estimate the generalization gap [26]] For instance. the 10-th percentile of the margin values in the oulput space on the training set (inverse, _ is often used to measure the generalization bound for neural networks 500 27 In terms of adversarial robustness, most of attack methods and defense methods utilize the probability margins morgin).......... Yang et al. [67 further proposed new measure, called boundary thickness (boundary_thice thices) that is a generalized version of margin and argued that it is highly correlated to the robust generalization gap.
average_ce.
Bryy (ar, w),
(12)
where Lee the cross-entropy loss. geverse_marginn
1/72, where y is 10th Otth-percentile of (o(f(x, w)) - maxizy (f w));} } for all æ, y. with the sigmoid function (.).
(13)
grob_marginn
Br o(f(x, w))
(14)
houndary_thickknee
Em |1:e 00122 //' Il(a < o(0, A) < bydd arg max (f(e, w)); & arg max (s((" [(",,)"),,
(15)
where 1(- is an indicator function, g(x, x", X) off(Ax + (1- \)o", w)); of +
A)æ² w)))". " ú arg max; of (a, w)); 1j" arg max; of ww)); and a, b are the hyper- parameters that controls the sensitivity of the boundary thickness. Following [67], we find x by
using PGD10 with Ly-nomm, c 1, a 0.2, then a 0. 0.75, and batch size 128. A higher value of boundary_ thi ckness implies larger margin in the output space.
line of work has focused on the smoothness for achieving adversarial robustness in adversarial training. Most simply, the KI. divergence between benign and adversarial logits (kl s<l_divergence) of TRADES (7]] can be regarded as a smoothness regularization due to its logit pairing. Yang et al. [66] investigated the theoretical benefit of local Lipschitzness Local_lipp and demonstrated that its value estimated on the test dataset correlates with the robust generalization gap.
Smoothness. Based prior works 6 10 28
kl_ divergence.
En
KL(S(x, ()), f(x", w)) | Se
max
(16)
where KL is KL-divergence and the maximization is conducted by PGDIO with the step-size 2/255. A lower value of kl, divergence implies that a model outputs similar outputs for both benign example and adversarial example.
local _ip.
[ w) - f(x", w)ll
Ex
(17)
||0-==
x=
where the maximization conducted by PGDIO with the step-size 2/255. This is the empirical version of the local Lipschitzness, resulting lower value of local_lip implies smoother model.
Flatness. Flatness a recently focused measure the generalization domain 43 30
Recent works
argue that flatter minima yield better generalization performance than sharper minima. The common strategy to achieve flatness is to minimize the estimated sharpness [I8 73), which the difference between the current loss and the maximum loss for given neighborhood (estimated_ sharteesd_srrpessss Kwon al. [36] investigated the scale-invariant sharpness (estimated_iin ii__iin__haa jiii___aarrpeess) Note that other diverse concept of estimated sharpness is actively researched in recent works R3 5 34
Adversarial weight perturbation (AWP) [4 also has dramatically improved adversarial robustness by minimizing the loss of perturbed weight. Recently, Stutz et al. (56) has demonstrated that their
24

---

proposed measure, average flatness (average_flat). is effective for estimating robust generalization gap.
>> pacbayes flat. Based on PAC-Bayesian framework
43 Jiang et al.
27 proposed a simplified
version of IAC-Bayesian bounds as follows:
1/o'
(18)
where o' is the largest value such that (u + u; { v)) E(w: {æ, [e,y < 0.1. Here, Uy ~ (0, o') and {ww; ()) denotes the error on the given set {x, y). Thus, higher value of pacbayes_ful flat implies flatter optimum in terms of error landscape.
estimated__harpnesss
E
E
max
Loe (f(x,, Coe (f ( Lox (f(x,))))))
(19)
w1125p
we calculate the estimated sharpness with single-step ascent with 0.1. A higher value of estimateddd ssstimmae__sharpeesss implies sharper optimum in terms of loss landscape.
Following 73
estimated l sharpness.
Lee w w)) - Leel (f w)
Exs .
max [|Tw U112Sp
(20)
where Two is ||/l| i.e., element-wise adaptive sharpness, and Lee the cross-entropy loss. Similar to the estimated sharpness, calculate the estimated invariant sharpness with single-step ascent with = 0.1 (36.. A higher value of est estimatte iimtteddddd_e inv ieviiivssaaae __sssapeassssss implies a sharper optimum in terms of loss landscape.
average_ flat.
Fyn (w))
max (e un +)) -|
max 1
Lools(x", un) (i))))
(21)
where (w)) = (w + 1 S pl 12/1 layers i). We use PGD10 with 8/255 for both inner maximizations. Following [56), we take 10 random weight perturbations with p 0.5. A higher value of average_ flat implies sharper optimum in terms of adversarial loss landscape.
Gradient-norm. Gradient-norm with respect input or weight is also a consistently researched area in terms of generalization. Recently, Zhao et al. (72) also demonstrated that regularizing the several tasks. Additionally, there are few works that emphasize the importance of regularizing the gradient norm 44). The gradient norm of inputs (x g_aadddooom) also can have underlying correlation between the robust generalization performance. Prior works utilized the input gradient for analyzing adversarially trained models and generating adversarial examples 35 16
gradient norm of weights (w_grad_novl _orad_norm) can achieve sufficient improvement
of weights [4))44.
> x
Eny Lo w)
(22)
w-grad_norm.
En.y [| V (f(x, w), v)11
(23)
Comment on batch normalization fusion Here, we provide comments on some further discussed things when estimating the above measures. In previous studies (27 17 it has been observed that considering batch normalization (batch-norm) layers can have an impact on common general- ization measures, such as sharpness [15]. To address this issue, the batch-norm layers and other moving statistics were fused with the preceding convolution layers before calculating the values of generalization measures. Thus, when estimating (num_paa goum_params, path_ __orr, Log_prod___ _f__pec_ log-prod_off pprod_offfro, pacbayes_flee ffatf,tt,, ), we apply batch-norm fusion to all ResNet blocks. However, for certain blocks, such as pre-activation ResNets, where the batch-norm layer is placed at the begin- ning, the fusion cannot be directly applied. To ensure consistency, we add an identity convolutional layer front of all batch-norm layers that do not have the preceding convolution layer. While there are various batch- norm fusion (or batch-norm folding) techniques, including those related to
25

---

generalization [27 17 51. quantization 669 37 49 and memory optimization domains 8 19 2]
there no precise solution address this problem the context of various model structures (ResNets. Nres ViT, etc.) and activation functions (SiL.U LeakyReI.U, etc.), which we leave as topic for future work.
C Training Details
In Section 3 1,344 models were trained using the CIFAR- dataset with 8/255. We here
provide the detailed training settings. We followed the common settings used in 42 71 48 21]
Given the higher Rademacher complexity [68 and larger sample complexity 54 of adversarial training, data augmentation [2]] and the utilization of extra data ( can significantly improve the adversarial robustness. Therefore,
also considered the impact of augmentation technique, including RandomCrop with padding 4 and Rundommloriizouuulii.. as well as the use additional
data collected by Carmon et al. (9).
Regarding model architectures, we employed three different models: ResNet18 (3), WRN28- 10 [70]. and WRN34-10 [70]. These models have been widely adopted and serve as benchmarks for evaluating the stability and performance the majority of models trained on CIFAR-10 10 in RobustBench 12 consist of WRN28-10 (15) and WRN34-10 (14) among the 63 available models.
adversarial training methods. It is worth noting that
For training methods, we considered four different approaches: Standard, AT TRADES [71]. and MART [61] Notably, AT, TRADES, and MART have been shown outperform other variations
by incorporating various training tricks (48) and integrating recent techniques 9 64 For all methods, generated adversarial examples using projected gradient descent (PGD)
During training, single-step approximation of the inner maximization in Eq. (I) can lead to faster adversarial training, but may suffer from catastrophic overfitting 663 31.. On the other hand, a large number of steps leads to stable robustness, but requires heavy computational costs. Therefore, we considered both 1 and 10 steps for each adversarial training method.
In terms of optimization, we used SGD with momentum 0.9 and weight decay of x 10-4, and a step-wist wise learning rate decay performed epochs 100 and 150 with decay rate of 0.1. In all the experiments, we trained the models for 200 epochs. As highlighted by Pang et al. [48). the batch size used during adversarial training has been found affect its performance. Thus, varied the batch size among (32, 64, 128, 256). Additionally, we also considered adversarial weight perturbation (AWP) (64), which can improve the robust generalization performance of models. AWP belongs to the class of sharpness-awand minimization methods [18 36). This can be formalized as follows:
min
maax aaddv =l(S
ax u + v),
(24)
where (w)) = (w + v b S pl/ws/12 Vi-th layer). As described by Wu et al. (64 x adv is calculated based on the non-pen non-perturbed model f(w), and single step of maximization with respect to is sufficient to improve robustness. We used the best-performing value of 5 x 10-3 from [4]
26


Reviews:
[{"review_id": "FYY1DfzXir", "reviewer": "Reviewer_PVgT", "summary": "The paper conducts a large scale correlation analysis for adversarially trained models to identify which existing robustness-related measures are predictive of robust overfitting. The correlation analysis is performed over large set of relevant dimensions (including model architectures, training procedures, attack strengths, optimizers, batch sizes, etc.). The overall experimental methodology seems fine.", "questions": "Do the key findings hold when considering only trained models which have high train & test robustness?\n\nTable 1 is not very easy to read at a high-level; have you attempted colour-mapping it (i.e. like a heatmap)?\n\nHave you tried training a classifier (e.g. just an MLP) using all measures as input features to predict a discretized robust generalization gap? Depending on how well the classifier generalizes, this could be useful for as a direct optimization proxy to improving the predicted robust generalization gap.\n \nIt seems a bit awkward to not qualify \"measures\" with a prefix; for example, in [32] the considered measures are named \"complexity measures\". E.g., have you considered something like \"robustness measures\", \"proxy measures\" etc.?\n\nSee also the section on weaknesses above.", "limitations": "Limitations are discussed briefly.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The experiments uncover some interesting findings regarding which metrics can be predictive of robust generalization.\n\nA definite strength of this work is that all code and trained models are made available online -- this will enable the adversarial ML community at large to build on the analysis explored herein with new measures, training methods/settings/etc.", "weaknesses": "1. A major significant weakness of this work is its title. The title is non-descriptive and incendiary. For an academic publication (i.e. not a blog post, news article or social media post) this seems wholly unacceptable. Please update it to something suitably descriptive instead.\n\n2. How related this work is with reference [23] (Jiang & Neyshabur et al., \"Fantastic Generalization Measures and Where to Find Them\") is severely downplayed. Essentially, this work follows almost the exact same playbook as [23]. A subsection on the relationship of this work with [23] should be included in this paper. For example, a small step in this direction is section 3.3 which already points towards the evaluation metrics being wholly adopted from [23] while changing the notation.\n\n3. The attack used for measuring the robustness of trained models on the test set is just PGD-10 and this is not strong enough for a proper evaluation of robustness. AutoAttack (at least), and e.g., Square and Multi-Targeted attacks should've been used instead to better approximate the models' adversarial robustness. To root out training procedures that are affected by gradient obfuscation, using just a simple gradient-based attack (with just 10 steps; no restarts; etc.), like PGD-10 for evaluation, is insufficient. What slightly ameliorates this point is the inclusion of analysis of models from RobustBench (e.g. Figure 5).\n\n4. Only when models are actually robust on the train set it is meaningful to measure the robustness gap; including the bottom left part of Figure 1 seems like it would just introduce noise into the whole estimation process.\n\n5. Two important sections, including the Key Findings and Section 3.3 (which introduces the evaluation metrics) should be re-written much more clearly. For example, key finding 1 states \"traditional metrics may be ineffective due to the high sensitivity with respect to training setups\" -- this is very unclear? What are traditional metrics? Saying that something \"may be ineffective\" is so imprecise that it becomes not useful for the reader. In key finding 3, it is said that \"flatness-based measures [...] perform poorly and even sharper minima can exhibit a lower robust generalisation gap\" -- what does it mean for sharper minima to exhibit a lower robust generalisation gap? Do you mean something like: \"neural networks which have sharper minima exhibit a lower generalisation gap in 70% of cases we evaluated?\". Please rephrase. For section 3.3, I had to carefully read [23] to try to be able to parse what section 3.3 is stating -- please clarify all the writing from 147-159; e.g., the point in lines 144-145 is completely unclear by itself; the point in lines 158-159 has typos and is unclear; similarly, what does \"generalization gap varies sensitively with respect to training setups\" mean? Could you motivate why the pi_k metric is effective (line 152)? An explanation of why this might be the case only shows up in section 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful assessment of the strengths and weaknesses of our paper. Your feedback provides valuable insights that help us improve the quality and clarity of our work. Below, we provide a detailed response addressing each of the weaknesses  [W#] and questions [Q#] raised with all references aligned to the main paper:\n****\n\n**[W1]** We are sorry to hear that the title of the paper is unacceptable. The initial title was motivated by [23] (Jiang & Neyshabur et al., \"Fantastic Generalization Measures and Where to Find Them\"), but we understand the need for a more accurately descriptive title that better reflects the content and objectives of our research. **We have taken this feedback seriously and are in the process of revising the title** to ensure that it appropriately represents the scholarly nature of our work. One of the potential titles that we are currently considering is 'Evaluating Generalization Measures in Adversarial Training Framework.' We believe this title encapsulates the core focus of our research, highlighting the evaluation of generalization measures within the context of an adversarial training framework. However, if you have any further suggestions or ideas for a more fitting title, we would greatly appreciate your input.\n\n****\n**[W2]** We appreciate the reviewer's feedback regarding the relationship between our work and the reference [23]. Indeed, our study draws inspiration from the valuable insights presented in [23], which conducted an evaluation of complexity measures in the standard training framework. However, due to space limitations in the main paper, we regrettably could not thoroughly delve into the connection between our work and [23]. In response to the reviewer's suggestion, **we introduce the following new subsection that explicitly addresses the connection between our study and [23] based on our answer to the first weakness in the global response:**\n\n\n\\subsection{Comparison to Jiang et al. [23]}\nThe pioneering study [23] explored the empirical correlations between complexity measures and generalization, with a primary focus on the standard training framework. Our main contribution is delving specifically into the realm of robustness measures within the adversarial training framework—a context having different generalization tendencies even with the same measures and experimental settings, as demonstrated by prior research [43, 52]. Of significance is the observation that the metric $\\psi_k$ proposed in [23] has limitations in accurately capturing the effectiveness of robustness measures due to its high variance with respect to training setups. By introducing a new metric, our work enhances the understanding of when and how robustness measures correlate with robust generalization. Moreover, while Jiang et al. [23] employed customized parameter-efficient neural networks, we adopt widely-used model architectures such as ResNets, thereby providing insights that are not only relevant to recent research but also offer more practical implications.\n\n****\n**[W3]** While we acknowledge the potential benefits of AutoAttack, the constraints outlined above guided our decision to use PGD-10. **We believe that Appendix “A.4 Robust Generalization Gap with AutoAttack”**, which demonstrates the comparison between the robust generalization gaps calculated using PGD and AutoAttack, **can help alleviate some of the concerns you've raised.** However, to address your concern in detail, we here explain three reasons why we primarily used PGD-10.\n\n**Firstly, the computational cost of AutoAttack is substantial.** Our experimental design involved training models across diverse adversarial settings and required adversarial examples for both training and test datasets to estimate the robust generalization gap. However, with AutoAttack, it takes 10 min/batch for WRN-34-10 on our resources. **Since we used 1300 models, we need at least 1 year to obtain all adversarial examples even with 6 GPUs.** Thus, the resource-intensive nature of AutoAttack led to impractical computation times, making it infeasible to execute extensive experiments.\n\n**Secondly, the prevalent usage of PGD as a baseline during training** among various methods. Indeed, all popular adversarial training methods, such as AT, TRADES, and MART, use PGD as a baseline during training, and thus early-stopping (or other similar techniques) is adopted with PGD on training or validation sets. This led us to consider PGD as a more practical choice for providing practical insights.\n\n**Lastly, the specific robustness measures we employed, namely boundary_thickness and local_lip, rely on PGD adversarial examples for their calculation.** As these robustness measures are often computed using PGD, the choice to use PGD for evaluation contributes to consistency across our experiments.\n\n****\n**[W4]** In the main paper, we mainly conducted experiments on all possible models to comprehensively validate the robustness measures across a wide range of models. However, we also have shared a curiosity about the implications of our analysis on models with high robustness. **To verify this, we have provided “A.2. Focusing on Adversarially Robust models” in Appendix.** Specifically, we selected models by conditioning on ‘average_ce(PGD)’ ≤ 1.5. Upon analyzing these adversarially robust models, we observed distinctive behaviors in certain robustness measures (’path_norm’, ‘average_ce’, ‘x_grad_norm’, ‘inverse_margin’, ‘prob_margin’, and ‘boundary_thickness’). For example, margin-based measures displayed more pronounced negative correlations, while the effectiveness of ‘boundary_thickness’ in estimating the robust generalization gap became effective. Please refer to Appendix A.2. for more detailed explanations and graphical representations of our findings.\n\n****\n*The remaining weakness and questions will be addressed in the continued official comment due to character limitations.*"}, {"review_id": "yUDt6FcL9e", "reviewer": "Reviewer_2BXc", "summary": "The authors propose a large-scale empirical analysis of the impact of several potential indicators of robust generalisation. This paper can be seen as a metaanalysis of many of the indicators proposed in the literature.", "questions": "1. I would assume that some measures only make sense within a given model with everything fixed except the weights. For example, flatness vs generalisation. There’s quite some literature in the non-adversarial setting arguing that flatness is correlated with generalisation. However, even within a model, with just the weights varying, it seems not to be the case. Correct me if I’m wrong but this is what Table 4, column “steps” indicates for example (since in that case, two models compared have the same loss landscape but are at a different location in weight space due to not having had as many steps). Do you have intuition why in the robust case this intuition doesn’t hold ? Mine is that it could be tied to some papers arguing that flatness in weight space and input space can be inversely correlated (e.g., “Adversarial Training Makes Weight Loss Landscape Sharper in Logistic Regression” by Yamada et al. 2021). It could also be related to using the training set instead of a validation set to evaluate flatness.", "limitations": "I would specify in limitations the choice of dataset, models and attack. And if relevant, the fact all measures were evaluated on the training set if that is correct.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. I appreciate a paper aiming to test and/or reproduce at a larger scale of experiments the viability of certain measures as indicators of the robust generalisation gap. Many papers claim that Z can be explained by X or Y in some particular setting, while others fail to observe that. This kind of study helps evaluate on a more equal footing several of the proposed correlated factors of robust generalisation. I believe this is always helpful for the research community as it helps develop intuition that might lead to novel methods. \n2. The paper is generally well-written and is quite accessible (see in “Typos and suggestions” questions/recommendations for fixing a part that can be improved; I understood out of intuition what the metric would be but it’d always be better to clarify that part since it is central to your results).\n", "weaknesses": "1. Intro is missing references (in fact, it only has 1 !). References are not optional there in my opinion because they contextualise your paper/motivations. I don’t think refs should come later; plus on your end it’s a matter of adding 10-15 references that are already in your bibliography in section 1, so it can be easily fixed.\n2. Only evaluated against PGD on CIFAR10 (that is ok in itself ! It should just be clarified in the claims). Please specify the norm ($L_\\infty$) and indicate that (attack choice and dataset) when you summarise your claims (abstract/intro/conclusion) since datasets and the choice of attack likely influences the results.\n3. Clarity is generally good but can be improved in 3.3. See “Typos and suggestions”.\n4. Correct me if I’m wrong, but it appears the measures are evaluated on the training set, and not a validation set. If so, could that be stated explicitly somewhere ? I believe that’s **very** important (and should also appear clearly in summaries of your results/methodology), especially since many notions/intuitions on flatness may apply better to a validation set, which could lead to different results. See my question, where this could be a possible explanation.\n\n\n**Typos and suggestions:**\n1. L134-135: “each $\\Theta_i$ corresponds to a tuple of training parameters” to clarify that they’re not individual parameters. If they are individual parameters, then the sentence “a search space $\\Theta$ = …” needs fixing because the right hand side of the equality is not a set (in fact, the right hand side needs fixing anyway to indicate a set).\n2. L146: isn’t $\\Theta_k$ a tuple of hyperparameters, and $\\theta_i$ a particular hyperparameter within such tuples ? I believe eq 5 and/or L146 could benefit from some clarifications. Namely, about what exactly $\\theta_k$ vs $\\Theta_k$ is. \n3. Table 1: I would also indicate in the caption what (PGD) means. Also, indicate what columns correspond to (yes it’s in the text but it’s always better when figures are as self-contained as possible in my opinion).\n4. Table 3: average_ce PGD x Model architecture should be bolded too.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful assessment of the strengths and weaknesses of our paper. Your feedback provides valuable insights that help us improve the quality and clarity of our work. Below, we provide a detailed response addressing each of the weaknesses  [W#], questions [Q#], and limitations [L#] raised with all references aligned to the main paper:\n****\n\n**[W1]** We appreciate the reviewer's valuable feedback regarding the references in the introduction section. During the refinement of the structure and its flow, the most of sentences containing references are relocated to Related Work. We will ensure the inclusion of references, such as the various phenomena [32, 43, 52, 41], measures [50, 51, 44], and relevant additional sources within Introduction.\n\n****\n**[W2]** We greatly appreciate your feedback. We agree that the importance of providing clear information regarding the dataset, attack choice, and norm used in our evaluations in the main paper. This will enhance the transparency, readability, and contextual understanding of our paper. We will definitely provide a detailed explanation of these key details in the Introduction and Methodology. We again appreciate the reviewer's thoughtful input.\n\n****\n**[W3]** Thank you for your detailed remarks. We will update all typos/suggestions and make sure to remove possible other ones.\n\n****\n**[W4]** We regret any confusion that may have arisen as a result. We would like to clarify that, apart from the weight-norm measure, all other measures are indeed evaluated on the training set, as established in prior works [22, 23, 51]. Although some of the settings are presented in Appendix “B. Measures”, we acknowledge that it has not been mentioned in the main paper. In our revised manuscript, we will take measures to prominently highlight the evaluation of measures on the training set. We will clarify the settings and add detailed settings at the beginning of “3. Experimental Methodology” and “4. Experiments”.\n\n****\n**[Q1]** First of all, your observation is correct. In the past, the statement that \"enhanced flatness corresponds to improved generalisation\" had been widely accepted in the deep learning community [13-15]. Indeed, Sharpness-aware minimization have demonstrated superior generalisation performance across various models and tasks, including adversarial training [15, 49]. However, **recent research [4] has challenged this statement** by revealing that flatness does not correlate well with generalisation in standard training framework, particularly within modern practical scenarios. Similarly, our work also finds that flatness does not correlate well with robust generalisation in adversarial training framework. Based on these observations, **we can conclude the following statement – that flatness should be considered as an objective to pursue during optimization, rather than a direct evaluation metric** for predicting a model's generalization performance.\n\nAdditionally, we greatly appreciate your insightful observation and the reference you provided, \"Adversarial Training Makes Weight Loss Landscape Sharper in Logistic Regression\" by Yamada et al. 2021. This paper indeed offers an interesting observation, both theoretically and empirically, on the relationship between perturbations during adversarial training and the sharpness of the loss landscape. Based on their observation, we recognize a potential connection between the observed high values of robust generalization gaps depicted in Figure 1 and the sharpening effect on the loss landscape resulting from larger perturbations during adversarial training. **Indeed, we also confirmed that adversarially trained models tend to exhibit sharper loss landscapes when compared to standard trained models (refer to Figure S2 in the PDF in global response).** We will definitely add a new subsection in our paper.\n\nIn assessing the flatness measures on the test set, we conducted an evaluation for both 'estimated_sharpness' and 'estimated_invariant_sharpness’. However, as shown in the below tables, there is **NO** significant correlation between test flatness and the robust generalization gap.\n\n| $\\pi_k$ | Model | Method | Steps | Optimizer | Batch | Aug | Semisup | flag | Total $\\tau$ |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| estimated_sharpness(test) | 0.10±0.01 | 0.15±0.03 | 0.12±0.20 | 0.09±0.07 | 0.08±0.04 | 0.14±0.05 | 0.10±0.03 | 0.15±0.03 | 0.09 |\n| estimated_invariant_sharpness(test) | 0.12±0.01 | 0.16±0.05 | 0.09±0.18 | 0.13±0.06 | 0.12±0.04 | 0.17±0.02 | 0.13±0.00 | 0.15±0.05 | 0.12 |\n\nCertainly, our analyses could be extended to test examples (or validation examples), but for our study, we chose to focus on train examples. This choice aligns with the approach outlined in [23], which argues that “the most direct and principled approach for studying generalization in deep learning is to prove a generalization bound which is typically an upper bound on the test error based on some quantity that can be calculated on the training set.” Moreover, it is noteworthy that some researches often use the statement that \"superior measure value lead to better generalization\" to support the claim that \"minimizing (or maximizing) measure value during training results in improved generalization,” which is not always true as demonstrated in [4]. Most importantly, the theoretical analyses in prior work [23, 49, 50] have often used the framework of PAC-learning that basically uses the training set for establishing upper bounds for the generalization gap. In this regard, we believe a comprehensive assessment of measures on the training dataset is relatively important to avoid wrong conclusions or misuses, such as employing these measures as optimization targets or early-stopping criteria.\n\n****\n**[L1]** We sincerely hope that the responses provided above address the outlined weaknesses and limitations. Thank you once again for your insightful assessment and guidance."}, {"review_id": "i6ifPokC1x", "reviewer": "Reviewer_vGwc", "summary": "This paper studies the correlation between previously proposed robustness measures and the robustness generalization gap. It studies a large number of models with different architectures and training parameters and shows that some prior beliefs about the usefulness of popular robust generalization measures are not well justified under such comprehensive study.", "questions": "1. On several occasions, it is mentioned that evaluating solely on the test set is undesirable. It’s also used as an explanation for obtaining opposite results to (Yang et al., 2020) but is not fully developed. Why is evaluating the metrics solely on the test set an issue?\n2. I found it a bit difficult to understand the purpose of $\\pi_k$, the new metric the paper proposes. It seems to be trying to handle cases similar to the Simpson’s paradox, where $k$ fixes a group that has a positive trend when considered in itself, but such that when all groups corresponding to different values of $k$ are considered, the trend becomes reversed. Would this be a valid explanation?\n3. (Jiang et al, 2020) show that sharpness-based measures are some of the most predictive ones for the standard generalization gap, while this paper shows that they are not very predictive for the robust generalization gap. How do you reconcile these two seemingly opposing results?\n4. In App. A.1, it is argued that there are measures with high correlation with the test robust accuracy but not the robust generalization gap. This is counterintuitive, so what would be possible explanation? High rank correlation with low $R^2$ for linear analysis doesn’t necessarily mean that the measures are not predictive, but rather that the relationship may be highly nonlinear.", "limitations": "I've identified no limitations or ethical concerns. \n\nHowever, here are some suggestions:\n\n1. Maybe instead of writing the mean and confidence intervals in Tabs. 1,2 and 3, you could show horizontal plots of the confidence intervals? That could make it much easier to visually compare the different effects.\n2. The measures are only defined in the appendix. I understand that this is for space purposes, but it is difficult to follow your Experiments section without knowing what all these names refer to. I’d recommend saying a couple of words each time you mention a new measure, keeping the formal definitions in the appendix. I think this would improve the readability.\n3. Line 199, there’s an unnecessary “to”.\n4. Line 699, “maximization” should be “maximizations”", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper comprehensively studies the ability of different robustness measures to predict robust generalization gap and highlights cases where their findings are at odds with prior work and previously held opinions.\n2. The methodology and the experiments are clearly and well described.\n3. The paper's findings can inform future experimental and theoretical work on the understanding of the robust overfitting problem.", "weaknesses": "1. The paper can benefit from further discussion on its findings. For example, the authors say that “margin maximization in adversarial training methods should be revisited” but it isn’t clear why it wouldn’t work. Or that “`boundary_thickness` is more applicable for models trained with AT than to other adversarial training methods” but no intuition as to why that’d be the case is offered. Similarly, they say that “smoothness does not guarantee low robust generalization gap”, which is at odds with the prior work they cite but there is no explanation as to why this discrepancy might occur.\n2. Methodologically, the paper seems quite similar to (Jiang et al., 2020). However, (Jiang et al., 2020) seem to have a more comprehensive analysis which attempts at explaining the possible causal factors for the generalization gap. I think this paper can benefit from such a deeper analysis as well.\n3. The paragraph starting on line 266 is sends a very conflicting message about the usefulness of `x_grad_norm` for predicting the robust generalization gap. It seems to be saying that it is both highly positively and highly negatively correlated. Perhaps a rewrite for clarity might help.\n4. Overall, I found it difficult to understand what are the main takeaways from this work. The conclusion says “Our results suggest practical guidelines for robust generalization” but I am not sure what they are. In several occasions the paper criticizes or is at odds with prior work, but does not provide sufficient evidence as to what are the causes of the discrepancies. I think the paper might benefit from sending a more clear message of what these guidelines are and why some of its findings are at odds with prior work.\n\n\nReferences:\n\nYiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. 2020\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful assessment of the strengths and weaknesses of our paper. Your feedback provides valuable insights that help us improve the quality and clarity of our work. Below, we provide a detailed response addressing each of the weaknesses  [W#], questions [Q#], and limitations [L#] raised with all references aligned to the main paper:\n****\n\n**[W1]** We sincerely apologize for the lack of discussion on our findings. We certainly agree that more discussions and explanations could benefit the novelty of our paper. To address your concerns, **we have undertaken a comprehensive analysis of 15 additional papers** related to “margin maximization”, “boundary thickness”, and “local Lipschitzness”. **We kindly request the reviewer to refer to our second response to \"Weaknesses\" in the global response for a more comprehensive explanation.** The forthcoming revised version of our manuscript will include the discussions, which will not only address the concerns but also provide a more robust foundation for our findings. Thank you for your diligence in providing these insights.\n\n****\n**[W2]** Thank you for your insightful comment regarding the suggestion of experiments on causal factors. In response to your suggestion, **we have performed an additional analysis in Table S1 available in the global response PDF,** focusing on the calculation of the conditional mutual information between robust measures and the robust gap. The cardinality of the set of hyper-parameters $S$ is less than or equal to two (i.e., $|S|\\leq2$), in accordance with (Jiang et al., 2020).\n\nWe here highlight new findings that align well with the results outlined in the main paper. Specifically, average_ce(PGD) consistently exhibits a significantly high conditional mutual information $\\hat{I}(V_\\mu, V_g|U_S)$ and shows the highest value of $\\kappa(\\mu)=0.409$. This indicates the concrete connection between average_ce(PGD) and the robust generalization gap. Similarly, prob_margin(PGD) shows a high value of $\\kappa(\\mu)=0.387$. local_lip, estimated_sharpness, estimated_inv_sharpness, and average_flat show a low $\\hat{I}(V_\\mu, V_g|U_S)$ and $\\kappa(\\mu)$, which is consistent to our observation. Notably, pacbayes_mag_flat(PGD) shows a decent $\\kappa(\\mu)$, yet it exhibits a relatively low $\\hat{I}(V_\\mu, V_g|U_S)$ for Steps, which is the important factor in discriminating robust and non-robust models, as discussed in Line 255. x_grad_norm yields a decent $\\kappa(\\mu)$ compared to other measures, as observed in Table 5 and Figure 5.\n\nWe appreciate your feedback, and by incorporating these deeper analyses, we believe our paper now provides a more comprehensive understanding of the underlying causal factors driving the robust generalization gap.\n\n****\n**[W3]** We apologize for any confusion caused by the phrasing in the paragraph starting on line 266. Allow us to clarify the relationship regarding the usefulness of x_grad_norm in predicting the robust generalization gap. Specifically, x_grad_norm exhibits a high negative correlation with the robust generalization gap compared to other measures. In practical terms, this indicates that models characterized by a lower robust generalization gap tend to have higher x_grad_norm. We will provide clearer paragraphs in the revised version, taking into careful account all the valuable feedback provided by the reviewers.\n\n****\n**[W4]** In the statement \"Our results suggest practical guidelines for robust generalization,\" the term 'practical guidelines' is intended to emphasize that our study verifies the viability of certain measures with a larger scale of experiments. **As Reviewer 2BXc said that “[…] Many papers claim that Z can be explained by X or Y in some particular setting, while others fail to observe that,”** we believe all our observation (whether certain measures correlated or not correlated to robust generalization) can be helpful for the research community. We have highlighted particularly noteworthy guidelines in blue, which we believe offer substantial insights for future investigations. As this paper primarily aims at an empirical evaluation of measures' effectiveness in predicting the robust generalization gap, our contribution lies more in the realm of empirical findings, similar to [17, 37], rather than being theoretical. **Furthermore, we would like to argue the careful usage of statements such as \"Model A is superior to Model B because Model A has a better measure value than Model B,\" a frequently employed in recent literature.** However, we understand that a lack of discussion on why some of our findings are at odds with prior work. We hope our response to the previous weakness (W1) can resolve this issue. Again, thank you for your valuable suggestion, which will undoubtedly contribute to the depth of our paper.\n\n****\n**[Q1]** As demonstrated in [23], “the most direct and principled approach for studying generalization in deep learning is to prove a generalization bound which is typically an upper bound on the test error based on some quantity that can be calculated on the training set.” Moreover, it is noteworthy that some researches often use the statement that \"superior measure value lead to better generalization\" to support the claim that \"minimizing (or maximizing) measure value during training results in improved generalization,” which is not always true as demonstrated in [4]. Most importantly, the theoretical analyses in prior work [23, 49, 50] have often used the framework of PAC-learning that basically uses the training set for establishing upper bounds for the generalization gap. In this regard, we believe a comprehensive assessment of measures on the training dataset is essential to avoid wrong conclusions or misuses, such as employing these measures as optimization targets or early-stopping criteria.\n\n****\n*The remaining questions and limitations will be addressed in the continued official comment due to character limitations.*"}, {"review_id": "p0Nm0UpRNk", "reviewer": "Reviewer_qzPP", "summary": "This paper assesses the correlation between existing measures and the robust generalization gap in various experimental settings. Specifically, the author evaluates the relationship between measures based on weight-norm, margin, smoothness, flatness, gradient-norm, and robust generalization under different conditions such as model architecture, training methods, inner maximization steps, optimizer, batch size, data augmentation, and early stopping. Through extensive experimentation, the author finds that these measures are highly sensitive to training setups and, therefore, not very effective.", "questions": "Please refer to the comments in Weaknesses.", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Conduct extensive experiments to explore the correlation between different measures and the robust generalization gap.  \n2. Present some new findings regarding robust overfitting.\n", "weaknesses": "1. Lack of innovation: The author mainly focuses on existing measures and does not propose new measures.    \n2. Limited contribution: The author's finding that existing measures are not effective does not provide substantial contributions since the underlying mechanism of robust overfitting remains unclear. If there were a measure capable of accurately evaluating the robust generalization gap, it would also offer a clear direction to the underlying mechanism of robust overfitting.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful assessment of the strengths and weaknesses of our paper. Your feedback provides valuable insights that help us improve the quality and clarity of our work. Below, we provide a detailed response addressing each of the weaknesses  [W#] raised with all references aligned to the main paper:\n****\n\n**[W1]** We are truly sorry to hear that the innovation of the paper does not match your expectation. Here is a short concrete motivation for studying existing measures. We discover that limitations exist when extending the findings of prior works to practical scenarios due to a restricted set of models [51, 49] and different evaluations [50, 44]. \nTo this end, we **newly propose a metric** $\\pi_k$ in Equation (6) to uncover the effectiveness of measures in the adversarial training framework by training **over 1300 models** and further **verify how and when are measures correlated with the robust generalization gap.** \nTo do so, we would very much appreciate it if the reviewer could further comment on what aspects they would judge as the most important to update in the paper.\n\n****\n**[W2]** The robust generalization gap is *complex* to study directly, particularly within the complexities of real-world datasets [43, 52, 55]. In this paper, we take a significant step towards understanding the robust overfitting by revisiting the measures and uncovering their connections with the robust generalization gap, which was not clearly verified due to the lack of experiments in prior works. We want to emphasize that **other reviewers**, Rev. 2BXc **\"This kind of study help evaluate on a more equal footing several of the proposed correlated factors of robust generalisation. I believe this is always helpful for the research community as it helps develop intuition that might lead to novel methods\"** and Rev. vGwc **\"The paper's findings can inform future experimental and theoretical work on the understanding of the robust overfitting problem,\"** agreed with the contribution of our paper.\nFurthermore, our study **provides an original contribution** **by uncovering the high correlation between the 'x_grad_norm' measure and the robust generalization gap**—a novel observation within this research domain. We also have conducted additional experiments in Appendix “A.3 Robust Measures with Regression Analysis”, which introduces the combination of ‘x_grad_norm’ and ‘average_ce(PGD)’ can more effectively predict the robust generalization gap.\n    \n****\nPlease refer to general responses for the common questions of other reviewers. **If there are still any unclear points, please feel free to ask detailed questions, we would be happy to provide further explanations.**"}]

Author rebuttal:
**Dear all,**

We would like to thank the editor and the reviewers for their careful comments and suggestions. 

We summarize the reviews according to our own perspective.
****
**Strengths.**

We are glad that Reviewers PVgT, 2BXc, and vGwc found that our results **“uncover some interesting findings regarding which metrics can be predictive of robust generalization”** and **“is always helpful for the research community as it helps develop intuition that might lead to novel methods.”** Additionally, Reviewers 2BXc and vGwc also highlighted that **“the paper is generally well-written”** and is rooted in **“clearly and well-described”** methodology and experiments.

$   $

**Weaknesses.**

We have welcomed all reviews and did our best to carefully addressed every concern.

Specifically, the reviewers raised the following common concerns.

1. Lack of detailed explanations on the relationship to [23] and distinctions between our study and the findings of [23]. (Rev. PVgT and vGwc)
2. More discussion of results and contributions. (Rev. vGwc and qzPP)
3. Visibility of Tables, which are not very easy to read at a high-level. (Rev. PVgT and vGwc)

**Our answer to the common concerns raised by the reviewers:**

1. We carefully revisited [23](Jiang & Neyshabur et al., "Fantastic Generalization Measures and Where to Find Them") and we here introduce a summarization of the differences between our study and the findings of [23].
To summarize, **(i) our work delves into the adversarial training framework**, diverging from the standard training framework in [23]. Furthermore, **(ii) our choice of widely-used models such as ResNets**, in contrast to the customized parameter-efficient structures of [23]. Importantly, **(iii) we introduce the novel metric $\pi_k$**, addressing the limitations of the metric $\psi_k$ from [23], facilitating a precise assessment of the correlations between measures and robust generalization.
We will add a new subsection that explicitly addresses clarification of the relationship to [23] in the revised version.
2. To augment the comprehensiveness of our results and their contributions, **we conducted an in-depth review of additional 15 papers (referenced as [S#])** ━will be updated in the continued comment due to character limitations━ related to “margin maximization”, “boundary thickness”, and “local Lipschitzness”.
   * Margin maximization: According to prior works [S1, S2], models trained with CrossEntropy serve as max-margin predictors by optimizing a lower bound of the margin within both standard and adversarial training frameworks. However, recent studies have highlighted that maximizing the margin might not necessarily be the optimal objective in adversarial training due to intricate gradient flow dynamics [S3] and the non-cognitive concept of using predicted probabilities [S4]. Moreover, in recent work [S5, S6], despite the similar robustness of TRADES and AT, their margin distributions on benign and adversarial examples are extremely different. This implies that the margin cannot be the sole determinant of adversarial robustness. Our findings also support these observations by revealing that margin alone does not correlate well with robust generalization; in fact, its excessive pursuit of margin maximization can potentially have an adverse impact on robust generalization. Considering other recent studies [S7, S8], the margin maximization should be accompanied by a consideration of other factors such as weight regularization or gradient information.
   * Boundary thickness: Boundary thickness often serves as a supportive measure to validate the robustness of new training algorithms [S10]. However, empirical validation of its efficacy remains sparse, primarily due to the limited focus on AT in the original paper. Intriguingly, our comprehensive experiment reveals that boundary thickness exhibits a relatively low correlation with the robust generalization gap within models trained by TRADES or MART. We believe that this is caused by the limitation of the margin because boundary thickness also sorely depends on the probability margin. Considering that AT shows a more prominent margin distribution compared to TRADES [S5], this margin-based measure, boundary thickness might be more aligned with AT —where robustness is attained through margin maximization—than with other training methodologies. Indeed, as shown in Figure 3, AT is placed as an outlier for both prob_margin and boundary_thickness. Consequently, our study cautions against the assumption that a higher boundary thickness implies better robust generalization. We emphasize that the concurrent work [S11] also points out that boundary complexities such as boundary thickness are highly abstract.
   * local Lipschitzness: While the foundational work [S12] demonstrates that imposing local Lipschitzness leads to better generalization in linear classification, recent research [50] presents an opposing perspective, suggesting that within neural networks, local Lipschitzness might hurt robust generalization. However, it's noteworthy that the conclusions drawn in [50] are built on limited models of fewer than 20 and are sorely based on test examples, warranting further investigation for comprehensive validation. In our large experiment, we cannot observe that local Lipschitzness itself negatively affects robust generalization. Moreover, in Appendix Tables 8 and 9, we observe that local Lipschitzness is also not perfectly aligned with robust accuracy in contrast to [50, S15]. These findings are consistent with recent works [S13, S14], which highlight the importance of model architecture or weight norms when evaluating models with local Lipshitzness.
3. To provide easy visualization of Tables, we will add the visualization of our tables by providing multiple point plots with error bars for each table in the Appendix. **Sample plots are available in the accompanying PDF in this “global” response.**

**↓Accompanying PDF**

Meta-review: The paper critically investigates the correlation between the robust generalization gap and multiple “robustness measures” studied in prior literature. It provides a medium-to-large scale study on 1,300 trained models, and it outlines robustness measures from multiple methodologies, including gradient-based, margin-based, and flatness-based. The main evaluation procedure of the paper follows that of Jiang et al. [23].

It's clear that the paper has contributions. However, I suggest the authors consider addressing the following issues after the paper is accepted.
- While the paper presents some interesting observational studies, two reviewers pointed out that the main conclusions lack insights. It is unclear how some measures correlate more with the robust generalization gap while others do not.
- It is also unclear if this paper presents a substantial conceptual improvement over the paper by Jiang et al. [23]. The paper seems to have followed mostly the evaluation procedures, as pointed out by one reviewer, and the authors have not discussed this point thoroughly in the paper.
- Other reviewers also provided useful feedback, such as on the title, writing clarity, and robustness of multiple attack methods (e.g., those used to avoid gradient obfuscation), including only those models exhibiting high training robustness.

I believe the authors have partially addressed some of their concerns during the rebuttal, but not all. I urge the authors to thoroughly address the remaining concerns in the final version.
Decision: Accept (poster)