SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hoki Kim
ghrl9613@snu.ac.kr
Jinseong Park
jinseong@snu.ac.kr
Yujin Choi
uznhigh@snu.ac.kr
Jaewook Lee
jaewook@snu.ac.kr
Adversarial training has become the de-facto standard method for improving the robustness of models against adversarial examples. However, robust overfitting remains a significant challenge, leading to a large gap between the robustness on the training and test datasets. To understand and improve robust generalization, various measures have been developed, including margin, smoothness, and flatness-based measures. In this study, we present a large-scale analysis of robust generalization to empirically verify whether the relationship between these measures and robust generalization remains valid in diverse settings. We demonstrate when and how these measures effectively capture the robust generalization gap by comparing over 1,300 models trained on CIFAR-10 under the  $L_{\infty}$  norm and further validate our findings through an evaluation of more than 100 models from RobustBench [12] across CIFAR-10, CIFAR-100, and ImageNet. We hope this work can help the community better understand adversarial robustness and motivate the development of more robust defense methods against adversarial attacks.
Deep neural networks have achieved tremendous success in various domains, but their vulnerability to subtle perturbations has been revealed through the existence of adversarial examples, which are not generally perceptible to human beings [57, 20]. To obtain robustness against these adversarial examples, numerous defense methods have been proposed, and among them, adversarial training has become a common algorithm because of its effectiveness and ease of implementation [42, 71, 61]. However, researchers have recently found that adversarial training methods also suffer from the problem of overfitting [63, 52], where an adversially trained model shows high robustness on training examples, yet significantly reduced robustness on test examples. As this robust overfitting progresses, the robust generalization gap increases, resulting in poor robustness for unseen examples.
To prevent robust overfitting and achieve high robust generalization, researchers have analyzed the properties of adversarial training and demonstrated the usefulness of some measures, such as margin-based measures, flatness-based measures, and gradient-based measures [53, 67, 66, 56]. Researchers have used them to estimate the robust generalization gap of given models and further developed new training schemes for improving its robustness [44, 66]. While these measures offer significant insights into robust generalization, we find that the evaluation of some measures is often limited due to the lack of models or training setups. These limitations can potentially lead to misleading conclusions, which may include inaccurate estimations of the robust generalization gap and misguided directions for the further development of adversarial training methods.
Therefore, to gain a more precise understanding of when and how these measures correlate with robust generalization, we train over 1,300 models on CIFAR-10 under the  $L_{\infty}$  norm across various training settings. We then investigate the relationships between a wide range of measures and their robust generalization gap. To further validate our findings, we also analyze over 100 models provided in RobustBench [12] across CIFAR-10, CIFAR-100, and ImageNet. Based on our large-scale study, we summarize our key findings as follows:
Key findings.
1. Due to the high sensitivity of the robust generalization gap to different training setups, the expectation of rank correlation across a wide range of training setups leads to high variance and may not capture the underlying trend.
2. Margin and smoothness exhibit significant negative correlations with the robust generalization gap. This suggests that, beyond a certain threshold, maximizing margin and minimizing smoothness can lead to a degradation in robust generalization performance.
3. Flatness-based measures, such as estimated sharpness, tend to exhibit poor correlations with the robust generalization gap. Rather, contrary to conventional assumptions, models with sharper minima can actually result in a lower robust generalization gap.
4. The norm of the input gradients consistently and effectively captures the robust generalization gap, even across diverse conditions, including fixed training methods and when conditioned on average cross-entropy.
To promote reproducibility and transparency in the field of deep learning, we have integrated the code used in this paper, along with pre-trained models, accessible to the public at https://github.com/Harry24k/MAIR. We hope that our findings and codes can help the community better understand adversarial robustness and motivate the development of more robust defense methods against adversarial attacks.
The primary distinction between standard and adversarial training is that adversarial training aims to correctly classify not only benign examples but also adversarial examples as follows:
$$
\min  _ {\boldsymbol {w}} \max  _ {\| \boldsymbol {x} ^ {a d v} - \boldsymbol {x} \| \leq \epsilon} \mathcal {L} (f (\boldsymbol {x} ^ {a d v}, \boldsymbol {w}), y), \tag {1}
$$
where  $(x,y)$  is drawn from the training dataset  $\mathcal{S}$  and  $f$  represents the model with trainable parameters  $\boldsymbol{w}$ . Among the distance metrics  $\| \cdot \|$ , in this paper, we focus on robustness with respect to the  $L_{\infty}$  norm. Note that the loss function  $\mathcal{L}$  can also include  $f(\boldsymbol{x},\boldsymbol{w})$  to minimize the loss with respect to  $\boldsymbol{x}$ . By optimizing (1), we hope to minimize the robust error on the true distribution  $\mathcal{D}$ , defined as:
$$
\mathcal {E} (\boldsymbol {w}; \epsilon , \mathcal {D}) = \mathbb {E} _ {(\boldsymbol {x}, y) \in \mathcal {D}} \left[ \max  _ {\| \boldsymbol {x} ^ {a d v} - \boldsymbol {x} \| \leq \epsilon} \mathbb {1} (f (\boldsymbol {x} ^ {a d v}, \boldsymbol {w}) \neq y) \right], \tag {2}
$$
where  $\mathbb{1}(\hat{y}\neq y)$  is an indicator function that outputs 0 if the prediction  $\hat{y}$  is same as the true label  $y$ , and 1 otherwise.
The majority of researches have focused on optimizing (1) through the development of new loss functions or training attacks. For instance, vanilla adversarial training (AT) [42] minimizes the loss of adversarial examples generated by projected gradient descent (PGD) [42] with multiple iterations. Following AT, several variations, such as TRADES [71] and MART [61], have achieved significant reductions in robust errors on various datasets through the adoption of KL divergence and the regularization on probability margins, based on theoretical and empirical analyses.
However, recent works have revealed that the adversarial training framework has a challenging generalization problem, characterized by higher Rademacher complexity [68] and larger sample complexity [54]. The overfitting problem in adversarial training has been observed as a common phenomenon across various settings [52], and it can even occur in a more catastrophic manner during single-step adversarial training [63, 31]. As robust overfitting progresses, the following robust generalization gap  $g(w)$  increases,
$$
g (\boldsymbol {w}) = \mathcal {E} (\boldsymbol {w}; \epsilon , \mathcal {D}) - \mathcal {E} (\boldsymbol {w}; \epsilon , \mathcal {S}). \tag {3}
$$
Thus, in order to minimize  $\mathcal{E}(\boldsymbol{w};\epsilon,\mathcal{D})$ , we should not only focus on minimizing the training objective function  $\mathcal{E}(\boldsymbol{w};\epsilon,\mathcal{S})$  but also on reducing the robust generalization gap  $g(\boldsymbol{w})$ .
To gain a deeper understanding of the robust overfitting and further reduce the robust generalization gap  $g(\boldsymbol{w})$ , a line of work has theoretically and empirically investigated measures, such as boundary thickness [67], local Lipschitzness [66], and flatness [56]. While these studies claim that these measures are reliable indicators of the robust generalization gap, the lack of consistency in experimental settings hinders us to identify their validity in practical scenarios. Therefore, in this work, we aim to investigate the clear relationship between these measures and the robust generalization gap by conducting a comprehensive analysis using a large set of models.
2.1 Comparison to Jiang et al. [27]
The pioneering study [27] explored the empirical correlations between complexity measures and generalization with a primary focus on the standard training framework. Our main contribution is delving into the realm of measures within the adversarial training framework—a context having different generalization tendencies from those of the standard training framework [53, 66]. Indeed, we observe that the metric  $\psi_{k}$  proposed in [27] has limitations in capturing the effectiveness of measures due to the high sensitivity of the robust generalization gap with respect to training setups. By introducing a new metric  $\pi_{k}$ , our work enhances the understanding of when and how robustness measures correlate with robust generalization. Furthermore, while Jiang et al. [27] employed customized parameter-efficient neural networks, we adopt widely-used model architectures such as ResNets, thereby providing insights that are not only relevant to recent research but also offer more practical implications.
3 Experimental Methodology
In the adversarial training framework, measures have played a crucial role by providing either theoretical upper bounds or empirical correlations with robust generalization. Previous works have leveraged these measures to propose new training schemes [61, 67, 64] and suggested directions to achieve high robustness [66, 56]. However, we discover that certain limitations and confusions exist when extending the findings of prior works to practical scenarios due to the use of a restricted set of models and training setups [67, 64, 66]. In order to gain a comprehensive understanding of the true efficacy of these measures, it is crucial to validate whether the effectiveness of measures remains valid in practical settings.
To address these challenges, our work aims to provide a comprehensive and accurate assessment of the effectiveness of measures for robust generalization in practical settings. Our objective is to address the fundamental question:
Do measures remain effective in correlating with robust generalization in practical settings? If so, how and when are measures correlated with robust generalization?
To this end, in Sections 3.1 and 3.2, we carefully construct the training space that considers practical scenarios within the adversarial training framework and gather a wide range of measures from previous works. In Section 3.3, we define the evaluation metrics and introduce specific variations to accurately capture the correlation between measures and robust generalization in practical settings.
3.1 Training Space
In the realm of adversarial training, various training procedures have been extensively explored to enhance adversarial robustness based on the development of AT, TRADES, and MART. Recently, to resolve the issue of robust overfitting, researchers have begun combining additional techniques, including commonly employed in the standard training framework, such as early-stopping [52], using additional data [9, 22], manipulating training tricks [48, 11], and adopting sharpness-aware minimization [64]. By integrating these techniques into AT, TRADES, and MART, high adversarial robustness have been achieved, outperforming other variants of adversarial training methods [21].
Based on these prior works, to mimic practical scenarios, we have carefully selected eight training parameters widely used for improving robust generalization: (1) Model architecture {ResNet18 [23],
Figure 1: (Left) Scatter plot of train robustness and test robustness. Bright color corresponds to high robust generalization gap, i.e., poor generalization. (Right) Boxplot of robust generalization gap for some training setups. All adversarial examples during training and testing are generated by PGD10 on CIFAR-10.
WRN28-10 [70], WRN34-10 [70}], (2) Training methods {Standard, AT [42], TRADES [71], MART [61]}, (3) Inner maximization steps  $\{1, 10\}$ , (4) Optimizer {SGD, AWP [64]}, (5) Batch-size \{32, 64, 128, 256\}, (6) Data augmentation {No Augmentation, Use crop and flip}, (7) Extra-data {No extra data, Use extra data [9]}, and (8) Early-stopping {No early-stopping, Use early-stopping}. Additional training details can be found in Appendix C, providing a comprehensive overview of the training procedures.
In total, 1,344 models were trained using the CIFAR-10 dataset with  $\epsilon = 8 / 255$ . Given these models, we evaluate their train/test robustness against PGD with 10 iterations (denoted as PGD10). While we acknowledge the existence of stronger adversarial attacks, such as AutoAttack [11], we primarily use PGD10 due to its prevalent use in adversarial training and the high computational demands of AutoAttack. Additionally, considering the usage of PGD in calculating specific measures, such as boundary Thickness and local_lip, ensures consistency in our analysis. For a detailed discussion, please refer to the Appendix A.4.
The statistics of the trained models are summarized in Figure 1. In the left plot, we can observe that the selected range of training parameters generates a diverse set of models, exhibiting robust generalization gaps ranging from  $0\%$  to  $70\%$ . Notably, certain models achieve  $100\%$  robustness on training data against PGD10, but their maximum robustness on the test set is only  $65\%$ , highlighting the importance of robust generalization gap. The right plot is a boxplot shows the distribution of robust generalization gaps for some training setups. As described in prior works [9, 52, 64], each training setup has a significant impact on robust generalization.
Beyond the measures proposed under the adversarial training frameworks [67, 66], previous works [27, 17] have demonstrated that certain measures can effectively capture the generalization gap under the standard training framework. Therefore, in this paper, we have gathered diverse measures from both the standard and adversarial training frameworks and categorized them into five different types based on their origins and formulas: (i) weight-norm (7, 8, 9, 10), (ii) margin (12, 13, 14, 15), (iii) smoothness (16, 17), (iv) flatness (18, 19, 20, 21), and (v) gradient-norm (22, 23). We denote the chosen measures in teletype font (e.g., path_norm). While we briefly introduce the concepts of measures in each paragraph in Section 4, we refer the readers to Appendix B for the details of measures including their mathematical definitions due to the page limit.
Given these measures, we calculate their value on whole training examples for each trained model. This choice aligns with prior works, which argue that the most direct approach for studying generalization is to prove a generalization bound that can be calculated on the training set [27] and offer a caution against the oversimplified notion that maximizing (or minimizing) a measure value inherently leads to improved generalization [5].
3.3 Evaluation Metrics
To uncover the relationship between measures and robust generalization performance, we adopt the Kendall rank correlation coefficient following prior works [27, 36]. We begin by defining a search space  $\Theta = \Theta_{1} \times \Theta_{2} \times \dots \times \Theta_{n}$ . Each  $\Theta_{i}$  corresponds to a search space for each training parameter defined in Section 3.1. Given the search space  $\Theta$ , we obtain the trained models  $f_{\theta}(\boldsymbol{w})$  for  $\theta \in \Theta$ . For each model  $f_{\theta}(\boldsymbol{w})$ , we measure the robust generalization gap  $g(f_{\theta}(\boldsymbol{w}))$  and the corresponding measure value  $\mu(f_{\theta}(\boldsymbol{w}))$ . For simplicity, we denote  $g(\theta) \coloneqq g(f_{\theta}(\boldsymbol{w}))$  and  $\mu(\theta) \coloneqq \mu(f_{\theta}(\boldsymbol{w}))$ . We then calculate Kendall's rank coefficient [29] as follows:
$$
\tau (\boldsymbol {\Theta}) = \frac {1}{| \boldsymbol {\Theta} | (| \boldsymbol {\Theta} | - 1)} \sum_ {\boldsymbol {\theta} \in \boldsymbol {\Theta}} \sum_ {\boldsymbol {\theta} ^ {\prime} \in \boldsymbol {\Theta}, \boldsymbol {\theta} \neq \boldsymbol {\theta} ^ {\prime}} \operatorname {s g n} \left(g (\boldsymbol {\theta}) - g \left(\boldsymbol {\theta} ^ {\prime}\right)\right) \cdot \operatorname {s g n} \left(\mu (\boldsymbol {\theta}) - \mu \left(\boldsymbol {\theta} ^ {\prime}\right)\right), \tag {4}
$$
where  $|\Theta|$  is the number of elements in  $\Theta$ , and  $\operatorname{sgn}(\cdot)$  is a sign function. The value of  $\tau$  becomes 1 when the pairs have the same rankings and  $-1$  when the pairs have reversed order rankings. Therefore, a higher value of  $\tau$  implies that as the value of a measure  $\mu$  increases, the robust generalization gap  $g$  also increases.
As noted by [27], the measure may strongly correlate with the robust generalization gap only when a specific training setting is varied. Therefore, Jiang et al. [27] introduced the following metric:
$$
\psi_ {k} (\Theta) = \mathbf {E} _ {\theta_ {1}, \dots , \theta_ {k - 1}, \theta_ {k + 1}, \dots , \theta_ {n}} \left[ \tau \left(\left\{\boldsymbol {\theta} = \left(\theta_ {1}, \dots , \theta_ {n}\right), \theta_ {k} \in \Theta_ {k} \right\}\right) \right], \tag {5}
$$
which captures the robust generalization gap when only the hyper-parameter  $\Theta_{k}$  changes. However, we demonstrate that  $\psi_{k}$  may not perform well in cases where Simpson's paradox exists. Simpson's paradox refers to a situation where each of the individual groups exhibits a specific trend, but it disappears (or reverses) when the groups are combined. Thus, when a parameter  $\Theta_{i\neq k}$  heavily affects the robust generalization gap,  $\psi_{k}$  becomes not effective as it captures the overall trends by taking expectation across all parameters including  $\Theta_{i\neq k}$ . In fact, within the adversarial training framework, the inner  $\tau$  in (5) shows extremely high variance due to the high sensitivity of the robust generalization gap with respect to training setups, which will be discussed in Table 1.
Therefore, we propose a metric for capturing the robust generalization performance by fixing the hyper-parameter  $\Theta_{k}$  as follows:
$$
\pi_ {k} (\Theta) = \mathbf {E} _ {\theta_ {k}} [ \tau (\{\boldsymbol {\theta} = \left(\theta_ {1}, \dots , \theta_ {n}\right), \theta_ {i} \in \Theta_ {i} \forall i \neq k \}) ]. \tag {6}
$$
Here,  $\pi_k(\Theta)$  represents the effectiveness of a measure  $\mu$  within a specific fixed training setup. This enables us to discover that some measures only work for specific training settings, e.g., uncovering the strong correlation between boundary Thickness and the robust generalization gap when AT is used as the training method. Furthermore, for measures exhibiting meaningful value of  $\pi_k$ , we additionally provide their scatter plots to mitigate the limitations of a correlation analysis.
Based on the measures and trained models defined in Section 3, we calculate the measures for each model and their robust generalization gap. Notably, in the realm of adversarial training, models encounter two types of examples: benign examples and adversarial examples. Therefore, to gain a deeper understanding of the relationship between robust generalization and measures, we also calculate the values of example-dependent measures for both benign examples and PGD10 examples.
Table 1 summarizes the results of  $\psi_{k}$  for each measure. Further, in order to consider the distributional correlation [17] and quantify the precision of  $\psi_{k}$ , we also report the corresponding standard deviation of the inner  $\tau$  in Eq. (5). A higher value of  $\psi_{k}$  indicates a stronger positive rank correlation, implying that as the measure value increases, the robust generalization gap increases. First of all, it is important to note that none of the measures are perfect. While certain measures show high  $\psi_{k}$ , all measures show high standard deviations. This observation indicates that no measure can provide a perfect estimation of the model's robust generalization gap. Moreover, with such high variances, it becomes challenging to clearly identify the underlying correlation of the measures.
To address this limitation, we conduct further analyses with our proposed metric,  $\pi_{k}$  in Eq. (6). The benefit of  $\pi_{k}$  is that it reveals the potential hidden relationship between the measures and the robust generalization gap by fixing training settings, which  $\psi_{k}$  cannot captures. Thus, from now on, we will report  $\pi_{k}$  of each measure and provide correlation analyses with the robust generalization gap, extending the connections beyond those presented in Table 1.
Table 1: Numerical results of  $\psi_{k}$  for each measure, along with its corresponding standard deviation. The total  $\tau$  indicates the Kendall's rank coefficient for the entire pairs  $(g,\mu)$ .  $*(\mathrm{PGD})$  indicates the same measure calculated on PGD10 examples for example-based measures.
Model Architecture	Training Methods	Steps	Optimizer	Batch-size	Aug	Extra-data	Early Stopping	Totalτ
num.params (7)	0.05±0.47	-	-	-	-	-	-	-	-0.03
path_norm (8)	0.22±0.59	0.47±0.39	0.38±0.92	0.41±0.69	0.20±0.47	0.32±0.73	0.29±0.72	0.28±0.63	0.35
log prod_of_spec (9)	-0.13±0.49	0.15±0.40	0.11±0.99	0.16±0.75	0.13±0.47	0.02±0.73	0.11±0.74	0.17±0.66	-0.13
log prod_of_fro (10)	0.07±0.44	0.48±0.39	0.62±0.79	0.56±0.63	0.45±0.46	0.38±0.65	0.35±0.70	0.37±0.69	0.18
euclid_init_norm (11)	0.07±0.47	0.32±0.34	0.00±1.00	0.08±0.74	-0.02±0.45	0.20±0.69	0.16±0.68	-0.06±0.70	0.18
average_ce (12)	-0.16±0.62	-0.25±0.52	-0.10±1.00	-0.26±0.74	-0.26±0.56	-0.20±0.73	-0.24±0.72	-0.21±0.65	-0.23
inverse_margin (13)	0.19±0.57	0.25±0.37	0.45±0.89	0.08±0.73	0.11±0.49	0.15±0.70	0.08±0.71	0.03±0.72	0.07
prob_margin (14)	0.19±0.61	0.23±0.52	0.09±1.00	0.26±0.74	0.30±0.53	0.25±0.72	0.30±0.70	0.20±0.65	0.24
boundary Thickness (15)	-0.06±0.61	0.05±0.45	0.07±1.00	-0.15±0.72	-0.17±0.53	-0.08±0.71	-0.16±0.70	-0.11±0.66	-0.02
kl_divergence (16)	-0.42±0.53	-0.36±0.36	-0.71±0.71	-0.26±0.70	-0.36±0.46	-0.51±0.66	-0.38±0.69	-0.45±0.74	-0.45
local_lip (17)	-0.25±0.56	-0.20±0.41	-0.48±0.88	-0.13±0.71	-0.17±0.48	-0.38±0.69	-0.20±0.69	-0.15±0.72	-0.23
pacbayes flats (18)	0.22±0.60	0.20±0.42	0.33±0.94	0.17±0.75	0.17±0.53	0.05±0.71	0.06±0.72	0.20±0.69	0.05
estimated_sharness (19)	0.07±0.60	0.07±0.41	0.19±0.98	0.15±0.80	0.05±0.56	-0.11±0.71	-0.03±0.74	-0.03±0.72	-0.07
estimated_inv_sharness (20)	0.16±0.60	0.20±0.35	0.42±0.91	0.27±0.77	0.14±0.55	0.00±0.71	0.07±0.73	0.06±0.78	0.04
averageflat (21)	-0.30±0.53	-0.31±0.41	-0.58±0.82	-0.21±0.68	-0.22±0.44	-0.43±0.66	-0.32±0.68	-0.30±0.71	-0.36
x_grad_norm (22)	-0.36±0.56	-0.29±0.36	-0.62±0.78	-0.31±0.74	-0.40±0.45	-0.51±0.65	-0.45±0.70	-0.50±0.69	-0.42
w_grad_norm (23)	0.17±0.56	0.23±0.36	0.50±0.86	0.31±0.74	0.16±0.52	0.03±0.70	0.10±0.72	0.10±0.78	0.07
average_ce(PGD) (12)	-0.59±0.52	-0.85±0.15	-0.86±0.51	-0.59±0.69	-0.63±0.48	-0.61±0.66	-0.59±0.68	-0.58±0.72	-0.78
inverse_margin(PGD) (13)	0.56±0.49	0.40±0.31	0.75±0.66	0.33±0.67	0.44±0.43	0.49±0.63	0.39±0.67	0.27±0.76	0.35
prob_margin(PGD) (14)	0.59±0.53	0.80±0.14	0.84±0.55	0.61±0.69	0.64±0.45	0.62±0.65	0.61±0.68	0.59±0.71	0.79
pacbayesflat(PGD) (18)	0.55±0.47	0.64±0.21	0.82±0.58	0.61±0.47	0.62±0.29	0.57±0.44	0.50±0.46	0.68±0.57	0.59
estimated_sharness(PGD) (19)	-0.05±0.64	-0.19±0.48	-0.10±0.99	0.08±0.77	-0.03±0.55	-0.15±0.74	-0.12±0.75	-0.00±0.67	-0.22
estimated_inv_sharness(PGD) (20)	-0.08±0.65	-0.20±0.52	-0.13±0.99	0.08±0.78	-0.06±0.57	-0.21±0.74	-0.12±0.75	-0.06±0.66	-0.25
x_grad_norm(PGD) (22)	-0.23±0.54	-0.21±0.38	-0.53±0.85	-0.19±0.70	-0.26±0.45	-0.38±0.68	-0.29±0.70	-0.33±0.73	-0.34
w_grad_norm(PGD) (23)	-0.14±0.63	-0.28±0.46	-0.19±0.98	0.01±0.78	-0.09±0.57	-0.19±0.76	-0.16±0.76	-0.13±0.68	-0.32