SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(b) log.prod_of_fro
Figure 12: Distributions of norm-based measures for model architectures. For each model architecture, the range of measure extremely varies.
In Fig. 12, we plot log.prod_of_spec and log_PROD_of_fro for each model architecture. We can observe that the range of each measure extremely varies with respect to the used model architecture. When the model architecture is fixed, they show some correlation with the robust generalization gap as described in the main paper.
A.6 Early Stopping and Estimated Sharpness
In the main paper, we discussed that estimated_sharpness exhibits a significant negative correlation when early stopping is not used. Fig. 13 shows the importance of early-stopping for estimated sharpness measures. Compared to Figures 4 and 13b, when early stopping is employed, the correlation approaches zero as shown in Figures 13a and 13c. This results supports the observation of prior study [56] that the importance of early stopping in the analysis of flatness.
(a) estimated_sharpness w/ early-stopping
Figure 13: Scatter plot of the estimated sharpness measures for the use of early-stopping. The same condition, average_ce (PGD)  $< 2$ , used as Fig. 4.
(b) estimated_inv_sharpness w/o early-stopping
(c) estimated_inv_sharpness w/ early-stopping
B Measures
In this section, we introduce the concept of each category of measures in the main paper, then explain the details of each measure and their mathematical definitions. Here, we denote  $\mathbf{W}_i$  as the weight tensor of  $i$ -th layer, following [17]. Given the number of layers  $d$ , the whole trainable parameters are denoted as  $\boldsymbol{w} = \mathrm{vec}(\mathbf{W}_1, \mathbf{W}_2, \dots, \mathbf{W}_d)$ .
Weight-norm. Based on some theoretical frameworks such as PAC-Bayes [43, 47, 38], weight norm-based measures are expected to be correlated with generalization performance. Among them, the product of Frobenius norm [46] (logprod_of_fro), the product of spectral norm [7] (logprod_of_spec), and path norm (path_norm) have been considered as important measures [27, 17]. Furthermore, the distance to the converged weight from the initial weight (euclid_init_norm) is also used to estimate the generalization gap [27]. Liu et al. [40] also argues that this distance can be used to judge the difficulty of optimization in adversarial training.  $\triangleright$  num.params.
$$
\sum_ {i = 1} ^ {d} k _ {i} c _ {i - 1} \left(c _ {i} + 1\right), \tag {7}
$$
where  $c_{i}$  is the number of channels and  $k_{i}$  is the kernel size at layer  $i$ . In the experiments, we calculated num.params by adding the number of parameters of all convolutional and linear layers. num.params is a fixed value when a model architecture is given.
$\triangleright$  path_norm.
$$
\sum_ {i} f _ {\boldsymbol {w} ^ {2}} (\mathbf {1}) [ i ], \tag {8}
$$
where  $w^2$  is the element-wise square operation and  $f(\cdot)[i]$  is the  $i$ -th logit output of the network. By setting all input variables as 1, this measure captures geometric properties of optimization under scale-invariant characteristics.
$\triangleright$  log.prod_of_spec.
$$
\log \left(\prod_ {i = 1} ^ {d} \| \mathbf {W} _ {i} \| _ {2} ^ {2}\right), \tag {9}
$$
where  $\| \cdot \|_2$  is a matrix  $L_2$ -norm, i.e., the largest singular value of each layer.
$\triangleright$  log_Prod_of_spec.
$$
\log \left(\prod_ {i = 1} ^ {d} \| \mathbf {W} _ {i} \| _ {F} ^ {2}\right), \tag {10}
$$
where  $\| \cdot \| _F$  is a Frobenius norm, i.e., the square root of the sum of the squares of the weight matrix.  $\triangleright$  euclid_init_norm.
$$
\frac {1}{d} \sum_ {i = 1} ^ {d} \| \boldsymbol {w} _ {i} - \boldsymbol {w} _ {i} ^ {0} \| _ {2}, \tag {11}
$$
where  $\pmb{w}_i = \mathrm{vec}(\mathbf{W}_i)$  and the initial weight of  $i$ -th layer  $\pmb{w}_i^0$ .
Margin. Margins are also actively researched measures to estimate the generalization gap [26]. For instance, the 10-th percentile of the margin values in the output space on the training set (inverse_margin) is often used to measure the generalization bound for neural networks [50, 27]. In terms of adversarial robustness, most of attack methods and defense methods utilize the probability margins (prob_margin). Yang et al. [67] further proposed a new measure, called boundary thickness (boundary Thickness), that is a generalized version of margin and argued that it is highly correlated to the robust generalization gap.
average_ce.
$$
\mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \left[ \mathcal {L} _ {c e} (f (\boldsymbol {x}, \boldsymbol {w}), \boldsymbol {y}) \right], \tag {12}
$$
where  $\mathcal{L}_{ce}$  is the cross-entropy loss.
$\triangleright$  inverse_margin.
$$
1 / \gamma^ {2}, \tag {13}
$$
where  $\gamma$  is 10th-percentile of  $\{\sigma(f(\pmb{x},\pmb{w}))_y - \max_{i \neq y} \sigma(f(\pmb{x},\pmb{w}))_i\}$  for all  $\pmb{x}, \pmb{y}$ , with the sigmoid function  $\sigma(\cdot)$ .
$\triangleright$  prob_margin.
$$
\mathbb {E} _ {\boldsymbol {x}, y} \left[ \sigma (f (\boldsymbol {x}, \boldsymbol {w})) _ {y} - \max  _ {i \neq y} \sigma (f (\boldsymbol {x}, \boldsymbol {w})) _ {i} \right]. \tag {14}
$$
$\triangleright$  boundary_thickness.
$$
\mathbb {E} _ {\boldsymbol {x}} \left[ \| \boldsymbol {x} - \boldsymbol {x} ^ {*} \| _ {2} \int_ {0} ^ {1} \mathbb {1} \{a <   g (\boldsymbol {x}, \boldsymbol {x} ^ {*}, \lambda) <   b \} d \lambda \mid \underset {i} {\arg \max } \sigma (f (\boldsymbol {x}, \boldsymbol {w})) _ {i} \neq \underset {i} {\arg \max } \sigma (f (\boldsymbol {x} ^ {*}, \boldsymbol {w})) _ {i} \right], \tag {15}
$$
where  $\mathbb{1}\{\cdot\}$  is an indicator function,  $g(\pmb{x}, \pmb{x}^*, \lambda) = \sigma(f(\lambda \pmb{x} + (1 - \lambda) \pmb{x}^*, \pmb{w}))_{\hat{y}} - \sigma(f(\lambda \pmb{x} + (1 - \lambda) \pmb{x}^*, \pmb{w}))_{\hat{y}^*}$ ,  $\hat{y} = \arg \max_i \sigma(f(\pmb{x}, \pmb{w}))_i$ ,  $\hat{y}^* = \arg \max_i \sigma(f(\pmb{x}^*, \pmb{w}))_i$  and  $a, b$  are the hyperparameters that controls the sensitivity of the boundary thickness. Following [67], we find  $\pmb{x}^*$  by using PGD10 with  $L_2$ -norm,  $\epsilon = 1$ ,  $\alpha = 0.2$ , then  $a = 0$ ,  $b = 0.75$ , and batch size 128. A higher value of boundary Thickness implies a larger margin in the output space.
Smootheness. Based on prior works [6, 10, 28], a line of work has focused on the smoothness for achieving adversarial robustness in adversarial training. Most simply, the KL divergence between benign and adversarial logits (kl_divergence) of TRADES [71] can be regarded as a smoothness regularization due to its logit pairing. Yang et al. [66] investigated the theoretical benefit of local Lipschitzness (local_lip) and demonstrated that its value estimated on the test dataset correlates with the robust generalization gap.
$\triangleright$  kl_divergence.
$$
\mathbb {E} _ {\boldsymbol {x}} \left[ \max  _ {\| \boldsymbol {x} - \boldsymbol {x} * \| \leq \epsilon} \operatorname {K L} (f (\boldsymbol {x}, \boldsymbol {w}), f (\boldsymbol {x} ^ {*}, \boldsymbol {w})) \right], \tag {16}
$$
where KL is KL-divergence and the maximization is conducted by PGD10 with the step-size 2/255. A lower value of k1_divergence implies that a model outputs similar outputs for both benign example and adversarial example.
$\triangleright$  local_lip.
$$
\mathbb {E} _ {\boldsymbol {x}} \left[ \max  _ {\| \boldsymbol {x} - \boldsymbol {x} ^ {*} \| \leq \epsilon} \frac {\| f (\boldsymbol {x} , \boldsymbol {w}) - f \left(\boldsymbol {x} ^ {*} , \boldsymbol {w}\right) \| _ {1}}{\| \boldsymbol {x} - \boldsymbol {x} ^ {*} \| _ {\infty}} \right], \tag {17}
$$
where the maximization is conducted by PGD10 with the step-size 2/255. This is the empirical version of the local Lipschitzness, resulting a lower value of local_lip implies a smoother model.
Flatness. Flatness is a recently focused measure in the generalization domain [43, 30]. Recent works argue that flatter minima yield better generalization performance than sharper minima. The common strategy to achieve flatness is to minimize the estimated sharpness [18, 73], which is the difference between the current loss and the maximum loss for a given neighborhood (estimated_sharpness). Kwon et al. [36] investigated the scale-invariant sharpness (estimated_inv_sharpness). Note that other diverse concepts of estimated sharpness are actively researched in recent works [33, 5, 34]. Adversarial weight perturbation (AWP) [64] also has dramatically improved adversarial robustness by minimizing the loss of perturbed weight. Recently, Stutz et al. [56] has demonstrated that their
proposed measure, average flatness (averageflat), is effective for estimating robust generalization gap.
$\triangleright$  pacbayesflat. Based on PAC-Bayesian framework [43], Jiang et al. [27] proposed a simplified version of PAC-Bayesian bounds as follows:
$$
1 / \sigma^ {\prime} \tag {18}
$$
where  $\sigma'$  is the largest value such that  $\mathbb{E}_{\boldsymbol{u}}[\mathcal{E}(\boldsymbol{w} + \boldsymbol{u}; \{\boldsymbol{x}, y\}) - \mathcal{E}(\boldsymbol{w}; \{\boldsymbol{x}, y\})] < 0.1$ . Here,  $u_j \sim \mathcal{N}(0, \sigma')$  and  $\mathcal{E}(\boldsymbol{w}; \{\boldsymbol{x}, y\})$  denotes the error on the given set  $\{\boldsymbol{x}, y\}$ . Thus, a higher value of pacbayesflat implies flatter optimum in terms of error landscape.
$\triangleright$  estimated_sharpness.
$$
\mathbb {E} _ {\boldsymbol {x}, y} \left[ \max  _ {\| \boldsymbol {v} \| _ {2} \leq \rho} \mathcal {L} _ {c e} (f (\boldsymbol {x}, \boldsymbol {w} + \boldsymbol {v}), y) - \mathcal {L} _ {c e} (f (\boldsymbol {x}, \boldsymbol {w}), y) \right]. \tag {19}
$$
Following [73], we calculate the estimated sharpness with a single-step ascent with  $\rho = 0.1$ . A higher value of estimated_sharpeness implies a sharper optimum in terms of loss landscape.
$\triangleright$  estimated_inv_sharpness.
$$
\mathbb {E} _ {\boldsymbol {x}, y} \left[ \max  _ {\| T _ {\boldsymbol {w}} ^ {- 1} \boldsymbol {v} \| _ {2} \leq \rho} \mathcal {L} _ {c e} (f (\boldsymbol {x}, \boldsymbol {w} + \boldsymbol {v}), y) - \mathcal {L} _ {c e} (f (\boldsymbol {x}, \boldsymbol {w}), y) \right], \tag {20}
$$
where  $T_{\boldsymbol{w}}$  is  $\| \boldsymbol{w} \|$ , i.e., element-wise adaptive sharpness, and  $\mathcal{L}_{ce}$  is the cross-entropy loss. Similar to the estimated sharpness, we calculate the estimated invariant sharpness with a single-step ascent with  $\rho = 0.1$  [36]. A higher value of estimated_inv_sharpness implies a sharper optimum in terms of loss landscape.
$\triangleright$  average-flat.
$$
\mathbb {E} _ {\boldsymbol {x}, y} \left[ \mathbb {E} _ {\boldsymbol {v} \in \mathcal {B} _ {\rho} (\boldsymbol {w})} \left[ \max  _ {\| \boldsymbol {x} - \boldsymbol {x} ^ {*} \| \leq \epsilon} \mathcal {L} _ {c e} (f (\boldsymbol {x} ^ {*}, \boldsymbol {w} + \boldsymbol {v}), y) \right] - \max  _ {\| \boldsymbol {x} - \boldsymbol {x} ^ {*} \| \leq \epsilon} \mathcal {L} _ {c e} (f (\boldsymbol {x} ^ {*}, \boldsymbol {w}), y) \right], \tag {21}
$$
where  $\mathcal{B}_{\rho}(\boldsymbol{w}) = \{\boldsymbol{w} + \boldsymbol{v} \mid \| \boldsymbol{v}_i \|_2 \leq \rho \| \boldsymbol{w}_i \|_2 \forall \text{ layers } i\}$ . We use PGD10 with  $\epsilon = 8/255$  for both inner maximizations. Following [56], we take 10 random weight perturbations with  $\rho = 0.5$ . A higher value of averageflat implies a sharper optimum in terms of adversarial loss landscape.
Gradient-norm. Gradient-norm with respect to input or weight is also a consistently researched area in terms of generalization. Recently, Zhao et al. [72] also demonstrated that regularizing the gradient norm of weights (w_grad_norm) can achieve sufficient improvement on several tasks. Additionally, there are a few works that emphasize the importance of regularizing the gradient norm of weights [53, 44]. The gradient norm of inputs (x_grad_norm) also can have underlying correlation between the robust generalization performance. Prior works utilized the input gradient for analyzing adversarially trained models [4] and generating adversarial examples [35, 16].
$\triangleright$  x_grad_norm.
$$
\mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \left[ \| \nabla_ {\boldsymbol {x}} \mathcal {L} _ {c e} (f (\boldsymbol {x}, \boldsymbol {w}), \boldsymbol {y}) \| _ {2} \right]. \tag {22}
$$
$\triangleright$  w_grad_norm.
$$
\mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \left[ \| \nabla_ {\boldsymbol {w}} \mathcal {L} _ {c e} (f (\boldsymbol {x}, \boldsymbol {w}), \boldsymbol {y}) \| _ {2} \right]. \tag {23}
$$
Comment on batch normalization fusion Here, we provide comments on some further discussed things when estimating the above measures. In previous studies [27, 17], it has been observed that considering batch normalization (batch-norm) layers can have an impact on common generalization measures, such as sharpness [15]. To address this issue, the batch-norm layers and other moving statistics were fused with the preceding convolution layers before calculating the values of generalization measures. Thus, when estimating {num.params, path_norm, log_PROD_of_spec, log_PROD_of_fro, pacbayes_FLAT}, we apply batch-norm fusion to all ResNet blocks. However, for certain blocks, such as pre-activation ResNets, where the batch-norm layer is placed at the beginning, the fusion cannot be directly applied. To ensure consistency, we add an identity convolutional layer in front of all batch-norm layers that do not have the preceding convolution layer. While there are various batch-norm fusion (or batch-norm folding) techniques, including those related to
generalization [27, 17, 5], quantization [69, 37, 49], and memory optimization domains [8, 19, 2], there is no precise solution to address this problem in the context of various model structures (ResNets, PreActResNets, ViT, etc.) and activation functions (SiLU, LeakyReLU, etc.), which we leave as a topic for future work.
In Section 3, 1,344 models were trained using the CIFAR-10 dataset with  $\epsilon = 8 / 255$ . We here provide the detailed training settings. We followed the common settings used in [42, 71, 48, 21].
Given the higher Rademacher complexity [68] and larger sample complexity [54] of adversarial training, data augmentation [21] and the utilization of extra data [9] can significantly improve the adversarial robustness. Therefore, we also considered the impact of augmentation technique, including RandomCrop with padding 4 and RandomHorizontalFlip, as well as the use of additional data collected by Carmon et al. [9].
Regarding model architectures, we employed three different models: ResNet18 [23], WRN28-10 [70], and WRN34-10 [70]. These models have been widely adopted and serve as benchmarks for evaluating the stability and performance of adversarial training methods. It is worth noting that the majority of models trained on CIFAR-10 in RobustBench [12] consist of WRN28-10 (15) and WRN34-10 (14) among the 63 available models.
For training methods, we considered four different approaches: Standard, AT [42], TRADES [71], and MART [61]. Notably, AT, TRADES, and MART have been shown to outperform other variations by incorporating various training tricks [48] and integrating recent techniques [9, 64]. For all methods, we generated adversarial examples using projected gradient descent (PGD) [42]. During training, a single-step approximation of the inner maximization in Eq. (1) can lead to faster adversarial training, but may suffer from catastrophic overfitting [63, 31]. On the other hand, a large number of steps leads to stable robustness, but requires heavy computational costs. Therefore, we considered both 1 and 10 steps for each adversarial training method.
In terms of optimization, we used SGD with momentum 0.9 and weight decay of  $5 \times 10^{-4}$ , and a step-wise learning rate decay was performed at epochs 100 and 150 with a decay rate of 0.1. In all the experiments, we trained the models for 200 epochs. As highlighted by Pang et al. [48], the batch size used during adversarial training has been found to affect its performance. Thus, we varied the batch size among {32, 64, 128, 256}. Additionally, we also considered adversarial weight perturbation (AWP) [64], which can improve the robust generalization performance of models. AWP belongs to the class of sharpness-aware minimization methods [18, 36]. This can be formalized as follows:
$$
\min  _ {\boldsymbol {w}} \max  _ {\| \boldsymbol {x} ^ {a d v} - \boldsymbol {x} \| \leq \epsilon , \boldsymbol {v} \in \mathcal {B} _ {\rho} (\boldsymbol {w})} \mathcal {L} (f (\boldsymbol {x} ^ {a d v}, \boldsymbol {w} + \boldsymbol {v}), y), \tag {24}
$$
where  $\mathcal{B}_{\rho}(\pmb{w}) = \{\pmb{w} + \pmb{v} \mid \| \pmb{v}_i \|_2 \leq \rho \| \pmb{w}_i \|_2 \forall i$ -th layer\}. As described by Wu et al. [64],  $\pmb{x}^{adv}$  is calculated based on the non-perturbed model  $f(\pmb{w})$ , and a single step of maximization with respect to  $\pmb{v}$  is sufficient to improve robustness. We used the best-performing value of  $\rho = 5 \times 10^{-3}$  from [64].