SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {Z _ {n}} \mathbb {E} _ {\Theta \sim \Theta_ {P | Z _ {n}}} \mathbb {E} _ {X} \| f (\Theta , X) - \bar {f} _ {P} (X) \| _ {1} ^ {2} \\ \leq 2 \mathbb {E} _ {Z _ {n}} \mathbb {E} _ {\Theta \sim \Theta_ {P | Z _ {n}}} \mathbb {E} _ {X} \left[ \| f (\Theta , X) - \bar {f} _ {P | Z _ {n}} (X) \| _ {1} ^ {2} + \| \bar {f} _ {P | Z _ {n}} (X) - \bar {f} _ {P} (X) \| _ {1} ^ {2} \right] \\ \leq 2 \mathbb {E} _ {Z _ {n}} \mathbb {E} _ {\Theta , \Theta^ {\prime} \sim \Theta_ {P | Z _ {n}}} \mathbb {E} _ {X} \| f (\Theta , X) - f (\Theta^ {\prime}, X) \| _ {1} ^ {2} + 2 \mathbb {E} _ {Z _ {n}} \mathbb {E} _ {X} \| \bar {f} _ {P | Z _ {n}} (X) - \bar {f} _ {P} (X) \| _ {1} ^ {2} \\ \leq 4 \mathbb {E} _ {Z _ {n}} \mathbb {E} _ {\Theta , \Theta^ {\prime} \sim \Theta_ {P \mid Z _ {n}}} \mathbb {E} _ {X} \mathrm {K L} (f (\Theta , X) | | f (\Theta^ {\prime}, X)) + 2 \mathbb {E} _ {Z _ {n}} \mathbb {E} _ {X} \| \bar {f} _ {P \mid Z _ {n}} (X) - \bar {f} _ {P} (X) \| _ {1} ^ {2} \\ = 4 \mathcal {C} _ {P} + 2 \mathbb {E} _ {Z _ {n}} \mathbb {E} _ {X} \| \bar {f} _ {P | Z _ {n}} (X) - \bar {f} _ {P} (X) \| _ {1} ^ {2} \leq 4 \mathcal {C} _ {P} + 2 \mathbb {E} _ {Z _ {n}} \mathbb {E} _ {Z _ {n} ^ {\prime}} \mathbb {E} _ {X} \| \bar {f} _ {P | Z _ {n}} (X) - \bar {f} _ {P | Z _ {n} ^ {\prime}} (X) \| _ {1} ^ {2} \\ \leq 4 \mathcal {C} _ {P} + 4 \mathbb {E} _ {Z _ {n}} \mathbb {E} _ {Z _ {n} ^ {\prime}} \mathbb {E} _ {X} \mathrm {K L} (\bar {f} _ {P \mid Z _ {n}} (X) | | \bar {f} _ {P \mid Z _ {n} ^ {\prime}} (X)) = 4 \left(\mathcal {C} _ {P} + \mathcal {S} _ {P}\right) = 4 \mathcal {D} _ {P}. \tag {2} \\ \end{array}
$$
Using (1), (2), and Lemma A.1, we obtain
$$
\lambda \mathbb {E} _ {Z _ {n}} \mathbb {E} _ {\Theta \sim \Theta_ {P | Z _ {n}}} \mathbb {E} _ {Z} \left[ n \ell_ {\Theta} (Z) - \sum_ {i = 1} ^ {n} \ell_ {\Theta} \left(X _ {i}, Y _ {i}\right) \right] \leq n \gamma^ {2} \psi (\lambda) \lambda^ {2} \mathcal {D} _ {P} + \mathbb {E} _ {Z _ {n}} \operatorname {K L} \left(\Theta_ {P | Z _ {n}} \| \Theta^ {0}\right). \tag {3}
$$
Set  $\Theta^0 = \mathbb{E}_{Z_n'}\Theta_{P|Z_n'}$  so that we have
$$
\mathcal {I} _ {P} = \mathbb {E} _ {Z _ {n}} \operatorname {K L} \left(\Theta_ {P \mid Z _ {n}} | | \Theta^ {0}\right). \tag {4}
$$
Also note that  $\bar{f}_P$  in  $\ell_{\Theta}$  cancels out since  $Z_{n}$  is iid samples of  $\mathbf{Z}$ , and so using the notation for the test loss and empirical loss defined in Theorem 2.1, we have
$$
\lambda \mathbb {E} _ {Z _ {n}} \mathbb {E} _ {\Theta \sim \Theta_ {P | Z _ {n}}} \mathbb {E} _ {Z} \left[ n \ell_ {\Theta} (Z) - \sum_ {i = 1} ^ {n} \ell_ {\Theta} \left(X _ {i}, Y _ {i}\right) \right] = n \lambda \mathbb {E} _ {Z _ {n}} \mathbb {E} _ {\Theta \sim \Theta_ {P | Z _ {n}}} \left[ \Phi_ {\mathbf {Z}} (\Theta) - \Phi (\Theta , Z _ {n}) \right]. \tag {5}
$$
(3), (4) and (5) imply the result.
B Additional figures
Figure 10 supplements 'Relation to disagreement' at the end of Section 2. It shows an example where the behavior of inconsistency is different from disagreement. Training was done on  $10\%$  of ImageNet with the seed procedure (tuned to perform well) with training length variations described in Table 6 below. Essentially, in this example, inconsistency goes up like generalization gap, and disagreement goes down like test error and goes up in the end, as training becomes longer.
Figure 11 supplements Figure 4 in Section 3.1. It shows inconsistency and instability of model outputs of the models trained with SGD with constant learning rates without iterate averaging. In this setting of high final randomness, larger learning rates make inconsistency larger while instability is mostly unaffected. By contrast, Figure 12 shows that when final randomness is low (due to iterate averaging in this case), both inconsistency and instability are predictive of generalization gap both within and across the learning rates.
Figure 13-15 supplement Figure 8 in Section 3.2. These figures show the relation of inconsistency and sharpness to generalization gap. Note that each graph has at least 16 points, and some of them (typically for the models trained with the same procedure) are overlapping. Inconsistency shows a stronger correlation with generalization gap than sharpness does.
All the experiments were done using GPUs (A100 or older).
C.1 Details of the experiments in Section 3.1
The goal of the experiments reported in Section 3.1 was to find whether/how the predictiveness of  $\mathcal{D}_P$  is affected by the diversity of the training procedures in comparison. To achieve this goal, we chose the training procedures to experiment with in the following four steps.
(a) Generalization gap
(b) Inconsistency
Figure 10: Inconsistency  $\mathcal{C}_P$  and disagreement  $(y$ -axis) in comparison with generalization gap and test error  $(y$ -axis). The  $x$ -axis is train loss. The arrows indicate the direction of training becoming longer. Each point is the average of 16 instances. Training was done on  $10\%$  of ImageNet with the seed procedure (tuned to perform well) with training length variations; see Table 6. In this example, essentially, inconsistency goes up like generalization gap, and disagreement goes down like test error and goes up in the end, as training becomes longer.
(c) Disagreement
(d) Test error
Figure 11: Inconsistency  $\mathcal{C}_P$  (left) and instability  $S_P$  (right) ( $x$ -axis) and generalization gap ( $y$ -axis). Supplement to Figure 4 in Section 3.1. SGD with a constant learning rate. No iterate averaging, and therefore, high randomness in the final state. Same procedures (and models) as in Figure 2. A larger learning rate makes inconsistency larger, but instability is mostly unaffected.
(a) Inconsistency (b) Instability
Figure 12: Inconsistency  $\mathcal{C}_P$  (left) and instability  $S_P$  (right) ( $x$ -axis) and generalization gap ( $y$ -axis). SGD with a constant learning rate with iterate averaging, and therefore, low randomness in the final state. Same procedures (and models) as in Figure 1. Both inconsistency and instability are predictive of generalization gap across the learning rates.
(c) Inconsistency (d) Instability
(e) Inconsistency (f) Instability
1. Choose a network architecture and the size of training set. First, for each dataset, we chose a network architecture and the size of the training set. The training set was required to be smaller than the official set so that disjoint training sets can be obtained for estimating instability. For the network architecture, we chose relatively small residual nets (WRN-28-2 for CIFAR-10/100 and ResNet-50 for ImageNet) to reduce the computational burden.
2. Choose a seed procedure. Next, for each dataset, we chose a procedure that performs reasonably well with the chosen network architecture and the size of training data, and we call this procedure a seed procedure. This was done by referring to the previous studies [32, 10] and performing some tuning on the development data considering that the training data is smaller than in [32, 10]. This step was for making sure to include high-performing (and so practically interesting) procedures in our empirical study.
3. Make core procedures from the seed procedure. For each dataset, we made core procedures from the seed procedure by varying the learning rate, training length, and the presence/absence of iterate averaging. Table 6 shows the resulting core procedures.
4. Diversify by changing an attribute. To make the procedures more diverse, for each dataset, we generated additional procedures by changing one attribute of the core procedure. This was done for all the pairs of the core procedures in Table 6 and the attributes in Table 7.
Legend for Case#10 (distillation)
+	-	△	▲	○	●
Teacher Student	standard standard	standard Consist.	Consist. standard	Consist. Consist.	Consist+Flat standard	Consist+Flat Consist.
Legend for Case#1-9: +Standard  $\triangle$  Consist.  $\times$  Flat.  $\circ$  Consist+Flat
(a) Case#1
(b) Case#2
(c) Case#3
(d) Case#4
(e) Case#5
(f) Case#6
(g) Case#7
(h) Case#8
(i) Case#9
(j) Case#10
Figure 13: Supplement to Figure 8 in Section 3.2. Inconsistency  $(x$ -axis) and generalization gap  $(y$ -axis) for all the 10 cases. All values are standardized so that the average is 0 and the standard deviation is 1.
(a) Case#1
Figure 14: Supplement to Figure 8 in Section 3.2. 1-sharppness  $(x$ -axis) and generalization gap  $(y$ -axis) for all the 10 cases. All values are standardized. Same legend as in Figure 13.
(a) Case#1
(f) Case#6
Figure 15: Supplement to Figure 8 in Section 3.2. Hessian  $(x$ -axis) and generalization gap  $(y$ -axis) for all the 10 cases. All values are standardized except that for the  $x$ -axis of Case#8 and 9, non-standardized values are shown in the log-scale for better readability. Same legend as in Figure 13.
Table 6: Core training procedures of the experiments in Section 3.1. The core training procedures consist of the exhaustive combinations of these attributes. The seed procedure attributes are indicated by * when there are multiple values. The optimizer was fixed to SGD with Nesterov momentum 0.9.
† More precisely, { 25, 50, ..., 250, 300, ..., 500, 600, ..., 1000, 1200, ..., 2000 } so that the interval gradually increased from 25 to 200. ‡ After training, a few procedures with very high training loss were excluded from the analysis. The cut-off was 0.3 (CIFAR-10), 2.0 (CIFAR-100), and 3.0 (ImageNet), reflecting the number of classes (10, 100, and 1000).
The choice of the constant scheduling is for starting the empirical study with simpler cases by avoiding the complex effects of decaying learning rates, as mentioned in the main paper; also, we found that in these settings, constant learning rates rival the cosine scheduling as long as iterate averaging is performed.
CIFAR-10/100	ImageNet
Network	WRN-28-2	ResNet-50
Training data size	4K	120K
Learning rate	{0.005, 0.01, 0.025*, 0.05}	{1/64, 1/32, 1/16*, 1/8}
Weight decay	2e-3	1e-3
Schedule	Constant	Constant
Iterate averaging	{EMA*, None}	{EMA*, None}
Epochs	{25, ..., 1000*, ..., 2000}†‡	{10, 20, ..., 200*}‡
Mini-batch size	64	512
Data augmentation	Standard+Cutout	Standard
Label smoothing	-	0.1
Table 7: Attributes that were varied for making variations of core procedures. Only one of the attributes was varied at a time.
CIFAR-10	CIFAR-100	ImageNet
Network	WRN-16-4	-	-
Weight decay	5e-4	-	1e-4
Schedule	Cosine	Cosine	Cosine
Mini-batch size	256	256	-
Data augmentation	None	-	-
Table 8: The values of the fixed attributes of the procedures shown in Figures 1-4 and 6 as well as 11-12. The training length and the learning rate were varied as shown in Table 6. The presence/absence of iterate averaging is indicated in each figure.
CIFAR-10	CIFAR-100	ImageNet
Network	WRN-28-2	ResNet-50
Training data size	4K	120K
Weight decay	2e-3	1e-3
Schedule	Constant	Constant
Mini-batch size	256	64	512
Data augmentation	Standard+Cutout	Standard
Label smoothing	-	0.1
Table 9: The values of the fixed attributes of the procedures shown in Figures 16-19. The training length and the learning rate were varied as shown in Table 6. The presence/absence of iterate averaging is indicated in each figure. The rest of the attributes are the same as in Table 8
CIFAR-10	CIFAR-100	ImageNet
Weight decay	5e-4	2e-3	1e-4
Mini-batch size	64	256	512
Note that after training, a few procedures with very high training loss were excluded from the analysis (see Table 6 for the cut-off). Right after the model parameter initialization, inconsistency  $\mathcal{C}_P$  is obviously not predictive of generalization gap since it is non-zero merely reflecting the initial randomness while generalization gap is zero. Similar effects of initial randomness are expected in the initial phase of training; however, these near random models are not of practical interest. Therefore, we excluded from our analysis.
C.1.1 SGD with constant learning rates (Figures 1-4 and 6)
In Section 3.1, we first focused on the effects of varying learning rates and training lengths while fixing anything else, with the training procedures that use a constant learning rate with or without iterate averaging. We analyzed all the subsets of the procedures that met this condition, and reported the common trend. That is, when the learning rate is constant, with iterate averaging,  $\mathcal{D}_P$  is predictive of generalization gap within and across the learning rates, and without iterate averaging,  $\mathcal{D}_P$  is predictive of generalization gap only for the procedures that share the learning rate; moreover, without iterate averaging, larger learning rates cause  $\mathcal{D}_P$  to overestimate generalization gap by larger amounts. Figures 1-4 and 6 show one particular subset for each dataset, and Table 8 shows the values of the attributes fixed in these subsets. To demonstrate the generality of the finding, we show the corresponding figures for one more subset for each dataset in Figures 16-19. The values of the fixed attributes in these subsets are shown in Table 9.
Smallest learning rate
2nd smallest l.r.
3rd smallest l.r.
Largest l.r.
(c) ImageNet
Figure 16:  $\mathcal{D}_P$  ( $x$ -axis) and generalization gap ( $y$ -axis). SGD with a constant learning rate and iterate averaging. Only the learning rate and training length were varied as in Figure 1 and the attributes were fixed to the values different from Figure 1; see Table 9 for the fixed values. As in Figure 1, a positive correlation is observed between  $\mathcal{D}_P$  and generalization gap.
CIFAR-10
(a) Inconsistency
(b) Instability
(c) Inconsistency
CIFAR-100
Figure 17:  $\mathcal{D}_P(x$ -axis) and generalization gap ( $y$ -axis). Constant learning rates. No iterate averaging. Only the learning rate and training length were varied as in Figure 2 and the attributes were fixed to the values different from Figure 2; see Table 9 for the fixed the values. As in Figure 2,  $\mathcal{D}_P$  is predictive of generalization gap for the procedures that share the learning rate, but not clear otherwise.
(d) Instability
ImageNet
(e) Inconsistency
(f) Instability
Figure 18: Inconsistency  $\mathcal{C}_P$  (left) and instability  $S_P$  (right) ( $x$ -axis) and generalization gap ( $y$ -axis). Same procedures (and models) as in Figure 17 (no iterate averaging). As in Figures 4 and 11 (also no iterate averaging), a larger learning rate makes inconsistency larger, but instability is mostly unaffected.
(a) CIFAR-10
Figure 19: Disagreement  $(x$ -axis) and test error  $(y$ -axis). Same models and legend as in Fig 16.
C.1.2 Procedures with low final randomness (Figures 5 and 7)
The procedures shown in Figures 5 and 7 are subsets (three subsets for three datasets) of all the procedures (the core procedures in Table 6 times the attribute changes in Table 7). The subsets consist of the procedures with either iterate averaging or a vanishing learning rate (i.e., going to zero) so
that they meet the condition of low final randomness. These subsets include those with the cosine learning rate schedule. With the cosine schedule, we were interested in not only letting the learning rate go to zero but also stopping the training before the learning rate reaches zero and setting the final model to be the iterate averaging (EMA) at that point, which is well known to be useful (e.g., [32]). Therefore, we trained the models with the cosine schedule for { 250, 500, 1000, 2000 } epochs (CIFAR-10/100) or 200 epochs (ImageNet), and saved the iterate averaging of the models with the interval of one tenth (CIFAR-10/100) or one twentieth (ImageNet) of the entire epochs.
C.2 Details of the experiments in Section 3.2
Section 3.2 studied inconsistency in comparison with sharpness in the settings where these two quantities are reduced by algorithms. The training algorithm with consistency encouragement (co-distillation) is summarized in Algorithm 1. These experiments were designed to study
- practical models trained on full-size training data, and
- diverse models resulting from diverse training settings,
in the situation where algorithms are compared after basic tuning is done, rather than the hyperparameter tuning-like situation in Section 3.1.
C.2.1 Training of the models
Table 10: Basic settings shared by all the models for each case (Case#1-7,10; images)
Training type	From scratch	Fine-tuning	Distillation
Dataset Network	ImageNet ResNet50	ViT	Food101 ViT / Mixer	CIFAR10 WRN28-2	Cars / Dogs EN-B0	Food101 ResNet-18
Batch size	512	4096	512	64	256	512
Epochs	100	300	200 / 100	-	-	400
Update steps	-	-	-	500K	4K / 2K	-
Warmup steps	0	10K	0	0	0	0
Learning rate	0.125	3e-3	3e-3	0.03	0.1	0.125
Schedule	Cosine	Linear/Cosine	Cosine	Constant	Cosine
Optimizer	Momentum	AdamW	AdamW	Momentum	Momentum	Momentum
Weight decay	1e-4	0.3	0.3	5e-4	1e-5	1e-3
Label smooth	0.1	0	0.1	0	0	0
Iterate averaging	-	-	-	EMA	EMA	-
Gradient clipping	-	1.0	1.0	-	20.0	-
Data augment	Standard	Cutout	Standard
Reference	[10]	[5]	[5]	[10],[32]	[10],[33]	[10]
Case#	1	2	3 / 4	5	6 / 7	10
'Momentum': SGD with Nesterov momentum 0.9.
This section describes the experiments for producing the models used in Section 3.2.
Basic settings Within each of the 10 cases, we used the same basic setting for all, and these shared basic settings were adopted/adapted from the previous studies when possible. Tables 10 and 11
Table 11: Basic settings shared by all the models for each case (Case#8-9; text). Hyperparameters for Case#8-9 (text) basically followed the RoBERTa paper [26]. The learning rate schedule was equivalent to early stopping of 10-epoch linear schedule after 4 epochs. Although it appears that [26] tuned when to stop for each run, we used the same number of epochs for all. Iterate averaging is our addition, which consistently improved performance.
Initial learning rate η0	1e-5
Learning rate schedule	Linear from η0 to 0.6η0
Epochs	4
Batch size	32
Optimizer	AdamW (β1=0.9, β2=0.98, ε=1e-6)
Weight decay	0.1
Iterate averaging	EMA with momentum 0.999
describe the basic settings and the previous studies that were referred to. Some changes to the previous settings were made for efficiency in our computing environment (no TPU); e.g., for Case#1, we changed the batch size from 4096 to 512 and accordingly the learning rate from 1 to 0.125. When adapting the previous settings to new datasets, minimal tuning was done for obtaining reasonable performance, e.g., for Cases#3 and 4, we changed batch size from 4096 to 512 and kept the learning rate without change as it performed better.
For CIFAR-10, following [32], we let the learning rate decay to  $0.2\eta_0$  instead of 0 and set the final model to the EMA of the models with momentum 0.999. For Cases#6-7 (fine-tuning), we used a constant learning rate and used the EMA of the models with momentum 0.999 as the final model, which we found produced reasonable performance with faster training. Cases#6-7 fine-tuned the publicly available EfficientNet-B0 $^2$  pretrained with ImageNet by [33]. The dropout rate was set to 0.1 for Case#3, and the stochastic depth drop rate was set to 0.1 for Case#4. The teacher models for Case#10 (distillation) were ensembles of ResNet-18 trained with label smoothing 0.1 for 200 epochs with the same basic setting as the student models (Table 10) otherwise.
For CIFAR-10, the standard data augmentation (shift and horizontal flip) and Cutout [8] were applied. For the other image datasets, only the standard data augmentation (random crop with distortion and random horizontal flip) was applied; the resolution was  $224 \times 224$ .
Table 12: Hyperparameters for SAM.
Case#	1	2	3	4	5	6	7	8,9	10
m-sharpness	128	256	32	32	32	16	16	2	128
ρ	0.05,0.1	0.05	0.1	0.1	0.1,0.2	0.1,0.2	0.1	0.005,0.01	0.1
Hyperparameters for SAM There are two values that affect the performance of SAM,  $m$  for  $m$ -sharpness and the diameter of the neighborhood  $\rho$ . Their values are shown in Table 12. [10] found that smaller  $m$  performs better. However, a smaller  $m$  can be less efficient as it can reduce the degree of parallelism, depending on the hardware configuration. We made  $m$  no greater than the reference study in most cases, but for practical feasibility we made it larger for Case#2.  $\rho$  was either set according to the reference when possible or chosen on the development data otherwise, from {0.05, 0.1, 0.2} for images and from {0.002, 0.005, 0.01, 0.02, 0.05, 0.1} for texts. For some cases (typically those with less computational burden), we trained the models for one additional value of  $\rho$  to have more data points.
Hyperparameters for the inconsistency penalty term The weight of the inconsistency penalty term for encouraging consistency was fixed to 1.
Number of the models For each training procedure (identified by the training objective within each case), we obtained 4 models trained with 4 distinct random sequences. Cases#1-9 consisted of either 4 or 6 procedures depending on the number of the values chosen for  $\rho$  for SAM. Case#10 (distillation) consisted of 6 procedures $^3$ , resulting from combining the choice of the training objectives for the teacher and the choice for the student. In total, we had 52 procedures and 208 models.
C.2.2 Estimation of the model-wise inconsistency and sharpness in Section 3.2
When the expectation over the training set was estimated, for a large training set such as ImageNet, 20K data points were sampled for this purpose. As described above, we had 4 models for each of the training procedures. For the procedure without encouragement of low inconsistency, the expectation of the divergence of each model was estimated by taking the average of the divergence from the three other models. As for the procedure with encouragement of low inconsistency, the four models were obtained from two runs as each run produced two models, and so when averaging for estimating inconsistency, we excluded the divergence between the models from the same run due to their dependency.
$^{2}$  https://github.com/google-research/sam
3Although the number of all possible combinations is 16, considering the balance with other cases, we chose to experiment with the following: teacher  $\{$  'Standard', 'Consist', 'Consist+Flat'  $\} \times$  student  $\{$  'Standard', 'Consist.'\}
Algorithm 1: Training with consistency encouragement. Co-distillation (named by [1], closely related to deep mutual learning [42]). Without additional unlabeled data.
Input & Notation: Labeled set  $Z_{n}$ ,  $\beta$  (default: 1), learning rate  $\eta$ . Let  $\phi$  be loss, and let  $f(\theta, x)$  be the model output in the form of probability estimate.
1 Sample  $\theta^{\mathrm{a}}$  and  $\theta^{\mathrm{b}}$  from the initial distribution.
2 for  $\bar{t} = 1,\dots ,T$  do
3 Sample two labeled mini-batches  $B^{\mathrm{a}}$  and  $B^{\mathrm{b}}$  from  $Z_{n}$
$$
\begin{array}{l} 4 \quad \left| \begin{array}{c} \theta^ {\mathrm {a}} \leftarrow \theta^ {\mathrm {a}} - \eta_ {t} \nabla_ {\theta^ {\mathrm {a}}} \left[ \frac {1}{\left| B ^ {\mathrm {a}} \right|} \sum_ {(x, y) \in B ^ {\mathrm {a}}} \phi (f (\theta^ {\mathrm {a}}, x), y) + \beta \frac {1}{\left| B ^ {\mathrm {a}} \right|} \sum_ {(x, \cdot) \in B ^ {\mathrm {a}}} \mathrm {K L} (f (\theta^ {\mathrm {b}}, x) | | f (\theta^ {\mathrm {a}}, x)) \right] \end{array} \right. \\ \left. \begin{array}{l} 5 \quad \left\lfloor \theta^ {\mathrm {b}} \leftarrow \theta^ {\mathrm {b}} - \eta_ {t} \nabla_ {\theta^ {\mathrm {b}}} \left[ \frac {1}{| B ^ {\mathrm {b}} |} \sum_ {(x, y) \in B ^ {\mathrm {b}}} \phi (f (\theta^ {\mathrm {b}}, x), y) + \beta \frac {1}{| B ^ {\mathrm {b}} |} \sum_ {(x, \cdot) \in B ^ {\mathrm {b}}} \mathrm {K L} (f (\theta^ {\mathrm {a}}, x) | | f (\theta^ {\mathrm {b}}, x)) \right] \right. \end{array} \right. \\ \end{array}
$$
1-sharppness requires  $\rho$  (the diameter of the local region) as input. We set it to the best value for SAM.
C.3 Details of the experiments in Sections 3.3
The optimizer was SGD with Nesterov momentum 0.9.