SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Disagreement (Fig 6,7) For the same procedures/models as in Figures 1 and 5, we show the relationship between disagreement and test error in Figures 6 and 7, respectively. We chose test error instead of generalization gap for the  $y$ -axis since the previous finding was 'Disagreement  $\approx$  Test error'. The straight lines are  $y = x$ . The relation in Fig 6 (a) and Fig 7 (a) is close to equality, but the relation appears to be more complex in the others. The procedures plotted in Fig 7 are those which satisfy the condition for  $\mathcal{D}_P$  to be predictive of generalization gap; thus, the results indicate that the condition for disagreement to be predictive of test error is apparently different.
(a) CIFAR-10
Figure 6: Disagreement  $(x$ -axis) and test error  $(y$ -axis). SGD with a constant learning rate and iterate averaging. Same models and legend as in Fig 1. "Disagreement  $\approx$  Test error" of the previous studies does not quite hold in (b) and (c).
(a) CIFAR-10
Figure 7: Disagreement  $(x$ -axis) and test error  $(y$ -axis). The models are the same as in Fig 5, which satisfy the condition for  $\mathcal{D}_P$  to be predictive of generalization gap. The results indicate that the condition for disagreement to be predictive of test error is different.
Note that while the previous empirical study of disagreement [27, 17] focused on the models with zero training error, we studied a wider range of models. Unlike CIFAR-10/100 (on which zero training error can be quickly achieved), a common practice for datasets like ImageNet is budgeted training [25] instead of training to zero error. Also note that zero training error does not necessarily lead to the best performance, which, for example, has been observed in the ImageNet experiments of this section.
3.2 On the predictiveness of inconsistency  $\mathcal{C}_P$ : from an algorithmic perspective
Based on the results above, this section focuses on inconsistency  $\mathcal{C}_P$ , which enables experiments with larger training data, and studies the behavior of inconsistency in comparison with sharpness from an algorithmic perspective.
Table 1: Datasets. Mostly full-size natural images and 2 texts.
Table 2: Networks.
Name	#class	#train	#dev	#test	Network	#param	Case
provided	used	ResNet-50 [12]	24M	1
ImageNet [7]	1000	1.28M	1.27M	10K	50K	ViT-S/32 [9]	23M	2
Food101 [3]	101	75750	70700	5050	25250	ViT-B/16 [9]	86M	3
Dogs [21]	120	12000	10800	1200	8580	Mixer-B/16 [35]	59M	4
Cars [23]	196	8144	7144	1000	8041	WRN-28-2 [40]	1.5M	5
CIFAR-10 [24]	10	50000	4000	5000	10000	EN-B0 [33]	4.2M	6,7
MNLI [36]	3	392702	382902	9800	9815	roberta-base[26]	124.6M	8,9
QNLI [36]	2	104743	99243	5500	5463	ResNet-18 [12]	11M	10
Training objectives Parallel to the fact that SAM seeks flatness of training loss landscape, a meta-algorithm co-distillation (named by [1], and closely related to deep mutual learning [42], summarized in Algorithm 1 in the Appendix) encourages consistency of model outputs. It simultaneously trains two (or more) models with two (or more) different random sequences while penalizing the inconsistency between the predictions of the two models. This is typically described as 'teaching each other', but we take a different view of consistency encouragement. Note that the penalty term merely 'encourages' low inconsistency. Since inconsistency by definition depends on the unknown data distribution, it cannot be directly minimized by training. The situation is similar to minimizing loss on the training data with the hope that loss will be small on unseen data. We analyzed the models that were trained with one of the following training objectives (shown with the abbreviations used below):
- 'Standard': the standard cross-entropy loss.
- 'Consist': the standard loss with encouragement of consistency.
- 'Flat': the SAM objective, which encourages flatness.
- 'Consist+Flat': encouraging both consistency and flatness. Coupling two instances of SAM training with the inconsistency penalty term.
**Cases** We experimented with ten combinations (cases) of dataset, network architecture, and training scenario. Within each case, we fixed the basic settings (e.g., weight decay, learning rate) to the ones known to perform well from the previous studies, and only varied the training objectives so that for each case we had at least four stochastic training procedures distinguished by the four training objectives (for some cases, we had more than four due to testing multiple values of  $\rho$  for SAM). We obtained four models (trained with four different random sequences) per training procedure. Tables 1 and 2 show the datasets and network architectures we used, respectively. Note that as a result of adopting high-performing settings, all the cases satisfy the condition of low final randomness.
3.2.1 Results
As in the previous section, we quantify the generalization gap by the difference of test loss from training loss (note that the loss is the cross-entropy loss; the inconsistency penalty or anything else is not included); the inconsistency values were again measured on the held-out unseen unlabeled data, disjoint from both training data and test data. Two types of sharpness values were measured, 1-sharppness (per-example loss difference in the adversarial direction) of [10] and the magnitude of the loss Hessian (measured by the largest eigenvalue), which represent the sharpness of training loss landscape and have been shown to correlate to generalization performance [10].
Inconsistency correlates to generalization gap (Figure 8) Figure 8 shows inconsistency or sharpness values  $(x$ -axis) and the generalization gap  $(y$ -axis). Each point in the figures represents a model; i.e., in this section, we show model-wise quantities (instead of procedure-wise), simulating the model selection setting. (Model-wise inconsistency for model  $\theta$  trained on  $Z_{n}$  with procedure  $P$  is  $\mathbb{E}_{\Theta \sim \Theta_{P|Z_n}}\mathbb{E}_X\mathrm{KL}(f(\Theta ,X)||f(\theta ,X))$ .) All quantities are standardized so that the mean is zero and the standard deviation is 1. In the figures, we distinguish the four training objectives. In Figure 8 (b) and (c), we confirm, by comparing the 'Flat.' models  $(\times)$  with the baseline standard models  $(+)$ , that encouragement of flatness (by SAM) indeed reduces sharpness  $(x$ -axis) as well as the generalization gap  $(y$ -axis). This serves as a sanity check of our setup.
Food101 ViT-B16, from scratch
(a) Inconsistency
(b) 1-sharppness
(c) Hessian
Figure 8: Inconsistency and sharpness  $(x$ -axis) and generalization gap  $(y$ -axis). Each point represents a model. Note that each graph plots 16-20 points, and some of them are overlapping.
MNLI, fine-tuning of RoBERTa-base
(d) Inconsistency
(e) 1-sharppness
(f) Hessian
We observe in Figure 8 that inconsistency shows a clear positive correlation with the generalization gap, but the correlation of sharpness with generalization gap is less clear (e.g., in (b)-(c), the generalization gap of the 'Consist.' models  $(\triangle)$  is much smaller than that of the 'Flat.' models  $(\times)$ , but their sharpness is larger). This is a general trend observed across multiple network architectures (ResNet, Transformers, MLP-Mixer, and so on), multiple datasets (images and texts), and multiple training scenarios (from scratch, fine-tuning, and distillation); the Appendix provides the figures for all 10 cases. Moreover, we found that while both sharpness reduction and inconsistency reduction lead to better generalization, in a number of cases inconsistency and generalization gap are reduced and yet sharpness remains relatively high. We view this phenomenon as an indication that inconsistency has a stronger (and perhaps more essential) connection with generalization gap than sharpness.
Inconsistency is more predictive of generalization gap than sharpness (Table 3) Motivated by the linearity observed in the figure above, we define a linear predictor of the generalization gap that takes the inconsistency or sharpness value as input. We measure the predictive power of the metrics through the prediction performance of this linear predictor trained with least square minimization. To make the inconsistency and sharpness values comparable, we standardize them and also generalization gap. For each case, we evaluated the metrics by performing the leave-one-out cross validation on the given set of models (i.e., perform least squares using all models but one and evaluate the obtained linear predictor on the left-out model; do this  $k$  times for  $k$  models and take the average). The results in Table 3 show that inconsistency outperforms sharpness, and the superiority still generally holds even when sharpness is given an 'unfair' advantage of additional labeled data.
Table 3: Generalization gap prediction error. The leave-one-out cross validation results, which average the generalization gap prediction residuals, are shown. See Table 2 for the networks used for each case. Smaller is better, and a value near 1.0 is very poor. Inconsistency: estimated on unlabeled data, 1-sharpness and Hessian: sharpness of training loss landscape. Inconsistency outperforms the sharpness metrics. Noting that inconsistency had access to additional data, even though unlabeled, we also show in the last 2 rows sharpness of test loss landscape estimated on the development data (held-out labeled data), which gives the sharpness metrics an 'unfair' advantage by providing them with additional labels. With this advantage, the predictiveness of sharpness mostly improves; however, still, inconsistency is generally more predictive. The bold font indicates that the difference is statistically significant at  $95\%$  confidence level against all others including the last 2 rows. The italic font indicates the best but not statistically significant.
Case#	1	2	3	4	5	6	7	8	9	10
Dataset	ImageNet	Food101	C10	Dog	Car	Mnli	Qnli	Food
Training scenario	From scratch	Fine-tuning	Distill.
Inconsistency	0.13	0.10	0.17	0.27	0.15	0.09	0.19	0.15	0.18	0.19
1-sharpness	0.21	0.48	0.75	0.74	0.84	0.34	0.24	0.58	0.98	0.75
Hessian	0.50	1.00	0.77	0.72	0.78	0.70	0.68	0.59	0.87	0.59
Giving the sharpness metrics an 'unfair' advantage by providing additional labels:
1-sharpsness of dev. loss	0.14	0.64	0.47	0.40	0.72	0.22	0.14	0.36	0.37	0.62
Hessian of dev. loss	0.45	0.82	0.58	0.75	0.13	0.27	0.15	0.91	0.81	0.89
3.3 Practical impact: algorithmic consequences
The results above suggest a strong connection of inconsistency of model outputs with generalization, which also suggests the importance of seeking consistency in the algorithmic design. To confirm this
+Standard  $\triangle$  Consist.  $\times$  Flat.  $\odot$  Consist+Flat
point, we first experimented with two methods (ensemble and distillation) and found that in both, encouraging low inconsistency in every stage led to the best test error. Note that in this section we report test error instead of generalization gap in order to show the practical impact.
(a) Ensemble
Figure 9: Inconsistency and test error with ensemble and distillation.
(b) Distillation. Teacher's knowledge level increases from left to right.
Ensemble (Fig 9 (a)) Figure 9 (a) shows that, compared with non-ensemble models (+), ensemble models (averaging logits of two models) reduce both inconsistency  $(x$ -axis) and test error  $(y$ -axis). It is intuitive that averaging the logits would cancel out the dependency on (or particularities of) the individual random sequences inherent in each single model; thus, outputs of ensemble models are expected to become similar and so consistent (as empirically observed). Moreover, we presume that reduction of inconsistency is the mechanism that enables ensemble to achieve better generalization. The best test error was achieved when the ensemble was made by two models trained with consistency encouragement ( $\bullet$  in Fig 9 (a)).
Distillation (Fig 9 (b)) We experimented with distillation [13] with encouragement of consistency for the teacher model and/or the student model. Figure 9 (b) shows the test error  $(y$ -axis) and the inconsistency  $(x$ -axis). The standard model  $(+)$  serves as the baseline. The knowledge level of teachers increases from left to right, and within each graph, when/whether to encourage consistency differs from the point to point. The main finding here is that at all the knowledge levels of the teacher, both inconsistency and test error go down as the pursuit of consistency becomes more extensive; in each graph, the points for the distilled models form a nearly straight line. (The line appears to shift away from the baseline  $(+)$  as the amount of external knowledge of the teacher increases. We conjecture this could be due to the change in  $\mathcal{I}_P$ .) While distillation alone reduces inconsistency and test error, the best results are obtained by encouraging consistency at every stage of training (i.e., training both the teacher and student with consistency encouragement). The results underscore the importance of inconsistency reduction for better performance.
On the pursuit of the state of the art (Table 4,5) We show two examples of obtaining further improvement of state-of-the-art results by adding consistency encouragement. The first example is semi-supervised learning with CIFAR-10. Table 4 shows that the performance of a state-of-the-art semi-supervised method can be further improved by encouraging consistency between two training instances of this method on the unlabeled data (Algorithm 2 in the Appendix). The second example is transfer learning. We fine-tuned a public ImageNet-trained EfficientNet with a more focused dataset Food101. Table 5 shows that encouraging consistency between two instances of SAM training improved test error. These examples demonstrate the importance of consistency encouragement in the pursuit of the state of the art performance.
4 Limitations and discussion
Theorem 2.1 assumes a bounded loss  $\phi(f, y) \in [0, 1]$  though the standard loss for classification is the cross-entropy loss, which is unbounded. This assumption can be removed by extending the theorem to a more complex analysis with other moderate assumptions. We, however, chose to present the simpler and so more intuitive analysis with a bounded loss. As noted above, this theorem is intended
Table 4: CIFAR-10 #train=4K, #unlabeled=41K, WRN28-2. Average and standard deviation of 5 runs.
Methods	Error (%)
Copied from [32]	MixMatch [2]	6.42
UDA [37]	4.88
FixMatch [32]	4.26
Our results	FixMatch	4.25±0.15
UDA (no sharpening)	4.33±0.10
UDA + 'Consist.'	3.95±0.12
RandAugment [6] was used for UDA and FixMatch.
Table 5: Food101, EfficientNet-B4. The average and standard deviation of 6 models are shown.
Methods	Error (%)
Previous results	EN-B4 [33]	8.5
EN-B7 [10]	7.17
EN-B7 SAM [10]	7.02
Our results	EN-B4 SAM †	6.00±0.05
EN-B4 SAM + 'Consist.'	5.77±0.04
The improvement over EN-B7 SAM of [10] is due to the difference in the basic setting; see Appendix.
for stochastic training procedures. The bound may not be useful for deterministic procedures, and this characteristics is not unique to our analysis but shared by the previous information-theoretic analysis of stochastic machine learning algorithms [38, 30, 28].
We acknowledge that due to resource constraints, there was a limitation to the diversity of the training procedures we empirically analyzed. In particular, our study of inconsistency and instability in Section 3.1 started with simpler cases of constant learning rate SGD and later included the cases of learning rate decay (Figure 5); consequently, the types of the training procedures studied in Section 3.1 were skewed towards the constant learning rate SGD. The models analyzed in Figure 5 were less diverse with CIFAR-100 and ImageNet than with CIFAR-10, as noted above, and again this asymmetry was due to the resource constraints. On the other hand, the models analyzed in this work are in a sense more diverse than the previous empirical studies [27, 17] of the disagreement metric (related to our inconsistency), which were restricted to the models with near zero train error.
In Section 3.2, we studied inconsistency from the algorithmic perspective in comparison with the sharpness metrics. We chose sharpness for comparison due to its known correlation to generalization performance and the existence of the algorithm (SAM) to reduce it. We acknowledge that there can be other metrics that are predictive of generalization gap.
Although the correlation of the disagreement metric with generalization gap is relatively poor in the settings of Section 3.1 (e.g., Figure 10), it is plausible that disagreement can be as predictive of generalization gap as inconsistency in some settings, for example, when the confidence level of predictions is sufficiently high since in that case, Theorem 2.1 should provide a theoretical basis also for disagreement though indirectly (details are provided in Appendix D.3). Moreover, for improving stochastic training of deep neural networks, it would be useful to understand the connection between generalization performance and discrepancies of model outputs in general (whether instability, inconsistency, disagreement, or something else), and we hope that this work contributes to pushing forward in this direction.
We presented a theory that relates generalization gap to instability and inconsistency of model outputs, which can be estimated on unlabeled data, and empirically showed that they are strongly predictive of generalization gap in various settings. In particular, inconsistency was shown to be a more reliable indicator of generalization gap than commonly used local flatness. We showed that algorithmic encouragement of consistency reduces inconsistency as well as test error, which can lead to further improvement of the state of art performance. Finally, our results provide a theoretical basis for existing methods such as co-distillation and ensemble.
[1] Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E. Dahl, and Geoffrey E. Hinton. Large scale distributed neural network training through online distillation. In International Conference on Learning Representations (ICLR), 2018.
[2] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. MixMatch: A holistic approach to semi-supervised learning. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.
As in the main paper, let  $Z = (X,Y)$  be a random variable representing a labeled data point (data point  $X$  and label  $Y$ ) with distribution  $\mathbf{Z}$ . Let  $Z_{n} = \{(X_{i},Y_{i}) : i = 1,2,\dots,n\}$  be a random variable representing iid training data of size  $n$  drawn from  $\mathbf{Z}$ .
We have the following lemma.
Lemma A.1. Given an arbitrary model parameter distribution  $\Theta^0$ , let
$$
\bar {f} _ {P} (x) = \mathbb {E} _ {Z _ {n} ^ {\prime}} \bar {f} _ {P | Z _ {n} ^ {\prime}} (x) = \mathbb {E} _ {Z _ {n} ^ {\prime}} \mathbb {E} _ {\Theta \sim \Theta_ {P | Z _ {n} ^ {\prime}}} f (\Theta , x),
$$
$$
\ell_ {\theta} (z) = \phi (f (\theta , x), y) - \phi (\bar {f} _ {P} (x), y) w h e r e z = (x, y)
$$
$$
- n \mathbb {E} _ {Z _ {n}} \mathbb {E} _ {\Theta \sim \Theta_ {P | Z _ {n}}} \ln \mathbb {E} _ {Z} \exp (- \lambda \ell_ {\Theta} (Z)) \leq \mathbb {E} _ {Z _ {n}} \mathbb {E} _ {\Theta \sim \Theta_ {P | Z _ {n}}} \sum_ {i = 1} ^ {n} \lambda \ell_ {\Theta} (X _ {i}, Y _ {i}) + \mathbb {E} _ {Z _ {n}} \mathrm {K L} (\Theta_ {P | Z _ {n}} | | \Theta^ {0}).
$$
Proof. Let  $\Theta_{*}$  be a model parameter distribution such that
$$
\boldsymbol {\Theta} _ {*} \propto \boldsymbol {\Theta} ^ {0} \exp \left[ \sum_ {i = 1} ^ {n} (- \lambda \ell_ {\boldsymbol {\Theta}} (X _ {i}, Y _ {i}) - \ln \mathbb {E} _ {Z} \exp (- \lambda \ell_ {\boldsymbol {\Theta}} (Z))) \right].
$$
$$
\begin{array}{l} \mathbb {E} _ {Z _ {n}} \exp \left[ \mathbb {E} _ {\Theta \sim \Theta_ {P | Z _ {n}}} \sum_ {i = 1} ^ {n} (- \lambda \ell_ {\Theta} (X _ {i}, Y _ {i}) - \ln \mathbb {E} _ {Z} \exp (- \lambda \ell_ {\Theta} (Z))) - \mathrm {K L} (\Theta_ {P | Z _ {n}} | | \Theta^ {0}) \right] \\ \leq \mathbb {E} _ {Z _ {n}} \sup  _ {\boldsymbol {\Theta}} \exp \left[ \mathbb {E} _ {\boldsymbol {\Theta} \sim \boldsymbol {\Theta}} \sum_ {i = 1} ^ {n} (- \lambda \ell_ {\boldsymbol {\Theta}} (X _ {i}, Y _ {i}) - \ln \mathbb {E} _ {Z} \exp (- \lambda \ell_ {\boldsymbol {\Theta}} (Z))) - \mathrm {K L} (\boldsymbol {\Theta} | | \boldsymbol {\Theta} ^ {0}) \right] \\ = \mathbb {E} _ {Z _ {n}} \sup  _ {\Theta} \mathbb {E} _ {\Theta \sim \Theta^ {0}} \exp \left[ \sum_ {i = 1} ^ {n} (- \lambda \ell_ {\Theta} (X _ {i}, Y _ {i}) - \ln \mathbb {E} _ {Z} \exp (- \lambda \ell_ {\Theta} (Z))) - \mathrm {K L} (\Theta | | \Theta_ {*}) \right] \\ = \mathbb {E} _ {Z _ {n}} \mathbb {E} _ {\Theta \sim \Theta^ {0}} \exp \left[ \sum_ {i = 1} ^ {n} (- \lambda \ell_ {\Theta} (X _ {i}, Y _ {i}) - \ln \mathbb {E} _ {Z} \exp (- \lambda \ell_ {\Theta} (Z))) \right] = 1. \\ \end{array}
$$
The first inequality takes sup over all probability distributions of model parameters. The first equality can be verified using the definition of the KL divergence. The second equality follows from the fact that the supreme is attained by  $\Theta = \Theta_{*}$ . The last equality uses the fact that  $(X_{i},Y_{i})$  for  $i = 1,\dots ,n$  are iid samples drawn from  $\mathbf{Z}$ . The desired bound follows from Jensen's inequality and the convexity of  $\exp (\cdot)$ .
Proof of Theorem 2.1. Using the notation of Lemma A.1, we have
$$
\begin{array}{l} \mathbb {E} _ {\Theta \sim \Theta_ {P \mid Z _ {n}}} \ln \mathbb {E} _ {Z} \exp (- \lambda \ell_ {\Theta} (Z)) \leq \mathbb {E} _ {\Theta \sim \Theta_ {P \mid Z _ {n}}} \mathbb {E} _ {Z} [ \exp (- \lambda \ell_ {\Theta} (Z) - 1 ] \\ \leq - \lambda \mathbb {E} _ {\Theta \sim \Theta_ {P | Z _ {n}}} \mathbb {E} _ {Z} \ell_ {\Theta} (Z) + \psi (\lambda) \lambda^ {2} \mathbb {E} _ {\Theta \sim \Theta_ {P | Z _ {n}}} \mathbb {E} _ {Z} \ell_ {\Theta} (Z) ^ {2} \\ \leq - \lambda \mathbb {E} _ {\Theta \sim \Theta_ {P \mid Z _ {n}}} \mathbb {E} _ {Z} \ell_ {\Theta} (Z) + \frac {\gamma^ {2}}{4} \psi (\lambda) \lambda^ {2} \mathbb {E} _ {\Theta \sim \Theta_ {P \mid Z _ {n}}} \mathbb {E} _ {X} \| f (\Theta , X) - \bar {f} _ {P} (X) \| _ {1} ^ {2}. \tag {1} \\ \end{array}
$$
The first inequality uses  $\ln u\leq u - 1$ . The second inequality uses the fact that  $\psi (\lambda)$  is increasing in  $\lambda$  and  $-\lambda \ell_{\theta}(z)\leq \lambda$ . The third inequality uses the Lipschitz assumption of the loss function.
We also have the following from the triangle inequality of norms, Jensen's inequality, the relationship between the 1-norm and the total variation distance of distributions, and Pinsker's inequality.