SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 13: Supplement to Figure 9 (a). Error rate  $(\%)$  and inconsistency of ensembles in comparison with non-ensemble models  $(+)$ . Food101, ResNet-18. The average and standard deviation of 4 are shown. Ensemble reduces test error and inconsistency.
Training method	Test error(%)	inconsistency
+	Non-ensemble	17.09±0.20	0.30±0.001
*	Ensemble of standard models	14.99±0.05	0.14±0.001
•	Ensembles of Consist. models	14.07±0.08	0.10±0.001
C.3.1 Ensemble experiments (Figure 9 (a))
The models used in the ensemble experiments were ResNet-18 trained for 200 epochs with label smoothing 0.1 with the basic setting of Case#10 in Table 10 otherwise. Table 13 shows the standard deviation of the values presented in Figure 9 (a). The ensembles also served as the teachers in the distillation experiments.
C.3.2 Distillation experiments (Figure 9 (b))
The student models were ResNet-18 trained for 200 epochs with the basic setting of Case#10 of Table 10 otherwise. The teachers for Figure 9(b) (left) were ResNet-18 ensemble models trained as described in C.3.1. The teachers for Figure 9(b) (middle) were ResNet-50 ensemble models trained similarly. The teachers for Figure 9(b) (right) were EfficientNet-B0 models obtained by fine-tuning the public ImageNet-trained model (footnote 2); fine-tuning was done with encouragement of consistency and flatness (with  $\rho = 0.1$ ) with batch size 512, weight decay 1e-5, the initial learning rate 0.1 with cosign scheduling, gradient clipping 20, and 20K updates. Table 14 shows the standard deviations of the values plotted in Figure 9(b).
Table 14: Supplement to Figure 9 (b). Test error  $(\%)$  and inconsistency of distilled-models in comparison with standard models  $(+)$ . The average and standard deviation of 4 are shown.
Test error (%)	inconsistency
left	middle	right	left	middle	right
+	17.09±0.20	0.30±0.001
△	15.74±0.09	14.95±0.17	14.61±0.14	0.19±0.001	0.21±0.001	0.22±0.002
▲	14.99±0.07	14.28±0.11	13.89±0.08	0.14±0.001	0.15±0.001	0.14±0.002
●	14.41±0.05	13.63±0.09	13.06±0.06	0.10±0.001	0.12±0.001	0.08±0.001
Algorithm 2: Our semi-supervised variant of co-distillation.
Input & Notation: Labeled set  $Z_{n}$ , unlabeled set  $U$ ,  $\beta$  (default: 1),  $\tau$  (default: 0.5), learning rate  $\eta$ , momentum of EMA (default: 0.999). Let  $\phi$  be loss, and let  $f(\theta, x)$  be the model output in the form of probability estimate.
1 Initialize  $\theta^{\mathrm{a}},\theta^{\mathrm{b}},\bar{\theta}^{\mathrm{a}}$  , and  $\bar{\theta}^{\mathrm{b}}$
2 for  $t = 1,\dots ,T$  do
3 Sample labeled mini-batches  $B^{\mathrm{a}}$  and  $B^{\mathrm{b}}$  from  $Z_{n}$ , and sample an unlabeled mini-batch  $B_U$  from  $U$ .
Let  $\psi (\theta ,\bar{\theta},x) = \beta \mathbb{I}(\max_i f(\bar{\theta};x)[i] > \tau)\mathrm{KL}(f(\bar{\theta},x)||f(\theta ,x))$  where  $\mathbb{I}$  is the indicator function.
$$
\theta^ {\mathrm {a}} \leftarrow \theta^ {\mathrm {a}} - \eta_ {t} \nabla_ {\theta^ {\mathrm {a}}} \left[ \frac {1}{| B ^ {\mathrm {a}} |} \sum_ {(x, y) \in B ^ {\mathrm {a}}} \phi (f (\theta^ {\mathrm {a}}, x), y) + \frac {1}{| B _ {U} |} \sum_ {x \in B _ {U}} \psi (\theta^ {\mathrm {a}}, \bar {\theta} ^ {\mathrm {b}}, x) \right]
$$
$$
\theta^ {\mathrm {b}} \leftarrow \theta^ {\mathrm {b}} - \eta_ {t} \nabla_ {\theta^ {\mathrm {b}}} \left[ \frac {1}{| B ^ {\mathrm {b}} |} \sum_ {(x, y) \in B ^ {\mathrm {b}}} \phi (f (\theta^ {\mathrm {b}}, x), y) + \frac {1}{| B _ {U} |} \sum_ {x \in B _ {U}} \psi (\theta^ {\mathrm {b}}, \bar {\theta} ^ {\mathrm {a}}, x) \right]
$$
$\bar{\theta}^{\mathrm{a}}$  and  $\bar{\theta}^{\mathrm{b}}$  keep the EMA of  $\theta^{\mathrm{a}}$  and  $\theta^{\mathrm{b}}$ , with momentum  $\mu$ , respectively.
C.3.3 Semi-supervised experiments reported in Table 4
The unlabeled data experiments reported in Table 4 used our modification of Algorithm 1, taylored for use of unlabeled data, and it is summarized in Algorithm 2. It differs from Algorithm 1 in two ways. First, to compute the inconsistency penalty term, the model output is compared with that of the exponential moving average (EMA) of the other model, reminiscent of Mean Teacher [34]. The output of the EMA model during the training typically has higher confidence (or lower entropy) than the model itself, and so taking the KL divergence against EMA of the other model serves the purpose of sharpening the pseudo labels and reducing the entropy on unlabeled data. Second, we adopted the masked divergence from Unsupervised Data Augmentation (UDA) [37], which masks (i.e., ignores) the unlabeled instances for which the confidence level of the other model is less than threshold. These changes are effective in the semi-supervised setting (but not in the supervised setting) for preventing the models from getting stuck in a high-entropy region.
In the experiments reported in Table 4, we penalized inconsistency between the model outputs of two training instances of Unsupervised Data Augmentation (UDA) [37], using Algorithm 2, by replacing loss  $\phi (\theta ,x)$  with the UDA objective. The UDA objective penalizes discrepancies between the model outputs for two different data representations (a strongly augmented one and a weakly augmented one) on the unlabeled data (the UDA penalty). The inconsistency penalty term of Algorithm 2 also uses unlabeled data, and for this purpose, we used a strongly augmented unlabeled batch sampled independently of those for the UDA penalty. UDA 'sharpens' the model output on the weakly augmented data by scaling the logits, which serves as pseudo labels for unlabeled data, and the degree of sharpening is a tuning parameter. However, we tested UDA without sharpening and obtained better performance on the development data (held-out 5K data points) than reported in [32], and so we decided to use UDA without sharpening for our UDA  $^+$  'Consist.' experiments. We obtained test error  $3.95\%$  on the average of 5 independent runs, which is better than  $4.33\%$  of UDA alone and  $4.25\%$  of FixMatch. Note that each of the 5 runs used a different fold of 4K examples as was done in the previous studies.
Following [32], we used labeled batch size 64, weight decay 5e-4, and updated the weights 500K times with the cosine learning rate schedule decaying from 0.03 to  $0.2 \times 0.03$ . We set the final model to the average of the last  $5\%$  iterates (i.e., the last 25K snapshots of model parameters). We used these same basic settings for all (UDA, FixMatch, and UDA+‘Consist.’). The unlabeled batch size for testing UDA and FixMatch was  $64 \times 7$ , as in [32]. For computing each of the two penalty terms for UDA+‘Consist,’ we set the unlabeled batch size to  $64 \times 4$ , which is approximately one half of that for UDA and FixMatch. We made it one half so that the total number of unlabeled data points used by each model ( $64 \times 4$  for the UDA penalty plus  $64 \times 4$  for the inconsistency penalty) becomes similar to that of UDA or FixMatch. RandAugment with the same modification as described in the FixMatch study was used as strong augmentation, and the standard data augmentation (shift and flip) was used as weak augmentation. The threshold for masking was set to 0.5 for both the UDA penalty and the inconsistency penalty and the weights of the both penalties were set to 1. Note that we fixed the
weights of penalties to 1, and only tuned the threshold for masking by selecting from  $\{0,0.5,0.7\}$  on the development data (5K examples).
C.3.4 Fine-tuning experiments in Table 5
The EfficientNet-B4 (EN-B4) fine-tuning experiments reported in Table 5 were done with weight decay 1e-5 (following [10]), batch size 256 and number of updates 20K following the previous studies. We set the learning rate to 0.1 and performed gradient clipping with size 20 to deal with a sudden surge of the gradient size. The diameter of the local region  $\rho$  for SAM was chosen from  $\{0.05, 0.1, 0.2\}$  based on the performance of SAM on the development data, and the same chosen value 0.2 was used for both SAM and SAM+‘Consist.’ Following the EN-B7 experiments of [10], the value  $m$  for  $m$ -sharpness was set to 16. Since SAM+‘Consist.’ is approximately twice as expensive as SAM as a result of training two models, we also tested SAM with 40K updates  $(20\mathrm{K} \times 2)$  and found that it did not improve performance. We note that our baseline EN-B4 SAM performance is better than the EN-B7 SAM performance of [10]. This is due to the difference in the basic setting. In [10], EN-B7 was fine-tuned with a larger batch size 1024 with a smaller learning rate 0.016 while the batch normalization statistics was fixed to the pre-trained statistics. Our basic setting allowed a model to go farther away from the initial pre-trained model. Also note that we experimented with smaller EN-B4 instead of EN-B7 due to resource constraints.
D Additional information
D.1 Additional correlation analyses using the framework of Jiang et al. (2020)
This section reports on the additional correlation analysis using the rigorous framework of Jiang et al. (2020) [18] and shows that the results are consistent with the results in the main paper and the previous work. The analysis uses correlation metrics proposed by [18], which seek to mitigate the effect of what [18] calls spurious correlations that do not reflect causal relationships with generalization. For completeness, we briefly describe below these metrics, and [18] should be referred to for more details and justification.
Notation In this section, we write  $\pi$  for a training procedure (or equivalently, a combination of hyperparameters including the network architecture, the data augmentation method, and so forth). Let  $g(\pi)$  be the generalization gap of  $\pi$ , and let  $\mu(\pi)$  be the quantity of interest such as inconsistency or disagreement.
Ranking-based Let  $\mathcal{T}$  be the set of the corresponding pairs of generalization gap and the quantity of interest to be considered:  $\mathcal{T} \coloneqq \cup_{\pi} \{(\mu(\pi), g(\pi))\}$ . Then the standard Kendall's ranking coefficient  $\tau$  of  $\mathcal{T}$  can be expressed as:
$$
\tau (\mathcal{T}):= \frac{1}{|\mathcal{T}|(|\mathcal{T}| - 1)}\sum_{(\mu_{1},g_{1})\in \mathcal{T}}\sum_{(\mu_{2},g_{2})\in \mathcal{T}\setminus (\mu_{1},g_{1})}\operatorname {sign}(\mu_{1} - \mu_{2})\operatorname {sign}(g_{1} - g_{2})
$$
[18] defines granulated Kendall's coefficient  $\Psi$  as:
$$
\Psi := \frac {1}{n} \sum_ {i = 1} ^ {n} \psi_ {i}, \psi_ {i} := \frac {1}{m _ {i}} \sum_ {\pi_ {1} \in \Pi_ {1}} \dots \sum_ {\pi_ {i - 1} \in \Pi_ {i - 1}} \sum_ {\pi_ {i + 1} \in \Pi_ {i + 1}} \dots \sum_ {\pi_ {n} \in \Pi_ {n}} \tau \left(\cup_ {\pi_ {i} \in \Pi_ {i}} (\mu (\pi), g (\pi))\right) \tag {6}
$$
where  $\pi_i$  is the  $i$ -th hyperparameter so that  $\pi = (\pi_1, \pi_2, \dots, \pi_n)$ ,  $\Pi_i$  is the set of all possible values for the  $i$ -th hyperparameter, and  $m_i := |\Pi_1 \times \dots \times \Pi_{i-1} \times \Pi_{i+1} \times \dots \times \Pi_n|$ .
Mutual information-based Define  $V_{g}(\pi, \pi') \coloneqq \mathrm{sign}(g(\pi) - g(\pi'))$ , and similarly define  $V_{\mu}(\pi, \pi') \coloneqq \mathrm{sign}(\mu(\pi) - \mu(\pi'))$ . Let  $U_{S}$  be a random variable representing the values of the hyperparameter types in  $S$  (e.g.,  $S = \{\text{learning rate, batch size}\}$ ). Then  $I(V_{\mu}, V_{g}|U_{S})$ , the conditional mutual information between  $\mu$  and  $g$  given the set  $S$  of hyperparameter types, and the
Table 15: Correlation scores of inconsistency and disagreement. For the training procedures with low final randomness (as in Figure 5), model-wise quantities (one model per procedure) were analyzed. (a)-(c) differ in the restriction on training loss; (b) and (c) exclude the models with high training loss while (a) does not. The average and standard deviation of 4 independent runs (that use 4 distinct subsamples of training sets as training data and distinct random seeds) are shown. Correlation scores: Two types of mutual information-based scores (' $\mathcal{K}$ ' as in (7) and '|S|=0':  $\frac{I(V_{\mu}, V_{g}|U_{S})}{H(V_{g}|U_{S})}$  with  $|S|=0$ ) and two types of Kendall's rank-correlation coefficient-based scores ( $\Psi$  as in (6) and overall  $\tau$ ). A larger number indicates a higher correlation. The highest numbers are highlighted. Tested quantities: 'Inconsistent': Inconsistency, 'Disagree': Disagreement, 'Random' (baseline): random numbers drawn from the normal distribution, 'Canonical' (baseline): a slight extension of the canonical ordering in [18]; it heuristically determines the order of two procedures by preferring smaller batch size, larger weight decay, larger learning rate, and presence of data augmentation (which are considered to be associated with better generalization) by adding one point for each and breaking ties randomly. Target quantities: Generalization gap (test loss minus training loss) as defined in the main paper, test error, and test error minus training error. Observation: Inconsistency correlates well to generalization gap, and disagreement correlates well to test error. With more aggressive exclusion of the models with high training loss (going from (a) to (c)), the correlation of disagreement to generalization gap improves and approaches that of inconsistency.
(a) No restriction on training loss
CIFAR-10	CIFAR-100	ImageNet
K	|S|=0	Ψ	τ	K	|S|=0	Ψ	τ	K	|S|=0	Ψ	τ
Correlation to generalization gap ('test loss - training loss')
Inconsistent.	0.23±.01	0.38±.01	0.60±.05	0.69±.01	0.51±.01	0.55±.01	0.68±.06	0.81±.01	0.75±.01	0.75±.01	0.73±.17	0.92±.00
Disagree.	0.14±.01	0.26±.01	0.54±.05	0.58±.01	0.11±.01	0.11±.01	0.48±.06	0.39±.01	0.11±.01	0.16±.01	0.53±.07	0.47±.01
Random	0.00±.00	0.00±.00	-0.02±.04	0.01±.03	0.00±.00	0.00±.00	-0.01±.07	0.02±.02	0.00±.00	0.01±.00	-0.00±.11	-0.04±.08
Canonical	0.01±.00	0.10±.00	0.32±.03	0.37±.01	0.00±.00	0.09±.00	0.14±.05	0.36±.01	0.00±.00	0.07±.00	0.36±.05	0.32±.00
Correlation to test error
Inconsistent.	0.08±.01	0.16±.01	0.51±.04	0.46±.01	0.00±.00	0.01±.00	0.33±.01	0.14±.01	0.03±.00	0.03±.00	0.20±.03	-0.20±.00
Disagree.	0.31±.01	0.37±.00	0.58±.05	0.68±.00	0.19±.01	0.31±.02	0.57±.05	0.63±.01	0.04±.00	0.05±.00	0.30±.07	0.25±.01
Random	0.00±.00	0.00±.00	0.02±.03	0.00±.01	0.00±.00	0.00±.00	-0.05±.07	-0.00±.04	0.00±.00	0.00±.00	-0.07±.08	0.01±.04
Canonical	0.01±.00	0.03±.00	0.15±.02	0.20±.01	0.00±.00	0.19±.01	0.31±.15	0.49±.01	0.00±.00	0.03±.00	0.08±.03	0.18±.00
Correlation to 'test error - training error'
Inconsistent.	0.11±.00	0.25±.01	0.53±.04	0.57±.01	0.41±.01	0.48±.01	0.60±.02	0.77±.01	0.73±.01	0.73±.00	0.91±.03	0.91±.00
Disagree.	0.16±.01	0.28±.01	0.54±.07	0.60±.01	0.12±.01	0.12±.01	0.45±.05	0.40±.01	0.12±.01	0.16±.01	0.52±.06	0.46±.01
Random	0.00±.00	0.00±.00	0.03±.04	0.00±.02	0.00±.00	0.00±.00	0.01±.07	-0.01±.03	0.00±.00	0.00±.00	0.06±.17	-0.03±.07
Canonical	0.01±.00	0.07±.00	0.31±.06	0.31±.01	0.00±.00	0.09±.00	0.21±.09	0.35±.01	0.00±.00	0.07±.00	0.24±.02	0.32±.00
(b) Excluding the models with very high training loss, as described in Appendix C.1 and Table 6.
CIFAR-10	CIFAR-100	ImageNet
K	|S|=0	Ψ	τ	K	|S|=0	Ψ	τ	K	|S|=0	Ψ	τ
Correlation to generalization gap ('test loss - training loss')
Inconsistent.	0.37±.01	0.57±.01	0.64±.05	0.83±.01	0.47±.01	0.52±.01	0.68±.07	0.79±.01	0.74±.01	0.75±.01	0.74±.16	0.92±.00
Disagree.	0.34±.02	0.53±.02	0.61±.05	0.80±.02	0.26±.03	0.33±.01	0.60±.05	0.65±.01	0.24±.02	0.33±.02	0.61±.07	0.65±.01
Random	0.00±.00	0.00±.00	-0.01±.05	-0.00±.03	0.00±.00	0.00±.00	0.06±.06	-0.01±.03	0.00±.00	0.00±.00	-0.10±.07	-0.05±.07
Canonical	0.02±.00	0.15±.00	0.28±.04	0.44±.01	0.00±.00	0.21±.01	0.23±.06	0.52±.01	0.00±.00	0.10±.00	0.46±.16	0.38±.00
Correlation to test error
Inconsistent.	0.10±.00	0.18±.01	0.51±.04	0.49±.01	0.01±.00	0.10±.01	0.42±.00	0.36±.02	0.01±.00	0.01±.00	0.21±.02	-0.10±.00
Disagree.	0.27±.00	0.33±.01	0.57±.06	0.65±.00	0.15±.01	0.27±.01	0.56±.05	0.59±.01	0.02±.00	0.02±.00	0.25±.07	0.16±.01
Random	0.00±.00	0.00±.00	0.03±.03	0.01±.02	0.00±.00	0.00±.00	0.04±.07	0.02±.02	0.00±.00	0.00±.00	0.04±.10	0.01±.06
Canonical	0.01±.00	0.03±.00	0.14±.02	0.21±.01	0.00±.00	0.20±.01	0.25±.14	0.51±.01	0.00±.00	0.02±.00	0.41±.03	0.19±.00
Correlation to 'test error - training error'
Inconsistent.	0.19±.01	0.37±.01	0.57±.04	0.69±.01	0.35±.01	0.44±.01	0.60±.02	0.74±.01	0.72±.01	0.73±.00	0.93±.03	0.91±.00
Disagree.	0.35±.02	0.52±.02	0.60±.06	0.80±.01	0.30±.03	0.35±.01	0.57±.05	0.67±.01	0.25±.02	0.32±.01	0.61±.06	0.64±.01
Random	0.00±.00	0.00±.00	0.01±.06	-0.02±.01	0.00±.00	0.00±.00	-0.01±.08	0.01±.03	0.00±.00	0.00±.00	-0.02±.09	-0.06±.06
Canonical	0.01±.00	0.10±.00	0.29±.03	0.36±.01	0.00±.00	0.20±.01	0.23±.06	0.50±.01	0.00±.00	0.11±.00	0.65±.03	0.38±.00
(c) Excluding the models with high training loss more aggressively with smaller cut-off values (one half of (b))
CIFAR-10	CIFAR-100	ImageNet
K	|S|=0	Ψ	τ	K	|S|=0	Ψ	τ	K	|S|=0	Ψ	τ
Correlation to generalization gap ('test loss - training loss')
Inconsistent.	0.36±.01	0.57±.01	0.65±.05	0.82±.01	0.44±.01	0.50±.01	0.69±.06	0.78±.01	0.77±.01	0.79±.01	0.74±.16	0.93±.00
Disagree.	0.39±.01	0.57±.01	0.62±.05	0.82±.01	0.31±.02	0.43±.01	0.65±.05	0.73±.01	0.43±.01	0.53±.01	0.69±.06	0.80±.01
Random	0.00±.00	0.00±.00	0.02±.03	0.00±.02	0.00±.00	0.00±.00	0.03±.11	0.04±.02	0.00±.00	0.00±.00	-0.05±.12	-0.02±.03
Canonical	0.02±.00	0.14±.00	0.36±.04	0.44±.01	0.00±.00	0.27±.01	0.23±.06	0.60±.01	0.00±.00	0.13±.00	0.44±.09	0.42±.00
Correlation to test error
Inconsistent.	0.12±.01	0.21±.01	0.53±.04	0.53±.01	0.04±.00	0.19±.01	0.48±.00	0.50±.02	0.01±.00	0.01±.00	0.31±.02	0.13±.00
Disagree.	0.27±.00	0.35±.01	0.58±.06	0.66±.01	0.19±.02	0.34±.01	0.58±.06	0.65±.01	0.05±.00	0.05±.00	0.27±.08	0.25±.00
Random	0.00±.00	0.00±.00	-0.01±.03	0.01±.03	0.00±.00	0.00±.00	0.01±.11	0.01±.04	0.00±.00	0.00±.00	-0.01±.15	-0.01±.01
Canonical	0.01±.00	0.03±.00	0.25±.03	0.22±.01	0.00±.00	0.24±.01	0.25±.03	0.56±.01	0.00±.00	0.06±.00	0.20±.10	0.29±.01
Correlation to 'test error - training error'
Inconsistent.	0.18±.01	0.37±.01	0.58±.04	0.68±.01	0.30±.01	0.42±.01	0.61±.02	0.72±.01	0.72±.01	0.76±.00	0.93±.03	0.92±.00
Disagree.	0.37±.01	0.54±.01	0.61±.06	0.80±.01	0.37±.03	0.46±.01	0.63±.05	0.75±.01	0.44±.01	0.52±.01	0.69±.06	0.79±.01
Random	0.00±.00	0.00±.00	0.01±.05	0.02±.02	0.00±.00	0.00±.00	0.00±.04	0.02±.03	0.00±.00	0.00±.00	-0.05±.18	-0.04±.04
Canonical	0.01±.00	0.09±.00	0.28±.04	0.35±.01	0.00±.00	0.26±.01	0.24±.02	0.58±.01	0.00±.00	0.14±.00	0.42±.09	0.44±.00
conditional entropy  $H(V_{g}|U_{S})$  can be expressed as follows.
$$
I (V _ {\mu}, V _ {g} | U _ {S}) = \sum_ {U _ {S}} p (U _ {S}) \sum_ {V _ {\mu} \in \{\pm 1 \}} \sum_ {V _ {g} \in \{\pm 1 \}} p (V _ {\mu}, V _ {g} | U _ {S}) \log \left(\frac {p (V _ {\mu} , V _ {g} | U _ {S})}{p (V _ {\mu} | U _ {S}) p (V _ {g} | U _ {S})}\right)
$$
$$
H \left(V _ {g} \mid U _ {S}\right) = - \sum_ {U _ {S}} p \left(U _ {S}\right) \sum_ {V _ {g} \in \{\pm 1 \}} p \left(V _ {g} \mid U _ {S}\right) \log \left(p \left(V _ {g} \mid U _ {S}\right)\right)
$$
Dividing  $I(V_{\mu}, V_{g}|U_{S})$  by  $H(V_{g}|U_{S})$  for normalization and restricting the size of  $S$  for enabling computation, [18] defines the metric  $\mathcal{K}(\mu)$  as:
$$
\mathcal {K} (\mu) := \min  _ {U _ {S} \text {s . t .} | S | \leq 2} \frac {I \left(V _ {\mu} , V _ {g} \mid U _ {S}\right)}{H \left(V _ {g} \mid U _ {S}\right)}. \tag {7}
$$
Results (Tables 15-16) In Table 15, we show the average and standard deviation of the correlation scores of model-wise quantities (for one model per training procedure), following [18]. The model-wise inconsistency for model  $\theta$  trained on  $Z_{n}$  with procedure  $P$  is  $\mathbb{E}_{\Theta \sim \Theta_{P|Z_n}}\mathbb{E}_X\mathrm{KL}(f(\Theta ,X)||f(\theta ,X))$ , and  $Z_{n}$  was fixed here; similarly, model-wise disagreement is  $\mathbb{E}_{\Theta \sim \Theta_{P|Z_n}}\mathbb{E}_X\mathbb{I}[c(\Theta ,X)\neq c(\theta ,X)]$  where  $c(\theta ,x)$  is the classification decision of model  $\theta$  on data point  $x$ . The average and standard deviation were computed over 4 independent runs that used 4 distinct subsamples of training sets as training data  $Z_{n}$  and distinct random seeds for model parameter initialization, data mini-batching, and so forth.
Table 15 compares inconsistency and disagreement in terms of their correlations with the generalization gap (test loss minus training loss as defined in the main paper), test error, and test error minus training error. The training procedures analyzed here are the procedures that achieve low final randomness by either a vanishing learning rate or iterate averaging as in Figure 5. Tables (a), (b), and (c) differ in the models included in the analysis. As noted in Appendix C.1, since near-random models in the initial phase of training are not of practical interest, procedures with very high training loss were excluded from the analysis in the main paper. Similarly, Table 15-(b) excludes the models with high training loss using the same cut-off values as used in the main paper, and (c) does this with smaller (and therefore more aggressive) cut-off values (one half of (b)), and (a) does not exclude any model. Consequently, the average training loss is the highest in (a) and lowest in (c).
Let us first review Table 15-(a). The results show that inconsistency correlates well to generalization gap (test loss minus training loss) as suggested by our theorem, and disagreement correlates well to test error as suggested by the theorem of the original disagreement study [17]. Regarding 'test error minus training error' (last 4 rows): on CIFAR-10, training error is relatively small and so it approaches test error, which explains why disagreement correlates well to it; on the other datasets, 'test error minus training error' is more related to 'test loss minus training loss', which explains why inconsistency correlates well to it. The standard deviations are relatively small, and so the results are solid. (The standard deviation of  $\Psi$  tends to be higher than the others for all quantities including the baseline 'Random', and this is due to the combination of the macro averaging-like nature of  $\Psi$  and the smallness of  $|\Pi_i|$  for some  $i$ 's, independent of the nature of inconsistency or disagreement.)
The overall trend of Table 15-(b) and (c) is similar to (a). That is, inconsistency correlates well to generalization gap (test loss minus training loss) while disagreement correlates well to test error, consistent with the results in the main paper and the original disagreement study. Comparing (a), (b), and (c), we also note that as we exclude the models with high training loss more aggressively (i.e., going from (a) to (c)), the correlation of disagreement to generalization gap (relative to that of inconsistency) improves and approaches that of inconsistency. For example, on CIFAR-100, the ratio of  $\mathcal{K}$  for (disagreement, generalization gap) with respect to  $\mathcal{K}$  for (inconsistency, generalization gap) improves from (a)  $0.11 / 0.51 = 22\%$  to (b)  $0.26 / 0.47 = 55\%$  to (c)  $0.31 / 0.44 = 70\%$ . With these models, empirically, high training loss roughly corresponds to the low confidence-level on unseen data, and so this observation is consistent with the theoretical insight that when the confidence-level on unseen data is high, disagreement should correlate to generalization gap as well as inconsistency, which is discussed in more detail in Appendix D.3.
Table 16 shows that the correlation of the estimate of  $\mathcal{C}_P$  (defined in Section 2) to the generalization gap is generally as good as the estimate of  $\mathcal{D}_P$  (defined in Theorem 2.1), which is consistent with the results in the main paper.
Table 16: Correlation analyses of the estimates of  $\mathcal{C}_P$  and  $\mathcal{D}_P$  (as defined in Section 2) using the rank-based and mutual information-based metrics from [18]. Training procedures with low final randomness of Figure 5. Correlation scores: see the caption of Table 15. Target quantities: Generalization gap (test loss minus training loss) analyzed in Theorem 2.1. Observation: The correlation of  $\mathcal{C}_P$  to generalization gap is generally as good as  $\mathcal{D}_P$ , which is consistent with the results in the main paper.