Inconsistency, Instability, and Generalization Gap of Deep Neural Network Training
Authors: Rie Johnson
Tong Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep neural network training
Generalization gap
Empirical study

Abstract: As deep neural networks are highly expressive, it is important to find solutions with small generalization gap (the difference between the performance on the training data and unseen data).  Focusing on the stochastic nature of training, we first present a theoretical analysis in which the bound of generalization gap depends on what we call inconsistency and instability of model outputs, which can be estimated on unlabeled data.  Our empirical study based on this analysis shows that instability and inconsistency are strongly predictive of generalization gap in various settings.  In particular, our finding indicates that inconsistency is a more reliable indicator of generalization gap than the sharpness of the loss landscape.  Furthermore, we show that algorithmic reduction of inconsistency leads to superior performance.  The results also provide a theoretical basis for existing methods such as co-distillation and ensemble.

Inconsistency, Instability, and Generalization Gap of Deep Neural Network Training
Rie Johnson
Tong Zhang* HKUST
RJ Research Consulting
New York, USA ri riejohnson@gmail.c
Hong Kong
tozzhnggtongzhangg-ll. org
Abstract
As deep neural networks are highly expressive, it is important to find solutions with small generalization gap (the difference between the performance the training data and unseen data). Focusing on the stochastic nature training, we first present theoretical analysis in which the bound of generalization gap depends on what we call inconsistency and instability of model outputs, which can be estimated on unlabeled data. Our empirical study based on this analysis shows that instability and inconsistency are strongly predictive of generalization gap in various settings. In particular, our finding indicates that inconsistency a more reliable indicator of generalization gap than the sharpness of the loss landscape. Furthermore, we show that algorithmic reduction of inconsistency leads to superior performance. The results also provide theoretical basis for existing methods such as co-distillation and ensemble.
Introduction
As deep neural networks highly expressive, the generalization gap (the difference between the performance on the training data and unseen data) can be serious issue. There has been intensive effort improve generalization, concerning, for example, network architectures [T), the training objective [10]. strong data augmentation and mixing [1]] In particular, there has been extensive effort understand the connection between generalization and the sharpness of the loss landscape
[O]. [8] conducted large-scale experiments with a number of metrics and found that the sharpness- based metrics predicted the generalization gap best. [TO] has shown that the sharpness measured by the maximum loss difference around the model (m-sharpness) correlates well to the generalization gap, which justifies their proposed method sharpness-aware minimization (SAM), designed to reduce the m-sharpness.
surrounding the model 14 20 [6
[8]
This paper studies generalization gap from different perspective. Noting that the standard procedure for neural network training is stochastic so that a different instance leads to a different model, we first present a theoretical analysis in which the bound of generalization gap depends on inconsistency and instability of model outputs, conceptually described as follows. Let P be a stochastic training procedure, e.g., minimization of the cross-entropy loss by stochastic gradient descent (SGD) starting from random initialization with a certain combination of hyperparameters. It can be regarded as random function that maps set labeled training data as input to model parameter as its output. We procedure P has high inconsistency of model outputs if the predictions made by the models trained with P using the same training data are very different from one another in expectation over the underlying (but unknown) data distribution. We also say P has high instability of model outputs different sampling of training data changes the expected predictions a lot over the underlying data distribution. Although both quantities are discrepancies of model outputs, the sources of the
This work was done when the second author was jointly with Google Research.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

discrepancies are different. The term stability comes from the related concept in the literature 4 31 Inconsistency related to the disagreement metric studied [27 I7 22], but there are crucial differences between them, as shown later. Both inconsistency and instability of model outputs can be estimated on unlabeled data.
Our high-level goal is to find the essential property of models that generalize well; such an insight would be useful for improving training algorithms. With this goal, we empirically studied the connection of inconsistency and instability with the generalization gap. As our bound also depends on property of model parameter distributions (for which we have theoretical insight but which we camnot easily estimate), we first experimented to find the condition under which inconsistency and instability of model outputs are highly correlated to generalization gap. The found condition low randomness in the final state is consistent with the theory, and it covers practically useful settings. We also found that when this condition met, inconsistency alone is almost as predictive as inconsistency and instability combined. This is practical advantage since estimation of instability requires multiple training sets and estimation of inconsistency does not. We thus focused on inconsistency, which enabled the use of full-size training data, and studied inconsistency in comparison with sharpness in the context of algorithmic reduction of these two quantities. We observed that while both sharpness reduction and inconsistency reduction lead to better generalization, there are a number of cases where inconsistency and generalization gap are reduced and yet sharpness remains relatively high. We view it indication that inconsistency has stronger (and perhaps more essential) connection with generalization gap than sharpness.
Our contributions are as follows.
We develop a theory that relates generalization gap to instability and inconsistency of model outputs, which can be estimated on unlabeled data.
Empirically, we show that instability and inconsistency are strongly predictive of generaliza- tion gap in various settings.
We show that algorithmic encouragement of consistency reduces inconsistency and improves performance, which can lead to further improvement of the state of the art performance.
Our results provide theoretical basis for existing methods such co-distillation and ensemble.
2 Theory
The theorem below quantifies generalization gap by three quantities: inconsistency of model outputs, instability of model outputs, and information-theoree instability of model parameter distributions.
Notation Let f0, x) be the output of model @ on data point in the form of probability estimates (e.g., obtained by applying softmax). We use the upright bold font for probability distributions. Let Z = =xx(X (X.Y) be random variable representing labeled data point (data point X and label Y) with given unknown distribution Z. Let Zn, = {(,, Y2) i 1, iid training data of size n drawn from Z. Let θ p z., be the distribution of model parameters resulting from applying a (typically stochastic) training procedure P to training set Zn
1,...,,?) } be a random variable representing
Inconsistency, instability, and information -foommttion-thhoretie instability Inconsistency of model outputs Cp ('C' for consistency) of training procedure P represents the discrepancy of outputs among the models trained on the same training data:
Cp = Ez,
(x
(Inconsistency of model outputs)
The source of inconsistency could be the random initialization, sampling of bani-bbtchess, randomized data augmentation, and so forth.
To define instability of model outputs, let friz,, (x) be the expected outputs (for data point x)) of the models trained by procedure P on training data Z.: FPi2,, (x) = Ee-er z., (O,,)) Instability of model outputs Sp ('S' for stability) of procedure P represents how much the expected prediction friza (x) would change with change of training data:
Sp = Ez.. Ex EX (X) (X)Il (X))
(Instability of model outputs)

---

Finally, the instability of model parameter distributions Zp is the mutual information between the random variable pp, which represents the model parameters produced by procedure P, and the random variable Zn. which represents training data. Ip quantifies the dependency of the model parameters on the training data, which is also called information theoretic (in )stability in [38:
Ip I(Op; Z,.) Ez, (L PIZ. Ezz ② PIZ;.) (Instability of model parameter distributions)
The second equality follows from the well-known relation of the mutual information to the KL divergence. The rightmost expression might be more intuitive, which essentially represents how much the model parameter distribution would change with change of training data.
Note that inconsistency and instability of model outputs can be estimated on unlabeled data. Ip cannot be easily estimated as it involves distributions over the entire model parameter space; however, have theoretical insight from its definition. (More precisely, it is possible to estimate Ip by sampling in theory, but practically, it is not quite possible to make a reasonably good estimate of Ip for deep neural networks with reasonably small amount of computation.)
Theorem 2.1. Using the definitions above, we consider Lipschitz loss function o(f y) E [0, 1] that (.(',,)) y)1 S Ilʃl- - [']11, where I and f' are probability estimates and 7/2 > satisfies (0(s,y) - OI'',
the Lipschitz constant. Let (A)= =²=A=1, which is an increasing function. Let Dp = Cp + Sp. Let (0) be test loss: (0) = (z==(x)))(f[L) X),Y) Y). Let Φ(0, Za) be empirical loss on 7. = {(X,, Vi)li= =1 =1. nn: $(0, 7n) E ((( (o),, Then for given training
procedure P, we have
Ip An
Ex, Ee-Oriz (z) ((,,
inf X=0
7 AWAAADD
The theorem indicates that the upper bound of generalization gap depends on the three quantities, instability of two types and inconsistency. described above. As a sanity check, note that right after random initialization, generalization gap the left-hand side is zero, and Ip and Sp in the right-hand side are also zero, which makes the right-hand side zero X O, Also note that this analysis is meant for stochastic training procedures. If P is a deterministic function of Z, and not constant, the mutual information Zp would become large while Dp O, which would make the bound loose.
Intuitively, when training procedure P is more random, model parameter distributions are likely to be flatter (less concentrated) and less dependent on the sampling of training data Zn, and so Ip is lower. However, high randomness would raise inconsistency of the model outputs Cp, which would raise Dr. Thus, the theorem indicates that there should be a trade-off with respect to the randomness of the training procedure.
The style of this general bound follows from the recent information theoretical generalization analyses of stochastic machine learning algorithms [38] 30 28 that employ Ip as a complexity measure, and such results generally hold in expectation. However, unlike the previous studies, we obtained a more refined Bernstein-style generalization result. Moreover, we explicitly incorporate inconsistency and instability of model outputs into our bound and show that smaller inconsistency and smaller instability lead to a smaller generalization bound on the right-hame hand side. In particular, if Ip relatively small so that we have Ip S m then setting À = /Ip (mpp and using (A) < 1 for S 1, we obtain a simpler bound
DrIr n
Ez, D2..E-OPiiz (02 (0) S 27V
that with models trained to zero training error, disagreement (in terms of classification decision) of identically trained mod- els is approximately equal to test error. When measured for the models that share training data, disagreement is closely related to inconsistency Cp above, and it can be expressed as Ez, E2.e" I c(O,X) # ((e)))) ]. where c(0,xz) x:) is classification decision c(f, x)) arg max; f(0, [i] and the indicator function I[u] = 1 if u is true and otherwise. In spite of the similarity, in fact, the behavior of disagreement and inconsistency Cp can be quite different. Disagreement is equivalent to sharpening the prediction to a one-hot vector and then taking 1-norm of the difference: [[ c(0, x) * c(0', x) = |onehot( (0, æ)) onehot(f(! x))]]-- This ultimate
Relation to disagreement It was empirically shown in 27 17
3

---

sharpening makes inconsistency and disagreement very different when the confidence-level of f low. This means that disagreement and inconsistency would behave more differently on more complex data (such as ImageNet) on which it is harder to train models to a state of high confidence. In Figure [0] (Appendix) we show an example of how different the behavior of disagreement and inconsistency can be, with ResNet-50 trained on 10% of ImageNet.
3 Empirical study
Our empirical study consists of three parts. As the bound depends not only Dp(= Cp + Sp) but also Ip, we first seek the condition under which Dp is predictive of generalization gap (Section B.I. Noting that, when the found condition is met, inconsistency Cp alone is as predictive of generalization gap as Dp, Section 3.2 focuses on inconsistency and shows that inconsistency is predictive of generalization gap in a variety of realistic settings that use full-size training sets. Finally, Section 3.3 reports on the practical benefit of encouraging low inconsistency in algorithmic design. The details for reproduction are provided in Appendix C
Remark: Predictiveness Dp relative Note that we are interested in how well the change in Dp matches the change in generalization gap; thus, evaluation of the relation between Dp and generalization gap always involves multiple training procedures to observe the changes/differences Given set S of training procedures, informally say Dp is predictive of generalization gap for S if the relative mallinsss/argeeness D of the procedures in essentially coincides with the relative smallness/laggeness of their generalization gap.
3.1 On the predictiveness of p Inconsistency Cp+ Instability Sp
Inconsistency Cp and instability Sp were estimated as follows. For each training procedure P (identified by combination of hyperparameters such the learning rate and training length), trained J models on each of K disjoint training sets. That is, K x J models were trained with each procedure P. The expectation values involved in Cp and Sp were estimated by taking the average; in particular, the expectation over data distribution Z was estimated on the held-out unlabeled data disjoint from training data or test data. Disagreement was estimated similarly. (K, J) was set to (4,8) for CIFAR- 10ARR-101111 100 and (4.4) for ImageNet, and the size of each training set was set to 4K for CIFAR- 10/100 and 120K (10%) for ImageNet.
3.1.1 Results
We start with the experiments that varied the learning rate and the length of training fixing anything else to see whether the change of Dp is predictive of the change of generalization gap caused by the change of these two hyperparameters. To avoid the complex effects of learning rate decay, we trained models with SGD with a constant learning rate (constant SGD in short).
Although constant SGD could perform poorly by itself, constant SGD with iterate averaging is known to competitive 29 16] Figure shows Dp (I-axis) and generalization gap (y-axis) of the training procedures that performed constant SGD with iterate averaging. Iterate averaging was performed by taking the exponential moving average with momentum 0.999. Each point represents procedure (identified by the combination of the learning rate and training length), and the procedures with the same learning rate are connected by a line in the increasing order of training length. A positive correlation is observed between Dp and generalization gap on all three datasets. Figure 3 plots generalization gap (left) and Dp (right) on the y-axis and training loss the r-aaxis for the same procedures as in Figure I observe that the up and down of DD is remarkably similar to the up and down of generalization gap. Also note that on CIFAR- 10/100, sometimes generalization gap first goes up and then starts coming down towards the www-training-low end of the lines (increase of training loss in some cases is due to the effects of weight decay). This 'turning around' of generalization gap (from going up to going down) in Figure 3 shows up as the "curling up' of the upper-right end of the lines in Figure] [ It is interesting to see that Dp and generalization gap matching at such detailed level.
Constant SGD with iterate averaging 13

---

Smallest learning rate In Figures 1-4 and 6: Only the learning rate and the length of training varied. Each point represents procedure. The procedures with the same learning rate are connected by line in the increasing order of training length; generally, training loss goes down and generalization gap goes up as the training length goes up, with some exceptions towards the end.
2nd smallest Lt.
3rd smallest Lr.
Largest Lr.
35
25
8
05
64
0.5
08
1.5
35
(a) CIFAR-10
(b) CIFAR-100
(c) ImageNet
(a) CIFAR-10 (b) CIFAR-100
(c) ImageNet
Figure 1: Dp estimates (æ-axis) and generalization gap axis). SGD with a constant learning rate and iterate averaging. Only the learning rate and training length were varied. A positive correlation is observed.
Figure 2: Dp estimates (æ-axis) and generalization gap (y- (y-axis). Constant learning rates. No iterate averaging. Dp is predictive of generalization gap only for the procedures that share the learning rate.
CIFAR-10
CIFAR-100
ImageNet
25
15
0.8
6
41.
696
0.5
02
04
0.3
65
0.5
0
000011
0101
0U903E
Dill
000001
0.DI
519011
113
10
0.1
Trining loss
Training los
Triining Joss
Training los
Training loss
Tiaining los.
(a) Gap
(b) Dr estimate
(c) Gap
(d) Dr estimate
(c) Gaa
(f) Dp estimate
Figure 3: Up and down of generalization gap (y-axis: left) and Dp (y-axis; right) as training proceeds. The z-axis training loss. SGD with constant learning rate with iterate averaging. The analyzed models and the legend are the same as in Fig [
For cach dataset, the left graph looks very similar to the right graph; up and down of Dr as training proceeds is a good indicator of and down of generalization gap.
3.5
Constant SGD without iterate averaging (Fig 2 While Dp is clearly predictive of generalization gap in the setting above, the results in Figure 2 are more complex. This fig- ure shows Dp (r-axis) and generalization gap (y-axis) for the training procedures that performed constant SGD and did not perform iterate averaging. As before, only the learning rate and training length were varied. In Figure 2] we observe that Dp is predictive only for those which share the learning rate. For fixed generalization gap. Dr is larger (instead of being the same) for a larger learning rate. Inspection of inconsistency and instability reveals that larger learning rates raise inconsistency although instability is mostly unaffected (see Fig 4). This is apparently the effect of high aadommmesshhoiiiiissseeereer of the final state. As the learning rate is constant, SGD updates near the end of training bounce around the local minimum, and the random noise in the gradient of the last mini-batch has a substantial influence on the final model parameter. The influence of this noise is amplified by larger learning rates. On the other hand, Ip is likely to be lower with higher randomness since higher randomness should make model parameter distributions flatter (less concentrated) and less dependent on the sampling of training data Z,. This means that in this case (i.e., where the final randomness is high and varies lot across the procedures), the interaction of Ip and Dp in the theoretical bound is complex, and Dp alone could be substantially less predictive than what is indicated by the bound.
1.5
1
Inomsitrrzyy
Instabiitt
(a) Inconsistency
(b) Instability
Figure 4: Inconsistency and instabil- ity (z-axiss and generalization gap (y- axis). Same procedures and legend as in Fig 2 Instability is relatively unaf- fected by the learning rate. CIFAR-100. Similar results on CIFAR- 10 and Ima- geNet (Figure 11 in the Appendix).
For Dp to be predictive, final randomness should be equally low (Fig S (a)- Therefore, we hypothesized and empirically confirmed that for Dp to be predictive, the degrees of final ran- domness/uncertain hateesss/uncrtaatttoooiisssss should be equally low, which can be achieved with either vanishing learning rate or iterate averaging. This means that, fortunately, Dp is mostly predictive for high- performing procedures that matter in practice.
Figure 5 (c) plot Dr (z-axis) and generalization gap (y-axis) for the procedures that satisfy the condition. In particular, those trained on CIFAR- 10 include a wide variety of procedures that differ
5

---

3: Dp = Cp + Sp, y: generalization gap
a:: Inconsistency Cp, generalization gap
13
Cop
637
05 Incosiesiry
Incondeetmny
Inareeidency
(a) CIFAR-10 (b) CIFAR-100
(c) ImageNet
(d) CIFAR-10 (e) CIFAR-100
(f) ImageNet
Figure 5: (a) Dp (z-axis) and generalization gap (y-axis). (d)-(f) Cr (z-axis) and generalization gap (g-axis). Both Dr and Cp are predictive generalization gap for training procedures with iterate averaging vanishing learning rate (so that randomness low), irrespective of differences the setting. In particular. the CIFAR-I0 10 results include the training procedures that differ in network architectures, mini-batch sizes, data augmentation, weight decay parameters, learning rate schedules, learning rates and training lengths.
in network architectures, mini-batch sizes, presence/absence of data augmentation, learning rate schedules (constant or vanishing), weight decay parameters, in addition to learning rates and training lengths. positive correlation between Dp and generalization gap is observed on all three datasets.
Inconsistency Cp vs. Dp (Fi Figure 5 (d)-(f) show that inconsistency Cp is almost as predictive as Dp when the condition of low randomness of the final states is satisfied. This is practical advantage since unlike instability Sp, inconsistency Cp does not require multiple training sets for estimation.
Disagreement (Fig For the same procedures/models as in Figures 1 and we show the relationship between disagreement and test error in Figures instead of generalization gap for the y-axis since the previous finding was 'Disagreement ~ Test error'. The straight lines are y = æ. The relation in Fig and Fig 7 (a) is close to equality, but the relation appears to be more complex in the others. The procedures plotted in Fig 7 are those which satisfy the condition for Dp to be predictive of generalization gap; thus, the results indicate that the condition for disagreement to be predictive of test error apparently different.
[n and respectively. We chose test error
Thst arver Dhl
Tea GbOK (R)
Tox arok R83
The orter
15
15
25
30
22
24
26
50
Disgreement
Disgrremeen
Disagreenet
Disgreener
Desagremment
Disgreeneet
(a) CIFAR-10 (b) CIFAR-100
ImageNet
CIFAR-I0
(b) CIFAR-100
ImageNet
Figure 6: Disagreement (æ-axis) and test error (y- axis). SGD a constant learning rate and iter- ate averaging. Same models and legend as in Fig T "Disagreemennt</Tee error" of the previous studics does not quite hold in (b) and (c).
Figure 7: Disagreement (æ-axis) and test (y-axis). The models are the same in Fig[S] which satisfy the condition for Dp predictive of generalization gap. The results indicate that the condition for disagrecment to be predictive of test error is different.
Note that while the previous empirical study of disagreement 27, 17 focused on the models with zero training error, studied wider range of models. Unlike CIFAR-10/100 (on which zero training error can quickly achieved), a common practice for datasets like ImageNet budgeted training [25] instead of training to zero error. Also note that zero training error does not necessarily lead to the best performance, which, for example, has been observed the ImageNet experiments of this section.
3.2
On the predictiveness of inconsistency Cp: from algorithmic perspective
Based the results above, this section focuses on inconsistency Cr, which enables experiments with larger training data, and studies the behavior inconsistency in comparison with sharpness from an algorithmic perspective.

---

Table 1: Datasets. Mostly full-size natural images and 2 lexts.
Table Networks.
#train
Network
#param
Case
Name
#class
#dcv
#test
provided
used
ResNet-50 12
24M
1
ImageNet
1000
1.28M
1.27M
10K
50K
ViT-S/32
0
23M
2
Food101
3
101
75750
70700
5050
25250
ViT-B/16 g
86M
3
Dogs
21
120
12000
10800
1200
8580
Mixer-B/16 35
59M
4
Cars
26
196
8144
7144
1000
8041
WRN-28-2
[40
1.5M
5
CIFAR-I0
24
10
50000
4000
5000
10000
EN-BO 33
4.2M
6,7
MNLI
392702
382902
9800
9815
roberta-base 26
124.6M
8,9
QNLI
36
2
104743
99243
5500
5463
ResNet- 18 12
11M
10
Training objectives Parallel the fact that SAM seeks flatness of training loss landscape, a meta- algorithm co-distillation (named by [. and closely related to deep mutual learning 442), summarized in Algorithm ! in the Appendix) encourages consistency trains two (or more) models with two (or more) different random sequences while penalizing the inconsistency between the predictions of the two models. This is typically described as 'teaching each other', but take a different view of consistency encouragement. Note that the penalty term merely "ecooragess' low inconsistency. Since inconsistency by definition depends on the unknown data distribution, it cannot be directly minimized by training. The situation is similar minimizing loss on the training data with the hope that loss will be small on unseen data. We analyzed the models that were trained with one of the following training objectives (shown with the abbreviations used below):
model outputs. simultancously
Standard': the standard cross-entropy loss.
Comsisst': the standard loss with encouragement of consistency.
Flatt' the SAM objective, which encourages flatness.
"Consist+Flatt encouraging both consistency and flatness. Coupling two instances of SAM training with the inconsistency penalty term.
Cases experimented with ten combinations (cases) of dataset, network architecture, and training scenario. Within each case, we fixed the basic settings (e.g.. weight decay, learning rate) to the ones known to perform well from the previous studies, and only varied the training objectives so that for each case we had at least four stochastic training procedures distinguished by the four training objectives (for some cases, we had more than four due to testing multiple values for SAM). We obtained four models (trained with four different random sequences) per training procedure. Tables [ show the datasets and network architectures we used, respectively. Note that as a result of adopting high- nigh-performing settings, all the cases satisfy the condition of low final randomness.
3.2.1 Results
As in the previous section, we quantify the generalization gap by the difference of test loss from training loss (note that the loss is the cross-entropy loss; the inconsistency penalty or anything else is not included); the inconsistency values were again measured on the held-out unseen unlabeled data, disjoint from both training data and test data. Two types of sharpness values were measured, 1-sharpness (per-example loss difference in the adversarial direction) of [10] and the magnitude of the loss Hessian (measured by the largest eigenvalue), which represent the sharpness training loss landscape and have been shown to correlate to generalization performance [0]
Inconsistency correlates to generalization gap (Figure 8) Figure g shows inconsistency or sharpness values (æ-axis) and the generalization gap (y-axis). Each point the figures represents model; i.e., in this section, we show model-wise quantities (instead of procedure-wise). simulating the model selection setting. (Model-wise inconsistency for model 0 trained on Zn with procedure P Ee-e PiZ., (L standard deviation In the figures, we distinguish the four training objectives. In Figure (b) and (c), we confirm, by comparing the 'Flat.' models (x) with the baseline standard models (+), that encouragement of flatness (by SAM) indeed reduces sharpness (x-axis) as well as the generalization gap (y-axis). This serves sanity check of our setup.
))). All quantities are standardized so that the mean is zero and
7

---

+ Standard A Consist.
X Flat.
O Consist+Flat
Food101 ViT-B 16. from scratch
MNLI, fine- tuning of RoBERTa-base
Centrataaiion B4p
Aennalittion DRg
x
2
(a) Inconsistency (b) I-sharpness
(c) Hessian
(d) Inconsistency (e) I-sharpness
(f) Hessian
Figure 8: Inconsisteney and sharpness (z-axis) and generalization gap (y-axis). Each point represents model. Note that each graph plots 16-20 points, and some of them are overlapping.
We observe in Figure 8 that inconsistency shows clear positíve correlation with the generalization gap, but the correlation of sharpness with generalization gap is less clear (e.g.. in (b)-(c) the generalization gap of the 'Consist.' models (Á) is much smaller than that of the 'Flat'' models (x), but their sharpness is larger). This is a general trend observed across multiple network architectures (ResNet, Transformers, MIP-Mixer, and so on), multiple datasets (images and texts), and multiple training scenarios (from scratch, fine-tuning, and distillation); the Appendix provides the figures for all 10 cases. Moreover, we found that while both sharpness reduction and inconsistency reduction lead to better generalization, in a number of cases inconsistency and generalization gap are reduced and yet sharpness remains relatively high. We view this phenomenon as indication that inconsistency has stronger (and perhaps more essential) connection with generalization gap than sharpness.
Inconsistency is more predictive of generalization gap than sharpness (Table Motivated by the linearity observed in the figure above, we define a linear predictor of the generalization gap that takes the inconsistency or sharpness value as input. We measure the predictive power of the metrics through the prediction performance of this linear predictor trained with least square minimization. To make the inconsistency and sharpness values comparable, we standardize them and also generalization gap. For each case, we evaluated the metrics by performing the leave-one-out cross validation on the given set of models (i.c., perform least squares using all models but one and evaluate the obtained linear predictor on the left-out model; do this & times for k models and take the average). The results in Table show that inconsistency outperforms sharpness, and the superiority still generally holds when sharpness is given 'unfair' advantage of additional labeled data.
Table 3: Generalization gap prediction error. The leave-one-out cross validation results, which average the generalization gap prediction residuals, are shown. See Table for the networks used for each case. Smaller is better, and a value near 1.0 is very poor. Inconsistency: estimated on unlabeled data, 1-sharpness and Hessian: sharpness of training loss landscape. Inconsistency outperforms the sharpness metrics. Noting that inconsistency had access to additional data, even though unlabeled, we also show in the last 2 rows sharpness of test loss landscape estimated on the development data (held-out labeled data), which gives the sharpness metrics an "unfair' advantage by providing them with additional labels. With this advantage, the predictiveness of sharpness mostly improves; however, still, sistnnsst is generally more predictive. The bold font indicates that the difference is statistically significant 95% confidence level against all others including the last 2 rows. The italic font indicates the best but not statistically significant.
Caselt
2
3
4
5
8
10
Datasel
ImageNet
Food101
C10
Dog
Car
Mnli
Qnli
Food
Training scenario
From scratch
Fine-tuning
Distill.
Inconsistency
0.13
0.10
0.17
0.27
0.15
0.09
0.19
0.15
0.18
0.19
1-sharpness
0.21
0.48
0.75
0.74
0.84
0.34
0.24
0.58
0.98
0.75
Hessian
0.50
1.00
0.77
0.72
0.78
0.70
0,68
0.59
0.87
0.59
Giving the sharpness metrics 'unfair' advantage by providing additional labels:
I 1-sharpness of dev. loss
0.14
0.64
0.47
0.40
0.72
0.22
0.14
0.36
0.37
0.62
Hessian of dev. loss
0.45
0,82
0.58
0.75
0.13
0.27
0.15
0.91
0.81
0.89
3.3 Practical impact: algorithmic consequences
The results above suggest a strong connection of inconsistency of model outputs with generalization, which also suggests the importance of seeking consistency in the algorithmic design. To confirm this
8

---

point, we first experimented with two methods (ensemble and distillation) and found that in both, encouraging low inconsistency in every stage led the best test error. Note that in this section we report test error instead of generalization gap in order to show the practical impact.
117
Tex crne (SS)
16
16
16
Tes erter
Tes ercer F4
15
13
Incossiiency
0.05 0.15 11225 Inconsisency
0.05 015 0.55
0.05
0.15 0.25
Incomsistency
Incumsistency
(a) Ensemble
(b) Distillation. Teacher's knowledge level increases from left to right.
Figure 9: Inconsistency and test error with ensemble and distillation.
(a) Ensemble reduces inconsistency (I-axis) and test error (y-axis). + Non-ensemble (haseline), Ensemble of standard models, Ensemble of the models trained with consistency encouragement. RN18 on Food101. (b) Test error (y-axis) and inconsistency of distilled models (r-axis). Without unlabeled data. No distillation. Standard distillation, Consistency encouragement
student, Consistency encouragement for both the student and teacher. The best performance is obtained when consistency is encouraged for both the teacher and student. Food101. Student: RNI8, Teacher: RNI8 (left), RN50 (middle), and fine-tuning of ImageNNt-trained ENBO (right). In both (a) and (b), the average of 4 models is shown; see Appendix for the standard deviations.
Ensemble (Fig (a)) F shows that, compared with non-ensemble models (+), ensemble models (averaging logits of two models) reduce both inconsistency (z-axxis) and test error (y-axis). It intuitive that averaging the logits would cancel out the dependency on (or particularities of) the individual random sequences inherent in each single model; thus, outputs of ensemble models expected to become similar and so consistent (as empirically observed). Moreover, we presume that reduction of inconsistency is the mechanism that enables ensemble to achieve better generalization. The best test error was achieved when the ensemble was made by two models trained with consistency encouragement (+ in Fig (a)).
Distillation (Fig 9 (b)) We experimented with distillation 13 with encouragement of consistency for the teacher model and/or the student model. F shows the test error (y-axis) and the inconsistency (æ-axis). The standard model (+) serves as the baseline. The knowledge level of teachers increases from left right, and within each graph, when/whether encourage consistency differs from the point to point. The main finding here is that at all the knowledge levels of the teacher, both inconsistency and test
go down the pursuit of consistency becomes more extensive; in each graph, the points for the distilled models form nearly straight line. (The line appears to shift away from the baseline (+) as the amount of external knowledge of the teacher increases. We conjecture this could be due to the change in Ip.) While distillation alone reduces inconsistency and test error, the best results are obtained by encouraging consistency at every stage of training (i.e., training both the teacher and student with consistency encouragement). The results underscore the importance of inconsistency reduction for better performance.
On the pursuit of the state of the art (Table 45 improvement of state-of-the-art art results by adding consistency encouragement. The first example is semi-supervised learning with CIFAR-10. Table 4shows that the performance of state-of-the-ae -upervuee method can be further improved by encouraging consistency between two training instances of this method on the unlabeled data (Algorithm{2) in the Appendix). The second example transfer learning. We fine-tuner public ImageNet-trait trained EfficientNet with more focused dataset Food 101. Tshows that encouraging consistency between two instances of SAM training improved test error. These examples demonstrate the importance of consistency encouragement in the pursuit of the state of the art performance.
We show two examples of obtaining further
4 Limitations and discussion
Theorem 2.1 assumes a bounded loss d(J,y) [0, 1] though the standard loss for classification the cross-entropy loss, which is unbounded. This assumption can be removed by extending the theorem to more complex analysis with other moderate assumptions. We, however, chose to present the simpler and more intuitive analysis with bounded loss. As noted above, this theorem is intended

---

Table 4:
CIFAR-10 10 #train=4K, #u
Table 5: Food101, EfficientNet-BE B4. The average and
WRN28-2. Average and standard deviation 5 runs.
standard deviation models are shown.
Methods
Error (%)
Methods
Error (%)
MixMatch
6.42
EN-B4 B-4 [33
Copied
Previous
UDA 37
4.88
EN-B7 [T]
7.17
from 32
results
FixMatch 32
4.26
EN-B7 SAM 10
7.02
FixMatch
1.2
EN-B4 SAM
6.000
Our results
Our results
UDA (no sharpening)
4.
BN- SAM "Consist.
5..77
UDA + "Consist."
3,95=0.12
The improvement over EN-B7 SAM of [10] is due to the difference in the basic setting; see Appendix.
RandAugment
6
was used for UDA and FixMatch.
for stochastic training procedures. The bound may not be useful for deterministic procedures, and this characteristics is not unique to our analysis but shared by the previous information chhoomation-thcortiic analysis of stochastic machine learning algorithms [38 30 28
We acknowledge that due to resource constraints, there was limitation to the diversity of the training procedures we empirically analyzed. particular, our study of inconsistency and instability in Section 3.1 started with simpler cases of constant learning rate SGD and later included the cases of learning rate decay (): consequently, the types of the training procedures studied Section 3.1 were skewed towards the constant learning rate SGD. The models analyzed in Figure 5 were less diverse with CIFAR-100 and ImageNet than with CIFAR-10 10, as noted above, and again this asymmetry was due to the resource constraints. On the other hand, the models analyzed in this work are a sense more diverse than the previous empirical studies [27] [7 of the disagreement metric (related to inconsistency), which were restricted to the models with near zero train error.
In Section 3.2 we studied inconsistency from the algorithmic perspective in comparison with the sharpness metrics. We chose sharpness for comparison due to its known correlation to generalization performance and the existence of the algorithm (SAM) to reduce it. We acknowledge that there can be other metrics that are predictive of generalization gap.
Although the correlation of the disagreement metric with generalization gap is relatively poor in the settings of Section [. (c.g.. Figure [0), it is plausible that disagreement can be as predictive of generalization gap as inconsistency in some settings, for example, when the confidence level of predictions is sufficiently high since in that case, Theorem 2. should provide theoretical basis also for disagreement though indirectly (details are provided in Appendix D.3). Moreover, for improving stochastic training of deep neural networks, it would be useful to understand the connection between generalization performance and discrepancies of model outputs in general (whether instability, inconsistency, disagreement, or something else), and we hope that this work contributes to pushing forward this direction.
Conclusion
We presented a theory that relates generalization gap to instability and inconsistency of model outputs, which can be estimated on unlabeled data, and empirically showed that they are strongly predictive of generalization gap in various settings. In particular, inconsistency was shown to be a more reliable indicator of generalization gap than commonly used local flatness. We showed that algorithmic encouragement of consistency reduces inconsistency as well as test error, which can lead to further improvement of the state of art performance. Finally, our results provide theoretical basis for existing methods such as co co-distillation and ensemble.
References
[1] Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George Dahl, and Geoffrey E. Hinton. Large scale distributed neural network training through online distillation. In International Conference on Learning Representations (ICLR), 2018
[2] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. MixMatch: A holistic approach to semi-supervised learning. In Proceedings of the 36th International Conference Machine Learning (ICML), 2019.
10

---

[3] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative components with random forests. In European Conference Computer Vision, 2014.
Olivier Bousquet and André Elisseef. Stability and generalization. Journal of Machine Learning Research, 2:499-526, 2002.
[5] Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. In International Conference on Learning Representations (ICLR), 2022.
[6] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. RandAugment: Practical automated data augmentation with reduced search space. In Advances Neural Information Processing Systems 33 (NeurIPS 2020), 2020.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li. Kai Li, and Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), 2009.
Terrance DeVries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv: (00.::/77 2017.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[10] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness minimization for efficiently improving generalization. International Conference on Learning Reprexentations (ICLR), 2021.
[11] Xavier Gastaldi. Shake-shake regularization. M.XKv:1705.07700 2017.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[13] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge a neural network. In Proceedings of Deep Learning and Representation Learning Workshop: NIPS 2014, 2014.
[14] Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9:1-48 1:1-2,, 1997.
[15] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. In Proceedings of European Conference Computer Vision (ECCV), 2016.
[16] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov; Dmitry Vetrov, and Andrew Gordon Wilson. Av- eraging weights leads wider optima and better generalization. In Proceedings of the Conference on Uncertainty Artificial Intelligence (UAI), 2018
[17] Yiding Jiang, Vaishnavh Nagarajan, Christina Back, and Zico Kolter. Assessing generalization of SGD via disagreement. In International Conference on Learning Representations (ICLR), 2022.
[18] Yiding Jiang. Behnam Neyshabur, Hossein Mobahi. Dilip Krishnan, and Samy Bengio. Fantastic gen- eralization measures and where to find them. In International Conference Learning Representations (ICLR), 2020.
[19] Rie Johnson and Tong Zhang, Guided learning nonconvex models through successive functional gradient optimization. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.
[20] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations (ICLR), 2017.
[21] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-gne ine-grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2011.
[22] Andreas Kirsch and Yarin Gal. A note on "Assessing generalization of SGD via disagreement". Transac- tions Machine Learning Research, 2022.

---

[23] Jonathan Krause, Michael Stark, Jia Deng. and Li Fei-Fei. 3D object representations for fine-grained categorization. In 4th International IEEE Workshop 3D Representation and Recognition (3dRR-13) 13),
Sydney, Australia, 2013.
[24] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[25] Mengtian Li, Ersin Yumer, and Deva Ramanan. Budgeted training: Rethinking deep neural network training under resource constraints. In International Conference on Learning Representations (ICLR), 2020.
[26] Yinhan Liu, Myle Ott, Naman Goyal. Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov, RoBERTa: A robustly optimized BERT pretraining approach. arXiv: 1907. 1692, 2019.
[27] Preetum Nakkiran and Yamini Bansal. Distributional generalization: new kind of generalization. arXiv:2009. 08892, 2020.
[28] Gergely Neu, Gintare Karolina Dziugaite, Mahdi Haghifam, and Daniel M Roy. intommtiiontthoreete generalization bounds for stochastic gradient descent. In Conference on Learning Theory, pages 3526-3545. PMLR, 2021.
[29] Gergely Neu and Lorenzo Rosasco. Iterate averaging as regularization for stochastic gradient descent. In Proceedings of Conference on Learning Theory (COLT),
[30] Daniel Russo and James Zou. How much does your data exploration overfit? Controlling bias via information usage. IEEE Transactions on Information Theory, 6611)330--333 2019.
[31] Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learnability, stability and uniform convergence. Journal of Machine Learning Research, 11 11:2635-2670, 2010.
[32] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. FixMatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020.
[33] Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning (ICML). 2019.
[34] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi- cun-ssupeeised deep learning results, In Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017.
[35] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov. Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner. Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, and Alexey Dosovitskiy Mario Lucie. MLP-Mixer: An alI-MLP architecture for vision. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021.
[36] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task henchmark and analysis platform for natural language understanding. International Conference on Learning Representations (ICLR), 2019.
[37] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoe V. Le. Unsupervised data aug- mentation for consistency training. In International Conference on Learning Representations (ICLR), 2020.
[38] Aolin Xu and Maxim Raginsky. information-theereele analysis of generalization capability of learning algorithms. In Advances in Neural Information P Processing Systems 30 (NIPS 2017). 2017.
[39] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of International Conference on Computer Vision (ICCV), 2019.
[40] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), 2016.
[41] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations (ICLR), 2018.
[42] Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018
12

---

Appendix
A Proof of Theorem 2.1
As in the main paper, let = be random variable representing a labeled data point (data point X and label Y) with distribution Z. Let Zn {(X,, Yi) 1,2, ........)) n) be random variable representing iid training data of size drawn from Z.
We have the following lemma.
Lemma A.1. Given an arbitrary model parameter distribution @ let
fr(x) =Bz (x22)()))) Ez, E0~ f(O,2)
( o(f(x)), 4) where z = (x,
then
-nBz, Ba-aMz, In Bz exp((>te(Z)) S Ez, Ra-en Xee(X,, V.)+E2, (T.(Op22,16))
Proof. Let O be a model parameter distribution such that
O, A exp
(X Y2) In Ez exp(- exp(-Xeo(21))
We have
Ez, exp
Egner
i=1
<E supexp exp
E
(-Xle -n KL()))) KL(
-Ez, sup E Eg - 0u exp
Σ (-Mee(X,,)) -xp -KL]))))
-Ez, E exp
Σ
Xex Y,)
In Ez cxp((xeo =1
The first inequality takes sup over all probability distributions of model parameters. The first equality can be verified using the definition of the KL. divergence. The second equality follows from the fact that the supreme is attained by @ @.. The last equality uses the fact that (X,, Y2) for i = 1, 1....,W are iid samples drawn from Z. The desired bound follows from Jensen's inequality and the convexity of exp(-).
Proof of Theorem 2.1 Using the notation of Lemmal
A.I
have
Ee-Oriz, In Ez exp(-xXeeee)) < Ee PIz. Ez (exp(~Nee -1]
--EE Ezle(Z) + vi Ee Ezl
22
< --EEgnenz, Ezlo(Z) + 4 1 (A)A Fx (0, X) fP(X)
(1)
The first inequality uses u S u - 1. The second inequality uses the fact that ()) is increasing in À and -xe((z) < X. The third inequality uses the Lipschitz assumption of the loss function.
13

---

We also have the following from the triangle inequality of norms, Jensen's inequality, the relationship between the 1-norm and the total variation distance of distributions, and Pinsker's inequality.
Bz, 1 PIZ. Ex ()( - fr(x)11
22E,, P Ex
[l) frzz. (x) F
22 00. Ex (x)
- ,(ee, 23,
- (x(X)
<4Ez, Bo, 50,00-0p.. Ex
Bx Bx -X)
=4Cp + 2Bz, Ex [ (X) Fx( - 4Cp E2, 12. (X) fo z (X)
<4Cp + 2E Ezj Ex BxKL (X) (X) (X)) = (Cp + Sp) = 40p.
(2)
Using 11. ( and Lemma A.1
we obtain
AE2, OpIZ. Ez nl E
<m>²u(A)2 +E, Ez. KL 10").
(3)
Set @ =Ez, θ piz; so that we have
Ip= KL(O P/, (())).))0
Also note that fo in te cancels out since Z, is iid samples of Z, and so using the notation for the test loss and empirical loss defined in Theorem 2.1
we have
λEz O.zz. Ez nla(2)
Σ (o(Xi,
= ndEZ, Eg~ [oz(0) - ( Z.)l
and 55) imply the result.
B
Additional figures
Figure 10 supplements 'Relation to disagreement the end of Section It shows an example where the behavior of inconsistency is different from disagreement. Training was done on 10% of ImageNet with the seed procedure (tuned to perform well) with training length variations described in Table below. Essentially, in this example, inconsistency goes up like generalization gap, and disagreement goes down like test error and goes up the end, as training becomes longer.
Figure I supplements Figure 4 in Section 3.1 shows inconsistency and instability of model outputs of the models trained with SGD with constant learning rates without iterate averaging. In this setting of high final randomness, larger learning rates make inconsistency larger while instability is mostly unaffected. By contrast, Figure 12 shows that when final randomness is low (due to iterate averaging in this case), both inconsistency and instability are predictive of generalization gap both within and across the learning rates.
These figures show the relation of inconsistency and sharpness to generalization gap. Note that each graph has at least 16 points, and some of them (typically for the models trained with the same procedure) are overlapping. Inconsistency shows a stronger correlation with generalization gap than sharpness does.
Figure B 13 supplement Figure [ in Section 3.2
C Experimental details
All the experiments were done using GPUs (A100 or older).
C.1 Details of the experiments in Section 3.1
The goal of the experiments reported in Section 3.1 was to find whether/how the predictiveness of Dp is affected by the diversity of the training procedures comparison. To achieve this goal, we chose the training procedures to experiment with in the following four steps.
14

---

203
Tox coL. (80)
08
022
12
Tramng
Trinnng Ine.
Truning Lons
Tramny
(a) Generalization gap (b) Inconsistency
(c) Disagreement
(d) Test error
Figure 10: Inconsistency Cp and disagreement (y-axis) in comparison with generalization gap and test error (y-axis). The z-axis is train loss. The arrows indicate the direction of training becoming longer. Each point is the average of 16 instances. Training was done on 10% of ImageNet with the seed procedure (tuned to perform well) with training length variations; see 6 In this example, essentially, inconsistency goes up like generalization gap, and disagreement goes down like test error and goes up in the end, as training becomes longer.
Smallest learning rate In Figures 11-12: Only the leurning and length of training varied. Each point represents procedure. The procedures with the same learning rate are connected by a line in the increasing order of training length; generally. generalization gap goes up as the training length goes up. with some exceptions towards the end.
2nd smallest Lr.
smallest Lr.
Largest lr.
CIFAR-10
CIFAR-100
ImageNet
15
35
111
8
25
500
Insoesiorncy
Iusubiliy
Inamsstrracy
Instabilty
Insresiorncy
Instabiity
(a) Inconsistency
(b) Instability
(c) Inconsistency
(d) Instability
(c) Inconsistency
(f) Instability
Figure 11: Inconsistency Cp (left) and instability Sp (right) (z-axis) and generalization gap (y- axis). Sup-
plement to Figure 4 in Section 3.1 SGD with constant learning rate. No iterate averaging, and therefore. high randomness in the final state. Same procedures (and models) as in Figure A larger learning rate makes inconsistency larger, but instability mostly unaffected.
CIFAR-10
CIFAR-100 100
ImageNet
2.5
8.0
05
04
06
DU5
0.5 Insubiity
055
15
0.1 03. 0.5
07
Iaconsitiones
Ircomistonnyy
lasubilty
Incossiteaayy
Instabliyy
(a) Inconsistency
(b) Instability
(c) Inconsistency
(d) Instability
(c) Inconsistency
(f) Instability
Inconsistency Cp (left) and instability Sp (right) (z-axis) and generalization gap (y-axis). SGD with a constant learning rate with iterate averaging, and therefore, low randomness the final state. Same procedures (and models) as in Figure I Both inconsistency and instability are predictive of generalization gap across the learning rates.
Figure 12:
1. Choose network architecture and the size of training set. First, for each dataset, chose a network architecture and the size of the training set. The training set was required to be smaller than the official set that disjoint training sets can be obtained for estimating instability. For the network architecture, we chose relatively small residual nets (WRN-28-2 for CIFAR-10/100 and ResNet-50 for ImageNet) to reduce the computational burden.
2. Choose a seed procedure. Next, for each dataset, we chose procedure that performs reasonably well with the chosen network architecture and the size of training data, and we call this procedure a seed procedure. This was done by referring to the previous studies [32 [0] and performing some tuning on the development data considering that the training data is smaller than in 32 10 This step was for making sure to include high- tigh-performing (and so practically interesting) procedures in our empirical study.
3. Make core procedures from the seed procedure. For each dataset, we made core pro- cedures from the seed procedure by varying the learning rate, training length, and the presence/absence of iterate averaging. 6 the resulting core procedures.
4. Diversify by changing altribute. To make the procedures more diverse, for each dataset, we generated additional procedures by changing one attribute the core procedure. This was done for all the pairs of the core procedures in Table 6 and the attributes in Table 7
15

---

Legend for Case#10 (distillation)
+
A
Teacher Student
standard standard
standard
Consist.
Consist.
Consist+Flat standard
Consist+Flat Consist.
Consist.
standard
Consist.
Legend for Case#1-99
Standard
Consist.
Flat.
Consist+Flat
Generalatioon pvp
Geveriliation pve
Conerallatioon pap
Generattzzion
6
0
(a) Case#1
(b) Casc#2
(c) Case#3
(d) Case#4
(e) Casc#5
Gonralgit pa
Comcealggtion pry
Gonraliaaionn sxp
Center aoteraltution
(f) Case#6
(g) Casc#7
(h) Case#8
(i) Case#9
(j) Casc#10
Figure 13: Supplement to Figure 8 in Section 3.2 Inconsistency (z-axis) and generalization gap (y-axis) for
all the 10 cases. All values are standardized so that the average is and the standard deviation is 1.
(a) Case#1
(b) Case#2
(c) Case#3
(d) Case#44
(e) Case#5
(f) Caset16
(g) Case#7
(h) Case#88
(i) Caselt9
(j) Case#10
Figure 14: Supplement to Figure 8 in Section 3.2 I-sharpness (æ-axis) and generalization gap (y-axis) for all
the 10 cases. All values standardized. Same legend as in Figure B3
2
2
22
(a) Case#1
(b) Case#2
(c) Case#3
(d) Case#4
(e) Case#5
x
2
30+2
5B+3
3Ee4
58+2
5E+3
5B-4
(f) Case#6
(g) Case#7
(h) Case#8
(i) Case#9
(i) Case#10
Hessian (x-axis) and generalization gap (y-axis) for all the 10 cases. All values standardized except that for the 7-axis of Case#88 and non-standardized values shown log-scale for better readability. Same legend as in Figure
Figure 15: Supplement to Figure
g in Section 3.2
16

---

Table Core training procedures experiments in Section 3.1 The core training procedures consist of the exhaustive combinations of these attributes. The seed procedure attributes are indicated by * when there are multiple values. The optimizer was fixed to SGD with Nesterov momentum 0.9.
i More precisely, 25, 50. ... 250, 300, ..... 500, 600. .... 1000, 1200, ..... 2000 that the interval gradually increased from 25 200. $ After training. few procedures with very high training loss were excluded from the analysis. The cut-off was (CIFAR-10), 2.0 (CIFAR-100), and 3.0 (ImageNet), reflecting the number of classes (10, 100, and 1000).
The choice of the constant scheduling is for starting the empirical study with simpler cases by avoiding the complex effects of decaying learning rates, mentioned in the main paper, also, we found that in these settings, constant learning rates rival the cosine scheduling as long as iterate averaging is performed.
CIFAR- 10/100
ImageNet ResNet-50)
Network
WRN-28-2
Training data size Learning rate Weight decay Schedule
4K
120K
[0.005, 0.01, 0.025", 0.05)
{1/64, 1/32, 1/16", 1/8) le-3 Constant (EMA*, None] (10, 20, .... 200 512
Constant
Iterate averaging Epochs Mini-batch size
(EMA*, None 1000'
25,
2000 k
64
Data augmentation
Standard+Cutout
Standard 0.1
Label smoothing
Table Attributes that were varied for making variations core procedures. Only one of the attributes was
varied at a time.
CIFAR-10
CIFAR-100
ImageNet
Network Weight decay
WRN-16-4
5e-4
le-4 Cosine
Schedule
Cosine 256
Cosine 256
Mini-batch size
Data augmentation
None
Table 8: The values of the fixed attributes of the procedures shown in Figures 1 4 and 6 as well as 11 12 The training length and the learning rate were varied as shown in Table The presence/absence of iterate averaging is indicated in each figure.
CIFAR-10 CIFAR-100
ImtgeNet ResNet-50
Network Training data size Weight decay Schedule Mini-batch size Data augmentation Label smoothing
WRN-28-2
4K
120K
2c-3
1c-3
Constant
Constant
256
64
$12
Standard+Cutout
Standard 0.1
Table The values of fixed attributes the procedures shown in Figures 16
training length and
the learning rate were varied as shown in Table 6 The presence/absence of iterate averaging is indicated in each figure. The rest of the attrihutes are the same as in Table 8
CIFAR-IO
CIFAR-100
ImageNet
Weight decay
Se-4
2e-3
le-4
Mini-batch size
64
256
512
Note that after training. a few procedures with very high training loss were excluded from the analysis (see Table/offo the cut-off). Right after the model parameter initialization, inconsistency Cp is obviously not predictive generalization gap since is non-zero merely reflecting the initial randomness while generalization gap is zero. Similar effects of initial randomness are expected in the initial phase of training; however, these near random models are not of practical interest. Therefore, excluded from analysis.
17

---

C.1.1 SGD with constant learning rates (Figures 14 and 6
In Section 3.1 we first focused on the effects of varying learning rates and training lengths while fixing anything else, with the training procedures that use constant learning rate with or without iterate averaging. We analyzed all the subsets of the procedures that met this condition, and reported the common trend. That is, when the learning rate is constant, with iterate averaging, Dp is predictive of generalization gap within and across the learning rates, and without iterate averaging. Dp is predictive of generalization gap only for the procedures that share the learning rate; moreover, without iterate averaging, larger learning rates cause Dr to overestimate generalization gap by larger amounts. Figures andd one particular subset for each dataset, and Table [8 shows the values of the attributes fixed in these subsets. To demonstrate the generality of the finding, we show the corresponding figures for one more subset for each dataset in Figures 16 The values of the fixed attributes in these subsets are shown in Table
Smallest learning rate In Figures 16 19: Only the learning rate and the length of training are varied. Each point represents a procedure. The procedures with the learning rate are connected a line the increasing order of training length; generally, generalization gup goes the training length goes up. with some exceptions towards the end.
2nd smullest Lr, 3rd smallest Lt.. Largest lr.
1.5
16
13
23
33
11
18
211
26
(a) CIFAR-10 (b) CIFAR- 100
(c) ImageNett
(a) CIFAR-I0
(b) CIFAR-100 100
(c) ImageNet
Dp (z-axis) and generalization gap (y- axis). SGD with a constant learning rate and iterate averaging, Only the learning were varied as in F and the attributes were fixed to the values different from Figure I see Table the fixed values. As in Figure positive correlation is observed between Dp and generalization gap.
Figure 16:
Figure 17: Dp (æ-axis) and generalization gap (y- axis). Constant learning rates. No iterate averaging. Only the learning rate and training length were varied as in Figure[ann and the attributes were fixed to the values different from Figure 2 see fable the fixed the val- ues. As in Figure 2 Dp is predictive of generalization gap for the procedures that share the learning rate, but not clear otherwise.
and training length
CIFAR-10
CIFAR- 100
ImageNet
1.1
00
0.8
122
0
0L2
57 DM L1
13 15
03 05 02 09 11
Incorsisuncy
Instabiltty
Inamssirzayy
Instabiiyy
Inarssieency
Instabilty
(a) Inconsistency
(b) Instability
(c) Inconsistency
(d) Instability
Inconsistency
(f) Instability
Figure 18: Inconsistency Cp (left) and instability Sp (right) (æ-axis) and generalization gap (g-axis). Same procedures (and models) as in Figure 17 (no iterate averaging). As Figures 4 and 1T (also no iterate
averaging), a larger learning rate makes inconsistency larger, but instability is mostly unaffected.
Test Trme 12
Toe cine
70
The
50
Diagecement
DNcagrement
Diagroemeze
(a) CIFAR-10 (b) CIFAR-100
(c) ImageNet
Figure 19: Disagreement (ı-axis) and test error (y- axis). Same models and legend as in Fig 16
C.1.2 Procedures with low final randomness (Figures and
The procedures shown in Figures 5 and 7 are subsets (three subsets for three datasets) of all the procedures (the core procedures Table times the attribute changes in Table The subsets consist of the procedures with either iterate averaging vanishing learning rate (i.c., going to zero) so
18

---

that they meet the condition of low final randomness. These subsets include those with the cosine learning rate schedule. With the cosine schedule, we were interested not only letting the learning rate go to zero but also stopping the training before the learning rate reaches 7ero and setting the final model to be the iterate averaging (EMA) at that point, which is well known to be useful (e.g.. (2). Therefore, we trained the models with the cosine schedule for { 250, 500, 1000, 2000) } epochs CCIFAR-10/100 or 200 epochs (ImageNet), and saved the iterate averaging of the models with the interval of one tenth (CIFAR-10/10) or one twentieth (ImageNet) of the entire epochs.
C.2 Details of the experiments in Section 3.2
Section 3.2 studied inconsistency in comparison with sharpness in the settings where these two quantities are reduced by algorithms. 'he training algorithm with consistency encouragement (co-distillation) is summarized in Algorithm[] [ These experiments
designed to study
practical models trained on full-size training data, and
diverse models resulting from diverse training settings,
in the situation where algorithms compared after basic tuning is done, rather than the hyperparam-
eter tuning-like situation in Section 3.1
C.2.1 Training of the models
Table 10: Basic settings shared by all the models each case (Case#1-7,110, images)
Training type
From scratch
Fine-tuning
Distillation
Dataset
ImageNet
Food101
CIFARIO
Cars / Dogs
Food101
Network
ResNet50
ViT
VIT Mixer
WRN28-2
EN-BO
ResNet- 18
Batch size
512
4096
512
64
256
$12
Epochs
300
200 100
400
Update steps
500K
4K 2K
Warmup steps
0
10K
0
0
0
0
Learning rate
0.125
3e-3
3e-3
0.03 Cosine Momentum
0.1
0.125
Schedule
Cosine
Linear/Cosine AdamW 0.3
Constant Momentum
Cosine
Optimizer
Momentum
AdamW 0.3
Momentum 1e-3
Weight decay
le-4
5c-4
1e-5
Label smooth
0.1
0
0.1
0
0
0
Iterate averaging
EMA
EMA 20.0
Gradient clipping
1.0
1.0
Data augment
Standard
Cutout
Standard
Reference
10
[5]
[5]
[10.32
[10
10
Casell
2
3/4
6/7
10
'Momentum': SGD with Nesterov momentum 0.9.
This section describes the experiments for producing the models used in Scction 3.2
Basic settings Within each of the 10 cases, we used the same basic setting for all, and these shared basic settings were adopted/adapted from the previous studies when possible. Tables 10 and I]
Table 11:
Basic settings shared by models for each case (Case#8-9; text). Hyperparameters for Case#8-9 (text) basically followed the RoBERTa paper (26). The learning rate schedule was equivalent to early stopping of 10-epoch linear schedule after 4 epochs. Although it appears that [26] tuned when to stop for each run, we used the same number of epochs for all. Iterate averaging is our addition, which consistently improved performance.
Initial learning rate 7]0 Learning rate schedule
lc-5
Linear from no to 0.67j0
Epochs
Batch size
32
Optimizer
AdamW (3,=0.9, 30=0.88, e=le-6
Weight decay
0.1
Iterate averaging
EMA with momentum 0.999
19

---

describe the basic settings and the previous studies that were referred to. Some changes to the previous settings were made for efficiency in our computing environment (no TPU); e.g., for Case#1, we changed the batch size from 4096 to 512 and accordingly the learning rate from 1 to 0. 125. When adapting the previous settings to new datasets, minimal tuning was done for obtaining reasonable performance, e.g., for Casesf3 and 4, we changed batch size from 4096 to 512 and kept the learning rate without change as it performed better.
For CIFAR-10, following [32] we let the learning rate decay to 0.2rjo instead of O and set the final model to the EMA of the models with momentum 0.999. For Cases#6-7 (fine-tuning), we used a constant learning rate and used the EMA of the models with momentum 0.999 as the final model, which we found produced reasonable performance with faster training. Cases#6-7 fine-tuned the publicly available EfficientNet-Bol BBC pretrained with ImageNet by [33]. The dropout rate was set to 0.1 for Case#33, and the stochastic depth drop rate was set to 0.1 for Casett4. The teacher models for Casef1 10 (distillation) were ensembles of ResNet-18 18 trained with label smoothing 0.1 for 200 epochs with the same basic setting as the student models (Table 10) otherwise.
For CIFAR-I0, the standard data augmentation (shift and horizontal flip) and Cutout [ were applied. For the other image datasets, only the standard data augmentation (random crop with distortion and random horizontal flip) was applied; the resolution was 224x224.
Table 12:
Hyperparameters for SAM.
Case#
2
3
4
6
7
8,9
10
m-sharpness
128
256
32
32
32
16
2
128
P
0.05,0.1
0.05
0.1
0.1
0.1,0.2
0.1,0.2
0.1
0.005,0.01
0.1
There are two values that affect the performance of SAM, m for m-sharpness and the diameter of the neighborhood p. Their values are shown in Table 12 [00 found that smaller m performs better. However, a smaller m can be less efficient as can reduce the degree of parallelism, depending on the hardware configuration. We made m no greater than the reference study in most cases, but for practical feasibility we made larger for Case#2., / was either set according the reference when possible chosen the development data otherwise, from {0.05, 0,1, 0.2) for images and from (0.002, 0.005, 0.01, 0.02, 0.05, 0.1] J for texts. For some cases (typically those with less computational burden), we trained the models for one additional value of to have more data points.
Hyperparameters for SAM
Hyperparameters for the inconsistency penalty term term for encouraging consistency was fixed to 1.
The weight of the inconsistency penalty
Number of the models For each training procedure (identified by the training objective within each case), we obtained models trained with 4 distinct random sequences. Cases#1 1-9 consisted of either 4 or 6 procedures depending the number of the values chosen for p for SAM. Case# 10 (distillation) consisted of 6 procedures resulting from combining the choice of the training objectives for the teacher and the choice for the student. In total, we had 52 procedures and 208 models.
C.2.2 Estimation of the model-wise inconsistency and sharpness in Section 3.2
When the expectation over the training set was estimated, for a large training set such as ImageNet, 20K data points were sampled for this purpose. As described above, had 4 models for each of the training procedures. For the procedure without encouragement of low inconsistency, the expectation of the divergence of each model was estimated by taking the average of the divergence from the three other models. As for the procedure with encouragement of low inconsistency, the four models were obtained from two runs each produced two models, and so when averaging for estimating inconsistency, we excluded the divergence between the models from the same run due to their dependency.
comm/ooole--eesarreee sam
a "Athhouhh the number of all possible combinations is 16, considering the balance with other cases, we chose to experiment with the following: teacher | 'Standard', 'Consist.' "Consist+Flat') } x student | 'Standard', 'Consist')
20

---

Algorithm 1: Training with consistency encouragement. Co-distillation (named by [I]. closely related deep mutual learning (2).. Without additional unlabeled data. Input & Notation: Labeled set Z. 3 (default: 1), learning rate Let ø be loss, and let f(0,z) be the model output in the form probability estimate.
Sample 0 and 0b from the initial distribution.
for t = 1...., T do
Sample two labeled
8 and 8 from Zn.
6 T/e V
1 E ( B ( .)..)) + T (E))) (11[(00,)11/(0,,)))))
V gb
Σ (L ,()(0,,))))))))))))))))))))))))) +B Te E (B ( .x)]
1 1-sharpness requires (the diameter of the local region) as input. We set it to the best value for SAM.
C.3 Details of the experiments in Sections 3.3
The optimizer was SGD with Nesterov momentum 0.9.
Table 13: Supplement to Figure 9 (a). Error rate (%) and inconsistency of ensembles in comparison with non-ensemble models (+). Food101, ResNet-18. The average and standard deviation of 4 are shown. Ensemble reduces test and inconsistency.
Training method
Test error(5))
inconsistency
Non-ensemble
17.09 20,20
0.30 001
Ensemble of standard models
14.99+0.05
0.14
Ensembles of Consist. models
14.07
0.1010.001
C.3.1 Ensemble experiments (Figure 9 (a))
The models used in the ensemble experiments were ResNet- 18 trained for 200 epochs with label smoothing 0.1 with the basic setting of Case#10 in Table 10 otherwise. Table 13 shows the standard deviation of the values presented Figure 9 (a). The ensembles also served as the teachers in the distillation experiments.
C.3.2 Distillation experiments (Figure (b))
The student models were ResNet-18 18 trained for 200 epochs with the basic setting of Case#f 10 of Table otherwise. The teachers for Figure 9 b) (left) were ResNet- 18 ensemble models trained
as described in C.3. The teachers for Figure [ (middle) were ResNet-50 ensemble models
trained similarly. The teachers for Figure b)) (right) were EfficientNet-BO models obtained by fine-tuning the public IrageNet-trained model (footnote(] fine-tuning was done with encouragement of consistency and flatness (with 1=0.1) with batch size 512, weight decay 1e-5, the initial learning rate 0.1 with cosign scheduling, gradient clipping 20, and 20K updates. Table 14 shows the standard deviations of the values plotted Figure (b).
Table 14: Supplement Figure (b). Test error (%) and inconsistency distilled modeels comparison with standard models (+). The average and standard deviation of are shown.
Test error (%)
inconsistency middle
left
middle
right
left
right
17.09
0.30
15.74=a.09
14.95+0.17
14.61
0. 0.194g.001
0.21 20.001
0.22+0.002
14.99=0.07
14.2810.11
13.8910.08
0. 0.14410.001
0.15 001
0. 0.1410.002
14.41 20.05
13.6340.00
13.0640.06
0. 0.104a.00
0.12
0.08 40.00
21

---

Algorithm 2: Our semi-supervised variant of co-distillation.
Input & Notation: Labeled set Zn unlabeled set U. 8 (default: 1), r (default: 0.5), learning rate
7), momentum of EMA (default: 0.999). Let e be loss, and let f(0, x) be the model output in the form of probability estimate.
Initialize 0%, 0b, 0%, and 0b.
2 for t = ,, ..'T'o do
3
Sample labeled mini-batches 8" and Bh from Zn, and sample an unlabeled mini-batch Bu from U.
(0000,))10 =)),(())))) where is the indicator 4 Let (0, 0. = [Imaxx (00 > +)
function.
0 0+
1/
Ve
2
En (000)))).))) + |Bo Exeny 200,00,00 E( (f((p), 2).,) v)
@ c H
Ni
g
[Bbb]
|Pul EZeBy (0)), 8e, x))
1
0 and @ keep the EMA of 0 and 09, with momentum μu, respectively.
7
C.3.3 Semi-supervised experiments reported in Table 4
The unlabeled data experiments reported in Table 4 used our modification of Algorithm [ taylored for use of unlabeled data, and it is summarized in Algorithm 2 It differs from Algorithm in two ways. First, to compute the inconsistency penalty term, the model output is compared with that of the exponential moving average (EMA) of the other model, reminiscent of Mean Teacher 34 The output of the EMA model during the training typically has higher confidence (or lower entropy) than the model itself, and taking the KL divergence against EMA of the other model serves the purpose of sharpening the pseudo labels and reducing the entropy on unlabeled data. Second, we adopted the masked divergence from Unsupervised Data Augmentation (UDA) 37,, which masks (i.e., ignores) the unlabeled instances for which the confidence level of the other model is less than threshold. These changes are effective in the semi-supervised setting (but not in the supervised setting) for preventing the models from getting stuck in a high-entropy region.
In the experiments reported in Table 4 we penalized inconsistency between the model outputs of two training instances of Unsupervised Data Augmentation (UDA) 37. using Algorithm 2 by replacing loss @(8, æ²) with the UDA objective. The UDA objective penalizes discrepancies between the model outputs for two different data representations (a strongly augmented one and a weakly augmented one) on the unlabeled data (the UDA penalty). The inconsistency penalty term of Algorithm 2 also uses unlabeled data, and for this purpose, we used a strongly augmented unlabeled batch sampled independently of those for the UDA penalty. UDA 'sharpens' the model output on the weakly augmented data by scaling the logits, which serves as pscudo labels for unlabeled data, and the degree of sharpening a tuning parameter. However, we tested UDA without sharpening and obtained better performance on the development data (held-out 5K data points) than reported in [32], and so we decided to use UDA without sharpening for our UDA+ experiments. We obtained test error 3.95% on the average of 5 independent runs, which is better than 4.33% of UDA alone and 4.25% of FixMatch. Note that each of the runs used a different fold of 4K examples as was done in the previous studies.
Following (32). we used labeled batch size 64, weight decay 5e-4, and updated the weights 500K times with the cosine learning rate schedule decaying from 0.03 to 0.03..03. We set the final model to the average of the last 5% iterates (i.e., the last 25K snapshots of model parameters). We used these same basic settings for all (UDA, FixMatch, and UDA+'Consisee for testing UDA and FixMatch was 64x7, 7, as in 32 For computing each of the two penalty terms for 'Donsssssss we set the unlabeled batch size to 64 x4, which is approximately one half of that for UDA and FixMatch. We made one half so that the total number of unlabeled data points used by each model (64x4 for the UDA penalty plus 64 for the inconsistency penalty) becomes similar to that of UDA or FixMatch. RandAugment with the same modification as described the Mixhatch study was used as strong augmentation, and the standard data augmentation (shift and flip) was used as weak augmentation. The threshold for masking was set to 0.5 for both the UDA penalty and the inconsistency penalty and the weights of the both penalties were set to L. Note that we fixed the
The unlabeled batch size
22

---

weights of penalties to 1, and only tuned the threshold for masking by selecting from {0, 0.5, 0.7} on the development data (5K examples).
C.3.4 Fine-tuning experiments in Table
The EfficientNer-B4 (EN-B4) fine-tuning experiments reported Table were done with weight decay le-5 (following [O]), batch size 256 and number of updates 20K following the previous studies. We set the learning rate to 0.1 and performed gradient clipping with size 20 to deal with sudden surge of the gradient size. The diameter of the local region p for SAM was chosen from {0.05, 0.1, 0.2} based on the performance of SAM on the development data, and the same chosen value 0.2 was used for both SAM and Consist.. Following the EN- experiments of [0], the value m for m-sharpness was set to 16. Since SAM+CConsiinn
is approximately twice expensive result of training two models, we also tested SAM with 40K updates (20Kx2) and
as SAM found that it did not improve performance. We note that our baseline EN-B4 SAM performance is better than the EN-B7 SAM performance of [T]. This due to the difference the basic setting. In [10, BN-B7 was fine-tuned with a larger batch size 1024 with smaller learning rate 0.016 while the batch normalization statistics was fixed to the pre-trained statistics. Our basic setting allowed a model to go farther away from the initial pre-trained model. Also note that we experimented with smaller EN-B4 instead of EN-B7 due to resource constraints.
D
Additional information
D.1 Additional correlation analyses using the framework of Jiang al. (2020)
This section reports the additional correlation analysis using the rigorous framework of Jiang et al. (2020) [8] and shows that the results are consistent with the results in the main paper and the previous work. The analysis uses correlation metries proposed by [18), which seek to mitigate the effect of what [18] calls spurious correlations that do not reflect causal relationships with generalization. For completeness, we briefly describe below these metrics, and [8] should be referred for more details and justification.
Notation In this section, we write æ for a training procedure (or equivalently, a combination of hyperparameters including the network architecture, the data augmentation method, and forth). Let g(Tr) be the generalization gap of ग, and let μu(T) be the quantity of interest such as inconsistency disagreement.
Ranking-based Let T be the set of the corresponding pairs of generalization gap and the quantity of interest to be considered: T : Un ((u(T))g(TT)) Then the standard Kendall's ranking coefficient of T can be expressed as:
Σ (M1.0))T (T
Σ
sign( - n22 (2)sign(g) -92)
18 defines granulated Kendall's coefficient Vi as:
Σ
Σ
Σ
Σ 7 (Un,em (v(T). (6)
Wi
mi
xilll
---EIIl-1 Ell+e
Ih Ella
where m; is the i-th hyperparameter so that (MI, nn). II is the set of all possible values for the i-th hyperparameter, and m, = [ x ... .....xx II,11 x I1 x ... x Il
Mutual information-based Define (T n²) = sign(g(T) g(*'),, and similarly define V,(T,,)) == sign(μ(T)) ==(='). Let Us be random variable representing the values of the hyperparameter types in (e.g.. S learning rate, batch size }). Then VVV,,VylUss) the con- ditional mutual information between μ and given the set S of hyperparameter types, and the
23

---

Table 15: Correlation scores of inconsistency and disagreement. For the training procedures with low final randomness (as in Figure(S], model-wise quantities (one model per procedure) were analyzed. (a)-(c) differ the restriction on training loss; (b) and (c) exclude the models with high training loss while (a) does not. The average and standard deviation of 4 independent runs (that use 4 distinct subsamples of training sets as training data and distinct random seeds) are shown. Correlation scores: Two types of mutual information- (CV,.V,lus. with [S|=0) and two types of Kendall's 's rank-corre coefficient- scores (1) as in and overall r). A larger number indicates higher correlation. The highest numbers are highlighted. Tested quantities: 'Inconsist.': : Inconsistency, "Disagree.": Disagreement, 'Random' (baseline): random numbers drawn from the normal distribution, 'Canonical' (baseline): slight extension of the canonical ordering in [18]: heuristically determines the order of two procedures by preferring smaller batch size, larger weight decay, larger learning rate, and presence of data augmentation (which are considered to be associated with better generalization) by adding one point for each and breaking ties randomly. Target quantities:
based scores ('K' as in 7 and =I")=0":
H(V,lUs)
Generalization gap (test loss minus training loss) as defined in the main paper, test error, and test error minus training error. Observation: Inconsistency correlates well to generalization gap, and disagreement correlates well to test error. With more aggressive exclusion of the models with high training loss (going from (a) to (c)), the correlation of disagreement to generalization gap improves and approaches that of inconsistency.
(a) No restriction on training loss
CIFAR-10
CIFAR- 100
ImageNet
K
[8] 3
V
-0
V
S
Correlation to generalization gap ('lest loss training loss")
Inconsist. 0.233.01
0..8...o1 os
0.600.os
0.69
0.69+.01
0.51+.01
0.58+.ox
0.68
+
0.78+.01
0.75
+
0.73
-.17
0.92
Disagree. 0.14
0.26g.01
0.541.05
0.58
0.11
0.11
0.481.08
0.391.01
0.11
0.166.01
0.53
0.47
Random 0.000 00 0.00 00
+0.02..0 04
0.01
oz
0.00
00
0.00
0.01
0.02 02
0.00
0g
0.01
00
0.00
13
0.04 06
Canonical 0.01 00
0.10+
0.32+ oz
0.37 01
0.00
0.09
0.144.05
0.36 .01
0.00
00
0.07
0.36
0.32
Correlation to test crror
Inconsist. 0.00 2. 01
0.
01
0.51 04
0.46
0.00 +
00
0.01
0.33 00
0.14 01
0.03 00
0.03
00
0.20 03
0.20
00
Disagree. 0.31
0.37
0.58
0.68
0.19
0.31 4.02
0.57+.06
0.63+.01
0..4
0.05
0.30
0,25
Random 0.00 -00 0.00+
00
0.02+.oo
0.00
0.00
0.00+..0
-0.05 +.07
-0.00
0.00+..0
0.00..00
000
-0.07
1.07--00
0.01
Canonical 0.01 oo
0.03
0o 0.152.0z
0.20 01
0.00
0.31
0.494.01
0.00 on
0.03
0.08
0.18
Correlation to 'lest errve training error'
Inconsist. 0.11
00
0.25 01
0.53 o0
0.57
0.41
0.48
0.60
0.77 0.00
0.73. 011
0.733..oo
000
0.91=
0.
0.91
Disagree. 0.16 166..00 01
0.28
0.54
0.60
0.12
0.12
0.45 1
0.401.o1
0.12 01
0.16
0.52
0.46_
Random 0.00
0.00
00
2.04
0.00
0.00
0.00
0.01 1.07
0.01
4.03
0.00 00
0.00
0.06
0.03
Canonical
0.01 00 0.00
0.31+.00
0.31 +.01
0 0.00+.00 0.09+.00
0.09
0.35+.01
0.004.00
0.07 .00
0.24 00
0.32 .00
(b) Excluding the models with very high training loss. as described in Appendix
and Table
CIFAR-10
CIFAR-
100
ImageNet
K
|S| -0
10
S =0 V
KC
5 =
Correlation generalizaaion gap ("test loss training loss
Inconsist. 0.37 0.
0.57 01
0.642.00
0.33 01 0,47 0.01
0.52
0.68
0.794.0
1
0.74 01
0.75
0.74 16
0.92 00
Disagree. 0.34 0.34 0.53
1.61
0.80
0.261.03
0.601.om 0.33
0.65 1.61
0.24
0.33
0.61
0.65
Random 0.00 no
0.00
00
0.01 o5
0.001.00
oz
0.00
oo 0.00
0.06 o0
0.01 + 03
0.00
00
0.00
00
0.10 07
0.05
Canonical 0.02 no
0.15
0.284.00
0.44
01
0.00
0.21 01
0.23
0.52
0.00
00
0.10
0.46
0.38
Correlation to test emoor
Inconsist. 0.100..00
0.18 01
0.51
0.49
0.01
0.10
0.42
0.36 .2
2.00
0.01 = 00
0.21
0.10_.00
Disagree. 0.27
0.33+
0.57+.00
0.65+.00
0.15+.01
0.27+.01
0.59 0.564.05
0.02
0.024.00
0.25
0.16-.02
Random 0.00
0.00
0o
0.03
0.01 t.02
0.00
2.000..0
0.00
0.04
0.02
0.00
0.00
000
0.04-
0.01
Canonical 0.01 1:
0.033
00
0.141.02
0.21
0.00
0.20
0.25
0.51
0.00 on
0.02
0.41
0.19
Correlation to Test error - training emor
Inconsist. 0.19 01
00.7
01 0.37 04
0.69
001
0.35 + 01
0.44 011
0.60
0.74 001
0.72 01
0.73 00
0.93 08
0.91 000
Disagree. 0.35
0.52+
0.60+.00
0.80
0.30+.o3
0.35+.01
0.57+.08
0.25 0.67+.01
oz
0.32+.01
0.61-000
0.64-.02
Random 0.00
0.00
0.01
o0
0.022
0.00
0.00
0.01
4.08
0.01
4.03
0.00 00
0.00
00
0.02
0.06
Canonical 0.00 0.10+.00 00
0.29+.02 2.0 0.36+.01
0 0.00+.00 0.20
0.23 2.0g
0.50+.01
001
0.00
0.
0.65 00
0.38
(c) Excluding the models with high training more aggressively smaller cut-off values (one (b)
CIFAR-10
CIFAR-1O0
ImageNet 1
IC
| =0 A
AC
[S] =0
KC
|||=
Correlation generalizzaz izaerialiaaionnoooo gap ("lest loss training loss.)
Inconsist. 0..6 01
0.57 o1 0.68
0.82 0.44
0.50
0.694.06
0.78
0.77
0.79
+
0.74_
16
0.93
Disagree. 0.39+.01
0.57+.01
0.62..06
0.82+.01
0.12
0.43
0.654.06
0.734.01
0.434.01
0.53
0.69
0.80
Random 0.00
0.00
00
0.02
0.00
0.00
0.00
0.03
0.04
0.00
0.00
00
-0.05
12
-..02..00
Canonical 0.02
0.14
0.36g.o1
0.44
0.00
0.27
0.23
0.60
0.00 oo
0.133
0.44
0.42
Correlation test emvor
Inconsist. 0.12 01
0.21 ox
0.53 04
0.53
0.04
0.19
0.48
0.50
.2
0.01
0.01
+: 000
0.31 02
Disugree. 0.27 00
0.35
0.581
00
0.661.01
0.19 19.0000
0.341.01
0.581.00
0.65
0.05
00
0.05 1g
0.27
0.25
Random 0.00 no
0.00
0o
0.01
0.
4.03
0.00 00
0.00
2.11
0.01
+
0.00
00
0.00
00
1.01
0.01
Canonical 0.01 00
0.033.00
0.25+.0a
0.22+
.1
0.000.00
0.24+.01
0.25+.08
0.56
0.00
oo
0.06
0,20+-10
0.29
Correlation to Tost crror - training crror"
Inconsist. 00. 00 00.37
01
2.58
0.68
0 0..00..1
0.42+ o1
0.61
0.72+.01
00.72
0..66..o
0.93
0.92_.00
Disagree. 0.37 0.54+
0.61+.0e
0.37 0.80
0.46
0.63
0.75
0.44
0.52
0.69
0.79_.0.
Random 0.000..0 0.00
0o
0.01
os
0.02
0.00 oo
0.00
0.00
0.021.03
0.00
0.00 00
-0.05
-0.44..4 04
Canonical 0.01 no 0.091..0
00
0.28
0.35 o1
0.00
0.26
0.24 .0
0.58
0.
0.00
00
0.14_
0.42
0.44 00
24

---

conditional entropy H (Vy)ss) can be expressed as follows.
p(Vn, V2JUs) ((VAIUs))(VV)UU))
T(V,, Vglus)) Σ ((U)))
E Σ
p(Vn, Vg log
Us
(EELL)
H(VgIUs)= - --==----- p(Us)
Σ p(V))Us) (og(o(()\)))
Us
(E((1)
Dividing I(V V2(Us) by HV)) for normalization and restricting the size of for enabling ns
computation, [18 defines the metric K(j)) as:
K(p) = min I(V VglUs)
Us s.. (V,
we show the average and standard deviation of the correlation scores of model-wise quantities (for one model per training procedure), follow- ing 18] The mosel-wise inconsistency for model 0 trained on Z, with procedure is E [x (0) X)), and Zn was fixed here; similarly, model-wise disagreement is Eg-e, r. 2., ExI[ ((O, X) & c(0, X)] where c(0, xr) is the classification decision of model 0 on data point x. The average and standard deviation were computed over independent runs that used distinct subsamples of training sets training data Z,, and distinct random seeds for model parameter initialization, data mini-batchinn and so forth.
Results (Tables
15 16)
In Table [5]
Table 1.5 compares inconsistency and disagreement in terms of their correlations with the gener- alization gap (test loss minus training loss as defined in the main paper), test error, and test error minus training error. The training procedures analyzed here are the procedures that achieve low final randomness by either vanishing learning rate or iterate averaging as in Figure Tables (a), (b), and (c) differ in the models included in the analysis. As noted in Appendix C.I since near-random models the initial phase of training are not of practical interest, procedures with very high training loss were excluded from the analysis in the main paper. Similarly, Table (5) excludes the models with high training loss using the same cut-off values as used in the main paper, and (c) does this with smaller (and therefore more aggressive) cut-off' values (one half of (b)). and (a) does not exclude any model. Consequently, the average training loss is the highest in (a) and lowest in (c).
Let us first review Table 15 (a). The results show that inconsistency correlates well to generalization gap (test loss minus training loss) as suggested by our theorem, and disagreement correlates well to test error as suggested by the theorem of the original disagreement study [17]. Regarding 'test error minus training error' (last rows): on CIFAR-10, training error is relatively small and so approaches test error, which explains why disagreement correlates well to it; on the other datasets, "test error minus training error' is more related to 'test loss minus training loss', which explains why inconsistency correlates well to it. The standard deviations are relatively small, and the results are solid. (The standard deviation of y tends to be higher than the others for all quantities including the baseline 'Random', and this is due to the combination of the macro averaging-lide nature of y and the smallness of [T:| for some i's, independent of the nature of inconsistency or disagreement.)
The overall trend of Table (b) and (c) is similar to (a). That is, inconsistency correlates well to generalization gap (test loss minus training loss) while disagreement correlates well to test error, consistent with the results in the main paper and the original disagreement study. Comparing (a), (b), and (c), we also note that as we exclude the models with high training loss more aggressively (i.e., going from (a) to (c)), the correlation of disagreement to generalization gap (relative to that of inconsistency) improves and approaches that of inconsistency. For example, on CIFAR-100 100, the ratio of K. for (disagreement, generalization gap) with respect to I. for (inconsistency, generalization gap) improves from (a) 1/0.55551 to (b) 0.22/0..475559 to (c) 0.31/0.44=7095 With these models, unseen data, and
empirically, high training loss roughly corresponds to the low confidence-level so this observation is consistent with the theoretical insight that when the confidence-level on unseen data is high, disagreement should correlate to generalization gap well as inconsistency, which is discussed in more detail in Appendix D.3
Table 16 shows that the correlation of the estimate of Cp (defined in Section 2p to the generalization gap is generally as good as the estimate of DD (defined Theorem 2. which is consistent with the results in the main paper.
25

---

Table 16: Correlation analyses of the estimates of Cp and Dp (as defined in Section 2 using the rank-based and mutual information- based metrics from [8]. Training procedures with low final randomness of Figure [S
Correlation scores: see the caption of Table 15 Target quantities: Gencralization gap (test loss minus training
loss) analyzed in Theorem 2.1 Observation: The correlation of Cp to generalization gap is generally good
as Dp, which is consistent with the results the main paper.
CIFAR-10
CIFAR- 100
ImageNet Net
MI-based
Ranking
MI-hased
Ranking
MI-based
Ranking
K
[
V/
7
K.
[
V
7
K.
V
7
Cr
0.38
0.59
0.71
0.83
0.49
0.53
0.80
0.80
0.75
0.75
0.59
0.92
Dp
0.38
0.59
0.80
0.84
0.42
0.47
0.75
0.76
0.81
0.81
0.60
0.94
D.2
Training error
Tables 17 and 18 show the training error values (the average, minimum, median, and the maximum) associated with the empirical results reported in Section 3.1 and 3.2 respectively.
Table 17: Training error of the models analyzed in Section 3.1
The four numbers represent the average.
minimum, median, and maximum values (%).
CIFAR-10
CIFAR- 100
ImageNet
Figure 3
0.4
0.0
0.0
8,8
4.7
0.0
0.1
53.6
24,9
3.0
23.5
56.6
Figure
3.3
0.0
2.6
10.9
18.2
0.2
13.6
56.3
44.5
16.8
45.6
65.3
Figure
0.5
0.0
0.0
10.1
4.0
0.0
0.0
54.5
16.3
0.1
11.3
59.0
Table 18:
Training error of the models analyzed in Section minimum, median, and maximum values (%).
3.2
The four numbers represent the average.
Case#1
Case#22
Case#3
Case#t
Case#55
10.0 8.8 10.1 11.6 4.5 2.1 4.5 6.7 0.06 0.02
0.04
0.17
0.04
0.01
0.03
0.11
0.0
0.0 0.0
0.0
Case#6
Case#7
Case#8
Case#9
Casc#T0
0.7 0.3 0.8 1.1 0.1 0.1 0.1 0.2 4.8
3.9
4.9
5.5
2.7
1.9
2.9
3.3 2.8 0.5 1.8 6.5
D.3 More on inconsistency and disagreement
Inconsistency takes how strongly the models disagree on each data point into account while disagree- ment ignores it. That is, the information disagreement receives on each data point is binary (whether the classification decisions of two models agree or disagree) while the information inconsistency receives is continuous and more complex. On the one hand, this means that inconsistency could use information ignored by disagreement and thus could behave quite differently from disagreement as seen in our empirical study. On the other hand, it should be useful also to consider the situation where inconsistency and disagreement are highly correlated since in this case our theoretical results can be regarded as providing a theoretical basis for the correlation of not only inconsistency but also disagreement with generalization gap though indirectly.
To simplify the discussion towards this end, let us introduce a slight variation of Theorem 2.1 which uses I-norm instead of the KL-divergence since disagreement is related 1-norm as noted in Section
2
Using the notation of Section 2 define 1-norm inconsistency CTPP and I-norm instability Sr.P which use the squared I-norm of the difference in place of the Kl-divergence as follows.
Proposition D.1 (1-norm variant of Theorem 2.1
Ci,r = E2, Re Ex X) - 1(0',
(1-norm inconsistency)
St,p = St,p=E,,.2 Ex Ilfiz, (X) friz; (X)112
(I-norm instability)
Then with the same assumptions and definitions as Theorem 2.1 we have
Ip
Ez, Be-eriz (0) - (600 < inf
b (A)) + +1,P) An
A5O 2
26

---

Sketch of proof. Using the triangle inequality of norms and Jensen's 's inequality, replace inequality of the proof of Theorem 2.1
Ez, OOI Ex)I (X S 4 +S)
with the following,
Bz Be.B Ex -
22Ez, B Ex [l] friz. (X)II I (X) fx())
<2E,, Bo,00-mm, Bx) - S(0', x)11 + 2Bz, Ezg Ex BxIlsp/111/..c. (X) - f 2, (x)
=2(
Now suppose that with the models of interest, the confidence level of model outputs is always high so that the highest probability estimate always near I, i.e., for any model 0 of interest, we have 1-max; (ar, [] < for positive constant such that e R 0 on any unseen data point x. Let c (0, x) be the classification decision of 0 on data point x as in Section 2 c(0,x) x)) = arg max; f(0, x [i]. Then it is easy to show that we have
S =1) x)), ~ I[000,)) x) # (000xx))
(8)
Disagreement measured for shared training data can be expressed as
Ez, 10 Ex1[ x(O, + x)]
Comparing (9) with the definition of CT,P above and considering (8) it is clear that under this high- confidence condition, disagreement (9) and 1 1-norm inconsistency CL.P should be highly correlated; therefore, under this condition, Proposition D.I suggests the relation of disagreement (measured for shared training data) to generalization gap indirectly through C1,P.
While this paper focused on the KL-divergence-basel inconsistency motivated by the use of the KL-divergence by the existing algorithm for encouraging consistency, the proposition above suggests that l-norm-based inconsistency might also be useful. We have conducted limited experiments in this regard and observed mixed results. In the settings of Appendix [D.] the correlation scores of 1-norm inconsistency with respect to generalization gap are generally either similar or slightly better, which promising. As for consistency encouragement during training, we have not seen a clear advantage of using 1-norm inconsistency penalty over using the KL-divergence inconsistency penalty is done in this paper, and more experiments would be required to understand its advantage/disauvaataag
In our view, however, for the purpose of encouraging consistency during training. KL-divergence inconsistency studied in this paper is more desirable than 1-norm inconsistency in least three ways. First, minimization of the KL-divergence inconsistency penalty is equivalent to minimization of the standard cross-entropy loss with soft labels provided by the other model; therefore, with the KL-divergence penalty, the training objective can be regarded cross-entropy loss terms, which are in the same range (while 1-norm inconsistency is not). This makes tuning of the weight for the penalty more intuitive and easier. Second, optimization of the standard cross- cross-entropy loss term with the KL-divergence inconsistency penalty has an interpretation of functional gradient optimization, as shown in [19]. The last (but not least) point is that optimization may be casier with KL-divergence inconsistency, which is smooth, than with 1-norm inconsistency, which is not smooth.
weighted average of two
Related the last point, disagreement, which involves arg max in the definition, cannot be easily integrated into the training objective, and this is crucial difference between inconsistency and disagreement from the algorithmic viewpoint.
Finally, we believe that for improving deep neural network training, it is useful to study the con- nection between generalization and discrepancies of model outputs general including instability, inconsistency, and disagreement, and we hope that this work contributes to progress in this direction.
27


Reviews:
[{"review_id": "eVeXgIQebn", "reviewer": "Reviewer_ARdH", "summary": "In this work, the authors introduce the ideas of “instability” and “inconsistency” of model outputs, and investigate the relationship between these quantities and the generalization gap. In particular, they empirically find a positive correlation between instability + inconsistency and the generalization gap. They further show that inconsistency can be more predictive of the generalization gap than m-flatness. ", "questions": "It would be helpful to include legends in each figure (e.g., figure 4 is missing a legend. I assume the legend is consistent with figure 2. However, it would be convenient to include the legend again.)\n\nWhat are the training errors for models in each experiment? \n", "limitations": "The authors have described the limitations.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is written cleary, well–organized, and well-motivated. The authors do a good job in providing some intuition behind the mathematical definitions of inconsistency and instability. \n2. The paper includes extensive experiments to support their hypothesis (e.g., use of various architectures, vision and text datasets, training methods, etc.)\n", "weaknesses": "1. In Figure 2, the authors observe that $\\mathcal{D_P}$'s predictive ability of generalization gap is sensitive to the learning rate. The authors further suggest that when the “final randomness is high,”  $\\mathcal{D_P}$’s predictive ability of the generalization gap is not as strong. In practice, practitioners may use a large learning rate and small batch size to obtain well-generalizing models (and so the final randomness would be high in this situation). Thus, $\\mathcal{D_P}$ may not be useful here. (In some sense, it is not surprising that low final randomness correlates with $\\mathcal{D}_{P}$’s predictive ability of generalization gap).\n2. The authors choose a constant learning rate schedule for the starting experiments to avoid confounding variables. However, the architectures used in the experiments have normalization layers, which can induce learning rate schedules. Perhaps running the preliminary experiments with at least one architecture w/out any normalization layers would be beneficial.\n3. The benefit of $\\mathcal{D}_{P}$ over prior metrics such as disagreement is not evident. In [17], test error is calculated at the end of training (when the train error is nearly zero). Since this is not true in figure 10, I would be interested in a plot of the gap between train accuracy and test accuracy  vs. training loss instead of figure 10d. (aside: the gap between train accuracy and test accuracy is an alternate definition of generalization gap to the definition used in this work). Perhaps this would lead to a more appropriate comparison of inconsistency vs. disagreement detailed in lines 113-127. \nMore generally, it would be nice to also include plots for this alternative definition of generalization gap, especially since experiment performance in the paper is often measured via test error.\n4. In the appendix (lines 600-604), $\\rho$ was set based on reference or the development data. However, the optimal value of $\\rho$ can be quite sensitive to the architecture used. Thus, it would be nice to use some type of grid search to choose $\\rho$ for fair comparison in e.g., figure 8.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> In practice, practitioners may use a large learning rate and small batch size to obtain well-generalizing models (and so the final randomness would be high in this situation). Thus, ${\\mathcal D_P}$ may not be useful here.\n\nEven when the initial learning rate is large, the final randomness can be reduced by either decaying the learning rate (e.g., by multiplying 0.1 a few times, using the cosine schedule, etc.) or performing iterate averaging, and both are commonly practiced.  Also, Section 3.2 experiments with known high-performing training procedures and shows that inconsistency is predictive of generalization gap.  And so we believe that inconsistency can be useful in practical settings.  \n\n> The authors choose a constant learning rate schedule for the starting experiments to avoid confounding variables. However, the architectures used in the experiments have normalization layers, which can induce learning rate schedules. Perhaps running the preliminary experiments with at least one architecture w/out any normalization layers would be beneficial.\n\nThanks for the suggestion. In our preliminary experiments, we tried some normalization-free network architectures and observed that inconsistency is predictive of generalization gap.  We designed the main experiments without them as their training was too expensive.  However, we will give them another try.\n\n> I would be interested in a plot of the gap between train accuracy and test accuracy vs. training loss instead of figure 10d.\n\nThe suggested plot turned out to be similar to 10a (the gap between training loss and test loss vs. training loss) in this particular setting of Fig 10, but it may be useful in some other settings.  \n\n> More generally, it would be nice to also include plots for this alternative definition of generalization gap, especially since experiment performance in the paper is often measured via test error.\n\nWe conducted additional correlation analyses, which include this alternative definition of gap (test error minus training error), and showed some results in the one-page pdf attached to the global response.  We hope that the analyses of this form, when added to the paper, will clarify the interesting difference from (and the merit over) disagreement.  \n\n> In the appendix (lines 600-604), $\\rho$ was set based on reference or the development data. However, the optimal value of $\\rho$ can be quite sensitive to the architecture used. Thus, it would be nice to use some type of grid search to choose for fair comparison in e.g., figure 8. \n\nLines 600-604 describes selection of $\\rho$ for training SAM models, and yes, $\\rho$ was set to the optimal value for each combination of the architecture and data.  When the optimal value is known from the literature, we used that value but also made sure it was actually optimal by testing the grid in the neighbor.  When the optimal value was not known for the combination of the architecture and data, we performed a grid search (on a held-out subsample of training data serving as development data).  \n\nAs for measuring 1-sharpness, we tested the values on a grid in the neighbor of the $\\rho$ value used for training SAM (= the optimum $\\rho$ for SAM) and found that the prediction power of 1-sharpness is similar in this neighbor.  Therefore, for simplicity, we used this optimum $\\rho$ for SAM also for 1-sharpness for producing the results in the paper, as described in Line 624.\n\n> It would be helpful to include legends in each figure (e.g., figure 4 is missing a legend. I assume the legend is consistent with figure 2. However, it would be convenient to include the legend again.)\n\nTo save space, the legend is sometimes described in the caption, e.g., \"Same procedures and legend as in Fig 2\" in the caption of Fig 4, but we will improve it if accepted.  \n\n> What are the training errors for models in each experiment?\n\nThe training error varies and we cannot describe it for all the experiments here, but we will include it in the paper if accepted.  Essentially, it is often near zero for CIFAR-10 (easier data; tiny images with only 10 classes) and for harder data (e.g., ImageNet; much larger images with 1000 classes), it varies more. \n"}, {"review_id": "cMRsDn4p4f", "reviewer": "Reviewer_k85B", "summary": "This paper investigates the generalization gap in deep neural networks, and propose that this gap is influenced by the inconsistency and instability of model outputs, two quantities which are defined by the authors, and justified theoretically via a new information theoretic generalization bound. The authors conduct empirical studies that confirm the predictive power of inconsistency and instability on the generalization gap, and demonstrate that explicitly reducing inconsistency during training improves performance. ", "questions": "- How does the bound Theorem 2.1 compare with the large existing literature of information-theoretic generalization bounds? Perhaps the simplest of these from Xu & Raginsky (2017) is of the form $\\sqrt{2\\sigma^2 I/n}$, upon which your result seems like a small improvement at best, and only when $\\mathcal{D}_P$ is very small. More recent bounds (e.g. Steinke and Zakynthinou, 2020 using the conditional mutual information) have made significant improvements on this, and so clarification as to which regimes your result provides an improvement would be helpful. \n- Relatedly, is there any evidence that $\\mathcal{D}_p$ is the dominant term in Theorem 2.1? Previous work has noted that mutual-information based bounds can be extremely large (though they are difficult to numerically estimate), and seemingly here the term $\\mathcal{I}_P$ would dominate.\n- Could the authors clarify what is being varied in the plots in, e.g. Figures 1 and 2? Are these plots illustrating the generalization gap/$\\mathcal{D}_P$ in time, i.e. as a function of the iteration of optimization? If so, it would be useful if there was some indication of the direction of time in this plot. \n- Do the authors have any hypotheses for why the disagreement/error relationship exhibits different behavior than the generalization gap/inconsistency relationship (maybe specifically in the zero training error regime, in which the generalization gap = test error)? It seems like this may have to do with the distribution of the logits in the trained models (which is ignored by the disagreement but not by the inconsistency metric), which would be very interesting to understand.\n- Could the authors explicitly state the penalty that is added to the loss function during training to encourage consistency and explain how it is computed?\n\n**References**\nThomas Steinke and Lydia Zakynthinou, Reasoning About Generalization via Conditional Mutual Information, 2020.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is generally well-written, and includes several interesting observations. While I'm unsure of the novelty of Theorem 2.1, its form is compelling and I like the fact that $\\mathcal{D}_P$ and $\\mathcal{C}_P$ can be estimated efficiently (unlike the mutual information term $\\mathcal{I}_P$ that also appears in other information theoretic bounds). I also find the results at the end on explicitly encouraging consistency interesting and a strong contribution -- in my opinion it would be good to expand on this aspect of the paper. ", "weaknesses": "While the form of the bound in Theorem 2.1 is compelling, I have some concerns with its novelty/improvement relative to existing results (see questions below). In particular, it's unclear to me that the bound represents an improvement on existing information theoretic generalization bounds in the literature.\n\nOn the empirical side, I think a more rigorous correlation analysis of the $\\mathcal{C}_P/\\mathcal{D}_P$ metrics along the lines of prior work (e.g. Jiang et al. 2020) would help strengthen the claims of the paper, since the empirical correlation between these measures and the generalization gap seems to be one of its main contributions. I also think it would be helpful to have a more detailed comparison of the inconsistency/error relationship compares with the disagreement/error relationship observed in prior work, as these seem very closely related.\n\nOverall, while the paper is well-written and contains some interesting insights, at the current stage I think it lacks sufficient novelty/improvement on the theoretical side, and comparison to prior work on the empirical side to recommend acceptance.  \n\n**References**\nYiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, Samy Bengio, Fantastic Generalization Measures and Where to Find Them, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Regarding theoretical novelty: in our view, our analysis is quite different from Xu & Raginsky (2017) as their bound is a sub-Gaussian bound which cannot lead to a faster rate than $\\sqrt{1/n}$ as long as the noise variance is nonzero. In comparison we have a Bernstein-style bound, which is needed for a faster rate in classical statistics and quite different from a sub-Gaussian bound.  Also our bound incorporates new quantities (inconsistency and instability of model outputs) different from the noise variance. Steinke and Zakynthinou (2020), using the conditional mutual information, is orthogonal and complementary to our work. We focus on new easily-measurable quantities and derive a new Bernstein bound to utilize them, but it will be interesting to incorporate conditional mutual information into the analysis in the future.\n\n> a more rigorous correlation analysis of the metrics along the lines of prior work (e.g. Jiang et al. 2020) would help strengthen the claims of the paper\n\nAs suggested, we performed correlation analyses using the metrics from [Jiang et al. 2020] and confirmed that the results are consistent with the submission.  We included a part of the results (comparison between inconsistency and disagreement) in the one-page pdf attached to the global response.  We will include the complete results in the paper (including other settings and the inconsistency+instability results).  It would indeed improve the paper.  We appreciate the suggestion.  \n\n> I also think it would be helpful to have a more detailed comparison of the inconsistency/error relationship compares with the disagreement/error relationship observed in prior work, as these seem very closely related.\n\nWe performed this comparison using the metrics from [Jiang et al. 2020] and included part of the results in the one-page pdf (global response).  The results show that essentially inconsistency is correlated to generalization gap while disagreement is correlated to test error.  This is consistent with our submission and the previous study -- the inconsistency/gap relation is suggested by our theorem and the disagreement/error relation is suggested by the theorem of the disagreement paper (disagreement = test error if the training procedure produces well-calibrated ensembles).\n\n> what is being varied in the plots in, e.g. Figures 1 and 2?\n\nIn Figures 1--4, the learning rate and training length were varied, and the points are connected in the order of training length.  This was said somewhere in the text, but now we realize that it should be said also in the captions and there should be arrows in the graphs, as suggested.  We will improve it if accepted.  \n\n> Do the authors have any hypotheses for why the disagreement/error relationship exhibits different behavior than the generalization gap/inconsistency relationship (maybe specifically in the zero training error regime, in which the generalization gap = test error)? It seems like this may have to do with the distribution of the logits in the trained models (which is ignored by the disagreement but not by the inconsistency metric), which would be very interesting to understand.\n\nAs noted, inconsistency takes how strongly the models disagree into account while the disagreement ignores it.  Suppose that the confidence level of model outputs (on unseen data) is fixed to a high value so that the models always either strongly agree or strongly disagree.  In this situation, inconsistency and disagreement should be highly correlated.  Further assume that test error and generalization gap are highly correlated, and then the disagreement/error relationship and the inconsistency/gap relationship should be similar.  However, in the reality, even if all the models have zero training error, their confidence level on unseen data could vary, thus, the strength of agreement between models would not be binary, which means that inconsistency could use information ignored by disagreement.  This is an interesting point, and we will add a discussion to the revision.\n\n> Could the authors explicitly state the penalty that is added to the loss function during training to encourage consistency and explain how it is computed?\n\nFor encouraging consistency, two models are trained simultaneously with two distinct random sequences (for initialization, mini-batch sampling, etc.) with the inconsistency penalty term, which is the KL-divergence of the model output with respect to the model output of the other model.  With the absence of unlabeled data, Algorithm 1 (in the Appendix) was used, and with unlabeled data (Table 4), Algorithm 2, which computes the inconsistency penalty term on the unlabeled data, was used.  \n"}, {"review_id": "vrwMKpIUg8", "reviewer": "Reviewer_3GRr", "summary": "The paper presents two measures for a stochastic training algorithm: inconsistency and instability. The former measures the inconsistency (or \"disagreement\") within the random ensemble of models trained from the same training set. The latter measures the inconsistency of two ensembled predictors, each obtained from an ensemble trained from an independent training set. A theorem is presented stating that the sum of inconsistency and instability modulates the mutual information (between the training set and algorithm output) in the generalization bound of Xu/Raginsky' 2017.  Empirical investigation is performed to assess the predictiveness of inconsistency and instability for generalization gap. Algorithmic implications are also investigated. ", "questions": "1. Are the notions of inconsistency and instability related to the notion of functional-CMI (or functional MI) in the work of Harutyunyan et al, \"Information-theoretic generalization bounds for black-box learning algorithms\", NeurIPS 2021? Exploring this connection might enhance this work.\n\n2. What is the loss function used in evaluating generalization gaps in the experiments?\n\n", "limitations": "Nothing to add.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "To this reviewer, a particularly novel and interesting aspect of this work is bringing the notion of inconsistency, or \"within-training-set agreement\" into the landscape of generalization bounds. This notion is akin to the notion of \"generalization disagreement equality\" (GDE) in the work of Jiang et al 2022 (reference [17] of this paper).  Notably -- although not adequately discussed by the authors -- a sufficient condition of GDE is a notion of calibration in [17]. A potential impact of this work is extending the development of information-theoretic generalization bounds to include the calibration-alike quantities. This, I found intriguing and inspiring.\n\nThe theoretical development is light. Nonetheless interesting.\n\nAmong various empirical results, the most interesting and novel aspect to this reviewer is the observation that inconsistency is more predictive for generalization than sharpness. Algorithmic exploitation of this aspect is also interesting.", "weaknesses": "Some conclusions from the empirical study appear speculative to this reviewer. For example, the authors hypothesized that a low degree of randomness is required for inconsistency+instability to be predictive, and in other places the author attributed complex phenomena arising from experiments to the interaction with the mutual information term. It is desirable that such claims are better corroborated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Are the notions of inconsistency and instability related to the notion of functional-CMI (or functional MI) in the work of Harutyunyan et al, \"Information-theoretic generalization bounds for black-box learning algorithms\", NeurIPS 2021? Exploring this connection might enhance this work.\n\nIt seems to us that the notions of inconsistency and instability are not the same as the notion of functional-CMI (or functional MI) because the latter does not lead to a Bernstein-style bound and does not use variance information. The latter might lead to the so-called first-order bound, but we need to study it further to understand the relationship.  We will be happy to add a discussion.\n\n> What is the loss function used in evaluating generalization gaps in the experiments?\n\nIt was the cross-entropy loss.  "}, {"review_id": "GE2d3kaoa7", "reviewer": "Reviewer_TByp", "summary": "This manuscript propose new notions of inconsistency, instability, and information-theoretifc instability based on the output confidence score to estimate the generalization gap of deep neural networks. Theoretical and empirical results are presented and show that the proposed notions, especially inconsistency, are correlated well with well-trained neural networks.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Three novel notions are proposed to measure the generalization gap of neural networks;\n\n2. Extensive experiments have been conducted to verfify the good correlation of instability and inconsistency with the generalization gap of neural networks;\n\n3. The manuscript is well organized and the writing is clear;\n\n4. Emprical study show the better correlation of the inconsistency than disagreement.", "weaknesses": "1. The novelty of proposed measurements are limited:\n    - The Inconsistency and Instability are very similar to the definition of disagreement, while the former notions replace the outout from one-hot predictions to softmax confidence score.\n    -  The Instabilty of model parameter distributions ($\\mathcal{I}_P$) can be regarded as a kind of algorithm stability. Therefore, I suggest the authors to provide a discussion on the differentce or advantages w.r.t. former definition of algorithm stabiliy.\n\n2. Marginal contributions on the theoretical results (Theorem 2.1):\n    - The upper bound in Theorem 2.1 is not a uniform convergence-based generalization bound, and does not show the relation to training sample size and may be loose, which greatly undermines the significance of the theoretical results;\n    - The right hand of the given upper bound is hard to estimate due to the existence of the Instabiliyt of model parameter distributions $\\mathcal{I}_P$, although In consistency $\\mathcal{C}_P$ and Instability $\\mathcal{S}_P$ can be convenient to estimate on unlabeled data.\n\n\nBased on above weaknesses and considering the marginal contributions on the proposed notions and theoretical results, I think this work is slightly below the acceptance bar.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The Inconsistency and Instability are very similar to the definition of disagreement, while the former notions replace the outout from one-hot predictions to softmax confidence score.\n\nWe find it interesting that in spite of the similarity in the definitions, they behave quite differently, as shown in the submission as well as the one-page pdf attached to the global response.  The theoretical justifications are also quite different -- the disagreement study shows that disagreement=test error (not the loss gap like our theorem) if the training procedure produces well-calibrated ensembles.  At a high level, we believe that it would be useful to study the connection between generalization performance and discrepancies of model outputs in general (whether instability, inconsistency, disagreement, or else), and considering the clear observable differences from disagreement, we feel that it is useful to study instability/inconsistency as well as disagreement.  \n\n> The Instabilty of model parameter distributions (${\\mathcal I_P}$) can be regarded as a kind of algorithm stability. Therefore, I suggest the authors to provide a discussion on the differentce or advantages w.r.t. former definition of algorithm stabiliy.\n\nIn our view, ${\\mathcal I_P}$ is more like the covering number or entropy than algorithmic stability.  ${\\mathcal I_P}$ appears in recently popularized information theoretical analyses, which we used to quantify randomness effects of learning algorithms into our generalization bound.  We will add a discussion in our revision.    \n\n> The upper bound in Theorem 2.1 is not a uniform convergence-based generalization bound, and does not show the relation to training sample size and may be loose, which greatly undermines the significance of the theoretical results\n\nIn the theorem, $n$ is the training sample size.  \n\nInformation theoretical results using mutual information inherently hold in expectation because mutual information is defined with respect to expectation. The results can be easily translated into a uniform convergence bound if we choose a prior and use the KL divergence with respect to the prior (instead of mutual information), and we thought this was well known. We will add a discussion. \n\n> The right hand of the given upper bound is hard to estimate due to the existence of the Instabiliyt of model parameter distributions, although Inconsistency and Instability can be convenient to estimate on unlabeled data.\n\nThe estimation of ${\\mathcal I_P}$ is possible using bootstrap, though not easy as noted, and we are happy to add a discussion in this regard.  On the other hand, ${\\mathcal I_P}$ is a standard quantity in the mutual information-based analyses, and so all the previous analyses using this quantity share the same trait.  While we employ ${\\mathcal I_P}$ to deal with the stochastic nature of neural network training, we would like to emphasize that the strength of this analysis is the inclusion of new easily-measurable quantities, as noted, and that the main focus of this paper is the empirical study of these new quantities.  \n"}]

Author rebuttal:
Thank you very much for the valuable feedback.  

In response to the suggestions, we conducted additional correlation analyses using the metrics from [Jiang et al. 2020], with generalization gap (loss difference), test error, and test error minus training error.  One of the metrics from [Jiang et al. 2020] is more rigorous in the sense that it was designed to eliminate spurious correlations.  The results are consistent with the submission.  We uploaded a one-page pdf that contains a part of the results.  We will include the complete results in the paper, if accepted, which would be a great addition.  We really appreciate the helpful suggestions.  

The rest of our response is to each reviewer individually.  

Meta-review: The paper presents an elegant framework for relevant know-hows in generalization of deep learning. The notion of inconsistency seamlessly interprets the generalization power of ensemble models while the notion of instability nicely complements the notion of stability in generalization of deep learning. I am mostly excited by the elegance of the framework and its relevance to the disagreement framework that was recently developed to explain the generalization of deep nets. While there are still many concerns for this framework, both practical and theoretical, I hope that this work might be able to bring more clarity to generalization of deep nets.
Decision: Accept (poster)