SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rie Johnson
RJ Research Consulting
riejohnson@gmail.com
tozhang@tongzhang-ml.org
As deep neural networks are highly expressive, it is important to find solutions with small generalization gap (the difference between the performance on the training data and unseen data). Focusing on the stochastic nature of training, we first present a theoretical analysis in which the bound of generalization gap depends on what we call inconsistency and instability of model outputs, which can be estimated on unlabeled data. Our empirical study based on this analysis shows that instability and inconsistency are strongly predictive of generalization gap in various settings. In particular, our finding indicates that inconsistency is a more reliable indicator of generalization gap than the sharpness of the loss landscape. Furthermore, we show that algorithmic reduction of inconsistency leads to superior performance. The results also provide a theoretical basis for existing methods such as co-distillation and ensemble.
As deep neural networks are highly expressive, the generalization gap (the difference between the performance on the training data and unseen data) can be a serious issue. There has been intensive effort to improve generalization, concerning, for example, network architectures [15, 11], the training objective [10], strong data augmentation and mixing [6, 41, 39]. In particular, there has been extensive effort to understand the connection between generalization and the sharpness of the loss landscape surrounding the model [14, 20, 16, 18, 10]. [18] conducted large-scale experiments with a number of metrics and found that the sharpness-based metrics predicted the generalization gap best. [10] has shown that the sharpness measured by the maximum loss difference around the model ( $m$ -sharpness) correlates well to the generalization gap, which justifies their proposed method sharpness-aware minimization (SAM), designed to reduce the  $m$ -sharpness.
This paper studies generalization gap from a different perspective. Noting that the standard procedure for neural network training is stochastic so that a different instance leads to a different model, we first present a theoretical analysis in which the bound of generalization gap depends on inconsistency and instability of model outputs, conceptually described as follows. Let  $P$  be a stochastic training procedure, e.g., minimization of the cross-entropy loss by stochastic gradient descent (SGD) starting from random initialization with a certain combination of hyperparameters. It can be regarded as a random function that maps a set of labeled training data as its input to model parameter as its output. We say procedure  $P$  has high inconsistency of model outputs if the predictions made by the models trained with  $P$  using the same training data are very different from one another in expectation over the underlying (but unknown) data distribution. We also say  $P$  has high instability of model outputs if different sampling of training data changes the expected predictions a lot over the underlying data distribution. Although both quantities are discrepancies of model outputs, the sources of the
*This work was done when the second author was jointly with Google Research.
discrepancies are different. The term stability comes from the related concept in the literature [4, 31]. Inconsistency is related to the disagreement metric studied in [27, 17, 22], but there are crucial differences between them, as shown later. Both inconsistency and instability of model outputs can be estimated on unlabeled data.
Our high-level goal is to find the essential property of models that generalize well; such an insight would be useful for improving training algorithms. With this goal, we empirically studied the connection of inconsistency and instability with the generalization gap. As our bound also depends on a property of model parameter distributions (for which we have theoretical insight but which we cannot easily estimate), we first experimented to find the condition under which inconsistency and instability of model outputs are highly correlated to generalization gap. The found condition - low randomness in the final state - is consistent with the theory, and it covers practically useful settings. We also found that when this condition is met, inconsistency alone is almost as predictive as inconsistency and instability combined. This is a practical advantage since estimation of instability requires multiple training sets and estimation of inconsistency does not. We thus focused on inconsistency, which enabled the use of full-size training data, and studied inconsistency in comparison with sharpness in the context of algorithmic reduction of these two quantities. We observed that while both sharpness reduction and inconsistency reduction lead to better generalization, there are a number of cases where inconsistency and generalization gap are reduced and yet sharpness remains relatively high. We view it as an indication that inconsistency has a stronger (and perhaps more essential) connection with generalization gap than sharpness.
Our contributions are as follows.
- We develop a theory that relates generalization gap to instability and inconsistency of model outputs, which can be estimated on unlabeled data.
- Empirically, we show that instability and inconsistency are strongly predictive of generalization gap in various settings.
- We show that algorithmic encouragement of consistency reduces inconsistency and improves performance, which can lead to further improvement of the state of the art performance.
- Our results provide a theoretical basis for existing methods such as co-distillation and ensemble.
The theorem below quantifies generalization gap by three quantities: inconsistency of model outputs, instability of model outputs, and information-theoretic instability of model parameter distributions.
Notation Let  $f(\theta, x)$  be the output of model  $\theta$  on data point  $x$  in the form of probability estimates (e.g., obtained by applying softmax). We use the upright bold font for probability distributions. Let  $Z = (X, Y)$  be a random variable representing a labeled data point (data point  $X$  and label  $Y$ ) with a given unknown distribution  $\mathbf{Z}$ . Let  $Z_{n} = \{(X_{i}, Y_{i}) : i = 1, \dots, n\}$  be a random variable representing iid training data of size  $n$  drawn from  $\mathbf{Z}$ . Let  $\Theta_{P|Z_{n}}$  be the distribution of model parameters resulting from applying a (typically stochastic) training procedure  $P$  to training set  $Z_{n}$ .
Inconsistency, instability, and information-theoretic instability Inconsistency of model outputs  $\mathcal{C}_P$  ('C' for consistency) of training procedure  $P$  represents the discrepancy of outputs among the models trained on the same training data:
$$
\mathcal {C} _ {P} = \mathbb {E} _ {Z _ {n}} \mathbb {E} _ {\theta , \theta^ {\prime} \sim \Theta_ {P \mid Z _ {n}}} \mathbb {E} _ {X} \mathrm {K L} (f (\Theta , X) | | f (\theta^ {\prime}, X)) \quad \text {(I n c o n s i s t e n c y o f m o d e l o u t p u t s)}
$$
The source of inconsistency could be the random initialization, sampling of mini-batches, randomized data augmentation, and so forth.
To define instability of model outputs, let  $\bar{f}_{P|Z_n}(x)$  be the expected outputs (for data point  $x$ ) of the models trained by procedure  $P$  on training data  $Z_{n}$ :  $\bar{f}_{P|Z_n}(x) = \mathbb{E}_{\Theta \sim \Theta_{P|Z_n}}f(\Theta ,x)$ . Instability of model outputs  $\mathcal{S}_P$  ( $S$  for stability) of procedure  $P$  represents how much the expected prediction  $\bar{f}_{P|Z_n}(x)$  would change with change of training data:
$$
\mathcal {S} _ {P} = \mathbb {E} _ {Z _ {n}, Z _ {n} ^ {\prime}} \mathbb {E} _ {X} \mathrm {K L} \left(\bar {f} _ {P | Z _ {n}} (X) \mid \mid \bar {f} _ {P | Z _ {n} ^ {\prime}} (X)\right) \quad (\text {I n s t a b i l i t y o f m o d e l o u t p u t s})
$$
Finally, the instability of model parameter distributions  $\mathcal{I}_P$  is the mutual information between the random variable  $\Theta_P$ , which represents the model parameters produced by procedure  $P$ , and the random variable  $Z_{n}$ , which represents training data.  $\mathcal{I}_P$  quantifies the dependency of the model parameters on the training data, which is also called information theoretic (in)stability in [38]:
$\mathcal{I}_P = I(\Theta_P;Z_n) = \mathbb{E}_{Z_n}\mathrm{KL}(\Theta_{P|Z_n}||\mathbb{E}_{Z_n'}\Theta_{P|Z_n'})$  (Instability of model parameter distributions)
The second equality follows from the well-known relation of the mutual information to the KL divergence. The rightmost expression might be more intuitive, which essentially represents how much the model parameter distribution would change with change of training data.
Note that inconsistency and instability of model outputs can be estimated on unlabeled data.  $\mathcal{I}_P$  cannot be easily estimated as it involves distributions over the entire model parameter space; however, we have theoretical insight from its definition. (More precisely, it is possible to estimate  $\mathcal{I}_P$  by sampling in theory, but practically, it is not quite possible to make a reasonably good estimate of  $\mathcal{I}_P$  for deep neural networks with a reasonably small amount of computation.)
Theorem 2.1. Using the definitions above, we consider a Lipschitz loss function  $\phi(f,y) \in [0,1]$  that satisfies  $|\phi(f,y) - \phi(f',y)| \leq \frac{\gamma}{2} \|f - f'\|_1$ , where  $f$  and  $f'$  are probability estimates and  $\gamma/2 > 0$  is the Lipschitz constant. Let  $\psi(\lambda) = \frac{e^{\lambda} - \lambda - 1}{\lambda^2}$ , which is an increasing function. Let  $\mathcal{D}_P = \mathcal{C}_P + \mathcal{S}_P$ . Let  $\Phi_{\mathbf{Z}}(\theta)$  be test loss:  $\Phi_{\mathbf{Z}}(\theta) = \mathbb{E}_{Z = (X,Y)}\phi(f(\theta,X),Y)$ . Let  $\Phi(\theta,Z_n)$  be empirical loss on  $Z_n = \{(X_i,Y_i) | i = 1,\dots,n\}$ :  $\Phi(\theta,Z_n) = \frac{1}{n}\sum_{i=1}^{n}\phi(f(\theta,X_i),Y_i)$ . Then for a given training procedure  $P$ , we have
$$
\mathbb {E} _ {Z _ {n}} \mathbb {E} _ {\Theta \sim \boldsymbol {\Theta} _ {P} | Z _ {n}} \left[ \Phi_ {\mathbf {Z}} (\Theta) - \Phi (\Theta , Z _ {n}) \right] \leq \inf _ {\lambda > 0} \left[ \gamma^ {2} \psi (\lambda) \lambda \mathcal {D} _ {P} + \frac {\mathcal {I} _ {P}}{\lambda n} \right].
$$
The theorem indicates that the upper bound of generalization gap depends on the three quantities, instability of two types and inconsistency, described above. As a sanity check, note that right after random initialization, generalization gap of the left-hand side is zero, and  $\mathcal{I}_P$  and  $S_P$  in the right-hand side are also zero, which makes the right-hand side zero as  $\lambda \to 0$ . Also note that this analysis is meant for stochastic training procedures. If  $P$  is a deterministic function of  $Z_{n}$  and not constant, the mutual information  $\mathcal{I}_P$  would become large while  $\mathcal{D}_P > 0$ , which would make the bound loose.
Intuitively, when training procedure  $P$  is more random, model parameter distributions are likely to be flatter (less concentrated) and less dependent on the sampling of training data  $Z_{n}$ , and so  $\mathcal{I}_P$  is lower. However, high randomness would raise inconsistency of the model outputs  $\mathcal{C}_P$ , which would raise  $\mathcal{D}_P$ . Thus, the theorem indicates that there should be a trade-off with respect to the randomness of the training procedure.
The style of this general bound follows from the recent information theoretical generalization analyses of stochastic machine learning algorithms [38, 30, 28] that employ  $\mathcal{I}_P$  as a complexity measure, and such results generally hold in expectation. However, unlike the previous studies, we obtained a more refined Bernstein-style generalization result. Moreover, we explicitly incorporate inconsistency and instability of model outputs into our bound and show that smaller inconsistency and smaller instability lead to a smaller generalization bound on the right-hand side. In particular, if  $\mathcal{I}_P$  is relatively small so that we have  $\mathcal{I}_P \leq n\gamma^2\mathcal{D}_P$ , then setting  $\lambda = \sqrt{\mathcal{I}_P / (n\gamma^2\mathcal{D}_P)}$  and using  $\psi(\lambda) < 1$  for  $\lambda \leq 1$ , we obtain a simpler bound
$$
\mathbb {E} _ {Z _ {n}} \mathbb {E} _ {\Theta \sim \Theta_ {P | Z _ {n}}} \left[ \Phi_ {\mathbf {Z}} (\Theta) - \Phi (\Theta , Z _ {n}) \right] \leq 2 \gamma \sqrt {\frac {\mathcal {D} _ {P} \mathcal {I} _ {P}}{n}}.
$$
Relation to disagreement It was empirically shown in [27, 17] that with models trained to zero training error, disagreement (in terms of classification decision) of identically trained models is approximately equal to test error. When measured for the models that share training data, disagreement is closely related to inconsistency  $\mathcal{C}_P$  above, and it can be expressed as  $\mathbb{E}_{Z_n}\mathbb{E}_{\Theta ,\Theta '\sim \Theta_{P|Z_n}}\mathbb{E}_X\mathbb{I}[c(\Theta ,X)\neq c(\Theta ',X)]$ , where  $c(\theta ,x)$  is classification decision  $c(\theta ,x) = \arg \max_i f(\theta ,x)[i]$  and  $\mathbb{I}$  is the indicator function  $\mathbb{I}[u] = 1$  if  $u$  is true and 0 otherwise. In spite of the similarity, in fact, the behavior of disagreement and inconsistency  $\mathcal{C}_P$  can be quite different. Disagreement is equivalent to sharpening the prediction to a one-hot vector and then taking 1-norm of the difference:  $\mathbb{I}[c(\theta ,x)\neq c(\theta ',x)] = \frac{1}{2}\| \mathrm{onehot}(f(\theta ,x)) - \mathrm{onehot}(f(\theta ',x))\| _1$ . This ultimate
sharpening makes inconsistency and disagreement very different when the confidence-level of  $f(\theta, x)$  is low. This means that disagreement and inconsistency would behave more differently on more complex data (such as ImageNet) on which it is harder to train models to a state of high confidence. In Figure 10 (Appendix) we show an example of how different the behavior of disagreement and inconsistency can be, with ResNet-50 trained on  $10\%$  of ImageNet.
3 Empirical study
Our empirical study consists of three parts. As the bound depends not only  $\mathcal{D}_P(= \mathcal{C}_P + \mathcal{S}_P)$  but also  $\mathcal{I}_P$ , we first seek the condition under which  $\mathcal{D}_P$  is predictive of generalization gap (Section 3.1). Noting that, when the found condition is met, inconsistency  $\mathcal{C}_P$  alone is as predictive of generalization gap as  $\mathcal{D}_P$ , Section 3.2 focuses on inconsistency and shows that inconsistency is predictive of generalization gap in a variety of realistic settings that use full-size training sets. Finally, Section 3.3 reports on the practical benefit of encouraging low inconsistency in algorithmic design. The details for reproduction are provided in Appendix C.
Remark: Predictiveness of  $\mathcal{D}_P$  is relative Note that we are interested in how well the change in  $\mathcal{D}_P$  matches the change in generalization gap; thus, evaluation of the relation between  $\mathcal{D}_P$  and generalization gap always involves multiple training procedures to observe the changes/differences. Given set  $S$  of training procedures, we informally say  $\mathcal{D}_P$  is predictive of generalization gap for  $S$  if the relative smallness/largeness of  $\mathcal{D}_P$  of the procedures in  $S$  essentially coincides with the relative smallness/largeness of their generalization gap.
3.1 On the predictiveness of  $\mathcal{D}_P =$  Inconsistency  $\mathcal{C}_P +$  Instability  $S_P$
Inconsistency  $\mathcal{C}_P$  and instability  $\mathcal{S}_P$  were estimated as follows. For each training procedure  $P$  (identified by a combination of hyperparameters such as the learning rate and training length), we trained  $J$  models on each of  $K$  disjoint training sets. That is,  $K \times J$  models were trained with each procedure  $P$ . The expectation values involved in  $\mathcal{C}_P$  and  $\mathcal{S}_P$  were estimated by taking the average; in particular, the expectation over data distribution  $\mathbf{Z}$  was estimated on the held-out unlabeled data disjoint from training data or test data. Disagreement was estimated similarly.  $(K, J)$  was set to (4,8) for CIFAR-10/100 and (4,4) for ImageNet, and the size of each training set was set to 4K for CIFAR-10/100 and 120K (10%) for ImageNet.
3.1.1 Results
We start with the experiments that varied the learning rate and the length of training fixing anything else to see whether the change of  $\mathcal{D}_P$  is predictive of the change of generalization gap caused by the change of these two hyperparameters. To avoid the complex effects of learning rate decay, we trained models with SGD with a constant learning rate (constant SGD in short).
Constant SGD with iterate averaging (Fig 1,3) Although constant SGD could perform poorly by itself, constant SGD with iterate averaging is known to be competitive [29, 16]. Figure 1 shows  $\mathcal{D}_P(x$ -axis) and generalization gap ( $y$ -axis) of the training procedures that performed constant SGD with iterate averaging. Iterate averaging was performed by taking the exponential moving average with momentum 0.999. Each point represents a procedure (identified by the combination of the learning rate and training length), and the procedures with the same learning rate are connected by a line in the increasing order of training length. A positive correlation is observed between  $\mathcal{D}_P$  and generalization gap on all three datasets. Figure 3 plots generalization gap (left) and  $\mathcal{D}_P$  (right) on the  $y$ -axis and training loss on the  $x$ -axis for the same procedures as in Figure 1; observe that the up and down of  $\mathcal{D}_P$  is remarkably similar to the up and down of generalization gap. Also note that on CIFAR-10/100, sometimes generalization gap first goes up and then starts coming down towards the low-training-loss end of the lines (increase of training loss in some cases is due to the effects of weight decay). This 'turning around' of generalization gap (from going up to going down) in Figure 3 shows up as the 'curling up' of the upper-right end of the lines in Figure 1. It is interesting to see that  $\mathcal{D}_P$  and generalization gap are matching at such a detailed level.
(c) ImageNet
Figure 1:  $\mathcal{D}_P$  estimates  $(x$ -axis) and generalization gap  $(y$ -axis). SGD with a constant learning rate and iterate averaging. Only the learning rate and training length were varied. A positive correlation is observed.
(a) Gap
CIFAR-10
(b)  $\mathcal{D}_P$  estimate
CIFAR-100
(c) Gap
Figure 2:  $\mathcal{D}_P$  estimates  $(x$ -axis) and generalization gap  $(y$ -axis). Constant learning rates. No iterate averaging.  $\mathcal{D}_P$  is predictive of generalization gap only for the procedures that share the learning rate.
(d)  $\mathcal{D}_P$  estimate
ImageNet
(e) Gap
(f)  $\mathcal{D}_P$  estimate
Figure 3: Up and down of generalization gap ( $y$ -axis; left) and  $\mathcal{D}_P$  ( $y$ -axis; right) as training proceeds. The  $x$ -axis is training loss. SGD with a constant learning rate with iterate averaging. The analyzed models and the legend are the same as in Fig 1. For each dataset, the left graph looks very similar to the right graph; up and down of  $\mathcal{D}_P$  as training proceeds is a good indicator of up and down of generalization gap.
(a) Inconsistency
(b) Instability
Figure 4: Inconsistency and instability  $(x$ -axis) and generalization gap  $(y$ -axis). Same procedures and legend as in Fig 2. Instability is relatively unaffected by the learning rate. CIFAR-100. Similar results on CIFAR-10 and ImageNet (Figure 11 in the Appendix).
Constant SGD without iterate averaging (Fig 2,4) While  $\mathcal{D}_P$  is clearly predictive of generalization gap in the setting above, the results in Figure 2 are more complex. This figure shows  $\mathcal{D}_P(x$ -axis) and generalization gap ( $y$ -axis) for the training procedures that performed constant SGD and did not perform iterate averaging. As before, only the learning rate and training length were varied. In Figure 2, we observe that  $\mathcal{D}_P$  is predictive only for those which share the learning rate. For fixed generalization gap,  $\mathcal{D}_P$  is larger (instead of being the same) for a larger learning rate. Inspection of inconsistency and instability reveals that larger learning rates raise inconsistency although instability is mostly unaffected (see Fig 4). This is apparently the effect of high randomness/noisiness/uncertainty of the final state. As the learning rate is constant, SGD updates
near the end of training bounce around the local minimum, and the random noise in the gradient of the last mini-batch has a substantial influence on the final model parameter. The influence of this noise is amplified by larger learning rates. On the other hand,  $\mathcal{I}_P$  is likely to be lower with higher randomness since higher randomness should make model parameter distributions flatter (less concentrated) and less dependent on the sampling of training data  $Z_{n}$ . This means that in this case (i.e., where the final randomness is high and varies a lot across the procedures), the interaction of  $\mathcal{I}_P$  and  $\mathcal{D}_P$  in the theoretical bound is complex, and  $\mathcal{D}_P$  alone could be substantially less predictive than what is indicated by the bound.
For  $\mathcal{D}_P$  to be predictive, final randomness should be equally low (Fig 5 (a)-(c)) Therefore, we hypothesized and empirically confirmed that for  $\mathcal{D}_P$  to be predictive, the degrees of final randomness/uncertainty/noisiness should be equally low, which can be achieved with either a vanishing learning rate or iterate averaging. This means that, fortunately,  $\mathcal{D}_P$  is mostly predictive for high-performing procedures that matter in practice.
Figure 5 (a)-(c) plot  $\mathcal{D}_P$  ( $x$ -axis) and generalization gap ( $y$ -axis) for the procedures that satisfy the condition. In particular, those trained on CIFAR-10 include a wide variety of procedures that differ
$x\colon \mathcal{D}_P = \mathcal{C}_P + S_P,y$  : generalization gap
(a) CIFAR-10
Figure 5: (a)-(c)  $\mathcal{D}_P$  ( $x$ -axis) and generalization gap ( $y$ -axis). (d)-(f)  $\mathcal{C}_P$  ( $x$ -axis) and generalization gap ( $y$ -axis). Both  $\mathcal{D}_P$  and  $\mathcal{C}_P$  are predictive of generalization gap for training procedures with iterate averaging or a vanishing learning rate (so that final randomness is low), irrespective of differences in the setting. In particular, the CIFAR-10 results include the training procedures that differ in network architectures, mini-batch sizes, data augmentation, weight decay parameters, learning rate schedules, learning rates and training lengths.
$x$ : Inconsistency  $\mathcal{C}_P$ ,  $y$ : generalization gap
(d) CIFAR-10
(e) CIFAR-100
(f) ImageNet
in network architectures, mini-batch sizes, presence/absence of data augmentation, learning rate schedules (constant or vanishing), weight decay parameters, in addition to learning rates and training lengths. A positive correlation between  $\mathcal{D}_P$  and generalization gap is observed on all three datasets.
Inconsistency  $\mathcal{C}_P$  vs.  $\mathcal{D}_P$  (Fig 5 (d)-(f)) Figure 5 (d)-(f) show that inconsistency  $\mathcal{C}_P$  is almost as predictive as  $\mathcal{D}_P$  when the condition of low randomness of the final states is satisfied. This is a practical advantage since unlike instability  $\mathcal{S}_P$ , inconsistency  $\mathcal{C}_P$  does not require multiple training sets for estimation.