Title: Assessing Generalization of SGD via Disagreement
Authors: Yiding Jiang
Vaishnavh Nagarajan
Christina Baek
J Zico Kolter
Venue: ICLR 2022
Keywords: Generalization
Deep Learning
Empirical Phenomenon
Accuracy Estimation
Stochastic Gradient Descent

Abstract: We empirically show that the test error of deep networks can be estimated by training the same architecture on the same training set but with two different runs of Stochastic Gradient Descent (SGD), and then measuring the disagreement rate between the two networks on unlabeled test data. This builds on -- and is a stronger version of -- the observation in Nakkiran&Bansal 20, which requires the runs to be on separate training sets. We further theoretically show that this peculiar phenomenon arises from the well-calibrated nature of ensembles of SGD-trained models. This finding not only provides a simple empirical measure to directly predict the test error using unlabeled test data, but also establishes a new conceptual connection between generalization and calibration.


Review discussion:
Strengths:
I think the GDE phenomenon presented in this paper is quite novel, and in my opinion, this message could be quite interesting to the DL theory / phenomena community. The core theoretical observation that calibration implies GDE does have a simple proof, admittedly, but I think the contribution is more in identifying the correct design choices to allow this theorem (e.g. consider ensemble of one-hot predictions rather than ensemble of probabilities), which I think is somewhat non-trivial.
Also, although it is still a bit opaque why calibration happens, the connection itself (between calibration and GDE) is kind of convincing and may motivate further research from both perspectives.
I also think the experiments in this paper are thorough and quite well-executed. The common variants (architectures, datasets, various notions of stochasticity, out-of-distribution performance) are sufficiently tested. Some of the more advanced questions arising from the theory (e.g. does ${\rm Dis}(h, h’)$ concentrate around its expectation, which I did wonder) are also tested via experiments.
Weaknesses:
It is a slight concern that the core observation in this paper is quite similar to Nakkiran and Bansal (2020)---Even though the authors already have a somewhat detailed comparison in the related work section, I am still left with some questions.
Specifically, I get an idea of what is the difference in the empirical results (DiffData vs. the newly proposed DiffInit and DiffOrder), but am still unclear in the theoretical / formulational aspects: Did they propose GDE as-is (in expectation over h, h’), just restricted to the “DiffData” version?
Also in the “they proved GDE specific to 1-NN models trained on different data”, did they not assume calibration but rather rely on something else to get that theorem?
Also, the main theoretical result is not end-to-end---calibration implies GDE, but no clues on when calibration happens in the first place. The authors discussed this on Page 9 and I suggest they also make sure various earlier claims reflect this limitation, when appropriate.
Update: I thank the authors for their response and updates to the paper. I am satisfied with the responses and would be glad to keep my current evaluation.
NO.
We are grateful for your encouraging feedback. We completely agree with your characterization of what aspects of our contribution are non-trivial.
It’s of course very important that we clarify our differences from N&B’20 since we build on them, so we’re glad you raised some important questions here:
***Question 1***
> Did they propose GDE as-is (in expectation over h, h’), just restricted to the `DiffData` version?
N&B’ 20 do formalize GDE in terms of the expectation for `AllDiff`. We formalize this just a bit more abstractly to include any stochasticity. **We did not mean to claim any sort of novelty in the math formulation -- we have rephrased the first para in Sec 4 to make this clearer.** Thanks for bringing this up.
***Question 2***
> “they proved GDE specific to 1-NN models trained on different data”, did they not assume calibration but rather rely on something else to get that theorem
**N&B’s 1-NN proof does not make any explicit connections to calibration. They make an assumption about the input distribution that could further imply something philosophically like calibration**. Their assumption is about how the nearest neighbors are distributed: for a random point $X$, and datasets $S_1$ and $S_2$, the following two distributions must be nearly identical:
$(X, NN_{S_1}(X)) \approx (NN_{S_2}(X), NN_{S_1}(X))$  ----> (1).
This assumption about the data points, then implies an assumption about the similarity in distribution of ground truth labels and predictions:
$(Y, \hat{Y}_1) \approx  (\hat{Y}_2, \hat{Y}_1)$ ----> (2)
[this then gives way to GDE].
Recall that calibration too, requires that the ground truth labels and the predicted labels are similar in distribution, but conditioned over the confidence level sets. i.e., we want
$(Y | \mathbb{E}[\hat{Y}_1] = q)  \approx (\hat{Y}_2 | \mathbb{E}[\hat{Y}_1] = q)$ → (3)
One could say that our calibration assumption in (3) in similar in spirit to their **implied** assumption in (2).
In other words,
- they don't make explicit connections to calibration
- it seems reasonable to say that their main assumption (1) about the (input, nearest neighbor) distribution is philosophically stronger than our calibration assumption.,
(As an aside, we have proven in our work that calibration is not a necessary condition for GDE.)
***Question 3***
> “no clues on when calibration happens in the first place”
We agree this is a limitation of our work.
While we admit it explicitly in the paper, we thoroughly agree that it’s important to be more upfront about this in the initial parts of the paper. As per your suggestion, **we have further highlighted this limitation right before the contributions and also in the third line of the contributions**. That being said, research on calibration and uncertainty estimation are rapidly growing within the deep learning literature and we hope our work can invite more attention to solving this important question.
## Strengths
The main empirical observation, that the disagreement rate is nearly equal to test error across variations in architecture and hyperparameters, is surprising to me. (I would probably have expected disagreement to be more than 1.3x lower than test error for variations in random seed.) This experimental finding is a valuable contribution to the literature in and of itself. It is, as the authors note, a tantalizing connection between generalization and calibration. And the authors' explanation of this observation by reduction to a calibration property is clever and convincing (though as they note it only explains it in expectation). The paper is well-written.
The notion of "class-aggregated calibration" introduced in this paper is natural. The proofs that I checked appear to be correct. I was glad to see an attempt at an analysis of the variance of disagreement in the form of Corollary A.1.1, weak though it is.
## Weaknesses
The correlation between disagreement rate and test error is strong for random initialization and order but not as strong as it is for freshly drawn training sets. It also appears that much of the correlation might be coming from the affect of data augmentation being turned on or off. I would have liked to see plots where the disagreement rate is averaged across more than one pair of runs—how much stronger does the correlation with test error get? This is crucial to judging how effective disagreement would be in practice as a tool for predicting test error with unlabeled data.
(The fact that there is correlation isn't surprising—it is intuitive that more error-prone classifiers will disagree more). More generally I would say that the main weakness of the paper is that it doesn't really attempt to present a strong case for the usefulness of the main observation in practice. That being said, the novelty of their empirical result and conceptual explanation are enough to make the paper worthy of acceptance in my opinion.
The proofs in the paper are quite straightforward (though somewhat long-winded because each small step is written explicitly); the theoretical innovation is in the statements, not the proofs.
## Other thoughts
As the authors mention, the study of calibration of neural network ensembles has primarily focused on their softmax probability outputs. Does softmax calibration imply anything about metrics of disagreement of softmax outputs (via, say, KL divergence)? I don't have much familiarity with the literature on calibration of neural network ensembles, so I don't know how novel this paper's discussion of that topic is.
NO.
**Concern 2:**
> correlation ... is strong for random initialization and order but not as strong as it is for freshly drawn training sets
We acknowledge that our correlation is indeed weaker than the fresh-data-based estimate. We admit this a few times in the paper, including in the introduction when we say our plots ”slightly deviate“ from N&B’20.
But perhaps we could’ve been clearer, so **we’ve added the following sentence in our introduction**: “Hence, ours is a more meaningful and practical estimator of test accuracy (albeit, a slightly less accurate estimate at that).”
It’s worth noting though that, **it’s not surprising to see stronger correlation when using fresh labeled data**. A fresh labeled set has enough information to directly and trivially compute test accuracy (like a held-out estimate!). Our **disagreement estimate uses strictly less information about test data** (by using only unlabeled data) and is able to achieve reasonably similar correlation.
**Concern 3:**
> Does much of the correlation come from data augmentation being turned on/off?
Great question.
- We agree that data augmentation is a hyperparamter that does contribute in part to higher tau/r^2 values.
- If we controlled for this hyperparameter and focused only on the other hyperparameters, the r^2 goes down but is still as large as [0.7.0.9], and tau ranges between [0.6, 0.77]-- please see Appendix C.3.
- We believe this is still noticeably high enough to imply strong correlation.
- Note that we also already had Fig 10 where for CNN we do not vary data augmentation and we observe very high values of 0.98 and 0.84.
- Having said that, we understand it’s always possible for one hyperparameter to dominate the correlation, which we expect more so at such minute variations. Disentangling the effects of each hyperparameter requires much larger scale experiments beyond the scope of our part-theoretical, part-empirical work. The qualitative point we were hoping to convey is how we see reasonably strong correlation even for minute variations.
Nevertheless, **to be transparent about this aspect, we have updated the main paper** under page 5 “observations”.
**Concern 4:**
> The fact that there is correlation isn't surprising—it is intuitive that more error-prone classifiers will disagree more
We agree a correlation is not altogether _that_ surprising.
But we'd like to mention a few minor points regarding why we think there's still a noteworthy surprise factor:
- _A priori_, we’d expect the scatter plot to be **arbitrarily** spread between x=0 line and the x=2y line.
So we think there is still some surprise in correlation.
- Nevertheless, we believe there's also an additional surprise in a **linear** correlation, and a further surprise in the proportionality constant lying in $1 \pm 0.3$.
- We also expected that two networks trained on the same data would have very low disagreement compared to `DiffData`, and as a result higher variance in the disagreement values and poorer correlation.
**Concern 5:**
>“plots where the disagreement rate is averaged across more than one pair of runs—how much stronger does the correlation with test error get?”
The variance of the disagreement is small as observed in Table 1 where we consider a 100-member ensemble.
Likewise, we have similar results for the scatter plot of distribution shift experiments. As per your request, **we've added plots in Appendix C.9**, specifically for `DiffInit` without data augmentation on ResNet18 + cifar10. **We observe only minor changes** after averaging over multiple (4) runs. We can't read much into these differences.
Overall, we believe adding more pairs would not significantly improve the correlation.
While writing the paper, we viewed this as an advantage since this suggests that we can get away with just two models for a reasonable estimate of generalization.
But we understand your point of view as well. We are optimistic that future work could find a different workaround to bring the plot closer to the “y=x” line and rank the models better.
*UPDATE*: Note that $R^2$ and $\tau$ only measure correlation, but doesn’t evaluate how close the points are w.r.t $x=y$ line. We evaluated “distance to x=y” through a different metric in C.9 and we observed that averaging over multiple runs does help (by up to a factor of 3). From a pure model selection perspective, $R^2$ and $\tau$ are sufficient, but this new metric demonstrates that using more pairs does benefit accuracy estimation.
**Question:**
> Does softmax calibration imply anything about metrics of disagreement of softmax outputs (via, say, KL divergence)?
Our proof is general enough to conclude that for softmax calibration, the expected disagreement between softmax predictions is equal to the expected softmax test error.
We are glad that you find the observations and theoretical discussion valuable, and share our excitement about the implication of our findings!
One of your biggest concerns is regarding practical implications. We agree we hadn't elaborated on this aspect in the paper initially, but we've updated the paper to address this.
In short, we believe it’s valuable to look at our contribution as a solution to in-distribution “**unsupervised accuracy estimation**” (u.a.e) i.e., estimating test performance with only unlabeled test data.
**TL;DR:**
U.a.e is **an existing practically well-motivated problem** (which we had mentioned briefly in the related work section). U.a.e is useful when we want **real-time evaluation, maintenance and debugging** (as motivated in [4]) and this does not require knowledge of test labels. We provide a very **simple** technique for in-distribution u.a.e, and ours works even with **blackbox** models where we don’t know weights/hidden representations. We think this makes our contribution practically relevant.
**Detailed answer:**
During test time, we may have access only to unlabeled data, because label annotation is costly (as motivated in [2]).
Further, as [1,3] write, due to privacy considerations, we may only have access to blackbox models trained by a different organization, using training sets with proprietary labels, and with no access to hold-out sets from those organizations, even from the same distribution.  Similar motivations have been discussed in [5,6,7,8].
In such a setting, how do we determine how well the blackbox models perform during deployment? How’d we pick the best blackbox model?
Existing approaches to u.a.e apply only to specialized settings (e.g., language models with logical constraints) or require structural assumptions (understandably, because they often also care about distribution-shift) and/or require **specialized training** approaches. **While there is also a separate line of work like [9] that predicts generalization performance using neural network weights, they do not work for blackbox models**.
Ours is a simple solution for u.a.e in the in-distribution setting: we simply measure disagreement between two **vanilla-trained** blackbox NN models.
Overall, we hope that the
- **simplicity** and
- **the applicability to blackbox models**,
**could make the case for its practicality, on top of the already-established practical motivation of unsupervised accuracy estimation**.
**We have updated the introduction and the related work with a discussion of this -- thank you for bringing this up! We think this addition helps motivate our paper better and informs the reader as to why our experimental results could be useful in practice.** We are eager to know what you think.
**Reference**
[1] Donmez, Lebanon, and Balasubramanian. Unsupervised supervised learning I: estimating classification and regression errors without labels. JMLR., 2010.
[2] Chen, Liu, Avci, Wu, Liang, and Jha. Detecting errors and estimating accuracy on unlabeled data with self-training ensembles. arXiv preprint arXiv:2106.15728, 2021.
[3] Jaffe, Nadler, and Kluger. Estimating the accuracies of multiple classifiers without labeled data. AISTATS 2015
[4] ElSahar and Galle. To annotate or not? predicting performance drop under domain shift. EMNLP 2019
[5]  Platanios, Poon,  Mitchell, and Horvitz. Estimating accuracy from unlabeled data: A probabilistic logic approach. NeurIPS 2017.
[6] Steinhardt and Liang. Unsupervised risk estimation using only conditional independence structure. NeurIPS 2016.
[7] Schelter, Rukat, and Bießmann. Learning to validate the predictions of black box classifiers on unseen data. ICMD 2020.
[8]  Chuang, Torralba, and Jegelka. Estimating generalization under distribution shifts via domain-invariant representations. ICML 2020.
[9] Jiang, Neyshabur, Mobahi, Krishnan, and Bengio. Fantastic generalization measures and where to find them. In ICLR 2020.
Thank you for the helpful response, and for making the edits to your paper.
I'm a little confused, actually, about the precise black box u.a.e. setting you're envisioning. Are you assuming the outside organization has trained the model twice using two random seeds, with their proprietary training set? (...in which case we need to trust that the two seeds are actually random.) Or that it's the evaluator's job to train the model twice? (...in which case the evaluator needs access to the proprietary training labels and to the model.)
Thanks a lot for reading our response and getting back to it.
Apologies for the confusion!
Privacy is one way to motivate why we are interested in black box models. In that particular case, we were envisioning the former, where we imagine having an agreed-upon contract with the org to release two independently trained black box models, but not the weights or the labeled training data themselves. (Hopefully, establishing this trust is not contradictory to the goal of privacy)
Having said that, there are also other reasons to want a black box approach, such as usability by non-experts and computational efficiency.
For example, Schelter et al., (see [7] above) point out the case where non-expert engineers use external services to train blackbox models. We simply require them to train twice with two random seeds. A blackbox approach may also be more efficient during test-time as we don't need to process the weights in anyway.
Nevertheless, we agree that in the example motivating the privacy considerations, it's indeed a limitation that we've to establish trust in the outside org.
But more broadly, we hope that our method is complementary to other u.a.e approaches. It has both advantages (e.g., simplicity, blackbox, no structural assumptions) and disadvantages (e.g., requires two independently trained models, in-distribution) which future work could improve upon.
Please let us know if you have any further questions!
Ah, okay, then I guess what's tripping me up is that if you trust the org, you might as well just ask them to tell you the model's accuracy on on their hold-out set.
Unless I'm still wrong about some aspect of the scenario?
In any case, I agree about the advantages and disadvantages of your method, and I consider discovering and partially explaining the emergent phenomenon sufficient in and of itself for acceptance regardless of the exact level of practicality of applications.
Agreed. Broadly, if there was a hold-out set, in-distribution u.a.e would be moot from a practical point of view.
However, we think that in cases where preserving a hold-out dataset is expensive and where one would rather use that to train the model, then our approach (and in-distribution u.a.e) would become practically relevant.
Thanks again for engaging with us during this process, and for your honest feedback!
**Strength**
The paper marks a significant advance in the understanding of the generalization of deep neural networks trained with SGD.
Specifically, a theoretical connection between generalization and calibration is provided. This opens up new opportunities and inspiring directions for the study of generalization in deep learning.  The theoretical analysis is sound and the experimental results are convincing. The paper is also beautifully written and a pleasure to read.
**Weakness**
As the authors also acknowledge, why and when SGD (or any given learning algorithm) is well-calibrated remains unclear and requires further characterization. It is also curious that the prediction disagreement of a pair of learned models from SGD appears to have low variance.
Finally to what extent the development of this paper will truly impact further understanding of deep learning is yet to be seen.
But rather than considering these as weaknesses, I tend to think of them more positively, as being inspiring and pointing to new directions. The contributions of this paper in my opinion well justify an acceptance decision.
The writing of the paper is occasionally sloppy, for example, in the proof of Theorem 4.1, the authors speak of ${\scr D_q}$ as if it is a set (rather than a distribution), in, for example,  "calculate the error on ${\scr D_q}$", "a $q$ fraction of ${\scr D_q}$" etc.
The authors are also encouraged to provide more intuition in the main body of the paper regarding the key insights in the proofs.
Although the proofs are not difficult to follow, the reader is likely to be buried in their technical details without truly appreciating the fundamental reasons for which the theorems are true.
Finally, the developed theory is not limited to the context of SGD. It might be a good idea to reflect the generality of the theoretical results in the paper title. -- a suggestion, not a weakness.
**Overall Recommendation**
Overall I recommend the acceptance of this paper without any reservation.
NO.
NONE
We are really happy to see that you view the weaknesses in these findings as opportunities!
Regarding your other concerns, we've done our best to address them. Details below:
- We sincerely apologize for the sloppy writing. **We have rewritten the proof sketch/notations to fix the issue you pointed out.**
- Also, we wholeheartedly agree that our paper would benefit from some more discussion of the fundamental intuition behind the theory. **We have elaborated on the brief intuition that we originally had after Theorem 1’s proof sketch (1 para → 2 para) and added some extra intuition for class-aggregated calibration in the appendix (due to space constraints).** Thank you for the thoughtful suggestion. We believe this makes our paper more accessible.
- Regarding the title, we are pleased to see your suggestion. Since our experiments are purely focused on SGD and deep learning, we are worried that our title might be an overclaim if we drop SGD from it. If you have an alternative suggestion or if you feel strongly about this, we’d love to hear that!
Thanks for your effort in replying to reviews and revising the paper.  I confirm that I will keep my original score. Cheers.
Strengths:
- The connection between calibration and generalization is novel. The theoretical results are sound.
- This paper shows stronger results than prior work (Nakkiran & Bansal 2020) with more practical implications-- being able to approximate the test error without a separately labeled dataset, given calibration.
- Experiments are through and sound.
- Paper is well written and easy to follow.
Weaknesses:
The data for DiffOrder and DiffInit deviates from the y=x curve more than the other two cases where a fresh dataset is used, so in practice, it’s still more accurate to use a fresh-labeled dataset.
Comments:
- On the last line on page 6 “namely ½ if x is hard, and 0 if easy”, is ½ supposed to be 1/k where k is the number of classes?
- Page 7: typo in “even though an ensembe of one or…”
--- Update ---
I have read all the reviews and I am satisfied with the authors' responses.
NO.
Thanks a lot for your time and for your encouraging feedback!
We'd like to respond to your thoughts below.
> “it’s still more accurate to use a fresh-labeled dataset”:
You’re right that our unlabeled-data-based estimate is indeed less accurate. **We believe though that our estimate is actually more practical** than the fresh-data-estimate because requiring labels on a **fresh held-out dataset is expensive in practice** [2] or even inaccessible due to privacy considerations [1, 3] as has been motivated in prior accuracy estimation work.
A labeled test dataset has enough information to (trivially and directly) compute the test accuracy while the **unlabeled dataset is strictly less informative** and naturally requires a more clever approach to estimation. We think **the loss in estimation accuracy power is a tradeoff for more practicality**, though future works may still address this issue.
Nevertheless, we completely agree and acknowledge the drop in estimation accuracy!
This is something we too admit a few times in the paper, including in the introduction when we say our plots ”slightly deviate“ from N&B’20.
But we could’ve been clearer, so **we’ve further added the following clause in our introduction**:
> Hence, ours is a more meaningful and practical estimator of test accuracy (albeit a slightly less accurate estimate at that).
Regarding other comments:
- Great catch! It should be $\frac{K-1}{K}$ and we have corrected it in the paper. It was ½ because we were thinking about binary classification.
- Fixed the typo, thanks!
**References** [also cited in the paper]:
[1] Pinar Donmez, Guy Lebanon, and Krishnakumar Balasubramanian. Unsupervised supervised learning I: estimating classification and regression errors without labels. JMLR., 2010.
[2] Jiefeng Chen, Frederick Liu, Besim Avci, Xi Wu, Yingyu Liang, and Somesh Jha. Detecting errors and estimating accuracy on unlabeled data with self-training ensembles. arXiv preprint arXiv:2106.15728, 2021.
[3] Ariel Jaffe, Boaz Nadler, and Yuval Kluger. Estimating the accuracies of multiple classifiers without labeled data. AISTATS 2015
We’d like to once again express our gratitude to the reviewers for their encouraging reviews and for their many constructive suggestions.
Below is a quick summary of how we’ve modified our paper to incorporate those suggestions:
1. We’ve added why our finding has practical relevance: it provides a simple blackbox approach to in-distribution unsupervised accuracy estimation.
2. We’ve rewritten the proof sketch and added more intuition in the main paper.
3. We have added an extra scatter plot in the appendix averaging over 4 pairs of models rather than just a single pair. We see that the plots get slightly closer to x=y line (although correlation-wise there’s not much difference). We also report correlation values individually for data-augmented and non-data-augmented models.
4. We’ve made it clearer, especially in the introduction that (a) our disagreement metric can be slightly less accurate than one that uses labeled test data (as is expected) and that (b) we don’t explain when/why calibration holds.
5. We've clarified our differences from Nakkiran and Bansal '20, especially the fact that they don't use calibration to prove their theorem for 1-nearest neigbhors.

Meta-review:
role: PC
data: comment: This article introduces an interesting variant of the work of Nakkiran & Bansal (2020). It shows empirically that the test error of deep models can be approximated from the disagreement on the unlabelled test data between two different trainings on the same data. The authors then show theoretically that a calibration property can explain such behaviour, and they report experiments showing that the relationship does exist in practical situations.  All reviewers agree on the practical and theoretical value of the article, which is very well organised and written. The ideas developed here are likely to lead to further work in the future, and they clearly deserve to be published at ICLR.

I agree with one of the reviewers that the title is somewhat misleading, as the reader expects an analysis based on SGD. The title could be shortened to "Assessing Generalization via Disagreement", and the experimental restriction to SGD could be mentioned in the abstract.

Decision: Accept (Spotlight)