SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now, suppose we were to learn the max-margin  $\hat{h}$  on the source distribution (Figure 2(b)). It is possible that the true labels are given by the worst-case boundary as depicted by  $y_{1}^{*}$  (pink), thus "flipping" the labels and causing  $\hat{h}$  to have 0 accuracy on  $\mathcal{T}$ . In this setting, a vacuous bound is correct. However, this seems rather unlikely to occur in practice--instead, recent experimental evidence [61, 34, 33] suggests that the true  $y^{*}$  will be much simpler. The maximum disagreement discrepancy here would be approximately 0.5, giving a test accuracy lower bound of 0.5--this is consistent with plausible alternative labeling functions such as  $y_{2}^{*}$  (orange). Even if  $y^{*}$  is not linear, we still expect that some linear function will induce larger discrepancy; this is precisely Assumption 3.3. Now suppose instead we learn  $\hat{h}$  as depicted in Figure 2(c). Then a simple ground truth such as  $y_{3}^{*}$  (green) is plausible, which would mean  $\hat{h}$  has 0 accuracy on  $\mathcal{T}$ . In this case,  $y_{3}^{*}$  is also a critic with disagreement discrepancy equal to 1, and so  $\mathrm{DIS}^2$  would correctly output an error upper bound of 1.
A setting where  $\mathsf{DIS}^2$  may be invalid. There is one setting where it should be clear that Assumption 3.5 is less likely to be satisfied: when the representation we are using is explicitly regularized to keep  $\max_{h' \in \mathcal{H}} \Delta(\hat{h}, h')$  small. This occurs for domain-adversarial representation learning methods such as DANN [18] and CDAN [38], which penalize the ability to discriminate between  $S$  and  $\mathcal{T}$  in feature space. Given a critic  $h'$  with large disagreement discrepancy, the discriminator  $D(x) = \mathbf{1}\{\arg \max_y \hat{h}(x)_y = \arg \max_y h'(x)_y\}$  will achieve high accuracy on this task (precisely,  $\frac{1 + \Delta(\hat{h}, h')}{2}$ ). By contrapositive, enforcing low discriminatory power means that the max discrepancy must also be small. It follows that for these methods  $\mathsf{DIS}^2$  should not be expected to hold universally, and in practice we see that this is the case (Figure 3). Nevertheless, when  $\mathsf{DIS}^2$  does overestimate accuracy, it does so by significantly less than prior methods.
4 Efficiently Maximizing the Disagreement Discrepancy
For a classifier  $\hat{h}$ , Theorem 3.6 clearly prescribes how to bound its test error: first, train a critic  $h'$  on the chosen  $\mathcal{X}$  to approximately maximize  $\Delta(\hat{h}, h')$ , then evaluate  $\epsilon_{\hat{S}}(\hat{h})$  and  $\hat{\Delta}(\hat{h}, h')$  using a holdout set. The remaining difficulty is in identifying the maximizing  $h' \in \mathcal{H}$ —that is, the one which minimizes  $\epsilon_{\mathcal{S}}(\hat{h}, h')$  and maximizes  $\epsilon_{\mathcal{T}}(\hat{h}, h')$ . We can approximately minimize  $\epsilon_{\mathcal{S}}(\hat{h}, h')$  by minimizing the sample average of the convex surrogate  $\ell_{\mathrm{logistic}} := -\frac{1}{\log|\mathcal{Y}|} \log \operatorname{softmax}(h(x))_y$  as justified by statistical learning theory. However, it is less clear how to maximize  $\epsilon_{\mathcal{T}}(\hat{h}, h')$ .
Figure 3:  $\mathbf{DIS}^2$  may be invalid when the features are regularized to violate Assumption 3.5. Domain-adversarial representation learning algorithms such as DANN [18] and CDAN [38] indirectly minimize  $\max_{h' \in \mathcal{H}} \Delta(\hat{h}, h')$ , meaning the necessary condition is less likely to be satisfied. Nevertheless, when  $\mathbf{DIS}^2$  does overestimate accuracy, it almost always does so by less than prior methods.
A few prior works suggest proxy losses for multiclass disagreement [12, 50, 23]. We observe that these losses are not theoretically justified, as they do not upper bound the 0-1 disagreement loss or otherwise do not meaningfully enforce that higher agreement causes higher loss. Furthermore, they are non-convex (or even concave) in the model logits, hindering optimization. Indeed, it is easy to identify simple settings in which minimizing these losses will result in a degenerate classifier with arbitrarily small loss but high agreement. Instead, we derive a new loss which satisfies the above desiderata and thus serves as a more principled approach to maximizing disagreement.
Definition 4.1. The disagreement logistic loss of a classifier  $h$  on a labeled sample  $(x, y)$  is defined as
$$
\ell_ {\mathrm {d i s}} (h, x, y) := \frac {1}{\log 2} \log \left(1 + \exp \left(h (x) _ {y} - \frac {1}{| \mathcal {Y} | - 1} \sum_ {\hat {y} \neq y} h (x) _ {\hat {y}}\right)\right).
$$
Fact 4.2. The disagreement logistic loss is convex in  $h(x)$  and upper bounds the 0-1 disagreement loss (i.e.,  $\mathbf{1}\{\arg \max_{\hat{y}}h(x)_{\hat{y}} = y\}$ ). For binary classification, the disagreement logistic loss is equivalent to the logistic loss with the label flipped.
We expect that  $\ell_{\mathrm{dis}}$  can serve as a useful drop-in replacement for any future algorithm which requires maximizing disagreement in a principled manner. We combine  $\ell_{\mathrm{logistic}}$  and  $\ell_{\mathrm{dis}}$  to arrive at the empirical disagreement discrepancy objective:
$$
\hat {\mathcal {L}} _ {\Delta} \left(h ^ {\prime}\right) := \frac {1}{| \hat {\mathcal {S}} |} \sum_ {x \in \hat {\mathcal {S}}} \ell_ {\text {l o g i s t i c}} \left(h ^ {\prime}, x, \hat {h} (x)\right) + \frac {1}{| \hat {\mathcal {T}} |} \sum_ {x \in \hat {\mathcal {T}}} \ell_ {\text {d i s}} \left(h ^ {\prime}, x, \hat {h} (x)\right).
$$
By construction,  $1 - \hat{\mathcal{L}}_{\Delta}(h')$  is concave and bounds  $\hat{\Delta}(\hat{h}, h')$  from below. However, note that the representations are already optimized for accuracy on  $\mathcal{S}$ , which suggests that predictions will have low entropy and that the  $1 / \log |y|$  scaling is unnecessary for balancing the two terms. We therefore drop the constant scaling factors; this often leads to higher discrepancy. In practice we optimize this objective with multiple initializations and hyperparameters and select the solution with the largest empirical discrepancy on a holdout set to ensure a conservative bound. Experimentally, we find that replacing  $\ell_{\mathrm{dis}}$  with any of the surrogate losses from [12, 50, 23] results in smaller discrepancy; we present these results in Appendix B.
Tightening the bound by optimizing over the logits. Looking at Theorem 3.6, it is clear that the value of the bound will decrease as the capacity of the hypothesis class is restricted. Since the number of features is large, one may expect that Assumption 3.5 holds even for a reduced feature set. In particular, it is well documented that deep networks optimized with stochastic gradient descent learn representations with small effective rank, often not much more than the number of classes [1, 2, 54, 29]. This suggests that the logits themselves should contain most of the features' information about  $S$  and  $\mathcal{T}$  and that using the full feature space is unnecessarily conservative. To test this, we evaluate  $\mathrm{DIS}^2$  on the full features, the logits output by  $\hat{h}$ , and various fractions of the top
principal components (PCs) of the features. We observe that using logits indeed results in tighter error bounds while still remaining valid—in contrast, using fewer top PCs also results in smaller error bounds, but at some point they become invalid (Figure C.2). The bounds we report in this work are thus evaluated on the logits of  $\hat{h}$ , except where we provide explicit comparisons in Section 5.
Identifying the ideal number of PCs via a "validity score". Even though reducing the feature dimensionality eventually results in an invalid bound, it is tempting to consider how we may identify approximately when this occurs, which could give a more accurate (though less conservative) prediction. We find that the optimization trajectory itself provides meaningful signal about this change. Specifically, Figure C.3 shows that for feature sets which are not overly restrictive, the critic very rapidly ascends to the maximum source agreement, then slowly begins overfitting. For much more restrictive feature sets (i.e., fewer PCs), the critic optimizes much more slowly, suggesting that we have reached the point where we are artificially restricting  $\mathcal{H}$  and therefore underestimating the disagreement discrepancy. We design a "validity score" which captures this phenomenon, and we observe that it is roughly linearly correlated with the tightness of the eventual bound (Figure C.4). Though the score is by no means perfect, we can evaluate  $\mathrm{DIS}^2$  with successively fewer PCs and only retain those above a certain score threshold, reducing the average prediction error while remaining reasonably conservative (Figure C.5). For further details, see Appendix C.
DA? Prediction Method	Coverage (↑)	Overest. (↓)	MAE (↓)
✘	✓	✘	✓	✘	✓
AC [25]	0.1000 ±.032	0.0333 ±.023	0.1194 ±.012	0.1123 ±.012	0.1091 ±.011	0.1091 ±.012
DoC [24]	0.1667 ±.040	0.0167 ±.0167	0.1237 ±.012	0.1096 ±.012	0.1055 ±.011	0.1083 ±.012
ATC NE [21]	0.2889 ±.048	0.1333 ±.044	0.0824 ±.009	0.0969 ±.012	0.0665 ±.007	0.0854 ±.011
COT [40]	0.2554 ±.0467	0.1667 ±.049	0.0860 ±.009	0.0948 ±.011	0.0700 ±.007	0.0808 ±.010
DIS2(Features)	1.0000	1.0000	0.0000	0.0000	0.2807 ±.009	0.1918 ±.008
DIS2(Logits)	0.9889 ±.011	0.7500 ±.058	0.0011 ±.000	0.0475 ±.007	0.1489 ±.011	0.0945 ±.010
DIS2(Logits w/o δ)	0.7556 ±.0475	0.4333 ±.065	0.0771 ±.013	0.0892 ±.011	0.0887 ±.009	0.0637 ±.008
Table 1: Comparing the  $\mathsf{DIS}^2$  bound to prior methods for predicting accuracy. DA denotes if the representations were learned via a domain-adversarial algorithm. We report what fraction of predictions correctly bound the true error (Coverage) and the average prediction error among shifts whose accuracy is overestimated (Overest.), along with overall MAE.  $\mathsf{DIS}^2$  has substantially higher coverage and lower overestimation error, though lower overall MAE. By dropping the concentration term in Theorem 3.6 we can get even better MAE—even beating the baselines on domain-adversarial representations—at some cost to coverage.
Datasets. We conduct experiments across 11 vision benchmark datasets for distribution shift on datasets that span applications in object classification, satellite imagery, and medicine. We use four BREEDs datasets: [65] Entity13, Entity30, Nonliving26, and Living17; FMoW [11] and Camelyon [4] from WILDS [35]; Officehome [69]; Visda [52, 51]; CIFAR10, CIFAR100 [36]; and Domainet [53]. Each of these datasets consists of multiple domains with different types of natural and synthetic shifts. We consider subpopulation shift and natural shifts induced due to differences in the data collection process of ImageNet, i.e., ImageNetv2 [60] and a combination of both. For CIFAR10 and CIFAR100 we evaluate natural shifts due to variations in replication studies [59] and common corruptions [27]. For all datasets, we use the same source and target domains commonly used in previous studies [22, 64]. We provide precise details about the distribution shifts considered in Appendix A. Because distribution shifts vary widely in scope, prior evaluations which focus on only one specific type of shift (e.g., corruptions) often do not convey the full story. We therefore emphasize the need for more comprehensive evaluations across many different types of shifts and training methods, as we present here.
Experimental setup and protocols. Along with source-only training with ERM, we experiment with Unsupervised Domain Adaptation (UDA) methods that aim to improve target performance with unlabeled target data (FixMatch [66], DANN [18], CDAN [38], and BN-adapt [37]). We experiment with Densenet121 [28] and Resnet18/Resnet50 [26] pretrained on ImageNet. For source-only ERM, as with other methods, we default to using strong augmentations: random horizontal flips, random crops, as well as Cutout [16] and RandAugment [13]. Unless otherwise specified, we default to full finetuning for source-only ERM and UDA methods. We use source hold-out performance to pick the best hyperparameters for the UDA methods, since we lack labeled validation data from the target distribution. For all of these methods, we fix the algorithm-specific hyperparameters to their original recommendations following the experimental protocol in [22]. For more details, see Appendix A.
Methods evaluated. We compare  $\mathrm{DIS}^2$  to four competitive baselines: Average Confidence (AC; [25]), Difference of Confidences (DoC; [24]), Average Thresholded Confidence (ATC; [21]), and Confidence Optimal Transport (COT; [40]). We give detailed descriptions of these methods in Appendix A. For all methods, we implement post-hoc calibration on validation source data with temperature scaling [25], which has been shown to improve performance. For  $\mathrm{DIS}^2$ , we report bounds evaluated both on the full features and on the logits of  $\hat{h}$  as described in Section 4. Unless specified otherwise, we set  $\delta = .01$  everywhere. We also experiment with dropping the lower order concentration term in Theorem 3.6, using only the sample average. Though this is of course no longer a conservative bound, we find it is an excellent predictor of test error and is worth including.
Metrics for evaluation. As our emphasis is on giving valid error bounds, we report the coverage, i.e. the fraction of predictions for which the true error does not exceed the predicted error. We also report the standard prediction metric, mean absolute error (MAE). Finally, we measure the conditional average overestimation: this is the MAE among predictions which overestimate the accuracy. This metric captures the idea that the most important thing is giving a valid bound—but if for some reason it is not, we'd at least like it to be as accurate as possible.
Results. Reported metrics for all methods can be found in Table 1. We aggregate results over all datasets, shifts, and training methods—we stratify only by whether the training method is domain-adversarial, as this affects the validity of Assumption 3.5. We find that  $\mathrm{DIS}^2$  achieves competitive MAE while maintaining substantially higher coverage, even for domain-adversarial features. When it does overestimate accuracy, it does so by much less, implying that it is ideal for conservative estimation even when any given error bound is not technically satisfied. Dropping the concentration term performs even better (sometimes beating the baselines), at the cost of some coverage. This suggests that efforts to better estimate the true maximum discrepancy may yield even better predictors. We also show scatter plots to visualize performance on individual distribution shifts, plotting each source-target pair as a single point. For these too we report separately the results for domain-adversarial (Figure 3) and non-domain-adversarial methods (Figure 1). To avoid clutter, these two plots do not include DoC, as it performed comparably to AC. Figure 4(a) displays additional scatter plots which allow for a direct comparison of the variants of  $\mathrm{DIS}^2$ . Finally, Figure 4(b) plots the observed violation rate (i.e. 1—coverage) of  $\mathrm{DIS}^2$  on non-domain-adversarial methods for varying  $\delta$ . We observe that it lies at or below the line  $y = x$ , meaning the probabilistic bound provided by Theorem 3.6 holds across a range of failure probabilities. Thus we see that our probabilistic bound is empirically valid all of the time—not in the sense that each individual shift's error is upper bounded, but rather that the desired violation rate is always satisfied.
Strengthening the baselines to improve coverage. Since the baselines we consider in this work prioritize predictive accuracy over conservative estimates, their coverage can possibly be improved without too much increase in error. We explore this option using LOOCV: for a desired coverage, we learn a parameter to either scale or shift a method's prediction to achieve that level of coverage on all but one of the datasets. We then evaluate the method on all shifts of the remaining dataset, and we repeat this for each dataset. Appendix D reports the results for varying coverage levels. We find that (i) the baselines do not achieve the desired coverage on the held out data, though they get somewhat close; and (ii) the adjustment causes them to suffer higher MAE than  $\mathsf{DIS}^2$ . Thus  $\mathsf{DIS}^2$  is on the Pareto frontier of MAE and coverage, and is preferable when conservative bounds are desirable. We believe identifying alternative methods of post-hoc prediction adjustment is a promising future direction.
Figure 4: (a): Scatter plots depicting  $\mathrm{Dis}^2$  estimated bound vs. true error for a variety of shifts. "w/o  $\delta$  indicates that the lower-order term of Theorem 3.6 has been dropped. (b): Observed bound violation rate vs. desired probability  $\delta$ . Observe that the true rate lies at or below  $y = x$  across a range of values.
The ability to evaluate trustworthy, non-vacuous error bounds for deep neural networks under distribution shift remains an extremely important open problem. Due to the wide variety of real-world shifts and the complexity of modern data, restrictive a priori assumptions on the distribution (i.e., before observing any data from the shift of interest) seem unlikely to be fruitful. On the other hand, prior methods which estimate accuracy using extra information—such as unlabeled test samples—often rely on opaque conditions whose likelihood of being satisfied is difficult to predict, and so they sometimes provide large underestimates of test error with no warning signs.
This work bridges this gap with a simple, intuitive condition and a new disagreement loss which together result in competitive error prediction, while simultaneously providing an (almost) guaranteed probabilistic error bound. We also study how the process of evaluating the bound (e.g., the optimization landscape) can provide even more useful signal, enabling better predictive accuracy. We expect there is potential to push further in each of these directions, hopefully extending the current accuracy-reliability Pareto frontier for test error bounds under distribution shift.
Thanks to Sam Sokota, Bingbin Liu, Yuchen Li, Yiding Jiang, Zack Lipton, Roni Rosenfeld, and Andrej Risteski for helpful comments. ER acknowledges the support of NSF via IIS-1909816, IIS-1955532, OAC-1934584. SG acknowledges Amazon Graduate Fellowship and JP Morgan AI Ph.D. Fellowship for their support.
[1] Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In International Conference on Machine Learning, pages 244-253. PMLR, 2018.
[2] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
[3] Christina Baek, Yiding Jiang, Aditi Raghunathan, and J Zico Kolter. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift. In Advances in Neural Information Processing Systems, 2022.
[4] Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging, 2018.
[5] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. In Advances in neural information processing systems, pages 6240-6249, 2017.
[60] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. DoImagenet classifiers generalize toImagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR, 2019.
[61] Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint arXiv:2202.06856, 2022.
[62] Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. An online learning approach to interpolation and extrapolation in domain generalization. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 2641-2657. PMLR, 28-30 Mar 2022.
[63] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252, 2015.
[64] Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, and Percy Liang. Extending the wilds benchmark for unsupervised adaptation. In NeurIPS Workshop on Distribution Shifts, 2021.
[65] Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. arXiv preprint arXiv:2008.04859, 2020.
[66] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33, 2020.
[67] Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2016.
[68] Antonio Torralba, Rob Fergus, and William T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(11):1958-1970, 2008.
[69] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018-5027, 2017.
[70] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.
[71] Richard Zhang. Making convolutional networks shift-invariant again. In ICML, 2019.
[72] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning. PMLR, 2019.
[73] Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P Adams, and Peter Orbanz. Non-vacuous generalization bounds at the imagenet scale: a pac-bayesian compression approach. arXiv preprint arXiv:1804.05862, 2018.
A.1 Description of Baselines
Average Thresholded Confidence (ATC). ATC first estimates a threshold  $t$  on the confidence of softmax prediction (or on negative entropy) such that the number of source labeled points that get a confidence greater than  $t$  match the fraction of correct examples, and then estimates the test error on on the target domain  $\mathcal{D}_{\mathrm{test}}$  as the expected number of target points that obtain a score less than  $t$ , i.e.,
$$
\mathrm {A T C} _ {\mathcal {D} _ {\mathrm {t e s t}}} (s) = \sum_ {i = 1} ^ {n} \mathbb {I} [ s (f (x _ {i} ^ {\prime})) <   t ],
$$
where  $t$  satisfies:  $\sum_{i=1}^{j} \mathbb{I}\left[\max_{j \in \mathcal{Y}}\left(f_{j}\left(x_{i}\right)\right)<t\right]=\sum_{i=1}^{m} \mathbb{I}\left[\arg \max_{j \in \mathcal{Y}} f_{j}\left(x_{i}\right) \neq y_{i}\right]$
Average Confidence (AC). Error is estimated as the average value of the maximum softmax confidence on the target data, i.e.,  $\mathbf{AC}_{\mathcal{D}_{\mathrm{test}}} = \sum_{i=1}^{n} \max_{j \in \mathcal{Y}} f_j(x_i')$ .