SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Elan Rosenfeld  
Machine Learning Department  
Carnegie Mellon University  
elan@cmu.edu
Saurabh Garg  
Machine Learning Department  
Carnegie Mellon University
We derive a new, (almost) guaranteed upper bound on the error of deep neural networks under distribution shift using unlabeled test data. Prior methods are either vacuous in practice or accurate on average but heavily underestimate error for a sizeable fraction of shifts. In particular, the latter only gives guarantees based on complex continuous measures such as test calibration, which cannot be identified without labels, and are therefore unreliable. Instead, our bound requires a simple, intuitive condition which is well justified by prior empirical works and holds in practice effectively  $100\%$  of the time. The bound is inspired by  $\mathcal{H}\Delta \mathcal{H}$ -divergence but is easier to evaluate and substantially tighter, consistently providing non-vacuous test error upper bounds. Estimating the bound requires optimizing one multiclass classifier to disagree with another, for which some prior works have used sub-optimal proxy losses; we devise a "disagreement loss" which is theoretically justified and performs better in practice. We expect this loss can serve as a drop-in replacement for future methods which require maximizing multiclass disagreement. Across a wide range of natural and synthetic distribution shift benchmarks, our method gives valid error bounds while achieving average accuracy comparable to—though not better than—competitive estimation baselines.
When deploying a model, it is important to be confident in how it will perform under inevitable distribution shift. Standard methods for achieving this include data dependent uniform convergence bounds [41, 6] (typically vacuous in practice) or assuming a precise model of how the distribution can shift [57, 10]. Unfortunately, it is difficult or impossible to determine how severely these assumptions are violated by real data [62], so practitioners usually cannot trust such bounds with confidence.
To better estimate test performance in the wild, some recent work instead tries to directly predict accuracy of neural networks using unlabeled data from the test distribution of interest [21, 3, 40]. While these methods predict the test performance surprisingly well, they lack pointwise trustworthiness and verifiability: their estimates are good on average over all distribution shifts, but they provide no signal of the quality of any individual prediction (here, each point is a single test distribution, for which a method predicts a classifier's average accuracy). Because of the opaque conditions under which these methods work, it is also difficult to anticipate their failure cases—indeed, it is reasonably common for them to substantially overestimate test accuracy for a particular shift, which is problematic when optimistic deployment would be very costly. Worse yet, we find that this gap grows with test error (Figure 1), making these predictions least reliable precisely when their reliability is most important. Although it is clearly impossible to guarantee upper bounds on test error for all shifts, there is still potential for error bounds that are intuitive and reasonably trustworthy.
Figure 1: Our bound vs. three prior methods for estimation across a wide variety of distribution shift benchmarks (e.g., WILDs, BREEDs, DomainNet) and training methods (e.g., ERM, FixMatch, BN-adapt). Prior methods are accurate on average, but it is difficult or impossible to know when a given prediction is reliable and why. Worse yet, they usually overestimate accuracy, with the gap growing as test accuracy decreases—this is precisely when a reliable, conservative estimate is most desirable. Instead,  $\mathrm{DIS}^2$  maximizes the disagreement discrepancy to give a reliable error upper bound which holds effectively  $100\%$  of the time. See Appendix F for stratification by training method.
In this work, we develop a method for (almost) provably bounding test error of classifiers under distribution shift using unlabeled test points. Our bound's only requirement is a simple, intuitive, condition which describes the ability of a hypothesis class to achieve small loss on a particular objective defined over the (unlabeled) train and test distributions. Inspired by  $\mathcal{H}\Delta\mathcal{H}$ -divergence [41, 7], our method requires training a critic to maximize agreement with the classifier of interest on the source distribution while simultaneously maximizing disagreement on the target distribution; we refer to this joint objective as the Disagreement Discrepancy, and so we name the method  $\mathrm{DIS}^2$ . We optimize this discrepancy over linear classifiers using deep features—or linear functions thereof—finetuned on only the training set. Recent evidence suggests that such representations are sufficient for highly expressive classifiers even under large distribution shift [61]. Experimentally, we find that our bound is valid effectively  $100\%$  of the time, $^1$  consistently giving non-trivial lower bounds on test accuracy which are reasonably comparable to competitive baselines.
Additionally, our proof of the bound leads to a natural (post-hoc) hypothesis test of the validity of its lone assumption. This provides an unusually strong positive signal: for more than half the datasets we evaluate we prove with high probability that the assumption holds; the corresponding indicator that it does not hold never occurs. We also show that it is possible to approximately test this bound's likelihood of being valid a priori using only unlabeled data: the optimization process itself provides useful information about the bound's validity, and we use this to construct a score which linearly correlates with the tightness of the bound. This score can then be used to relax the original bound into a sequence of successively tighter-yet-less-conservative estimates, interpolating between robustness and accuracy and allowing a user to make estimates according to their specific risk tolerance.
While maximizing agreement is statistically well understood, our method also calls for maximizing disagreement on the target distribution. This is not so straightforward in the multiclass setting, and we observe that prior works use unsuitable losses which do not correspond to minimizing the 0-1 loss of interest and are non-convex (or even concave) in the model logits [12, 50, 23]. To rectify this, we derive a new "disagreement loss" which serves as an effective proxy loss for maximizing multiclass disagreement. Experimentally, we find that minimizing this loss results in lower risk (that is, higher disagreement) compared to prior methods, and we believe it can serve as a useful drop-in replacement for any future methods which require maximizing multiclass disagreement.
Experiments across numerous vision datasets (BREEDs [65], FMoW-WILDs [35], Visda [51], Domainnet [53], CIFAR10, CIFAR100 [36] and OfficeHome [69]) demonstrate the effectiveness of our bound. Though  $\mathrm{DIS}^2$  is competitive with prior methods for error estimation, we emphasize that our focus is not on improving raw predictive accuracy—rather, we hope to obtain reliable (i.e.,
<sup>1</sup>The few violations are expected a priori, have an obvious explanation, and only occur for a specific type of learned representation. We defer a more detailed discussion of this until after we present the bound.
correct), reasonably tight bounds on the test error of a given classifier under distribution shift. In particular, while existing methods tend to severely overestimate accuracy as the true accuracy drops, our bound maintains its validity while remaining non-vacuous, even for drops in accuracy as large as  $70\%$ . In addition to source-only training, we experiment with unsupervised domain adaptation methods that use unlabeled target data and show that our observations continue to hold.
Estimating test error with unlabeled data. The generalization capabilities of overparameterized models on in-distribution data have been extensively studied using conventional machine learning tools [46, 47, 45, 48, 17, 5, 73, 39, 42]. This research aims to bound the generalization gap by evaluating complexity measures of the trained model. However, these bounds tend to be numerically loose compared to actual generalization error [70, 43]. Another line of work instead explores the use of unlabeled data for predicting in-distribution generalization [56, 55, 20, 44, 32]. More relevant to our work, there are several methods that predict the error of a classifier under distribution shift with unlabeled test data: (i) methods that explicitly predict the correctness of the model on individual unlabeled points [14, 15, 8]; and (ii) methods that directly estimate the overall error without making a pointwise prediction [9, 24, 12, 21, 3].
To achieve a consistent estimate of the target accuracy, several works require calibration on the target domain [32, 24]. However, these methods often yield poor estimates because deep models trained and calibrated on a source domain are not typically calibrated on previously unseen domains [49]. Additionally, [14, 24] require a subset of labeled target domains to learn a regression function that predicts model performance—but thus requires significant a priori knowledge about the nature of shift that, in practice, might not be available before models are deployed in the wild.
Closest to our work is [12], where the authors use domain-invariant predictors as a proxy for unknown target labels. However, like other works, their method only estimates the target accuracy—the actual error bounds they derive are not computable in practice. Second, even their estimate is computationally demanding and relies on multiple approximations, tuning of numerous hyperparameters, e.g. lagrangian multipliers; as a result, proper tuning is difficult and the method does not scale to modern deep networks. Finally, they suggest minimizing the (concave) negative cross-entropy loss, but we show that this can be a poor proxy for maximizing disagreement, instead proposing a more suitable replacement which we find performs much better.
Uniform convergence bounds. Our bound is inspired by classic analyses using  $\mathcal{H}$ - and  $\mathcal{H}\Delta \mathcal{H}$ -divergence [41, 6, 7]. These provide error bounds via a complexity measure that is both data- and hypothesis-class-dependent. This motivated a long line of work on training classifiers with small corresponding complexity, such as restricting classifiers' discriminative power between source and target data [18, 67, 38, 72]. Unfortunately, such bounds are often intractable to evaluate and are usually vacuous in real world settings. We provide a more detailed comparison to our approach in Section 3.1.
3 Deriving an (Almost) Provable Error Bound
Notation. Let  $\mathcal{S}, \mathcal{T}$  denote the source and target (train and test) distributions, respectively, over labeled inputs  $(x,y) \in \mathcal{X} \times \mathcal{Y}$ , and let  $\hat{\mathcal{S}}, \hat{\mathcal{T}}$  denote sets of samples from them with cardinalities  $n_S$  and  $n_T$  (they also denote the corresponding empirical distributions). Recall that we observe only the covariates  $x$  without the label  $y$  when a sample is drawn from  $\mathcal{T}$ . We consider classifiers  $h: \mathcal{X} \to \mathbb{R}^{|\mathcal{Y}|}$  which output a vector of logits, and we let  $\hat{h}$  denote the particular classifier whose error we aim to bound. Generally, we use  $\mathcal{H}$  to denote a hypothesis class of such classifiers. Where clear from context, we use  $h(x)$  to refer to the argmax logit, i.e. the predicted class. We treat these classifiers as deterministic throughout, though our analysis can easily be extended to probabilistic classifiers and labels. For a distribution  $\mathcal{D}$  on  $\mathcal{X} \times \mathcal{Y}$ , let  $\epsilon_{\mathcal{D}}(h,h') := \mathbb{E}_{\mathcal{D}}[\mathbf{1}\{\arg \max_y h(x)_y \neq \arg \max_y h'(x)_y\}]$  denote the one-hot disagreement between classifiers  $h$  and  $h'$  on  $\mathcal{D}$ . Let  $y^*$  represent the true labeling function such that  $y^*(x) = y$  for all samples  $(x,y)$ ; with some abuse of notation, we write  $\epsilon_{\mathcal{D}}(h)$  to mean  $\epsilon_{\mathcal{D}}(h,y^*)$ , i.e. the 0-1 error of classifier  $h$  on distribution  $\mathcal{D}$ .
The bound we derive in this work is extremely simple and relies on one new concept:
Definition 3.1. The disagreement discrepancy  $\Delta(h, h')$  is the disagreement between  $h$  and  $h'$  on  $\mathcal{T}$  minus their disagreement on  $\mathcal{S}$ :
$$
\Delta (h, h ^ {\prime}) := \epsilon_ {\mathcal {T}} (h, h ^ {\prime}) - \epsilon_ {\mathcal {S}} (h, h ^ {\prime}).
$$
We leave the dependence on  $S, T$  implicit. Note that this term is symmetric in its arguments and signed—it can be negative. With this definition, we now have the following lemma:
Lemma 3.2. For any classifier  $h$ ,  $\epsilon_{\mathcal{T}}(h) = \epsilon_{\mathcal{S}}(h) + \Delta (h,y^{*})$
Proof. By definition,  $\epsilon_{\mathcal{T}}(h) = \epsilon_{\mathcal{S}}(h) + (\epsilon_{\mathcal{T}}(h) - \epsilon_{\mathcal{S}}(h)) = \epsilon_{\mathcal{S}}(h) + \Delta (h,y^{*})$
We cannot directly use Lemma 3.2 to estimate  $\epsilon_{\mathcal{T}}(\hat{h})$  because the second term is unknown. However, observe that  $y^{*}$  is fixed. That is, while a learned  $\hat{h}$  will depend on  $y^{*}$  and  $\mathcal{S}$  and therefore  $\Delta (\hat{h},y^{*})$  may be large under large distribution shift— $y^{*}$  is not chosen to maximize  $\Delta (\hat{h},y^{*})$  in response to the  $\hat{h}$  we have learned. This means that for a sufficiently expressive hypothesis class  $\mathcal{H}$ , it should be possible to identify an alternative labeling function  $h^{\prime}\in \mathcal{H}$  for which  $\Delta (\hat{h},h^{\prime})\geq \Delta (\hat{h},y^{*})$  (we refer to such  $h^\prime$  as the critic). In other words, we should be able to find an  $h^\prime \in \mathcal{H}$  which, if it were the true labeling function, would imply at least as large of a drop in accuracy from train to test as occurs in reality. This key observation serves as the basis for our bound, and we discuss it in greater detail in Section 3.1.
In this work we consider the class  $\mathcal{H}$  of linear critics, with the features  $\mathcal{X}$  defined as source-finetuned neural representations or the logits output by the classifier  $\hat{h}$ . Prior work provides strong evidence that this class has surprising capacity under distribution shift, including the possibility that functions very similar to  $y^{*}$  lie in  $\mathcal{H}$  [61, 34, 33]. We formalize this intuition with the following assumption:
Assumption 3.3. Define  $h^* \coloneqq \arg \max_{h' \in \mathcal{H}} \Delta(\hat{h}, h')$ . We assume
$$
\Delta (\hat {h}, y ^ {*}) \leq \Delta (\hat {h}, h ^ {*}).
$$
Note that this statement is necessarily true whenever  $y^{*} \in \mathcal{H}$ ; it only becomes meaningful when considering restricted  $\mathcal{H}$ , as we do here. Note also that this assumption is made specifically for  $\hat{h}$ , i.e., on a per-classifier basis. This is important because while the above may not hold for every classifier  $\hat{h}$ , it need only hold for the classifiers whose error we would hope to bound, which is in practice a very small subset of classifiers (such as those which can be found by approximately minimizing the empirical training risk via SGD). From Lemma 3.2, we immediately have the following result:
Proposition 3.4. Under Assumption 3.3,  $\epsilon_{\mathcal{T}}(\hat{h}) \leq \epsilon_{\mathcal{S}}(\hat{h}) + \Delta(\hat{h}, h^*)$ .
Unfortunately, identifying the optimal critic  $h^*$  is intractable, meaning this bound is still not estimable—we present it as an intermediate result for clarity of presentation. To derive the practical bound we report in our experiments, we need one additional step. In Section 4, we derive a "disagreement loss" which we use to approximately maximize the empirical disagreement discrepancy  $\hat{\Delta}(\hat{h}, \cdot) = \epsilon_{\hat{T}}(\hat{h}, \cdot) - \epsilon_{\hat{S}}(\hat{h}, \cdot)$ . Relying on this loss, we instead make the assumption:
Assumption 3.5. Suppose we identify the critic  $h' \in \mathcal{H}$  which maximizes a concave surrogate to the empirical disagreement discrepancy. We assume  $\Delta(\hat{h}, y^*) \leq \Delta(\hat{h}, h')$ .
This assumption is slightly stronger than Assumption 3.3—in particular, Assumption 3.3 implies with high probability a weaker version of Assumption 3.5 with additional terms that decrease with increasing sample size and a tighter proxy loss. $^2$  Thus, the difference in strength between these two assumptions shrinks as the number of available samples grows and as the quality of our surrogate objective improves. Ultimately, our bound holds without these terms, implying that the stronger assumption is reasonable in practice. We can now present our main result:
Theorem 3.6 (Main Bound). Under Assumption 3.5, with probability  $\geq 1 - \delta$
$$
\epsilon_ {\mathcal {T}} (\hat {h}) \leq \epsilon_ {\hat {\mathcal {S}}} (\hat {h}) + \hat {\Delta} (\hat {h}, h ^ {\prime}) + \sqrt {\frac {(n _ {S} + 4 n _ {T}) \log^ {1 / \delta}}{2 n _ {S} n _ {T}}}.
$$
$^2$ Roughly, Assumption 3.3 implies  $\Delta (\hat{h},y^*)\leq \Delta (\hat{h},h') + \mathcal{O}\left(\sqrt{\frac{\log 1 / \delta}{\min(n_S,n_T)}}\right) + \gamma$ , where  $\gamma$  is a data-dependent measure of how tightly the surrogate loss bounds the 0-1 loss in expectation.
Proof. Assumption 3.5 implies  $\epsilon_{\mathcal{T}}(\hat{h}) \leq \epsilon_{\mathcal{S}}(\hat{h}) + \Delta(\hat{h}, h') = \epsilon_{\mathcal{S}}(\hat{h}, y^*) + \epsilon_{\mathcal{T}}(\hat{h}, h') - \epsilon_{\mathcal{S}}(\hat{h}, h')$ , so the problem reduces to upper bounding these three terms. We define the random variables
$$
r _ {\mathcal {S}, i} = \left\{ \begin{array}{l l} 1 / _ {n _ {S}}, & h ^ {\prime} (x _ {i}) = \hat {h} (x _ {i}) \neq y _ {i}, \\ - 1 / _ {n _ {S}}, & h ^ {\prime} (x _ {i}) \neq \hat {h} (x _ {i}) = y _ {i}, \\ 0, & \text {o t h e r w i s e}, \end{array} \right. \quad r _ {\mathcal {T}, i} = \frac {\mathbf {1} \{\hat {h} (x _ {i}) \neq h ^ {\prime} (x _ {i}) \}}{n _ {T}}
$$
for source and target samples, respectively. By construction, the sum of all of these variables is precisely  $\epsilon_{\hat{S}}(\hat{h},y^{*}) + \epsilon_{\hat{T}}(\hat{h},h^{\prime}) - \epsilon_{\hat{S}}(\hat{h},h^{\prime})$  (note these are the empirical terms). Further, observe that
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {\hat {\mathcal {S}}} r _ {\mathcal {S}, i} \right] = \mathbb {E} _ {\mathcal {S}} [ \mathbf {1} \{\hat {h} (x _ {i}) \neq y _ {i} \} - \mathbf {1} \{\hat {h} (x _ {i}) \neq h ^ {\prime} (x _ {i}) \} ] = \epsilon_ {\mathcal {S}} (\hat {h}, y ^ {*}) - \epsilon_ {\mathcal {S}} (\hat {h}, h ^ {\prime}), \\ \mathbb {E} \left[ \sum_ {\hat {\tau}} r _ {\mathcal {T}, i} \right] = \mathbb {E} _ {\mathcal {T}} [ \mathbf {1} \{\hat {h} (x _ {i}) \neq h ^ {\prime} (x _ {i}) \} ] = \epsilon_ {\mathcal {T}} (\hat {h}, h ^ {\prime}), \\ \end{array}
$$
and thus their expected sum is  $\epsilon_{S}(\hat{h},y^{*}) + \epsilon_{\mathcal{T}}(\hat{h},h^{\prime}) - \epsilon_{S}(\hat{h},h^{\prime})$ , which are the population terms we hope to bound. Now we apply Hoeffding's inequality: the probability that the expectation exceeds their sum by  $t$  is no more than  $\exp \left(-\frac{2t^2}{n_S(2 / n_S)^2 + n_T(1 / n_T)^2}\right)$ . Solving for  $t$  completes the proof.
Remark 3.7. While we state Theorem 3.6 as an implication, Assumption 3.5 is equivalent to the stated bound up to finite-sample terms. Our empirical findings (and prior work) suggest that Assumption 3.5 is reasonable in general, but this equivalence allows us to actually prove that it holds in practice for many shifts. We elaborate on this in Appendix E.
The core message behind Theorem 3.6 is that if there is a simple (i.e., linear) critic  $h'$  with large disagreement discrepancy, the true  $y^*$  could plausibly be this function, implying  $\hat{h}$  could have high error—likewise, if no simple  $y^*$  could hypothetically result in high error, we should expect low error.
Remark 3.8. Bounding error under distribution shift is fundamentally impossible without assumptions. Prior works which estimate accuracy using unlabeled data rely on experiments, suggesting that whatever condition allows their method to work holds in a variety of settings [21, 3, 40, 32, 24]; using these methods is equivalent to implicitly assuming that it will hold for future shifts. Understanding these conditions is thus crucial for assessing in a given scenario whether they can be expected to be satisfied.<sup>3</sup> It is therefore of great practical value that Assumption 3.5 is a simple, intuitive requirement: below we demonstrate that this simplicity allows us to identify potential failure cases a priori.
3.1 How Does  $\mathsf{DIS}^2$  Improve over  $\mathcal{H}$  and  $\mathcal{H}\Delta \mathcal{H}$ -Divergence?
To verifiably bound a classifier's error under distribution shift, one must develop a meaningful notion of distance between distributions. One early attempt at this was  $\mathcal{H}$ -divergence [6, 41] which measures the ability of a binary hypothesis class to discriminate between  $S$  and  $\mathcal{T}$  in feature space. This was later refined to  $\mathcal{H}\Delta \mathcal{H}$ -divergence [7], which is equal to  $\mathcal{H}$ -divergence where the discriminator class comprises all exclusive-ors between pairs of functions from the original class  $\mathcal{H}$ . Though these measures can in principle provide non-vacuous bounds, they usually do not, and evaluating them is intractable because it requires maximizing an objective over all pairs of hypotheses. Furthermore, these bounds are overly conservative even for simple function classes and distribution shifts because they rely on uniform convergence. In practice, we do not care about bounding the error of all classifiers in  $\mathcal{H}$  -we only care to bound the error of  $\hat{h}$ . This is a clear advantage of  $\mathrm{DIS}^2$  over  $\mathcal{H}\Delta \mathcal{H}$ .
The true labeling function is never worst-case.4 More importantly, we observe that one should not expect the distribution shift to be truly worst case, because the test distribution  $\mathcal{T}$  and ground truth
<sup>3</sup>Whether and when to trust a black-box estimate that is consistently accurate in all observed settings is a centuries-old philosophical problem [30] which we do not address here. Regardless, Figure 1 shows that these estimates are not consistently accurate, making interpretability that much more important.
4If it were, we'd see exactly  $0\%$  test accuracy-and when does that ever happen?
(b)
Figure 2: The advantage of  $\mathsf{DIS}^2$  over bounds based on  $\mathcal{H}$  and  $\mathcal{H}\Delta \mathcal{H}$ -divergence. Consider the task of classifying circles and squares (triangles are unlabeled). (a): Because  $h_1$  and  $h_2 \oplus h_3$  perfectly discriminate between  $S$  (blue) and  $\mathcal{T}$  (red),  $\mathcal{H}$  and  $\mathcal{H}\Delta \mathcal{H}$ -divergence bounds are always vacuous. In contrast,  $\mathsf{DIS}^2$  is only vacuous when  $0\%$  accuracy is induced by a reasonably likely ground truth (such as  $y_3^*$  in (c), but not  $y_1^*$  in (b)), and can often give non-vacuous bounds (such as  $y_2^*$  in (b)).
$y^{*}$  are not chosen adversarially with respect to  $\hat{h}$ . Figure 2 gives a simple demonstration of this point. Consider the task of learning a linear classifier to discriminate between squares and circles on the source distribution  $S$  (blue) and then bounding the error of this classifier on the target distribution  $\mathcal{T}$  (red), whose true labels are unknown and are therefore depicted as triangles. Figure 2(a) demonstrates that both  $\mathcal{H}$ - and  $\mathcal{H}\Delta \mathcal{H}$ -divergence achieve their maximal value of 1, because both  $h_1$  and  $h_2 \oplus h_3$  perfectly discriminate between  $S$  and  $\mathcal{T}$ . Thus both bounds would be vacuous.