Title: (Almost) Provable Error Bounds Under Distribution Shift via Disagreement Discrepancy
Venue: NeurIPS 2023
Accepted: yes
Keywords: accuracy estimation

error bounds

distribution shift

unsupervised domain adaptation
Abstract: We derive a new, (almost) guaranteed upper bound on the error of deep neural networks under distribution shift using unlabeled test data. Prior methods are either vacuous in practice or accurate on average but heavily underestimate error for a sizeable fraction of shifts. In particular, the latter only give guarantees based on complex continuous measures such as test calibration, which cannot be identified without labels, and are therefore unreliable. Instead, our bound requires a simple, intuitive condition which is well justified by prior empirical works and holds in practice effectively 100\% of the time. The bound is inspired by $\mathcal{H}\Delta\mathcal{H}$-divergence but is easier to evaluate and substantially tighter, consistently providing non-vacuous test error upper bounds. Estimating the bound requires optimizing one multiclass classifier to disagree with another, for which some prior works have used sub-optimal proxy losses; we devise a "disagreement loss" which is theoretically justified and performs better in practice. We expect this loss can serve as a drop-in replacement for future methods which require maximizing multiclass disagreement. Across a wide range of natural and synthetic distribution shift benchmarks, our method gives valid error bounds while achieving average accuracy comparable to—though not better than—competitive estimation baselines.

(Almost) Provable Error Bounds Under Distribution Shift via Disagreement Discrepancy

Abstract

We derive a new, (almost) guaranteed upper bound on the error of deep neural networks under distribution shift using unlabeled test data. Prior methods are either vacuous in practice or accurate on average but heavily underestimate error for a sizeable fraction of shifts. In particular, the latter only give guarantees based on complex continuous measures such as test calibration, which cannot be identified without labels, and are therefore unreliable. Instead, our bound requires a simple, intuitive condition which is well justified by prior empirical works and holds in practice effectively 100% of the time. The bound is inspired by H ∆ H -divergence but is easier to evaluate and substantially tighter, consistently providing nonvacuous test error upper bounds. Estimating the bound requires optimizing one multiclass classifier to disagree with another, for which some prior works have used sub-optimal proxy losses; we devise a 'disagreement loss' which is theoretically justified and performs better in practice. We expect this loss can serve as a drop-in replacement for future methods which require maximizing multiclass disagreement. Across a wide range of natural and synthetic distribution shift benchmarks, our method gives valid error bounds while achieving average accuracy comparable tothough not better than-competitive estimation baselines.

1 Introduction

When deploying a model, it is important to be confident in how it will perform under inevitable distribution shift. Standard methods for achieving this include data dependent uniform convergence bounds [41, 6] (typically vacuous in practice) or assuming a precise model of how the distribution can shift [57, 10]. Unfortunately, it is difficult or impossible to determine how severely these assumptions are violated by real data [62], so practitioners usually cannot trust such bounds with confidence.

To better estimate test performance in the wild, some recent work instead tries to directly predict accuracy of neural networks using unlabeled data from the test distribution of interest [21, 3, 40]. While these methods predict the test performance surprisingly well, they lack pointwise trustworthiness and verifiability: their estimates are good on average over all distribution shifts, but they provide no signal of the quality of any individual prediction (here, each point is a single test distribution , for which a method predicts a classifier's average accuracy). Because of the opaque conditions under which these methods work, it is also difficult to anticipate their failure cases-indeed, it is reasonably common for them to substantially overestimate test accuracy for a particular shift, which is problematic when optimistic deployment would be very costly. Worse yet, we find that this gap grows with test error (Figure 1), making these predictions least reliable precisely when their reliability is most important. Although it is clearly impossible to guarantee upper bounds on test error for all shifts, there is still potential for error bounds that are intuitive and reasonably trustworthy.

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

Target Accuracy

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

DIS 2

(ours)

ATC

COT

AC

y = x

In this work, we develop a method for (almost) provably bounding test error of classifiers under distribution shift using unlabeled test points. Our bound's only requirement is a simple, intuitive, condition which describes the ability of a hypothesis class to achieve small loss on a particular objective defined over the (unlabeled) train and test distributions. Inspired by H ∆ H -divergence [41, 7], our method requires training a critic to maximize agreement with the classifier of interest on the source distribution while simultaneously maximizing disagreement on the target distribution; we refer to this joint objective as the Disagreement Discrepancy , and so we name the method DIS 2 . We optimize this discrepancy over linear classifiers using deep features-or linear functions thereoffinetuned on only the training set. Recent evidence suggests that such representations are sufficient for highly expressive classifiers even under large distribution shift [61]. Experimentally, we find that our bound is valid effectively 100% of the time, 1 consistently giving non-trivial lower bounds on test accuracy which are reasonably comparable to competitive baselines.

Additionally, our proof of the bound leads to a natural (post-hoc) hypothesis test of the validity of its lone assumption. This provides an unusually strong positive signal: for more than half the datasets we evaluate we prove with high probability that the assumption holds; the corresponding indicator that it does not hold never occurs. We also show that it is possible to approximately test this bound's likelihood of being valid a priori using only unlabeled data: the optimization process itself provides useful information about the bound's validity, and we use this to construct a score which linearly correlates with the tightness of the bound. This score can then be used to relax the original bound into a sequence of successively tighter-yet-less-conservative estimates, interpolating between robustness and accuracy and allowing a user to make estimates according to their specific risk tolerance.

While maximizing agreement is statistically well understood, our method also calls for maximizing dis agreement on the target distribution. This is not so straightforward in the multiclass setting, and we observe that prior works use unsuitable losses which do not correspond to minimizing the 0-1 loss of interest and are non-convex (or even concave) in the model logits [12, 50, 23]. To rectify this, we derive a new 'disagreement loss' which serves as an effective proxy loss for maximizing multiclass disagreement. Experimentally, we find that minimizing this loss results in lower risk (that is, higher disagreement) compared to prior methods, and we believe it can serve as a useful drop-in replacement for any future methods which require maximizing multiclass disagreement.

Experiments across numerous vision datasets (BREEDs [65], FMoW-WILDs [35], Visda [51], Domainnet [53], CIFAR10, CIFAR100 [36] and OfficeHome [69]) demonstrate the effectiveness of our bound. Though DIS 2 is competitive with prior methods for error estimation, we emphasize that our focus is not on improving raw predictive accuracy -rather, we hope to obtain reliable (i.e.,

correct), reasonably tight bounds on the test error of a given classifier under distribution shift. In particular, while existing methods tend to severely overestimate accuracy as the true accuracy drops, our bound maintains its validity while remaining non-vacuous, even for drops in accuracy as large as 70% . In addition to source-only training, we experiment with unsupervised domain adaptation methods that use unlabeled target data and show that our observations continue to hold.

2 Related Work

Estimating test error with unlabeled data. The generalization capabilities of overparameterized models on in-distribution data have been extensively studied using conventional machine learning tools [46, 47, 45, 48, 17, 5, 73, 39, 42]. This research aims to bound the generalization gap by evaluating complexity measures of the trained model. However, these bounds tend to be numerically loose compared to actual generalization error [70, 43]. Another line of work instead explores the use of unlabeled data for predicting in-distribution generalization [56, 55, 20, 44, 32]. More relevant to our work, there are several methods that predict the error of a classifier under distribution shift with unlabeled test data: (i) methods that explicitly predict the correctness of the model on individual unlabeled points [14, 15, 8]; and (ii) methods that directly estimate the overall error without making a pointwise prediction [9, 24, 12, 21, 3].

To achieve a consistent estimate of the target accuracy, several works require calibration on the target domain [32, 24]. However, these methods often yield poor estimates because deep models trained and calibrated on a source domain are not typically calibrated on previously unseen domains [49]. Additionally, [14, 24] require a subset of labeled target domains to learn a regression function that predicts model performance-but thus requires significant a priori knowledge about the nature of shift that, in practice, might not be available before models are deployed in the wild.

Closest to our work is [12], where the authors use domain-invariant predictors as a proxy for unknown target labels. However, like other works, their method only estimates the target accuracy-the actual error bounds they derive are not computable in practice. Second, even their estimate is computationally demanding and relies on multiple approximations, tuning of numerous hyperparameters, e.g. lagrangian multipliers; as a result, proper tuning is difficult and the method does not scale to modern deep networks. Finally, they suggest minimizing the (concave) negative cross-entropy loss, but we show that this can be a poor proxy for maximizing disagreement, instead proposing a more suitable replacement which we find performs much better.

Uniform convergence bounds. Our bound is inspired by classic analyses using H - and H ∆ H -divergence [41, 6, 7]. These provide error bounds via a complexity measure that is both data- and hypothesis-class-dependent. This motivated a long line of work on training classifiers with small corresponding complexity, such as restricting classifiers' discriminative power between source and target data [18, 67, 38, 72]. Unfortunately, such bounds are often intractable to evaluate and are usually vacuous in real world settings. We provide a more detailed comparison to our approach in Section 3.1.

3 Deriving an (Almost) Provable Error Bound

Notation. Let S , T denote the source and target (train and test) distributions, respectively, over labeled inputs ( x, y ) ∈ X × Y , and let ˆ S , ˆ T denote sets of samples from them with cardinalities n S and n T (they also denote the corresponding empirical distributions). Recall that we observe only the covariates x without the label y when a sample is drawn from T . We consider classifiers h : X → R |Y| which output a vector of logits, and we let ˆ h denote the particular classifier whose error we aim to bound. Generally, we use H to denote a hypothesis class of such classifiers. Where clear from context, we use h ( x ) to refer to the argmax logit, i.e. the predicted class. We treat these classifiers as deterministic throughout, though our analysis can easily be extended to probabilistic classifiers and labels. For a distribution D on X ×Y , let ϵ D ( h, h ′ ) := E D [ 1 { arg max y h ( x ) y = arg max y h ′ ( x ) y } ] denote the one-hot disagreement between classifiers h and h ′ on D . Let y ∗ represent the true labeling function such that y ∗ ( x ) = y for all samples ( x, y ) ; with some abuse of notation, we write ϵ D ( h ) to mean ϵ D ( h, y ∗ ) , i.e. the 0-1 error of classifier h on distribution D .

̸

The bound we derive in this work is extremely simple and relies on one new concept:

Definition 3.1. The disagreement discrepancy ∆( h, h ′ ) is the disagreement between h and h ′ on T minus their disagreement on S :

We leave the dependence on S , T implicit. Note that this term is symmetric in its arguments and signed-it can be negative. With this definition, we now have the following lemma:

Lemma 3.2. For any classifier h , ϵ T ( h ) = ϵ S ( h ) + ∆( h, y ∗ ) .

We cannot directly use Lemma 3.2 to estimate ϵ T ( ˆ h ) because the second term is unknown. However, observe that y ∗ is fixed . That is, while a learned ˆ h will depend on y ∗ and S -and therefore ∆( ˆ h, y ∗ ) may be large under large distribution shifty ∗ is not chosen to maximize ∆( ˆ h, y ∗ ) in response to the ˆ h we have learned. This means that for a sufficiently expressive hypothesis class H , it should be possible to identify an alternative labeling function h ′ ∈ H for which ∆( ˆ h, h ′ ) ≥ ∆( ˆ h, y ∗ ) (we refer to such h ′ as the critic ). In other words, we should be able to find an h ′ ∈ H which, if it were the true labeling function , would imply at least as large of a drop in accuracy from train to test as occurs in reality. This key observation serves as the basis for our bound, and we discuss it in greater detail in Section 3.1.

In this work we consider the class H of linear critics, with the features X defined as source-finetuned neural representations or the logits output by the classifier ˆ h . Prior work provides strong evidence that this class has surprising capacity under distribution shift, including the possibility that functions very similar to y ∗ lie in H [61, 34, 33]. We formalize this intuition with the following assumption:

Assumption 3.3. Define h ∗ := arg max h ′ ∈H ∆( ˆ h, h ′ ) . We assume

Note that this statement is necessarily true whenever y ∗ ∈ H ; it only becomes meaningful when considering restricted H , as we do here. Note also that this assumption is made specifically for ˆ h , i.e. on a per-classifier basis. This is important because while the above may not hold for every classifier ˆ h , it need only hold for the classifiers whose error we would hope to bound, which is in practice a very small subset of classifiers (such as those which can be found by approximately minimizing the empirical training risk via SGD). From Lemma 3.2, we immediately have the following result:

Proposition 3.4. Under Assumption 3.3, ϵ T ( ˆ h ) ≤ ϵ S ( ˆ h ) + ∆( ˆ h, h ∗ ) .

Unfortunately, identifying the optimal critic h ∗ is intractable, meaning this bound is still not estimable-we present it as an intermediate result for clarity of presentation. To derive the practical bound we report in our experiments, we need one additional step. In Section 4, we derive a 'disagreement loss' which we use to approximately maximize the empirical disagreement discrepancy ˆ ∆( ˆ h, · ) = ϵ ˆ T ( ˆ h, · ) -ϵ ˆ S ( ˆ h, · ) . Relying on this loss, we instead make the assumption:

Assumption 3.5. Suppose we identify the critic h ′ ∈ H which maximizes a concave surrogate to the empirical disagreement discrepancy. We assume ∆( ˆ h, y ∗ ) ≤ ∆( ˆ h, h ′ ) .

This assumption is slightly stronger than Assumption 3.3-in particular, Assumption 3.3 implies with high probability a weaker version of Assumption 3.5 with additional terms that decrease with increasing sample size and a tighter proxy loss. 2 Thus, the difference in strength between these two assumptions shrinks as the number of available samples grows and as the quality of our surrogate objective improves. Ultimately, our bound holds without these terms, implying that the stronger assumption is reasonable in practice. We can now present our main result:

Theorem 3.6 (Main Bound) . Under Assumption 3.5, with probability ≥ 1 -δ ,

Proof. Assumption 3.5 implies ϵ T ( ˆ h ) ≤ ϵ S ( ˆ h ) + ∆( ˆ h, h ′ ) = ϵ S ( ˆ h, y ∗ ) + ϵ T ( ˆ h, h ′ ) -ϵ S ( ˆ h, h ′ ) , so the problem reduces to upper bounding these three terms. We define the random variables

̸

̸

̸

for source and target samples, respectively. By construction, the sum of all of these variables is precisely ϵ ˆ S ( ˆ h, y ∗ ) + ϵ ˆ T ( ˆ h, h ′ ) -ϵ ˆ S ( ˆ h, h ′ ) (note these are the empirical terms). Further, observe that

̸

̸

̸

and thus their expected sum is ϵ S ( ˆ h, y ∗ ) + ϵ T ( ˆ h, h ′ ) -ϵ S ( ˆ h, h ′ ) , which are the population terms we hope to bound. Now we apply Hoeffding's inequality: the probability that the expectation exceeds their sum by t is no more than exp ( -2 t 2 n S ( 2 / n S ) 2 + n T ( 1 / n T ) 2 ) . Solving for t completes the proof.

Remark 3.7 . While we state Theorem 3.6 as an implication, Assumption 3.5 is equivalent to the stated bound up to finite-sample terms . Our empirical findings (and prior work) suggest that Assumption 3.5 is reasonable in general, but this equivalence allows us to actually prove that it holds in practice for many shifts. We elaborate on this in Appendix E.

The core message behind Theorem 3.6 is that if there is a simple (i.e., linear) critic h ′ with large disagreement discrepancy, the true y ∗ could plausibly be this function, implying ˆ h could have high error-likewise, if no simple y ∗ could hypothetically result in high error, we should expect low error.

Remark 3.8 . Bounding error under distribution shift is fundamentally impossible without assumptions. Prior works which estimate accuracy using unlabeled data rely on experiments, suggesting that whatever condition allows their method to work holds in a variety of settings [21, 3, 40, 32, 24]; using these methods is equivalent to implicitly assuming that it will hold for future shifts. Understanding these conditions is thus crucial for assessing in a given scenario whether they can be expected to be satisfied. 3 It is therefore of great practical value that Assumption 3.5 is a simple, intuitive requirement: below we demonstrate that this simplicity allows us to identify potential failure cases a priori .

3.1 How Does DIS 2 Improve over H - and H ∆ H -Divergence?

To verifiably bound a classifier's error under distribution shift, one must develop a meaningful notion of distance between distributions. One early attempt at this was H -divergence [6, 41] which measures the ability of a binary hypothesis class to discriminate between S and T in feature space. This was later refined to H ∆ H -divergence [7], which is equal to H -divergence where the discriminator class comprises all exclusive-ors between pairs of functions from the original class H . Though these measures can in principle provide non-vacuous bounds, they usually do not, and evaluating them is intractable because it requires maximizing an objective over all pairs of hypotheses. Furthermore, these bounds are overly conservative even for simple function classes and distribution shifts because they rely on uniform convergence. In practice, we do not care about bounding the error of all classifiers in H -we only care to bound the error of ˆ h . This is a clear advantage of DIS 2 over H ∆ H .

The true labeling function is never worst-case. 4 More importantly, we observe that one should not expect the distribution shift to be truly worst case, because the test distribution T and ground truth

(a)

(b)

(c)

！

h2

hi

h3

y2

*

h

*

yi

y3

*

公

y ∗ are not chosen adversarially with respect to ˆ h . Figure 2 gives a simple demonstration of this point. Consider the task of learning a linear classifier to discriminate between squares and circles on the source distribution S (blue) and then bounding the error of this classifier on the target distribution T (red), whose true labels are unknown and are therefore depicted as triangles. Figure 2(a) demonstrates that both H - and H ∆ H -divergence achieve their maximal value of 1, because both h 1 and h 2 ⊕ h 3 perfectly discriminate between S and T . Thus both bounds would be vacuous.

Now, suppose we were to learn the max-margin ˆ h on the source distribution (Figure 2(b)). It is possible that the true labels are given by the worst-case boundary as depicted by y ∗ 1 (pink), thus 'flipping' the labels and causing ˆ h to have 0 accuracy on T . In this setting, a vacuous bound is correct. However, this seems rather unlikely to occur in practice-instead, recent experimental evidence [61, 34, 33] suggests that the true y ∗ will be much simpler. The maximum disagreement discrepancy here would be approximately 0 . 5 , giving a test accuracy lower bound of 0 . 5 -this is consistent with plausible alternative labeling functions such as y ∗ 2 (orange). Even if y ∗ is not linear, we still expect that some linear function will induce larger discrepancy; this is precisely Assumption 3.3. Now suppose instead we learn ˆ h as depicted in Figure 2(c). Then a simple ground truth such as y ∗ 3 (green) is plausible, which would mean ˆ h has 0 accuracy on T . In this case, y ∗ 3 is also a critic with disagreement discrepancy equal to 1, and so DIS 2 would correctly output an error upper bound of 1 .

A setting where DIS 2 may be invalid. There is one setting where it should be clear that Assumption 3.5 is less likely to be satisfied: when the representation we are using is explicitly regularized to keep max h ′ ∈H ∆( ˆ h, h ′ ) small. This occurs for domain-adversarial representation learning methods such as DANN [18] and CDAN [38], which penalize the ability to discriminate between S and T in feature space. Given a critic h ′ with large disagreement discrepancy, the discriminator D ( x ) = 1 { arg max y ˆ h ( x ) y = arg max y h ′ ( x ) y } will achieve high accuracy on this task (precisely, 1+∆( ˆ ′

h,h ) 2 ). By contrapositive, enforcing low discriminatory power means that the max discrepancy must also be small. It follows that for these methods DIS 2 should not be expected to hold universally, and in practice we see that this is the case (Figure 3). Nevertheless, when DIS 2 does overestimate accuracy, it does so by significantly less than prior methods.

4 Efficiently Maximizing the Disagreement Discrepancy

For a classifier ˆ h , Theorem 3.6 clearly prescribes how to bound its test error: first, train a critic h ′ on the chosen X to approximately maximize ∆( ˆ h, h ′ ) , then evaluate ϵ ˆ S ( ˆ h ) and ˆ ∆( ˆ h, h ′ ) using a holdout set. The remaining difficulty is in identifying the maximizing h ′ ∈ H -that is, the one which minimizes ϵ S ( ˆ h, h ′ ) and maximizes ϵ T ( ˆ h, h ′ ) . We can approximately minimize ϵ S ( ˆ h, h ′ ) by minimizing the sample average of the convex surrogate ℓ logistic := -1 log |Y| log softmax ( h ( x )) y as justified by statistical learning theory. However, it is less clear how to maximize ϵ T ( ˆ h, h ′ ) .

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Target Accuracy

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Dis 2

ATC

COT

AC

y = x

A few prior works suggest proxy losses for multiclass disagreement [12, 50, 23]. We observe that these losses are not theoretically justified, as they do not upper bound the 0-1 disagreement loss or otherwise do not meaningfully enforce that higher agreement causes higher loss. Furthermore, they are non-convex (or even concave) in the model logits, hindering optimization. Indeed, it is easy to identify simple settings in which minimizing these losses will result in a degenerate classifier with arbitrarily small loss but high agreement. Instead, we derive a new loss which satisfies the above desiderata and thus serves as a more principled approach to maximizing disagreement.

Definition 4.1. The disagreement logistic loss of a classifier h on a labeled sample ( x, y ) is defined as

̸

Fact 4.2. The disagreement logistic loss is convex in h ( x ) and upper bounds the 0-1 disagreement loss (i.e., 1 { arg max ˆ y h ( x ) ˆ y = y } ). For binary classification, the disagreement logistic loss is equivalent to the logistic loss with the label flipped.

We expect that ℓ dis can serve as a useful drop-in replacement for any future algorithm which requires maximizing disagreement in a principled manner. We combine ℓ logistic and ℓ dis to arrive at the empirical disagreement discrepancy objective:

By construction, 1 -ˆ L ∆ ( h ′ ) is concave and bounds ˆ ∆( ˆ h, h ′ ) from below. However, note that the representations are already optimized for accuracy on S , which suggests that predictions will have low entropy and that the 1 / log |Y| scaling is unnecessary for balancing the two terms. We therefore drop the constant scaling factors; this often leads to higher discrepancy. In practice we optimize this objective with multiple initializations and hyperparameters and select the solution with the largest empirical discrepancy on a holdout set to ensure a conservative bound. Experimentally, we find that replacing ℓ dis with any of the surrogate losses from [12, 50, 23] results in smaller discrepancy; we present these results in Appendix B.

Tightening the bound by optimizing over the logits. Looking at Theorem 3.6, it is clear that the value of the bound will decrease as the capacity of the hypothesis class is restricted. Since the number of features is large, one may expect that Assumption 3.5 holds even for a reduced feature set. In particular, it is well documented that deep networks optimized with stochastic gradient descent learn representations with small effective rank, often not much more than the number of classes [1, 2, 54, 29]. This suggests that the logits themselves should contain most of the features' information about S and T and that using the full feature space is unnecessarily conservative. To test this, we evaluate DIS 2 on the full features, the logits output by ˆ h , and various fractions of the top

principal components (PCs) of the features. We observe that using logits indeed results in tighter error bounds while still remaining valid -in contrast, using fewer top PCs also results in smaller error bounds, but at some point they become invalid (Figure C.2). The bounds we report in this work are thus evaluated on the logits of ˆ h , except where we provide explicit comparisons in Section 5.

Identifying the ideal number of PCs via a 'validity score'. Even though reducing the feature dimensionality eventually results in an invalid bound, it is tempting to consider how we may identify approximately when this occurs, which could give a more accurate (though less conservative) prediction. We find that the optimization trajectory itself provides meaningful signal about this change. Specifically, Figure C.3 shows that for feature sets which are not overly restrictive, the critic very rapidly ascends to the maximum source agreement, then slowly begins overfitting. For much more restrictive feature sets (i.e., fewer PCs), the critic optimizes much more slowly, suggesting that we have reached the point where we are artificially restricting H and therefore underestimating the disagreement discrepancy. We design a 'validity score' which captures this phenomenon, and we observe that it is roughly linearly correlated with the tightness of the eventual bound (Figure C.4). Though the score is by no means perfect, we can evaluate DIS 2 with successively fewer PCs and only retain those above a certain score threshold, reducing the average prediction error while remaining reasonably conservative (Figure C.5). For further details, see Appendix C.

5 Experiments

Datasets. We conduct experiments across 11 vision benchmark datasets for distribution shift on datasets that span applications in object classification, satellite imagery, and medicine. We use four BREEDs datasets: [65] Entity13, Entity30, Nonliving26, and Living17; FMoW [11] and Camelyon [4] from WILDS [35]; Officehome [69]; Visda [52, 51]; CIFAR10, CIFAR100 [36]; and Domainet [53]. Each of these datasets consists of multiple domains with different types of natural and synthetic shifts. We consider subpopulation shift and natural shifts induced due to differences in the data collection process of ImageNet, i.e., ImageNetv2 [60] and a combination of both. For CIFAR10 and CIFAR100 we evaluate natural shifts due to variations in replication studies [59] and common corruptions [27]. For all datasets, we use the same source and target domains commonly used in previous studies [22, 64]. We provide precise details about the distribution shifts considered in Appendix A. Because distribution shifts vary widely in scope, prior evaluations which focus on only one specific type of shift (e.g., corruptions) often do not convey the full story. We therefore emphasize the need for more comprehensive evaluations across many different types of shifts and training methods , as we present here.

Experimental setup and protocols. Along with source-only training with ERM, we experiment with Unsupervised Domain Adaptation (UDA) methods that aim to improve target performance with unlabeled target data (FixMatch [66], DANN [18], CDAN [38], and BN-adapt [37]). We experiment with Densenet121 [28] and Resnet18/Resnet50 [26] pretrained on ImageNet. For source-only ERM, as with other methods, we default to using strong augmentations: random horizontal flips, random crops, as well as Cutout [16] and RandAugment [13]. Unless otherwise specified, we default to full finetuning for source-only ERM and UDA methods. We use source hold-out performance to pick the best hyperparameters for the UDA methods, since we lack labeled validation data from the target distribution. For all of these methods, we fix the algorithm-specific hyperparameters to their original recommendations following the experimental protocol in [22]. For more details, see Appendix A.

Methods evaluated. We compare DIS 2 to four competitive baselines: Average Confidence (AC; [25]), Difference of Confidences (DoC; [24]), Average Thresholded Confidence (ATC; [21]), and Confidence Optimal Transport (COT; [40]). We give detailed descriptions of these methods in Appendix A. For all methods, we implement post-hoc calibration on validation source data with temperature scaling [25], which has been shown to improve performance. For DIS 2 , we report bounds evaluated both on the full features and on the logits of ˆ h as described in Section 4. Unless specified otherwise, we set δ = . 01 everywhere. We also experiment with dropping the lower order concentration term in Theorem 3.6, using only the sample average. Though this is of course no longer a conservative bound, we find it is an excellent predictor of test error and is worth including.

Metrics for evaluation. As our emphasis is on giving valid error bounds, we report the coverage , i.e. the fraction of predictions for which the true error does not exceed the predicted error. We also report the standard prediction metric, mean absolute error (MAE). Finally, we measure the conditional average overestimation : this is the MAE among predictions which overestimate the accuracy. This metric captures the idea that the most important thing is giving a valid bound-but if for some reason it is not, we'd at least like it to be as accurate as possible.

Results. Reported metrics for all methods can be found in Table 1. We aggregate results over all datasets, shifts, and training methods-we stratify only by whether the training method is domainadversarial, as this affects the validity of Assumption 3.5. We find that DIS 2 achieves competitive MAE while maintaining substantially higher coverage, even for domain-adversarial features. When it does overestimate accuracy, it does so by much less, implying that it is ideal for conservative estimation even when any given error bound is not technically satisfied. Dropping the concentration term performs even better (sometimes beating the baselines), at the cost of some coverage. This suggests that efforts to better estimate the true maximum discrepancy may yield even better predictors. We also show scatter plots to visualize performance on individual distribution shifts, plotting each source-target pair as a single point. For these too we report separately the results for domainadversarial (Figure 3) and non-domain-adversarial methods (Figure 1). To avoid clutter, these two plots do not include DoC, as it performed comparably to AC. Figure 4(a) displays additional scatter plots which allow for a direct comparison of the variants of DIS 2 . Finally, Figure 4(b) plots the observed violation rate (i.e. 1 -coverage) of DIS 2 on non-domain-adversarial methods for varying δ . We observe that it lies at or below the line y = x , meaning the probabilistic bound provided by Theorem 3.6 holds across a range of failure probabilities. Thus we see that our probabilistic bound is empirically valid all of the time-not in the sense that each individual shift's error is upper bounded, but rather that the desired violation rate is always satisfied.

Strengthening the baselines to improve coverage. Since the baselines we consider in this work prioritize predictive accuracy over conservative estimates, their coverage can possibly be improved without too much increase in error. We explore this option using LOOCV: for a desired coverage, we learn a parameter to either scale or shift a method's prediction to achieve that level of coverage on all but one of the datasets. We then evaluate the method on all shifts of the remaining dataset, and we repeat this for each dataset. Appendix D reports the results for varying coverage levels. We find that (i) the baselines do not achieve the desired coverage on the held out data, though they get somewhat close; and (ii) the adjustment causes them to suffer higher MAE than DIS 2 . Thus DIS 2 is on the Pareto frontier of MAE and coverage, and is preferable when conservative bounds are desirable. We believe identifying alternative methods of post-hoc prediction adjustment is a promising future direction.

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Target Accuracy

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Prediction

Features

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Target Accuracy

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Prediction

Logits

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Target Accuracy

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Prediction

Logits w/o

d

(a)

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

Violation Probability

d

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

Observed Violation Rate

y = x

(b)

6 Conclusion

The ability to evaluate trustworthy , non-vacuous error bounds for deep neural networks under distribution shift remains an extremely important open problem. Due to the wide variety of real-world shifts and the complexity of modern data, restrictive a priori assumptions on the distribution (i.e., before observing any data from the shift of interest) seem unlikely to be fruitful. On the other hand, prior methods which estimate accuracy using extra information-such as unlabeled test samplesoften rely on opaque conditions whose likelihood of being satisfied is difficult to predict, and so they sometimes provide large underestimates of test error with no warning signs.

This work bridges this gap with a simple, intuitive condition and a new disagreement loss which together result in competitive error prediction , while simultaneously providing an (almost) guaranteed probabilistic error bound . We also study how the process of evaluating the bound (e.g., the optimization landscape) can provide even more useful signal, enabling better predictive accuracy. We expect there is potential to push further in each of these directions, hopefully extending the current accuracyreliability Pareto frontier for test error bounds under distribution shift.

Acknowledgments and Disclosure of Funding

Thanks to Sam Sokota, Bingbin Liu, Yuchen Li, Yiding Jiang, Zack Lipton, Roni Rosenfeld, and Andrej Risteski for helpful comments. ER acknowledges the support of NSF via IIS-1909816, IIS1955532, OAC-1934584. SG acknowledges Amazon Graduate Fellowship and JP Morgan AI Ph.D. Fellowship for their support.

References

Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In International Conference on Machine Learning , pages 244-253. PMLR, 2018.

Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems , 32, 2019.

Christina Baek, Yiding Jiang, Aditi Raghunathan, and J Zico Kolter. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift. In Advances in Neural Information Processing Systems , 2022.

Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging , 2018.

Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. In Advances in neural information processing systems , pages 6240-6249, 2017.

Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances in Neural Information Processing Systems , volume 19. MIT Press, 2006.

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning , 79: 151-175, 2010.

Jiefeng Chen, Frederick Liu, Besim Avci, Xi Wu, Yingyu Liang, and Somesh Jha. Detecting errors and estimating accuracy on unlabeled data with self-training ensembles. Advances in Neural Information Processing Systems , 34:14980-14992, 2021.

Mayee Chen, Karan Goel, Nimit S Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Ré. Mandoline: Model evaluation under distribution shift. In International Conference on Machine Learning , pages 1617-1629. PMLR, 2021.

Yining Chen, Elan Rosenfeld, Mark Sellke, Tengyu Ma, and Andrej Risteski. Iterative feature matching: Toward provable domain generalization with logarithmic environments. In Advances in Neural Information Processing Systems , volume 35, pages 1725-1736. Curran Associates, Inc., 2022.

Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018.

Ching-Yao Chuang, Antonio Torralba, and Stefanie Jegelka. Estimating generalization under distribution shifts via domain-invariant representations. International conference on machine learning , 2020.

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 702-703, 2020.

Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15069-15078, 2021.

Weijian Deng, Stephen Gould, and Liang Zheng. What does rotation prediction tell us about classifier accuracy under varying testing environments? arXiv preprint arXiv:2106.05961 , 2021.

Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017.

Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008 , 2017.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. J. Mach. Learn. Res. , 17(1):2096-2030, jan 2016. ISSN 1532-4435.

Jacob Gardner, Geoff Pleiss, Kilian Q Weinberger, David Bindel, and Andrew G Wilson. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems , pages 7576-7586, 2018.

Saurabh Garg, Sivaraman Balakrishnan, J Zico Kolter, and Zachary C Lipton. Ratt: Leveraging unlabeled data to guarantee generalization. arXiv preprint arXiv:2105.00303 , 2021.

Saurabh Garg, Sivaraman Balakrishnan, Zachary Chase Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict out-of-distribution performance. In International Conference on Learning Representations , 2022.

Saurabh Garg, Nick Erickson, James Sharpnack, Alex Smola, Siva Balakrishnan, and Zachary Lipton. Rlsbench: Domain adaptation under relaxed label shift. In International Conference on Machine Learning (ICML) , 2023.

Tom Ginsberg, Zhongyuan Liang, and Rahul G Krishnan. A learning based hypothesis test for harmful covariate shift. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=rdfgqiwz7lZ .

Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence on unseen distributions. arXiv preprint arXiv:2107.03315 , 2021.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning (ICML) , 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Computer Vision and Pattern Recognition (CVPR) , 2016.

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261 , 2019.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017.

Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks, 2022. URL https://openreview. net/forum?id=dn4B7Mes2z .

David Hume. An enquiry concerning human understanding: A critical edition , volume 3. Oxford University Press on Demand, 2000.

Junguang Jiang, Yang Shu, Jianmin Wang, and Mingsheng Long. Transferability in deep learning: A survey, 2022.

Yiding Jiang, Vaishnavh Nagarajan, Christina Baek, and J Zico Kolter. Assessing generalization of SGD via disagreement. In International Conference on Learning Representations , 2022.

Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations , 2020.

Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.02937 , 2022.

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML) , 2021.

Alex Krizhevsky and Geoffrey Hinton. Learning Multiple Layers of Features from Tiny Images. Technical report, Citeseer, 2009.

Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv:1603.04779 , 2016.

Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems , 31, 2018.

Philip M Long and Hanie Sedghi. Generalization bounds for deep convolutional neural networks. arXiv preprint arXiv:1905.12600 , 2019.

Yuzhe Lu, Zhenlin Wang, Runtian Zhai, Soheil Kolouri, Joseph Campbell, and Katia Sycara. Predicting out-of-distribution error with confidence optimal transport. In ICLR 2023 Workshop on Pitfalls of limited data and computation for Trustworthy ML , 2023.

Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.3430 , 2009.

Vaishnavh Nagarajan and J Zico Kolter. Deterministic pac-bayesian generalization bounds for deep networks via generalizing noise-resilience. arXiv preprint arXiv:1905.13344 , 2019.

Vaishnavh Nagarajan and J. Zico Kolter. Uniform convergence may be unable to explain generalization in deep learning. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Preetum Nakkiran and Yamini Bansal. Distributional generalization: A new kind of generalization. ArXiv , abs/2009.08092, 2019.

Behnam Neyshabur. Implicit regularization in deep learning. arXiv preprint arXiv:1709.01953 , 2017.

Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory , pages 1376-1401, 2015.

Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. Exploring generalization in deep learning. arXiv preprint arXiv:1706.08947 , 2017.

Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of over-parametrization in generalization of neural networks. In International Conference on Learning Representations , 2018.

Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. arXiv preprint arXiv:1906.02530 , 2019.

Matteo Pagliardini, Martin Jaggi, François Fleuret, and Sai Praneeth Karimireddy. Agree to disagree: Diversity through disagreement for better transferability. In The Eleventh International Conference on Learning Representations , 2023.

Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge, 2017.

Xingchao Peng, Ben Usman, Kuniaki Saito, Neela Kaushik, Judy Hoffman, and Kate Saenko. Syn2real: A new benchmark forsynthetic-to-real visual domain adaptation. arXiv preprint arXiv:1806.09755 , 2018.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1406-1415, 2019.

Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems , 34:1256-1272, 2021.

Emmanouil A Platanios, Hoifung Poon, Tom M Mitchell, and Eric Horvitz. Estimating accuracy from unlabeled data: A probabilistic logic approach. arXiv preprint arXiv:1705.07086 , 2017.

Emmanouil Antonios Platanios, Avinava Dubey, and Tom Mitchell. Estimating accuracy from unlabeled data: A bayesian approach. In International Conference on Machine Learning , pages 1416-1425. PMLR, 2016.

Hamed Rahimian and Sanjay Mehrotra. Distributionally robust optimization: A review. arXiv preprint arXiv:1908.05659 , 2019.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv:1806.00451 , 2018.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? 2018. https://arxiv.org/abs/1806.00451 .

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning , pages 5389-5400. PMLR, 2019.

Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint arXiv:2202.06856 , 2022.

Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. An online learning approach to interpolation and extrapolation in domain generalization. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , volume 151 of Proceedings of Machine Learning Research , pages 2641-2657. PMLR, 28-30 Mar 2022.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115(3):211-252, 2015.

Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, and Percy Liang. Extending the wilds benchmark for unsupervised adaptation. In NeurIPS Workshop on Distribution Shifts , 2021.

Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. arXiv preprint arXiv:2008.04859 , 2020.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semisupervised learning with consistency and confidence. Advances in Neural Information Processing Systems , 33, 2020.

Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence , 2016.

Antonio Torralba, Rob Fergus, and William T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 30(11):1958-1970, 2008.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5018-5027, 2017.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530 , 2016.

Richard Zhang. Making convolutional networks shift-invariant again. In ICML , 2019.

Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning . PMLR, 2019.

Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P Adams, and Peter Orbanz. Non-vacuous generalization bounds at the imagenet scale: a pac-bayesian compression approach. arXiv preprint arXiv:1804.05862 , 2018.

Appendix

A Experimental Details

A.1 Description of Baselines

Average Thresholded Confidence (ATC). ATC first estimates a threshold t on the confidence of softmax prediction (or on negative entropy) such that the number of source labeled points that get a confidence greater than t match the fraction of correct examples, and then estimates the test error on on the target domain D test as the expected number of target points that obtain a score less than t , i.e.,

where t satisfies: ∑ j i =1 I [max j ∈Y ( f j ( x i )) < t ] = ∑ m i =1 I [ arg max j ∈Y f j ( x i ) = y i ]

Average Confidence (AC). Error is estimated as the average value of the maximum softmax confidence on the target data, i.e, AC D test = ∑ n i =1 max j ∈Y f j ( x ′ i ) .

̸

̸

Difference Of Confidence (DOC). We estimate error on the target by subtracting the difference of confidences on source and target (as a surrogate to distributional distance [24]) from the error on source distribution, i.e., DOC D test = ∑ n i =1 max j ∈Y f j ( x ′ i ) + ∑ m i =1 I [ arg max j ∈Y f j ( x i ) = y i ] -∑ m i =1 max j ∈Y f j ( x i ) . This is referred to as DOC-Feat in [24].

Confidence Optimal Transport (COT). COT uses the empirical estimator of the Earth Mover's Distance between labels from the source domain and softmax outputs of samples from the target domain to provide accuracy estimates:

where S n = { f ( x ′ i ) } n i =1 are the softmax outputs on the unlabeled target data and Y m = { y j } j m =1 are the labels on holdout source examples.

For all of the methods described above, we assume that { ( x ′ i ) } n i =1 are the unlabeled target samples and { ( x i , y i ) } i m =1 are hold-out labeled source samples.

A.2 Dataset Details

In this section, we provide additional details about the datasets used in our benchmark study.

CIFAR10 We use the original CIFAR10 dataset [36] as the source dataset. For target domains, we consider (i) synthetic shifts (CIFAR10-C) due to common corruptions [27]; and (ii) natural distribution shift, i.e., CIFAR10v2 [58, 68] due to differences in data collection strategy. We randomly sample 3 set of CIFAR-10-C datasets. Overall, we obtain 5 datasets (i.e., CIFAR10v1, CIFAR10v2, CIFAR10C-Frost (severity 4), CIFAR10C-Pixelate (severity 5), CIFAR10-C Saturate (severity 5)).

CIFAR100 Similar to CIFAR10, we use the original CIFAR100 set as the source dataset. For target domains we consider synthetic shifts (CIFAR100-C) due to common corruptions. We sample 4 CIFAR100-C datasets, overall obtaining 5 domains (i.e., CIFAR100, CIFAR100C-Fog (severity 4), CIFAR100C-Motion Blur (severity 2), CIFAR100C-Contrast (severity 4), CIFAR100C-spatter (severity 2) ).

FMoW In order to consider distribution shifts faced in the wild, we consider FMoW-WILDs [35, 11] from WILDS benchmark, which contains satellite images taken in different geographical regions and at different times. We use the original train as source and OOD val and OOD test splits as target domains as they are collected over different time-period. Overall, we obtain 3 different domains.

Camelyon17 Similar to FMoW, we consider tumor identification dataset from the wilds benchmark [4]. We use the default train as source and OOD val and OOD test splits as target domains as they are collected across different hospitals. Overall, we obtain 3 different domains.

BREEDs We also consider BREEDs benchmark [65] in our setup to assess robustness to subpopulation shifts. BREEDs leverage class hierarchy in ImageNet to re-purpose original classes to be the subpopulations and defines a classification task on superclasses. We consider distribution shift due to subpopulation shift which is induced by directly making the subpopulations present in the training and test distributions disjoint. BREEDs benchmark contains 4 datasets Entity13 , Entity-30 , Living-17 , and Non-living-26 , each focusing on different subtrees and levels in the hierarchy. We also consider natural shifts due to differences in the data collection process of ImageNet [63], e.g, ImageNetv2 [60] and a combination of both. Overall, for each of the 4 BREEDs datasets (i.e., Entity-13, Entity-30, Living-17, and Non-living-26), we obtain four different domains. We refer to them as follows: BREEDsv1 sub-population 1 (sampled from ImageNetv1), BREEDsv1 sub-population 2 (sampled from ImageNetv1), BREEDsv2 sub-population 1 (sampled from ImageNetv2), BREEDsv2 sub-population 2 (sampled from ImageNetv2). For each BREEDs dataset, we use BREEDsv1 sub-population A as source and the other three as target domains.

OfficeHome We use four domains (art, clipart, product and real) from OfficeHome dataset [69]. We use the product domain as source and the other domains as target.

DomainNet We use four domains (clipart, painting, real, sketch) from the Domainnet dataset [53]. We use real domain as the source and the other domains as target.

Visda We use three domains (train, val and test) from the Visda dataset [52]. While 'train' domain contains synthetic renditions of the objects, 'val' and 'test' domains contain real world images. To avoid confusing, the domain names with their roles as splits, we rename them as 'synthetic', 'Real-1' and 'Real-2'. We use the synthetic (original train set) as the source domain and use the other domains as target.

A.3 Setup and Protocols

Architecture Details For all datasets, we used the same architecture across different algorithms:

CIFAR-10: Resnet-18 [26] pretrained on Imagenet

CIFAR-100: Resnet-18 [26] pretrained on Imagenet

Camelyon: Densenet-121 [28] not pretrained on Imagenet as per the suggestion made in [35]

FMoW: Densenet-121 [28] pretrained on Imagenet

BREEDs (Entity13, Entity30, Living17, Nonliving26): Resnet-18 [26] not pretrained on Imagenet as per the suggestion in [65]. The main rationale is to avoid pre-training on the superset dataset where we are simulating sub-population shift.

Officehome: Resnet-50 [26] pretrained on Imagenet

Domainnet: Resnet-50 [26] pretrained on Imagenet

Visda: Resnet-50 [26] pretrained on Imagenet

Except for Resnets on CIFAR datasets, we used the standard pytorch implementation [19]. For Resnet on cifar, we refer to the implementation here: https://github.com/kuangliu/pytorch-cifar . For all the architectures, whenever applicable, we add antialiasing [71]. We use the official library released with the paper.

For imagenet-pretrained models with standard architectures, we use the publicly available models here: https://pytorch.org/vision/stable/models.html . For imagenet-pretrained models on the reduced input size images (e.g. CIFAR-10), we train a model on Imagenet on reduced input size from scratch. We include the model with our publicly available repository.

Hyperparameter details First, we tune learning rate and ℓ 2 regularization parameter by fixing batch size for each dataset that correspond to maximum we can fit to 15GB GPU memory. We set the number of epochs for training as per the suggestions of the authors of respective benchmarks. Note that we define the number of epochs as a full pass over the labeled training source data. We summarize learning rate, batch size, number of epochs, and ℓ 2 regularization parameter used in our study in Table A.3.

For each algorithm, we use the hyperparameters reported in the initial papers. For domain-adversarial methods (DANN and CDANN), we refer to the suggestions made in Transfer Learning Library [31]. We tabulate hyperparameters for each algorithm next:

DANN, CDANN, As per Transfer Learning Library suggestion, we use a learning rate multiplier of 0 . 1 for the featurizer when initializing with a pre-trained network and 1 . 0 otherwise. We default to a penalty weight of 1 . 0 for all datasets with pre-trained initialization.

FixMatch We use the lambda is 1.0 and use threshold τ as 0.9.

Compute Infrastructure Our experiments were performed across a combination of Nvidia T4, A6000, and V100 GPUs.

B Comparing Disagreement Losses

We define the alternate losses for maximizing disagreement:

Chuang et al. [12] minimize the negative cross-entropy loss, which is concave in the model logits. That is, they add the term log softmax ( h ( x ) y ) to the objective they are minimizing. This loss results in substantially lower disagreement discrepancy than the other two.

Pagliardini et al. [50] use a loss which is not too different from ours. They define the disagreement objective for a point ( x, y ) as

̸

For comparison, ℓ dis can be rewritten as

̸

where the incorrect logits are averaged and the exponential is pushed outside the sum. This modification results in (2) being convex in the logits and an upper bound to the disagreement 0-1 loss, whereas (1) is neither.

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Disagreement Discrepancy

0

20

40

60

Count

Train

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Disagreement Discrepancy

Test

Negative Cross-Entropy

D-BAT

/lscript

dis

(ours)

Loss

Neg. X-Ent [12]

0

.

3555

±

.

0124

0

.

1694

±

.

0105

D-BAT [50]

0

.

8145

±

.

0177

0

.

3224

±

.

0212

ℓ

dis

(Ours)

0

.

8333

.

0132

0

.

3322

.

0205

±

±

Figure B.1 displays histograms of the achieved disagreement discrepancy across all distributions for each of the disagreement losses (all hyperparameters and random seeds are the same for all three losses). The table below it reports the mean disagreement discrepancy on the train and test sets. We find that the negative cross-entropy, being a concave function, results in very low discrepancy. The DBAT loss (Equation (1)) is reasonably competitive with our loss (Equation (2)) on average, seemingly because it gets very high discrepancy on a subset of shifts. This suggests that it may be particularly suited for a specific type of distribution shift, though it is less good overall. Though the averages are reasonably close, the samples are not independent, so we run a paired t-test and we find that the increases to average train and test discrepancies achieved by ℓ dis are significant at levels p = 0 . 024 and p = 0 . 009 , respectively. With enough holdout data, a reasonable approach would be to split the data in two: one subset to validate critics trained on either of the two losses, and another to evaluate the discrepancy of whichever one is ultimately selected.

C Exploration of the Validity Score

To experiment with reducing the complexity of the class H , we evaluate DIS 2 on progressively fewer top principal components (PCs) of the features. Precisely, for features of dimension d , we evaluate DIS 2 on the same features projected onto their top d / k components, for k ∈ [1 , 4 , 16 , 32 , 64 , 128] (Figure C.2). We see that while projecting to fewer and fewer PCs does reduce the error bound value, unlike the logits it is a rather crude way to reduce complexity of H , meaning at some point it goes too far and results in invalid error bounds.

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Target Accuracy

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Prediction

Full Features

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Target Accuracy

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Prediction

Top 1/4 PCs

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Target Accuracy

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Prediction

Top 1/16 PCs

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Target Accuracy

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Prediction

Top 1/32 PCs

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Target Accuracy

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Prediction

Top 1/64 PCs

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Target Accuracy

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Prediction

Top 1/128 PCs

However, during the optimization process we observe that around when this violation occurs, the task of training a critic to both agree on S and disagree on T goes from 'easy' to 'hard'. Figure C.3 shows that on the full features, the critic rapidly ascends to maximum agreement on S , followed by slow decay (due to both overfitting and learning to simultaneously disagree on T ). As we drop more and more components, this optimization becomes slower.

SourceAgreementDuringTraining

Epoch

0.8

Agreement on source

0.6

0.4

original

PCA / 4

0.2

PCA/16

PCA / 32

PCA / 64

PCA /128

0

20

40

60

80

100

We therefore design a 'validity score' intended to capture this phenomenon which we refer to as the cumulative ℓ 1 ratio . This is defined as the maximum agreement achieved, divided by the cumulative sum of absolute differences in agreement across all epochs up until the maximum was achieved.

Formally, let { a i } T i =1 represent the agreement between h ′ and ˆ h after epoch i , i.e. 1 -ϵ ˆ S ( ˆ h, h ′ i ) , and define m := arg max i ∈ [ T ] a i . The cumulative ℓ 1 ratio is then a m a 1 + ∑ i m =2 | a i -a i -1 | . Thus, if the agreement rapidly ascends to its maximum without ever going down over the course of an epoch, this ratio will be equal to 1, and if it non-monotonically ascends then the ratio will be significantly less. This definition was simply the first metric we considered which approximately captures the behavior we observed; we expect it could be greatly improved.

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

Cumulative

/lscript

1

Ratio

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Accuracy Minus Bound

Figure C.4 displays a scatter plot of the cumulative ℓ 1 ratio versus the difference in estimated and true error for DIS 2 evaluated on the full range of top PCs. A negative value implies that we have underestimated the error (i.e., the bound is not valid). We see that even this very simply metric roughly linearly correlates with the tightness of the bound, which suggests that evaluating over a range of top PC counts and only keeping predictions whose ℓ 1 ratio is above a certain threshold can improve raw predictive accuracy without reducing coverage by too much. Figure C.5 shows that this is indeed the case: compared to DIS 2 evaluated on the logits, keeping all predictions above a score threshold can produce more accurate error estimates, without too severely underestimating error in the worst case.

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Target Accuracy

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Logits (0.15/0.99)

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Target Accuracy

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Ratio

>

0.95 (0.14/0.96)

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Target Accuracy

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Ratio

>

0.9 (0.13/0.88)

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Target Accuracy

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Ratio

>

0.8 (0.10/0.73)

D Making Baselines More Conservative with LOOCV

To more thoroughly compare DIS 2 to prior estimation techniques, we consider a strengthening of the baselines which may give them higher coverage without too much cost to prediction accuracy. Specifically, for each desired coverage level α ∈ [0 . 9 , 0 . 95 , 0 . 99] , we use all but one of the datasets to learn a parameter to either scale or shift a method's predictions enough to achieve coverage α . We then evaluate this scaled or shifted prediction on the distribution shifts of the remaining dataset, and we repeat this for each one.

The results, found in Table D.4, demonstrate that prior methods can indeed be made to have much higher coverage, although as expected their MAE suffers. Furthermore, they still underestimate error on the tail distribution shifts by quite a bit, and they rarely achieve the desired coverage on the heldout dataset-though they usually come reasonably close. In particular, ATC [21] and COT [40] do well with a shift parameter, e.g. at the desired coverage α = 0 . 95 ATC matches DIS 2 in MAE and gets 94.4% coverage (compared to 98.9% by DIS 2 ). However, its conditional average overestimation is quite high, almost 9%. COT gets much lower overestimation (particularly for higher coverage levels), and it also appears to suffer less on the tail distribution shifts in the sense that α = 0 . 99 does not induce nearly as high MAE as it does for ATC. However, at that level it only achieves 95.6% coverage, and it averages almost 5% accuracy overestimation on the shifts it does not correctly bound (compared to 0.1% by DIS 2 ). Also, its MAE is still substantially higher than DIS 2 , despite getting lower coverage. Finally, we evaluate the scale/shift approach on our DIS 2 bound without the lower order term, but based on the metrics we report there appears to be little reason to prefer it over the untransformed version, one of the baselines, or the original DIS 2 bound.

Taken together, these results imply that if one's goal is predictive accuracy and tail behavior is not important (worst ~10%), ATC or COT will likely get reasonable coverage with a shift parameterthough they still significantly underestimate error on a non-negligible fraction of shifts. If one cares about the long tail of distribution shifts, or prioritizes being conservative at a slight cost to average accuracy, DIS 2 is clearly preferable. Finally, we observe that the randomness which determines which shifts are not correctly bounded by DIS 2 is 'decoupled' from the distributions themselves under Theorem 3.6, in the sense that it is an artifact of the random samples, rather than a property of the distribution (recall Figure 4(b)). This is in contrast with the shift/scale approach which would produce almost identical results under larger sample sizes because it does not account for finite sample effects. This implies that some distribution shifts are simply 'unsuitable' for prior methods because they do not satisfy whatever condition these methods rely on, and observing more samples will not remedy this problem. It is clear that working to understand these conditions is crucial for reliability and interpretability, since we are not currently able to identify which distributions are suitable a priori.

E Proving that Assumption 3.5 Holds for Some Datasets

Here we describe how the equivalence of Assumption 3.5 and the bound in Theorem 3.6 allow us to prove that the assumption holds with high probability. By repeating essentially the same proof as Theorem 3.6 in the other direction, we get the following corollary:

Corollary E.1. If Assumption 3.5 does not hold, then with probability ≥ 1 -δ ,

Note that the concentration term here is different from Theorem 3.6 because we are bounding the empirical target error, rather than the true target error. The reason for this change is that now we can make direct use of its contrapositive:

Corollary E.2. With probability ≥ 1 -δ over the randomness of the samples ˆ S and ˆ T , if it is the case that

then Assumption 3.5 must hold.

We evaluate this bound on non-domain-adversarial shifts with δ = 10 -6 . As some of the BREEDS shifts have as few as 68 test samples, we restrict ourselves to shifts with n T ≥ 500 to ignore those where the finite-sample term heavily dominates; this removes a little over 20% of all shifts. Among the remainder, we find that the bound in Corollary E.2 holds 55.7% of the time when using full features and 25.7% of the time when using logits. This means that for these shifts, we can be essentially certain that Assumption 3.5-and therefore also Assumption 3.3-is true.

Note that the fact that the bound is not violated for a given shift does not at all imply that the assumption is not true. In general, the only rigorous way to prove that Assumption 3.5 does not hold would be to show that for a fixed δ , the fraction of shifts for which the bound in Theorem 3.6 does not hold is larger than δ (in a manner that is statistically significant under the appropriate hypothesis test). Because this never occurs in our experiments, we cannot conclude that the assumption is ever false. At the same time, the fact that the bound does hold at least 1 -δ of the time does not prove that the assumption is true-it merely suggests that it is reasonable and that the bound should continue to hold in the future. This is why it is important for Assumption 3.5 to be simple and intuitive, so that we can trust that it will persist and anticipate when it will not.

However, Corollary E.2 allows us to make a substantially stronger statement. In fact, it says that for any distribution shift, with enough samples, we can prove a posteriori whether or not Assumption 3.5 holds, because the gap between these two bounds will shrink with increasing sample size.

F Figure 1 Stratified by Training Method

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

Target Accuracy

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

ERM-aug-imagenet

ERM-aug-rand

BN adapt

FixMatch

y = x

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper try to propose a new metric on evaluation of the distribution shift of dataset. The authors first prove the proposed error estimation bound with some assumptions. Then they demonstrate the effectiveness of their method through training a surrogate model maximizing the disagreement discrepancy. The authors conduct experiments on different dataset, showing that their method never overestimate the prediction accuracy on the target domain.\n\nSTRENGTHS:\n1. The authors try to use unlabeled data in the target domain, which is believed can bring benefits on predict the accuracy on the target domain.\n2. The author introduce an interesting conception, named disagreement discrepancy, that represents the maximum difference between the target domain and source domain in the hypothesis set. This conception may provide some insights on the investigation of domain shift.\n\nWEAKNESSES:\n1. Certain evaluation metrics seem unclear. The reviewer finds it confusing why the method's characteristic of never overestimating the target accuracy—consistently underestimating it—is considered advantageous, particularly when its Mean Absolute Error (MAE) significantly underperforms compared to other methods. The reviewer recommends the authors to illustrate the benefits of this particular feature.\n2. When compared with the baseline methods, the proposed metric falls short in terms of some crucial metrics, such as the Mean Absolute Error (MAE)\n3. The authors attempt to train an additional network by maximizing the discrepancy in disagreement. However, this approach may be time-consuming and unreliable due to its sensitivity to the training hyperparameters.\n\nQUESTIONS:\nMost questions are listed in the weakness part.\n\nMinor:\n1. Please explain the meaning of '(almost) Provable'\n\nLIMITATIONS:\nShown in weakness part", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method that under certain assumptions provides upper bounds on the accuracy under distribution shift when provided with unlabelled test data from the shift's target distribution.\n\nSTRENGTHS:\nThe assumptions necessary to obtain the bounds are clearly stated and discussed.\n\nThe problem of estimating performance under distribution shift is important, and it is not clear whether meaningful bounds are possible. As such, the attempt of providing almost provable bounds in the way that it is done here could be a good contribution in this direction.\n\nThe experimental evaluations are quite extensive and reasonable. Particularly the careful examinations of when the assumptions are fulfilled are important and support these assumptions.\n\nWEAKNESSES:\nThe method strongly depends on the function class in which the optimal critic is searched for, and potentially also on its approximation quality within that class. This means that when one allows a perfect critic to be found, which intuitively would look like it should be allowed since it definitely exists if the source and target distributions have smaller overlap than the source test error, then the obtained bounds are vacuous.\nOn the other hand, one can not know a priori whether a function class is large enough to contain a critic that satisfies Assumption 3.3 and thus provides a trustworthy guaranteed bound.\nThese issues are clearly stated and discussed, and it is shown that the necessary assumptions easily hold when considering linear functions on features for standard networks and benchmarks.\nHowever, it is not clear how reliably the method could be transferred to new, potentially less regular datasets and/or models. The bounds after all rely on assumptions that can only be confirmed empirically after the true labels of the target distribution are known.\n\nCompared to other methods, the proposed $Dis^2$ method makes estimates on the accuracy on the target distribution that are worse than simple baselines. This means that the issues of the provability of the bounds mentioned above are important weaknesses.\n\nIn l. 215, it is not clear that the logistic surrogate is still a valid approximation after it is combined with the disagreement logistic loss. The combination of the two losses should be discussed as the sum looks straightforward but has as far as I can see no intrinsic reason to be natural. There is balancing done by leaving out the $1/\\log|Y|$ term, which seems very ad hoc. Other combinations like e.g. sum of squares or other powers of the individual losses would appear just as valid. Would it make sense (for certain function classes) to do a constraint optimization of the disagreement on the target data while fixing the decisions on the source (i.e. $\\epsilon_S{\\hat{h}, h') = 0$)?\n\nQUESTIONS:\nWhy is Method [1] (Agreement-on-the-line) not compared to in the evaluations? This should be possible after it is fit on some other shifted distributions.\n\nCertain assumptions on the distribution shift seem to be necessary, besides the discussed model-specific settings. For example, the bounds would not hold for a shift with similar images but switched class labels. Can such extreme cases be formalized and excluded?\n\nLIMITATIONS:\nWhether the necessary assumptions hold is extensively discussed and tested for standard distribution shifts. It is however not clear to what kinds of situations one can or cannot expect the method to generalize.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to propose a way to characterize the error bounds under distribution shift with provable error guarantees. Based on several assumptions, the authors show that the target error bound can be reasonably estimated by maximizing the agreement within the desired classifier on source distribution and maximizing the disagreement on the target distribution, i.e. $DIS^2$. The authors justified the validity of $DIS^2$ by comparing it to other existing methods both from theoretical and empirical perspectives, and showed that it successfully upper bounds the error under distribution shift. The authors also discussed to what extent the assumptions hold in practice with empirical findings.\n\nSTRENGTHS:\nThe problem of justifying the validity of the trained models under distribution shift is important in modern machine learning. The prior work based on data dependent uniform convergence or assumptions of how distribution shifts are relatively vacuous in practice. The author proposes a method to characterize the error bound of the trained model on target distribution that is theoretically sound with assumptions almost empirically verifiable, making it a significant progress in this area.\n\nWEAKNESSES:\nWeaknesses:\n\n1. Although empirical evidence shows that both Assumptions 3.3 and 3.5 are reasonable, the observations are still based on limited data. There could be cases when the assumptions fail to hold. The paper discussed a setting (domain-adversarial learning) where the proposed method $DIS^2$ may be invalid. It is still unclear to me when the assumptions do or do not hold in practice. From a theoretical perspective, is it related to the representativeness of the hypothesis class H? On the empirical side, although except for the adversarial learning case the target function is rarely the worst case, I still feel there could be a possibility. After all that is the whole point of worst-case analysis. Hence, this point of view needs to be supported by more substantive evidence, probably based on real data.\n\n2. The representativeness of the hypothesis class is partly based on how many features the algorithm uses in training. There is a section talking about how the number of features affects the value of the error bound (the logits). It would be interesting to discuss these empirical findings together with hypothesis class representation.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes, the limitations have been adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper provides new error bound of distribution shift based on a notion called disagreement discrepancy. By assuming that the model class has enough expressiveness, it is theoretically proven that the error can be bounded by the worst-case disagreement discrepancy. Empirical investigation shows that such a method provides valid error bounds.

STRENGTHS:
1.	The proposed method is novel and interesting. The paper does a great work in demonstrating why such a measure is better than $\mathcal{H}-$ and $\mathcal{H}\Delta \mathcal{H}$-divergence.

2.	This paper provides exhaustive experiments to test the performance of proposed measure.

WEAKNESSES:
1.	The writing of this paper can be further polished. For example, the authors can briefly introduce $\mathcal{H}-$ and $\mathcal{H}\Delta \mathcal{H}$-divergence in Section 3.

2.	According to the experiments, the proposed measure is not better than existing baselines.

QUESTIONS:
1.	In line 246, what does “the logits output by $\hat{h}$” mean?

2.	The method bounds the  disagreement discrepancy between h and true label by the worst-case disagreement discrepancy with respect to h. However, when the model class is highly expressive, how we can expect the worst-case bound to be tight?

3.	Does the method preserve the order of error? That is, if we have the error of model A is larger than model B, can we observe that the Dis^2 of model A is also larger than model B?

LIMITATIONS:
Please refer to “weakness”.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes a novel method to estimate the error of a classifier under a shift of distribution. It relies on finding a worst case \"critic\" classifier which maximizes disagreement on some unlabeled target domain and hence bounds the disagreement with the true labeling function. The authors apply several necessary ticks to make the bound tractable and non-vacuous: (i) they introduce a new disagreement loss which can be efficiently maximized to find the critique and (ii) they only consider linear critics. While those two aspects make it possible that computed bounds are violated, in practice they show their bound is competitive in terms of mean absolute error while providing a better coverage than other methods.\n\nSTRENGTHS:\nClarity: I found the paper well written and enjoyable to read. Presenting complex ideas in simple terms is one of the contribution of this work. Related works are cited adequately, I would only suggest adding [1] which seems also relevant. \n\nOriginality: While I find the insights and derivation not entirely novel---the derivation in [2] also relies on the same idea of bounding disagreement on the target domain with a worst-case critic while agreeing on the source domain---using those with the goal to provide an error bound on the target distribution is novel to me. Furthermore, the authors propose a novel disagreement loss. \n\nSignificance: The proposed approach is simple to implement yet competitive, I can see it being useful to the community. \n\nReferences:\n\n[1] Predicting Out-of-Distribution Error with the Projection Norm (https://arxiv.org/abs/2202.05834)\n\n[2] Agree to Disagree: Diversity through Disagreement for Better Transferability (https://arxiv.org/abs/2202.04414)\n\nWEAKNESSES:\nWhile they are an extreme case, datasets with completely spurious correlations such as the ones in [3,4,2] might be interesting to consider. For those datasets, the best critic is actually $y^*$. You can then see if your approach recovers the right critic despite your added linearity constraint. As those datasets are shown in [4] to be a failure case when relying on logits, this might fail unless other features are used.\n\nReferences:\n\n[3] Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalization (https://arxiv.org/abs/2105.05612)\n\n[4] Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations (https://arxiv.org/abs/2204.02937)\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nLimitations of the approach are mentioned adequately in various parts of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel approach to estimate error bounds for deep neural networks under distributions shift. The proposed bounds remedy the limitations of exiting bounds that either provide vacuous bounds or underestimate the error for a big fraction of shifts while also often requiring access to test labels. In contrast, the bounds proposed by the authors only use unlabeled test data under a simple, intuitive assumption about the hypothesis class's ability to achieve small loss on the unlabeled train and test distributions. More specifically, their approach involves optimizing a disagreement discrepancy between two classifiers using a novel disagreement loss. The experimental results show that the obtained bounds are non-vacuous and comparable to existing competitive baselines.\n\nSTRENGTHS:\n- The presentation is clear and the paper is overall well-written. The notation is also introduced properly and make the theory easy to follow.\n- The limitations of previous works are also clearly stated and provide a good motivation for this work. \n- One of the main promises of this approach is allowing the user to interpolate between robustness and accuracy depending on the preferred level of risk tolerance.\n- The approach is overall simple and leads to non-vacuous bounds without using test labels.\n\nWEAKNESSES:\n- The presentation of the results is the main drawback of this work for me! The choice of aggregating all the results over all datasets, shifts, and training methods is strange and does not allow an objective evaluation of the performance of the bounds and analysis of their drawbacks and limitations for the different scenarios. It is possible that the proposed bounds underperform severely on most of the datasets, shifts and training methods and perform exceptionally well on a subset of them. It is also concerning that even the appendix does not include the detailed evaluation for each dataset and shift pair separately. \n- Additionally, we see from Table 1 that the only scenario where the proposed bounds outperform existing baselines is if the concentration term in Theorem 3.6 s dropped at the expense of coverage. Hence, a more detailed and transparent presentation of the results is crucial to verify the claims made by the authors.\n- Including standard errors in Table 1 would also be crucial to verifying the validity of the results given the wide range of scenarios for which the performance was aggregated.\n\nQUESTIONS:\n- How does the calculation of the bound scale with the size of the dataset and model architecture in practice?\n- In which cases would you expect your bound to be vacuous?\n\nLIMITATIONS:\nThe authors identified a setting where their proposed bounds may be invalid. The above questions and feedback may help identify more limitations of this work to be explicitly discussed in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}