SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that the fact that the bound is not violated for a given shift does not at all imply that the assumption is not true. In general, the only rigorous way to prove that Assumption 3.5 does not hold would be to show that for a fixed  $\delta$ , the fraction of shifts for which the bound in Theorem 3.6 does not hold is larger than  $\delta$  (in a manner that is statistically significant under the appropriate hypothesis test). Because this never occurs in our experiments, we cannot conclude that the assumption is ever false. At the same time, the fact that the bound does hold at least  $1 - \delta$  of the time does not prove that the assumption is true—it merely suggests that it is reasonable and that the bound should continue to hold in the future. This is why it is important for Assumption 3.5 to be simple and intuitive, so that we can trust that it will persist and anticipate when it will not.
However, Corollary E.2 allows us to make a substantially stronger statement. In fact, it says that for any distribution shift, with enough samples, we can prove a posteriori whether or not Assumption 3.5 holds, because the gap between these two bounds will shrink with increasing sample size.
F Figure 1 Stratified by Training Method
Figure F.6: Error prediction stratified by training method. Stars denote  $\mathrm{DIS}^2$ , circles are ATC NE. We see that  $\mathrm{DIS}^2$  maintains its validity across different training methods.