SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Having established that confidence-based deferral may be sub-optimal in certain settings, we now consider the viability of deferral rules that are learned in a post-hoc manner. Compared to confidence-based deferral, such rules aim to explicitly account for both the confidence of model 1 and 2, and thus avoid the failure cases identified above.
The key idea behind such post-hoc rules is to directly mimic the optimal deferral rule in (3). Recall that this optimal rule has a dependence on the output of  $h^{(2)}$ ; unfortunately, querying  $h^{(2)}$  defeats the entire purpose of cascades. Thus, our goal is to estimate (3) using only the outputs of  $p^{(1)}$ .
We summarise a number of post-hoc estimators in Table 1, which are directly motivated by the One-hot, Probability, and Relative Confidence Oracle respectively from §3.3. The first is to learn when model 1 is incorrect, and model 2 is correct. For example, given a validation set, suppose we construct samples  $S_{\mathrm{val}} \doteq \{(x_i, z_i^{(1)})\}$ , where  $z_i^{(1)} = 1[y = h^{(2)}(x_i)] - 1[y = h^{(1)}(x_i)]$ . Then, we fit
$$
\min_{g\colon \mathfrak{X}\to \mathbb{R}}\frac{1}{|S_{\mathrm{val}}|}\sum_{(x_{i},z_{i}^{(1)})\in S_{\mathrm{val}}}\ell (z_{i}^{(1)},g(x_{i})),
$$
where, e.g.,  $\ell$  is the square loss. The score  $g(x)$  may be regarded as the confidence in deferring to model 2. Similarly, a second approach is to perform regression to predict  $z_{i}^{(2)} = p_{y}^{(2)}(x_{i}) - p_{y}^{(1)}(x_{i})$ . The third approach is to directly estimate  $z_{i}^{(3)} = \max_{y'} p_{y'}^{(2)}(x_{i})$  using predictions of the first model.
As shall be seen in §5.2, such post-hoc rules can learn to avoid the failure cases for confidence-based deferral identified in the previous section. However, it is important to note that there are some conditions where such rules may not offer benefits over confidence-based deferral.
Failure mode: Bayes  $p^{(2)}$ . Suppose that the model  $p^{(2)}$  exactly matches the Bayes-probabilities, i.e.,  $p_{y'}^{(2)}(x) = \mathbb{P}(y' \mid x)$ . Then, estimating  $\max_{y'} p_{y'}^{(2)}(x)$  is equivalent to estimating  $\max_{y'} \mathbb{P}(y' \mid x)$ . However, the goal of model 1 is precisely to estimate  $\mathbb{P}(y \mid x)$ . Thus, if  $p^{(1)}$  is sufficiently accurate, in the absence of additional information (e.g., a fresh dataset), it is unlikely that one can obtain a better estimate of this probability than that provided by  $p^{(1)}$  itself. This holds even if the  $\mathbb{P}(y \mid x)$  is non-deterministic, and so the second model has non-trivial error.
Failure mode: non-predictable  $p^{(2)}$  error. When the model  $p^{(2)}$ 's outputs are not strongly predictable, post-hoc deferral may devolve to regular confidence-based deferral. Formally, suppose we seek to predict  $z \doteq \max_{y'} p_{y'}^{(2)}(x)$ , e.g., as in MaxProb. A non-trivial predictor must achieve an average square error smaller than the variance of  $z$ , i.e.,  $\mathbb{E}[(z - \mathbb{E}[z])^2]$ . If  $z$  is however not strongly predictable, the estimate will be tantamount to simply using the constant  $\mathbb{E}[z]$ . This brings us back to the assumption of model 2 having a constant probability of error, i.e., confidence-based deferral.
4.3 Finite-Sample Analysis for Post-Hoc Deferral Rules
We now formally quantify the gap in performance between the learned post-hoc rule and the Bayes-optimal rule  $r^* = 1[g^*(x) > c]$  in Proposition 3.1, where  $g^*(x) = \eta_{h^{(2)}(x)}(x) - \eta_{h^{(1)}(x)}(x)$ . We will consider the case where the validation sample  $S_{\mathrm{val}}$  is constructed with labels  $z_i^{(1)}$ . We pick a scorer  $\hat{g}$  that minimises the average squared loss  $\frac{1}{|S_{\mathrm{val}}|} \sum_{(x_i, z_i) \in S_{\mathrm{val}}} (z_i - g(x_i))^2$  over a hypothesis class  $\mathcal{G}$ . We then construct a deferral rule  $\hat{r}(x) = 1[\hat{g}(x) > c]$ .
Lemma 4.2. Let  $\mathcal{N}(\mathcal{G},\epsilon)$  denote the covering number of  $\mathcal{G}$  with the  $\infty$ -norm. Suppose for any  $g\in \mathcal{G}$ ,  $(z - g(x))^2\leq B,\forall (x,z)$ . Furthermore, let  $\tilde{g}$  denote the minimizer of the population squared loss  $\mathbb{E}\left[(z - g(x))^2\right]$  over  $\mathcal{G}$ , where  $z = 1[y = h^{(2)}(x)] - 1[y = h^{(1)}(x)]$ . Then for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$  over draw of  $S_{\mathrm{val}}$ , the excess risk for  $\hat{r}$  is bounded by
$$
\begin{array}{l} R (\hat {r}; h ^ {(1)}, h ^ {(2)}) - R (r ^ {*}; h ^ {(1)}, h ^ {(2)}) \\ \leq 2 \cdot \left(\underbrace {\mathbb {E} _ {x} \left[ (\tilde {g} (x) - g ^ {*} (x)) ^ {2} \right]} _ {\text {A p p r o x i m a t i o n e r r o r}} + \underbrace {4 \cdot \inf _ {\epsilon > 0} \left\{B \sqrt {\frac {2 \cdot \log \mathcal {N} (\mathcal {G} , \epsilon)}{| S _ {\mathrm {v a l}} |}} \right\}} _ {\text {E s t i m a t i o n e r r o r}} + \mathcal {O} \left(\sqrt {\frac {\log (1 / \delta)}{| S _ {\mathrm {v a l}} |}}\right)\right) ^ {1 / 2}. \\ \end{array}
$$
We provide the proof in §A.4. The first term on the right-hand side (RHS) is an irreducible approximation error, quantifying the distance between the best possible model in the class to the oracle scoring function. The second and the third terms on RHS quantify total estimation error.
4.4 Relation to Existing Work
As noted in §3.4, learning a deferral rule for a two-model cascade is closely related to existing literature in learning to defer to an expert. This in turn is a generalisation of the classical literature on learning to reject [27, 7], which refers to classification settings where one is allowed to abstain from predicting on certain inputs. The population risk here is a special case of (2), where  $h^{(2)}(x)$  is assumed to perfectly predict  $y$ . The resulting Bayes-optimal classifier is known as Chow's rule [11, 60], and exactly coincides with the deferral rule in Lemma 4.1. Plug-in estimates of this rule are thus analogous to confidence-based deferral, and have been shown to be similarly effective [53].
In settings where one is allowed to modify the training of  $h^{(1)}$ , it is possible to construct losses that jointly optimise for both  $h^{(1)}$  and  $r$  [1, 12, 60, 73, 8, 21, 36]. While effective, these are not applicable in our setting involving pre-trained, black-box classifiers. Other variants of post-hoc methods have been considered in Narasimhan et al. [49], and implicitly in Trapeznikov and Saligrama [74]; however, here we more carefully study the different possible ways of constructing these methods, and highlight when they may fail to improve over confidence-based deferral.
5 Experimental Illustration
In this section, we provide empirical evidence to support our analysis in §4.1 by considering the three failure modes in which confidence-based deferral underperforms. For each of these settings, we
(a) Standard ImageNet (100% non-dog training images)
Figure 1: Test accuracy vs deferral rate of plug-in estimates (\$3.3) for the oracle rule. Here,  $h^{(1)}$  is a MobileNet V2 trained on all ImageNet classes, and  $h^{(2)}$  is a dog specialist trained on all images in the dog synset plus a fraction of non-dog training examples, which we vary. As the fraction decreases,  $h^{(2)}$  specialises in classifying different types of dogs. By considering the confidence of  $h^{(2)}$  (Relative Confidence), one gains accuracy by selectively deferring only dog images.
(b)  $4\%$  non-dog training images
(c)  $2\%$  non-dog training images
compute deferral curves that plot the classification accuracy versus the fraction of samples deferred to the second model (which implicitly measures the overall compute cost). In line with our analysis in §4.1, post-hoc deferral rules offer better accuracy-cost trade-offs in these settings.
5.1 Confidence-Based versus Oracle Deferral
We begin by illustrating the benefit of considering confidence of the second model when constructing a deferral rule. In this experiment,  $h^{(1)}$  is a generalist (i.e., trained on all ImageNet classes), and  $h^{(2)}$  is a dog specialist trained on all images in the dog synset, plus a fraction of non-dog training examples, which we vary. There are 119 classes in the dog synset. We use MobileNet V2 [65] as  $h^{(1)}$ , and a larger EfficientNet B0 [71] as  $h^{(2)}$ . For hyperparameter details, see Appendix C.
Figure 1 shows the accuracy of confidence-based deferral (Confidence) and Relative Confidence (Equation (6)) on the standard ImageNet test set as a function of the deferral rate. We realise different deferral rates by varying the value of the deferral threshold  $c$ . In Figure 1a, the fraction of non-dog training images is  $100\%$  i.e., model 2 is also a generalist trained on all images. In this case, we observe that Relative Confidence offers little gains over Confidence.
However, in Figure 1b and Figure 1c, as the fraction of non-dog training images decreases, the non-uniformity of  $h^{(2)}$ 's error probabilities increases i.e.,  $h^{(2)}$  starts to specialise to dog images. In line with our analysis in §4.1, confidence-based deferral underperforms when model 2's error probability is highly non-uniform. That is, being oblivious to the fact that  $h^{(2)}$  specialises in dog images, confidence-based deferral may erroneously defer non-dog images to it. By contrast, accounting for model 2's confidence, as done by Relative Confidence, shows significant gains.
5.2 Confidence-Based versus Post-Hoc Deferral
From §5.1, one may construct better deferral rules by querying model 2 for its confidence. Practically, however, querying model 2 at inference time defeats the entire purpose of cascades. To that end, we now compare confidence-based deferral and the post-hoc estimators (Table 1), which do not need to invoke model 2 at inference time. We consider each of the settings from §4.1, and demonstrate that post-hoc deferral can significantly outperform confidence-based deferral. We present more experimental results in Appendix E, where we illustrate post-hoc deferral rules for  $K > 2$  models.
Post hoc model training. For a post-hoc approach to be practical, the overhead from invoking a post-hoc model must be small relative to the costs of  $h^{(1)}$  and  $h^{(2)}$ . To this end, in all of the following experiments, the post-hoc model  $g\colon \mathcal{X}\to \mathbb{R}$  is based on a lightweight, three-layer MultiLayer Perceptron (MLP) that takes as input the probability outputs from model 1. That is,  $g(x) = \mathrm{MLP}(p^{(1)}(x))$  where  $p^{(1)}(x)\in \Delta_L$  denotes all probability outputs from model 1. Learning  $g$  amounts to learning the MLP as the two base models are fixed. We train  $g$  on a held-out validation set. For full technical details of the post-hoc model architecture and training, see Appendix C. We use the objectives described in Table 1 to train  $g$ .
(a) Standard ImageNet.
(b)  $4\%$  non-dog training images.
(c)  $2\%$  non-dog training images.
(d) Standard CIFAR 100.
(g) Standard CIFAR 100.
Figure 2: Test accuracy vs deferral rate of the post-hoc approaches in Table 1 under the three settings described in §4.1: 1) specialist (row 1), 2) label noise (row 2), and 3) distribution shift. Row 1: As the fraction of non-dog training images decreases, model 2 becomes a dog specialist model. Increase in the non-uniformity in its error probabilities allows post-hoc approaches to learn to only defer dog images. Row 2: As label noise increases, the difference in the probability of correct prediction under each model becomes zero (i.e., probability tends to chance level). Thus, it is sub-optimal to defer affected inputs since model 2's correctness is also at chance level. Being oblivious to model 2, confidence-based deferral underperforms. For full details, see §5.2. Row 3: As the skewness of the label distribution increases, so does the difference in the probability of correct prediction under each model (recall the optimal rule in Proposition 3.1), and it becomes necessary to account for model 2's probability of correct prediction when deferring. Hence, confidence-based deferral underperforms.
(e) Label noise on first 10 classes.
(h) Head classes: 50.
(f) Label noise on first 25 classes.
(i) Head classes: 25.
Specialist setting. We start with the same ImageNet-Dog specialist setting used in §5.1. This time, we compare six methods: Confidence, Random, MaxProb, Diff-01, Diff-Prob, and Entropy. Random is a baseline approach that defers to either model 1 or model 2 at random; MaxProb, Diff-01, and Diff-Prob are the post-hoc rules described in Table 1; Entropy defers based on the thresholding the entropy of  $p^{(1)}$  [31, 36], as opposed to the maximum probability.
Results for this setting are presented in Figure 2 (first row). We see that there are gains from post-hoc deferral, especially in the low deferral regime: it can accurately determine whether the second model is likely to make a mistake. Aligning with our analysis, for the generalist setting (Figure 2a), confidence-based deferral is highly competitive, since  $h^{(2)}$  gives a consistent estimate of  $\mathbb{P}(y \mid x)$ .
Label noise setting. In this setting, we look at a problem with label noise. We consider CIFAR 100 dataset where training examples from pre-chosen  $L_{\mathrm{noise}} \in \{0,10,25\}$  classes are assigned a uniformly drawn label. The case of  $L_{\mathrm{noise}} = 0$  corresponds to the standard CIFAR 100 problem. We set  $h^{(1)}$  to be CIFAR ResNet 8 and set  $h^{(2)}$  to be CIFAR ResNet 14, and train both models on the noisy data. The results are shown in Figure 2 (second row). It is evident that when there is label noise, post-hoc approaches yield higher accuracy than confidence-based on a large range of deferral rates, aligning with our analysis in §4.1. Intuitively, confidence-based deferral tends to forward noisy samples to  $h^{(2)}$ , which performs equally poorly, thus leading to a waste of deferral budget. By contrast, post-hoc rules can learn to "give up" on samples with extremely low model 1 confidence.
(a) Training performance
(b) Test performance
Figure 3: Training and test accuracy of post-hoc approaches in the ImageNet-Dog specialist setting. Model 2 (EfficientNet B0) is trained with all dog images and  $8\%$  of non-dog images. Observe that a post-hoc model (i.e., Diff-01) can severely overfit to the training set and fail to generalise.
Distribution shift setting. To simulate distribution shift, we consider a long-tailed version of CIFAR 100 [40] where there are  $h \in \{100, 50, 25\}$  head classes, and  $100 - h$  tail classes. Each head class has 500 training images, and each tail class has 50 training images. The standard CIFAR 100 dataset corresponds to  $h = 100$ . Both models  $h^{(1)}$  (CIFAR ResNet 8) and  $h^{(2)}$  (CIFAR ResNet 56) are trained on these long-tailed datasets. At test time, all methods are evaluated on the standard CIFAR 100 balanced test set, resulting in a label distribution shift.
We present our results in Figure 2 (third row). In Figure  $2\mathrm{g}$ , there is no distribution shift. As in the case of the specialist setting, there is little to no gain from post-hoc approaches in this case since both models are sufficiently accurate. As  $h$  decreases from 100 to 50 (Figure 2h) and 25 (Figure 2i), there is more distribution shift at test time, and post-hoc approaches (notably Diff-  $\varpi 1$ ) show clearer gains. To elaborate, the two base models are of different sizes and respond to the distribution shift differently, with CIFAR ResNet 56 being able to better handle tail classes overall. Diff-  $\varpi 1$  is able to identify the superior performance of  $h^{(2)}$  and defer input instances from tail classes.
5.3 On the Generalisation of Post-Hoc Estimators
Despite the benefits of post-hoc approaches as demonstrated earlier, care must be taken in controlling the capacity of the post-hoc models. We consider the same ImageNet-Dog specialist setting as in the top row of Figure 2. Here, model 2 is trained on all dog images, and a large fraction of non-dog images  $(8\%)$ . Since model 2 has access to a non-trivial fraction of non-dog images, the difference in the probability of correct prediction of the two models is less predictable. We report deferral curves on both training and test splits in Figure 3. Indeed, we observe that the post-hoc method Diff-01 can overfit, and fail to generalise. Note that this is despite using a feedforward network with two hidden layers of only 64 and 16 units (see Appendix C for details on hyperparameters) to control the capacity of the post-hoc model. Thoroughly investigating approaches to increase generalisation of post-hoc models will be an interesting topic for future study.
The Bayes-optimal deferral rule we present suggests that key to optimally defer is to identify when the first model is wrong and the second is right. Based on this result, we then study a number of estimators (Table 1) to construct trainable post hoc deferral rules, and show that they can improve upon the commonly used confidence-based deferral.
While we have identified conditions under which confidence-based deferral underperforms (e.g., specialist setting, label noise), these are not exhaustive. An interesting direction for future work is to design post-hoc deferral schemes attuned for settings involving other forms of distribution shift, such as the presence of out-of-distribution samples. It is also of interest to study the efficacy of more refined confidence measures, such as those based on conformal prediction [69]. Finally, while our results have focussed on image classification settings, it would be of interest to study analogous trends for natural language processing models.
[76] Tom Veniat and Ludovic Denoyer. Learning time/memory-efficient deep architectures with budgeted super networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3492-3500, 06 2018. doi: 10.1109/CVPR.2018.00368.
[77] P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 1, pages I-I, 2001. doi: 10.1109/CVPR.2001.990517.
[78] T. Vu, M. Eder, T. Price, and J. Frahm. Any-width networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3018-3026, Los Alamitos, CA, USA, jun 2020. IEEE Computer Society. doi: 10.1109/CVPRW50498.2020.00360. URL https://doi.ieeecomputersociety.org/10.1109/CVPRW50498.2020.00360.
[79] Byron C. Wallace and Issa J. Dahabreh. Class probability estimates are unreliable for imbalanced data (and how to fix them). In 2012 IEEE 12th International Conference on Data Mining, pages 695-704, 2012. doi: 10.1109/ICDM.2012.115.
[80] Xiaofang Wang, Dan Kondratyuk, Eric Christiansen, Kris M. Kitani, Yair Movshovitz-Attias, and Elad Eban. Wisdom of committees: An overlooked approach to faster and more accurate models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=Mv02tovbs4-.
[81] Xin Wang, Yujia Luo, Daniel Crankshaw, Alexey Tumanov, Fisher Yu, and Joseph E. Gonzalez. IDK cascades: Fast deep learning by learning not to overthink. In Amir Globerson and Ricardo Silva, editors, Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pages 580-590. AUAI Press, 2018.
[82] Ji Xin, Rodrigo Nogueira, Yaoliang Yu, and Jimmy Lin. Early exiting BERT for efficient document ranking. In Proceedings of SustainNLP: Workshop on Simple and Efficient Natural Language Processing, pages 83-88, Online, November 2020. Association for Computational Linguistics.
[83] Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. DeeBERT: Dynamic early exiting for accelerating BERT inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2246-2251, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.204. URL https://aclanthology.org/2020.acl-main.204.
[84] Cha Zhang and Paul Viola. Multiple-instance pruning for learning efficient cascade detectors. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2008.
[85] Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. BERT loses patience: Fast and robust inference with early exit. In Advances in Neural Information Processing Systems, volume 33, pages 18330-18341. Curran Associates, Inc., 2020.
[86] Shlomo Zilberstein. Using anytime algorithms in intelligent systems. AI Magazine, 17(3):73, Mar. 1996. doi: 10.1609/aimag.v17i3.1232.