SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
p _ {\mathrm {P A}, m} (y \mid \boldsymbol {x}) := \frac {1}{Z _ {m}} \prod_ {i = 1} ^ {m} \max  \left(f _ {i} (\boldsymbol {x}) _ {y}, 0\right) ^ {w _ {i}}, \quad Z _ {m} = \sum_ {y ^ {\prime} \in \{1, \dots , K \}} \prod_ {i = 1} ^ {m} \max  \left(f _ {i} (\boldsymbol {x}) _ {y ^ {\prime}}, 0\right) ^ {w _ {i}} \tag {4}
$$
where  $\max (\cdot ,\cdot)$  returns the maximum of the two arguments. We set the ensemble weights to  $w_{i} = i / M$  since we expect later experts to be stronger predictors. Note that PA's predictive distribution may degenerate into a zero distribution such that  $p_{\mathrm{PA},m}(y|\boldsymbol {x}) = 0,\forall y\in \mathcal{V}$ . This occurs when the ensemble's experts have non-overlapping support and are thus unable to arrive at a consensus prediction. If a prediction is required, we fall back on the softmax predictive distribution based on the logits of the classifier at the current exit.
Learning and Post-Hoc Application Equation 4 can be used for training an EENN directly via the negative log-likelihood. However, one can also take pre-trained EENNs (e.g., as open-sourced by the original authors) and apply Equation 4 post-hoc. For this post-hoc application, the EENN's logits are input into Equation 4 rather than into the usual softmax transformation. As we will demonstrate in the experiments, post-hoc application endows the model with conditional monotonicity without any substantial degradation to the original model's accuracy. In cases such as CIFAR-100, we find that our method actually improves the accuracy of the pre-trained model. The only metric we find to be sensitive to training from scratch vs
post-hoc application is confidence calibration, with the former providing better calibration properties. This result will be demonstrated in Figure 6. Thus, we currently recommend using our method via post-hoc application, as it requires minimal implementation overhead (about three lines of code) and allows the user to easily toggle between anytime and traditional behavior.
For a visualization of how our post-hoc PA affects probability trajectories, see Figure 2. PA manages to rectify the backbone EENN's non-monotone behavior to a large extent. As a concrete example of how this can help with anytime predictions, we can consider again the scenario of Android
Figure 2: Ground-truth probability trajectories (i.e.,  $\{\hat{\gamma}_m^p (\boldsymbol {x})\}_{m = 1}^M)$  for 10 random test data points after applying our PA method. For comparison, MSDNet's trajectories from Figure 1 are reproduced in the background.
In the most widely used softmax parameterization of the categorical likelihood, the exponential function is used when transforming logits into probabilities.
phones introduced in Section 1. Deploying the original MSDNet might result in certain data points receiving inferior predictions on a higher-spec device compared to a device with limited computational capabilities. However, if we apply our PA to an MSDNet, such inconsistencies become far less likely due to the model's monotonic behavior.
Anytime Algorithms [Dean and Boddy, 1988, Zilberstein, 1996] have long been studied outside the context of neural networks. For instance, Wellman and Liu [1994] considered Bayesian networks in anytime settings, while Grubb and Bagnell [2012] proposed an anytime Ada-Boost algorithm [Bishop, 2007] and also studied Haar-like features [Viola and Jones, 2001] for anytime object detection. More recently, modern neural networks have been used in models with only marginal anytime properties [Huang et al., 2018, Hu et al., 2019, Liu et al., 2022]. Our research improves upon this work by bringing these models a step closer to having per-data-point anytime prediction, a much stronger form of anytime computation.
Early-Exit Neural Networks [Teerapittayanon et al., 2016, Yang et al., 2020, Han et al., 2022a] have been investigated for applications in computer vision [Huang et al., 2018, Kaya et al., 2019] and natural language processing [Schwartz et al., 2020, Schuster et al., 2021, Xu and McAuley, 2023]. A wide variety of backbone architectures has been explored, encompassing recurrent neural networks [Graves, 2016], convolutional networks [Huang et al., 2018], and transformers [Wang et al., 2021]. Furthermore, ensembling of EENNs intermediate predictions has been considered [Qendro et al., 2021, Wolczyk et al., 2021, Meronen et al., 2023], primarily to enhance their uncertainty quantification capabilities. Our objective is to leverage the recent advancements in early-exit networks and make them suitable for true (conditional) anytime predictive modeling.
Related to our critique of current EENNs, Kaya et al. [2019] and Wolczyk et al. [2021] also observed that prediction quality can decline at later exits, calling the phenomenon 'overthinking.' These works address the problem via methods for early termination. Our work aims to impose monotonicity on the ground-truth probabilities, which eliminates 'overthinking' by definition. We further examine the connection to Kaya et al. [2019] and Wolczyk et al. [2021] in Appendix B.5.
Product-of-Experts [Hinton, 1999, 2002] ensembles have been extensively studied in the generative modeling domain due to their ability to yield sharper distributions compared to Mixture-of-Experts ensembles [Wu and Goodman, 2018, Huang et al., 2022]. However, in the supervised setting, PoE ensembles have received relatively little attention [Pignat et al., 2022]. PoE is often applied only post-hoc, likely due to training difficulties [Cao and Fleet, 2014, Watson and Morimoto, 2022].
We conduct two sets of experiments. First, in Section 6.1, we verify that our method (PA) maintains strong average performance while significantly improving conditional monotonicity in state-of-the-art EENNs making them more suitable for the anytime prediction task. To check that our findings generalize across data modalities, we also conduct NLP experiments and present the results in Appendix B.4. In the second set of experiments, detailed in Section 6.2, we move beyond prediction quality and shift focus to monotonicity's effect on uncertainty quantification in the anytime setting. Lastly, we point out some limitations in Section 6.3.
Datasets We consider CIFAR-10, CIFAR-100 [Krizhevsky et al., 2009], and ILSVRC 2012 (ImageNet; Deng et al. [2009]). The CIFAR datasets each contain 50k training and 10k test images, while ImageNet is a larger dataset with 1.2M training and 50k test instances. The three datasets consist of 10, 100, and 1000 classes, respectively.
Baselines We use four state-of-the-art EENNs as baselines and backbone architectures. The first is the Multi-Scale Dense Network (MSDNet; Huang et al. [2018]), which consists of stacked convolutional blocks. To maximize performance at each early-exit, the authors introduce: (1) feature maps on multiple scales, ensuring that coarse-level features are available all throughout the network,
Our code is publicly available at https://github.com/metodj/AnytimeClassification.
Figure 3: Test accuracy on the CIFAR and ImageNet datasets. Our PA method maintains a competitive performance compared to the baseline models. For each model and dataset, we use the same number of early exits as proposed by the authors. We plot the average accuracy from  $n = 3$  independent runs. To enhance the readability of the plot, we omit the error bars here as well as the results of the L2W model and report them in Appendix B.11.
and (2) dense connectivity to prevent earlier classifiers from negatively impacting later ones. The second baseline is the IMTA network [Li et al., 2019], which shares MSDNet's architecture but is trained with regularization that improves collaboration across the classifiers of each exit. The third baseline is the L2W model [Han et al., 2022b], which enhances MSDNet by learning to weight easy and hard samples differently at each exit during training. The intuition is that easier samples should contribute more to the fitting of earlier exits, while harder samples should be more influential for later exits. Lastly, we consider the Dynamic Vision Transformer (DViT; Wang et al. [2021]), which is composed of stacked transformer networks that differ in the number of patches into which they divide an input image. We report results for the T2T-ViT-12 instantiation of DViT. During testing, all baselines employ a softmax parameterization for their categorical likelihood and solely rely on the most recent prediction at each early exit.
6.1 Anytime Prediction with Conditional Monotonicity
Test Accuracy We first verify that post-hoc application of our PA transformation preserves the original model's test accuracy. Maintaining test accuracy is essential, as simply examining conditional monotonicity in isolation does not provide a complete picture. For instance, a model that predicts a ground-truth probability of zero at every exit would exhibit perfect conditional monotonicity while being useless in practice. Figure 3 reports accuracy per exit for the three backbones models, comparing the original model  $(\text{一})$  to our proposals PA  $(\text{一} -)$  and CA  $(\cdots)$ . We find that our monotonicity methods do not significantly degrade test accuracy in any case. In some instances, such as the MSDNet results on CIFAR-100 (middle, blue), our PA even improves accuracy. Comparing our proposals, PA tends to outperform CA in accuracy, although the advantage is often slight.
Conditional Monotonicity Next, we quantify the degree to which each model is conditionally monotonic, again using the maximum probability decrease metric  $(\mathrm{MPD}^*)$  used in Figure 1 (right). Figure 4 reports the percentage of test cases that exhibit a drop in the ground-truth probability at any exit. Firstly, we see that CA and PA drastically improve monotonicity in all cases. Moreover, PA is clearly superior to CA for CIFAR-100 and ImageNet. On CIFAR-10, PA and CA perform on par for smaller decrease thresholds, with PA outperforming CA again for larger thresholds. Notably, on ImageNet, our PA method does not exhibit a drop in ground-truth probability larger than 0.01 for any test example for all three models.
From the results in Figures 3 and 4, one might wonder if a simpler change to the backbone EENN could lead to similar improvements in conditional monotonicity as our PA. To test this, we looked at 'vanilla' early-exit ensembles where predictive distributions from different exits are combined using either mixture-of-experts (MoE) or product-of-experts (PoE), see Appendix B.2 for the results. While using ensembles is beneficial, it is insufficient to achieve improvements in monotonicity similar to those of PA. This highlights the importance of the activation function (that maps logits
Max Ground-Truth  $(y^{*})$  Probability Decrease
Figure 4: The  $\%$  of test examples with a ground-truth probability drop exceeding a particular threshold. Our PA significantly improves the conditional monotonicity of ground-truth probabilities across various datasets and backbone models. To better illustrate the different behavior of various methods, a log scale is used for the y-axis. We report average monotonicity together with one standard deviation based on  $n = 3$  independent runs. For DViT [Wang et al., 2021], we were unable to perform multiple independent runs, and report the results for various models instantiations instead, see Appendix B.11. Note that for PA, the monotonicity curves on ImageNet (almost perfectly) overlap at  $y = 0$  for all backbone models considered. We also report results for the L2W model in Appendix B.11, which are largely the same as for MSDNet.
to probabilities) when it comes to achieving more monotone behavior. We additionally explored if calibrating the backbone EENN would be enough to make the underlying model monotone. While post-hoc calibrating every exit does help with monotonicity, it does not perform as well as PA; see Appendix B.9.
Monotonicity in Correctness Our PA approach yields close to perfect conditional monotonicity when considering probability as the estimator of prediction quality. Yet it is also worth checking if other estimators are monotonic, such as the correctness of the prediction, i.e.,  $\hat{\gamma}_m^c (\pmb {x})$ . For correctness, we observe similar improvements in conditional monotonicity; see Appendix B.1.2 for a detailed analysis. For example, on CIFAR-100, MSDNet 'forgets' the correct answer in  $\sim 30\%$  of test cases, whereas applying post-hoc PA reduces these monotonicity violations to  $\sim 13\%$  of cases.
6.2 Anytime Uncertainty
We also examine the uncertainty quantification abilities of these anytime models. Again consider Figure 2, the probability trajectories obtained from post-hoc PA. The probabilities start low at first few exits, indicating high uncertainty in the early stages of anytime evaluation. The left plot of Figure 5 further supports this insight, showing that the average entropy of the PA predictive distributions is high and reduces at each early exit. Lastly, in the right plot, we present the average size of conformal sets at each exit. The set size reflects the model's predictive uncertainty, with larger sets indicating more uncertainty. Thus we see that PA is best at uncertainty quantification: it is quite uncertain at the earliest exits (set size of  $\sim 11$ ) but grows twice as confident by the last exit (set size of  $\sim 4$ ). CA and regular MSDNet keep a relatively constant set size by comparison, only varying between
Figure 5: Uncertainty measures for the MSDNet model on the CIFAR-100. Left: average test entropy of (categorical) predictive distributions for each early exit. Right: mean size of conformal sets at each exit point. To obtain conformal scores, we used Regularized Adaptive Predictive Sets algorithm [RAPS; Angelopoulos et al., 2021].  $20\%$  of test examples were used as a hold-out set to calculate conformal quantiles at each early exit.
Conformal sets are prediction sets over  $\mathcal{V}$  that are marginally guaranteed to contain the ground-truth label (without making any assumptions on the underlying data distribution). For more on conformal sets, see Shafer and Vovk [2008].
11 and 7.5. Hence, the monotonicity properties of PA result in an appealing link between uncertainty and computation time; and we term this property as anytime uncertainty. This is an appropriate inductive bias for anytime predictive models since, intuitively, making quicker decisions should entail a higher level of uncertainty. Lastly, in Appendix B.7, we present results of conditional level experiments where we find that our PA yields significantly more monotone (i.e., non-increasing) uncertainty patterns.
Label Sets with Small Cardinality We observe that PA's performance in terms of conditional monotonicity improves as the number of classes  $|\mathcal{V}|$  increases. For instance, PA yields fully monotone trajectories on ImageNet, but not for datasets with smaller number of classes like CIFAR-10, where we detect small violations. This finding is further supported by our NLP experiments in Appendix B.4, where  $|\mathcal{V}|$  is smaller compared to the image classification datasets considered in this section. Based on this finding, we recommend using the CA approach over PA for scenarios in which  $|\mathcal{V}| < 5$ .
Calibration Gap Due to its aforementioned underconfidence bias, post-hoc PA generates poorly calibrated probabilities during the initial exits, as illustrated in the left plot of Figure 6. Yet this is primarily observed in expected calibration error (ECE) and related metrics. It disappears when considering some alternative metrics to ECE, as shown in the right plot of Figure 12. Finetuning with PA as an optimization objective, as opposed to applying it post-hoc, closes the calibration gap in most cases, albeit at some cost to accuracy and monotonicity. See Figure 6 (right) for concrete examples of how PA with finetuning affects ground-truth probability trajectories. In addition, we experimented with adaptive thresholding in PA, using different thresholds  $b$  for distinct data points, rather than employing 0 for all data points as with ReLU, and found that it can also be beneficial in terms of calibration. For more details on finetuning and adaptive thresholding, as well as a discussion on the relevance of ECE-like metrics in the anytime setting, see Appendix B.3.
Figure 6: Calibration analysis for MSDNet on CIFAR-100. Left: expected calibration error (ECE) during anytime evaluation. Right: Ground-truth probability trajectories for 10 random test data points after applying our PA with finetuning (PA-FT) method. For comparison, MSDNet's trajectories from Figure 1 are reproduced in the background.
We have demonstrated that expending more time during prediction in current early-exit neural networks can lead to reduced prediction quality, rendering these models unsuitable for direct application in anytime settings. To address this issue, we proposed a post-hoc modification based on Product-of-Experts and showed through empirical and theoretical analysis that it greatly improves conditional monotonicity with respect to prediction quality over time while preserving competitive overall performance.
In future work, we aim to further improve EENNs by imbuing them with other critical anytime properties, such as diminishing returns and consistency. Moreover, it would be interesting to study the impact of monotonicity in other settings, such as budgeted batch classification. We report some preliminary results in Appendix B.10, finding the monotonicity to be less beneficial there compared to the anytime setting, which was the focus of our work. We also plan to delve deeper into the concept of anytime uncertainty. Although we have (briefly) touched upon it in this study, we believe a more comprehensive and formal definition is necessary for the development of robust and reliable anytime models.
We thank Alexander Timans and Rajeev Verma for helpful discussions. We are also grateful to the anonymous reviewers who helped us improve our work with their constructive feedback. MJ and EN are generously supported by the Bosch Center for Artificial Intelligence. JUA acknowledges funding from the EPSRC, the Michael E. Fisher Studentship in Machine Learning, and the Qualcomm Innovation Fellowship.
J. U. Allingham and E. Nalisnick. A Product of Experts Approach to Early-Exit Ensembles. Technical report, 2022.
A. N. Angelopoulos, S. Bates, M. I. Jordan, and J. Malik. Uncertainty sets for image classifiers using conformal prediction. In 9th International Conference on Learning Representations, ICLR, 2021.
J. Antorán, J. U. Allingham, and J. M. Hernández-Lobato. Depth uncertainty in neural networks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, 2020.
C. M. Bishop. Pattern recognition and machine learning, 5th Edition. Information science and statistics. Springer, 2007. ISBN 9780387310732.
Y. Cao and D. J. Fleet. Generalized product of experts for automatic and principled fusion of gaussian process predictions. CoRR, 2014. URL http://arxiv.org/abs/1410.7827.
E. Daxberger, A. Kristiadi, A. Immer, R. Eschenhagen, M. Bauer, and P. Hennig. Laplace latex - effortless bayesian deep learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems, 2021.
T. L. Dean and M. S. Boddy. An analysis of time-dependent planning. In Proceedings of the 7th National Conference on Artificial Intelligence. AAAI Press / The MIT Press, 1988.
J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
A. Graves. Adaptive computation time for recurrent neural networks. CoRR, 2016. URL http://arxiv.org/abs/1603.08983.
A. Grubb and D. Bagnell. Speedboost: Anytime prediction with uniform near-optimality. In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, AISTATS, 2012.
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML, 2017.
Y. Han, G. Huang, S. Song, L. Yang, H. Wang, and Y. Wang. Dynamic neural networks: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 2022a.
Y. Han, Y. Pu, Z. Lai, C. Wang, S. Song, J. Cao, W. Huang, C. Deng, and G. Huang. Learning to weight samples for dynamic early-exiting networks. In Computer Vision - ECCV. Springer, 2022b.
G.E.HintonPRODUCTsofexperts.Technical report,1999.
G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput., 2002.
H. Hu, D. Dey, M. Hebert, and J. A. Bagnell. Learning anytime predictions in neural networks via adaptive loss balancing. In The Thirty-Third Conference on Artificial Intelligence, AAAI. AAAI Press, 2019.
M. P. Wellman and C. Liu. State-space abstraction for anytime evaluation of probabilistic networks. In UAI '94: Proceedings of the Tenth Annual Conference on Uncertainty in Artificial Intelligence, 1994.
M. Wolczyk, B. Wojcik, K. Balazy, I. T. Podolak, J. Tabor, M. Smieja, and T. Trzcinski. Zero time waste: Recycling predictions in early exit neural networks. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems, 2021.
M. Wu and N. D. Goodman. Multimodal generative models for scalable weakly-supervised learning. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, 2018.
C. Xu and J. J. McAuley. A survey on dynamic neural networks for natural language processing. In *Findings of the Association for Computational Linguistics: EACL*, 2023.
L. Yang, Y. Han, X. Chen, S. Song, J. Dai, and G. Huang. Resolution adaptive networks for efficient inference. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
J. Yu, L. Yang, N. Xu, J. Yang, and T. S. Huang. Slimmable neural networks. In 7th International Conference on Learning Representations, ICLR, 2019.
S. Zilberstein. Using anytime algorithms in intelligent systems. AI Magazine, 1996.
- We provide a proof of conditional guarantees in EENNs for (hard) PoE in Appendix A.
- We present conditional monotonicity results using alternative estimators of performance quality  $\gamma_{m}$  in Appendix B.1.
- We conduct an ablation study for our PA model in Appendix B.2.
- We propose two modifications to our PA that help with reducing the calibration gap in Appendix B.3.
- We report results of NLP experiments in Appendix B.4.
- We delve into the connections between conditional monotonicity and previously introduced concepts in EENNs literature in Appendix B.5.
- We discuss anytime regression and deep ensembles in Appendix B.6.
- We present conditional monotonicity results using uncertainty measures in Appendix B.7.
- We propose a technique for controlling the violations of conditional monotonicity in PA in Appendix B.8.
- We study the impact of calibration on the conditional monotonicity in Appendix B.9.
- We provide budgeted batch classification results in Appendix B.10.
- We provide additional plots in Appendix B.11.
In the following, we provide a proof for conditional monotonicity of (hard) Product-of-Experts ensembles presented in Section 4.2.
Proposition. Let  $p_{\Pi,m}(\mathbf{y} \mid \mathbf{x})$  represent the Product-of-Experts ensemble of the first  $m$  early-exits. Furthermore, let  $y \in \mathcal{Y}$  denote any class that is present in the support of the full-NN product distribution, i.e.,  $p_{\Pi,M}(y \mid \mathbf{x}) > 0$ . Under the assumption that the Heaviside activation function is used to map logits to probabilities in each  $p_m(y \mid \mathbf{x})$ , it follows that:
$$
p _ {\Pi , m} (y \mid \boldsymbol {x}) \leq p _ {\Pi , m + 1} (y \mid \boldsymbol {x})
$$
for every early-exit  $m$  and for every input  $\pmb{x}$
Proof. Using the definition of the PoE ensemble and that of the Heaviside activation function, the predictive likelihood at the  $m$ th early exit can be written as