SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Metod Jazbec
UvA-Bosch Delta Lab
m.jazbec@uva.nl
Dan Zhang
Bosch Center for AI &
dan.zhang2@de.bosch.com
James Urquhart Allingham
jua23@cam.ac.uk
Eric Nalisnick
e.t.nalisnick@uva.nl
Modern predictive models are often deployed to environments in which computational budgets are dynamic. Anytime algorithms are well-suited to such environments as, at any point during computation, they can output a prediction whose quality is a function of computation time. Early-exit neural networks have garnered attention in the context of anytime computation due to their capability to provide intermediate predictions at various stages throughout the network. However, we demonstrate that current early-exit networks are not directly applicable to anytime settings, as the quality of predictions for individual data points is not guaranteed to improve with longer computation. To address this shortcoming, we propose an elegant post-hoc modification, based on the Product-of-Experts, that encourages an early-exit network to become gradually confident. This gives our deep models the property of conditional monotonicity in the prediction quality—an essential stepping stone towards truly anytime predictive modeling using early-exit architectures. Our empirical results on standard image-classification tasks demonstrate that such behaviors can be achieved while preserving competitive accuracy on average.
When deployed in the wild, predictive models are often subject to dynamic constraints on their computation. Consider a vision system for an autonomous vehicle: the same system should perform well in busy urban environments and on deserted rural roads. Yet the former environment often requires quick decision-making, unlike the latter. Hence, it would be beneficial for our models to be allowed to 'think' as long as its environment allows. Deploying models to resource-constrained devices has similar demands. For example, there are roughly 25,000 different computing configurations of Android devices [Yu et al., 2019], and developers often need to balance poor user experience in low-resource settings with under-performance in high-resource settings.
Anytime algorithms can ease, if not resolve, these issues brought about by dynamic constraints on computations. An algorithm is called anytime if it can quickly produce an approximate solution and subsequently employ additional resources, if available, to incrementally decrease the approximation error [Dean and Boddy, 1988, Zilberstein, 1996].<sup>1</sup> Anytime computation is especially valuable when resources can fluctuate during test time. Returning to the example of mobile devices, an anytime
<sup>1</sup>We give a complete definition in Section 2.
algorithm would be able to gracefully adapt to the computational budget of whatever class of device it finds itself on: low-performing devices will depend on intermediate (but still good) solutions, while high-performing devices can exploit the model's maximum abilities.
Early-exit neural networks (EENNs) [Teerapittayanon et al., 2016] have attracted attention as anytime predictors, due to their architecture that can generate predictions at multiple depths [Huang et al., 2018]. However, as we demonstrate, EENNs do not exhibit anytime behavior. Specifically, current EENNs do not possess conditional monotonicity, meaning that the prediction quality for individual data points can deteriorate with longer computation at test time. This lack of monotonicity leads to a decoupling of computation time and output quality—behavior that is decisively not anytime.
In this work, we improve EENNs' ability to perform anytime predictive modeling. We propose a lightweight transformation that strongly encourages monotonicity in EENNs' output probabilities as they evolve with network depth. Our method is based on a Product-of-Experts formulation [Hinton, 1999] and exploits the fact that a product of probability distributions approximates an 'and' operation. Thus by taking the product over all early-exits computed thus-far, the aggregated predictive distribution is gradually refined over time (exits) to have a non-increasing support. This transformation has an inductive bias that encourages anytime uncertainty: the first exits produce high-entropy predictive distributions—due to their large support—and the EENN becomes progressively confident at each additional exit (by decreasing the support and reallocating probability).
In our experiments, we demonstrate across a range of vision classification tasks that our method leads to significant improvements in conditional monotonicity in a variety of backbone models without re-training. Moreover, our transformation preserves the original model's average accuracy and even improves it in some cases (e.g. CIFAR-100). Thus our work allows the large pool of efficient architectures to be endowed with better anytime properties, all with minimal implementation overhead.
2 Background & Setting of Interest
Data Our primary focus is multi-class classification. Let  $\mathcal{X} \subseteq \mathbb{R}^d$  denote the feature space, and let  $\mathcal{V}$  denote the label space, which we assume to be a categorical encoding of multiple  $(K)$  classes.  $\pmb{x} \in \mathcal{X}$  denotes a feature vector, and  $y \in \mathcal{Y}$  denotes the class (1 of  $K$ ). We assume  $\pmb{x}$  and  $y$  are realizations of the random variables  $\mathbf{x}$  and  $\mathbf{y}$  with the (unknown) data distribution  $p(\mathbf{x},\mathbf{y}) = p(\mathbf{y}|\mathbf{x})p(\mathbf{x})$ . The training data sample is  $\mathcal{D} = \{(x_n,y_n)\}_{n=1}^N$ . Our experiments focus on vision tasks in which  $\pmb{x}$  is a natural image; in the appendix, we also report results on natural language.
Early-Exit Neural Networks: Model and Training Our model of interest is the early-exit neural network [Teerapittayanon et al., 2016] (EENN). For a given data-point  $\pmb{x}$ , an EENN defines  $M$  total models— $p(\mathbf{y} \mid \mathbf{x} = \pmb{x}, \pmb{\phi}_m, \pmb{\theta}_m)$ ,  $m = 1, \dots, M$ , where  $\phi_m$  denotes the parameters for the  $m$ th classifier head and  $\pmb{\theta}_m$  are the parameters of the backbone architecture. Each set of parameters  $\pmb{\theta}_m$  contains the parameters of all previous models:  $\pmb{\theta}_1 \subset \pmb{\theta}_2 \subset \dots \subset \pmb{\theta}_M$ . Due to this nested construction, each model can be thought of as a preliminary 'exit' to the full architecture defined by  $\pmb{\theta}_M$ . This enables the network's computation to be 'short-circuited' and terminated early at exit  $m$ , without evaluating later models. Training EENNs is usually done by fitting all exits simultaneously:
$$
\ell \left(\phi_ {1}, \dots , \phi_ {M}, \boldsymbol {\theta} _ {1}, \dots , \boldsymbol {\theta} _ {M}; \mathcal {D}\right) = - \sum_ {n = 1} ^ {N} \sum_ {m = 1} ^ {M} w _ {m} \cdot \log p (\mathbf {y} = y _ {n} \mid \mathbf {x} = \boldsymbol {x} _ {n}, \phi_ {m}, \boldsymbol {\theta} _ {m}) \tag {1}
$$
where  $w_{m} \in \mathbb{R}^{+}$  are weights that control the degree to which each exit is trained. The weights are most often set to be uniform across all exits. For notational brevity, we will denote  $p(\mathbf{y} = y \mid \mathbf{x} = x, \phi_{m}, \theta_{m})$  as  $p_{m}(y \mid x)$  henceforth.
Setting of Interest At test time, there are various ways to exploit the advantages of an EENN. For example, computation could be halted if an input is deemed sufficiently easy such that a weaker model (e.g.  $p_1$  or  $p_2$ ) can already make an accurate prediction, thus accelerating inference time. Such efficient computation has been the central focus of the EENNs's literature thus far [Matsubara et al., 2023] and is sometimes referred to as budgeted batch classification [Huang et al., 2018].
See Appendix B.6 for discussion of anytime regression.
However, in this work we focus exclusively on the related, yet distinct, anytime setting motivated in the Introduction. We assume computation is terminated when the current environment cannot support computing and evaluating the  $(m + 1)$ th model. These constraints originate solely from the environment, meaning the user or model developer has no control over them. We assume the model will continue to run until it either receives such an external signal to halt or the final—i.e.,  $M$ th—model is computed [Grubb and Bagnell, 2012]. Furthermore, in line with previous literature on anytime algorithms [Huang et al., 2018, Hu et al., 2019], we assume the model sees one data point at a time when deployed, i.e., the batch size is one at test time.
Anytime Predictive Models To have the potential to be called 'anytime,' an algorithm must produce a sequence of intermediate solutions. In the case of classification, an anytime predictive model must hence output a series of probability distributions over  $\mathcal{V}$  with the goal of forming better and better approximations of the true conditional data distribution  $p(\mathbf{y}|\mathbf{x})$ . Specifically, we are interested in the evolution of the conditional error for a given  $\boldsymbol{x} \in \mathcal{X}$ , defined here as  $\epsilon_t(\boldsymbol{x}) \coloneqq D(p(\mathbf{y}|\boldsymbol{x}), p_t(\mathbf{y}|\boldsymbol{x}))$  where  $D$  denotes a suitable distance or divergence (e.g., KL-divergence or total-variation), and  $p_t$  represents a model's predictive distribution after computing for time  $t \in \mathbb{R}_{\geq 0}$ . In addition to conditional, i.e., instance-level, error we can also consider the evolution of a marginal error defined as  $\bar{\epsilon}_t \coloneqq \mathbb{E}_{\boldsymbol{x} \sim p(\mathbf{x})}[\epsilon_t(\boldsymbol{x})]$ . For the remainder of this section, we will use  $\epsilon_t$  to refer to both versions of the error and revisit their difference in Section 3. Since this paper focuses on anytime EENNs, we use the early-exit index  $m$  as the time index  $t$ .
Properties of Anytime Models Zilberstein [1996] enumerates properties that an algorithm should possess to be anytime. Below we summarize Zilberstein [1996]'s points that are relevant to predictive modeling with EENNs, ordering them in a hierarchy of necessity:
1. Interruption: The most basic requirement is that the model can be stopped and produce an answer at any time. Traditional neural networks do not satisfy interruptibility since they produce a single output after evaluating all layers. However, EENNs do by returning the softmax probabilities for the  $m$ th model if the  $(m + 1)$ th model has not been computed yet.
2. Monotonicity: The quality of the intermediate solutions should not decrease with computation time. Or equivalently, expending more compute time should guarantee performance does not degrade. For EENNs, monotonicity is achieved if the error  $\epsilon_{m}$  is non-increasing across early-returns:  $\epsilon_{1} > \ldots >\epsilon_{M}$ .
3. Consistency: The quality of the intermediate solutions should be correlated with computation time. For EENNs, consistency is of most concern when the amount of computation varies across sub-models. For example, if computing the  $(m + 1)$ th model requires evaluating  $10\times$  more residual blocks than the  $m$ th model required, then a consistent architecture should produce a roughly  $10\times$  better solution ( $\epsilon_{m + 1}\approx \epsilon_m / 10$ ).
4. Diminishing Returns: The difference in quality between consecutive solutions is gradually decreasing, with the largest jumps occurring early, e.g.  $(\epsilon_{1} - \epsilon_{2}) \gg (\epsilon_{M-1} - \epsilon_{M})$ . Monotonicity and consistency need to be satisfied in order for this stronger property to be guaranteed.
For a traditional EENN (trained with Equation 1), property #1 is satisfied by construction (by the presence of early-returns). Property #2 is not explicitly encouraged by the typical EENN architecture or training objective. Hence, we begin the next section by investigating if current EENNs demonstrate any signs of monotonicity. Properties #3 and #4 are left to future work since, as we will demonstrate, achieving property #2 is non-trivial and is thus the focus of the remainder of this paper.
Estimating Anytime Properties The error sequence  $\{\epsilon_m\}_{m=1}^M$  is essential for studying the anytime properties of EENNs. We can equivalently consider the evolution of prediction quality measures  $\gamma_m$  and aim for a model that exhibits increasing quality, i.e.,  $\gamma_m \leq \gamma_{m+1}, \forall m$ . For the remainder of the paper, we will work with the notion of prediction quality unless otherwise stated. Observe that each  $\gamma_m$  (and equivalently,  $\epsilon_m$ ) is always unknown, as it depends on the unknown data distribution  $p(\mathbf{y} \mid \mathbf{x})$ . Consequently, we must rely on estimates  $\hat{\gamma}_m$ . In the presence of a labeled hold-out dataset, we can use ground-truth labels  $y^* \in \mathcal{V}$  and consider the correctness of the prediction as our estimator:
3While EENNs are the focus of our work, they are not the only model class that can be used for anytime prediction. In theory, any model that yields a sequence of predictions could be considered.
$\hat{\gamma}_{m}^{c}(\pmb{x}) \coloneqq [\arg \max_{y} p_{m}(y \mid \pmb{x}) = y^{*}]$  where  $[\cdot]$  is the Iverson bracket. However, such an estimator provides a crude signal (0 or 1) on a conditional level (per data point). As such, it is not congruent with conditional monotonicity or consistency. Due to this, we focus primarily on the probability of the ground-truth class  $\hat{\gamma}_{m}^{p}(\pmb{x}) \coloneqq p_{m}(y^{*} \mid \pmb{x})$  when examining anytime properties at the conditional level. Finally, estimating marginal prediction quality (error) requires access to  $p(\mathbf{x})$ . Following common practice, we approximate  $\mathbb{E}_{\pmb{x} \sim p(\mathbf{x})}$  by averaging over the hold-out dataset. Averaging  $\hat{\gamma}_{m}^{c}(\pmb{x})$  and  $\hat{\gamma}_{m}^{p}(\pmb{x})$  corresponds to the test accuracy and the average ground-truth probability at the  $m$ th exit, respectively.
3 Checking for Monotonicity in Early-Exit Neural Networks
In this section, we take a closer look at state-of-the-art early-exit neural networks and their monotonicity, a key property of anytime models. Our results show that EENNs are marginally monotone but lack conditional monotonicity in the prediction quality. For ease of presentation, we focus here on a particular EENN (Multi-Scale Dense Net (MSDNet); Huang et al. [2018]) and dataset (CIFAR-100). Section 6 shows that this observation generalizes across other EENNs and datasets.
Marginal Monotonicity The left plot of Figure 1 reports MSDNet's test accuracy (blue) on CIFAR-100 as a function of the early exits. Accuracy has been the primary focus in the anytime EENN literature [Huang et al., 2018, Hu et al., 2019], and as expected, it increases at each early exit  $m$ . Thus MSDNet clearly exhibits marginal monotonicity: monotonicity on average over the test set. The same figure also shows that the average ground-truth probability (orange) increases at each  $m$ , meaning that MSDNet demonstrates marginal monotonicity in this related quantity as well.
Conditional Monotonicity Yet for real-time decision-making systems, such as autonomous driving, it is usually not enough to have marginal monotonicity. Rather, we want conditional monotonicity: that our predictive model will improve instep with computation for every input. If conditional monotonicity is achieved, then marginal monotonicity is also guaranteed. However, the converse is not true, making conditional monotonicity the stronger property.
To check if MSDNet demonstrates conditional monotonicity, examine the middle plot of Figure 1. Here we plot the probability of the true label, i.e.,  $\hat{\gamma}_m^p (\pmb {x})$ , as a function of the early exits for 10 randomly sampled test points (again from CIFAR-100). We see that—despite MSDNet being marginally monotonic—it is quite unstable in the probabilities it produces per exit. For example, the purple line exhibits confidence in the correct answer  $(\sim 75\%)$  at the second exit, drops to near zero by the fourth exit, becomes confident again  $(\sim 80\%)$  at exits 5 and 6, and finally falls to near zero again at the last exit. Thus MSDNet clearly does not exhibit the stronger form of monotonicity.
We next quantify the degree to which MSDNet is violating conditional monotonicity. First, we define the maximum decrease in the ground-truth probability as  $\mathrm{MPD}^*(\boldsymbol{x}) \coloneqq \max_{m' > m} \left\{ \max (\hat{\gamma}_m^p(\boldsymbol{x}) - \hat{\gamma}_{m'}^p(\boldsymbol{x}), 0) \right\}$ . For various thresholds  $\tau \in [0,1]$ , we count how many test instances exhibit a probability decrease that exceeds the given threshold:  $N_\tau \coloneqq \sum_n [\mathrm{MPD}^*(\boldsymbol{x}_n) \geq \tau]$ . The results are depicted in the right panel of Figure 1. We see that for  $\sim 25\%$  of test points, the ground-truth probability decreases by at least 0.5 at some exit. In a separate analysis (see Appendix B.1.2), when examining the correctness of prediction measure  $\hat{\gamma}_m^c(\boldsymbol{x})$ , we find that  $\sim 30\%$  of test examples exhibit decreasing trajectories across early exits. A similar finding was reported by Kaya et al. [2019]. Both observations demonstrate that MSDNet, a state-of-the-art EENN, does not exhibit conditional monotonicity nor is it close to achieving it.
4 Transformations for Conditional Monotonicity
In light of the aforementioned challenges with current EENNs in the anytime setting, we explore potential lightweight modifications that could encourage, or even enforce, conditional monotonicity. We first suggest a strong baseline for encouraging monotonicity, taking inspiration from Zilberstein [1996]. We then introduce our primary methodology, which is based on product ensembles.
4The Iverson bracket [cond] is 1 if cond is true and 0 otherwise.
This implies that the model incorrectly classifies the data point after correctly predicting it at some of the earlier exits:  $\hat{\gamma}_m^c (\pmb {x}) = 1$  and  $\hat{\gamma}_{m'}^c (\pmb {x}) = 0$  for some  $m^{\prime} > m$ .
Figure 1: Multi-Scale Dense Network (MSDNet; Huang et al. [2018]) results on the test set of CIFAR-100. Left: test accuracy and average ground-truth probability at each early exit, the two measures of marginal monotonicity we study in this work. Middle: ground-truth probability trajectories (i.e.,  $\{\hat{\gamma}_m^p (\pmb {x})\}_{m = 1}^M$  ) for 10 random test data points. Right: aggregation of analysis on conditional monotonicity for MSDNet. Specifically, we compute the maximum decrease in ground-truth probability for each test data point and then plot the percentage of test examples where the drop exceeds various thresholds. A red dashed line denotes the performance of an oracle model with perfect conditional monotonicity, i.e., none of the data points exhibit any decrease in the ground-truth probability.
4.1 Caching Anytime Predictors
As noted by Zilberstein [1996], conditional monotonicity can be encouraged by caching the best prediction made so far and returning it (instead of the latest prediction). Yet this requires the ability to measure the prediction quality of intermediate results. While previously introduced quantities, such as ground-truth probability  $\hat{\gamma}_m^p (\pmb {x})$ , can serve to examine anytime properties before deploying the model, their reliance on the true label  $y^{*}$  renders them unsuitable for measuring prediction quality at test time. As a proxy, we propose using the EENN's internal measure of confidence. We found that simply using the softmax confidence performs well in practice. At an exit  $m$ , we compute  $C(p_{m},\pmb {x})\coloneqq \max_{y}p_{m}(y|\pmb {x})$  and overwrite the cache if  $C(p_j,\pmb {x}) < C(p_m,\pmb {x})$ , where  $j$  is the exit index of the currently cached prediction. We call this approach Caching Anytime (CA) and consider it as a (strong) baseline. While CA is rather straightforward, it has been thus far overlooked in the anytime EENN literature, as the standard approach is to return the latest prediction [Huang et al., 2018, Hu et al., 2019]. Note that if the anytime model displays conditional monotonicity to begin with, applying CA has no effect since the most recent prediction is always at least as good as the cached one.
4.2 Product Anytime Predictors
Our proposed method for achieving conditional monotonicity in EENNs is motivated by the following idealized construction.
Guaranteed Conditional Monotonicity via Hard Product-of-Experts Consider an EENN whose sub-classifiers produce 'hard' one-vs-rest probabilities: each class has either probability 0 or 1 (before normalization). Now define the predictive distribution at the  $m$ th exit to be a Product-of-Experts [Hinton, 1999] (PoE) ensemble involving the current and all preceding exits:
$$
p _ {\Pi , m} (y \mid \boldsymbol {x}) := \frac {1}{Z _ {m}} \prod_ {i = 1} ^ {m} [ f _ {i} (\boldsymbol {x}) _ {y} > b ] ^ {w _ {i}}, \quad Z _ {m} = \sum_ {y ^ {\prime} \in \{1, \dots , K \}} \prod_ {i = 1} ^ {m} [ f _ {i} (\boldsymbol {x}) _ {y ^ {\prime}} > b ] ^ {w _ {i}} \tag {2}
$$
where  $b$  is a threshold parameter,  $[\cdot ]$  is an Iverson bracket,  $f_{i}(\pmb {x})\in \mathbb{R}^{K}$  represents a vector of logits at ith exit, and  $w_{i}\in \mathbb{R}^{+}$  are ensemble weights. Assuming that a class  $y$  is in the support of the final classifier for a given  $\pmb{x}$ , i.e.  $p_{\Pi ,M}(y|\pmb {x}) > 0$ , then such a model is guaranteed to be monotone in the probability of  $y$ :
$$
p _ {\Pi , m} (y \mid \boldsymbol {x}) \leq p _ {\Pi , m + 1} (y \mid \boldsymbol {x}), \forall m. \tag {3}
$$
The formal proof can be found in Appendix A. Consequently, if the true class  $y^{*}$  is in the final support, an EENN with this construction is conditionally monotonic in the ground-truth probability  $\hat{\gamma}_m^p(\boldsymbol{x})$  (as defined in Section 2).
The intuition behind this construction is that at each exit, the newly computed ensemble member casts a binary vote for each class. If a class has received a vote at every exit thus far, then it remains in the support of  $p_{\Pi,m}(\mathbf{y} \mid \mathbf{x})$ . Otherwise, the label drops out of the support and never re-enters. At each
exit, the set of candidate labels can only either remain the same or be reduced, thus concentrating probability in a non-increasing subset of classes. This construction also has an underconfidence bias: the probability of all labels starts small due to many labels being in the support at the earliest exits. Consequently, the model is encouraged to exhibit high uncertainty early on and then to become progressively confident at subsequent exits.
A Practical Relaxation using ReLU Although the above construction provides perfect conditional monotonicity (for labels in the final support), it often results in a considerable decrease in marginal accuracy (see Appendix B.2). This is not surprising since the  $0 - 1$  probabilities result in 'blunt' predictions. At every exit, all in-support labels have equal probability, and thus the model must select a label from the support at random in order to generate a prediction. The support is often large, especially at the early exits, and in turn the model devolves to random guessing.
To overcome this limitation, first notice that using  $0 - 1$  probabilities is equivalent to using a Heaviside function to map logits to probabilities. In turn, we propose to relax the parameterization, replacing the Heaviside with the ReLU activation function. The ReLU has the nice property that it preserves the rank of the 'surviving' logits while still 'nullifying' classes with low logits. Yet this relaxation comes at a price: using ReLUs allows conditional monotonicity to be violated. However, as we will show in Section 6, the violations are rare and well worth the improvements in accuracy. Moreover, the number of violations can be controlled by applying an upper limit to the logits, as detailed in Appendix B.8.
Therefore our final Product Anytime (PA) method for conditional monotonicity is written as: