SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Specifically, we employ an early-exit model from Schwartz et al. [2020], in which the BERT model is adapted for accelerated inference. The considered datasets are (1) IMDB, a binary sentiment classification dataset; (2) AG, a news topic identification dataset with  $|\mathcal{Y}| = 4$ ; and (3) SNLI, where the objective is to predict whether a hypothesis sentence agrees with, contradicts, or is neutral in relation to a premise sentence  $(|\mathcal{Y}| = 3)$ . We adhere to the same data preprocessing and model training procedures as outlined in Schwartz et al. [2020].
To account for the smaller number of classes in these NLP datasets, we shift logits for a (small) constant prior to applying ReLU activation function. Hence the PA likelihood at  $m$ th exit has the form (before applying product ensembling):
$$
p _ {m} (y \mid \boldsymbol {x}) = \max  \left(f _ {m} (\boldsymbol {x}) _ {y} + 1 / C, 0\right) ^ {w _ {i}}, \tag {5}
$$
where  $C$  represents the cardinality of  $\mathcal{V}$ . This helps with reducing the possibility of PA collapsing to a zero distribution.
The results are depicted in Figure 17. Similar to our image classification experiments discussed in Section 6.1, we find that our proposed modifications yield comparable overall accuracy, as shown in the left plot. Considering conditional monotonicity (middle plot), at least one of our two suggested methods outperforms the baseline model on every dataset. However, the degree of improvement in monotonicity is considerably smaller in comparison to our image classification experiments, with the PA model even failing to yield consistent improvement on certain datasets, such as IMDB. We attribute this not to the change in data modality, but rather to the fewer number of classes in the NLP datasets used in this section as compared to the image datasets used in Section 6.1.
Figure 17: NLP experiments results using the BERT model from Schwartz et al. [2020] as the backbone. Despite the findings being qualitatively similar to those presented in Figures 3 and 4 for image classification datasets, the enhancement in terms of conditional monotonicity provided by our proposed methods is less pronounced for the NLP classification tasks considered in this section.
B.5 Overthinking and Hindsight Improvability
In this section, we delve deeper into concepts that have been previously introduced in EENN literature and that share a connection with the idea of conditional monotonicity, which is the focus of our study.
In Kaya et al. [2019], the authors discovered that EENNs, for some data points, yield correct predictions early on but then produce incorrect results at later exits<sup>13</sup>—a phenomenon they term overthinking. Consequently, the authors noted that early-exiting in deep neural networks is not only beneficial for speeding up inference but can sometimes even lead to improved overall test accuracy. To measure overthinking, they considered the difference in performance between an oracle model that exits at the first exit with the correct answer<sup>14</sup> and the performance of the full EENN  $p_M(\mathbf{y} \mid \mathbf{x})$ . In the anytime setting, we desire a model with the lowest overthinking measure, i.e., 0, as this indicates that evaluating more layers in the early-exit network does not negatively impact performance.
In Figure 18, we present the results of the overthinking analysis for the MSDNet [Huang et al., 2018] model. We observe that both of our proposed methods significantly improve in terms of the overthinking measure (left), making them more applicable in anytime scenarios.
Figure 18: Overthinking analysis based on Kaya et al. [2019] using the MSDNet model. Our methods substantially improve upon the baseline MSDNet in terms of the overthinking measure (left), making them more suitable for an anytime setting. However, most of these improvements can be attributed to a decrease in the theoretical performance of the oracle model (middle) rather than to the improvement of the full EENN performance (right). For a more detailed explanation of the quantities depicted, please refer to Section B.5.
We also note that a decrease in the overthinking measure can result from either (1) a reduction in the (theoretical) performance of the oracle model or (2) an improvement in the (actual) performance of the full EENN. As can be seen in Figure 18 (middle and right), our proposed modifications mainly improve the overthinking measure through (1). To explore this further, in Figure 19 we plot the number of test points that the model learned, i.e., correctly predicted for the first time, and the number of points that the model forgot, i.e., incorrectly predicted for the first time, at each early exit. Both PA and CA show a clear strength in minimizing forgetting, although they do so at the cost of diminishing the baseline model's ability to learn correct predictions. $^{15}$  These results indicate that there is further room for improvement in the adaptation of EENNs for anytime computation, as the ideal model would minimize forgetting without compromising its learning abilities.
Using notation introduced in Section 2 this implies:  $\hat{\gamma}_m^c (\pmb {x}) = 1$  and  $\hat{\gamma}_{m'}^c (\pmb {x}) = 0$  for some  $m^{\prime} > m$
Such a model cannot be implemented in practice, as it requires access to ground-truth values. Nevertheless, it is interesting to study it as a way to better understand the upper bound on the achievable performance of the underlying model.
These observations are consistent with the analysis we presented in Section B.1.2 where we used correctness of prediction as a measure for conditional monotonicity.
Figure 20: Results for CIFAR-100 using Zero-Time-Waste (ZTW; Wolczyk et al. [2021]) model. Applying our PA method to ZTW model leads to large improvements in terms of conditional monotonicity (middle) and hindsight improvability (right), while only causing a minor decrease in marginal accuracy (left).
Figure 19: Learn-vs-forget analysis for the CIFAR-100 dataset using the MSDNet model. Specifically, we plot the number of data points that are correctly classified for the first time (left) and the number of data points that are incorrectly predicted for the first time (right) at each early exit. We observe that our modifications help the model 'forget' less, though this comes with a minor trade-off in the model's ability to 'learn'.
In a follow-up study by Wolczyk et al. [2021], the hindsight improvability (HI) measure is introduced. For each early exit, this measure takes into account the set of incorrectly predicted points and computes the percentage of those that were accurately predicted at any of the previous exits. In an anytime setting, an ideal model would consistently exhibit an HI measure of 0. As evidenced by the results in Figure 20, our post-hoc modifications substantially enhance the HI measure (right plot) while only causing a minor negative impact on overall accuracy (left plot). Similar to the overthinking measure, the observation that  $HI > 0$  for our proposed methods suggests a potential for further improvements.
B.6 Deep Ensembles and Regression
Although we have presented our results for classification with early-exit neural networks, we can easily extend them to handle generic deep ensembles and regression settings.
Deep Ensembles While applying our method to deep ensembles might seem strange since they have no implicit ordering as in the case of early-exit NNs, we note that ensembles are often evaluated sequentially in low resource settings where there is not enough memory to evaluate all of the ensemble members concurrently. In such settings, there is an implicit, if arbitrary, ordering, and we may still require anytime properties so that we can stop evaluating additional ensemble members at any point. Therefore, we can apply both Product Anytime and Caching Anytime—as in Section 4—to deep ensembles, where  $p_m(y|x)$  now represents the  $m$ -th ensemble member, rather than the  $m$ -the early exit. An earlier version of this work [Allingham and Nalisnick, 2022] focused on product anytime in the deep ensembles case. However, in that work, the focus was also on training from scratch rather than the post-hoc setting we consider in this work. Training from scratch is challenging due to, among other things,[16] instabilities caused by the Product-of-Experts and the bounded support of the likelihoods. Thus, the PA results from Allingham and Nalisnick [2022] are worse in terms of predictive performance than the standard ensemble baseline. Note that our theoretical results in Appendix A also apply in the deep ensembles case.[17] Figure 21 shows qualitative results for our anytime ensembles applied to a simple classification task. As expected, the support shrinks as we add more multiplicands to the product.
Figure 21: Evolution of the support for each of 4-classes  $(\square, \square, \square, \square)$  in a spiral-arm classification task with a 5-member anytime ensemble, as 1 to 5 (left to right) members are included in the prediction.
Regression Product Anytime can be applied to regression by using a uniform likelihood at each early-exit (or ensemble member)
$$
p _ {m} \left(y \mid x\right) = \mathcal {U} \left(a _ {m}, b _ {m}\right)
$$
which results in a product likelihood
$$
p _ {1: m} (y \mid \boldsymbol {x}) = \mathcal {U} \left(a _ {\max }, b _ {\min }\right)
$$
where  $a_{\mathrm{max}} = \max(\{a_m\})$  and  $b_{\mathrm{min}} = \min(\{b_m\})$ . That is, the lower and upper bounds of the product are, respectively, the maximum-lower and minimum-upper bounds of the multiplicands. As we add more multiplicands, the width of the product density can only get smaller or stay the same, and thus the probability of the modal prediction  $p_{1:m}(\tilde{y} | x)$  can only increase or stay the same, as in the classification case. Figure 22 shows qualitative results for anytime ensembles applied to a simple regression task. We see that the order of evaluation is unimportant. Firstly, as expected, the final result is the same, and in both cases, the support size always decreases or is unchanged. Secondly, the fit has almost converged within a few evaluations, and the last few multiplicands tend to have smaller effects. Note that the prediction bounds cover the data well even for the poorer early fits. Finally, note that there is no support outside of the training data range—we consider this data as out-of-distribution.
<sup>16</sup>For example, Allingham and Nalisnick [2022] use a mixture of the product and individual likelihoods as their loss function to encourage the individual ensemble members to fit the data well. However, they use the standard softmax likelihood—with an exponential activation function—for the individual ensemble members, which conflicts with the Heaviside activations in the product likelihood and results in reduced performance.
<sup>17</sup>We note that Allingham and Nalisnick [2022] neglect an important property required for monotonicity of the modal probability:  $p_m(y \mid x)$  must be uniform in addition to having finite support. I.e., in the classification setting, the activation function used to map logits to probabilities must be the Heaviside function.
Figure 22: Evolution of the the product support (■) and mode (—) for the simple 1d [Antorán et al., 2020] regression task with a 5-member anytime ensemble, as 1 to 5 (left to right) members are included in the prediction. In each column the support of the multiplicand to be included in the next column's product are shown (■). The top and bottom show two different arbitrary orderings of the ensemble members.
B.7 Anytime Uncertainty: Conditional Monotonicity
In Section 6.2, we presented the results of our uncertainty experiments where we observed that, marginally, our PA model demonstrates an anytime uncertainty pattern: the model starts with high uncertainty early on and then progressively becomes more confident at later exits.
In this section, we supplement these experiments with an analysis at the conditional level. Specifically, for a given  $\pmb{x} \in \mathcal{X}$ , we compute the uncertainty (for instance, entropy) at each early exit:  $\{U_m(\pmb{x})\}_{m=1}^M$ . Following the approach we used in Section 3 for performance quality sequences, we define a maximum increase in uncertainty as  $\mathrm{MUI}(\pmb{x}) := \max_{m' > m} \left\{ \max(U_{m'}(\pmb{x}) - U_m(\pmb{x}), 0) \right\}$ . Next, for a range of suitably chosen thresholds  $\tau \in \mathbb{R}_{\geq 0}$ , we compute the number of test instances that exhibit an uncertainty increase surpassing a given threshold  $N_\tau := \sum_n [\mathrm{MUI}(\pmb{x}_n) \geq \tau]$ . Note that a monotone oracle model would exhibit entirely non-increasing uncertainty trajectories, meaning  $N_\tau = 0$ ,  $\forall \tau$ .
Figure 23: Conditional monotonicity using entropy as an uncertainty measure and MSDNet model as a backbone. To better illustrate the different behavior of various methods, a log scale is used for the y-axis. Our PA exhibits a significantly more monotone behavior compared to the baseline MSDNet [Huang et al., 2018] model.
The results for entropy and conformal set size are depicted in Figures 23 and 24, respectively. We observe that an EENN like MSDNet exhibits non-monotone behavior also in its uncertainty estimates across different exits. This means that there are data points where the model starts with high confidence in the initial exits but becomes less certain in subsequent stages. Such a trend is problematic in the anytime setting, similar to the issues of non-monotonicity in performance quality (c.f., Section 3). It suggests that the model expends additional computational resources only to diminish its confidence in its response. Reassuringly, our PA aids in ensuring monotonicity not only
in terms of performance quality (e.g., ground-truth probability) but also in its uncertainty estimates (e.g., conformal set size) as evident in Figures 24 and 23.
Figure 24: Conditional monotonicity using conformal set size as an uncertainty measure and MSDNet model as a backbone. To better illustrate the different behavior of various methods, a log scale is used for the y-axis. Same as in Figure 5, we used Regularized Adaptive Predictive Sets algorithm [RAPS; Angelopoulos et al., 2021] to compute conformal scores.  $20\%$  of test examples were used as a hold-out set to calculate conformal quantiles at each early exit. Our PA exhibits a significantly more monotone behavior compared to the baseline MSDNet [Huang et al., 2018] model.
B.8 Controlling the Violations of Conditional Monotonicity
In Section 4.2, we introduced our Product Anytime (PA) method. It relies on the ReLU activation function to map logits to (unnormized) probabilities, which unfortunately compromises the strong conditional monotonicity guarantees that are present when using the Heaviside activation function.
In this section, we propose a technique that allows for a more nuanced control of violations of conditional guarantees. To illustrate this, observe first that ReLU and Heaviside are specific cases of the following activation function:
$$
a (x) := C _ {b} \cdot \max  (\min  (x, b), 0) \tag {6}
$$
where  $C_b \coloneqq \max(1, 1/b)$ . Notice how  $\lim_{b \to \infty} a(x) = \mathrm{ReLU}(x)$  and  $\lim_{b \to 0^+} a(x) = \mathrm{Heaviside}(x)$ . Therefore, by adjusting the clipping threshold  $b$ , we can navigate the accuracy-monotonicity trade-off as needed. This is demonstrated on the CIFAR-10 dataset in Figure 25.
Figure 25: Test accuracy (left) and conditional monotonicity (middle and right) on CIFAR-10 dataset for various instantiations of the PA model with the MSDNet backbone. We vary the clipping threshold  $b$  used in the activation function when mapping logits to probabilities. When  $b$  is close to 0, the activation function approximates the Heaviside function, resulting in more monotone behavior, although this comes with a decrease in (marginal) accuracy. As  $b$  is increased, the activation function moves closer to ReLU, leading to more violations of monotonicity, but also enhancing accuracy.
B.9 Impact of Calibration on Conditional Monotonicity
In this section, we study whether monotonicity could be achieved by calibrating the underlying EENN rather than employing our proposed PA. To answer this, we conducted an experiment where we calibrated MSDNet and analyzed its effect on monotonicity. For the calibration, we utilized three standard techniques: temperature scaling [Guo et al., 2017], deep ensembles [Lakshminarayanan et al., 2017], and last-layer Laplace [Daxberger et al., 2021]. In Figure 26, we present results for MSDNet on CIFAR-100. While all three approaches lead to better calibration in terms of ECE (right plot), our PA significantly outperforms them in conditional monotonicity (middle plot). Moreover, all three baselines are (arguably) more complex compared to our PA: temperature scaling requires validation data, deep ensemble has M-times more parameters (M being the ensemble size), and Laplace is significantly slower compared to PA since we need test-time sampling to estimate the predictive posterior at each exit.
However, simply improving calibration might not be expected to improve monotonicity. Thus, in Figure 26, we also provide results for combining the above uncertainty calibration techniques with our CA baseline. We see that this combination provides further improvements w.r.t. monotonicity -- the improved calibration allows for better caching of the correct prediction. However, the monotonicity of these caching-and-calibrated methods still underperforms compared to PA despite being more complicated. This demonstrates that monotonicity can not be achieved via calibration alone and further underlines the value of our proposed approach.
Figure 26: Comparison of our PA with calibrated baselines using MSDNet as a backbone on CIFAR-100 dataset. For calibration, we use last-layer Laplace, deep ensembles with  $M = 5$ , and temperature scaling. Additionally, we consider both caching (—) and non-caching (---) versions of calibrated baselines. None of the considered calibrated baselines outperforms our PA in terms of monotonicity (middle), despite being more complex. The blue and purple solid lines overlap in the accuracy plot (left), indicating that the temperature scaling does not affect accuracy for the non-cached version, as expected.
B.10 Budgeted Batch Classification
In this section, we conduct a budgeted batch classification experiment using MSDNet [Huang et al., 2018] as the backbone. In budgeted batch classification, a limited amount of compute is available to classify a batch of datapoints. The goal is to design a model that can allocate resources smartly between easy and hard examples to maximize the overall performance, such as accuracy.
The results are shown in Figure 27. Since MSDNet outperforms our PA at most computational budgets, we conclude that the monotonicity is less beneficial in budgeted batch classification. However, as seen on CIFAR datasets, the non-monotonicity of MSDNet is also concerning in this setting – for larger values of FLOPs, MSDNet performance starts to drop, while PA keeps monotonically increasing and surpasses MSDNet. Based on these results, we suggest “turning on” our PA only for anytime prediction and recommend that the user sticks to the original EENN in the budgeted batch classification. However, this certainly warrants further investigation (as seen by the performance benefits of our PA on CIFAR for larger FLOPs).
Figure 27: Budgeted batch classification results for the MSDNet model. To measure the available computational resources, we use FLOPs. Lower FLOPs will encourage the model to rely more on initial exits, while a larger number of FLOPs will encourage the model to propagate more points to the later exits. Our PA modification proves to be less beneficial in this setting, as seen by its inferior performance compared to MSDNet for most of the computational budgets.
FLOPs [1e9]
B.11 Additional Plots
Here, we provide additional results that were omitted in Section 6.1 to enhance the readability of the plots there. Specifically, in Figure 28, we include test accuracy results, along with their respective error bars, for the MSDNet [Huang et al., 2018] and IMTA [Li et al., 2019] models. In Figures 31 and 32, we present results for the L2W model [Han et al., 2022b]. We observe that our findings also generalize to this EENN – our PA preserves the marginal accuracy while significantly improving conditional monotonicity across all considered datasets.
As for the DViT [Wang et al., 2021] model, our attempts to train this model from scratch using the published code<sup>18</sup> were unsuccessful. Hence, we resorted to working with the pre-trained models provided by the authors. They only published a single pre-trained model for each model version, and as a result, instead of presenting results for independent runs with varying random seeds, we display the results for all provided DViT model instantiations in Figure 29 and 30.
Figure 28: Test accuracy on the CIFAR and ImageNet datasets. Our PA method maintains a competitive performance compared to the baseline models. For each model and dataset, we use the same number of early exits as proposed by the authors. We plot the average accuracy from  $n = 3$  independent runs.
$^{18}$ https://github.com/blackfeather-wang/Dynamic-Vision-Transformer
Figure 29: Test accuracy on the CIFAR and ImageNet datasets for various instantiations of DViT model [Wang et al., 2021].
Figure 30: The  $\%$  of test examples with a ground-truth probability drop exceeding a particular threshold for various instantiations of DViT model [Wang et al., 2021]. Our PA significantly improves the conditional monotonicity of ground-truth probabilities across various datasets and backbone models. To better illustrate the different behavior of various methods, a log scale is used for the y-axis.
Figure 31: Test accuracy on the CIFAR and ImageNet datasets for L2W model [Han et al., 2022b]. We plot the average accuracy from  $n = 3$  independent runs.
Figure 32: The  $\%$  of test examples with a ground-truth probability drop exceeding a particular threshold for L2W model [Han et al., 2022b]. Results are averaged over  $n = 3$  independent runs. To better illustrate the different behavior of various methods, a log scale is used for the y-axis.