Title: Towards Anytime Classification in Early-Exit Architectures by Enforcing Conditional Monotonicity
Venue: NeurIPS 2023
Accepted: yes
Keywords: anytime algorithms

early-exit neural networks

conditional monotonicity

anytime uncertainty
Abstract: Modern predictive models are often deployed to environments in which computational budgets are dynamic. Anytime algorithms  are well-suited to such environments as, at any point during computation, they can output a prediction whose quality is a function of computation time. Early-exit neural networks have garnered attention in the context of anytime computation due to their capability to provide intermediate predictions at various stages throughout the network. However, we demonstrate that current early-exit networks are not directly applicable to anytime settings, as the quality of predictions for individual data points is not guaranteed to improve with longer computation. To address this shortcoming, we propose an elegant post-hoc modification, based on the Product-of-Experts, that encourages an early-exit network to become gradually confident. This gives our deep models the property of *conditional monotonicity* in the prediction quality---an essential building block towards truly anytime predictive modeling using early-exit architectures. Our empirical results on standard image-classification tasks demonstrate that such behaviors can be achieved while preserving competitive accuracy on average.

Towards Anytime Classification in Early-Exit Architectures by Enforcing Conditional Monotonicity

Abstract

Modern predictive models are often deployed to environments in which computational budgets are dynamic. Anytime algorithms are well-suited to such environments as, at any point during computation, they can output a prediction whose quality is a function of computation time. Early-exit neural networks have garnered attention in the context of anytime computation due to their capability to provide intermediate predictions at various stages throughout the network. However, we demonstrate that current early-exit networks are not directly applicable to anytime settings, as the quality of predictions for individual data points is not guaranteed to improve with longer computation. To address this shortcoming, we propose an elegant post-hoc modification, based on the Product-of-Experts, that encourages an early-exit network to become gradually confident. This gives our deep models the property of conditional monotonicity in the prediction quality-an essential stepping stone towards truly anytime predictive modeling using early-exit architectures. Our empirical results on standard image-classification tasks demonstrate that such behaviors can be achieved while preserving competitive accuracy on average.

1 Introduction

When deployed in the wild, predictive models are often subject to dynamic constraints on their computation. Consider a vision system for an autonomous vehicle: the same system should perform well in busy urban environments and on deserted rural roads. Yet the former environment often requires quick decision-making, unlike the latter. Hence, it would be beneficial for our models to be allowed to 'think' as long as its environment allows. Deploying models to resource-constrained devices has similar demands. For example, there are roughly 25 , 000 different computing configurations of Android devices [Yu et al., 2019], and developers often need to balance poor user experience in low-resource settings with under-performance in high-resource settings.

Anytime algorithms can ease, if not resolve, these issues brought about by dynamic constraints on computations. An algorithm is called anytime if it can quickly produce an approximate solution and subsequently employ additional resources, if available, to incrementally decrease the approximation error [Dean and Boddy, 1988, Zilberstein, 1996]. 1 Anytime computation is especially valuable when resources can fluctuate during test time. Returning to the example of mobile devices, an anytime

algorithm would be able to gracefully adapt to the computational budget of whatever class of device it finds itself on: low-performing devices will depend on intermediate (but still good) solutions, while high-performing devices can exploit the model's maximum abilities.

Early-exit neural networks (EENNs) [Teerapittayanon et al., 2016] have attracted attention as anytime predictors, due to their architecture that can generate predictions at multiple depths [Huang et al., 2018]. However, as we demonstrate, EENNs do not exhibit anytime behavior. Specifically, current EENNs do not possess conditional monotonicity , meaning that the prediction quality for individual data points can deteriorate with longer computation at test time. This lack of monotonicity leads to a decoupling of computation time and output quality-behavior that is decisively not anytime .

In this work, we improve EENNs' ability to perform anytime predictive modeling. We propose a lightweight transformation that strongly encourages monotonicity in EENNs' output probabilities as they evolve with network depth. Our method is based on a Product-of-Experts formulation [Hinton, 1999] and exploits the fact that a product of probability distributions approximates an ' and ' operation. Thus by taking the product over all early-exits computed thus-far, the aggregated predictive distribution is gradually refined over time (exits) to have a non-increasing support. This transformation has an inductive bias that encourages anytime uncertainty : the first exits produce highentropy predictive distributions-due to their large support-and the EENN becomes progressively confident at each additional exit (by decreasing the support and reallocating probability).

In our experiments, we demonstrate across a range of vision classification tasks that our method leads to significant improvements in conditional monotonicity in a variety of backbone models without re-training . Moreover, our transformation preserves the original model's average accuracy and even improves it in some cases (e.g. CIFAR-100). Thus our work allows the large pool of efficient architectures to be endowed with better anytime properties, all with minimal implementation overhead.

2 Background & Setting of Interest

Data Our primary focus is multi-class classification. 2 Let X ⊆ R d denote the feature space, and let Y denote the label space, which we assume to be a categorical encoding of multiple ( K ) classes. x ∈ X denotes a feature vector, and y ∈ Y denotes the class ( 1 of K ). We assume x and y are realizations of the random variables x and y with the (unknown) data distribution p ( x , y ) = p ( y | x ) p ( x ) . The training data sample is D = { ( x n , y n ) } N n =1 . Our experiments focus on vision tasks in which x is a natural image; in the appendix, we also report results on natural language.

Early-Exit Neural Networks: Model and Training Our model of interest is the early-exit neural network [Teerapittayanon et al., 2016] (EENN). For a given data-point x , an EENN defines M total modelsp ( y | x = x , ϕ m , θ m ) , m = 1 , . . . , M , where ϕ m denotes the parameters for the m th classifier head and θ m are the parameters of the backbone architecture. Each set of parameters θ m contains the parameters of all previous models: θ 1 ⊂ θ 2 ⊂ . . . ⊂ θ M . Due to this nested construction, each model can be thought of as a preliminary 'exit' to the full architecture defined by θ M . This enables the network's computation to be 'short-circuited' and terminated early at exit m , without evaluating later models. Training EENNs is usually done by fitting all exits simultaneously:

where w m ∈ R + are weights that control the degree to which each exit is trained. The weights are most often set to be uniform across all exits. For notational brevity, we will denote p ( y = y | x = x , ϕ m , θ m ) as p m ( y | x ) henceforth.

Setting of Interest At test time, there are various ways to exploit the advantages of an EENN. For example, computation could be halted if an input is deemed sufficiently easy such that a weaker model (e.g. p 1 or p 2 ) can already make an accurate prediction, thus accelerating inference time. Such efficient computation has been the central focus of the EENNs's literature thus far [Matsubara et al., 2023] and is sometimes referred to as budgeted batch classification [Huang et al., 2018].

However, in this work we focus exclusively on the related, yet distinct, anytime setting motivated in the Introduction. We assume computation is terminated when the current environment cannot support computing and evaluating the ( m + 1) th model. These constraints originate solely from the environment, meaning the user or model developer has no control over them. We assume the model will continue to run until it either receives such an external signal to halt or the final-i.e., M th-model is computed [Grubb and Bagnell, 2012]. Furthermore, in line with previous literature on anytime algorithms [Huang et al., 2018, Hu et al., 2019], we assume the model sees one data point at a time when deployed, i.e., the batch size is one at test time.

Anytime Predictive Models To have the potential to be called 'anytime,' an algorithm must produce a sequence of intermediate solutions. In the case of classification, an anytime predictive model must hence output a series of probability distributions over Y with the goal of forming better and better approximations of the true conditional data distribution p ( y | x ) . Specifically, we are interested in the evolution of the conditional error for a given x ∈ X , defined here as ϵ t ( x ) := D ( p ( y | x ) , p t ( y | x ) ) where D denotes a suitable distance or divergence (e.g., KLdivergence or total-variation), and p t represents a model's predictive distribution after computing for time t ∈ R ≥ 0 . In addition to conditional, i.e., instance-level, error we can also consider the evolution of a marginal error defined as ¯ ϵ t := E x ∼ p ( x ) [ ϵ t ( x ) ] . For the remainder of this section, we will use ϵ t to refer to both versions of the error and revisit their difference in Section 3. Since this paper focuses on anytime EENNs, 3 we use the early-exit index m as the time index t .

Properties of Anytime Models Zilberstein [1996] enumerates properties that an algorithm should possess to be anytime . Below we summarize Zilberstein [1996]'s points that are relevant to predictive modeling with EENNs, ordering them in a hierarchy of necessity:

Interruptibility : The most basic requirement is that the model can be stopped and produce an answer at any time. Traditional neural networks do not satisfy interruptibility since they produce a single output after evaluating all layers. However, EENNs do by returning the softmax probabilities for the m th model if the ( m +1) th model has not been computed yet.

Monotonicity : The quality of the intermediate solutions should not decrease with computation time. Or equivalently, expending more compute time should guarantee performance does not degrade. For EENNs, monotonicity is achieved if the error ϵ m is non-increasing across early-exits: ϵ 1 ≥ . . . ≥ ϵ M .

Diminishing Returns : The difference in quality between consecutive solutions is gradually decreasing, with the largest jumps occurring early, e.g. ( ϵ 1 -ϵ 2 ) ≫ ( ϵ M -1 -ϵ M ) . Monotonicity and consistency need to be satisfied in order for this stronger property to be guaranteed.

Consistency : The quality of the intermediate solutions should be correlated with computation time. For EENNs, consistency is of most concern when the amount of computation varies across sub-models. For example, if computing the ( m +1) th model requires evaluating 10 × more residual blocks than the m th model required, then a consistent architecture should produce a roughly 10 × better solution ( ϵ m +1 ≈ ϵ m / 10) .

For a traditional EENN (trained with Equation 1), property #1 is satisfied by construction (by the presence of early-exits). Property #2 is not explicitly encouraged by the typical EENN architecture or training objective. Hence, we begin the next section by investigating if current EENNs demonstrate any signs of monotonicity. Properties #3 and #4 are left to future work since, as we will demonstrate, achieving property #2 is non-trivial and is thus the focus of the remainder of this paper.

Estimating Anytime Properties The error sequence { ϵ m } M m =1 is essential for studying the anytime properties of EENNs. We can equivalently consider the evolution of prediction quality measures γ m and aim for a model that exhibits increasing quality, i.e., γ m ≤ γ m +1 , ∀ m . For the remainder of the paper, we will work with the notion of prediction quality unless otherwise stated. Observe that each γ m (and equivalently, ϵ m ) is always unknown, as it depends on the unknown data distribution p ( y | x ) . Consequently, we must rely on estimates ˆ γ m . In the presence of a labeled hold-out dataset, we can use ground-truth labels y ∗ ∈ Y and consider the correctness of the prediction as our estimator:

ˆ γ c m ( x ) := [arg max y p m ( y | x ) = y ∗ ] where [ · ] is the Iverson bracket. 4 However, such an estimator provides a crude signal (0 or 1) on a conditional level (per data point). As such, it is not congruent with conditional monotonicity or consistency. Due to this, we focus primarily on the probability of the ground-truth class ˆ γ p m ( x ) := p m ( y ∗ | x ) when examining anytime properties at the conditional level. Finally, estimating marginal prediction quality (error) requires access to p ( x ) . Following common practice, we approximate E x ∼ p ( x ) by averaging over the hold-out dataset. Averaging ˆ γ c m ( x ) and ˆ γ p m ( x ) corresponds to the test accuracy and the average ground-truth probability at the m th exit, respectively.

3 Checking for Monotonicity in Early-Exit Neural Networks

In this section, we take a closer look at state-of-the-art early-exit neural networks and their monotonicity, a key property of anytime models. Our results show that EENNs are marginally monotone but lack conditional monotonicity in the prediction quality. For ease of presentation, we focus here on a particular EENN ( Multi-Scale Dense Net (MSDNet); Huang et al. [2018]) and dataset (CIFAR-100). Section 6 shows that this observation generalizes across other EENNs and datasets.

Marginal Monotonicity The left plot of Figure 1 reports MSDNet's test accuracy ( blue ) on CIFAR100 as a function of the early exits. Accuracy has been the primary focus in the anytime EENN literature [Huang et al., 2018, Hu et al., 2019], and as expected, it increases at each early exit m . Thus MSDNet clearly exhibits marginal monotonicity : monotonicity on average over the test set. The same figure also shows that the average ground-truth probability ( orange ) increases at each m , meaning that MSDNet demonstrates marginal monotonicity in this related quantity as well.

Conditional Monotonicity Yet for real-time decision-making systems, such as autonomous driving, it is usually not enough to have marginal monotonicity. Rather, we want conditional monotonicity : that our predictive model will improve instep with computation for every input . If conditional monotonicity is achieved, then marginal monotonicity is also guaranteed. However, the converse is not true, making conditional monotonicity the stronger property.

To check if MSDNet demonstrates conditional monotonicity, examine the middle plot of Figure 1. Here we plot the probability of the true label, i.e., ˆ γ p m ( x ) , as a function of the early exits for 10 randomly sampled test points (again from CIFAR-100). We see that-despite MSDNet being marginally monotonic-it is quite unstable in the probabilities it produces per exit. For example, the purple line exhibits confidence in the correct answer ( ∼ 75% ) at the second exit, drops to near zero by the fourth exit, becomes confident again ( ∼ 80% ) at exits 5 and 6 , and finally falls to near zero again at the last exit. Thus MSDNet clearly does not exhibit the stronger form of monotonicity.

We next quantify the degree to which MSDNet is violating conditional monotonicity. First, we define the maximum decrease in the ground-truth probability as MPD ∗ ( x ) := max m ′ >m { max(ˆ γ p m ( x ) -ˆ γ p m ′ ( x ) , 0) } . For various thresholds τ ∈ [0 , 1] , we count how many test instances exhibit a probability decrease that exceeds the given threshold: N τ := ∑ n [ MPD ∗ ( x n ) ≥ τ ] . The results are depicted in the right panel of Figure 1. We see that for ∼ 25% of test points, the ground-truth probability decreases by at least 0 . 5 at some exit. In a separate analysis (see Appendix B.1.2), when examining the correctness of prediction measure ˆ γ c m ( x ) , we find that ∼ 30% of test examples exhibit decreasing trajectories across early exits. 5 Asimilar finding was reported by Kaya et al. [2019]. Both observations demonstrate that MSDNet, a state-of-the-art EENN, does not exhibit conditional monotonicity nor is it close to achieving it.

4 Transformations for Conditional Monotonicity

In light of the aforementioned challenges with current EENNs in the anytime setting, we explore potential lightweight modifications that could encourage, or even enforce, conditional monotonicity. We first suggest a strong baseline for encouraging monotonicity, taking inspiration from Zilberstein [1996]. We then introduce our primary methodology, which is based on product ensembles.

)

1

2

3

4

5

6

7

Early Exit

0.65

0.70

Test Accuracy

y

∗

Prob.

1

2

3

4

5

6

7

Early Exit

0.0

0.5

1.0

y

∗

Probability

0.01

0.10

0.20

0.33

0.50

y

∗

Prob. Decrease

0

25

50

75

%of Test Samples (

↓

MSDNet

4.1 Caching Anytime Predictors

As noted by Zilberstein [1996], conditional monotonicity can be encouraged by caching the best prediction made so far and returning it (instead of the latest prediction). Yet this requires the ability to measure the prediction quality of intermediate results. While previously introduced quantities, such as ground-truth probability ˆ γ p m ( x ) , can serve to examine anytime properties before deploying the model, their reliance on the true label y ∗ renders them unsuitable for measuring prediction quality at test time. As a proxy, we propose using the EENN's internal measure of confidence. We found that simply using the softmax confidence performs well in practice. At an exit m , we compute C ( p m , x ) := max y p m ( y | x ) and overwrite the cache if C ( p j , x ) < C ( p m , x ) , where j is the exit index of the currently cached prediction. We call this approach C aching A nytime (CA) and consider it as a (strong) baseline. While CA is rather straightforward, it has been thus far overlooked in the anytime EENN literature, as the standard approach is to return the latest prediction [Huang et al., 2018, Hu et al., 2019]. Note that if the anytime model displays conditional monotonicity to begin with, applying CA has no effect since the most recent prediction is always at least as good as the cached one.

4.2 Product Anytime Predictors

Our proposed method for achieving conditional monotonicity in EENNs is motivated by the following idealized construction.

Guaranteed Conditional Monotonicity via Hard Product-of-Experts Consider an EENN whose sub-classifiers produce 'hard' one-vs-rest probabilities: each class has either probability 0 or 1 (before normalization). Now define the predictive distribution at the m th exit to be a Product-of-Experts [Hinton, 1999] (PoE) ensemble involving the current and all preceding exits:

where b is a threshold parameter, [ · ] is an Iverson bracket, f i ( x ) ∈ R K represents a vector of logits at i th exit, and w i ∈ R + are ensemble weights. Assuming that a class y is in the support of the final classifier for a given x , i.e. p Π ,M ( y | x ) > 0 , then such a model is guaranteed to be monotone in the probability of y :

The formal proof can be found in Appendix A. Consequently, if the true class y ∗ is in the final support, an EENN with this construction is conditionally monotonic in the ground-truth probability ˆ γ p m ( x ) (as defined in Section 2).

The intuition behind this construction is that at each exit, the newly computed ensemble member casts a binary vote for each class. If a class has received a vote at every exit thus far, then it remains in the support of p Π ,m ( y | x ) . Otherwise, the label drops out of the support and never re-enters. At each

exit, the set of candidate labels can only either remain the same or be reduced, thus concentrating probability in a non-increasing subset of classes. This construction also has an underconfidence bias : the probability of all labels starts small due to many labels being in the support at the earliest exits. Consequently, the model is encouraged to exhibit high uncertainty early on and then to become progressively confident at subsequent exits.

A Practical Relaxation using ReLU Although the above construction provides perfect conditional monotonicity (for labels in the final support), it often results in a considerable decrease in marginal accuracy (see Appendix B.2). This is not surprising since the 0 -1 probabilities result in 'blunt' predictions. At every exit, all in-support labels have equal probability, and thus the model must select a label from the support at random in order to generate a prediction. The support is often large, especially at the early exits, and in turn the model devolves to random guessing.

To overcome this limitation, first notice that using 0 -1 probabilities is equivalent to using a Heaviside function to map logits to probabilities. 6 In turn, we propose to relax the parameterization, replacing the Heaviside with the ReLU activation function. The ReLU has the nice property that it preserves the rank of the 'surviving' logits while still 'nullifying' classes with low logits. Yet this relaxation comes at a price: using ReLUs allows conditional monotonicity to be violated. However, as we will show in Section 6, the violations are rare and well worth the improvements in accuracy. Moreover, the number of violations can be controlled by applying an upper limit to the logits, as detailed in Appendix B.8.

Therefore our final P roduct A nytime (PA) method for conditional monotonicity is written as:

where max( · , · ) returns the maximum of the two arguments. We set the ensemble weights to w i = i/M since we expect later experts to be stronger predictors. Note that PA's predictive distribution may degenerate into a zero distribution such that p PA ,m ( y | x ) = 0 , ∀ y ∈ Y . This occurs when the ensemble's experts have non-overlapping support and are thus unable to arrive at a consensus prediction. If a prediction is required, we fall back on the softmax predictive distribution based on the logits of the classifier at the current exit.

Learning and Post-Hoc Application Equation 4 can be used for training an EENN directly via the negative log-likelihood. However, one can also take pre-trained EENNs (e.g., as open-sourced by the original authors) and apply Equation 4 post-hoc . For this post-hoc application, the EENN's logits are input into Equation 4 rather than into the usual softmax transformation. As we will demonstrate in the experiments, post-hoc application endows the model with conditional monotonicity without any substantial degradation to the original model's accuracy. In cases such as CIFAR-100, we find that our method actually improves the accuracy of the pre-trained model. The only metric we find to be sensitive to training from scratch vs

1

2

3

4

5

6

7

0.0

0.5

1.0

y

∗

Probability

post-hoc application is confidence calibration, with the former providing better calibration properties. This result will be demonstrated in Figure 6. Thus, we currently recommend using our method via post-hoc application, as it requires minimal implementation overhead (about three lines of code) and allows the user to easily toggle between anytime and traditional behavior.

For a visualization of how our post-hoc PA affects probability trajectories, see Figure 2. PA manages to rectify the backbone EENN's non-monotone behavior to a large extent. As a concrete example of how this can help with anytime predictions, we can consider again the scenario of Android

phones introduced in Section 1. Deploying the original MSDNet might result in certain data points receiving inferior predictions on a higher-spec device compared to a device with limited computational capabilities. However, if we apply our PA to an MSDNet, such inconsistencies become far less likely due to the model's monotonic behavior.

5 Related work

Anytime Algorithms [Dean and Boddy, 1988, Zilberstein, 1996] have long been studied outside the context of neural networks. For instance, Wellman and Liu [1994] considered Bayesian networks in anytime settings, while Grubb and Bagnell [2012] proposed an anytime Ada-Boost algorithm [Bishop, 2007] and also studied Haar-like features [Viola and Jones, 2001] for anytime object detection. More recently, modern neural networks have been used in models with only marginal anytime properties [Huang et al., 2018, Hu et al., 2019, Liu et al., 2022]. Our research improves upon this work by bringing these models a step closer to having per-data-point anytime prediction, a much stronger form of anytime computation.

Early-Exit Neural Networks [Teerapittayanon et al., 2016, Yang et al., 2020, Han et al., 2022a] have been investigated for applications in computer vision [Huang et al., 2018, Kaya et al., 2019] and natural language processing [Schwartz et al., 2020, Schuster et al., 2021, Xu and McAuley, 2023]. A wide variety of backbone architectures has been explored, encompassing recurrent neural networks [Graves, 2016], convolutional networks [Huang et al., 2018], and transformers [Wang et al., 2021]. Furthermore, ensembling of EENNs intermediate predictions has been considered [Qendro et al., 2021, Wolczyk et al., 2021, Meronen et al., 2023], primarily to enhance their uncertainty quantification capabilities. Our objective is to leverage the recent advancements in early-exit networks and make them suitable for true (conditional) anytime predictive modeling.

Related to our critique of current EENNs, Kaya et al. [2019] and Wolczyk et al. [2021] also observed that prediction quality can decline at later exits, calling the phenomenon 'overthinking.' These works address the problem via methods for early termination. Our work aims to impose monotonicity on the ground-truth probabilities, which eliminates 'overthinking' by definition. We further examine the connection to Kaya et al. [2019] and Wolczyk et al. [2021] in Appendix B.5.

Product-of-Experts [Hinton, 1999, 2002] ensembles have been extensively studied in the generative modeling domain due to their ability to yield sharper distributions compared to Mixture-of-Experts ensembles [Wu and Goodman, 2018, Huang et al., 2022]. However, in the supervised setting, PoE ensembles have received relatively little attention [Pignat et al., 2022]. PoE is often applied only post-hoc, likely due to training difficulties [Cao and Fleet, 2014, Watson and Morimoto, 2022].

6 Experiments

We conduct two sets of experiments. 7 First, in Section 6.1, we verify that our method (PA) maintains strong average performance while significantly improving conditional monotonicity in state-of-the-art EENNs making them more suitable for the anytime prediction task. To check that our findings generalize across data modalities, we also conduct NLP experiments and present the results in Appendix B.4. In the second set of experiments, detailed in Section 6.2, we move beyond prediction quality and shift focus to monotonicity's effect on uncertainty quantification in the anytime setting. Lastly, we point out some limitations in Section 6.3.

Datasets We consider CIFAR-10, CIFAR-100 [Krizhevsky et al., 2009], and ILSVRC 2012 (ImageNet; Deng et al. [2009]). The CIFAR datasets each contain 50 k training and 10 k test images, while ImageNet is a larger dataset with 1.2M training and 50 k test instances. The three datasets consist of 10, 100, and 1000 classes, respectively.

Baselines We use four state-of-the-art EENNs as baselines and backbone architectures. The first is the Multi-Scale Dense Network (MSDNet; Huang et al. [2018]), which consists of stacked convolutional blocks. To maximize performance at each early-exit, the authors introduce: (1) feature maps on multiple scales, ensuring that coarse-level features are available all throughout the network,

0.96

0.97

0.98

CIFAR-10

1

2

3

4

5

6

7

0.89

0.90

0.91

0.92

0.93

0.84

0.88

CIFAR-100

1

2

3

4

5

6

7

Early Exit

0.64

0.68

0.72

baseline

PA (ours)

CA

baseline

PA (ours)

CA

1

2

3

4

5

0.60

0.65

0.70

0.75

0.80

ImageNet

MSDNet

IMTA

DViT

Test Accuracy (

↑

)

and (2) dense connectivity to prevent earlier classifiers from negatively impacting later ones. The second baseline is the IMTA network [Li et al., 2019], which shares MSDNet's architecture but is trained with regularization that improves collaboration across the classifiers of each exit. The third baseline is the L2W model [Han et al., 2022b], which enhances MSDNet by learning to weight easy and hard samples differently at each exit during training. The intuition is that easier samples should contribute more to the fitting of earlier exits, while harder samples should be more influential for later exits. Lastly, we consider the Dynamic Vision Transformer (DViT; Wang et al. [2021]), which is composed of stacked transformer networks that differ in the number of patches into which they divide an input image. We report results for the T2T-ViT-12 instantiation of DViT. During testing, all baselines employ a softmax parameterization for their categorical likelihood and solely rely on the most recent prediction at each early exit.

6.1 Anytime Prediction with Conditional Monotonicity

Test Accuracy We first verify that post-hoc application of our PA transformation preserves the original model's test accuracy. Maintaining test accuracy is essential, as simply examining conditional monotonicity in isolation does not provide a complete picture. For instance, a model that predicts a ground-truth probability of zero at every exit would exhibit perfect conditional monotonicity while being useless in practice. Figure 3 reports accuracy per exit for the three backbones models, comparing the original model ( ) to our proposals PA ( ) and CA ( ). We find that our monotonicity methods do not significantly degrade test accuracy in any case. In some instances, such as the MSDNet results on CIFAR-100 ( middle , blue ), our PA even improves accuracy. Comparing our proposals, PA tends to outperform CA in accuracy, although the advantage is often slight.

Conditional Monotonicity Next, we quantify the degree to which each model is conditionally monotonic, again using the maximum probability decrease metric (MPD ∗ ) used in Figure 1 ( right ). Figure 4 reports the percentage of test cases that exhibit a drop in the ground-truth probability at any exit. Firstly, we see that CA and PA drastically improve monotonicity in all cases. Moreover, PA is clearly superior to CA for CIFAR-100 and ImageNet. On CIFAR-10, PA and CA perform on par for smaller decrease thresholds, with PA outperforming CA again for larger thresholds. Notably, on ImageNet, our PA method does not exhibit a drop in ground-truth probability larger than 0 . 01 for any test example for all three models.

From the results in Figures 3 and 4, one might wonder if a simpler change to the backbone EENN could lead to similar improvements in conditional monotonicity as our PA. To test this, we looked at 'vanilla' early-exit ensembles where predictive distributions from different exits are combined using either mixture-of-experts (MoE) or product-of-experts (PoE), see Appendix B.2 for the results. While using ensembles is beneficial, it is insufficient to achieve improvements in monotonicity similar to those of PA. This highlights the importance of the activation function (that maps logits

0.0

0.1

0.2

0.3

0.4

0.5

0

10 0

10 1

%of Test Samples (

↓

)

CIFAR-10

baseline

PA (ours)

CA

0.0

0.1

0.2

0.3

0.4

0.5

y

∗

) Probability Decrease

CIFAR-100

0.0

0.1

0.2

0.3

0.4

0.5

ImageNet

MSDNet

IMTA

DViT

to probabilities) when it comes to achieving more monotone behavior. We additionally explored if calibrating the backbone EENN would be enough to make the underlying model monotone. While post-hoc calibrating every exit does help with monotonicity, it does not perform as well as PA; see Appendix B.9.

Monotonicity in Correctness Our PA approach yields close to perfect conditional monotonicity when considering probability as the estimator of prediction quality. Yet it is also worth checking if other estimators are monotonic, such as the correctness of the prediction, i.e., ˆ γ c m ( x ) . For correctness, we observe similar improvements in conditional monotonicity; see Appendix B.1.2 for a detailed analysis. For example, on CIFAR-100, MSDNet 'forgets' the correct answer in ∼ 30% of test cases, whereas applying post-hoc PA reduces these monotonicity violations to ∼ 13% of cases.

6.2 Anytime Uncertainty

We also examine the uncertainty quantification abilities of these anytime models. Again consider Figure 2, the probability trajectories obtained from post-hoc PA. The probabilities start low at first few exits, indicating high uncertainty in the early stages of anytime evaluation. The left plot of Figure 5 further supports this insight, showing that the average entropy of the PA predictive distributions is high and reduces at each early exit. Lastly, in the right plot, we present the average size of conformal sets 8 at each exit. The set size reflects the model's predictive uncertainty, with larger sets indicating more uncertainty. Thus we see that PA is best at uncertainty quantification: it is quite uncertain at the earliest exits (set size of ∼ 11 ) but grows twice as confident by the last exit (set size of ∼ 4 ). CA and regular MSDNet keep a relatively constant set size by comparison, only varying between

1

2

3

4

5

6

7

Early Exit

0

2

4

Entropy

1

2

3

4

5

6

7

Early Exit

5.0

7.5

10.0

12.5

↓

)

11 and 7 . 5 . Hence, the monotonicity properties of PA result in an appealing link between uncertainty and computation time; and we term this property as anytime uncertainty . This is an appropriate inductive bias for anytime predictive models since, intuitively, making quicker decisions should entail a higher level of uncertainty. Lastly, in Appendix B.7, we present results of conditional level experiments where we find that our PA yields significantly more monotone (i.e., non-increasing) uncertainty patterns.

6.3 Limitations

Label Sets with Small Cardinality We observe that PA's performance in terms of conditional monotonicity improves as the number of classes |Y| increases. For instance, PA yields fully monotone trajectories on ImageNet, but not for datasets with smaller number of classes like CIFAR-10, where we detect small violations. This finding is further supported by our NLP experiments in Appendix B.4, where |Y| is smaller compared to the image classification datasets considered in this section. Based on this finding, we recommend using the CA approach over PA for scenarios in which |Y| < 5 .

Calibration Gap Due to its aforementioned underconfidence bias, post-hoc PA generates poorly calibrated probabilities during the initial exits, as illustrated in the left plot of Figure 6. Yet this is primarily observed in expected calibration error (ECE) and related metrics. It disappears when considering some alternative metrics to ECE, as shown in the right plot of Figure 12. Finetuning with PA as an optimization objective, as opposed to applying it post-hoc, closes the calibration gap in most cases, albeit at some cost to accuracy and monotonicity. See Figure 6 ( right ) for concrete examples of how PA with finetuning affects ground-truth probability trajectories. In addition, we experimented with adaptive thresholding in PA, using different thresholds b for distinct data points, rather than employing 0 for all data points as with ReLU, and found that it can also be beneficial in terms of calibration. For more details on finetuning and adaptive thresholding, as well as a discussion on the relevance of ECE-like metrics in the anytime setting, see Appendix B.3.

7 Conclusion

We have demonstrated that expending more time during prediction in current early-exit neural networks can lead to reduced prediction quality, rendering these models unsuitable for direct application in anytime settings. To address this issue, we proposed a post-hoc modification based on Product-of-Experts and showed through empirical and theoretical analysis that it greatly improves conditional monotonicity with respect to prediction quality over time while preserving competitive overall performance.

In future work, we aim to further improve EENNs by imbuing them with other critical anytime properties, such as diminishing returns and consistency. Moreover, it would be interesting to study the impact of monotonicity in other settings, such as budgeted batch classification. We report some preliminary results in Appendix B.10, finding the monotonicity to be less beneficial there compared to the anytime setting, which was the focus of our work. We also plan to delve deeper into the concept of anytime uncertainty. Although we have (briefly) touched upon it in this study, we believe a more comprehensive and formal definition is necessary for the development of robust and reliable anytime models.

1

2

3

4

5

6

7

Early Exit

0.0

0.2

0.4

0.6

0.8

1.0

y

∗

Probability

PA with Finetuning

1

2

3

4

5

6

7

Early Exit

0.0

0.2

0.4

0.6

ECE (

↓

)

MSDNet

PA (ours)

PA-FT (ours)

CA

Acknowledgments and Disclosure of Funding

We thank Alexander Timans and Rajeev Verma for helpful discussions. We are also grateful to the anonymous reviewers who helped us improve our work with their constructive feedback. MJ and EN are generously supported by the Bosch Center for Artificial Intelligence. JUA acknowledges funding from the EPSRC, the Michael E. Fisher Studentship in Machine Learning, and the Qualcomm Innovation Fellowship.

References

U. Allingham and E. Nalisnick. A Product of Experts Approach to Early-Exit Ensembles. Technical report, 2022.

N. Angelopoulos, S. Bates, M. I. Jordan, and J. Malik. Uncertainty sets for image classifiers using conformal prediction. In 9th International Conference on Learning Representations, ICLR , 2021.

Antorán, J. U. Allingham, and J. M. Hernández-Lobato. Depth uncertainty in neural networks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems , 2020.

M. Bishop. Pattern recognition and machine learning, 5th Edition . Information science and statistics. Springer, 2007. ISBN 9780387310732.

Cao and D. J. Fleet. Generalized product of experts for automatic and principled fusion of gaussian process predictions. CoRR , 2014. URL http://arxiv.org/abs/1410.7827 .

Daxberger, A. Kristiadi, A. Immer, R. Eschenhagen, M. Bauer, and P. Hennig. Laplace redux - effortless bayesian deep learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems , 2021.

L. Dean and M. S. Boddy. An analysis of time-dependent planning. In Proceedings of the 7th National Conference on Artificial Intelligence . AAAI Press / The MIT Press, 1988.

Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) , 2009.

Graves. Adaptive computation time for recurrent neural networks. CoRR , 2016. URL http: //arxiv.org/abs/1603.08983 .

Grubb and D. Bagnell. Speedboost: Anytime prediction with uniform near-optimality. In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, AISTATS , 2012.

Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML , 2017.

Han, G. Huang, S. Song, L. Yang, H. Wang, and Y. Wang. Dynamic neural networks: A survey. IEEE Trans. Pattern Anal. Mach. Intell. , 2022a.

Han, Y. Pu, Z. Lai, C. Wang, S. Song, J. Cao, W. Huang, C. Deng, and G. Huang. Learning to weight samples for dynamic early-exiting networks. In Computer Vision - ECCV . Springer, 2022b.

E. Hinton. Products of experts. Technical report, 1999.

E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput. , 2002.

Hu, D. Dey, M. Hebert, and J. A. Bagnell. Learning anytime predictions in neural networks via adaptive loss balancing. In The Thirty-Third Conference on Artificial Intelligence, AAAI . AAAI Press, 2019.

Huang, D. Chen, T. Li, F. Wu, L. van der Maaten, and K. Q. Weinberger. Multi-scale dense networks for resource efficient image classification. In 6th International Conference on Learning Representations, ICLR , 2018.

Huang, A. Mallya, T. Wang, and M. Liu. Multimodal conditional image synthesis with product-ofexperts gans. In Computer Vision - ECCV . Springer, 2022.

Kaya, S. Hong, and T. Dumitras. Shallow-deep networks: Understanding and mitigating network overthinking. In Proceedings of the 36th International Conference on Machine Learning, ICML . PMLR, 2019.

D. Kivlichan, Z. Lin, J. Z. Liu, and L. Vasserman. Measuring and improving model-moderator collaboration using uncertainty estimation. CoRR , 2021. URL https://arxiv.org/abs/2107. 04212 .

Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems , 2017.

Li, H. Zhang, X. Qi, R. Yang, and G. Huang. Improved techniques for training adaptive deep networks. In IEEE/CVF International Conference on Computer Vision, ICCV . IEEE, 2019.

Liu, Z. Xu, H. Wang, T. Darrell, and E. Shelhamer. Anytime dense prediction with confidence adaptivity. In The Tenth International Conference on Learning Representations, ICLR , 2022.

Matsubara, M. Levorato, and F. Restuccia. Split computing and early exiting for deep learning applications: Survey and research challenges. ACM Comput. Surv. , 2023.

Meronen, M. Trapp, A. Pilzer, L. Yang, and A. Solin. Fixing overconfidence in dynamic neural networks. CoRR , 2023. doi: 10.48550/arXiv.2302.06359. URL https://doi.org/10.48550/ arXiv.2302.06359 .

Pignat, J. Silvério, and S. Calinon. Learning from demonstration using products of experts: Applications to manipulation and task prioritization. Int. J. Robotics Res. , 2022.

Qendro, A. Campbell, P. Liò, and C. Mascolo. Early exit ensembles for uncertainty quantification. In Machine Learning for Health, ML4H@NeurIPS 2021 , Proceedings of Machine Learning Research. PMLR, 2021.

Schuster, A. Fisch, T. S. Jaakkola, and R. Barzilay. Consistent accelerated inference via confident adaptive transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP , 2021.

Schwartz, G. Stanovsky, S. Swayamdipta, J. Dodge, and N. A. Smith. The right tool for the job: Matching model and instance complexities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL , 2020.

Shafer and V. Vovk. A tutorial on conformal prediction. J. Mach. Learn. Res. , 2008.

Teerapittayanon, B. McDanel, and H. T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 23rd International Conference on Pattern Recognition, ICPR . IEEE, 2016.

A. Viola and M. J. Jones. Rapid object detection using a boosted cascade of simple features. In 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) , 2001.

Wang, R. Huang, S. Song, Z. Huang, and G. Huang. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems , 2021.

Watson and T. K. Morimoto. Learning non-parametric models in real time via online generalized product of experts. IEEE Robotics Autom. Lett. , 2022.

P. Wellman and C. Liu. State-space abstraction for anytime evaluation of probabilistic networks. In UAI '94: Proceedings of the Tenth Annual Conference on Uncertainty in Artificial Intelligence , 1994.

Wolczyk, B. Wójcik, K. Balazy, I. T. Podolak, J. Tabor, M. Smieja, and T. Trzcinski. Zero time waste: Recycling predictions in early exit neural networks. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems , 2021.

Wu and N. D. Goodman. Multimodal generative models for scalable weakly-supervised learning. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems , 2018.

Xu and J. J. McAuley. A survey on dynamic neural networks for natural language processing. In Findings of the Association for Computational Linguistics: EACL , 2023.

Yang, Y. Han, X. Chen, S. Song, J. Dai, and G. Huang. Resolution adaptive networks for efficient inference. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR , 2020.

Yu, L. Yang, N. Xu, J. Yang, and T. S. Huang. Slimmable neural networks. In 7th International Conference on Learning Representations, ICLR , 2019.

Zilberstein. Using anytime algorithms in intelligent systems. AI Magazine , 1996.

The Appendix is structured as follows:

We provide a proof of conditional guarantees in EENNs for (hard) PoE in Appendix A.

We present conditional monotonicity results using alternative estimators of performance quality γ m in Appendix B.1.

We conduct an ablation study for our PA model in Appendix B.2.

We propose two modifications to our PA that help with reducing the calibration gap in Appendix B.3.

We report results of NLP experiments in Appendix B.4.

We delve into the connections between conditional monotonicity and previously introduced concepts in EENNs literature in Appendix B.5.

We discuss anytime regression and deep ensembles in Appendix B.6.

We present conditional monotonicity results using uncertainty measures in Appendix B.7.

We propose a technique for controlling the violations of conditional monotonicity in PA in Appendix B.8.

We study the impact of calibration on the conditional monotonicity in Appendix B.9.

We provide budgeted batch classification results in Appendix B.10.

We provide additional plots in Appendix B.11.

A Theoretical Results

In the following, we provide a proof for conditional monotonicity of (hard) Product-of-Experts ensembles presented in Section 4.2.

Proposition. Let p Π ,m ( y | x ) represent the Product-of-Experts ensemble of the first m early-exits. Furthermore, let y ∈ Y denote any class that is present in the support of the full-NN product distribution, i.e., p Π ,M ( y | x ) > 0 . Under the assumption that the Heaviside activation function is used to map logits to probabilities in each p m ( y | x ) , it follows that:

for every early-exit m and for every input x .

Proof. Using the definition of the PoE ensemble and that of the Heaviside activation function, the predictive likelihood at the m th early exit can be written as

where f i ( x ) ∈ R K denotes a vector of logits at the i -th early-exit, b represents a chosen threshold, and w i ∈ R + represent ensemble weights. Since we assume that class y is in the support of the full NN ensemble, i.e., p Π ,M ( y | x ) > 0 , it follows that

It then remains to show that the normalization constant is non-increasing, i.e., Z m ≥ Z m +1 , ∀ m . To this end observe that [ f m +1 ( x ) y ′ > b ] w m +1 ∈ { 0 , 1 } , ∀ y ′ ∈ Y , using which we proceed as

B Additional Experiments

B.1 Conditional Monotonicity

In the main paper, the ground-truth probability was our focus when studying conditional monotonicity in early-exit networks. Here, we supplement these findings with additional results, utilizing alternative estimators of prediction quality γ m .

B.1.1 Full-Model Prediction Probability

Suppose only an unlabeled hold-out dataset is available. In that case, the anytime properties of EENNs can still be investigated by relying on the full model prediction ˆ y := arg max y p M ( y | x ) , instead of y ∗ . Naturally, the merit of this alternative hinges on the underlying model being accurate in most cases. Furthermore, besides serving as a substitute for the ground-truth probability in the absence of a labeled dataset, the evolution of the full-model prediction probability can be used as a stand-alone measure of how well early-exits in an EENN approximate the full model.

In Figure 7, we present conditional monotonicity results using ˆ y probability as the prediction quality measure. Similar to the results presented in Section 6.1 for y ∗ probability, we conclude that our PA outperforms both the baseline models and CA approach across all datasets considered.

0.0

0.1

0.2

0.3

0.4

0.5

0

10 0

10 1

%of Test Samples (

↓

)

CIFAR-10

baseline

PA (ours)

CA

0.0

0.1

0.2

0.3

0.4

0.5

CIFAR-100

0.0

0.1

0.2

0.3

0.4

0.5

ImageNet

MSDNet

IMTA

DViT

y

) Probability Decrease

B.1.2 Correctness of Prediction

We next shift our focus to the correctness of prediction as an estimator of performance quality. Recall from Section 2 its definition: ˆ γ c m ( x ) := [arg max y p m ( y | x ) = y ∗ ] . It is a binary measure that can only take on the values of 0 or 1, and thus provides a less nuanced signal about the evolution of the performance quality compared to probability-based measures that evolve more continuously in the interval [0 , 1] .

To aggregate the results of the conditional monotonicity analysis, we consider the following two quantities:

% mono: the percentage of test data points that exhibit fully monotone trajectories in the correctness of the prediction, i.e., ˆ γ c m ( x ) ≤ ˆ γ c m +1 ( x ) , ∀ m . This implies that if the model managed to arrive at the correct prediction at the m th exit, it will not forget it at later exits. An oracle model with perfect conditional monotonicity will result in % mono = 100% .

% zero: the number of points for which the EENN model is wrong at all exits. This is used to account for trivial solutions of the monotonicity desideratum: ˆ γ c m ( x ) = 0 , ∀ m . Naturally, the smaller the % zero , the better.

Results are displayed in Table 1. Comparing our PA model against the MSDNet baseline, we observe a significant improvement in terms of the percentage of test data points that exhibit perfect conditional monotonicity, i.e., % mono. However, this improvement comes at a cost, as fewer points have a correct prediction for at least one exit. Despite this trade-off, the magnitude of improvement in % mono is larger compared to the degradation of % zero across all datasets considered, which demonstrates that our PA method improves the conditional monotonicity of ˆ γ c m ( x ) in a non-trivial manner.

Although the Caching Anytime (CA) method outperforms PA in terms of % mono across all datasets, it is worth noting that a considerable portion of CA's outperformance can be attributed to trivial solutions (i.e., ˆ γ c m ( x ) = 0 , ∀ m ), which is reflected by CA's poor performance in terms of % zero. Furthermore, the fact that neither CA nor PA fully achieves conditional monotonicity in prediction correctness (i.e., % mono = 100 ) highlights that there is still potential for further improvement in endowing EENNs with conditional monotonicity.

B.2 Product Anytime (PA) Ablations

To better understand the role of different components in our proposed PA method, we conduct several ablations and present the results in Figure 8. In accordance with the theoretical results we presented in Section 4.2, we observe that the PoE ensemble combined with the Heaviside activation ( ) yields a model that exhibits complete conditional monotonicity in the full model prediction (i.e., ˆ y ) probability. 9 However, this combination also leads to a model with inferior performance in terms of overall test accuracy (see left plot). As mentioned, this occurs because the model assigns an equal probability to all classes in its support. The ReLU activation resolves this issue by performing on par with Heaviside in terms of monotonicity and on par with exponential, i.e., softmax, in terms of accuracy, thus combining the best of both worlds.

0.65

0.70

0.75

Test Accuracy (

↑

)

1

2

3

4

5

6

7

Early Exit

0.025

0.050

PoE

No Ens.

MoE

PoE

No Ens.

MoE

0.0

0.1

0.2

0.3

0.4

0.5

y

Probability Decrease

0

10 0

10 1

%of Test Samples (

↓

)

Conditional Monotonicity

ReLU

exponential

Heaviside

0.0

0.1

0.2

0.3

0.4

0.5

y

∗

Probability Decrease

0

10 0

10 1

%of Test Samples (

↓

)

Conditional Monotonicity

Delving deeper into the ReLU activation function and examining various ensembling styles, we also observe in Figure 8 that Mixture-of-Experts with ReLU (MoE-ReLU; ) demonstrates performance comparable to our proposed PA (PoE-ReLU; ) in terms of both overall accuracy and conditional monotonicity. The MoE predictive distribution at the m th exit is defined as:

where a : R K → R ≥ 0 represents a chosen activation function, such as ReLU, and we utilize uniform weights w i = 1 /m . Given this observation, one might question why we prefer PoE over MoE in our final PA model. The rationale for this decision is two-fold. First, the proof presented in Appendix A, which serves as the inspiration for our method, is only valid when PoE ensembling is employed. Second, unlike the PoE-based model, the MoE-based model does not exhibit anytime uncertainty. Instead, it demonstrates constant underconfidence across all exits, as illustrated in Figure 9. Considering these factors, we find PoE to be superior to MoE for anytime prediction purposes and incorporate it into our final proposed solution.

1

2

3

4

5

6

7

Early Exit

0.2

0.3

0.4

0.5

0.6

ECE (

↓

)

MSDNet

PoE-ReLU (PA)

MoE-ReLU

No Ens.-ReLU

1

2

3

4

5

6

7

Early Exit

0.0

0.2

0.4

0.6

0.8

1.0

Avg. Max Class Probability

Early-Exit Ensembles Baselines MoE-exponential ( ) and PoE-exponential ( ) can also be interpreted as early-exit ensemble baselines for our PA approach. Instead of relying only on the current predictive distributions (as we do when reporting backbone EENNs results in Section 6), we ensemble distributions across all exits up to and including the current one. However, it is clear from Figure 8 that ensembling softmax predictions is insufficient for achieving conditional monotonicity, as both approaches significantly underperform compared to our PA. This underscores the importance of the activation function that maps logits to probabilities for improving monotonicity properties in EENNs. In Figure 10, we verify that these findings hold on the ImageNet dataset as well, where we again find that our PA is the only configuration that performs well both in terms of marginal accuracy and conditional monotonicity.

Sigmoid Activation Function So far, we have discussed why the ReLU function is preferred over the Heaviside or exponential (softmax) functions for transforming logits into probabilities in our setting. We also considered the sigmoid activation function; however, upon further examination, we identified two drawbacks that make it less suitable in our context. Firstly, the sigmoid function lacks the "nullifying" effect of ReLU. As a result, the support for classes diminishes more slowly, and we do not observe the same anytime uncertainty behavior as we do with ReLU (see Section 6.2). Secondly, logits with larger values all map to the same value, approximately 1, under the sigmoid function. This can negatively impact accuracy, as it does not preserve the ranking. Figure 11 illustrates the results using the sigmoid activation function for the CIFAR-100 dataset with MSDNet as a backbone: our choice of ReLU clearly outperforms the sigmoid activation.

0.55

0.60

0.65

0.70

Test Accuracy (

↑

)

1

2

3

4

5

Early Exit

0

PoE

No Ens.

MoE

PoE

No Ens.

MoE

0.0

0.1

0.2

0.3

0.4

0.5

y

Probability Decrease

0

10 0

10 1

%of Test Samples (

↓

)

Conditional Monotonicity

ReLU

exponential

Heaviside

0.0

0.1

0.2

0.3

0.4

0.5

y

∗

Probability Decrease

0

10 0

10 1

%of Test Samples (

↓

)

Conditional Monotonicity

2

4

6

Early Exit

0.60

0.65

0.70

0.75

Test Accuracy (

↑

)

baseline

ReLU (PA)

Sigmoid

0.0

0.1

0.2

0.3

0.4

0.5

y

∗

) Prob. Decrease

0

10 0

10 1

%of Test Samples (

↓

)

2

4

6

Early Exit

0.2

0.3

0.4

0.5

0.6

ECE (

↓

)

B.3 Closing the Calibration Gap

Here we outline two modifications to our proposed PA method aimed at reducing the calibration gap reported in Section 6.3.

B.3.1 Adaptive Thresholds

We first introduce an extension of our Product Anytime (PA) method, which enables navigating the monotonicity-calibration trade-off in scenarios when a hold-out validation dataset is available.

Recall that ReLU is used as an activation function in PA to map logits to probabilities. This implies that 0 is used as a threshold, determining which classes 'survive' and get propagated further, and which do not. However, 0 is arguably an arbitrary threshold, and one could consider another positive constant b ∈ R + instead. It is important to note that the choice of b directly influences the size of the support of the predictive likelihood. As illustrated in Figure 13, a larger value of b results in a smaller number of classes with positive probability (as expected). This larger threshold does also lead to a reduction in the coverage of PA's predictive sets with respect to the ground-truth class. However, the

1

2

3

4

5

6

7

Early Exit

0.0

0.2

0.4

0.6

ECE (

↓

)

1

2

3

4

5

6

7

Early Exit

0.80

0.83

0.85

0.88

0.90

0.93

OC AUROC @ 0.5% (

↑

)

MSDNet

PA (ours)

PA-FT (ours)

CA

1

2

3

4

5

6

7

10

20

30

40

PA Set Size

1

2

3

4

5

6

7

Early Exit

0.88

0.90

0.92

0.94

0.96

0.98

1.00

PA Set

y

∗

Coverage

1

2

3

4

5

6

7

0.66

0.68

0.70

0.72

0.74

0.76

Test Accuracy (

↑

)

PA Threshold

0

1

3

5

7

decrease in coverage does not significantly affect the overall accuracy of the model, as shown in the far right plot.

Expanding on this concept, one can also consider adaptive thresholding: rather than using the same threshold value for all data points, the model could dynamically adjust the threshold in the activation function based on the current example. Ideally, when the model is confident in its prediction, a higher value of b is employed, resulting in smaller predictive sets. Conversely, when faced with a more challenging example, a lower threshold is utilized to capture the model's uncertainty.

Switching to ReLU with adaptive thresholding, the predictive likelihood at i th exit (before ensembling) is given by:

where τ : R K → R + maps the vector of logits f ( x ) ∈ R K to a non-negative threshold. 10

Here, we consider threshold functions of the form τ ( f ( x ) ) = C · p ψ ( f ( x ) ) , where p ψ : R K → [0 , 1] is a model that estimates the probability of the datapoint x being classified correctly, and C is a constant that ensures the outputs of τ have a similar magnitude to that of logits. To estimate C and ψ , we rely on the hold-out validation dataset. Specifically, we fit a binary logistic regression model using labels y τ n := [ˆ y n = y ∗ n ] . Additionally, we only use the K = 3 largest logits as input to the regression model, as we have empirically found that excluding the rest does not significantly impact the performance. To find C , we perform a simple grid search and select the value that yields the best monotonicity-calibration trade-off on the validation dataset. We fit a thresholding model only at the first exit and then reuse it at later exits, as this approach yielded satisfactory performance for the purposes of our analysis. It is likely that fitting a separate thresholding model at each exit would lead to further performance benefits.

In Figure 14, we observe that PA with adaptive thresholding still improves over baseline in terms of monotonicity (middle plot, note the log-scale of y -axis). Although the adaptive thresholding approach demonstrates worse monotonicity when compared to the PA algorithm with a static threshold (i.e., τ = 0 ), it outperforms the PA method in terms of calibration, as measured by the Expected Calibration Error (ECE). To further elucidate the influence of adaptive thresholding, Figure 15 presents the groundtruth probability trajectories for a random selection of test data points. In general, it is apparent

1

2

3

4

5

6

7

Early Exit

0.62

0.64

0.66

0.68

0.70

0.72

Test Accuracy (

↑

)

MSDNet

PA

PA (Adapt Thres)

0.0

0.1

0.2

0.3

0.4

0.5

y

∗

Probability Decrease

0

10 0

10 1

%of Test Samples (

↓

)

1

2

3

4

5

6

7

Early Exit

0.1

0.2

0.3

0.4

0.5

0.6

ECE (

↓

)

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

0.8

1.0

Ground-Truth Probability

MSDNet

1

2

3

4

5

6

7

Early Exit

PA

1

2

3

4

5

6

7

PA (Adapt Thres)

that adaptive thresholding can serve as an effective strategy for improving the calibration of the PA method, albeit at some expense to both conditional monotonicity and overall accuracy.

B.3.2 PA Finetuning

The post-hoc nature of our proposed PA method is attractive due to its minimal implementation overhead. However, as mentioned in Section 6.3, it seems to adversely affect calibration, according to ECE-like metrics, when compared to baseline models. This discrepancy might be due to a mismatch between the objectives during training and testing in the PA method. To examine this hypothesis, we consider training our model using the PA predictive likelihood (as given by Equation 4) and assess its effect on calibration.

Given the product structure and the non-smooth activation function (ReLU) employed in PA's likelihood, the training can be quite challenging. To enhance the stability of the training process, we introduce two modifications. Firstly, instead of training the model from scratch, we begin by training with a standard EENN objective (see Equation 1) with a softmax likelihood and then finetune the model using the PA likelihood at the end. 11 Secondly, we substitute the ReLU activation function with a Softplus function 12 , which we found aids in stabilizing the model's convergence during training. Consequently, we also utilize Softplus to map logits to (unnormalized) probabilities during testing. Moreover, we found that using ensemble weights w m = 1 is more effective for models finetuned

1

2

3

4

5

6

7

0.89

0.90

0.91

0.92

0.93

Test Accuracy (

↑

)

MSDNet

PA

PA Finetune

0.1

0.2

0.3

0.4

0.5

0

10 0

10 1

%of Test Samples (

↓

)

CIFAR-10

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

ECE (

↓

)

1

2

3

4

5

6

7

0.65

0.70

0.75

Test Accuracy (

↑

)

MSDNet

PA

PA Finetune

0.1

0.2

0.3

0.4

0.5

0

10 0

10 1

%of Test Samples (

↓

)

CIFAR-100

1

2

3

4

5

6

7

0.0

0.2

0.4

0.6

ECE (

↓

)

1

2

3

4

5

Early Exit

0.55

0.60

0.65

0.70

Test Accuracy (

↑

)

MSDNet

PA

PA Finetune

0.1

0.2

0.3

0.4

0.5

y

∗

Probability Decrease

0

10 0

10 1

%of Test Samples (

↓

)

ImageNet

1

2

3

4

5

Early Exit

0.0

0.2

0.4

0.6

ECE (

↓

)

with the PA objective, as opposed to our default choice of w m = m/M presented in the main paper for post-hoc PA.

The results are presented in Figure 16. Focusing initially on (marginal) test accuracy ( right ), we note that PA training ( ) has a slight negative impact on performance, particularly when contrasted with the post-hoc PA ( ). Similarly, it underperforms post-hoc PA in terms of conditional monotonicity ( middle ), but it considerably surpasses the baseline MSDNet model ( ). When considering ECE, it becomes clear that fine-tuning with PA leads to significant improvements over post-hoc PA, thus substantiating our hypothesis that the calibration gap mainly arises from the discrepancy between train-time and test-time objectives. On the CIFAR-100 dataset, PA with finetuning even outperforms the baseline MSDNet in terms of ECE.

Based on these observations, we conclude that integrating PA during the training of EENNs is an effective strategy for bridging the calibration gap, though it incurs a minor cost in terms of overall accuracy and conditional monotonicity. We note that these costs could be mitigated (or ideally, completely eliminated) through further stabilization of product training. As such, we view this as a promising avenue for future research.

B.3.3 Relevance of ECE in Anytime Prediction Setting

In Section 6.3, we pointed out poor Expected Calibration Error (ECE) in the initial exits as one of the limitations of our proposed PA method. It is important to note, however, that no decision in the anytime setting is based on model (or prediction) confidence, i.e., max y p m ( y | x ) . This contrasts with the budgeted batch classification setting [Huang et al., 2018], where the decision on when to exit could be based on the model confidence. Thus, we believe that our PA potentially undermining

model confidence, as evident by poor ECE in the earlier exits, is not a limitation that would seriously impact the merit of our solution. Especially since we focus solely on the anytime setting, where, we would argue, non-monotone behavior in terms of performance quality is a much more serious issue.

However, it is true that model/prediction confidence, i.e. max y p m ( y | x ) , could be useful in the anytime scenario as a proxy for model uncertainty. This allows the model-when prompted by its environment to make a prediction-to also offer an estimate of uncertainty alongside that prediction. However, given that our PA potentially adversely affects model confidence in earlier exits, we caution against its use. Instead, we recommend alternative measures of uncertainty that better align with our model, such as the conformal set size (c.f., Section 6.2).

B.4 NLP Experiments

In this section, we conduct NLP experiments with the aim of demonstrating that our proposed approaches for adapting EENNs for anytime inference are generalizable across different data modalities.

Specifically, we employ an early-exit model from Schwartz et al. [2020], in which the BERT model is adapted for accelerated inference. The considered datasets are (1) IMDB, a binary sentiment classification dataset; (2) AG, a news topic identification dataset with |Y| = 4 ; and (3) SNLI, where the objective is to predict whether a hypothesis sentence agrees with, contradicts, or is neutral in relation to a premise sentence ( |Y| = 3 ). We adhere to the same data preprocessing and model training procedures as outlined in Schwartz et al. [2020].

To account for the smaller number of classes in these NLP datasets, we shift logits for a (small) constant prior to applying ReLU activation function. Hence the PA likelihood at m th exit has the form (before applying product ensembling):

where C represents the cardinality of Y . This helps with reducing the possibility of PA collapsing to a zero distribution.

The results are depicted in Figure 17. Similar to our image classification experiments discussed in Section 6.1, we find that our proposed modifications yield comparable overall accuracy, as shown in the left plot. Considering conditional monotonicity ( middle plot), at least one of our two suggested methods outperforms the baseline model on every dataset. However, the degree of improvement in monotonicity is considerably smaller in comparison to our image classification experiments, with the PA model even failing to yield consistent improvement on certain datasets, such as IMDB. We attribute this not to the change in data modality, but rather to the fewer number of classes in the NLP datasets used in this section as compared to the image datasets used in Section 6.1.

1

2

3

4

Early Exit

0.75

0.80

0.85

0.90

Test Accuracy (

↑

)

baseline

PA (ours)

CA

0.0

0.1

0.2

0.3

0.4

0.5

y

∗

Probability Decrease

0

10 0

10 1

%of Test Samples (

↓

)

0.0

0.1

0.2

0.3

0.4

0.5

y

Probability Decrease

0

10 0

10 1

%of Test Samples (

↓

)

IMDB

AG

SNLI

B.5 Overthinking and Hindsight Improvability

In this section, we delve deeper into concepts that have been previously introduced in EENN literature and that share a connection with the idea of conditional monotonicity, which is the focus of our study.

In Kaya et al. [2019], the authors discovered that EENNs, for some data points, yield correct predictions early on but then produce incorrect results at later exits 13 -a phenomenon they term overthinking . Consequently, the authors noted that early-exiting in deep neural networks is not only beneficial for speeding up inference but can sometimes even lead to improved overall test accuracy. To measure overthinking, they considered the difference in performance between an oracle model that exits at the first exit with the correct answer 14 and the performance of the full EENN p M ( y | x ) . In the anytime setting, we desire a model with the lowest overthinking measure, i.e., 0, as this indicates that evaluating more layers in the early-exit network does not negatively impact performance.

In Figure 18, we present the results of the overthinking analysis for the MSDNet [Huang et al., 2018] model. We observe that both of our proposed methods significantly improve in terms of the overthinking measure ( left ), making them more applicable in anytime scenarios.

CIFAR100

ImageNet

0.00

0.03

0.05

0.08

0.10

0.12

0.15

Overthinking Measure (

↓

)

MSDNet

PA (ours)

CA

CIFAR100

ImageNet

0.70

0.75

0.80

0.85

0.90

↑

)

CIFAR100

ImageNet

0.60

0.65

0.70

0.75

0.80

↑

)

We also note that a decrease in the overthinking measure can result from either (1) a reduction in the (theoretical) performance of the oracle model or (2) an improvement in the (actual) performance of the full EENN. As can be seen in Figure 18 ( middle and right ), our proposed modifications mainly improve the overthinking measure through (1). To explore this further, in Figure 19 we plot the number of test points that the model learned , i.e., correctly predicted for the first time, and the number of points that the model forgot , i.e., incorrectly predicted for the first time, at each early exit. Both PA and CA show a clear strength in minimizing forgetting, although they do so at the cost of diminishing the baseline model's ability to learn correct predictions. 15 These results indicate that there is further room for improvement in the adaptation of EENNs for anytime computation, as the ideal model would minimize forgetting without compromising its learning abilities.

1

3

5

7

9

11

13

15

Early Exit

0.55

0.60

0.65

0.70

0.75

Test Accuracy (

↑

)

0.0

0.1

0.2

0.3

0.4

0.5

y

∗

Probability Decrease

0

20

40

60

80

%of Test Samples (

↓

)

ZTW

PA (ours)

CA

1

3

5

7

9

11

13

15

Early Exit

0.0

0.1

0.2

0.3

0.4

0.5

Hindsight Improvability (

↓

)

2

3

4

5

6

7

Early Exit

0

2

4

6

8

10

%of Test Samples

Learn (

↑

)

MSDNet

PA (ours)

CA

2

3

4

5

6

7

Early Exit

Forget (

↓

)

In a follow-up study by Wolczyk et al. [2021], the hindsight improvability (HI) measure is introduced. For each early exit, this measure takes into account the set of incorrectly predicted points and computes the percentage of those that were accurately predicted at any of the previous exits. In an anytime setting, an ideal model would consistently exhibit an HI measure of 0 . As evidenced by the results in Figure 20, our post-hoc modifications substantially enhance the HI measure ( right plot) while only causing a minor negative impact on overall accuracy ( left plot). Similar to the overthinking measure, the observation that HI > 0 for our proposed methods suggests a potential for further improvements.

B.6 Deep Ensembles and Regression

Although we have presented our results for classification with early-exit neural networks, we can easily extend them to handle generic deep ensembles and regression settings.

Deep Ensembles While applying our method to deep ensembles might seem strange since they have no implicit ordering as in the case of early-exit NNs, we note that ensembles are often evaluated sequentially in low resource settings where there is not enough memory to evaluate all of the ensemble members concurrently. In such settings, there is an implicit, if arbitrary, ordering, and we may still require anytime properties so that we can stop evaluating additional ensemble members at any point. Therefore, we can apply both Product Anytime and Caching Anytime-as in Section 4-to deep ensembles, where p m ( y | x ) now represents the m -th ensemble member, rather than the m -the early exit. An earlier version of this work [Allingham and Nalisnick, 2022] focused on product anytime in the deep ensembles case. However, in that work, the focus was also on training from scratch rather than the post-hoc setting we consider in this work. Training from scratch is challenging due to, among other things, 16 instabilities caused by the Product-of-Experts and the bounded support of the likelihoods. Thus, the PA results from Allingham and Nalisnick [2022] are worse in terms of predictive performance than the standard ensemble baseline. Note that our theoretical results in Appendix A also apply in the deep ensembles case. 17 Figure 21 shows qualitative results for our anytime ensembles applied to a simple classification task. As expected, the support shrinks as we add more multiplicands to the product.

Regression Product Anytime can be applied to regression by using a uniform likelihood at each early-exit (or ensemble member)

which results in a product likelihood

where a max = max ( { a m } ) and b min = min ( { b m } ) . That is, the lower and upper bounds of the product are, respectively, the maximum-lower and minimum-upper bounds of the multiplicands. As we add more multiplicands, the width of the product density can only get smaller or stay the same, and thus the probability of the modal prediction p 1: m (˜ y | x ) can only increase or stay the same, as in the classification case. Figure 22 shows qualitative results for anytime ensembles applied to a simple regression task. We see that the order of evaluation is unimportant. Firstly, as expected, the final result is the same, and in both cases, the support size always decreases or is unchanged. Secondly, the fit has almost converged within a few evaluations, and the last few multiplicands tend to have smaller effects. Note that the prediction bounds cover the data well even for the poorer early fits. Finally, note that there is no support outside of the training data range-we consider this data as out-of-distribution.

B.7 Anytime Uncertainty: Conditional Monotonicity

In Section 6.2, we presented the results of our uncertainty experiments where we observed that, marginally, our PA model demonstrates an anytime uncertainty pattern: the model starts with high uncertainty early on and then progressively becomes more confident at later exits.

In this section, we supplement these experiments with an analysis at the conditional level. Specifically, for a given x ∈ X , we compute the uncertainty (for instance, entropy) at each early exit: { U m ( x ) } M m =1 . Following the approach we used in Section 3 for performance quality sequences, we define a maximum increase in uncertainty as MUI ( x ) := max m ′ >m { max( U m ′ ( x ) -U m ( x ) , 0) } . Next, for a range of suitably chosen thresholds τ ∈ R ≥ 0 , we compute the number of test instances that exhibit an uncertainty increase surpassing a given threshold N τ := ∑ n [ MUI ( x n ) ≥ τ ] . Note that a monotone oracle model would exhibit entirely non-increasing uncertainty trajectories, meaning N τ = 0 , ∀ τ .

0.0

0.2

0.4

0.6

0.8

1.0

0

10 0

10 1

10 2

%of Test Samples (

↓

)

CIFAR-10

baseline

PA (ours)

CA

0.0

0.2

0.4

0.6

0.8

1.0

CIFAR-100

0.0

0.2

0.4

0.6

0.8

1.0

ImageNet

The results for entropy and conformal set size are depicted in Figures 23 and 24, respectively. We observe that an EENN like MSDNet exhibits non-monotone behavior also in its uncertainty estimates across different exits. This means that there are data points where the model starts with high confidence in the initial exits but becomes less certain in subsequent stages. Such a trend is problematic in the anytime setting, similar to the issues of non-monotonicity in performance quality (c.f., Section 3). It suggests that the model expends additional computational resources only to diminish its confidence in its response. Reassuringly, our PA aids in ensuring monotonicity not only

in terms of performance quality (e.g., ground-truth probability) but also in its uncertainty estimates (e.g., conformal set size) as evident in Figures 24 and 23.

1

2

3

4

5

0

10 0

10 1

%of Test Samples (

↓

)

CIFAR-10

baseline

PA (ours)

CA

1

2

3

4

5

CIFAR-100

1

2

3

4

5

ImageNet

B.8 Controlling the Violations of Conditional Monotonicity

In Section 4.2, we introduced our Product Anytime (PA) method. It relies on the ReLU activation function to map logits to (unnormalized) probabilities, which unfortunately compromises the strong conditional monotonicity guarantees that are present when using the Heaviside activation function.

In this section, we propose a technique that allows for a more nuanced control of violations of conditional guarantees. To illustrate this, observe first that ReLU and Heaviside are specific cases of the following activation function:

where C b := max (1 , 1 /b ) . Notice how lim b →∞ a ( x ) = ReLU ( x ) and lim b → 0 + a ( x ) = Heaviside ( x ) . Therefore, by adjusting the clipping threshold b , we can navigate the accuracymonotonicity trade-off as needed. This is demonstrated on the CIFAR-10 dataset in Figure 25.

1

2

3

4

5

6

7

Early Exit

0.2

0.4

0.6

0.8

Test Accuracy (

↑

)

0.0

0.1

0.2

0.3

0.4

0.5

y

∗

Probability Decrease

0

2

4

6

%of Test Samples (

↓

)

0.0

0.1

0.2

0.3

0.4

0.5

y

Probability Decrease

0.0

0.5

1.0

1.5

2.0

%of Test Samples (

↓

)

b

0.1

1

5

10

100

B.9 Impact of Calibration on Conditional Monotonicity

In this section, we study whether monotonicity could be achieved by calibrating the underlying EENN rather than employing our proposed PA. To answer this, we conducted an experiment where we calibrated MSDNet and analyzed its effect on monotonicity. For the calibration, we utilized three standard techniques: temperature scaling [Guo et al., 2017], deep ensembles [Lakshminarayanan et al., 2017], and last-layer Laplace [Daxberger et al., 2021]. In Figure 26, we present results for MSDNet on CIFAR-100. While all three approaches lead to better calibration in terms of ECE (right plot), our PA significantly outperforms them in conditional monotonicity (middle plot). Moreover, all three baselines are (arguably) more complex compared to our PA: temperature scaling requires validation data, deep ensemble has M-times more parameters (M being the ensemble size), and Laplace is significantly slower compared to PA since we need test-time sampling to estimate the predictive posterior at each exit.

However, simply improving calibration might not be expected to improve monotonicity. Thus, in Figure 26, we also provide results for combining the above uncertainty calibration techniques with our CA baseline. We see that this combination provides further improvements w.r.t. monotonicity -- the improved calibration allows for better caching of the correct prediction. However, the monotonicity of these caching-and-calibrated methods still underperforms compared to PA despite being more complicated. This demonstrates that monotonicity can not be achieved via calibration alone and further underlines the value of our proposed approach.

1

2

3

4

5

6

7

Early Exit

0.650

0.675

0.700

0.725

0.750

0.775

Test Accuracy (

↑

)

no CA

CA

no CA

CA

0.0

0.1

0.2

0.3

0.4

0.5

y

∗

) Prob. Decrease

0

10 0

10 1

10 2

%of Test Samples (

↓

)

1

2

3

4

5

6

7

Early Exit

0.0

0.2

0.4

0.6

ECE (

↓

)

baseline

PA

Laplace

DE(5)

Temp

B.10 Budgeted Batch Classification

In this section, we conduct a budgeted batch classification experiment using MSDNet [Huang et al., 2018] as the backbone. In budgeted batch classification, a limited amount of compute is available to classify a batch of datapoints. The goal is to design a model that can allocate resources smartly between easy and hard examples to maximize the overall performance, such as accuracy.

The results are shown in Figure 27. Since MSDNet outperforms our PA at most computational budgets, we conclude that the monotonicity is less beneficial in budgeted batch classification. However, as seen on CIFAR datasets, the non-monotonicity of MSDNet is also concerning in this setting for larger values of FLOPs, MSDNet performance starts to drop, while PA keeps monotonically increasing and surpasses MSDNet. Based on these results, we suggest 'turning on' our PA only for anytime prediction and recommend that the user sticks to the original EENN in the budgeted batch classification. However, this certainly warrants further investigation (as seen by the performance benefits of our PA on CIFAR for larger FLOPs).

0.02

0.03

0.04

0.05

0.06

90

91

92

Test Accuracy (

↑

)

CIFAR-10

MSDNet

PA

CA

0.02

0.04

0.06

FLOPs [1e9]

65.0

67.5

70.0

72.5

CIFAR-100

MSDNet

PA

CA

0.4

0.6

0.8

1.0

1.2

60

65

70

ImageNet

MSDNet

PA

CA

B.11 Additional Plots

Here, we provide additional results that were omitted in Section 6.1 to enhance the readability of the plots there. Specifically, in Figure 28, we include test accuracy results, along with their respective error bars, for the MSDNet [Huang et al., 2018] and IMTA [Li et al., 2019] models. In Figures 31 and 32, we present results for the L2W model [Han et al., 2022b]. We observe that our findings also generalize to this EENN - our PA preserves the marginal accuracy while significantly improving conditional monotonicity across all considered datasets.

As for the DViT [Wang et al., 2021] model, our attempts to train this model from scratch using the published code 18 were unsuccessful. Hence, we resorted to working with the pre-trained models provided by the authors. They only published a single pre-trained model for each model version, and as a result, instead of presenting results for independent runs with varying random seeds, we display the results for all provided DViT model instantiations in Figure 29 and 30.

1

2

3

4

5

6

7

0.89

0.90

0.91

0.92

0.93

Test Accuracy (

↑

)

CIFAR-10

1

2

3

4

5

6

7

Early Exit

0.62

0.64

0.66

0.68

0.70

0.72

0.74

0.76

CIFAR-100

baseline

PA (ours)

CA

baseline

PA (ours)

CA

1

2

3

4

5

0.58

0.60

0.62

0.65

0.68

0.70

0.73

ImageNet

MSDNet

IMTA

1

2

3

0.965

0.970

0.975

0.980

0.985

Test Accuracy (

↑

)

CIFAR-10

DVT T2t vit 14

DVT Deit small

DVT T2t vit 12

1

2

3

Early Exit

0.84

0.86

0.88

0.90

CIFAR-100

1

2

3

0.70

0.75

0.80

ImageNet

baseline

PA (ours)

CA

baseline

PA (ours)

CA

0.0

0.1

0.2

0.3

0.4

0.5

0

10 0

10 1

%of Test Samples (

↓

)

CIFAR-10

baseline

PA (ours)

CA

0.0

0.1

0.2

0.3

0.4

0.5

y

∗

) Probability Decrease

CIFAR-100

0.0

0.1

0.2

0.3

0.4

0.5

ImageNet

DVT T2t vit 14

DVT Deit small

DVT T2t vit 12

1

2

3

4

5

0.89

0.90

0.91

0.92

0.93

Test Accuracy (

↑

)

CIFAR-10

1

2

3

4

5

Early Exit

0.625

0.650

0.675

0.700

0.725

0.750

CIFAR-100

baseline

PA (ours)

CA

baseline

PA (ours)

CA

1

2

3

4

5

0.625

0.650

0.675

0.700

0.725

0.750

ImageNet

L2W

0.0

0.1

0.2

0.3

0.4

0.5

0

10 0

10 1

%of Test Samples (

↓

)

CIFAR-10

0.0

0.1

0.2

0.3

0.4

0.5

y

∗

) Probability Decrease

CIFAR-100

baseline

PA (ours)

CA

0.0

0.1

0.2

0.3

0.4

0.5

ImageNet

L2W

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies an interesting problem of early-exit classification networks when applied to anytime prediction tasks. Specifically, the authors empirically demonstrate that in an early-exit network, the prediction confidence scores of its different exits for the same sample are not monotonically increased. Then the authors propose a method to recalculate the prediction confidence of different exits. This method is shown effective in improving the conditional monotonicity of different exits' predictions.\n\nSTRENGTHS:\n1. The studied problem is interesting and important;\n2. The proposed method is theoretically guaranteed;\n3. The experiment is comprehensive, covering different architectures and datasets.\n\nWEAKNESSES:\nI have some concerns as follows:\n\n1. The results on CIFAR datasets (Fig. 2 left and middle) are confusing: the IMTA baselines achieved lower/similar accuracy than the MSDNet baseline in Fig. 2. However, IMTA is apparently a stronger method than the original baseline. This makes the results much less convincing to me. Moreover, I did not find the ImageNet results using the proposed PA for MSDNet and IMTA (blue and green dashed lines) in Fig 3 (right). Only DVT results (orange dashed line) are presented. \n\n2. The experiment results in Fig. 4 indeed show that the ground-truth probability trajectories monotonically increase by using the proposed PA. However, **how can such monotonicity improve the anytime prediction task**, is not discussed. Is there any metric that can measure the anytime performance of a model? Now the authors only show us that the overall accuracy can be preserved (or improved a little).\n\n3. It is recommended to conduct experiments on more recent multi-exit models, such as L2W-MSDNet [1].\n\n4. I'm curious whether better monotonicity is beneficial in the budgeted dynamic inference setting (the smooth accuracy-FLOPs curves reported in MSDNet and IMTA papers).\n\n[1] Han, Yizeng, et al. \"Learning to Weight Samples for Dynamic Early-Exiting Networks.\" ECCV 2022.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nAs I stated in the weakness part, the benefits of the obtained better monotonicity are not discussed in depth.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission focuses on multi-exit early networks, from the perspective of anytime inference that can accommodate varying computational budgets. Such models produce a progressive refinement of the final prediction, offering the opportunity to \"exit-early\", providing a meaningful prediction. The main motivation of this work is to enforce conditional monotonicity on the predictions of successive early-exits, such as the output quality consistently improves as the inference process progresses. The proposed solution adopts the Product-of-Experts formulation and can be applied post-training, to achieve monotonicity on prediction quality as well as prediction uncertainty.\n\nSTRENGTHS:\n-The submission identifies and studies an interesting challenge with early-exit models, in the context of anytime inference, which is often overlooked by relevant literature.\n\n-The papers analysis is wide and insightful, providing numerous empirical results that can motivate further research in the topic. \n\n-The proposed methodology is clearly explained, and experiments are conducted on a range of traditional benchmarks, consistently to the practise of relevant literature.\n\nWEAKNESSES:\n-Certain design choices are not adequately justified and further comparisons/ablations should be included on the main paper. More specifically, it will be beneficial to examine: i) how the proposed methodology would behave in frozen backbone early-exit models and ii) in comparison to simple ensemble between early-exit models that comprises a intuitive baseline.\n\n-Additionally, it is claimed that the proposed approach can also be applied at training time, but this scenario is not adequately evaluated. Would the proposed methodology incur additional error or under-confidence on the  shallower exits (instead of improving the deeper ones) in a quest to achieve the desired monotonicity? And if so how can this be mitigated?\n\n-Figure 2-right may suggest a scalability issue for the proposed approach as the baseline dominates the depth-accuracy trade-off in contrast to other datasets. This limitation should be further investigated.\n\n-The motivation of progressively increasing predictive confidence, may contradict the general benefits of uncertainty estimation in early-exit models (e.g. for input dependent inference). One could argue that uncertainty should ideally be calibrated with the probability of the ground-truth class; rather than penalised on early-classifier even if the provided prediction is correct. \n\n-Finally, the discussion of the results (e.g. on Fig. 2) could benefit from providing some quantitative conclusions on text (e.g. average/median improvement etc.).\n\nQUESTIONS:\nIn accordance to the discussion in \"Weaknesses\" please clarify:\n1. How does the proposed methodology compare to vanilla early-exit ensembles in terms of both accuracy and uncertainty monotonicity (should be added in Fig.2 analysis).\n2. Is the proposed methodology still relevant if applied on early-exit models trained via the frozen-backbone methodology (i.e. is the behaviour of Fig1-middle still evident in this case; and resolved by the proposed approach)?\n3. Is there a scalability issue when the proposed method is applied on ImageNet? What aspect (sample size, number of classes etc) seem to mostly cause this effect ?\n4. Is the predictive accuracy and uncertainty of early-classifier reduced when applying the proposed methodology at training time ?\n5. Is it indeed beneficial to decrease prediction confidence on shallow classifiers, even when their predictions are correct (e.g. on easier samples) ?\n6. In the case of the IMTA early-exit model, where knowledge distillation is applied, it is expected that the prediction uncertainty of the resulting model will be affected. How does this affect the analysis of Sec.6.2; do the results remain consistent for IMTA models ?\n\nLIMITATIONS:\nSome limitations are clearly discussed in a dedicated subsection of the paper. Following the above discussion, it may be beneficial to broaden this discussion accordingly.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new approach, called Product Anytime (PA), for monotonic confidence estimation in early-exit neural networks. Inspired by the Product-of-Experts (PoE) approach, PA takes the normalized confidence with respect to the product of ReLU thresholded prediction logits. The authors show through experiments that PA achieves similar accuracy curves as the original softmax logits, but improves the monotonicity of early-exited prediction confidences.\n\nSTRENGTHS:\n1. The authors tackle a practical and important problem that should be solved in early-exit neural networks.\n2. The intuition of monotonicity from the Product-of-Experts (PoE) makes sense to me.\n3. The provided evaluations are comprehensive and support the claims made in the paper.\n\nWEAKNESSES:\n1. Although I agree with the authors monotonicity is an important property in early-exit NNs. I am not completely convinced whether it is meaningful to design a specialized algorithm to achieve it. My point is, can we use the confidence/uncertainty calibration algorithms [1, 2] in the literature, that seem to produce relatively accurate confidence estimation to replace the proposed algorithm? On one hand, the uncertainty calibration algorithm better reflects the accuracy of the model prediction. At the same time both approaches (uncertainty estimation and the proposed algorithm) can not provide theoretical guarantee as PoE does. I think uncertainty quantification should provide better monotonicity than softmax confidence. On the other hand, uncertainty calibration gives better intuition on which sample should be early exited, since it does not have the \"calibration gap\" issue observed in the proposed PA method.\n2. I am unsure how the proposed algorithm should be used in the practical runtime deployment scenario.\n3. The proposed approach seems to have multiple obvious limitations, such as the zero distribution due to non-overlapping labels among stages, the poor confidence at initial stages, and suboptimal performance for applications with the limited number of classes.\n\n\n[1] Gal, Yarin, and Zoubin Ghahramani. \"Dropout as a Bayesian approximation: Representing model uncertainty in deep learning.\" international conference on machine learning. PMLR, 2016.\n\n[2] Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. \"Simple and scalable predictive uncertainty estimation using deep ensembles.\" Advances in neural information processing systems 30 (2017).\n\nQUESTIONS:\nWhat is the unique advantage of the proposed approach compared to conventional uncertainty estimation algorithms applied in early-exit neural networks?\n\nLIMITATIONS:\nYes, the authors have a separate section to discuss the limitations of the proposed technical solution.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper first reviews the definition & key properties of anytime models (Zilberstein, 1996) and argues to focus on equipping the EENNs with conditional monotonicity (in addition to their built-in interruptibility, Sec 2). The paper then verifies that existing EENNs lack this critical property (Fig 1), proposes the PoE-based post-hoc modification with ReLU relaxation to encourage conditional monotonicity (Sec 4.2), and validates its accuracy (Fig 2), monotonicity (Fig 3) and uncertainty (Fig 4) on CIFAR-10/100 and ImageNet with MSDNet, IMTA and DViT. The paper also presents results on ablations (Sec B.2), calibration gap (adaptive thresholding & fine-tuning, Sec B.3), NLP (Sec B.4), ensembling, regression (Sec B.6) and different degrees of ReLU relaxation (Sec B.8).\n\nSTRENGTHS:\n+ [Originality] The paper novelly modifies the PoE formulation to achieve conditional monotonicity, making it sufficiently and advantageously different from existing works e.g. (Wolczyk et al., 2021).\n+ [Clarity] The paper is well written and easy to follow. All figures are well made and the appendix is very informative.\n+ [Quality] The paper is of good quality, with the proposed method relatively well motivated and designed, and the empirical evaluation (including results in the appendix) relatively comprehensive, although further improvements can be made (see Weaknesses).\n+ [Significance] Given the importance of the topic, even though the results are not necessarily perfect (but decent enough), I think this work is reasonably promising and could aid/inspire future research.\n\nWEAKNESSES:\n- [Evaluation] \n1) Given the criticality of the relaxation/approximation of the Heaviside function, it’s highly desirable to see more thorough analyses of and comparisons between different approximations, e.g. sigmoid (seemingly much more logical than ReLU), clipped ReLU, learnable Padé approximation [1], etc., in addition to Sec B.8.\n2) The exponent $i/M$ of the proposed PA method (Eq 4) is clearly a powerful inductive bias ensuring the network starts with low confidence, which should deserve more analyses such as: (a) How do the baselines perform when equipped with this simple feature or corresponding adjustments on the softmax temperature? (b) How does $i/M$ compare to other progressions or constant exponents? (c) Does $i/M$ adversely limit the confidence on simpler test samples such that e.g. early exits become ineffective?\n\n[1] Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks, ICLR, 2020.\n\nQUESTIONS:\n- Are $w_i$ and $b_i$ in Eq 4 learnable during fine-tuning? Why or why not?\n\nLIMITATIONS:\nThe authors have reasonably addressed the paper’s limitations in Sec 6.3.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a post-hoc modification of early-exit neural networks. Specifically, the authors focus on the conditional monotonicity out of four properties presented in a prior work. A training-based method is also proposed to improve the model confidence.\n\nSTRENGTHS:\nThe proposed method based on product-of-experts improves conditional monotonicity and its effectiveness is also supported by theoretical results.\n\nWEAKNESSES:\n- Some writing is somewhat verbose/redundant. For example, while properties of anytime models take a considerable amount of space, they are not that meaningful at last.\n\n- Plots on \"% of Test Samples vs. Max y* Prob. Decrease\" should be described in a better way, in that what it means and why do we care about it.\n\n- As addressed in Fig. 5, the proposed method results in hurting model confidence. It is good to see that fine-tuning with the proposed learning objective can improve this, but it makes the point of this work distracted; recall that this paper means to propose a post-hoc method.\n\n- How PA Finetune is done is not explained enough.\n\n- I am not sure if we really need to care about monotonicity. Enforcing to have monotonicity might be harmful, as shown in some curves in Fig. 2. Instead, assuming that the model is calibrated well, we could rely on the confidence of intermediate result and exit early regardless of the time budget for the best performance.\n\nQUESTIONS:\nPlease address concerns in weaknesses.\n\n--- post rebuttal\n\nI appreciate the authors' active responses during the author-reviewer discussion period. Most of my concerns have been solved, however, the main concern on the necessity of (conditional) monotonicity for EENNs is still not yet solved. Hence, I increase my rating, but not by much.\n\nIn my opinion, when proposing a new metric, the metric itself should either 1) be meaningful by itself or 2) have a strong positive correlation with the main goal. The monotonicity is not meaningful by itself (a model producing a constant output has the perfect monotonicity, which is not useful in practice) and the correlation with the overall performance (accuracy) is not so significant. To provide successful cases as examples, ECE in confidence calibration is crucial when considering the reliability of model outputs, and forgetting in continual learning has a positive correlation with the average accuracy.\n\nRegarding the scalability, the monotonicity measure seems scalable, but ECE is not for the proposed method. To me, confidence calibration is one of the most important topic in deep learning for a successful deployment to the real world applications, so non-scalable ECE is my major concern. In short, increasing the monotonicity leads to slightly improve the overall accuracy, while significantly deteriorating confidence calibration in the large-scale experiment on ImageNet.\n\nHowever, given that conditional monotonicity is important (and given that ECE is somewhat problematic), I agree that the post-hoc method proposed by the authors is simple yet effective in terms of the proposed monotonicity metric.\n\nLIMITATIONS:\nNothing special.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}