Fast yet Safe: Early-Exiting with Risk Control
Authors: Metod Jazbec
Alexander Timans
Tin Hadži Veljković
Kaspar Sakmann
Dan Zhang
Christian A. Naesseth
Eric Nalisnick
Venue: NeurIPS 2024 poster (2024)
Keywords: Early-exit neural networks
risk control
efficiency
robustness
Primary area: safety_in_machine_learning
TL;DR: We leverage risk control to tune the exit threshold of early-exit neural networks, thus equipping early-exit outputs with guarantees.

Abstract: Scaling machine learning models significantly improves their performance.  However, such gains come at the cost of inference being slow and resource-intensive.  Early-exit neural networks (EENNs) offer a promising solution: they accelerate inference by allowing intermediate layers to exit and produce a prediction early.  Yet a fundamental issue with EENNs is how to determine when to exit without severely degrading performance.  In other words, when is it 'safe' for an EENN to go 'fast'?  To address this issue, we investigate how to adapt frameworks of risk control to EENNs. Risk control offers a distribution-free, post-hoc solution that tunes the EENN's exiting mechanism so that exits only occur when the output is of sufficient quality.  We empirically validate our insights on a range of vision and language tasks, demonstrating that risk control can produce substantial computational savings, all the while preserving user-specified performance goals.

Fast yet Safe: Early-Exiting with Risk Control
Metod Jazbec' Alexander Timans 1. Tin Hadži Veljković'
Kaspar Sakmann²
Dan Zhang"
Christian A. Naesseth'
Eric Nalisnick" 1,3
UUA-Bosch Delta Lab, University of Amsterdam
Bosch Center for AI, Robert Bosch GmbH
Johns Hopkins University
Abstract
Scaling machine learning models significantly improves their performance, How- ever, such gains come the cost of inference being slow and resource-intensive. Early-exit neural networks (EENNs) offer promising solution: they accelerate inference by allowing intermediate layers 'exit' and produce a prediction early. Yet a fundamental issue with EENNS is how to determine when to exit without severely degrading performance. In other words, when is it 'safe" for an EENN go 'fast"? To address this issue, we investigate how to adapt frameworks of risk control EENNs. Risk control offers a distribution-fres post-hoc solution that tunes the EENN's exiting mechanism that exits only occur when the output is of sufficient quality. We empirically validate our insights on a range vision and lan- guage tasks, demonstrating that risk control can produce substantial computational savings, all the while preserving user-specified performance goals.
Introduction
As predictive models continue to grow in size, so do the costs of running them at inference time [12, 80). This presents challenge to domains ranging from mobile computing to smart appliances to autonomous vehicles all of which require models that operate on sssource-constraine [62, 50, 47]. Even if computation is not limited by hardware, concerns over the energy usage and carbon footprint of large models motivates their efficient implementation [55, 44]. Additionally, since computational constraints can be dynamic, e.g., due variable loads in web traffic energy demand, is desirable for models to be able to adjust their computational needs to changing conditions [63].
hardware
Early-exit neural networks (EENNs) present simple yet effective approach such dynamic com- putation [70, 36). Leveraging the neural network's compositional nature, EENNS can generate predictions at intermediate layers, thereby 'exiting' the computation 'early' when stop condition met. This early-exit ability has proven useful in settings ranging from vision and language to recommendations ([29), see 4). Yet the flexibility of EENNs does not come for free: predictions generated at early exits are usually inferior to those produced by the full model. In turn, dilemma arises in which the exit condition must balance computational savings with predictive performance.
In this work, we address the EENN'S efficiency ys. performance trade-off via statistical frameworks of risk control (RC) [4. 8). By tuning the EENN's exiting mechanism based on user-specified notion of risk, RC aims to enhance the safety of early-exit outputs. We consider several risks that quantify the difference between the early-exit and full model's outputs, both in terms of prediction quality and uncertainty estimation. Moreover, we study RC frameworks that control the risk with varying degrees with high probability). We demonstrate the effectiveness of
of stringency (i.e.. in expectation this weight, post-hoc solution across a range of tasks, including image classification, semantic segmentation, language modeling, image generation with diffusion, and speculative decoding in large language models. In particular, we make the following contributions:
"qqual contributions. Corresponding authors: <m. wm.jaabeceuva..o nl, a.r. timansduva.nn
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

We formalize EENNs as risk-controlling predictors, ensuring risk control is amenable to the early-exit setting by explicitly linking risk control and early-exit requirements (Prop. 1 & Prop. 2).
We propose risk functions to control early-exit performance both in terms of model predictions and their underlying predictive distributions (Eq. 6 & Eq. 8). Previous work has considered only prediction quality [65], not uncertainty quality (as we do).
We improve upon prior work for language modeling [65], demonstrating that adaptions of risk control allow for less conservative early-exiting and result in larger efficiency gains ($ 3.3, $ 5.3).
We apply, for the first time, risk control early-exiting in image classification, semantic seg- mentation, and image generation, as well as speculative decoding in large language models ($ 5).
2 Background
Data, Let X' x y denote the sample space and assume a data-generating distribution P over it. We consider = -I,,... K} for classification and C IR for regression. Observed samples from P are split into disjoint train, calibration and test sets, denoted Dtraii. Deat, and Drest- We assume samples (x. y) in D cat and Diest to be drawn i.i.d. from P, whereas Dtrein is permitted to be drawn randomly from different distribution (of same support).
Early-Exe Eait Neural Networks. EENNs extend traditional static network models by dynamically adjusting computations during the model's 's forward pass (e.g.. .. the number of evaluated network layers) on the basis of an input sample's complexity or 'difficulty.. More formally, we define an EENN as sequence of prohabilistic classifiers p(y x = æ;; Φ), 01), where 1 1....... I. enumerates the model's exit layers, and Φi and 0 define the model's classification head and backbone parameters at the I-th exit, respectively. The final index L denotes the full model, i.e., all layers are evaluated for a given input sample The obtained predictive distribution p(y in short as P (z). permits to retrieve both predicted class label ÿt = arg maxyey Pi x)) and associated confidence score [0. 1]. which aims to capture model'ss 's certainty about the exit's current prediction. One common choice for classification tasks is the maximum class probability possible, as we explore in 5.2.
σ;; Φi. 0₂) at the (-th exit layer, denoted
é maXyEy (3 x). However, different notions of confidence At test-time, these confidence scores can be leveraged to determine the early-exit model's 's required computations for samples via thresholding. For given test input æ. the EENN exits² at the first layer for which its confidence exceeds a pre-specified threshold value x E [0, 1]. For simplicity, a single threshold value \ X, V/ (1, L 1} is often fixed across exit layers, setup we also consider here. The predictive distribution obtained from the model's 's early-exit mechanism is then given by
(x(y = (y(lx), where min E
i + 0 if = E =l= (1,..., 2). (I)
The threshold parameter À regulates the trade-off between the EENN's accuracy and efficiency gains. Lower values equate larger speed-ups by increasing the likelihood of an early exit (and vice versa), at the cost of generally inferior predictions. Such marginally monotone behavior, where model performance improves on average across exits, is core assumption for the practical use EENNS (see also Fig. 1). We formalize it as
E( (((r((y)))) v)1 2
v)] =I
for some arbitrary loss function f, and elaborate on its connection to risk control in $ 3.3. It is common determine early-exiting criteria by investigating these trade-offs between performance and efficiency on hold-out data, selecting thresholds that ensure the EENN meets a user's computational budget [36, 70] or performance goals [16, 21, 48]. A standard practice is to treat the EENN's predictive confidence (e.g., its softmax scores) as a heuristic for prediction quality. However, this is fallible, EENNS can exhibit fluctuating poorly-calibrated confidences [40, 37, 51], motivating more principled threshold selection.
Such model-driven exiting is distinct from anytime settings, where exits citrromment-drivveer [85].

---

Risk Control. Statistical frameworks for risk control (RC) [1, 4, 8] aim to improve prediction reliability by equipping Hureshold-basee models with safety assurances. Specifically, consider pre-trained prediction model fÅ whose outputs depend on a threshold A. For example, given a classification task, the set predictor fA X + 2" described by Bates et al. [8] includes a class label in the set if its probability exceeds the threshold, i.e. fxx == (y E y p(y] 2 A}. Next, a notion of error for fx is captured by defining a problem-specifer loss function f: E: x 3 + R. For instance, meaningful choice for the set predictor could be the miscoverage loss ((fx()), y) 1/y & fx()),, where [-] is the indicator function. The risk associated with a particular threshold À E is then defined as the expected loss
(3) with A the set of potential threshold candidates. RC frameworks leverage different probabilistic tools which we detail further in $ 3.3 to determine subset Å S A for which the risk in Eq. is guaranteed to be small. Note that Å is retrieved in a post-hoc manner by leveraging the calibration set
R(A) y) [e(fA(z))
Deal sampled i.i.d from P. Thus, R²)) is a random quantity dependent on Deal for any Å E A.
Given such a risk, desired safety assurances may vary in strength. For a tolerated risk level E (0, 1). risk control in expectation seeks to guarantee that
ED [R())] Sc V
(4)
where the outer expectation is taken over randomly drawn calibration data of finite size |Deatl = n. A stronger statement on risk control with high probability requires additionally specifying a probability level 8 E (0, 1), and aims to ensure that
De (R()) 20 V.
(5)
That is, rather than the average control over calibration data in Eq. 4, risk control according to Eq. 5 holds with high probability for any particular sampled set Deat- In both cases, we may refer to fx for any À E Å as a controlling predictor. The risk level and probability level 8 are user-specified parameters dictating how tightly the risk is controlled, and a particular choice has to consider the setting and loss É. For example, reasonable choice for the stated miscoverage loss may be (c, 5) = (0.05, 0.1). Observing Å = 9 implies that there is no risk-controllinng predictor for the selected (c, 6), indicating overly stringent risk control conditions which fx cannot satisfy.
The prediction guarantees obtained via RC are highly practical, since they are (i) distribution-free. i.e. they do not impose any particular assumptions on the generating distribution P. (ii) are post-hoc applicable to any arbitrary choice of underlying predictor fx. and (iii) hold in finite samples, thus not relying on asymptotic limit statements. Indeed, experimentally find that the provided assurances hold even for remarkably small calibration sets (n ss 100, see $ 5).
3 Safe Early-Exiting via Risk Control
We now detail our approach for early-exiting with safety guarantees based on risk control. We begin by outlining EENNs as risk-controlling predictors ($ 3.1). Next, we describe two general types of risk to measure performance drops resulting from early-exiting, Importantly, these risks can be employed to assess the quality of both predictions and predictive distributions ($ 3.2). Finally, we motivate and formalise how different risk control frameworks can be adapted to the early-exit setting ($ 3.3).
3.1 EENNS as Cosk-Controlling Predictors
As mentioned in $ 2, risk control requires a predictor fx whose outputs depend on a threshold A A. The EENN's confidence-based thresholding behaviour (following Eq. 1) lends itself naturally to such a formulation. For a particular exit threshold X E [0. 1]. the EENN px xx) will act as such threshold predictor. To ensure that the EENN satisfies the user's control requirements, the risk-controlling threshold set Á needs to be identified. Importantly, this can be done post-hoc using pre-trained EENN with fixed weights, since only the exit threshold X is modified. In order to maximize computational savings while ensuring that the user-defined risk is managed, we select Å = min A, since a low threshold encourages earlier exiting. If À is empty, we default to ā 1, the equivalent of relying strictly on the full model output PL (z)
3

---

3.2 Early-Exiting Risks
We next detail two types of risk which can be employed to guard against performance drops due to early-exiting. Similarly to Schuster et al. [65], our risks are defined in terms of relative exit performance, permitting their calculation for both labelled and unlabelled calibration data. Moving beyond their setting, we suggest these risks for controlling the quality of both model predictions and predictive distributions, from which confidence scores can be derived.
When calibration labels are present, these can be used to measure the early-exit performance through supervised losses. Let (x) denote general EENN output for some input æ. It takes the form ÿt for predictions and Pr (y)=)) for the underlying predictive distribution. The supervised performance gap risk then defined as
Performance Gap Risk.
(6) where Ōx (x) and (x)x) refer to early-exit and full model outputs, respectively. The choice of loss function l is task-specific, and we outline relevant choices in $ 5, such as the 0-1 loss for image classification. For predictive distribution control, we suggest leveraging squared distributional loss which, when averaged across samples, recovers the Brier score [13]. Specifically, we define such a 'Brier loss' for classification tasks as
RC) == E -)p (( ((()) - (())(x).
(n(Di(ylz). = Σ (pi(k)z) 1ly k:)
where pr (k(()) denotes the predicted probability of a particular class k; and 1/y k:] its one-hot encoded label. The Brier score is a strictly proper scoring rule [26, 27], ensuring its suitability to assess probabilistic forecasts. Moreover, its mathematical formulation lends itself favorably to risk control when compared to other widely used probabilistic metrics. We defer further details to $ A.3. Addressing risk control of the underlying predictive distribution Pr: (y| (z)) is compelling extension, as confidence or uncertainty estimates are typically derived from it. Particularly in safety-critical scenarios where reliable uncertainties are essential [32, 9], such control can thus prove very useful.
Consistency Risk. In the case of unlabelled calibration data, an unsupervised version of Eq. can be obtained by replacing the ground truth labels y with labels ÿL obtained from the full model. We define the unsupervised consistency risk as
RC (A) == Eep (O)( 9L) - ((01()), 91)],
(8)
where only input samples a ~ P are required for its evaluation.
For regular prediction losses, Eq. 8 collapses to evaluating the per-sample loss {()x, ý1), since EØ1., ( = 0. For predictive distribu- tion control, the loss difference remains, and we substitute y Eq. by sampling a label 9r. (x(x)) from the EENN's final layer. Our reliance on the last layer's output is mo- tivated by the EENN's marginal monotonicity (Eq. 2 and Fig. 1). Finally, we note that both the
ImageNet
Accuracy (t)
0.80
0.70
MSDNet
L2W-DEN
DVVT
Dyn=Pew
Brier (L)
0.50
0.40
performance gap risk (x) and consistency risk (A) are quite agnostic to the BENN'S actual predictive performance. In both cases, the risk formulation aims to ensure prediction consistency via the relative performance gap be- tween exits, as opposed to absolute performance with respect to observed ground truth labels.
2
Early Exit
Figure I: Accuracy and Brier score [13] across exits for different EENNS for image classification on ImageNet ($ 5.1). Marginally monotone perfor- mance trends (Eq. 2) are generally observed across models, with last-layer exits performing best.
3.3 Risk Control Frameworks
After having defined our early-exit risks, we next outline how desired cisk-controlling exit threshold Á can be computed based on calibration data. We begin by considering a 'naive' empirical approach, followed by risk control in expectation (Prop. 1) and with high probability (Prop. 2).

---

Empirical Approach. For tolerated risk level e E (0, 1), 'naive' empirical threshold can be selected by picking the smallest threshold [0. 1] from the candidate set A for which the risk on the calibration set Dent is controlled, i..,
Kamp == min E E A R(A; Deat) Se..
Note that (eat) - (ox(2))) yi) the empirical calibration risk, an approximation of the true risk in Eq. computed on Dent (and likewise R(A; Drest) denotes the empirical test risk). For the risks introduced in $ 3.2 the threshold demp is always well-defined, since RR(A 1) is zero
Risk Control in Expectation. The threshold Remp is a straight- choice, but can fail to control the risk on test data if the approximation quality of ((A) by R(A Deat) is poor, e.g., due to badly drawn calibration data. Perhaps surprisingly, only a slight modification of Eq. 9 is required to ensure risk control in expectation. Specifically, for a bounded loss function & & B where B > 0, and assuming a monotone risk (A), the threshold*
72 R(A; cat) ACRC == min { AAEA n+1
B n+1 Scj
(10)
guarantees Eq. 4, thus shielding against bad sample draws on average. It is easy to show that Aapp lim ÅcRC, and since our losses designed to be upper-bounded by B E (1, the two thresholds already coincide for small calibration sets (n se 100). We formalize risk control in expectation in the following proposition for early-exit setting:
B] be a right-continuous bounded loss, and assume a marginally monotone EENN (Eq. 2). Then the exit threshold doRC ensures risk control in expectation, i.e., it holds that E Pp [R((ARR)] Se for any E (0, 1).
Proposition 1. Let E A + (-o0,
Our proposition is an extension of Conformal Risk Control [4] (CRC) to the early-exit setting, and a proof can be found in $ A.1. Our main technical insight is that risk control can be relaxed to assume monotone risks, rather than monotone losses as in the original formulation [4]. This relaxation is crucial for the early-exit setting, since we can relate monotone risks to assumptions of marginal monotonicity on the EENN (see Lemma 1). In contrast, monotone losses translate to assuming conditional monotonicity, a much stronger requirement suggesting the EENN's performance improves across exits per sample, and which has been shown to be violated practice [40, 76, 37].
Risk control with High Probability. A stronger guarantee can be obtained by ensuring risk control with high probability for any drawn calibration set. We employ the Upper Confidence Bound (UCB) from Bates al. [8] for this purpose. First, an empirical upper bound R. (A; Deat) is derived to bound the risk R() with high probability. That is, for a probability level å E (0, 1) it holds that
P (R(A) < RR (A; 2 1 11- A.
(11)
An exit threshold ensuring risk control according to Eq. 5 is then selected as
λUcR == min(X E A R (X;Doa)) < VX 2 A).
(12)
Similarly to Prop. I, we can now formalize risk control with high probability for the early-exit setting: Proposition 2. Let é: A + [-B, B] be a bounded loss, and assume a marginally monotone EENN (Eq. 2). Then the exit threshold λuen ensures risk control with high probability, i.e., holds that PD (P(us-x"((AUcc)) S <) 2 8 for any (c, E 10,
This reformulation of the main theorem from Bates et al. [8] (Thm. A.1) is proven in $ A. and an algorithmic description is given in Appendix B. We employ their suggested Saalby--mmth-Raaddae bound [75] (WSR) to compute R- (A; Deaat), but relax the bounding requirements on the loss from é E [0, 1] to É E [-B, B) for B > This change has important implications for the early-exit setting, since it admits "rewarding" the EENN when an earlier exit performs better than the final exit for some samples, phenomenon known as overthinking [40, 37]. In practice, this results in a better risk estimate and less conservative carly-exiting, See $ A.2 for more details on the effect of loss bounds.
In Eq. 10 and Eq. 12, we default to Á = if Á 0, implying that risk control does not permit exily-exiting.
5

---

Learn-then-Tee Test and CALM [65]. Learn- tesn [1] (LTT) is another framework for high-probability risk control, where threshold se- lection is framed as a multiple hypothesis testing problem. In contrast to UCB (Prop. 2), LTT does not require risk monotonicity, and can thus also be employed when the EENN is suspected to vio- late marginally monotone behaviour. LTT in the early-exit setting has been employed by Schus- ter et al. [65] (CALM), presumably motivated by the avoidance of this assumption. However, ex- pecting an EENN to marginally improve across exits is core requirement which implicitly un- derlies any practical implementation. Since this assumption is usually also empirically satisfied (Fig. 1), there is no obvious reason to explicitly avoid it. Furthermore, correcting for multiple testing in LTT via fixed sequence testing - as is done for CALM will only yield practical sav- ings if monotonicity is satisfied. We stress these observations since we find that UCB provides greater computational savings than LTT under the same guarantees (Fig. 2 and 5.3), including
CALM (15-large), CNN/ DM, Necd 100
Test Risk 72
0.10
0.05
0.00
Exii Layer (1)
15
LTT
UCB
10
CRC
0.02
0.04
0.06
0.08
0.10
0.12
0.14
Risk Level e
Figure 2: Empirical test risk (top) and efficiency gains (bottom) for the CALM model [65] for text summarization on CNN/DM. Our adaptation of UCB [8] (Prop. 2) outperforms the LTT [1] ap- proach in CALM by yielding larger efficiency gains under the same risk control assurances (see & 5.3 for details). Shading denotes the standard deviation across S = 100 calibration/test splits.
for small-sample regimes (n z 100), which are of high practical interest and were not explored by Schuster et al. [65]. Moreover, due LTT's reliance on the Hoeffding-Bemne account for instances of model overthinking (see A.2). Thus, unless Eq. 2 is known to be violated, we recommend UCB over LTT in the early-exit setting.
bound [11], it cannot
Related Work
Early-Exiting [70, 29] dynamic approach to accelerate model inference is both orthogonal and complementary to static model compression techniques such as pruning, quantization, and [6. 79, 68, 84, 25]. Its wide-ranging applicability has been demonstrated across numerous vision [36, 48, 15, 69. 22] and language tasks [21, 83, 65, 78, 5, 53]. While most prior work has focused on the trade-off between performance quality and computational savings, the safety of early-exit models has received less attention to date [64, 65, 51, 38). Risk Control has gained traction due to interest in efficient, post-hoc approaches with safety assurances for large models. Most related, conformal prediction [66] has been popularized as an effective method for uncertainty quantification with guarantees on the miscoverage risk [24, 3]. Recently, multiple proposals address the control of more general risk notions [4, 8, 1, 67, 54, 45J, with explored applications ranging from imaging [71, 2, 81, 23, 43, 61, 10, 72] language [86, 20, 45, 56] and beyond [39, 46). Most closely related to our work is research by Schuster et al. [65]. who first employ risk control for safe early-exiting in language modeling. We move beyond their setting by (i) controlling the quality of both model predictions and uncertainty estimates ($ 3.2), (ii) obtaining better efficiency gains through careful selection of our risk control framework ($ 3.3), and (iii) extending early-exit risk control to novel tasks ($ 5). Ringel et al. [60]'s work is also related: they apply risk control to exit early for time series prediction task. Yet their emphasis is on exiting from stream of input features, whereas we exit from the model itself (i.e., a stream of model layers).
5 Experiments
We empirically validate early-exiting via risk control on a suite of different tasks: image classification (5 5.1), semantic segmentation ($ 5.2), language modelling ($ 5.3), image generation with diffusion (S 5.4), and speculative decoding ($ 5.5). Our code is publicly available at :/tgi thub. com/ metodj/RC-Es EENN. We begin by outlining our general risk control design and evaluation metrics.
Risk control design.
We target control of the performance gap and consistency risks defined in $ 3.2. For predictions ÿ we employ target-specific losses, and, when applicable, for predictive

---

(0)10--1)
RP) (Brier)
(9
R (p) (Brier)
0.3
Test Risk
0.2
0.1
MSDNet
CRC
Exit Layer
DWI
UCB
L2W-DEN
Dyn-Pere
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
Risk Level
Figure 3: Empirical test risk (top) and efficiency gains (bottom) for different early-exit models. risks ($ 3.2) and risk levels e on ImageNet (for calibration set size n 100). In line with theoretical results, the test risk is controlled across models, risk types, and levels. Despite guaranteeing control expectation (CRC, Prop. 1) or with high probability (UCB, Prop. 2), obtained gains are substantial.
distributions Pr (y (ae) our Brier score formulation. We denote these four risks in short as R (g). (p). RC ( and RC (f).. Risk control requirements of different strength are assessed by varying the risk level e. Note that our approach is entirely post-hoc, and our experiments leverage existing pretrained EENNS when possible. Thus the underlying models typically not modified, and omit the commonly seen performance vs. FLOP curves to instead appropriately benchmark against different risk levels. For risk control with high probability, we set ò 0.1 (i.e., % probability). Reported numbers are averaged across multiple trials of calibration and test splitting (S = 100) to account for sampling effects. Additional results across experiments can be found Appendix
Evaluation metrics. We evaluate results based obtained test risks and efficiency gains. We assess whether the guarantees stated in $ 3.3 are satisfied by checking if the empirical test risk for given cist-contro threshold is controlled, i.e., R(,; Drest) S holds. Ideally, the measured test prevent overly conservative early-exiting. We measure
risk should also approach from below, efficiency gains by reporting the average exil layer across test samples, or its relative improvement last-layer exiting (in %.. Similar gains terms of arithmetic operations (FLOPS) are reported in Appendix D. We favour approaches which, while controlling the test risk, exit as early as possible.
5.1 Image Classification
For image classification, we focus on the ImageNet dataset [19]. We employ four state-of-the- art EENNS to demonstrate that our findings generalize across different models and architectures: for details). We employ standard 0-1 loss for predictions, and the Brier loss formulation from Eq. 7 for predictive distributions. Fig. 3 displays results for the sample calibration regime (n = 100). In line with our thcoretical guarantees, the test risk remains controlled across all models, risk types, and risk levels (top row). The steeply decreasing efficiency curves affirm that even under strict control requirements, substantial efficiency gains be obtained (bottom row). For example, controlling the prediction gap risk at strict 5% (RC(1) (ÿ) for c = 0.05) results in model average ~ 61% less layers evaluated for control in expectation (CRC, Prop. 1), and 46% for control with high probability (UCB, Prop. 2), Table 2. Naturally, UCB produces more cautious early-exiting due to its stronger safety assurance, but these differences decrease for larger calibration sets (see D.I for 1000). This highlights the practical benefit of allocating more calibration samples: a larger sample size can aid to mitigate the price paid by high--rohhee probbbbrbbbblltyy guarantee terms of obtained inference speed-ups.
MSDNet [36], DViT [74], I.2W-DEN [30], and Dyn-Pere [31] (see
5.2 Semantic Segmentation
For this task, we explore the effect of different confidence measures used in Eq. on realizable speed-ups. We use the EENN with four exits proposed by Liu al. [48] (ADP-C, see $ C.2). ADP-C permits pixel-level carly-cxiting with per- per-pixel confidence scores. Since desire to carly-exit the
7

---

Table I: Efficiency gains for semantic segmentation with risk control via UCB (Prop. 2) on Cityscapes.
We evaluate for different risks ($ 3.2), confidence measures ($ 5.2) and risk levels e. Displayed values denote relative improvement over last-layer exiting in terms of mean exit layer (in %).
Risk
(r) (mloU)
(t) (Brier)
( (Miscov.)
R (0) (Brier)
Level
0.01
0.05
0.1
0.01
0.05
0.1
0.01
0.05
0.1
0.01
0.05
0.1
Mean
Top-1
6.3
33.7
53.5
0.0
13.6
43.4
6.3
39.2
61.8
0.0
39.3
58.4
Top-Diff'
9.3
35,5
54.4
0.0
17.5
44.3
6.3
39.9
62.4
0.0
38.6
57.9
Entropy
5.2
36.0
54.3
0.0
17.9
41.0
5.1
40.4
61.3
0.0
40.1
58.3
Patch
Top-1
10.0
35.7
53.3
18.4
45.3
8.8
39.1
61.5
0.0
38.0
58.3
Top-Diff
10.0
35.2
53.4
0.0
19.4
45.9
8.8
40.5
62.2
0.0
38.4
58.8
Entropy
9.1
34.8
53.5
0.0
18.0
45.8
8.1
38.9
61.5
0.0
37.3
57.1
Groun Truth
First layer
Last (1 4)
0.20
Brier A/s
Early exit
0.15
Conficeence
0.30
0.08
Late exit
0.00
Exit
Figure 4: Right: Example of our method's early-exiting Cityscapes [17]. For two samples that exit early (1 = 1) and exit late (I = 4), we display ground truth segmentation masks and confidence maps the first and last model layer. Left: For every sample, we compute the Brier loss difference A (p) (y (z), y) - e (f4(y)aa [ y) between first and last model layer (Eq. 7), and stratify values across respective exit layers; the red dot denotes the mean. For both figures, we consider the simplest combination of Top-1 confidence score and mean image-level aggregation (for 0.08).
entire image instead, we explore image-level aggregations alongside different confidence scores, which are briefly outlined below. As task-specific prediction losses, we consider the commonly used mean intersection-ovi civerioonooee-iiioo (mloU) and miscoverage for the labelled and unlabelled cases, respectively. For predictive distribution control, we employ pixel-averaged versions of the Brier loss in Eq. 7 (see $ A.3). We evaluate our approaches on Cityscapes validation data (80% Dent. 20% Drest); ; in addition, we finetune and evaluate ADP-C on subset of the GTA5 dataset [59] in $ D.2.
We consider three pixel-level confidence scores: the top class softmax probability (Top-1), the difference between top two class probabilities (Top-Diff), and the normalized entropy over pixel's predictive distribution (Entropy). In addition, we consider three image-level aggregation strategies: the image's 's average pixel confidence (Mean), its 0.25-th quantile (Quantile), and a patch-base approach (Patch), wherein sliding window of fixed size (e.g., 50 x pixels) computes the mean confidence over pixels per patch, and the min over such patch scores is retrieved. These aggregations consider both varying levels of prudence (Mean vs. Quantile) and granularity (Mean vs. Patch).
Table displays obtained efficiency gains for risk control via UCB (Prop. 2) across different risk levels (0.01, 0.05, 0.1). In line with Fig. 3, increased speed-ups observed as the risk requirements are relaxed (i.e., increases). Notably, for a given 6 the gains for Brier-based risks tend to be smaller than for prediction risks, affirming more challenging risk control. The differences between combinations of per-pixel confidence and image-level aggregation are most pronounced for small e, where Patch records highest gains while Quantile is more conservative (see D.2 for full results). In Fig. 4, we display qualitative example of the model's 's exiting behaviour. For sample which exits at the first layer (top row), the EENN's confidence map remains fairly stable across subsequent layers, suggesting an accurate model assessment has been reached early on. In contrast, sample which exits at the final layer (bottom row) will see a substantial improvement model certainty, justifying additional computations. Such behaviour is also visible when stratifying all samples across their respective model exits (Fig. 4, left). For samples which exit later, the difference between distributional losses at the first and final layer increases, affirming that compute is spent meaningfully.

---

Test Risk π
0.0
13
Exit (L)
10
CRC
UCB
0.00
0.10
"Full model" <
0.01
c 0.05
c 0.10
0.15
Risk Level e
Figure 5: Results for early-exit diffusion with DeeDiff [69] on CelebA [49]. Left: The quality of generated images is directly related the target risk control level Right: Empirical test risks controlled for both CRC (Prop. 1) and UCB (Prop. 2) (for calibration set size n 500).
5.3 Language Modeling
For this task, we replicate the main experiments from Schuster et al. [65] (CALM), using their early- exit version of the T5 model [57] for text summarization on CNN/DM [33] and question answering on SQuAD [58]. Recall that CALM makes use of the Searn-then-Tess [1] (LTT) framework for early-exit prediction control, whereas we suggest the Upper Confidence Bound [8] (UCB). In contrast their experiments which involve excessively large calibration sets (n = 8000), we explore more practical settings of low calibration sample counts with n E {100, 1000). Our results for the performance gap risk (Eq. 6) based on task-specific losses (ROUGE-L for CNN/DM and Token-F! for SQuAD) are displayed Fig. and $ D.3. all cases, UCB exit thresholds provide larger computational savings over LTT, while ensuring the same risk control with high probability (Eq. 5). Since particularly pronounced for n = 100, these results highlight the need for careful framework selection in order to minimize the cost of providing guarantees. Once more, risk control in expectation (CRC, Prop. 1) permits faster exiting due to its weaker safety assurance. Encouragingly, even with as few as n 100 calibration samples, CRC cxit thresholds reach near-optimal exiting, indicated by their proximity to the ideal (diagonal) risk line. This suggests that even for modern language tasks, equipping an EENN with notions of safety does not necessitate strong compromise inference efficiency.
5.4 Image Generation with Early-Exit Diffusion
To demonstrate the wide-ranging applicability of our proposal, we lastly consider carly-exiting for image generation with diffusion. We employ the DeeDiff model [69], which performs early-exiting on the denoising network each sampling step during the reverse diffusion process We target control of the perceptual difference between images generated by the accelerated and full diffusion processes, which we measure with the LPIPS score [82], and where lower values indicate perceptually closer images. Our results on the CelebA dataset [49] are shown in Fig. 5 for both risk control via CRC (Prop. and UCB (Prop. 2), asserting that the risk is controlled all levels e. The impact of the risk level on image generation is additionally visualized for two examples. For strict control requirements the early-exit generations perceptually resemble the full model, whereas generated image quality visibly deteriorates for larger e (but remains controlled). For smaller e, the speed-up within each sampling step are relatively modest (e.g., 155% for e 0.05). However, such gains accrue over the large number of sampling steps in the image generation process (~500), resulting in overall meaningful savings. Similar observations for CIFAR [42] are reported D.4.
5.5 Speculative Decoding for Large Language Models
While we primarily focus on early-exiting, our final experiment highlights how risk con- trol can also be applied to other techniques for efficient inference. Here we con- sider accelerating the inference of large language models (LLMss) using the (soft) specu- lative decoding approach BiLD [41]. BiLD uses small draft model to generate mul- tiple tokens autoregressively while the original LLM is only employed for verification.
Since the code for DeeDiff not publicly available, implement it ourselves (see $ C.4).

---

This step can be performed in a single forward pass for all proposed tokens, necessitating less computations from the larger, more expensive model. During verification the difference in to- ken distributions between the models is com- puted, and the tokens generated by the draft model are rejected if the difference exceeds tolerated threshold, triggering a "rollback" (see Eq. 3 in [41]). We apply our risk control frame- works to this rollback threshold, which dictates (i) the similarity of generated text to the output produced solely by the original LLM,, and (ii) the associated inference speed-ups in terms of sentences (or samples) per second. Our results in Fig. 6 for the performance gap risk (Eq. 6), as defined via the difference in sentence-level BLEU scores, corroborate our previous findings for language modeling (Fig. 2 and $ 5.3). That is, our approaches via CRC (Prop. 1) and UCB (Prop. 2) provide meaningful speed-ups and im- prove upon the LTT method [1] used by CALM [65], while maintaining the desired risk control across test samples.
BiLD (Soft Speculative Decoding)
0,010
Test Risk π
0.005
0.000
Samples /Se (1)
3.6
CRC
UCB
LTT
3.4
0.000
0/102
11004
00066
0
0.m0
Risk Levele
Figure 6: Empirical test risk (top) and efficiency gains (bottom) for the BiLD model [41] for ma- chine translation task (De-En) on IWSLT [14] (with
500). We fix the fallback threshold to 0.5, and apply risk control to the rollback thresh- old. Our adaptation of UCB again outperforms LTT by yielding larger efficiency gains under the same guarantees. Shading denotes the standard deviation across S = 100 calibration/test splits.
6 Discussion
Our work addresses how to select a 'safe' exiting mechanism for early-exit neural networks (EENNs). We propose balancing the EENN's efficiency vs. performance trade-off via risk control, ensuring that accelerated inference does not compromise the quality of the early-exit outputs. We validate our light- post-hoc solution on a variety of tasks and improve upon prior work [65] (see $ 4).
single shared exit threshold among layers (Eq. 1). While using a shared exit threshold is common [69, 48, 40, 76, 83, 77], relaxing this condition could to further efficiency gains. This, however, introduces new challenges both in terms of theory (e.g.. defining monotonicity requirements) and practice (e.g., substantially larger search spaces). Overcoming them by adopting recently proposed risk control techniques for nigh-dimensional thresholds [60, 71] could prove promising for future extensions. Another problem back to a single threshold by use
Limitations and Future Work. A key limitation of our work is the reliance
(simple) workaround is to reduce the of a threshold function, as partially explored in [65]. Instead of working directly with multiple thresholds, our risk control framework can then be applied to this scalar parameter. Additionally, multiple risk control extensions provide natural avenues for future work. Firstly, risk control as used in our work achieved only marginally across observations (Eq. 3), and one could aspire for more granular exil-conditional control [60]. Secondly, the employed risk control frameworks define risk in terms of the expected loss. One could instead aim to control the tails of the loss, e.g., via specific quantiles of interest [67]. Lastly, relaxing the i.i.d assumption on calibration and test data could help extend cist-contoolling EENNs to scenarios with test-time distribution shifts [86] or to online updating strategies [23].
Broader Impacts. EENNS provide a simple and effective approach to dynamic computation [29]. Their computational savings can reduce energy costs and the carbon footprint, as well as allow the model to be deployed on resource-constraing hardware. By incorporating a 'safe' exit mechanism into these models, we improve their trustworthiness and strengthen the reliability of their intermediate outputs, along with any decisions based on them. This facilitates safer model deployment in real- world applications and contributes to more responsible decision-maker making.. While we do not foresee any direct negative consequences from our work, improper use of our risk control framework can lead to violations or misinterpretations of its provided guarantees. This, in turn, can risk instilling a false sense of security. Overall, we believe that our work outlines an effective approach to improve the reliability of EENNS and to safely balance their inherent efficiency vs. performance trade-off. In doing it contributes to the goal of developing models that are ultimately fast yet safe.
10

---

Acknowledgments and Disclosure of Funding
We thank Mona Schirmer, Rajeev Verma, Christoph-Nikolas Strachle, Patrick Forré and Stephen Bates for helpful discussions and clarifications. We are also grateful to the anonymous reviewers who helped improve the work with their constructive feedback. This project was generously supported by the Bosch Center for Artificial Intelligence. Eric Nalisnick did not utilize resources from Johns Hopkins University for this project.
References
[1] Anastasios N. Angelopoulos, Stephen Bates, Emmanuel Candès, Michael I. Jordan, and Lihua Lei. Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control. arXiv
Preprint (arXiv:111000005), 2021. 3, 6, 9, 10, 11
[2] Anastasios N. Angelopoulos, Amit Pal Kohli, Stephen Bates, Michael Jordan, Jitendra Malik, Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano. Amage-to-Imane Regression with Distribution-Fres Uncertainty Quantification and Applications Imaging. International Conference Muchine Leurning, 2022. 6
[3] Anastasios N Angelopoulos, Stephen Bates, et al. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 2023. 6
[4] Anastasios N Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Con- formal risk control. International Conference Learning Representations, 2024. 3,
2
[5] Sangmin Bae, Jongwoo Ko, Hwanjun Song, and Se- Young Yun. Fast and robust carly-exiting framework for autoregressive language models with synchronized parallel decoding. Conference on Empirical Methods in Natural Language Processing, 2023. 6,
[6] Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Mengdan Zhu, Yifei Zhang, al. Beyond efficiency: systematic survey of resource-efficient large language models. arXiv Preprint (arXiv:2401.0066
2024. 6
[7] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. Conference on Computer Vision and Pattern Recognition, 2023.
[8] Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution-free, risk-controllingg prediction sets. Journal of the ACM, 2021. 1, 3, 5, 6, 9, 2, 11
[9] Edmon Begoli, Tanmoy Bhattacharya, and Dimitri Kusnezov. The need for uncertainty quan- tification in machine-assisted medical decision making. Nature Machine Intelligence, 2019.
4
[10] Omer Belhasin, Yaniv Romano, Daniel Freedman, Ehud Rivlin, and Michael Elad. Principal uncertainty quantification with spatial correlation for image restoration problems. Transactions on Pattern Analysis and Machine Intelligence, 2023. 6
[11] Vidmantas Bentkus. On hoeffding's incqualities. Annals Probability, 2004. 6,
[12] Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. International Conference on Machine Learning, 2017.
[13] Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 1950. 3
[14] Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Nichues. Sebastian Stüker, Katsuitho Sudoh, Koichiro Yoshino, and Christian Federmann. Overview of the iwslt 2017 evaluation campaign. International Workshop on Spoken Language Translation, 2017.
[15] Joud Chataoui, Mark Coates, et al. fointly-learned exit and inference for a dynamic neural network. International Conference on Learning Representations, 2023. 6

---

[16] Xinshi Chen, Hanjum Dai, Yu Li, Xin Gao, and Le Song. Learning to stop while learning to predict. International Conference on Machine Learning. 2020. 2
[17] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. Conference on Computer Vision and Pattern Recognition, 2016. 8, 6
[18] A Philip Dawid. Probability forecasting. Encyclopedia of Statistical Sciences, 2004. 3
[19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scand scale hierarchical image database. Conference Computer Vision and Pattern Recognition, 2009. 7
[20] Zhun Deng, Thomas Zollo, Jake Snell, Toniann Pitassi, and Richard Zemel. Distribution- free statistical dispersion control for societal applications. Advances in Neural Information Processing Systems, 2024. 6
[21] Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. International Conference on Learning Representations, 2020. 2,
[22] Zhengcong Fei, Xu Yan, Shuhui Wang, and Qi Tian. Deccap: Dynamic early exiting for efficient image captioning. Conference on Computer Vision and Pattern Recognition, 2022. 6
[23] Shai Feldman, Liran Ringel, Stephen Bates, and Yaniv Romano. Achieving risk control in
online learning settings. Transactions on Machine Learning Research, 2023. 10
[24] Matteo Fontana, Gianluca Zeni, and Simone Vantini. Conformal prediction: unified review of theory and new challenges. Bernoulli, 2023, 6
[25] Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. survey of quantization methods for efficient neural network inference. Low-Power Computer Vision, 2022. 6
[26] Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 2007. 4,
[27] Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2007.
4. 3
(28) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. International Conference on Machine Learning, 2017. 3
[29] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey, Transactions on Pattern Analysis and Machine Intelligence, 2021.
1, 6, 10
[30] Yizeng Han, Yifan Pu, Zihang Lai, Chaofei Wang, Shiji Song, Junfeng Cao, Wenhui Huang, Chao Deng, and Gao Huang. Learning to weight samples for dynamic early-exiting networks. European Conference on Computer Vision, 2022. 7, 6
[31] Yizeng Han, Dongchen Han, Zeyu Liu, Yulin Wang, Xuran Pan, Yifan Pu, Chao Deng, Jun- lan Feng, Shiji Song, and Gao Huang. Dynamic perceiver for efficient visual recognition. International Conference on Computer Vision, 2023.
[32] Geoffrey Heal and Antony Millner. Reflections: Uncertainty and decision making in climate change economics. Review of Environmental Economics and Policy, 2014. 4
[33] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espcholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in Neural Information Processing Systems, 2015.
[34] Hans Hersbach. Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather and Forecasting, 2000. 3
12

---

[35] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020.
[36] Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Wein- berger. Multi-scale dense networks for resource efficient image classification. International Conference on Learning Representations, 2018. 1, 2, 6,
[37] Metod Jazbec, James Allingham, Dan Zhang, and Eric Nalisnick. Towards anytime classifi- cation in early-exit architectures by enforcing conditional monotonicity, Advances in Neural Information Processing Systems, 2024. 2, 3
[38] Metod Jazbec, Patrick Forré, Stephan Mandt, Dan Zhang, and Eric Nalisnick. Early-exit neural networks with nested prediction sets. Conference on Uncertainty Artificial Intelligence, 2024.
[39] Ying Jin, Zhimei Ren, and Emmanuel J Candès. Sensitivity analysis of individual treatment effects: A robust conformal inference approach. Proceedings of the National Academy of Sciences, 2023. 6
[40] Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. Shallow-deep networks: Understanding and mitigating network overthinking. International Conference on Machine Learning, 2019. 2, 5, 10, 3
[41] Schoon Kim, Karttikeya Mangalam. Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. Advances in Neural Information Processing Systems, 2024. 9, 10
[42] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Toronto, ON, Canada, 2009. 9, 12
[43] Gilad Kutiel, Regev Cohen, Michael Elad, and Daniel Freedman. What's behind the mask: Estimating uncertainty in soage-to-imaag problems. arXiy Preprint (arX1v:2211.12211), 2022.
[44] Loie Lannelongue, Jason Grealey, and Michael Inouye. Green algorithms: quantifying the carbon footprint of computation. Advanced Science, 2021. I
[45] Bracha I controlling multiple risks with pareto testing. International Conference on Learning Represen- tations, 2022. 6
Adam Fisch, Regina Barzilay, and Tommi S Jaakkola. Efficiently
[46] Bracha Genffer-Goldsheinn. Adam Fisch, Regina Barzilay, and Tommi Jaakkola. Risk-controlling model selection via guided bayesian optimization. arXiv Preprint (arXiv:2312.01692, 2023.
[47] Shiya Liu, Dong Sam Ha, Fangyang Shen, and Yang Yi. Efficient neural networks for edge devices. Computers & Electrical Engineering, 2021.
[48] Zhuang Liu, Zhiqiu Xu, Hung-Ju Wang, Trevor Darrell, and Evan Shelhamer. Anytime dense prediction with confidence adaptivity. International Conference on Learning Representations, 2021. 2. 6, 7, 10
[49] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. International Conference on Computer Vision, 2015. 9
[50] Arnab Neelim Mazumder, Jian Meng, Hasib-Al Rashid, Utteja Kallakuri, Xin Zhang, Jae-Sun Seo, and Tinoosh Mohsenin. A survey on the optimization of neural network accelerators for miero-ai on-device inference. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2021.
[51] Lassi Meronen, Martin Trapp, Andrea Pilzer, Le Yang, and Arno Solin. Fixing overconfidence in dynamic neural networks. Winter Conference on Applications of Computer Vision, 2024. 2,
[52] Allan H Murphy, new vector partition of the probability score. Journal of Applied Meteorology and Climatoloyy, 1973. 3
13

---

[53] Xuchen Pan, Yanxi Chen, Yaliang Li, Bolin Ding, and Jingren Zhou. Ee-tuning: An eco- nomical yet scalable solution for tuning early-exit large language models. arXiv Preprint (arXiv:2402.00518) 2024. 6
[54] Sangdon Park, Osbert Bastani, Nikolai Matni, and Insup Lee. Pac confidence sets for deep neural networks via calibrated prediction. International Conference on Learning Representations, 2020.
[55] David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv Preprint ww.Xi::2004.03330 2021.
[56] Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S Jaakkola, and Regina Barzilay. Conformal language modeling. International Conference Learning Representations, 2023. 6
[57] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020.
[58] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. Conference Empirical Methods in Natural Language Processing, 2016. 9
[59] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. European Conference on Computer Vision, 2016. 8, 6
[60] Liran Ringel, Regev Cohen, Daniel Freedman, Michael Elad, and Yaniv Romano. Early time classification with accumulated accuracy gap control. International Conference on Machine Learning, 2024. 6, 10
[61] Swami Sankaranarayanan, Anastasios Nikolas Angelopoulos, Stephen Bates, Yaniv Romano, and Phillip Isola. Semantic uncertainty intervals for disentangled latent spaces. Advances in Neural Information Processing Systems, 2022. 6
[62] Kavya Saravanan and Abbas Z Kouzani. Advancements in on-device deep neural networks. Information, 2023.
[63] Simone Scardapane, Michele Scarpiniti, Enzo Baccarelli, and Aurelio Uncini. Why should we add early exits to neural networks? Cognitive Computation, 2020.
[64] Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina Barzilay. Consistent accelerated inference via confident adaptive transformers. Conference on Empirical Methods in Natural Language Processing, 2021. 6
[65] Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 2022. 2, 4, 6, 9, 10, 3, 7.
[66] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 2008. 6
[67] Jake Snell, Thomas Zollo, Zhun Deng, Toniann Pitassi, and Richard Zemel. Quantile risk con- trol: A flexible framework for bounding the probability of high-loss predictions. International Conference on Learning Representations, 2022. 6, 10
[68] Max Sponner, Bernd Waschneck, and Akash Kumar. Adapting neural networks at runtime: Current trends at-runtime optimizations for deep learning. ACM Computing Surveys, 2024.
[69] Shengkun Tang, Yaqing Wang, Caiwen Ding, Yi Liang, Yao Li, and Dongkuan Xu. Deediff: Dynamic uncertainty-aware early exiting for accelerating diffusion model generation. arXiv Preprint (arXiv:2309.17077) 2023. 6, 9, 10, 7, 12
14

---

[70] Surat Teerapitaayaoon Bradley McDanel, and Hsiang- Kung. Branchynet: Fast inference via early exiting from deep neural networks. International Conference on Pattern Recognition. 2016. 2, 6
[71] Jacopo Teneggi, Matthew Tivnan, Web Stayman, and Jeremias Sulam. How to trust your diffusion model: A convex optimization approach to conformal risk control. International Conference Machine Learning. 2023, 6, 10
[72] Alexander Timans, Christoph-Nikolas Straehle, Kaspar Sakmann, and Eric Nalisnick. Adaptive bounding box uncertainties via two-step conformal prediction. European Conference on Computer Vision 2024. 6
[73] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep iighrresolution representation learning for visual recognition. Transactions on Pattern Analysis and Machine Intelligence, 2020. 6
[74] Yulin Wang. Rui Huang, Shiji Song. Zeyi Huang, and Gao Huang. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. Advances in Neural Information Processing Systems, 2021. 7,6 6
[75] Ian Waudby-Smith and Aaditya Ramdas. Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2024. 5, 1, 2
[76] Maciej Wolezyk, Bartosz Wojcik, Klaudia Balazy, Igor T Podolak, Jacek Tabor, Marek Smieja, and Tomasz Trzcinski. Zero time waste: Recycling predictions in early exit neural networks. Advances in Neural Information Processing Systems, 2021. 5, 10
[77] Ji Xin, Raphael Tang, Jaejun Lee. Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference. Proceedings of the Association for Computational Linguistics, 2020. 10
[78] Canwen Xu and Julian McAuley. A survey on dynamic neural networks for natural language processing. Findings of the Association for Computational Linguistics, 2023. 6
[79] Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, al. A survey of resource-efficiens Ilm and multimodal foundation models. arXiv Preprint (arXiv:2401.08092) 2024. 6
[80] Xiaowei Xu, Yukun Ding, Sharon Xiaobo Hu, Michael Niemier, Jason Cong, Yu Hu, and Yiyu Shi. Scaling for edge inference of deep neural networks. Nature Electronics, 2018.
[81] Yunpeng Xu, Wenge Guo, and Zhi Wei. Conformal risk control for ordinal classification. Conference on Uncertainty in Artificial Intelligence, 2023. 6
[82) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. Conference on Computer Vision and Pattern Recognition, 2018. 9,7
[83] Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 2020. 6, 10
[84] Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. A survey on efficient inference for large language models. arXiv Preprint 0orXiv:2444.14294 2024. 6
[85]
Shlomo Zilberstein. Using anytime algorithms intelligent systems, Al Magazine, 1996. 2
[86] Thomas P Zollo, Todd Morrill, Zhun Deng, Jake Snell, Toniann Pitassi, and Richard Zemel. Prompt risk control: A rigorous framework for responsible deployment of large language models. International Conference on Learning Representations, 2023. 6, 10
15

---

APPENDIX
The appendix is organized follows:
Appendix A contains mathematical details, namely proofs for propositions ($ A.1), elaboration on bounding conditions of the loss function ($ A.2), and further details on our Brier score formulation ($ A.3).
Appendix B contains algorithmic descriptions of risk control via the Upper Confidence Bound [8] (UCB, Prop. 2) with the Waudby-Smiith-Ramdee bound [75] (WSR, Prop. 3) in Algorithm I and Algorithm 2.
Appendix C contains various implementation details of our four experiments: image classification ($ C.1), semantic segmentation ($ C.2), language modeling ($ C.3), and image generation ($ C.4).
Appendix D contains additional results across our suite of experiments (in the same order as above), including risk control and efficiency curves for varying risk levels ($ D.1, $ D.2, D.3, $ D.4), as well as additional efficiency gain tables ($ D.I, $ D.2).
A Mathematical Details
A.1 Proofs for Risk Control
We begin by formalizing the connection between marginal monotonicity requirements on the EENN (Eq. 2) and the monotonicity of risks (Eq. 3) in Lemma below.
Lemma 1. marginally monotone EENN satisfying Eq. 2 for some arbitrary loss function & implies
monotone decreasing risks of the form in 3.2, i.e., have that R(x1) 2 R((2) for A1 S A2-
Proof. For a given test sample æ, denote the exit layers corresponding exit thresholds Å₁ and X2 as l and l2. From the EENN's confidence-bee beadddence-beed exiting mechanism in Eq. it follows that 4 S (2. ie., smaller exit threshold X will result in exits that are earlier or equal to the larger threshold X2- From our risk formulation in terms of relative exit performances in 3.2 and marginal monotonicity according to Eq. it then follows that
R(A1) B( (((o (z). (()) y)]
(Ey 2) 2 (( (z) = R(22).
Next, we prove Prop. 1, our adaptation of the Conformal Risk Control [4] (CRC) guarantee on risk control in expectation for the carly-exit setting. Our proof closely follows the proof for Thm. 1 in Angelopoulos et al. [4], but we relax the condition on monotone losses to that on monotone risks, which implies assuming marginal monotonicity the EENN according Lemma 1. We restate proposition from the main paper for completeness first.
Proposition 1. Let A A + (-oo, B) be right-continuous bounded loss, and assume a marginally monotone EENN (Eq. 2). Then the exit threshold doRe ensures risk control in expectation, i.e., it holds that P [R(ACRC)] Se for any E (0,
Proof. Consider the calibration set Deat [(x,, y:) D and some test sample (xc y) P drawn i.i.d from P, and denote their union set as D == Deal U (æ, y). Additionally, define (A()) = (()x( yr) as the loss of the EENN's early-exit output for the i-th sample. In particular, (x41()) (X) now denotes the test sample's loss. Let first recall the definition of òcRc (Eq. 10):
B
ÂCRC min E A: n + Dea)
Se). S...
n+1
Note that A is a discrete grid of values over [0, 1], e.g., equidistant values (0.01, 0.02, ,..,,,,,,,, 1,, and ff Å = 0 and the risk control condition is never met, we default to Ả = for all threshold selection

---

procedures. Thus, the min is well-defined. Next, consider the empirical risk RA+1(A; D) computed over 'D using the + available samples, and define the following threshold choice:
An+1 := -ii {A A (>; S E.).
(13)
An+1 is always well-defined, since Rm,+1 1 (A = 1) zero for the risks introduced in $ 3.2. As we assume bounded loss function 8 < B, we observe that for any À E A have
Ru++ (A;
R(A; + ln+1(A)
n2 7+1
Rea:
B
n2 11
n n+
which implies that An+1 S ACRC- Since assume marginally monotone EENN, by Lemma 1 it follows that the risk is monotone decreasing and by R(Ax+1) 2 R(ACRC) also have that
EDaa~pp [R(An+1) 2 ----p~" (R(cR(c)))))
(14)
By Eq. 3 and our loss definition above, we can rewrite for general À the expression
Pe [R()] ED (x.))) (fx (A)]] Lid (en4p
in short E[en+1 1 (A)]. The remainder of the proof follows Thm. I in Angelopoulos al. [4]. Assume a particular set D is given. Then the threshold λn+1 is void randomness and constant, and by the i.dd condition we also have that (A) Unif((e1,.... (x+1) }) is uniformly distributed. Combining these observations with the law of total expectation (l.o.te.) and right-continuity (r.c.) of the loss, the final result follows:
El (ACRc)]
(Eq. 14)
S
[f
(2++1)
Loic
B(E(lat1(An---
[
Unif
E
n+1
4 (Am+1)
(E4. 13) &rc. S
Ec] =
Next, we sketch a proof for Prop. 2, our adaptation of the Upper Confidence Bound [8] (UCB) guarantee on risk control with high probability for the early-exit setting. Our main change includes modifying the
bound [75] (WSR) to relax the bounding condition on the loss from E [0, 1] to [- , B] (Prop. 3). We first restate our proposition from the main paper. Proposition 2. Let : + [-B, B) be bounded loss, and assume marginally monotone EENN (Eq. 2). Then the exit threshold λuce ensures risk control with high probability, i.e., il holds that any PD (R(AUcC) E) 1 8 for any (c, E (0,
Proof. Our result follows almost directly from the proofs Bates et al. [8] (for their Thm. A.1), where we leverage the required risk monotonicity by Lemma 1. We omit the technical requirement on risk continuity from the original proof, since a relaxation to non-continuous risks is permitted ([8), Remark 3). A proof that the WSR bound can be used to construct valid upper confidence bound can be found in Bates et al. [8], Sec. 3.1.3. However, an assumption on losses bounded to é E [0. is made, which is overly restrictive for the early-exit setting. We relax this assumption to l [- B in Prop. 3 below, concluding the result.
Since risk definitions in $ 3.2 can naturally assume negative values, and thus want to account for the occurrence of earlier exits performing better, we relax the bounding condition on the loss function for the Waudhy-S San---mttt---ada bound [75] (WSR) in the following proposition. Proposition 3. valid upper confudence bound (Eq. 11) based on the Waudby- Saiii-mmmtth-mmiit bound can be constructed for losses f E I-- B-B,, B] with B > 0.
Proof. Observe the definitions of individual components (pi of Vi, Ks)) for the WSR bound in Algorithm 2. In particular, define v; min ( / log(11))) }. Since the second term is always no?
non-negative, it follows that v4 [0. 1/2B]. For the loss & E [-B, B, B), we then have that (&+ B[4]) E [-2B, 2B.. Hence, it follows that 1 - (e: - E((,)) 2 0. which implies that K; = I My (e, E({G])) is a neganvge martingale. The rest of the proof follows Prop. from Bates al. [8]. concluding the result.

---

Observe that the proof for Prop. 3 utilizes the fraction 1/2B in its definition of 1/; which can make the WSR bound less tight. However, in the case of a marginally monotone EENN (Lemma we can relax this fraction to 1/B instead, comment we formalize in the following remark.
Remark 1. Note that in the case of marginally monotone EENN, the bound from Prop. 3 can be further optimized. For a bounded loss & E [0, B), the relative early-exit loss {(,, L) := ((xx(z),))
(()r(z), y) will fall in the [-B, B] range (see also $ 3.2 and A.2). Additionally, due to the marginal monotonicity assumption (Eq. 2) the risk based on the relative loss will be non-negative,
ie.. [((x, L) = E[e(>, L)] [0, B). This implies that (E(X, L) R((x, L)) E [-2B, B). Hence, Ki will be a non-negative martingale even when the upper bound 1/ B is used for v/ instead of 1/2B.
A.2 On Bounding of the Loss Function
The risks outlined in 3.2 rely on an early-exit loss definition in terms of relative exit performance. That is, our risks take the general form
R(A) = E(m,y)> [e(2, L(), := (()x(x),)) == 3) {()((x),))
(15)
with E(,, I.) denoting the relative early-exit loss. Recall that Qg and OG. are based on px and Pt. respectively. For a bounded loss & E [0, BJ, we then have that f(). L) E [- B, BJ. In our early-exit setting, negative losses have an intuitive interpretation. The associated test samples are cases where the EENN overthinks [40, 37], i.e., the early-exit px performs better than the final exit pr.
Risk control via CRC (Prop. 1) or UCB (Prop. 2) with the relaxed WSR bound (Prop. 3) conveniently account for such occurrences. In contrast, this presents a challenge for the Lear--then-Tee [1] (LTT) framework employed by Schuster al. [65], since the underlying Hoeffding-Bentkus Bentkus [11] (HB)
bound requires E(\, L) E [0, 1]. As a workaround, Schuster et al. [65] instead impose a lower loss limit of zero, i.e., they use
((A, = max (()x(z).)) - (().
(16)
While solving their technical requirement, it introduces a key drawback in that the risk control procedure cannot account for samples where the risk requirement is satisfied 'for free'. This introduces substantially more conservative early-exiting (see Fig. 7), since an upper bound on the empirical calibration risk is used for threshold tuning.
In addition, observe that the Brier score is naturally bounded by [0, 2] for the multi-class setting [13]. Thus, our relative early-exit Brier losses assume values in the range of [-2, While acceptable for risk control via CRC and UCB (by setting B = 2), it once again does not align with the LTT requirement ((). E [0, 1]. Thus, applying LTT requires additional restrictions such as scaling (e.g., with a 1/2 term) and Eq. 16 to satisfy the bounds. This highlights another drawback where the intuitive risk interpretation as a Brier score difference is partially lost (see $ A.3 below).
Note that while CRC and UCB are amenable to E(), L) E [- B, B). it can still be beneficial to ensure non-negative losses (e.g.. by Eq. 16) in order to improve the marginal monotonicity of the EENN (Eq. 2). Namely, it can happen that an EENN is not marginally monotone for an unrestricted loss (Eq. 15), but is for its zero-bounded counterpart (Eq. 16). Hence, such approaches might be useful when there is reason to believe that EENN violates its marginal monotonicity assumption, though in such cases practitioner might better opt for the pruning of unnecessary model layers instead.
A.3 Brier score formulation
Brier score motivation. The Brier score is a strictly proper scoring rule [26, 27], ensuring its suitability to assess probabilistic forecasts. This can be demonstrated by its decomposition structure, which highlights that both calibration and sharpness properties of the forecaster are taken into account [52, 18]. Moreover, its mathematical formulation lends itself favorably to risk control when compared to other widely used probabilistic metrics. These include the expected calibration error (ECE [28]), which requires binning and thus cannot be defined at per-sample level; the negative log- ligelikedhoodd which is less interpretable and unbounded; the ranked probability score (RPS), which does not treat class distances equally; and the continuous ranked probability score (CRPS [34]) or f divvergeenees like the Hellinger distance, which can be overly conservative by aiming to control the (potentially long) tail of the distribution, and require access to the full (ground truth) distribution. While amenable to our overall risk control framework, such probabilistic metrics, or any derived top-k uncertainty measures, seem either less practical or less principled.
3

---

0115
Risk R
0.10
max(e(A)
K(A) 4(1)
f[1],0]
Test Risk R
0.10
CRC
UCB
Exit Layer (L)
Layer
[0
ITT
10
0.05
0.10
0.30
A
Risk Level
Risk Level
Figure 7: We display risk curves based on relative early-exit losses E(X, L) for the CALM model [65] on CNN/DM (left). The risk based on the pero-bounded losses (-; Eq. 16) naturally upper-bounds the one without (-; Eq. 15). Since LTT requires E(A, L) E [o, 1], threshold tuning is performed based the larger risk (-), despite aiming to control the actual risk (-). This results in overly conservative early-exiting, as indicated by the LTT test risk (--) deviating furthest from the optimal (diagonal) risk line (middle), and its lowest empirical gains (right). Since CRC and UCB permit negative losses, we display their results for both zero-bounded (--; Eq. 16) and unrestricted (-; Eq. 15) settings, Also here, the unrestricted setting results in controlled risks with larger efficiency gains, highlighting its relevance for the early-exit setting with relative losses of the form E(X, L).
Brier loss and Brier score. For convenience (but easily transferable), consider the supervised setting where target risk control of the performance gap risk (Eq. 6) for predictive distributions, which we denote short as (p).. This requires computing the Brier loss (g (pr (y x). which define for the multi-class setting in Eq. 7. Now consider dataset D P of size N (e.g., the calibration set D Deat). The associated Brier score [13] denoted Brier( for the I-th exit layer is then defined as the mean Brier loss across samples, i.e., have
N (B.)(D))
N N (B(pi(y
K Σ
Brier (pi) 1 N
1
Σ
Eg 7)
1
Σ
(p(()))m)) 1 = k))
where B.., (fi) is an abbreviation for the n-th sample's Brier loss. The risk R RT)) that we aim to control is approximated by its empirical equivalent (C) D, and can then be interpreted the difference Brier scores between our EENN with threshold À and the full (last-layer exit) model:
N N
RC
(n(sx(y(en) ( y.)
n=1
(x))
BB Brier(fx) Brier(pt).
N
Mean-pixel Brier score. Since our semantic segmentation experiment ($ 5.2) is a pixel-level classification task, obtain pixel-level predictive distributions, and thus compute per-pixel Brier losses. For an image of height H and width W, we can denote by (Pr.i,)) the n--hh sample's Brier loss the pixel location (i,j) E H W. Since we target image-level early-exiting, these per-pixel Brier losses are averaged across pixels to compute per-image Brier loss, which in turn is averaged across samples to obtain the 1-th exit layer's Brier score. That is, the layer's associated Brier score Brier(fr is defined as
Brier(
NHW Σ
Σ
(p.())
NHW Σ Σ
( Un..i) (I.))EHW
==1 (x.
and can be interpreted as the mean-pixel Brier score of the particular exit layer. Note that if we interpret every image pixel as an individual sample (i.e., define Ñ == NHW), the Brier score formulation sample-averaged Brier loss continues hold. Similar above, the targeted risk is then interpreted as the mean-pixel Brier score difference.

---

B Algorithmic Details
Here, sketch the algorithm for computing the risk-controlling threshold λUcr (Eqq 12) via the Upper Confidence Bound [8] (UCB, Prop. 2) with the Waudby-Smith--n Smith- bound [75] (WSR, Prop. 3). The algorithm is adaptation of the approach presented in Bates et al. [8] the early-exit setting. Note that our practical code implementation differs slightly from the pseudo-code presented here, as omit here some code optimization steps such vectorization improve readability.
Algorithm 1: Risk control for EENNS via UCB (Prop. 2)
input EENN px. Deats 6, ó, loss function l, grid step output: AUcB
grid ar, 0,, # Construct the UCB (Eq. 11) UCB = np. onen (1en(ggid)
for Y an grid do
L f(xx, Deat) l(pL, Deat), # (n,)
UCB [i] WSR (L, 5) Algorithm 2
Find Auen (Eq. 12)
rcp = -1
for i, ucd in enumerate (UCB [1:]) do
if' ucb >= then
rcp = i break
return grid[rep]
Algorithm 2: WSR bound (Prop. for UCB
(see also Section 3.1.3 in Bates et al. [8])
input losses L, 8, grid step A, V bound B (default
1)
output: UCB R+
(L)
# init arrays H,, ,v, v,n = np ones (n),
for in (n)ge(n) do
p|i] (1/2+ E (i)) 1)
0 (1/4 + - ~b)))
r|i] min(1/B, V 2log(1/0)
n-1
# define function
x[i = lambda e: II -1
grid np. www.aaaeee,,,0 1)
for in grid do max, x[]](e) > 1/6 then
return
5

---

C Implementation Details
Since our approach is post-hoc, the required compute resources needed are minimal. The primary requirement is sufficient memory to process the required calibration and test data, which can be quite large (e.g., images of size 2048 x 1024 pixels for Cityscapes). All our experiments can be performed and replicated on single A 100 GPU with experiment runtimes of <1 day. We detail further case-specific implementations below for each experiment.
C.1 Image Classification
Model details. The models we consider are: the multi-scale dense network (MSDNet; [36]), an adaptation of traditional convolutional NNs for the early-exit setting; the dynamic vision transformer (DViT; [74]), which comprises multiple transformers with an increasing number of imput patches; an enhanced MSDNet model that weights easy and hard examples differently during training (L2W- DEN: [30]); and recently proposed dynamic perceiver (Dyn-Pere: [31]), which decouples feature extraction and early prediction tasks via a novel dual-branch architecture. For all models, we either work with the publicly available pretrained checkpoints or train the models ourselves, closely following the original implementation details.
C.2 Semantic Segmentation
Model details. We consider the EENN proposed by Liu et al. [48] (ADP-C), which adds three intermediate exit heads the HRNet segmentation model [73] (for total of four exits) and is trained end-to-end on Cityscapes [17]. The model comes in two sizes, small (W18) and large (W48). We focus on the larger model (ADP-C-WAL) W48), but find results hold equivalently for the smaller one (in fact, larger gains can be obtained), Across experiments we employ the publicly available model checkpoints from the original implementation'.
Model finetuning. Since the model is trained on Cityscapes, we consider additional finetuning evaluate ADP-C on GTAS [59]. We take the available pre-trained model checkpoint (APD-C- W48) and finetune the model for 50 epochs on the GTA5 training set (~~ 12000 images). For this purpose, we employ the original training script and training parameters (e.g., learning rate, batch size, etc.). However, we find that our finetuned model does not perform on par with the original. i.e. performance on GTAS substantially inferior to that on Cityscapes. In particular, the performance improvement across subsequent exits on GTA5 is marginal, resulting in an EENN that is less suitable for early-exiting (see also $ D.2). Yet, we find that risk control frameworks still apply, highlighting their robust model-agnostic properties even in light of an inferior underlying predictor.
Image-level aggregation. ADP-C provides exiting mechanism following Eq. I on pixel-level, which less useful for down-stream applications and decision making. For details on the exact mechanism, refer to Liu al. [48]. Rather than exiting only for selected image pixels, we instead want to early-exit the entire image whilst ensuring risk control. Thus, alongside different per- per-pixel confidence scores C,,,j (i.j) И W. we also consider confidence aggregations o(.) which produce single image-level confidence measure è to perform image-level risk control. Note that our prediction losses mean intersection-over-eeee lmage-level, whereas our distributional loss (Eq. 7) is adapted to additionally average over pixels, resulting in mean-pixel Brier score interpretation (see $ A.3).
and miscoverage already aggregate from pixel- to
Risk control evaluation. We evaluate the original ADP-C-W48 on Cityscapes validation data, with split of 80% Deat (i.e., 400 images) and 20% Drest (i.e., 100 images). Similarly, we randomly select a subset of 1000 images from the GTAS validation set and evaluate our finetuned model using 80% Deat (i.e., 800 images) and 20% Dtest (i.e., 200 images). In both cases, average risk control results over 100 trials of random calibration and test splits.
SSee httpp://gittie com haamm//luhuuaagiin tiyt

---

C.3 Language Modeling
Model details. For our language modeling experiments, we employ the early-exit pretrained model based on T5-large (770M parameters) from Bae et al. [5] While this model closely follows the implementations Schuster al. [65], we found it easier work with than the original code?. Note that Schuster et al. [65] report results for s5-small (60M parameters) and T5-base (220M parameters), whereas we use the larger T5-large. For risk control evaluation, we follow their exact exiting mechanism. Specifically, we compute softmax-based confidences at every exit and deploy their threshold decay mechanism, where early-exiting more conservative for initial tokens and becomes progressively more permissive ([65), Eq. 5).
C.4 Image Generation with Early-Exit Diffusion
Model details. For our image generation experiment, we re-implement the early-exit diffusion model proposed by Tang et al. [69] (DeeDiff), since the original code is not publicly available. We model our training procedure
closely as possible to the original. suggested the paper, we use the U- UTV transformer [7] as a backbone denoising network. Farly-exiting in DeeDiff is performed on the denoising network each sampling step. Specifically, for every sampling step t and exit layer
11, ................................ L, per-pixel confidence map Ce,r is obtained. Then, Ce, is used to compute the global (scalar) confidence score Ce.t by averaging the confidence across all pixels. the scalar confidence score satisfies the exit condition Ce.t 2 X, we proceed to the next denoising step / + 1, employing the output (i.e., the predicted noise) the e-th exit layer at the f-th sampling step. The model trained using standard diffusion denoising loss [35] and two uncertainty-awet losses, closely following the approach described in Tang et al. [69].
LPIPS metric. Our task-specific prediction loss measures the perceptual difference between early- exit and full model image generations. For this, we employ the LPIPS metric [82], which computes the similarity between activations of image patches for selected pre-trained network. LPIPS values are the range [o, 1], with smaller values indicating perceptually more similar images.
D Further Experimental Results
D.1 Image Classification
Additional test risk and efficiency curves for calibration set size n. 1000 on ImageNet are displayed in Fig. 8, while tables with efficiency gain values on ImageNet are given in Table 2.
(1)
RP) (Brier)
RC (0) 13
RP) (Brier)
Test Risk
0.3
0.1
00
Exit Layer (3)
MSONA
CRC
LOW-DEN
Dyn-Por
0.
0.1
0.2
0.
0.4
0.0
0.1
022
0.3
0.4
0.0
0.1
02
0.4
0.0
0.1
0.3
0.6
Risk Levele
Figure 8: Empirical test risk (top) and efficiency gains (bottom) for different early-exit models, risks (S 3.2) and risk levels ImageNet (for calibration set size 1000). In line with theoretical results, the test risk is controlled across models, risk types, and levels. Despite guaranteeing control in expectation (CRC, Prop. 1) with high probability (UCB, Prop. 2), obtained gains substantial.
Seee http:://gggtllaocooaa//////////s roreetrrrrertteeeeeeeeettt "Se http://hhw....ooooooeee//eee///a

---

Table 2: Efficiency gains for various EENNs on ImageNet, for risk control via CRC (Prop. or UCB (Prop. 2) and calibration set size {100, 1000}. Displayed values denote relative improvement over last-layer exiting in tenms of mean exit layer (in %). The test risk successfully controlled in all cases. Results focus small risk levels E {0.01, 0.05), which are of higher practical interest.
(a) UCB and 100
Risk
(O)((0--1)
RC (Brier)
R (0)(0-1) 1)
RC (Brier)
Level E
0.01
0.05
0.01
0.05
0.01
0.05
0.01
0.05
MSDNet
0.0
42.51
0.0
30.62
0.0
34.21
0.0
30.47
DVTT
0.0
46.5
0.0
35.4
0.0
45.14
0.0
39.74
L2W-DEN
0.0
49.47
0.0
33,09
0.0
41.77
0.0
35.77
Dyn-Perc
0.0
45.42
0.0
35.67
0.0
31.71
0.0
34.85
(b) CRC and
100
Risk
(C)
RC)) (Brier)
RC) 1)
(f) (Brier)
Level
0.01
0.05
0.01
0.05
0.01
0.05
0.01
0.05
MSDNet
41.64
58.6
10.99
43.58
30.71
43.68
4.46
40.54
DVIT
49.32
58.6
5.02
51.09
40.36
51.83
4.58
46.89
L2W-DEN
50.82
64.38
8.09
50.4
35.09
50.57
0.0
44.66
Dyn-Perc
44.39
63.9
36.86
59.42
30.09
57.99
31.19
58.05
(c) UCB and 1000
Risk
RC(P (p) (Brier)
(0) 1)
RP) (Brier)
Level e
0.01
0.05
0.01
0.05
0.01
0.05
0.01
0.05
MSDNet
33.23
56.0
27.94
46.11
27.05
44.0
28.05
43.65
DVIT
38.24
58.12
34.93
52.31
34.61
50.02
34.5
48.31
L2W-DEN
38.11
60.87
28.96
47.97
32.04
49.71
26.46
43.6
Dyn-Perc
42.24
63.77
49.9
62.51
30.48
56.01
50.34
60.28
(d) CRC and n 1000
Risk
(0)((0)))
RC (f) (Brier)
RC(())
RC (p) (Brier)
Level
0.01
0.05
0.01
0.05
0.01
0.05
0.01
0.05
MSDNet
46.56
62.55
33.11
50.56
33.01
46.12
34.09
46.26
DVIT
48.68
61.6
39.05
54.89
40.75
52.21
38.48
49.86
L.2-DDEN
47.73
65.04
35.09
53.81
37.08
51.7
33.76
48.48
Dyn-Perc
55.16
66.73
54.28
65.25
43.35
58.62
53.67
62.04

---

D.2 Semantic Segmentation
We report full tables with efficiency gains across risks and confidence measure combinations for Cityscapes (Table 3) and GTA5 (Table 4) in terms of mean exit layer and GFLOPS. In addition, we display test risk and efficiency curves on both Cityscapes (Fig. 9) and GTAS (Fig. 10) for risk control via CRC (Prop. 1) and UCB (Prop. 2). The figures are for the simplest combination of Top-1 pixel-level confidence and mean image-level aggregation. Note that the figures for other confidence combinations are similar and thus omitted, with the test risk being controlled in all cases.
GTA5 results. We observe that for GTA5 the performance gap risk for prediction control R (ÿ) (mloU) seems particularly easy to control, with high gains reached even for very strict 0.01. This relates to the underlying predictor's generally inferior performance due to finetuning (see C.2). The obtained model records lower performance and marginal improvements across exit layers, resulting in small risk that is casy to control. Intuitively, the price paid by exiting early is marginal, since the early-cxit layer performs almost on par with the final layer. Thus, the scale of the risk deviates from that other risks, and more meaningful risk control should consider both improving the underlying predictor, and selecting different scale of risk levels c.
12 (91tmloo)
R (Uccee)
R2 (M)(Msscon))
Rº (P)1Brien)
0:33
0.22
0:33
Tel
0.0
000
Esst Joyer (3)
110
Layar
CRC
0.33
0.22
0.25
3.0
0.25
0.00
0.3
0.8
0.30
Risk Level
Risk Levele
Figure 9: Empirical test risk (top) and efficiency gains (hottom) for different risks ($ 3.2) and risk levels on Cityscapes. In line with theoretical results, the test risk controlled across risk types and levels. Despite guaranteeing control in expectation (CRC, Prop. 1) with high probability (UCB, Prop. 2), obtained gains are meaningful. For both figures, we consider the simplest combination of Top-1 confidence score and mean image-level aggregation.
RS (Missco)))
(p))(rrr))))
0.230
0.25
0115
Rask
0.05
0.000
Test Rbsk
0.10
0/0
005
0300
0.5
0.5
0.31
Eott Laver (1)
Fsid Layer (1)
13C
Risk Tarvel
Risk T Lavela
Figure 10: Empirical test risk (top) and efficiency gains (bottom) for different risks ($ 3.2) and risk levels on GTA5. In line with theoretical results, the test risk is controlled across risk types and levels. Despite guaranteeing control expectation (CRC, Prop. 1) with high probability (UCB, Prop. 2), obtained gains are meaningful. For both figures, we consider the simplest combination of Top-1 confidence score and mean image-level aggregation.

---

Table 3: Efficiency gains for semantic segmentation with risk control via UCB (Prop. 2) on Cityscapes.
We evaluate for different risks ($ 3.2), confidence measures ($ 5.2) and risk levels c. Displayed values denote relative improvement over last-layer exiting (in %) in terms of mean exit layer or floating point operations (GFLOPS). The test risk is successfully controlled in all cases.
(a) Efficiency gains in terms of mean exit layer improvement.
Risk
(y) (mloU)
( (Brier)
(g) (Miscov.)
R (p) (Brier)
Level e
0.01
0.05
0.1
0.01
0.05
0.1
0.01
0.05
0.1
0.01
0.05
0.1
Mean
Top-1
6.3
33.7
53.5
0.0
13.6
43.4
6.3
39.2
61.8
0.0
39.3
58.4
Top-Diff
9.3
54.4
44.3
39,9
62.4
38.6
Entropy
5.2
36.0
54.3
0.0
17.9
41.0
5.1
40.4
61.3
0.0
40.1
58.3
Quant
Top-1
0.0
36.7
54.6
0.0
14.9
45.0
0.0
41.2
63.4
0.0
39.1
59.4
Top-Diff
37.1
55.2
17.2
45.2
41.2
63.7
0.0
40,4
Entropy
6.1
37.0
54,0
0.0
17.9
44.7
6.1
41.0
63.1
0.0
39.1
58.7
Patch
Top-1
10.0
35.7
53.3
0.0
18.4
45.3
8.8
39.1
61.5
0.0
38.0
58.3
Top-Diff
10.0
35.2
53.4
0.0
19.4
45.9
8.8
40.5
62.2
0.0
38.4
58.8
Entropy
9.1
53.5
18.0
45.8
38.9
61.5
37.3
57.1
(b) Efficiency gains terms of GFLOPS improvement.
Risk
Ro(g) (mloU)
( (Brier)
Ro( (Miscov.)
RC(p) (p) (Brier)
Level
0.01
0.05
0.1
0.01
0.05
0.1
0.01
0.05
0.1
0.01
0.05
0.1
Mean
Top-1
6.2
33.4
53,2
0.0
13.5
43,1
6.2
38,9
61.4
0.0
39.0
58.0
Top-Diff
9.2
35.3
54.0
0.0
17.4
44.0
6.3
39.6
62.0
0.0
38.3
57.5
Entropy
5.1
35.7
53.9
17.7
40.7
5.1
40.1
60.9
0.0
39.8
57.9
Quant.
Top-1
0.0
36.4
54.2
0.0
14.8
44.6
0.0
40,9
62.9
0.0
38.8
58.9
Top-Diff
0.1
36.8
54.8
0.0
17.1
44.8
0.0
40.9
63.3
0.0
40.1
59.1
Entropy
53.6
44.4
40.7
Patch
Top-1
9.9
35.4
52,9
18.2
44,9
8.7
38.8
61.0
0.0
37.7
57.8
Top-Diff
9.9
34.9
53.0
0.0
19.2
45.6
8.7
40.2
61.7
0.0
38.1
58.4
Entropy
34.6
53.1
17.8
45.5
38.6
61.1
0.0
37.0
56.7
Table 4: Efficiency gains for semantic segmentation with risk control via UCB (Prop. 2) on GTA5. We evaluate for different risks ($ 3.2), confidence measures ($ 5.2) and risk levels c. Displayed values denote relative improvement over last-layer exiting (in %) in terms of mean exit layer or floating point operations (GFLOPS). The test risk is successfully controlled in all cases.
(a) Efficiency gains in terms of mean exit layer improvement.
Risk
RC (z)) (mloU)
R ( (Brier)
R R (Miscov.)
RC( (f) (Brier)
Level
0.01
0.05
0.1
0.01
0.05
0.1
0.01
0.05
0.1
0.01
0.05
0.1
Quant. Mcan
Top-1
63.3
75.0
75.0
0.2
4.0
10.3
3.9
21.2
37.4
3.7
21.5
37.6
Top-DDff
75.0
0.3
12.0
21.8
39.2
3.0
23.6
Entropy
62.5
75.0
75.0
0.2
2.9
12.5
2,8
18.5
39.9
2.8
18.7
42.9
Tup-1
63.4
75.0
75.0
0.0
4.6
12.4
2.5
23.1
42,3
2.4
23.4
42.6
Top-Diff
63.8
75.0
75.0
0.0
12.6
4.2
22.8
42.1
3.0
24.1
43.4
Entropy
61.1
75.0
75.0
0.1
14.0
3.7
23.6
41.8
3.6
23.9
43.6
Patch
Top-1
60.1
75.0
75.0
2.9
18.3
35.6
3.9
18.9
35.5
2.9
18.3
35.6
Top-Diff
60.2
75.0
75.0
3.5
19.7
37.2
4.7
19.5
37.1
3.5
19.7
37.2
Entropy
58.9
75.0
75.0
2.2
19.0
36.7
19.0
36.5
2.2
19.0
36.7
(b) Efficiency gains in of GFLOPS improvement.
Risk
RC Rg) (g)) (mloU)
R (Brier)
( (Miscov.)
RC() (p) (Brier)
Level
0.01
0.05
0.1
0.01
0.05
0.1
0.01
0.05
0.1
0.01
0.05
0.1
Top-1
62.8
75.0
75.0
0.2
3.9
10.2
3.9
21.1
37.2
3.7
21.3
37.3
Top-Diff
62.5
75.0
75.0
0.3
11.9
21.6
38.9
3.0
23.5
42.7
Entropy
62.0
75.0
75.0
0.2
2.9
12.4
2,8
18.3
39.6
2.8
18.6
42.6
Quant
Tup-1
63.0
75.0
75.0
0.0
4.5
12.4
2.5
22.9
42.0
2.4
23.2
42.3
Top-Diff
63.4
75.0
75.0
0.0
12.5
4.2
22.7
41.7
3.0
24.0
43.1
Entropy
60.6
75.0
75.0
0.1
13.9
3.7
23.4
41.5
3.5
23.7
43.2
Patch
Top-1
59.7
75.0
75.0
2.9
18.2
35.4
3.9
18.7
35.2
2.9
18.2
35.4
Top-Difr
59.8
75.0
75.0
3.4
19.5
36.9
4.7
19.4
36.8
3.4
19.5
36.9
Entropy
58.4
75.0
75.0
2.2
18.8
36.4
18.8
36.2
2.2
18.8
36.4
10

---

D.3
Language Modeling
CNN/DM (Naat = 100)
CNN/DM (Ma = 1000)
SQuAD (Mat 100)
5Qu QuAA (Aoal 1000)
0.20
0.20
0.20
0.20
0.15
0.15
0.15
0.15
Test Risk
0.10
0.10
0.10
0.10
15.0
15.0
Exit Layer (L)
12.5
12.5
3.0
CRC
10.0
2.5
2.5
LIT
7.5
2.0
2.0
1.5
0.10 0.20
0.10
0.20
0.10
0.20
0.10
0.20
Figure 11: Empirical test risk (top) and efficiency gains (bottom) for the CALM model [65] for text summarization on CNN/DM and question answering on SQuAD. Our adaptation of UCB [8] (Prop. 2) outperforms the LTT [1] approach CALM by yielding larger efficiency gains under the same risk control assurances. Shading denotes the standard deviation across S 100 calibration/test splits.
State-Saturation
Meta-Classifier
Softmax
Test Risk
0.10
0.10
0.10
0.05
0.05
0.05
Exit Layer (1)
20
20
20
CRC
LIT
0.05
0.10
0.15
0.05
0.10
0.15
0.05
0.10
0.15
Risk Level
Figure 12: Empirical test risk (top) and efficiency gains (bottom) for the CALM model for text summarization on CNN/DM across different confidence measures (see Schuster et al. [65], $3.5). From left to right: Hidden state saturation, meta-classifiers, and top-2 softmax difference. Our employed risk control frameworks based on CRC and UCB continue to outperform LTT across all measures of confidence. Shading denotes the standard deviation across 100 splits.
D.4 Image Generation with Early-Exit Diffusion

---

Test Risk R
0.00
13
CRC
Exit (
10
UCB
5
0.00
0.05
0.10
"Full model" é = 0.01
c = 0.05
6 = 0.10
6 0.15
Risk Level
Results for early-exit diffusion with DeeDiff [69] on CIFAR [42]. Right: Empirical test risks are controlled for both CRC (Prop. 1) and UCB (Prop. 2) (for calibration set size = 500). Left: The quality of generated images is directly related to the targeted risk control level
Figure 13:
12

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We provide all theoretical (see 3) and experimental (see $ 5) results backing up our claims.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We clearly list out our key limitations in $
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We state our theoretical results in $ 3.3 and provide the proofs in $ A.1.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe our experiments in 5 and provide further implementation details in Appendix C.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code publicly available at https: //t::://hhtt/t com/metodjj /RC- /RCCEENCCN.NE In $ 5 and Appendix C, we describe our experimental setup in detail. All employed datasets are publicly available.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe our experiments 5 and provide further implementation details in Appendix C.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We display I-sigma standard deviations across multiple trials of random calibration and test data splitting in all figures with empirical test risk and efficiency curves. The exceptions are Fig. and Fig. where we omit them to improve readability, without affecting any interpretation of results.
8. Experiments Compute Resources
13

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Since our approach is post-hoc, the required compute resources needed are minimal. The primary requirement sufficient memory jointly load the required calibra- tion and test data, which can be quite large (e.g., for 2048 x 1024 images on Cityscapes). All our experiments can be performeedreplicateed on single A100 GPU with runtimes of < ld. We mention this in Appendix C.
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the Cwwublic/Ethssuiieei
NeurlPS Code of Ethics ptttts://neurdii
Answer: [Yes]
Justification: We checked the Code of Ethics carefully and our work complies with all the points listed there.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We clearly mention what we believe to be the impacts of our work in $
Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work doesn't introduce any new foundation generation models, hence additional safeguards are not needed.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We made sure properly cite all the related work and datasets used.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation
provided alongside the assets?
Answer: [Yes]
Justification: We describe our theory in 3.3. We also additionally provide pseudo-code for our main algorithm in Appendix B, and reference all underlying papers. Our code is com/metod //Reettdddjo/Booeeeee TRC-EENN.
publicly available at
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No experiments involving crowd-sourcing human experiments were con- ducted in our research.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
14

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No experiments involving crowd-sourcing human experiments were con- ducted in our research.
15


Reviews:
[{"review_id": "XJ5ysQZrVe", "reviewer": "Reviewer_qAvq", "summary": "This manuscript revisits the confidence-based thershold tuning for early-exit networks following a risk-control formulation. The proposed approach aims to provide an enhanced mechanism to identify confident predictions early-on during the computation, and exit early to improve inference efficiency with minimal effect on accuracy. The underlying trade-off between accuarcy and latency is balanced by a post-hoc solution, that determines the thershold value for the exits, through a risk control formulation considering a labelled calibration set (Performance Gap Risk), or the predictions of the final exit of the original model (Consistency Risk).", "questions": "- Please consider the comments raised on the Weaknesses section and share any insights you may have on them.\n- Can the proposed Consistency Risk formulation be used for online adaptation of the thresholds, as during runtime calibration labels are not available, but input distribution shifts may occur?\n\n**Post-Rebuttal Edit:**\nProvisionally increasing my score from WA to A, given the thorough clarifications provided by the authors that adequately addressed most of my concerns.", "limitations": "Limitations and future work are adequately discussed in the manuscript.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The manuscript studies a very interesting and timely problem, often overlook by the booming filed of early-exit models. \n- The proposed approach is post-hoc and can be applied to existing approaches without the need of re-training/finetunning the model, and does not add any computational overhead at inference time. \n- The effectiveness of the proposed approach is experimentally demonstrated on a wide range of predictive (image classification and segmentation) and generative tasks (language modelling and image generation), proving the generality of the proposed approach. \n- The paper is well-written and offers adequate background information and a nice formulation.", "weaknesses": "- The reliance of the proposed approach to marginal monotonicity between performance and depth can be a limiting factor in some applications. E.g. recent works on EE-based LLMs indicate the predictive accuracy fluctuates notably across exits (see Elhoushi et al, LayerSkip, 2024 - Fig.2). \n- As stated in the limitations, the adoption of a single and static threshold across exits also significantly reduces the search space of the proposed approach. The inefficiency imposed by this design choice has not been quantified, but is likely to be significant without any calibratuon technics on exit confidence or special training of the model. \n- It is unclear how the proposed approach can be applied to the emerging line of works with learnable exit policies, as well as whether it will maintain its performance on different confidence metrics, such as state-saturation, used in LLMs etc", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer qAvq, we thank you for your time and engaged questions.\n\n> The reliance of the proposed approach to marginal monotonicity between performance and depth can be a limiting [...] (see Elhoushi et al, LayerSkip, 2024 - Fig.2).\n\nThis is a great point and we fully agree that for a particular sample the predictive accuracy can fluctuate across exits. We refer to this effect as *overthinking* (following [5, 12]) and mention it several times throughout the paper (L77-80, L196-198, L211-214). However, note that our approaches do not require such strong assumptions on *conditional* monotonicity, i.e., monotonic behaviour per sample. Rather, we only require the milder assumption of *marginal* monotonicity, i.e., that predictive accuracy is monotone across exits *on average* over samples (Eq. 2 in the paper). Such a marginal condition is often implicitly assumed in the early-exit literature, and validated by the commonly shown performance vs. FLOP curves which monotonically grow as the budget increases. Indeed, Fig. 2 in Elhoushi et al. [13] is showcasing a lack of *conditional* monotonicity, since the model outputs are shown for a particular input prompt. Looking at Fig. 6 and 8 from the same paper, we observe that performance is generally monotonic w.r.t. exits on average (marginally). In fact, **the difference between conditional and marginal monotonicity serves as a motivation for one of our main technical contributions** - in the original formulation, CRC requires conditional monotone risks, while we extend this to marginal monotone ones (see Prop. 1). As we state in the paper (L192-198), such an extension is crucial to make CRC amenable to EENNs because of the overthinking issue.\n\n> As stated in the limitations, the adoption of a single and static threshold across exits also significantly reduces the search space [...] inefficiency imposed by this design choice has not been quantified... .\n\nPlease consider our response under point (1) in the general rebuttal, and kindly provide additional questions in case you feel it has not been sufficiently addressed.\n\n> how the proposed approach can be applied to the emerging line of works with learnable exit policies\n\nWe assume you refer to papers such as JEI-DNN [14], where exits are determined via direct modeling of exit probabilities. If so, it is correct that our procedure does not directly transfer since we require a threshold-based exiting mechanism. We will ensure to better define the scope of our contribution, and we thank you for pointing this out. Note, however, that models with learnable exit policies are less adaptive compared to their threshold counterparts, as they require retraining the model from scratch for every new computational budget. In contrast, threshold-based models can be adapted in a post-hoc fashion by simply tuning the threshold. \n\nLet us also add here that thresholding is not used only in early-exiting, but also in some other emerging lines of work like (soft) speculative decoding [15], where the rollback policy is governed by thresholding. Our risk control framework could be applied to such settings as well, and we will try to add an experiment to the camera-ready version to better exemplify the scope of our contribution.\n\n> whether it will maintain its performance on different confidence metrics, such as state-saturation, used in LLMs\n\nWe note that **our approach is agnostic to the choice of confidence measure** used when early-exiting. We aim to demonstrate this in the paper with our segmentation experiment in §5.2 (see Table 1; and Table 3 in §D.2), where we consider different confidence measures and aggregators. Based on your comment, **we have extended our language modeling experiment to include additional confidence measures**: (i) hidden-state saturation and (ii) meta-classifiers, as proposed by [2]. We observe in the Figure B of the attached rebuttal PDF that our employed risk control frameworks based on CRC and UCB continue to outperform LTT across all confidence measures, and will include these results in the camera-ready version.\n\n> Can the proposed Consistency Risk formulation be used for online adaptation [...], but input distribution shifts may occur?\n\nWe are not entirely sure what form of online adaptation is meant here, so kindly inform us if the question was not properly addressed.\n\nFor distribution shifts between training and calibration data (i.e., $P_{train} \\neq P_{cal}$) risk control continues to be applicable as long as the model’s marginal monotonicity is not violated. Our post-hoc approach treats the EENN as a black-box and makes no assumptions on the training distribution (as stated in §2, L50-51). For shifts between calibration and test data (i.e., $P_{cal} \\neq P_{test}$) the provided guarantees cease to hold, since an *i.i.d.* assumption on those samples is made. However, under benign shifts the empirical test risks may continue to be controlled.\n\nSince we guarantee risk control even for small calibration set sizes ($n \\approx 100$) and are computationally light-weight, a naive online update could see a periodical re-calibration of the exit threshold based on collected test samples, which, in the case of our *consistency risk*, would indeed not even require any test labels. We fully agree that it is an interesting direction for future work to explore proper online updating. \n\nWe thank you for your comments and would appreciate you considering raising your score if you believe they have been addressed sufficiently.\n\n**References**\n\n[12] Jazbec, M., et al. Towards anytime classification in early-exit architectures by enforcing conditional monotonicity. *NeurIPS 2023*\n\n[13] Elhoushi, Mostafa, et al. Layer skip: Enabling early exit inference and self-speculative decoding. *arXiv preprint* (2024)\n\n[14] Regol, F., et al. Jointly-learned exit and inference for a dynamic neural network: Jei-dnn. *ICLR 2024*\n\n[15] Kim, S., et al. Speculative decoding with big little decoder. *NeurIPS 2023*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VWOkUaMNg6", "reviewer": "Reviewer_c7rs", "summary": "This paper proposes a method for improving the efficiency of early-exit neural networks (EENNs) while maintaining performance. ​ EENNs allow for predictions to be made at intermediate layers, resulting in faster inference. ​ However, the challenge is determining when it is safe to exit without sacrificing performance. ​ The authors address this issue by adapting risk control frameworks to EENNs. ​ They propose risk functions to measure the performance drop resulting from early-exiting and demonstrate the effectiveness of their approach on various vision and language tasks. ​", "questions": "Refer to Weaknesses.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper addresses an important problem in machine learning - improving the efficiency of inference without sacrificing performance. ​\n- The authors propose a novel approach of adapting risk control frameworks to EENNs, which provides a post-hoc solution for determining when it is safe to exit. ​\n- The paper provides empirical validation of their approach on a range of vision and language tasks, demonstrating substantial computational savings while preserving performance goals. ​\n- The authors consider both prediction quality and uncertainty estimation in their risk control framework, which is important for safety-critical scenarios. ​", "weaknesses": "- The empirical validation of the approach is limited to a range of vision and language tasks. It would be beneficial to see the application of the method to other domains as well.\n- The paper does not discuss the limitations or potential drawbacks of the proposed approach. It would be helpful to have a discussion on the potential trade-offs or challenges in implementing the risk control framework in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer c7rs, we thank you for your time and comments, and address your two raised concerns in the following.\n\n> The empirical validation of the approach is limited to a range of vision and language tasks. It would be beneficial to see the application of the method to other domains as well.\n\nWe politely disagree with this statement. Our experiments incorporate a wide variety of models, confidence measures and risk types across multiple machine learning tasks, datasets and data modalities (image classification on ImageNet with MSDNet, DViT, L2W-DEN and Dyn-Perc for 0—1 and Brier losses; semantic segmentation on Cityscapes & GTA5 with ADP-C for mIoU, miscoverage and Brier losses and several confidence measures; text summarization on CNN/DM and question answering on SQuAD with T5 for ROUGE-L and Token-F1 losses; image generation with early-exit diffusion on CelebA and CIFAR for LPIPS loss).  Moreover, our extensive experiments have been highlighted as a key strength by all other reviewers (qAvq: \"*experimentally demonstrated on a wide range of predictive ... and generative tasks*\"; LhAs: \"*extensive experimental evidence across multiple domains*\"; q2CX: \"*The experiments on different vision tasks can be strong evidence to show the effectiveness…*”).  \n\nLastly, we are surprised since your review also lists our experimental evaluation as a strength (”*The paper provides empirical validation of their approach on a range of vision and language tasks, demonstrating substantial computational savings while preserving performance goals*”); perhaps one of the mentions of our experiments was in error?\n\nIf you have any concrete additional experiments in mind, please let us know and we would be happy to consider them.\n\n> The paper does not discuss the limitations or potential drawbacks of the proposed approach. It would be helpful to have a discussion on the potential trade-offs or challenges in implementing the risk control framework in practice.\n\nWe politely point to §6, which is titled “Conclusion, Limitations and Future Work”.  There we state our main limitation (a single exit threshold) and point to several limitations and opportunities for future work that are rooted in risk control itself, rather than our adaptation to the early-exit setting.  Furthermore, we are explicit about our marginal monotonicity requirement for the EENN (e.g., in our propositions) and discuss it throughout the paper. The presence of Limitations has also been acknowledged by the other reviewers (qAvq: \"*Limitations and future work are adequately discussed in the manuscript.*\"). Lastly, we will use the additional page of the camera-ready version to further expand on the limitations. \n\n> 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.\n\nBased on your current review, it is not entirely clear to us why you are recommending rejection. We would very much appreciate further elaboration of your concerns, or a consideration to raise your score if you believe they have been addressed sufficiently.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OITljIKfzo", "reviewer": "Reviewer_LhAs", "summary": "This paper introduces a risk control framework for early-exit neural networks to balance the trade-off between inference efficiency and model performance. Compared to the conventional methods that manually set the confidence thresholds for early exiting, this work proposes a method to determine exit points from the perspective of risk control. Experiments on vision and language tasks demonstrate the effectiveness.", "questions": "Please see weaknesses.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Novelty: The integration of risk control into EENNs is novel, and the studied problem is an interesting and important challenge in deploying early-exiting models;\n* Empirical Validation: The paper provides extensive experimental evidence across multiple domains, demonstrating the practical applicability of the proposed methods;\n* Soundness: The risk control framework is well-developed, offering both theoretical foundations and practical algorithms for determining exit thresholds.", "weaknesses": "Overall I like the paper. A small suggestion is to demonstrate the accuracy-FLOPs curves as in the compared methods (MSDNet, L2W-DEN, and Dyn-Perceiver), because the current x-axis, \"risk level\", might be insufficiently straightforward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer LhAs, we thank you for your time and comments, and address your raised point in the following.\n\n> A small suggestion is to demonstrate the accuracy-FLOPs curves as in the compared methods (MSDNet, L2W-DEN, and Dyn-Perceiver), because the current x-axis, \"risk level\", might be insufficiently straightforward\n\nWe would politely point out here that our risk control approach is entirely *post-hoc*. As such, we do not change the performance of any underlying model but work with existing pretrained EENNs, and thus our model performances are equivalent to the numbers provided by the original authors. While we provide performance curves in Fig. 1 for some models, this was to stress the marginal monotonicity condition rather than testing model’s ability. Thus, we consider reporting the commonly seen performance vs. FLOP curves found in early-exit papers as redundant for our work, since we do not modify the underlying models.\n\nPerhaps it's also helpful to add here that we may interpret our risk control threshold selection as a framework to navigate the performance vs. FLOP curve based on the user’s performance requirements, i.e. we find the point on the curve such that the EENN’s test performance is *guaranteed* to be at most $\\epsilon$ (e.g., at most 5%) worse than the full model (i.e., last-layer exit). Hence, we believe that evaluating against different risk levels $\\epsilon$ is the appropriate way to benchmark our procedure. Also note that provided “test risk” figures such as Fig. 2 and 3 in the paper are standard in the risk control domain (see, e.g., Fig. B.1 in [2]). Moreover, the fact that the test risk curves are close to the diagonal (e.g., see Figure 3, upper row) is encouraging, as it suggests that with our framework, there is very little efficiency cost for the added safety of having risk-control guarantees.\n\nWe thank you for the suggestion, highlighting the need to better explain provided test risk figures. We attempted to do so in §5 (L272-278), but shall expand upon it for the camera-ready version based on your feedback.\n\n[2] Schuster, T., et al. Confident adaptive language modeling. *NeurIPS 2022*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LNjq77UKoa", "reviewer": "Reviewer_q2CX", "summary": "This paper investigate how to adapt frameworks of risk control to early-exit neural networks (EENN). The new framework is designed to exit without severely degrading performance as well as to guarantee the reliability of the prediction during early exiting.  The applied risk control technique offers a distribution-free, post-hoc solution that tunes the EENN’s exiting mechanism so that exits only occur when the output is of sufficient quality. Experiments on a series of tasks, including image classification, semantic segmentation, language modelling, and image generation show the effectiveness of the proposed method.\n\n-------------after rebuttal----------\n\nThe authors address most of my concerns. Although the single exiting threshold seems somehow a little flaw, the proposed new method is interesting and new.", "questions": "See weakness. I think more evaluation metrics, such as test accuracy for classification tasks, or the IOU for segmentation tasks, should be included to show the effectiveness of the proposed method.", "limitations": "See weakness. I think it is acceptable to use the same threshold for theoretical analysis. However, the real performance of the evaluated EENNs should also be attached in the manuscript.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The experiments on different vision tasks can be strong evidence to show the effectiveness of the proposed method. \n2. The theoretical analysis seems reasonable to be.", "weaknesses": "The main weakness of this paper is under the strong assumption that all exiting thresholds are set to the same values. I admit that relaxing this assumption by adopting RC techniques for a high-dimensional threshold can lead to some difficulties in theoretical analysis, while using the same threshold for all exits may harm the overall performance of EENN. Only empirical test risk is used in all experiments. The general performance of EENN models in different tasks is also an important evaluation metric for EENN. \n\nThe applicability of the proposed RC method can be impeded if the prediction performance is unsatisfactory when using the same exiting threshold for every exit classifiers, even though it has a lower risk in the prediction.\n\nIt is also suggested to review the related important works in early exiting neural network areas, such as:\n[1] Improved techniques for training adaptive deep networks. in ICCV, 2019.\n[2] Resolution Adaptive Networks for Efficient Inference, in CVPR, 2020.\n[3] Learning to Weight Samples for Dynamic Early-exiting Networks, in ECCV, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer q2CX, we thank you for your time and helpful comments, and address your two key concerns in the following. \n\n> The main weakness of this paper is under the strong assumption that all exiting thresholds are set to the same values. I admit that relaxing this assumption by adopting RC techniques for a high-dimensional threshold can lead to some difficulties in theoretical analysis, while using the same threshold for all exits may harm the overall performance of EENN.\n\n> The applicability of the proposed RC method can be impeded if the prediction performance is unsatisfactory when using the same exiting threshold for every exit classifiers, even though it has a lower risk in the prediction.\n\nPlease consider our response under point (1) in the general rebuttal to this question, and kindly provide additional questions in case you feel it has not been sufficiently addressed.\n\n> Only empirical test risk is used in all experiments. The general performance of EENN models in different tasks is also an important evaluation metric for EENN.\n\n> I think more evaluation metrics, such as test accuracy for classification tasks, or the IOU for segmentation tasks, should be included to show the effectiveness of the proposed method.\n\n> However, the real performance of the evaluated EENNs should also be attached in the manuscript.\n\nThanks for raising these points as they made us realise there is some misunderstanding about the interpretation of our proposed risks, and their relationship to standard performance metrics. As we explicitly state in L164-168 and seen from the equations in §3.2, our risks are formulated as the mean relative performance difference between full (i.e., last-layer exit) and early-exit model, and provide a generic template in which we may plug in different labels, model outputs, confidence measures, and loss functions.\n\nWe explore a wide range of loss functions across tasks for prediction control, and in particular **use standard task-specific performance losses** (and precisely the ones you mention): 0-1 loss for classification, mIOU loss and miscoverage loss for segmentation, ROUGE-L loss and Token-F1 loss for language modelling, and LPIPS loss for image generation. Thus, the empirical test risk equates the difference in test accuracy between the full and early-exit model for our classification experiments in §5.1, the difference in mIOU between the full and early-exit model for some of our segmentation experiments in §5.2, etc., providing interpretable assessments of the EENN’s performance based on risk control.\n\nWe would also emphasis here that our risk control approach is entirely *post-hoc*. As such, **we do not change the performance of any underlying model but work with existing pretrained EENNs**, and thus our model performances are equivalent to the numbers provided by the original authors. Thus, we consider reporting the commonly seen performance vs. FLOP curves found in early-exit papers as redundant for our work, since we do not modify the underlying models (except for switching to a single exit threshold for some models like MSDNet, see our point (1) in meta rebuttal for a longer discussion on this).\n\nPerhaps it's also helpful to add here that we may interpret our risk control threshold selection as a framework to navigate the performance vs. FLOP curve based on the user’s performance requirements, i.e. we find the point on the curve such that the EENN’s test performance is guaranteed to be at most $\\epsilon$ (e.g., at most 5%) worse than the full model (i.e., last-layer exit). As such, evaluating against different risk levels $\\epsilon$ is the appropriate way to benchmark our procedure, and provided “test risk” figures such as Fig. 2 and 3 in the paper are common in the risk control domain (see, e.g., Fig. B.1 in [2]). Moreover, the fact that the test risk curves are close to the diagonal (e.g., see Figure 3, upper row) is encouraging, as it suggests that with our framework, there is very little efficiency cost for the added safety of having risk-control guarantees.\n\nWe thank you for raising these points, highlighting the need to better explain the provided test risk figures. We attempted to do so in §5 (L272-278), but shall expand upon it for the camera-ready version. Likewise, we will aim to clarify that model performance is not affected by our post-hoc risk control procedure.\n\n[2] Schuster, T., et al. Confident adaptive language modeling. *NeurIPS 2022*\n\n> It is also suggested to review the related important works in early exiting neural network areas, such as: [1] Improved techniques for training adaptive deep networks. in ICCV, 2019. [2] Resolution Adaptive Networks for Efficient Inference, in CVPR, 2020. [3] Learning to Weight Samples for Dynamic Early-exiting Networks, in ECCV, 2022.\n\nWe appreciate the additional references. We would politely point out that reference [3] is already contained in our image classification experiments, where we refer to it as L2W-DEN (see Figure 3 in the paper). Following your advice, **we have also added the suggested reference [1] to our experiments**. We refer to the model as IMTA and observe similar results to the other models, see Figure A of the attached rebuttal PDF. We will incorporate this model alongside your other suggested reference [2] (RANet) for the camera-ready version (the ImageNet training of the RANet unfortunately didn't finish before the rebuttal deadline), either into the main figure or into an appendix figure to preserve readability.\n\nWe thank you again for your comments and would appreciate you considering raising your score if you believe they have been addressed sufficiently.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their efforts, time and comments, which are greatly appreciated. 

We are glad that you found the work **tackles an interesting and important problem** (qAvq: “very interesting and timely problem”; c7rs: “addresses an important problem”; LhAs: “interesting and important challenge”), proposes a **novel and well-grounded approach** (c7rs: “propose a novel approach”; LhAs: “framework is well-developed”, “integration of risk control into EENNs is novel”; q2CX: “theoretical analysis seems reasonable”) and appreciate the **extensive experimental validation** (qAvq: "experimentally demonstrated on a wide range of predictive ... and generative tasks"; c7rs: “empirical validation … on a range of vision and language tasks”; LhAs: "extensive experimental evidence across multiple domains"; q2CX: "The experiments on different vision tasks can be strong evidence to show the effectiveness…”). Reviewers also appreciated the method's post-hoc nature (qAvq: “does not add any computational overhead”), uncertainty aspect (c7rs: “consider both prediction quality and uncertainty estimation”), and the paper's writing style (qAvq: ”The paper is well-written … and a nice formulation.”).

We next address and clarify a key point raised in the reviews.

 **(1) Reliance on a single exit threshold (qAvq, q2CX)**

We agree with the reviewers that our reliance on a one-dimensional shared threshold among exit layers can be considered a limitation, as we highlight ourselves in §6. However, we think that **addressing the simpler, single-threshold case first provides a necessary stepping stone for extending our approach** to scenarios with higher-dimensional thresholds for EENNs. Note that for higher-dimensional cases new challenges arise both in terms of theoretical aspects (e.g., defining monotonicity requirements) and practical ones (e.g., substantially larger search spaces). As we highlight in §6, combining our work with some of the recently proposed risk control extensions for high-dimensional thresholds [1] could help address those challenges, and we think this is a very promising future direction.

Morever, we note that **single exit thresholds are quite common in the early-exit literature** [2, 3, 4, 5, 6, 7, 8, 9, 10], most likely due to the complications of designing principled threshold selection mechanisms for higher-dimensional cases. For instance, in [11] they resort to an (inefficient) heuristic based on evaluating randomly sampled threshold vectors on hold-out data. While we agree that the use of exit-specific thresholds might lead to even faster early-exit models, our work provides a principled selection mechanism for one-dimensional thresholds first, with hopes of extending to more complex higher dimensions in the future. Also, we are not aware of any work that studies in detail the potential inefficiencies induced by relying on a single threshold vs. multiple ones in early-exit architectures and we agree that this is an interesting an important direction for future work.

Lastly, as a workaround, one could **reduce the multi-dimensional problem to a single threshold** selection by defining the so-called *threshold function*. As an example, we would point to our language modeling experiment (§5.3 and L1120-1122 in Appendix C.3) based on the CALM model [2]. While in CALM the threshold is the same across exits for a given token, it changes between different tokens in a sequence (concretely, the threshold function is given by $f(\lambda,t) = \text{clip}_{[0,1]}(0.9 \cdot \lambda + 0.1 \cdot e^{-\tau \cdot t / N})$, where $\tau, N$ are fixed values, $t$ is the token index and $\lambda$ is the parameter to tune via risk control). Note that our current framework still supports such a dynamic threshold by performing risk control not on the threshold itself but on the parameter $\lambda$ of the threshold function. Similar ideas could also be used to support dynamic thresholds not only across tokens but also across layers/exits, allowing for the flexibility of exit-specific thresholds while keeping the risk control computations tractable.

We thank reviewers for pointing out this important aspect, and will add these details to the camera-ready version to better clarify our threshold assumption.

We further address each reviewers' concerns and questions in our individual responses.

**References**

[1] Teneggi, J., et al. How to trust your diffusion model: A convex optimization approach to conformal risk control. *ICML 2023*

[2] Schuster, T., et al. Confident adaptive language modeling. *NeurIPS 2022*

[3] Tang, S., et al. Deediff: Dynamic uncertainty-aware early exiting for accelerating diffusion model generation. *arXiv preprint 2023*

[4] Liu, Z., et al. Anytime dense prediction with confidence adaptivity. *ICLR 2022*

[5] Kaya, Y., et al. Shallow-deep networks: Understanding and mitigating network overthinking. *ICML 2019*

[6] Wołczyk, M., et al. Zero time waste: Recycling predictions in early exit neural networks. *NeurIPS 2022*

[7] Zhou, W., et al. Bert loses patience: Fast and robust inference with early exit. *NeurIPS 2020*

[8] Schwartz, R., et al. The right tool for the job: Matching model and instance complexities. *ACL 2020*

[9] Xin, J., et al. DeeBERT: Dynamic early exiting for accelerating BERT inference. *ACL 2020*

[10] Xin, J., et al. BERxiT: Early exiting for BERT with better fine-tuning and extension to regression. *EACL 2021*

[11] Elbayad, M., et al. Depth-adaptive transformer. *ICLR 2020*

Meta-review: ### Summary
This paper tries to improve the efficiency of deep models by exploring a type of Early-Exit Neural Network (EENN). EENNs accelerate inference by allowing intermediate layers to exit and produce a prediction early. However, this paper improves them by using approaches inspired by leveraging a method called as Risk Control (RC.)  RC manually sets the confidence thresholds for early exiting. The paper presents promising experimental results on vision and language tasks.

### Decision

The paper is well-written and clear. The approach is interesting, and the problem is important and potentially impactful. There were several criticisms directed toward the paper during the rebuttal period. However, the authors did a good job replying to them during the rebuttal period. The authors provided clarifications for many of the questions raised by the reviewers, and the proposed approach has merit on its own, as it studies a very interesting and often overlooked aspect of EE models and offers a holistic approach that demonstrates its effectiveness across models and tasks. Thus, I would recommend this paper for acceptance. 

However, I recommend that the authors to read the rebuttals carefully and address the reviewers' concerns in the camera-ready of the paper.
Decision: Accept (poster)