Efficient Inference With Model Cascades
Authors: Luzian Lebovitz
Lukas Cavigelli
Michele Magno
Lorenz K Muller
Venue: Accepted by TMLR ()

Abstract: State-of-the-art deep learning models are becoming ever larger. However, many practical applications are constrained by the cost of inference. Cascades of pretrained models with conditional execution address these requirements based on the intuition that some inputs are easy enough that they can be processed correctly by a smaller model allowing for an early exit. If the smaller model is not sufficiently confident in its prediction, the input is passed on to a larger model. The selection of the confidence threshold allows to trade off computational cost against accuracy. In this work we explore the effective design of model cascades, thoroughly evaluate the impact on the accuracy-efficiency trade-off, and provide a reproducible state-of-the-art baseline that is currently missing for related research. We demonstrate that model cascades dominate the ImageNet Pareto front already with 2-model cascades, achieving an average reduction in compute effort at equal accuracy of almost 3.1x above 86% and more than 1.9x between 80% and 86% top-1 accuracy, while 3-model cascades achieve 4.4x above 87% accuracy. We confirm wider applicability and effectiveness of the method on the GLUE benchmark. We release the code to reproduce our experiments in the supplementary material and use only publicly available pretrained models and datasets.

Published in Transactions on Machine Learning Research (08/2023)
Efficient Inference With Model Cascades
Luzian Lebovitz
luzion.kcboviizt0ppp ec. ee.etha.ch
Department of Electrical Engineering Information Technology ETH Zurich
Lukas Cavigelli
tukas, tukas. wwl.ssccriigll@@@uwww...comm
Computing Systems Lab
Huawei Technologies
Michele Magno
michele. michele. ee..th.cc
Department of Electrical Engineering Information Technology
ETH Zurich
Lorenz Müller
lorene.maeller@@uaaass.com
Computing Systems Lab
Huowei Technologies
Reviewed on OpenReview: https: // optp//veee.r.vee...mm net/ forum? id= obB415rg8g
Abstract
State-of-the-art deep learning models are becoming ever larger. However, many practical applications are constrained by the cost of inference. Cascades of pretrained models with conditional execution address these requirements based on the intuition that some inputs easy enough that they can be processed correctly by smaller model allowing for an early exit. If the smaller model is not sufficiently confident in its prediction, the input is passed on to a larger model. The selection of the confidence threshold allows to trade off computational cost against accuracy. In this work we explore the effective design of model cascades, thoroughly evaluate the impact on the accuracy-efficienc trade-off, and provide a reproducible statc-of- stato-of-the-art thc-art bascline that currently missing for related rescarch. We demonstrate that model cascades dominate the ImageNet Pareto front already with 2-model cascades, achieving 3.1x x above 86% and more than 1.9x between 80% and 86% top-1 accuracy, while 3-model cascades achieve 4.4x above 87% accuracy. We confirm wider applicability and effectiveness of the method on the GLUF benchmark. We release the code to reproduce our experiments in the supplementary material and use only publicly available pretrained models and datasets.
average reduction in compute effort at equal accuracy of almost
1 Introduction
The trade-off between accuracy and efficiency fundamentally important to deep learning. While state-of- the-art results are achieved by models of ever-growing size, practical applications are constrained by the cost of inference. Reducing the energy consumption is critical for energy-limited systems like wcarable and mobile devices. For inference at large scale such in data centers, minimizing the compute resource requirements is important economically and environmentally, and enables wider adoption of beneficial new technology.
Early exit model cascades are a cheap, simple, and effective approach to improve the accuracy-efficinncy trade-off. The cost. of training and inference is dominated by the largest model in the cascade, which means that the cost added by smaller models usually negligible. Pretrained models can be used as is or finetuned. When the input is processed by at least 2 models the cascade, the predictions can be ensembled. Gontijo- Lopes al. (2022) show that ensembles are more accurate when the models are more diverse due to the lower correlation, which also be leveraged by cascades. Cascading complementary approach and

---

Published in Transactions on Machine Learning Research (08/2023)
can be combined with many other approaches for increasing efficiency, particularly methods that improve the individual models, such as better architecture and training or model compression. Importantly, model cascades can be applied at any scale and tend to be more effective for larger models as can be seen from our experiments.
Although early exit model cascades are effective, little prior work on them has been published. In Section 2 we identify a lack of comparison between different cascading methods, knowledge gaps about when cascades perform well, and limited evaluation. Furthermore, different works use different datasets and metrics. Effi cacy is commonly measured by comparison with models trained for the experiments. Code and models are usually not released, and specifies about model size are not always provided. This makes comprehensive comparison between different works infeasible.
We address these points by conducting a comprehensive evaluation and creating a solid foundation for future work. The choice of bascline is critical to the empirical evaluation as it determines the mcasured efficacy. To ensure valid and relevant results, we compare with Parelo fronts of state-of-the-arl pretrained models, which represent the best achievable trade-off between accuracy and efficiency. HaatipllyAAcuumultte (MAC) operations are a widely used measure of computation in deep learning because the computation in models is typically dominated by matrix multiplications, which consist entirely of MAC operations. Furthermore, MAC count enables hardware agnostic benchmarking and gives an indication of energy consumption and to a lesser extent inference speed. We therefore focus on MAC count as measure efficiency in our evaluation. No training is done in our experiments. Instead, we use publicly available datasets and pretrained models. We release our code make our work easily reproducible. The code designed so that the results can be updated with new state-of-the-art models.
Most of our experiments are conducted on ImageNet [Russakovsky et al. 2015 due to its significance and the large amount of pretrained models available from PyTorrc Image Models (Wightman, 2019). We explore how to build cascades effectively and compare many different cascading methods across continuous Pareto fronts multiple settings using a diverse selection models for a thorough and reliable evaluation. We then test their wider applicability on text classification tasks from the GLUE (Wang et al. 2019) benchmark using models from Hugging Face (Wolf et al., 2020). Our contributions are:
We demonstrate that already 2-model cascades dominate the ImageNet Pareto front.
We provide insight into how to construct model cascades by investigating how choose models combine, what confidence metric to use and whether to aggregate predictions.
We investigate the impact of a distribution shift on cascades.
We provide an easily reproducible baseline for future research.
2 Related Work
Accuracy-efficiency trade-off Improving the ecluracyyeffcciency trade-off' is a central objective for a wide breadth of research with many different approaches. MobileNetV3 (Howard et al. 2019 and EfficientNet (Tan & Le. 2019) represent how architectures have become highly optimized. Model efficiency can be improved further by applying compression techniques like quantization and pruning (Han al. 2016). Once- for-all (Cai et al.) 2020) utilizes progressive shrinking of input resolution, kernel size, network width and depth together with knowledge distillation to obtain more efficient models than conventional neural architecture scarch. Training is very important with recent advances most notably through data augmentation (Shorten & Khoshgoftaar 2019) and pretraining on more data (Ridnik et al. 2021), which is enabled further by self- supervised (Devlin et al.) 2019) and seni-super (Pham et al.] 2021] methods. Many other procedures exist such as model soups (Wortsman et al. 2022), which finc-tunes model with multiple hyperparameter configurations and averages the weights. We focus on carly exiting from cascade of models, which is a complementary approach that is comparatively simple yet effective.
2

---

Published in Transactions on Machine Learning Research (08/2023)
Dynamic models More closely related to our work are dynamic neural networks (Han et al. 2021 Xu &
McAuley 2023). An example of dynamic depth is SkipNet (Wang et al. 2018b), which adds gating modules and a learned skipping policy to skip network layers based on the input. BranchyNet (Téerapittayanon et al,
2016) adds branches to the original net for early evaluation and exits when prediction entropy is below a threshold. Similarly, Shallow-Deep Networks {Kaya el al., 2019) insert early classifiers. |Wotzyyk el al. (202T) improve carly exiting from within a model by recycling predictions of carlier classifiers. However, Huang
et al. (018)) describe how early classifiers lack coarse features, and training the model for early classification
can lower the accuracy of later classifiers Furthermore, modern convolutional neural networks are highly optimized and their depth, width, and resolution are carefully scaled, gradually lowering the resolution and
increasing the number of channels as features become more complex (Tan & Le 2019). For early classification outputs, this optimization and scaling is disrupted, which results in a lower accuracy than a separate model at equal compute effort can achieve. We demonstrate experimentally that a smaller accuracy difference between classifiers allows for more frequent carly exiting. This motivates using a separate model for carly classification, which enables us to increase the early exit rate at the cost of small overhead when forwarding to the larger model since features are no longer shared between classifiers However, early exiting from within single model and between cascaded models are two separate and complementary approaches that can be combined (Bolukbasi et al. 2017
Early exit model cascades Cascades are commonly used in machine learning and have been popular- ized by influential works such as Viola & Jones (2001). We focus on specific type of cascade that uses multiple models in sequence with the capability to exit early if confident enough in a prediction. Park et al.) (2015) propose the big/LITTLE architecture, which is a cascade of a small and large model with static or dynamic softmax margin threshold as the exit condition to reduce energy consumption. Waan et al. (0118a) demonstrate reduced computational cost by cascading up to 3 models. They compare different classifiers for the objective of achieving the desired accuracy, which includes a trained decision network, cost-oblivious grid search for both entropy and maximum softmax thresholds, and cost-aware learned entropy threshold. They find that only the last three methods manage to satisfy the accuracy objective, that entropy is better confidence metric than maximum softmax, and that the cost-aware method performs better. Guan et al. (2018) Irain agent lo exit belween models to reduce the computational cost. Similarly, BBolkbasi et al. (2017) present trained policies to exit early between as well as within models for a trade-off between accuracy and inference time. Streeter (2018) proposes an algorithm to automatically construct efficient cascades from a selection of available models. Inoue (2019) shows that exiting early from an ensemble of 10 or 20 models can greatly reduce the computational cost. while preserving most of the static ensemble's accuracy. Tnoue averages the softmax to ensemble the predictions and exit based on a confidence interval condition. [Wang] et al. (2022) cascade up to 4 models of the same architecture families such as EfficientNet, ensemble by averaging the logits, use maximum softmax as confidence metric, and optimize for accuracy and computa- tional cost with exhaustive scarch. They also demonstrate the efficacy of self-cascading by varying the input resolution for the same model.
From prior work, we notice a lack of comparison between different cascading methods, knowledge gaps about when cascades perform well, and limitod evaluation. We address this by comparing many different cascading methods in multiple settings using a diverse selection of pretrained models for thorough and reliable evaluation. We investigate when cascades work well and unlike prior work we evaluate improvement across continuous Pareto fronts. No training is done in our work. Instead, we use publicly available statc-of-the-al models and release the code to reproduce our results, thereby providing a strong bascline that is currently missing for future related research.
3 Method
The idea behind early exit cascades is that inference can be done on a smaller model first and if confident in its prediction we can save time and computation by exiting early and skipping the larger model. A cascade contains at least pretrained models. A condition is needed to decide whether to exit carly. This condition uses a confidence score computed from the model prediction and a threshold set in advance that needs to be satisfied. Algorithm ] shows the implementation of the cascading method we found to be most effective.
3

---

Published in Transactions on Machine Learning Research (08/2023)
Algorithm 1 Early exit model cascade with maximum softmax confidence metric and no ensembling
Require: input tensor X, models (M1, M,) ordered by increasing cost, thresholds (fI,..., In in-1), for l.....n n do
2; (X(
P: softmax(z:)
if i = n or max(p:) 2 f then
return arg max
Di cascade returns predicted class
Cost Let C2 be the cost of model M, which represents metrics we want to improve, such as the number of Multiply-Accumllat (MAC) operations or time of inference. We order the models within a cascade so that
C, < Ci+1; meaning the cheaper model makes a prediction first. When there are n models in the cascade, there are n possible costs when inferring single imput. In the case n 2, when an early exit is made, the cost is C1, otherwise, the cost is Ci + C2. They represent the lower and upper bound for the cascade. Assuming we exit carly after the first model at rate E1, the cost becomes C1 + (1 E1)C2 average. We want to evaluate how elficient. cascades are by comparing them to a baseline at equal accuracy. The improvement factor I by which we reduce the cost is / = Co/(Oi (1)), where C. is the baseline cost and E is set so that the cascade matches the baseline accuracy. Therefore, the improvement we can achiove is limited by the early exit rate we can reach, and the cost difference between the models, where larger is better for both.
Early exit condition We need a condition to decide when to exit. early. Such a condition typically evaluated by obtaining confidence score from the model prediction and comparing it threshold t. This is done after each model in the cascade except the last, resulting in n-1 thresholds. For classification tasks, the model output for input tensor X is usually logits vector x and the predicted class is arg max(2). We can use the softmax function
e
softmax(21)
=
(1)
Σ/=1
e²
to transform the logits into pseudo-probabllities p for all K classos. One confidence metric the maximum softmax, which we can use as the probability that the prediction is correct. We exit early max (p) l. Various other conditions exist in related literature (Han al. 2021 Xu & McAuley 2023) and the entropy threshold appears to be the most common. There, after applying the softmax function, we can compute the Shannon entropy
H(P) - pa log(p:).
Σ
(2)
A lower entropy represents higher confidence in the prediction and exit early when H(p) < t. Other confidence metrics are the logits margin (Streeter 2018) and the softmax margin (Park et al. 2015). which are the largest logit or softmax minus the second largest respectively. They represent. how much more confident the model is in its first prediction and we exit if the margin is above or equal to a threshold.
Ensemble When the cascade does not exit after the first model, we have multiple predictions that we can ensemble in the hope of improving performance. Typical ways to ensemble models are plurality vote (Hansen& Salamon 1990p or averaging of the outputs (Bishop et al. 1995). To mean ensemble the outputs, we compute the arithmetic mean of softmax or logits. An intuitive alternative to compare the prediction confidence of all models and choose the most confident, which we call comparison ensemble. Weighting calibration can be used when model confidences are mismatched so that the most overconfident model does not dominate the ensemble. Guo el al. (2017) demonstrate that temperature scaling is effective calibration tool and Ashukha et al. (2020) recommend it for ensembles. Temperature scaling is achieved by dividing the logits by a temperature factor T so that the calibrated probabilities become
e+
p
(3)
Σ
K
4

---

Published in Transactions on Machine Learning Research (08/2023)
where T is set to minimize the negative log-likelihodd on validation set. We examine the calibration of models used in our ImageNet experiments in Appendix F
Evaluation The performance of a cascade can be evaluated on a validation set. First, the logits are obtained for each model with a batched forward pass on the entire validation set. From the logits, the confidence score of each input can be calculated according to the used metric. For 2-model cascades, a k xx3 3 array is constructed containing the confidence score for the first model and prediction correctness for both models for cach of the examples in the validation set. The array is sorted along the dimension according to confidence score from worst to best. To oblain the cascade accuracy, the cunulative sum of correct examples from the second model and the reversed cumulative sum for the first model are added together and divided by k. The cumulative sums represent the number of correct predictions for each model. Now, both the threshold boundaries and cascade accuracy are known for all k + 1 possible early exit rates. The average inference cost can be obtained with linear interpolation from C1 at 100% early exit to C1 + C2 at 0% early exit.
The time complexity for all operations Ok) except for the sort, which is Olllog)), Therefore, evaluation is very cheap for 2-model cascades. The process is dominated by the model inference on the validation set. However, the complexity increases exponentially more models are added to the cascade and becomes Ook k) for models with n-1 thresholds. Evaluation with more models can be done by sweeping the thresholds from the last to the first. Heuristic search can be used to reduce the complexity. Alternatively, in order to avoid the exponentially increasing complexity altogether, all thresholds can be set to be the same. However, this would negatively impact performance. When ensembling is used, this basic recipe needs to be adjusted by using the ensemble prediction correctness and confidence score instead.
We evaluate Parcto improvement by comparing the cost of cascade to achieve the same accuracy baseline Pareto front of single models. We conduct Pareto optimal linear interpolation between base models to enable continuous comparison instead of just at specific models. This linear interpolation represents the trade-off achieved when randomly selecting one of the two models with varying probabilities.
Threshold selection A fixed threshold can be selected for testing and deployment based on the evaluation. This is done once ahead of time. There is a constant overhead during deployment to compute the confidence score and compare to the threshold, which is extremely small compared to the cost of the models. The selection process depends on the desired trado-off' between efficiency and accuracy. For example, one may select the threshold which achieves the greatest Pareto improvement, or the greatest reduction in inference cost while preserving the accuracy of the best model. One concern of a fixed threshold is that the validation data may not align with the real world data encountered during deployment. To address this, analyze the impact of distribution shifts at the end of Section for example by defining a computational budget that must be satisfied (Huang et al. 2018
4.1 Alternatively, dynamic thresholds can be used,
4 Experiments
In order to evaluate the impact of early exit model cascades, we verify whether they reliably outperform state-of-the-art single models across entire Pareto fronts on different tasks. We rigorously compare many different cascading methods to determine the most effective. To understand when cascades work well, we figure out how properties such the size and accuracy of cascaded models affect performance. We begin by conducting a set of experiments on ImageNet (Russakovsky et al. 2015) (Section 4.1 duc to its relevance and evaluate robustness to distribution shifts on ImageNet (Recht et al. 2019 As step to indicate the general utility of model cascades, we demonstrate Pareto improvement on text classification tasks from the GLUE (Wang et al.] 2019h benchmark (Section 4.2). All pretrained models used for image and text classification are publicly available from PyTorch Image Models (Wightman) 2019) and Hugging Face (Wolf et al. 2020) respectively. The code to reproduce these experiments is available in the supplementary material.
5

---

Published in Transactions on Machine Learning Research (08/2023)
4.1 Image classification ImageNet
We select from 668 pretrained models with labulated accuracy for the ImageNet validation set from PyTorch Image Models [Wightman 2019). For the computational cost of models, we obtain MAC count with the fucore library. To acquire the time cost we use benchmark numbers in inferred samples per second on an RTX 3090 with NHWC data format and automatic mixed precision from Wightman (2019). From this, we can establish baseline Pareto fronts for both MAC and time cost. We infer the ImageNet validation set for models the Pareto fronts to confirm their accuracy and obtain logits, from which we compute the prediction correctness and confidence score for every image. From these, can efficiently compute exact accuracy, threshold, and average inference cost at all possible early exit rates for every cascade. Threshold selection can then be made based on the desired trade-off. For example, we choose the threshold at the point of maximal validation Pareto improvement. for our ImageNet test server submissions.
As a strong and contimuous baseline, we use the Pareto front established by linear interpolation between pretrained model pairs, which represents the trade-off achieved when randomly selecting one of the two models with varying probabilities. No interpolation done for cascades in order to have direct comparison between individual cascades and random model selection. We evaluate improvement by cascade Pareto fronts as factor by which average inference cost in MAC or time is reduced to achieve the same accuracy. The average cost represents the total cost for the entire validation set at specific thresholds divided by the number of images.
90
90
ImageNet Validation Top-1 %
85
85
80
80
75
75
70
70
Pretrained Models
65
Baseline Pazeto
Suftmax Cascade Pareto
Tested Cascades
+
ImageNet Tests
10s
10P
1030
10n
10t2
4
Average Inference MAC
Improvement Factor
Figure 1: The maximum softmax confidence metric without calibration and ensembling is one of the simplest yet most effective for cascading and achieves Pareto improvement across most of the MAC range with 2- model cascades. The pretrained models show that the accuracy benofit from increasing model size diminishes. Cascade performance improves as the accuracy difference between models decreases compared to the MAC difference. Three points at the cascade Pareto front with locally maximal improvement were evaluated on the ImageNet test set. The results are summarized in Table [an confirm the validation results. Notable are that cascading also increases the top accuracy by 0.306% to 88.90%, and that the improvement factor drops to almost 1 at 77.92% accuracy due to outlier model, a MobileNet V3-Larg with much higher accuracy than similarly sized models, which was achieved by pretraining on ImageNet-22K (Ridnik et al. 2021
For the tradc-off between accuracy and MAC count, the bascline Pareto front consists of 27 models. For 2-model cascades, there are (27) 351 possible combinations. Figure shows the Pareto front and im- provement factor for the cascading method which uses maximum softmax confidence without calibration or ensembling. The cascades achieve improvement along most of the Pareto front with reduction in compu- tational cost by factor of up to 3.83 for the same accuracy. This achieved by the cascade containing the models EfficcentNeet B4 NS Taan & Le. 2019), convolutional neural network pretrained on 300M unlabeled

---

Published in Transactions on Machine Learning Research (08/2023)
Table 1: Three points at the locally maximal improvement from the softmax 2-model cascade Pareto front were evaluated the ImageNet test set as shown in Figure 1 They were chosen the strongest points the cascade Pareto front to lest against validation set overfitting, Each point corresponds to cascade with specific threshold. Values are rounded to 4 significant digits.
Cascade Models
Validation
Test
Softmax
First
Second
Top-1
MAC
Top-1 1
MAC 4
Threshold
MN3-S 0.75 (Howard al. 2019
MN3-L Riddiik al.
2021
75.88%
136.2M
75.91%
135.9M
0.4551
MN3-L Ridnik et al. 2021
EN B3 NS Xie et al.
212U
83.39%
732.4M
83.13%
735.2M
0.7112
EN B4 NS Xie et a..
2020
2ETT
Bao et
2022
87.32%
14.39G
87.21%
14.38G
0.5658
images from the JFT dataset (Hinton et al. transformer pretrained on ImageNet- 222- which shows the benefit of model diversity. Model families which contribute to the cascade Parcto front in desconding order of importance are EfficientNet NoisyStudent (Xie
2015 Chollei 2017) and BEiT [aao et al. 2022 a vision
jet al.
2020), MobileNed (Howard et al.) 2019 Ridnik al. 2021), BETT (Bao et al. 2022 TinyNet (Han
jet al.
2020b), GhostNet (Han al. 2020a) and LeViT (Graham et al. 2021).
To test against validation set overfitting, points of the cascade Pareto front with locally maximal improvement were evaluated the ImageNet test set. As shown in Table [ the test set performance is close enough to confirm the validation set results. However, test accuracy is slightly lower average. Possible explanations are that the optimal points are shifted for the test set, that the cascade Pareto front envelops noise, and that the pretrained models themselves are slightly overfitted to the validation set. To investigate the last point, the pretrained EffficientNet. B4 NoisyStudent was evaluated on the ImageNet test set and achieved a top-1 accuracy 0.04% lower than on the validation set, which accounts for almost half of the 0.09% difference for the 3rd test point.
67.3
Jucuncy Izup
Average Imnptoveneaa Over Pactoo
Moldle Top-1 Difforence:
0.35
Top-1 Differrmmnt
0.25
20
87.0
0:30
Imngeeee Validatica
86.3
0.15
0.15
86.0
ElliiiatN B4
0.30
3EIT Lanje 220
0.66
0.06
0.5
disciae Parsto
5% Relative
0.00
40
50
10
70
50
50
100
107
10'
107
10a
10a
Avotage Inferencc MAC
1e10
Early Extt: %
MAC Factor
The cascades tend to maintain the accuracy of the better model for a plateau or even exceed it before accuracy starts to break down. Middle: Each point represents the largest early exit rate before cascade accuracy drops by 5% of the top-1 difference below the better model. Right: The average Pareto improvement of all individual cascades. While lower accuracy difference between cascaded models is always desirable, there exists sweel spot for the size difference. While the length of the plateau before accuracy drops increases with smaller accuracy difference between models, for the baseline Pareto this plateau grows with the size range as the accuracy benefit of increased model size diminishes.
Figure 2: Left:
Figure 2 shows the relationship between the accuracy difference of the models in the cascade and the early exit rates achievable by cascades before accuracy starts to break down. The cascades perform better when the accuracy difference between models small while the MAC difference large. Therefore, the Parcto improvement is minimal in the lowest MAC regime and becomes larger higher MAC regimes because of the diminishing accuracy benefit. from increased model size, which be observed in Figure 1
7

---

Published in Transactions on Machine Learning Research (08/2023)
Cascading method comparison We compare the 4 confidence metrics entropy, maximum softmax, softmax margin, and logits margin both without ensembling and with comparison ensembling enabled by calibration using temperature scaling, Furthermore, using the maximum softmax confidence metric we evaluate mean ensembling for both logits and softmax, with and without temperature scaling. This results in 12 different cascading methods compared in Figure 3
From the methods without ensembling, the entropy confidence metric performs worst. Eutropy is influenced by probabilities assigned to false labels while for early exiting. Softmax margin is the only cascading method that achieves the entire Pareto froet-cxcept at the smallest model, whose computational cost cannot be reduced further with cascading. It also reaches the largest average Pareto improvement of all 12 methods with 1.502x across the entire accuracy range. This is because softmax margin dominates below 78% accuracy, which represents almost 2/3 of the accuracy range but only about 1/4 of the log-scaled MAC range. The maximum softmax metric has the sccond- largest average improvement 1.497x but shows the most MAC range dominance of all methods.
only interested in whether the predicted label is correct
improvement >1 across
When ensembling, calibration may be required, because otherwise there is guarantee that the uncalibrated confidence scores of the different models are comparable. Temperature scaling was done on the validation set, which means that the results for methods using temperature scaling represent an ideal scenario where the calibration is optimally fitted to the data. Yet, calibration proves ineffective for mean ensemble cascades and the comparison ensemble methods achieve the least improvement of all. Mean ensembling shows potential the upper end of the accuracy range. Therefore, we use the maximum softmax confidence metric, which is worse overall but better at higher accuracies than the softmax margin (see Appendix C for details). While the mean softmax method performs better overall, uncalibrated mean logits ensembling achieves the best average improvement above 86% accuracy of all cascading methods, as well as the best top accuracy of 89.03%. A possible explanation for the poor performance of ensembling is that Pareto optimal cascade scttings rely on high carly cxit rates, where only the lcast cortain predictions are forwarded. These are less useful for or may even hurt ensembling.
No Ensemble
Comparison Ensemble
Mean Ensemble
88
ImageNot Validation Top-1 %
84
82
82
52
80
80
Max Saltmax
Max Softmax
Entropy
TS Entropy
Mean Logis
Max Softmax
TS. Max Saftmax
Mean Softmax
Softmax Margin
Saftmax Margin
Mean TS Logits
Logits Margin
TS Logits Margin
Mean TS Softmax
2
3
2
Improvement Factor
Improvement Factor
Improvement Factor
Figure 3: A comparison of Pareto improvement achieved for the 12 lested cascading methods. To improve legibility, only the accuracy range above 75% is shown as improvement achieved below that point is small for all methods. The maximum softmax method from the left plotted again as a black line to have a bascline for comparison. TS stands for temperature-scaled.
8

---

Published in Transactions on Machine Learning Research (08/2023)
3-model cascades Cascading more models increases the complexity since well-suited models required for improvement and multiple thresholds need to be selected, which increases the problem dimensionalityy The ImageNet validation set consists of 50000 images, which means that there are 50001 choices per threshold ranging from always to never exiting carly. For the 27 Pareto optimal models, there are (3) 2925 possible combinations. Since compuling cascade performance is cheap, an exhaustive search still feasible for model cascades and has the advantages of simplicity and guarantee that the optimum will be found. To do so, for every model combination conduct sweep on the first threshold. We can speed up the process by only sampling every tenth sorted value for the first threshold since neighboring values are almost identical.
Interostingly, ensembling performs slightly worse relative to no ensembling for 3 instead of 2 models, and mean softmax is now superior to mean logits. The maximum softmax method achieves the largest average Pareto improvement of all cascading methods both across the entire accuracy range and above 80% only. Figure shows that there is diminished benefit. from adding a third model but clear improvement at higher accuracies. 2-model cascades offer a strong and practical bascline, but for optimal performance adding more models should be considered while keeping in mind the trade-off between diminishing returns and increased complexity, which could also harm generalizability.. To test this, evaluated the point of the 3-model maximum softmax Pareto front with the largest improvement on the ImageNet test set. This is the cascade with models EfficientNet B4 NS (Tan & Le. 2019). 2 (Bao et al. 2022), and EfficientNet L2 NS at the thresholds 4 = O. 0.6299 and l2 = 0.4905. While test accuracy at 87.86% is only 0.08% lower than on the validation set, the average MAC count is also 1.1% larger at 26.32G. This indicates a slightly worse generalization than the 3rd test result in Table 1 for cascade with only the first models.
90
90
ImageNet Validation Top-1 %
80
75
75
Pretrained Models
70
Baseline Pareto
70
2-model Max Softmax
65
3-model Max Softmax
3-model Mean Logits
60
+
Wang et al. Base
60
Wang et al. Committee
10a
10a
10
10
Average Inference MAC
Improvement Factor
Figure 4: 3-model cascades start to outperform 2-model cascades at higher MAC regimes. Left: We compare our results to Wang al. (2022). who use the mean logits ensemble method. Right: The maximum softmax method achieves the best performance of all cascading methods and outperforms the mean logits method
used by Wang et al. 2022 across most of the Parcto front.
Wolczyk et al. [2021
report results for exiting early from within model ImageNet. They compare three different methods for adding multiple early classifiers to pretrained ResNet-50 (He et al. 2016) Shallow- Deep Networks Kaya et al. 2019). Patience-based Early Exit (Zhou et al.] (020)), and Zero Time Waste (Wotezyk et al. 2021). For an average reduction in inference MAC by 25%, the accuracy drops by 6.1%, 10.4%, and 5.7% for the respective methods. Accuracy degrades more rapidly as the average inference cost is reduced further. The inference cost of a ResNet-50 is 4.111 GMAC. At this point of our bascline Parcto front we obtain 59% reduction in inference cost while achieving equal accuracy for 3-model maximum softmax cascades.
9

---

Published in Transactions on Machine Learning Research (08/2023)
Inference time Radosavovic et al. (2020) show that the mumber of MAC operations is imperfect predictor for GPU inference speed. Therefore, we also evaluate cascade performance for the trade-off bet ween accuracy and speed measured in average time per inference, Figure 5 shows that 2-model cascades already achiove improvement everywhere except near the outlying LeViT-128S (Graham et al. 2021 As can be seen in greater detail Appendix [ maximumn softmax again performs best of all tested cascading methods, which demonstrates its reliability and consistency. Notably, it outperforms mean ensemble methods even above 86% accuracy for inference speed, achieving an average improvement of 2.618x compared to 2.203x by the mean logits cascading method. 'he baseline Pareto front consists of 25 models. There is a large model overlap between MAC and speed Pareto fronts. The main difference is that Efficient (Tan & Le. 2019 Xie et al. 2020) and the related NenyNet (Han et al. 2020b) architecture families perform worse for perform better.
inference speed while LeViT (Graham et al. 2021 and ConvNeXi
Liu et al.
2022
90
90
ImageNet Validation Top-1 %
85
80
75
70
Prefrained Models
65
Baseline Pareto
65
Max Softmax
Softmax Margin
60
60
Mean Logits
10-6
10
10-3
10-2
10-1
3
Average Seconds/ IInference
Improvement Factor
Figure 5: Inference speed is measured in average time per image on an RTX 3090 GPU. The images are batched with a batch size of 1024 or smaller when necessary to fit the GPU memory. Therefore, the actual latency is larger. Less Pareto improvement is achieved than for MAC count because the time cost Pareto front spans orders of magnitude instcad of 4. The 2-model cascades perform particularly well concave points of the baseline Pareto front.
Robustness to distribution shifts Training data may not align with real-world inputs. concern is that model cascades are vulnerable to distribution shifts, particularly when the data used to determine the threshold differs from the deployment data. We test robustncss by evaluating Parcto improvement on the ImageNet V2 MalchedFrequency dataset (Recht al. 2019h for identical cascades and threshold ranges that make up the original ImageNet cascade Pareto front for the maximum softmax confidence metric. Figure shows that while the distribution shift harms cascade. performance, they remain beneficial and achieve improveentt across most of the Pareto front. Two contributing factors that decrease cascade performance are that model accuracies on ImageNet are 7.2 - 13.8% lower and accuracy differences between the models are larger with the overall accuracy range increasing by 5.7%. Furthermore, model accuracy is no longer monotonically increasing with size. Additional analysis, which shows that the distribution shifted ImageNet cascade Pareto front resembles the new ImageNettV2 cascade Pareto front, can be seen in Appendix D
4.2
Text classification on GLUE benchmark
Early exit model cascades are effective for image classification with both convolutional neural networks and vision transformers. We expect that the cascades also work other domains where prediction confidence or quality can be quantified, Indeed, beneficial early exiting has been demonstrated for image segmentation (Wang et al. 2022 and rein reinforcement learning (Wotczyk et al. 2021). As a first. step to test that the identical
10

---

Published in Transactions on Machine Learning Research (08/2023)
90
90
Top-1 %
Pretrained Models ImageNet
60
Baseline Pareto ImageNet
60
Softnax Casvade Purelo
Protrained Models Ie2
Baseline Pareto Netge
Distribution Shiftod Cascade Pareto
10
10a
10
10
10
Average Inference MAC
Improverennt Factor
Figure 6: Robustness to distribution shifts is evaluated by comparing the Pareto improvement achievod for identical cascades and threshold ranges on ImageNet and Ima
cascade structure can be used in different domain, we experiment on the text classification tasks SST-2 (Socher et al. 2013). MRPC (Dolan & Brockett 2005). QNLI (Rajpurkar et al. 2016 and QQP (Chen et al. 2018) from the GLUE (Wang et al. 2019)) benchmark. The results are shown in Figure and confirm the efficacy of early exit cascades. Since the tasks are binary classification, the 4 confidence metrics entropy, maximum softmax, softmax margin, and logits margin result in identical order when sorting the confidence scores of all predictions, and the resulting cascades are equivalent. We searched for fine-tuned models on Hugging Face (Wolf al.) 2020 Models used include BERT (Devlin et al.
2019), RoBERTa (Liu et al.
2019), DeBERTa (He et al. 2021), ELECTRA (Clark et al. 2020), ALBERT [Lan al. 2020 DynaBERT
(Hou al.) (020)), DistiBEERT (Sanh al.) 2019), M-FAC (Frantar et al.,
2021). MiniLM (Wang et al,
2020 and XLNet Yang et al.
2019
Postarned Konkla
Canada Parto
Bodaee Partia
SST-2 Valddaiiin Ausuaay 35
XIRPE: Validaaion Astumer
MBPPCC Valiaasie
08
37
127
37'
1279
QNL1 Valdattia Aecuuter 36.
Valiiation Accuuncy
375
81
QUP
37
332
1011
1/
104
11
219
121
Aseagi Infreece MAC
Irppommeent Pantos
Asueage Informce MAG
Ieppoomeent Pactot
Averago Inference MAG
Inprommenn Partoc
Figure 7: Early exit cascades achieve Pareto improvement for text classification tasks from the GLUE benchmark despite the large accuracy differences between models. The results for the MRPC and QQP tasks are similar for both accuracy and F1 score. 18 models are used for SST-2, 16 for MRPC, 12 for QQP, and 15 for QNLI.
11

---

Published in Transactions on Machine Learning Research (08/2023)
5 Conclusion
Early exit model cascades can be used to trade off inference cost against quality, to optimize for maximum Pareto improvement, or to maintain and even exceed the performance of the largest model while lowering cost. We demonstrate the efficacy of early exit cascades on a wide selection of publicly available pretrained models for image classification on ImageNct and text classification tasks from the GLUE benchmark. We cascade Pareto optimal models independent of their architecture or training and provide a strong and reproducible baseline that currently missing for related research. To investigate the effective design of cascades, we compare various confidence metries and ensembling strategies. We find that the maximum softmax confidence metric achieves the largest improvement overall while softmax margin excels in low confidence scenarios and the commonly used entropy metric performs worst, that ensembling predictions can increase the maximum accuracy the cascade achieves but fails to improve the Pareto front, and that temperature scaling is ineffective at alleviating this. We demonstrate a relationship between the accuracy diffcrence of cascaded modcls and the achievable early exit rale as well as the desirable size difference, and show that while distribution shifts degrade cascade performance, cascades remain beneficial.
References
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, el al. A review of uncertainly quantification in deep learning: Techniques, applications and challenges. Information fusion, 76:243-291 2021.
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncer- tainty estimation and ensembling in deep learning. International Conference Learning Representations, 2020.
Hangbo Bao, Dong, Songhao Piao, and Furu Wei. BEIT: BERT pre-training of image transformers. International Conference on Leurning Representations, 2022.
Christopher M Bishop al. Neural networks for pattern recognation. Oxford university press, 1995.
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient PMLR, 2017.
inference. In International Conference on Machine Learning, pp.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. International Conference on Learning Representations, 2020.
Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. Quora question pairs, 2018.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the
IEEE conference compuler vision and pattern recognition, pp. 1251 1258, 2017.
Kevin Clark, Minh-
Minh-Thang Thang Luong, Quoe V. Le, and Christopher D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. International Conference on Learning Representations, 2020.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirec- tional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), 2019.
Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrasos. In Third International Workshop on Paraphrasing (IWP2005), 2005.
Elias Frantar, Eldar Kurtic, and Dan Alistarh. M-fac: Efficient matrix-free approximations of second-order information. Advances in Neural Information Processing Systems, 34, 2021.
12

---

Published in Transactions on Machine Learning Research (08/2023)
Jakob Gawlikowski, Cedrique Rovile Njieutchen Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, pp. 1 77, 2023.
Raphael Gontijo-Topes, Yann Dauphin, and Fkin Dogus Cubuk. No representation to rule them all: Overlapping features of training methods. International Conference Learning Representations, 2022.
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer convnct's clothing for faster inference. In Proceedings of the TEEE/CVE International Conference on Compuler Vision, pp. 12259-122000 2021.
Jiaqi Guan, Yang Liu, Qiang Liu, and Jian Peng. Energy-efficient amortized inference with cascaded deep classifiers. 27th International Joint Conference on Artificial Intelligence, 2018.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1580-1589, 2020a.
Kai Han, Yunhe Wang, Qiulin Zhang, Wei Zhang, Chunjing Xu, and Tong Zhang, Model rubik's cube: Twisting resolution, depth and width for tinynets. Advances in Neural Information Processing Systems, 33:19353-19366 2020b.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations, 2016.
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. IEEE Transactions on Pollern Analysis and Machine Intelligence, 2021.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions on pottern analysis and machine intelligence, 12(10):993-1001, 1990.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 000,778, 2016.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding- enhanced bert with discntangled attention. International Conference on Learning Representations, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. NIPS Deep Learning and Representation Learning Workshop, 2015.
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33:9782 9793, 2020.
Androw Howard, Mark Sandler, Grace Chu, Liang- Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. Proceedings of the IFFF/CVE /CVF International Conference on Computer Vision, pp. 1314-1324, 2019.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Weinberger. Multi- scale donse networks for resource efficient image classification. International Conference on Learning Representations, 2018.
Hiroshi Inoue. Adaptive ensemble prediction for deep neural networks based on confidence level. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1284-1293. PMLR, 2019.
Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. Shallow-degy deep networks: Understanding and mitigating network overthinking. In International conference machine learning, pp. PMIRR, 2019.
13

---

Published in Transactions on Machine Learning Research (08/2023)
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite bert for self-superviser learning of language representations. International Conference on Learning Representations, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoycr, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint XrX::11907.. 11692, 2019.
Zhuang Liu, Hanzi Mao, Chao Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. 2022 IEEE/CVP Conference on Computer Vision and Pollern Recognition (CVPR), pp. 11966 11976, 2022.
Eunhyeok Park, Dongyoung Kim, Soobeom Kim, Yong-Deok Kim, Gunhee Kim, Sungroh Yoon, and Sungjoo Yoo. Big/little deep neural network for ultra low power inference. In 2015 international conference on hardware/ sooftaar codesign and system synthesis (codes+ isss), pp. 122-132. IEEE, 2015.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVF Conference Computer Vision and Pallern Recognation, pp. 11557-11568, 2021.
Tija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 10428 10436, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, pp. Linguistics, 2016.
Association for Computational
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 5389-5440. PMLR, 2019.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zehnik-Manor. Imagenet-21k pretraining for the masses. In Thirty-fifth Conference on Neural Information Processing Systems Datasels and Benchmarks Track (Round 1), 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 111(3):211-252, 2015. doi: N0.1007/<11263-0155
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. NIPS Energy Efficient Machine Learning and Cognitive Computing Workshop, 2019.
Connor Shorten and Taghi M Khoshgoftaar. A survey image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christo- pher Polts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, pp. 1631-1642, 2013.
Matthew Streeter. Approximation algorithms for cascading prediction models. In International Conference on Machine Learning, pp. 4752 4760. PMLR, 2018
Mingxing Tan and Quoe Le. Efficientnet; Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105-6114. PMLR, 2019.
Surat Tecrapittayanon, Bradley McDanel, and Hsiang- Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), pp. TEFE, 2016.
14

---

Published in Transactions on Machine Learning Research (08/2023)
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In European Conference Springer, 2022.
Computer Vision, pp.
Paul Viola and Michael Jones. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE compuler sociely conference on compuler vision and pallern recognition. CVPR 2001, volume 1, pp. 1-1. leee, 2001.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. International Conference on Learning Representations, 2019.
Deng-Bao Wang, Lei Feng, and Min-Ling Zhang. Rethinking calibration of deep neural networks: Do not be afraid of overconfidence. Advances in Neural Information Processing Systems, 111188091182)) 2021.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Newral Information Processing Systems, 33:5776 5788, 2020.
Xiaolang Wang, Dan Kondratyuk, Eric Christiansen, Kris M Kitani, Yair Movshovitz-Attis, and Elad Eban. Wisdom of committees: An overlooked approach to faster and more accurate models. International Conference Learning Representations, 2022.
Xin Wang, Yujia Luo, Daniel Crankshaw, Alexey Tumanov, Fisher Yu, and Joseph E. Gonzalez. IDK cascades: Fast deep learning by learning not to overthink. Thirty- Fourth Conference on Uncertainty in Artificial Intelligence, UAI, 2018a.
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 409 424, 2018b.
Ross Wightman. PyTorch Image Models. ptttps://githute https//tttttrggggttb...o//raaaan http://tww....oo.o..t.oaooo//rne
2019.
Maciej Wolezyk, Bartosz Wójeik, Klaudia Balazy, Igor T Podolak, Jacek Tabor, Marek Smieja, and Tomasz Trzcinski. Zero time waste: Recycling predictions in early exit neural networks. Advances in Neural Information Processing Systems, 34:2516 2528, 2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julion Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-arl natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Rapha Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Yair Carmon, Ali Farhadi, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23965 23965-23998, 2022.
Qizhc Xic, Minh-Thang Luong, Eduard Hovy, and Quoc V Lc. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 10687-10698, 2020.
Canwen Xu and Julian McAuley. A survey on dynamic neural networks for natural language processing. EACL 2023 Findings, 2023.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Gen- eralized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Wangchunshu Zhou, Canwen Xu, Tao Gc, Julian McAuley, Ke Xu, and Furu Wei. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 2020.
15

---

Published in Transactions on Machine Learning Research (08/2023)
A
Additional Pareto improvement comparison data
No Ensemble
Comparison Ensemble
Mean Ensemble
86
ImageNet Validation Top-1 %
82
80
80
80
Max Softmax
Max Sollmax
7%
Enirope
78
Entropy
78
Mona Logits
Merr SoBmax
Softmusz Margia
Softmax Margin
TS Logits
76
Lugits Margin
Logits Margin
Meaa T8 Solsaaa
2
Improvement Factor
Improvement Factor
Improvement Fuctor
Figure 3-model cascade Pareto improvement for the accuracy-MAC trade-off. Maximum softmax confi- dence without ensembling is the best cascading method overall and plotted again in the middle and right black line to have baseline for comparison. The softmax margin cascading method performs best in the lower accuracy range and mean ensembling shines at the upper end of the accuracy range.
No Ensemble
Comparison Ensemble
Mean Ensemble
88
86
86
86
ImageNet Validation Top-1
84
84
82
Max Soltmax
Max Soltmax
78
Estropy
TS Ensropy
Monn Logits
Mas: Softmax
TS Max Softmax
Mewn Sofmaax
Softmax Margin
TS Seftmax Margin
Mous TS Logits
76
Logits Margin
76
TS Logits Margin
76
Menn TS Safimax
Improvement Factor
Improvement Factor
Improvement Factor
Figure 9: 2-model cascade Parcto improvement for the accuracy-time tradc-off. Again, maximum softmax confidence without ensembling is the best. cascading method of all and is plotted again in the middle and right as black line to have baseline for comparison. Notable is that mean ensembling shows worse performance for inference speed relative to the maximum softmax method. Furthermore, the mcan softmax method now outperforms the mean logits method. This indicates that cascading methods which use ensembling are less reliable.
16

---

Published in Transactions on Machine Learning Research (08/2023)
B Top-5 accuracy
97.5
97.5
ImageNet Validation Top-5.
95.0
95.0
92.5
92.5
90.0
90
Postraized Models
Boscine Pareto
85.0
Eettrpy
85.0
Max Softman
82.5
Soflamxx Sum
82.5
Logits
10P
10P
1010
10R1
1022
Average Inference MAC
Improvement Factor
Figure 10: For some applications, such as content or autocomplete recommendation, it is less important that the top-1 prediction is correct and instead it matters more that the correct prediction among the top. To investigate this scenario, we evaluate the trade-off belween top-5 accuracy and MAC count. 2-model cascades perform better at the lower range but worse at the higher range for top-5 accuracy despite the smaller accuracy differences. Here, the entropy confidence metric beats maximum softmax since takes into account more than the top-1 prediction. However, the top-5 softmax sum, which takes into account all relevant probabilities, outperforms entropy across most of the Pareto front. In attempt to improve the cascading further, we try mean ensembling. However, neither the mean logits nor the mean softmax (not pictured) method perform well outside of narrow points at the upper limit of the accuracy range.
C Confidence metric for mean ensembles
89
88
ImageNet Validation Top-1
87
85
84
Max Softmax Mean Logits
Max Softmax Mean Softmax
83
Softmax Margin Mean Logits
Softmax Margin Mean Softmax
82
Improvement Factor
A comparison of confidence metrics for mean ensembled 2-model cascades in the high accuracy regime, where they perform best. While the difference is very small, the softmax margin confidence metric is better overall than the maximum softmax for mean ensembled cascades. However, maximum softmax confidence leads to better improvement in the high accuracy regime, where mean ensembles perform best, and is therefore the metric selected.
Figure 11:
17

---

Published in Transactions on Machine Learning Research (08/2023)
D Robustness to distribution shifts
80
80
ImageNet Test Top-1 %
75
65
65
60
60
55
Pretrained Models ImageNetV2
Baseline Pareto 22
Softmax Cascade Pareto ImageNetV2
Distribution Shifted Cascade Pareto
10a
10P
10
1Q
10
Average Inference MAC
Improvement Factor
Figure 12: To get a fine-grained view of the impact distribution shifts have on fixed thresholds, we compare the distribution-shfteed ImageNet cascade Parcto front with the new ImageNet cascade Pareto front for the maximum softmax confidence metric. The distribution shifted Pareto front is obtained by using the same cascades at identical confidence threshold ranges that make up the original ImageNNtt Pareto front and evaluating them on ImageNett On the left, we show the individual distribution shifted segments, which represent separate cascades and threshold ranges that make up the original Pareto front. The distribution shift causes these segments to no longer be connected as identical thresholds lead changed early exit rates for different data and cascades are affected to a different degree. The new ImageNetV2 cascade Pareto front is obtained with the same algorithm that has been used to get the original Pareto front, which obtains the Pareto front across all possible 2-model cascades. The distribution shifted front remains close to Pareto optimal across much of the range.
E
Calibration
Early exit model cascades rely on a relationship between the confidence metric and the model accuracy where more confident predictions should be more accurate. This enables exiting early for predictions more likely to be correct and forwarding predictions less likely to be correct. However, Guo et al. (011)) report that modern networks are poorly calibrated. direction of overconfidence (Gawlikowski al. 2023 Abdar et al., 2021 Wang al. 2021).
often stated that this miscalibration is typically in the
We examine the calibration of models used in our ImageNet experiments in Figure 13 for the maximum softmax confidence metric. The baseline Pareto front consists of 27 models for the trade-off between accuracy and MAC count, and 25 models for the trade-off between accuracy and inference speed. Combined, 43 unique models are cascaded. Their calibration is visualized by comparing moving averages of both accuracy and confidence for 1000 examples across all 50000 examples of the validation set ordered by their confidence.
Contrary to what is often reported, underconfidence is more common than overconfidence among these state- of-the-art models. The largest model families BETT (Bao et. al.] 2022), ConvNeXt (Liu al.) 2022),, and DeiT3 [Touvron al. 2022) seem to have better than average calibration and are also among the most modern models used in our experiments.
What matters for cascading is to what exlent the miscalibration is shift and to what extent the positive relationship between accuracy and confidence disrupted. The former, for example when the model is over- or underconfident, addressed by shifting the carly exit threshold. However, the latter is a concern because it can decrease the model's accuracy for examples that satisfy threshold. Figure 13 shows that there is a noisy but generally consistent relationship between confidence and accuracy for all models.
18

---

Published in Transactions on Machine Learning Research (08/2023)
Bijetod
Expaht
Spatrd
Man
ceneett real
oceract Sirge bottolk
097 bur peoate
tezerk
lop packti
GF 221 Jazinik
greit 2
Sarlicneat
Sheing
Jame 1000
24
24
05
64 30
Mostraa
U25
12f 12:r
ker__20
Sieing Zvregr
Mroie Jurup 100
$100
Vlateme Jest_ 250
Marienn Salwer
Mir
Monterr Sedener
Martean Saltes
lent
Vissman Sullwa
Mosess Salees
Ma Sollmas
Montman Seree
Maiiem Salless
Prpuret
Peance
Madoes: Schoot
Macesas Schtnss
Malsent Safteas
Powred
IC.
KLonkioetc und
022
120
34
Mastaae Softuut
Mtutaaa Scttaacs
Modtzza Sertaer
Meges_
atd_ R. 201
100
Figure 13: The moving average of 1000 examples for both accuracy and ordered confidence (maximum softmax) for all 43 models used in the ImageNet experiments to visualize how well they are calibrated. The titles are the names of the models according to the PyTorch Image Models library (Wightman) 2019).
19

---

Published in Transactions on Machine Learning Research (08/2023)
We attempt further investigation by inspecting curves of model accuracy above and below respective thresh- olds representing various early exit rates as shown in Figure 14 For ideal cascade performance want a confidence score threshold above which every prediction is correct and below which every prediction is false. We compare the uncalibrated with the temperature scaled predictions. Temperature scaling is done on the validation set and therefore maximally fitted to the data, which means this represents an upper limit and calibration performance during deployment is expected be worse. However, temperature scaled calibration seems have little impact.
To what extent is calibration desirable? One idea is to simulate perfect calibration by using temperature scaling to set the average maximum softmax equal to the model accuracy and then assume the calibration is perfect by setting the prediction accuracy to be the maximum softmax. Figure [4 shows improved but worse performance for EfficientNet B4 NoisyStudent. This indicates
performance for MobileNetV33-Latte that calibration not necessarily desirable. Instead, cascade performance determined by combination of the inference cost and accuracy difference between the modcls, the prediction accuracy above versus below the threshold, and how oflen the next model makes correct predictions for inputs which do not exit early.
mobilenetv3 large 100 miil
Aocuracy Threshold
0.8
0.8
Maximam Softmax Theshold
Accuracy Delow Threshold
0.4
Accuray Above Threshald
Uucalibrated
Temperature Sealed
Assameed Perfect Calibration
Ideal
0.0
ImageNet Test Submissson
0
10000
20000
30000
40000
50000
Validation Set Example
effirientnet, ns
1.0
Amurcy Threshold
0.8
0.6
Maxcimmn Thresheld
Accuracy Below Thresook
0.4
Avcuracy Abore Threshold
Uncaltbrated
Tempessure Scaled
0.2
Assunee Perlect Calbrattia
Idaal
ImageNet Test Sulmisston
0.0
10000
20000
30000
40000
50000
Validation Set Example
The ordered maximum softmax with prediction accuracy for examples above and below the respective threshold is shown for uncalibrated and temperature scaled predictions, as well as the idcal scenario where there a threshold above which every prediction correct and below which every prediction is false, and assumed perfect calibration where temperature scaling is used to set the average maximum softmax equal to the accuracy and the prediction confidence is then treated as the prediction accuracy. The confidence threshold used for ImageNet test submissions as listed in Table 1 is highlighted to show points of maximal Pareto improvement.
Figure 14:
20


Reviews:
[{"review_id": "jhnvGk72oh", "reviewer": "Reviewer_RnjH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is an investigative work. It summarizes past work and compares different methods on equal footing. Some analysis is done to understand what are the right design choices for this category of models, and to give us intuition on the method. By comparing lots of models and setups it also is a trustworthy data point.\n\nWhile I appreciate this work, it feels more like a practical guide than a scientific investigation. I mean by that that it investigates the \"how\" (how do I make a good cascade) more than the \"why\" (what makes a good cascade good?). Generally I think this is where this paper could benefit the most from, unfortunately I don't have specific advice on what direction. I do think doing better on this front would help spur more precise and informative follow up work (something the authors seem to suggest is missing to the field). \n\nI do have a number of comments on the work, see the following section, but I'm otherwise content with this being a TMLR paper. It was easy to read and understand, and appears to be empirically sound.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper conducts a fairly thorough investigation of so-called model cascades, summarizing existing literature and trying to properly compare existing methods.\nA model cascade is when a sequence of models (of growing computational cost and accuracy) is queried iteratively, but when the iteration can be stopped if a model's prediction is judged confident enough. This allows to trade off accuracy and cost in a natural way, and can be done with pretrained models without any need for finetuning.\nThe paper makes the following main empirical observations:\n- Choosing models with large computational cost differences for the cascade induces the most cost-efficient trade offs\n- The ideal exit strategy and ensembling strategy are problem and ensemble-choice dependent, but generally thresholding (with thresholds induces from the validation set) and taking the most confident prediction works\n- 2-model cascades are already good, 3-model cascades only marginally improve performance\n\n\nGenerally the paper suggests that we can already do interesting things with very simple cascades, and so that we may be able to do better with more research on the topic.\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "This wouldn't be a review if it didn't ask for more baselines, and so I do wonder how this compares against dynamic models, either \"early exit\"-type models or even conditional-type models (e.g. large mixture of experts, like [1] and its descendants) that use a cheap model to route the input to the appropriate model.  \nI think this is an important baseline class because (a) it also allows for natural trade offs, and (b) in the \"limit\" of us getting better at ML I'd bet on the model class where the whole model is \"aware\" of its different parts over an amalgamation of agnostic models combined after the fact (and not necessarily \"tuned\" with each other).\n\n[1] Shazeer, Noam, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. \"Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.\" arXiv preprint arXiv:1701.06538 (2017)\n\nI'm in some sense surprised these methods work at all given the known lack of calibration of deep neural networks [2, 3]. I think it would make sense to spend a little more time on this (and perhaps even empirically justify the use of softmax logits almost as-is for iterative model selection -- this is the kind of \"scientific investigation\" I'm trying to allude to above).\n\n[2] Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. \"On calibration of modern neural networks.\" In International conference on machine learning, pp. 1321-1330. PMLR, 2017.   \n[3] Wang, Deng-Bao, Lei Feng, and Min-Ling Zhang. \"Rethinking calibration of deep neural networks: Do not be afraid of overconfidence.\" Advances in Neural Information Processing Systems 34 (2021): 11809-11820.\n\n\nSome more comments:\n- Maybe I'm missing something but on several occasion the authors write about \"research that is currently missing.\" I'm not sure I see specifically what this is about; it would be helpful to expand on this.\n- I'm not sure I understand the whole paragraph at the top of page 4. $A_2$ is introduced but never used, and then the grammar of the sentences following that is a bit weird. Would it be possible to rephrase?\n- Also on page 4, after Eq (3), the NLL is minimized on what? A single example? A data set? I'm assuming the validation set?\n- Figure 6, it would make sense if the Pareto front was shifted down by a constant factor (as much as the performance of the base models of the cascade is decreased by), but it could also be slightly better (the cascade could induce some robustness somehow). Is this the case? If not I'm not sure why this particular experiment is relevant in the paper. Are cascades normally known to _not_ be robust?\n- When evaluating Pareto fronts, it's common to reduce them to some quantity so that different methods can be quantitatively compared. This can be done with indicators like the hypervolume, generational distance, and so on. Might be worth considering it here, since it does appear that methods in this paper do not strictly dominate one another.\n- It's interesting that, similarly to ensembles, 2 or 3 models are sometimes plenty enough. I do wonder if this could be stretched further, and this is perhaps a part that's lacking in this paper. What could we do with 5, 10 models? Would this test the very idea of cascades? In some sense, if a model is too trivial, then we cannot trust its own self-confidence. What are the limits of that?\n- Obviously the aim of this paper was not to do so, but it would be interesting to speculate more on how much better the trade off could be if models were trained _with the intent_ of using them in a cascade. Intuitively, the gap should exist, since it exists in e.g. pruned and quantized models.\n", "broader_impact_concerns": "There is no Broader Impact Concerns statement in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5LdpnLQRpS", "reviewer": "Reviewer_FkTW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n - The paper evaluates a simple idea: model cascades with conditional early exit, and backs it up with abundant experimental results.\n - The goal is clear: demonstrating that model cascades constructed from pretrained models of an existing Pareto front further improves the Pareto front \n - Experiments cover both the image domain and the text domain\n - It further explores ensemble methods and their effects\n - The writing is clear, and the paper is structured to be easy to follow\n\n\n## Weaknesses\n - The biggest weakness of this paper is that it is unclear how the thresholds are obtained. It seems to me that it is optimized for every cascade and exit strategy, at lest for 2-model cascades that produced Figure 1 and Table 1. That has too problems essentially. Firstly, having to optimize for thresholds reduces the validity of the method — the saved inference compute from early exit would be canceled out from the compute for threshold optimization. Secondly, it is the paper's job to document the process clearly. When said \"we choose the threshold at the point of maximal validation Pareto improvement for our ImageNet test server submissions\" it remains unclear if the threshold is obtained via a grid search, binary search, or something else. In the 3-model cascade case, it is also unclear how \"we set the first threshold\"\n - The definition of MAC was never spelled out. Since it is such an important concept throughout the paper, it should be clearly defined early on, with equations and explanations.\n - The conclusion is not very convincing. Although the experiment shows that using cascades can allow for optimizing for maxim Pareto improvement, if it incurs extra compute finding proper thresholds, it will not be worth it eventually. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper validates and explores the design space of model cascades -- combining 2 or more pretrained models with conditional execution to perform standard classification tests. The work thoroughly evaluates the impact on the accuracy-efficiency trade-off, contains abundant ablation study, and provide a solid baseline for the direction of utilizing more than one pretrained models combined with early exits to boost performances. Experiments contain both image classification (ImageNet) and text classification (GLUE) tasks, covering a wide range of usability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the paper can be much stronger once these issues are addressed and corresponding changes made:\n\n- Clarify how the thresholds are obtained for each experiment\n- Address the added cost from finding/optimizing thresholds\n- Add additional experiments with fixed thresholds (e.g. 0.5) or fixed exit rate and see how it performs\n- Add equation or clear definition of MAC counts\n\nOther, minor issues:\n - In Algorithm 1, it should be made clear that models {M1, ..., Mn} are ordered by model size\n - This sentence is unclear, consider rewrite or further clarify: \"No linear interpolation is done between points at cascade Pareto fronts to have a direct comparison between individual cascades and random model selection.\"\n - Also nuclear what this means: \"due to the outlying MobileNetV3-Large pretrained on ImageNet-22K\"\n - In 4.2, it's unclear how many models in total are used\n\n\n\n\n\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UHcR2Ls5uf", "reviewer": "Reviewer_aYFZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-organized and easy to follow.\n2. Although the proposed method is simple, it demonstrates its effectiveness in Experiments.\n3. The paper verifies some current used techniques and establishes a reproducible baseline, which are both useful for future research.\n\nWeakness:\n1. What are \"Pareto fronts\" and \"Multiply-Accumulate (MAC)\"? The critical citations are missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a simple yet effective method for model cascades. The proposed method combines two or more models, and the inference process terminates early when the Softmax output from a model is above a threshold. The authors conduct experiments with hundreds of models on both image and text datasets to verify the effectiveness of the proposed method, demonstrating good performance on the accuracy-efficiency trade-off. The code for the proposed method will be made public as a reproducible state-of-the-art baseline for future research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The authors give six contributions in Introduction. It would be beneficial to condense them into three or four key points.\n2. Some important citations should be supplemented with additional information.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers recommend to accept this paper, after several points were clarified in the discussion with the authors. The reviewers comment that the paper asks its field to take a pause and think, rather than being a paper with a groundbreaking novel contribution. The reviewers feel this paper presents a straightforward yet effective approach for modeling cascades, striking an optimal balance between accuracy and efficiency. The paper is well written, the edits post-review are helpful, and the technical contribution feels interesting and relevant to move things forward in that subfield.
Decision: Accept (poster)