Conditional Execution Of Cascaded Models Improves The Accuracy-Efficiency Trade-Off
Authors: Luzian Lebovitz
Lukas Cavigelli
Michele Magno
Lorenz K Muller
Venue: Submitted to ICLR 2023 (2023)
Keywords: inference
efficiency
cascades
pretrained

Abstract: The compute effort required to perform inference on state-of-the-art deep learning models is ever growing. Practical applications are commonly limited to a certain cost per inference. Cascades of pretrained models with conditional execution address these requirements based on the intuition that some inputs are easy enough that they can be processed correctly by a small model allowing for an early exit. If the small model is not sufficiently confident in its prediction, the input is passed on to a larger model. The selection of the confidence threshold allows to trade off compute effort against accuracy. In this work, we explore the effective design of model cascades, and thoroughly evaluate the impact on the accuracy-compute trade-off. We find that they not only interpolate favorably between pretrained models, but that this trade-off curve commonly outperforms single models. This allows us to redefine most of the ImageNet Pareto front already with 2-model cascades, achieving an average reduction in compute effort at equal accuracy of almost 3.1x above 86% and more than 1.9x between 80% and 86% top-1 accuracy. We confirm the wide applicability and effectiveness of the method on the GLUE benchmark. We release the code to reproduce our experiments in the supplementary material and use only publicly available models and datasets.

Under review as a conference paper at ICLR 2023
CONDITIONAL EXECUTION OF CASCADED MODELS IMPROVES THE ACCURACY-EFTICIENCY TRADE-OFF
Anonymous authors Paper under double-blind review
ABSTRACT
The compute effort required to perform inference on state-of-the-art deep learning models is ever growing. Practical applications are commonly limited certain cost per inference. Cascades of pretrained models with conditional execution ad- dress these requirements based on the intuition that some inputs are easy enough that they be processed correctly by small model allowing for early exit. If the small model is not sufficiently confident in its prediction, the input is passed larger model. The selection of the confidence threshold allows to trade off compute effort against accuracy. In this work, we explore the effective design of model cascades, and thoroughly evaluate the impact the eccuracy-compute trade-off. We find that they not only interpolate favorably between pretrained models, but that this trade-off curve commonly outperforms single models. This allows us to redefine most of the ImageNet Pareto front already with 2-model cascades, achieving an average reduction in compute effort equal accuracy of almost 3.1x above 86% and more than 1.9x between 80% and 86% top-1 ac- euracy. We confirm the wide applicability and effectiveness of the method on the GLUE benchmark. We release the code to reproduce our experiments in the supplementary material and use only publicly available models and datasets.
INTRODUCTION
The trade-off between accuracy and efficiency fundamentally important to deep learning. While state-of-the--at results are achieved by models of ever growing size, practical applications are con- strained. Reducing the energy consumption important for energy limited systems like wearable and mobile devices. For inference at large scale such as in data centers, minimizing the compute resource requirements is important economically and environmentally.
There are many different approaches to improve the accuracy-efficiency trade-off (see Section 2) We explore early exiting from cascades of pretrained models for classification tasks. Such an ap- proach can work when some inputs easier to classify than others and therefore can be processed by a smaller model to save computation. Assuming the input's difficulty is unknown, a first clas- sification be made by the smaller model and if it is not confident in its prediction, the input can be classified again by a larger and more accurate model. The models in an early exit cascade are ordered by cost from cheapest most expensive and an early exit is made after the first model where the prediction confidence satisfies the exit condition.
Early exiting is often explored from within a model like in BranchyNet (Teerapittayanon al. 2016) or Shallow-Deep Networks (Kaya et al.) 2019), by inserting additional classifier outputs earlier in the model. Huang al. (2017) describe how early classifiers lack coarse features and training the model for early classification can lower the accuracy of later classifiers. Furthermore, modern con- volutional neural networks are highly optimized and their depth, width and resolution is carefully scaled, gradually lowering the resolution and increasing the number of channels as features become more complex (Tan & Le 2019). For early classification outputs, this optimization and scaling is disrupted, which results in a lower accuracy than a separate model at equal compute effort can achieve. We demonstrate experimentally that a smaller accuracy difference between classifiers al- lows for more frequent early exiting. This motivates using separate model for early classification, which enables us to increase the early exit rate at the cost of a small overhead when forwarding to the larger model, since features longer shared between classifiers. However, early exit-

---

Under review as a conference paper at ICLR 2023
ing from within single model and between cascaded models are two separate and complementary approaches that can be combined (Bolukbasi et al. 2017).
A disadvantage of early exit model cascades is that more than model is required, which increases memory as well as worst case latency and compute effort, where the worst case is no early exit. However, the costs models is usually negligible. Key advantages are the ease of use and efficacy. Pretrained models can be used as is finetuned. Cascading is a complementary approach and can be combined with many other approaches for increasing efficiency, particularly methods that improve the individual models, such as better architecture and training. When the input processed by at least 2 models in the cascade, the predictions can be ensembled. (Gontjo-Lopes et al. (2021) show that ensembles are more accurate when the models are more diverse due to the lower error correlation, which can also be leveraged by cascades.
dominated by the largest model, which means that the added cost of smaller
Although early exit model cascades are simple and effective, little prior work on them has been published (see Section 2). We explore how to build such cascades effectively on ImageNet (Rus-) sakovsky et al., 2015). For this we compare many different cascading methods across continuous Pareto fronts in multiple settings using diverse selection of pretrained models for thorough and reliable evaluation. We then confirm their wide applicability on text classification tasks from the
GLUE (Wang et al. 2018a) benchmark. Our contributions and findings are:
We demonstrate that already 2-model cascades dominate the entire ImageNet Pareto front.
We provide insight into how to construct the model cascades: We empirically demonstrate relationship between accuracy difference of cascaded models and achievable early exit rate as well as desirable size difference in Figure 2.
The maximum softmax confidence metric achieves most improvement overall while soft- max margin excels in low confidence scenarios and the commonly used entropy metric performs worst.
Ensembling predictions, when inference is done for least 2 models in a cascade, gen- erally increases the top accuracy achieved by the cascade but lowers Pareto improvement. Temperature scaled calibration ineffective at alleviating this.
For our evaluation models to ensure our results are credible and relevant. This means we provide strong and reproducible baseline for future related research that is currently missing.
rely an external baseline comprised of Pareto optimal public
2 RELATED WORK
Accuracy-efficiency trade-off Improving the accuracy-efficinncy Pareto front is a central objec- tive for wide breadth of research with many different approaches. MobileNetV3 (Howard et al.) 2019) and EfficientNet (Tan & 2019) represent how architectures have become highly optimized. Model efficiency can improved further by applying compression techniques like quantization and pruning (Han et al. 2015). Once-for-all (Cai et al. 2019) utilizes progressive shrinking of input resolution, kernel size, network width and depth together with knowledge distillation to obtain more efficient models than conventional neural architecture search. Training very important with recent advances most notably through data augmentation (Shorten & Khoshgoftaar (009) and pretraining on more data (Ridnik et al., (021). which is enabled further by cell-supervise (Devlin et al.) 2018) and semi-supervised (Pham et al. 2021) methods. Many other procedures exist such as model soups (Wortsman al. 2022). which line-tunes model with multiple hyperparameter configurations and averages the weights. More closely related to our work are dynamic neural networks (Han et al. 2021 Xu & McAuley, (022). An example for dynamic depth is SkipNet (Wang et al.) (018b), which adds gating modules and learned skipping policy to skip network layers based on the input. BranchyNet (Teerapittayanon
al., 2016) adds branches to the original net for early evaluation and exits when prediction entropy is below threshold. Wolczyk et al. (202I) improve early exiting from within a model by recyeling predictions of earlier classifiers. We focus on early exiting from a cascade of models, which is a complementary approach that is comparatively simple yet effective.
Early exit model cascades Cascades are commonly used in machine learning and have been pop- ularized by influential works such as Viola & Jones (001). We focus on specific type of cascade
2

---

Under review as a conference paper at ICLR 2023
which uses multiple models in sequence with the capability to exit early if confident enough in a pre- diction. Park et al. (2015) propose the big/LITTLE architecture, which is a cascade of a small and large model with static dynamic softmax margin threshold exit condition to reduce energy con- sumption. [Wang et al. 2017) demonstrate reduced computational cost by cascading up to 3 models. They compare different classifiers for the objective of achieving desired accuracy, which include trained decision network, cost-oblivious grid search for both entropy and maximum softmax thresh- olds, and cost-aware learned entropy threshold. They find that only the last three methods manage to satisfy the accuracy objective, that entropy is a better confidence metric than maximum softmax and that the aost-aware method performs better. [uaan al. (2017) train an agent to exit between models to reduce computational cost. Similarly. [Bolukbasi et al. (2017) present trained policies to exit early between as well as within models for a trade-off between accuracy and inference time. Streeter (2018) proposes an algorithm to automatically construct efficient cascades from selection of available models. [nooe (2019) show that exiting early from an ensemble of 10 or 20 models can greatly reduce computational cost while preserving most of the static ensemble's accuracy. They use average softmax to ensemble the predictions and exit based on a confidence interval condition. Wang et al. (2022) cascade up to 4 models of the same architecture families such EfficientNet (Tan & Le 2019). ensemble by averaging the logits, use maximum softmax as confidence metric, and optimize for accuracy and computational cost with exhaustive search. They also demonstrate the efficacy of self-cascading by varying the input resolution for the same model.
From prior work, we notice lack of comparison between different cascading methods, knowledge gaps about when cascades perform well, and limited evaluation. We address this by comparing many different cascading methods in multiple settings using diverse selection of pretrained models for a thorough and reliable evaluation. We investigate when cascades work well and unlike prior work we evaluate improvement across continuous Pareto fronts. No training is done in our work. Instead, we use publicly available state-of-the-arr models and make available code to reproduce our results, thereby providing a strong baseline for future related research that currently missing.
3 METHOD
The idea behind early exit cascades is that inference can be done on a smaller model first and if confident in its prediction we can save time and computation by exiting early and skipping the original model. A cascade contains at least 2 pretrained models. A condition is needed to decide whether to exit early. This condition uses a confidence score computed from the model prediction and a threshold set in advance which needs to be satisfied. Algorithm shows the implementation of the cascading method we found to be most effective.
Algorithm 1 Early exit model cascade with maximum softmax confidence metric and no ensembling
Require: input tensor X, models (M1,.., M,). thresholds (f1:-.., .i..-..,,-1l 1), n 2 2, E N
for i = I,..., n do
2; (X)
p; = softmax(z;)
if (i < and max(p,) 2 t2) or == n then
return argmax(p:)
D cascade returns predicted class
Cost Let Cj be the cost of a model M, which represents metries we want improve, such as number of -ucupllyAccuumllte (MAC) operations or time of inference. We order the models within a cascade so that C, < C(+1, meaning the cheaper model makes a prediction first. When there are models in the cascade, there are n possible costs when inferring a single input. In the case n = 2, when an early exit is made, the cost C1, otherwise the cost is C1 + Cz. They represent the lower and upper bound for the cascade. Assume we exit early after the first model at a rate E the cost becomes C + (1-&) OC2 in average. For cost reduction, we want to achieve C₁ + +1--E) < C2- Therefore, our early exit rate must satisfy ει > C1/C2, Assume we want to maintain the second model's 's accuracy A2. For a two-model cascade, the improvement factor / by which we reduce the cost is I = CC/ + (1 C2). Therefore, the improvement we can achieve is limited by the early exit rate we can reach before cascade accuracy declines and the cost difference between the models, where larger is better for both.
3

---

Under review as a conference paper at ICLR 2023
Early exit condition We need a condition to decide when to exit early. Such condition is typ- ically evaluated by obtaining a confidence score from the model prediction and comparing to a threshold t. This is done after each model in the cascade except the last, resulting in n- thresholds. For classification tasks, the model output for an input tensor X is usually a logits vector z and the predicted class is argmax(z). We the softmax function
e
softmax(2:)
ΣØ E
Pi
(1)
to transform the logits into pseudo-proon heuudo-robabiilties for all K classes. One confidence metric is the maximum softmax, which we can use as probability that the prediction is correct. We exit early max(p) 2 t. Various other conditions exist in literature (Han et al., 2021 Xu & McAAley) 2022) and the entropy threshold appears to be most common. After applying the softmax function we can compute the Shannon entropy
H(p) = - p. log(p,).
(2)
lower entropy represents higher confidence in the prediction and we exit early when Hp) I Other confidence metries found in literature are the logits margin (Streeter, (2188) and the softmax margin (Park al.) 2015), which are the largest logit or softmax minus the second largest respec- tively. They represent by how much the model is more confident in its first prediction and we exit if the margin is above or equal to a threshold.
Ensemble When the cascade does not exit after the first model, we have multiple predictions which we can ensemble in the hope of improving performance. Typical ways to ensemble models plurality vote (Hansen & Salamon, T990p averaging of the outputs Bishop et al. (1995). To mean ensemble the outputs, we compute the arithmetic mean of softmax or logits. An intuitive alternative is compare the prediction confidence of all models and choose the most confident, which we call comparison ensemble. Weighting or calibration can be used when model confidences are mismatched so that the most overconfident model does not dominate the ensemble. Guo et al. (2017) demonstrate that temperature scaling is an effective calibration tool AnduAsh et al. (2020) recommend it for ensembles. Temperature scaling is achieved by dividing the logits by a temperature factor T so that the calibrated probabilities become
p
(3)
Σ
where T is set to minimize the negative log- log-liiel
4 EXPERIMENTS
In order to evaluate the impact of early exit model cascades, we verify whether they reliably out- perform state-of-the-art single models across entire Pareto fronts on different tasks. We rigorously compare many different cascading methods to determine the most effective. To understand when cascades work well, we figure out how properties such as size and accuracy of cascaded models affect performance. We begin by conducting set of experiments 2015) (Section 4.1) due to its relevance in related work. As a step to indicate general utility of demonstrate Pareto improvement
ImageNet (Russakovsky al.
text classification tasks from the GLUE (Wang et al.] 201 8a) benchmark (Section 4.2). All pretrained models used for image and text clas- sification are publicly available from PyTorch Image Models (Wightman 2019) and Hugging Face (Wolf et al. (2200) respectively. We use a GeForce RTX 3090 GPU for all inference. The code to reproduce these experiments is available the supplementary material.
model cascades,
4.1 IMAGE CLASSIFICATION ON IMAGENET
We select from 668 pretrained models with tabulated accuracy for the ImageNet validation set from PyTorch Image Models (Wightman) 2019). For the computational cost of models we obtain MAC with the fvcore library. To acquire the time cost use benchmark numbers in inferred samples per
4

---

Under review as a conference paper at ICLR 2023
second on an RTX 3090 with NHWC data format and automatic mixed precision from Wightman (2019). From this we can establish baseline Pareto fronts for both MAC and time cost. We infer the ImageNet validation set for models at the Pareto fronts to confirm their accuracy and obtain logits, from which we compute prediction correctness and confidence score for every image. From these efficiently compute exact accuracy, threshold and average inference cost all possible early exit rates for every cascade. Threshold selection can then be made based on desired trade-off. For example choose the threshold the point of maximal validation Pareto improvement for our ImageNet test server submissions.
As a strong and continuous baseline we use the Pareto front established by linear interpolation between pretrained model pairs, which represents the trade-off achieved when randomly selecting one of the two models with varying probability. No linear interpolation is done between points at cascade Pareto fronts to have direct comparison between single cascades and random model selection. We evaluate improvement by cascade Pareto fronts as factor by which average inference cost in MAC or time is reduced to achieve the same accuracy. The average cost represents the total cost for the entire validation set at specific thresholds divided by number of images.
90
90
IntageNet Validaton Top-1 %
85
78
15
70
Presrained Models
Baseline Pareto
Sofizuax Coocade Pareto
60
Tested Chacades
80
ImagaNes Tests
10
10
10
10
Average Inference MAC
Improvement Factor
Figure I: The maximum softmax confidence metric without calibration and ensembling is one of the simplest yet most effective for cascading and achieves Pareto improvement across of the MAC range with 2-model cascades. The pretrained models show that the accuracy benefit from increasing model size diminishes. Cascade performance improves as accuracy difference between models decreases compared to MAC difference. Two points at the cascade Pareto front with locally maximal improvement were evaluated on the ImageNet test set. The results are summarized in Table I and confirm the validation results. Notable are that cascading also increases the top accuracy by 0.306% to 98.90% and that the improvement factor drops to almost at 77.92% accuracy due to the outlying MobileNetV3-LLncm pretrained ImageNet-22 -2K (Ridnik et al., 2021).
Two points at locally maximal improvement from the softmax 2-model cascade Pareto front were evaluated on the ImageNet test set as shown in Figure [I] They were chosen strongest points at the cascade Pareto front to test against validation set overfitting. Each point corresponds to cascade with specific threshold. Values are rounded to 4 significant digits.
Table 1:
Cascade Models
Validation
Test
Softmax
First
Second
Top-1
MAC L
Top-1
MAC 1
Threshold
MN3-S 0.75 (Howard et al. 2019
MN3-L Ridnik al
2021
75.88%
136.2M
75.91 %
135.9M
0.4551
MN3-L Ridmk et 2021
EN B3 NS Xie al. 2020
83.39 %
732.4M
83.13%
735.2M
0.7112
EN B4 (Xie al. 2020
2022
87.32%
14.39G
87.21%
14.38G
0.5658
For the trade-off between accuracy and MAC, the baseline Pareto front consists of 27 models. For 2-model cascades, there are (27) = 351 possible combinations. Figure shows the Pareto front and improvement factor for the cascading method which uses maximum softmax confidence without calibration or ensembling. The cascades achieve improvement along most of the Pareto front with reduction in computational cost by factor of up 3.83 for the same accuracy. This is achieved
5

---

Under review as a conference paper at ICLR 2023
by the cascade containing the models EfficientNet B4 NS Tan & Le 2019). convolutional neural network pretrained on 300M unlabeled images from the JFT dataset (Hinton et al. 2015 Chollet
2017). and 224 (Bao et al. 202T), vision transformer pretrained on ImageNet-224. which shows the benefit of model diversity. Model families which contribute to the cascade Pareto front in descending order of importance are EfficientNet NoisyStudent (Xie et al. 2020), MobileNetV3
(Howard et al. 2019 Ridnik et al.) (021), BEiT (Bao et al. 2021). TinyNet (Han et al. 2020b),
GhostNet (Han et al. 2020a) and LeViT (Graham al. 2021
To test against validation set overfitting, points of the cascade Pareto front with locally maximal improvement were evaluated on the ImageNet test set. As shown in Table test set performance is close enough to confirm the validation set results. However, test accuracy is slightly lower in average. Possible explanations are that the optimal points are shifted for the test set, that the cascade Pareto front envelops noise and that the pretrained models themselves are slightly overfitted to the validation set. To investigate the last point, the pretrained EfficientNet B4 NoisyStudent was evalu- ated on the ImageNet test set and achieved a top-1 accuracy 0.04% lower than on the validation set, which accounts for almost half of the 0.09% difference for the 3rd test point.
$7.5
5i Relattre Aesurayy
InoggNee Valodatonn Tog-1
Stodok Top-2 Diffeceuse
0335
Modhll Top-1 Eiflecunee
0:35
2.
0.20
11
0.55
0.55
-18
08.
Eiliatte B4 33
0.00
0.00
BBT
Sehrean Cur
Duedde Placts
Selatino
0.00
0.30
117
237
Average Informne MAC
1410
Eaaly Palt
36odals MAC Eifiecence
The cascades tend to maintain the accuracy of the better model for plateau before accuracy starts to break down. Middle: Each point represents the largest early exit rate before cascade accuracy drops by 5% of the top-1 difference below the better model. Right: The average Pareto improvement of all individual cascades. While a lower accuracy difference between cascaded models is always desirable, there exists sweet spot for the size difference. While the length of the plateau before accuracy drops increases with smaller accuracy difference between models, for the baseline Pareto this plateau grows with the size range the accuracy benefit of increased model size diminishes.
Figure 2: Left:
Figure 2 shows the relationship between the accuracy difference of the models the cascade and the early exit rate cascades can achieve before accuracy starts to break down. The cascades perform better when the accuracy difference between models is small while the MAC difference is large. Therefore, the Pareto improvement is minimal in the lowest MAC regime and becomes larger higher MAC regimes because of the diminishing accuracy benefit from increased model size, which can be observed in Figure 1
Cascading method comparison We compare the confidence metrics entropy, maximum soft- max, softmax margin and logits margin both without ensembling and with comparison ensembling enabled by calibration using temperature scaling. Furthermore, using the maximum softmax confi- dence metric we evaluate mean ensembling for both logits and softmax, with and without tempera- ture scaling. This results in 12 different cascading methods compared in Figure 3
From the methods without ensembling, the entropy confidence metric performs worst. Entropy influenced by probabilities assigned to false labels while we are only interested in whether the predicted label is correct for early exiting. Softmax margin is the only cascading method which achieves an improvement > across the entire Pareto front (except at the smallest model, whose computational cost cannot be reduced further with cascading). It also reaches the largest average Pareto improvement of all 12 methods with x 1.5022 across the entire accuracy range. This is because softmax margin dominates below 78% accuracy, which represents almost 2/3 of the accuracy range but only about 1/4 of the log scaled MAC range. The maximum softmax metric has the second largest average improvement 1, 1.497 and shows most MAC range dominance of all methods.

---

Under review as a conference paper at ICLR 2023
When ensembling, calibration may be required, because otherwise there is no guarantee that the un- calibrated confidence scores of the different models are comparable. Temperature scaling was done on the validation set, which means that the results for methods using temperature scaling represent an ideal scenario where the calibration is optimally fitted to the data. Yet, calibration proves ineffective for mean ensemble cascades and the comparison ensemble methods achieve the least improvement of all. Mean ensembling shows potential at the upper end of the accuracy range. Therefore, we use the maximum softmax confidence metric, which is worse overall but better at higher accuracies than the softmax margin (see Appendix [ for details). While the mean softmax method performs better overall, uncalibrated mean logits ensembling achieves the best average improvement above 86% accuracy of all cascading methods, as well as the best top accuracy of 89.03%. A possible explanation for the poor performance of ensembling is that Pareto optimal cascade settings rely on high early exit rates, where only the least certain predictions are forwarded. These are less useful for or may even hurt ensembling.
No Ensemble
C Comparison Ensemble ble
Mean Ensemble
ImaggnNet Validation Top-1 %%
66
80
80
80
Max SoEsmax
Max Softzaa
78
Eatzopy
78
TS Encropy
Mean Logits
Mix Sof:max
TS Mix Saftmax
Moun Softmax
Soltmax Margin
Softmix Margis
Mean TS Logits
Logits Margin
76
TS Logits Margan
Sof-mox
2
Improvement Factor
Improvement Factor
Improvement Factor
Figure 3: A comparison of Pareto improvement achieved for the 12 tested cascading methods. To improve legibility, only the accuracy range above 75% is shown as improvement achieved below that point is small for all methods. The maximum softmax method from the left is plotted again as black line to have a baseline for comparison. TS stands for temperature scaled.
3-model cascades Cascading more models increases the complexity since well suited models are required for improvement and multiple thresholds need to be selected, which increases the problem dimensionality. The ImageNet validation consists of 50000 images, which means that there are 50001 choices per threshold ranging from always to never exiting early. For the 27 Pareto optimal models there are (?7) 2925 possible combinations. Since computing cascade performance is cheap, exhaustive search is still feasible for 3-model cascades and has the advantages of simplicity and a guarantee that the optimum will be found. To do so, for every model combination we set the first threshold and then compute the cascade performance for all choices of the second threshold. We can speed up the process by only sampling every tenth sorted value for the first threshold since neighboring values are almost identical.
Interestingly, ensembling performs slightly worse relative to no ensembling for 3 instead of 2 models and mean softmax is now superior to mean logits. The maximum softmax method achieves the largest average Pareto improvement of all cascading methods both across the entire accuracy range and above 80% only. Figure/sshow> that there is diminished benefit from adding a third model but improvement at higher accuracies. 2-model cascades offer strong and practical baseline, but for optimal performance adding more models should be considered while keeping in mind the trade- off between diminishing returns and increased complexity, which could also harm generalizability. To test this, we evaluated the point of the 3-model maximum softmax Pareto front with largest improvement on the ImageNet test set. This is the cascade with models EfficientNet B4 NS (Tan &
7

---

Under review as a conference paper at ICLR 2023
Le 2019), 221 (Bao et al. 2021) and EfficientNet L2 NS at the thresholds t 0.6299 and t2 0.4905. While test accuracy at 87.86% is only 0.08% lower than on the validation set, average MAC is also 1.1% larger 26.32G. This indicates slightly worse generalization than the 3rd test
result Table D for cascade with only the first 2 models.
InngeNNt Valdation Top-1 %
90
78
15
Pretzmined Modals
Baseline Pazeto
70
2-model Max Softmax
Jamodal Max Soitmax
3-model Mean Logits
Wang el al. Baee
Wong
10a
10
1090
10P4
1012
Average Inference MAC
Improvement
Factor
Figure 4: 3-model cascades start to outperform 2-model cascades at higher MAC regimes. Left: We compare results to Wang et al. (2022), who the mean logits ensemble method. Right: The maximum softmax method achieves the best performance of all cascading methods and outperforms the mean logits method used by Wang al. (022) across most of the Pareto front.
Inference time Radosavovic et al. (2020) show that the number of MAC operations is imper- fect predictor for GPU inference speed. Therefore, we also evaluate cascade performance for the trade-off between accuracy and speed measured in average time per inference. Figure [ shows that 2-model cascades already achieve improvement everywhere except near the outlying LeViT-128S (Graham et al. 202T). As be in greater detail in Appendix A maximum softmax again performs best of all tested cascading methods, which demonstrates its reliability and consistency. Notably, outperforms mean ensemble methods even above 86% accuracy for inference speed, achieving average improvement of x2.618 compared to x2.203 by the mean logits cascading method. The baseline Pareto front consists of 25 models. There a large model overlap between MAC and speed Pareto fronts. The biggest difference is that EfficientNet( (Tn & Le (2019); (2020) and the related TinyNet Han et al. (2020b) architecture families perform worse for inference speed while LeViT Graham et al. 1021 and ConvNeXt Liu et al. (2022) perform better.
(0019) Xie al.
InungeNNe Valiaation Top-1 %
45
85
75
70
70
Preizsiued Modele
Rasolino Pardoo
Max: Sofimax
Sofzzox Margin
Mesu Logits
10
10-4
10
10
10-4
A Average Seconds/ Inference
Improvement
Factor
Figure 5: Inference speed is measured in average time per image an RTX 3090 GPU. The images are batched, therefore the actual latency larger. Less Pareto improvement achieved than for MAC because the time cost spans 4 orders of magnitude instead of 5. The 2-model cascades perform particularly well at concave points of the baseline Pareto front.
8

---

Under review as a conference paper at ICLR 2023
4.2 TEXT CLASSIFICATION ON GLUE BENCHMARK
Early exit model cascades are effective for image classification with both convolutional neural net- works and vision transformers. We expect that the cascades also work in other domains where prediction confidence can be quantified. Indeed, beneficial early exiting has been demonstrated for image segmentation (Wang al. 2022) and reinforcement learning (Wotezyk et al. (02T). As first step to test that the identical cascade structure can be used in a different domain, we experiment on the text classification tasks SST-2 (Socher al. 2013), MRPC (Dolan & Brocket 20055. QNLI (Rajpurkar et al.) 2016) and QQP (Chen et al.) 20188) from the GLUE (Wang et al. (2118)) bench- mark. The results shown in Figure 6 and confirm the efficacy of early exit cascades, Since the tasks are binary classification, the 4 confidence metries entropy. maximum softmax, softmax margin and logits margin result in identical order when sorting the confidence scores of all predictions, and the resulting cascades are equivalent. We searched for fine-tuned models on Hugging, Face (Wolf) al. 2020). Models used include BERT (Devlin et al. 2018), RoBERTa (Liu et al. 2019), De- BERTa (He et al. 2020), ELECTRA (Clark et al. 2020). ALBERT (Lan et al., 2019), Dyn
(Hou et al. 2020), , DistilBERT (Sanh et al. 2019). M-FAC (Frantar et al. 2021
MiniLM (Wang
ct al.
2020 and XLNet (Yang al. 2019)).
Postralned Medek
Casaade Parelo
Boseline Parelo
SET-2 Valdiation Accuncy %
MRPC Valiettion Accarccy %
02-
00-
MRPC Valibation FL
50
02-
-6-
06
83-
80
QNL Valitocion Acmuary
05.0
QqP Valiiattooo Accutarcy %
QQP Valiiation F1 %%
15
00-
67.5
64
83.0
107
102
100
1011
12
10P
10
100
100
Avorage Inference MAC
Average
Average Inference MAC
Figure 6: Early exit cascades achieve Pareto improvement for text classification tasks from the GLUE benchmark in spite of the large accuracy differences between models. The results for the MRPC and QQP tasks are similar for both accuracy and Fl score.
5 CONCLUSION
We demonstrate the efficacy of carly exit cascades on wide selection of publicly available pre- trained models for image classification on ImageNet (Russakovsky et al.) (003)) and text classifica- tion tasks from the GLUE benchmark (Wang et al. (018a).. We compare various early exit condi- tions, ensembling strategies and temperature scaled calibration. Notably we find that the commonly used entropy threshold performs worst, that ensembling can increase the maximum accuracy but fails to improve the Pareto front and that temperature scaling is ineffective. Which method performs best varies based on the cascaded models. Since comparing cascade performance of different meth- ods is computationally cheap and fast, the best situational method can be determined. Otherwise, our recommendation to use the maximum softmax threshold condition without ensembling and calibration, which is not just simple and reliable but also the most effective of all methods tested.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We provide code reproduce our experiments in the supplementary material. Our code can be executed on regular consumer hardware in few GPU and CPU hours except for the 3-model cascade exhaustive search, which may take up to a few CPU days. For the GLUE benchmark experiments we also provide all data generated by the submitted code, which includes stored Pareto fronts, logits, inference data, dataframes with model information and more. For the ImageNet experiments we provide all Pareto fronts but do not include the stored cascades, logits and inference data due to their large file size. No training was done and all datasets and pretrained models used in this work are publicly available. Therefore, we provide strong and reproducible baseline for future related research that is currently missing.
REFERENCES
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. arXiv preprint arXiv:2002.06470, 2020.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2/06.0888 08254,, 2021.
Christopher M Bishop et al. Neural networks for paltern recognition. Oxford university press, 1995.
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In International Conference on Machine Learning, pp. 527-536. PMLR, 2017.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.00991, 2019.
Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. Quora question pairs, 2018.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher 1) Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint 20Xx::0003310555 2020.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv: 1110:/88000000 2018.
Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (/WP2005), 2005.
Elias Frantar, Eldar Kurtie, and Dan Alistarh. M-fac: Efficient matrix-free approximations of second-order information. Advances in Neural Information Processing Systems, 34, 2021.
Raphael Gontijo-Lopes, Yann Dauphin, and Ekin D Cubuk. No one representation to rule them all: Overlapping features of training methods. arXiv preprint arXiv:2 10. 2021.
Benjamin Graham, Alaacldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: vision transformer convnet's clothing for faster inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12259- 2021.
Jiaqi Guan, Yang Liu, Qiang Liu, and Jian Peng. Energy-efficient amortized inference with cascaded deep classifiers. arXiv preprint arXiv: 170.0336, 2017.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1580-1589, 2020a.
10

---

Under review as a conference paper at ICLR 2023
Kai Han, Yunhe Wang, Qiulin Zhang, Wei Zhang. Chunjing Xu, and Tong Zhang. Model rubik's cube: Twisting resolution, depth and width for tinynets. Advances in Neural Information Pro- cessing Systems, 33:19353-19364 2020b.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.0014 2015.
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions on pattern analysis and machine intelligence, 12000, 1990.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.0566 0365,, 2020.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv: 1xX3v:5503.02555 02.50.3..51,,,. 2(7), 2015.
Lu Hou, Zhiqi Huang. Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33: 9782-9793, 2020.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE/CVF International Conference Computer Vision, pp. 1314-1324, 2019.
Gao Huang. Danlu Chen, Tianhong Li, Felix Wu, Laurens Van Der Maaten, and Kilian Q Wein- berger. Multi-scale dense networks for resource efficient image classification. arXiv preprint 1703.0040.9444 2017.
Hiroshi Inoue. Adaptive ensemble prediction for deep neural networks based on confidence level. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1284-1293. PMLR, 2019.
Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. Shallow-deep networks: Understanding and mitigating network overthinking. In International conference on machine learning, pp. 3301- 3310. PMLR, 2019.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: lite bert for supe-supe learning of language representations. arXiv preprint arXiv:1909.1 11922, 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv:1907. 1692, 2019.
Zhuang Liu, Hanzi Mao, Chao-Yuee Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. arXiv preprint 0rXiv:2201.03545. 2022.
Eunhyeok Park, Dongyoung Kim, Soobeom Kim, Yong-Deok Kim, Gunhee Kim, Sungroh Yoon, and Sungjoo Yoo. Big/little deep neural network for ultra low power inference. In 2015 in- ternational conference on hardware/software codesign and system synthesis (codes+ isss), pp- 124-132. IEEE, 2015.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11557-11568, 2021.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, pp. 2383-2392. Association for Computational Linguistics, 2016.
11

---

Under review as a conference paper at ICLR 2023
Tal Ridnik, Emanuel Ben-Baruchs Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104. 10972, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 1153 2015. doi: 10.1007/s1 1263-015-0816-.y.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01100 2019.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, pp. 1631-1642, 2013.
Matthew Streeter. Approximation algorithms for cascading prediction models. In International Conference on Machine Learning, pp. 4752-4760. PMLR, 2018
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 1105-6114. PMLR, 2019.
Surat Teerapittayanon, Bradley McDanel, and Sailang-Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference Pattern Recognition (ICPR), pp. 2464-2469, IEEE, 2016.
Paul Viola and Michael Jones. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001, volume 1, pp. I-I. Icce, 2001.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:
2018a.
Wenhui Wang. Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neu- ral Information P Processing Systems, 33:5776-5788, 2020.
Xiaofang Wang, Dan Kondratyuk, Eric Christiansen, Kris M Kitani, Yair Movshovitz-Attias and Elad Eban. Wisdom of committees: An overlooked approach to faster and more accurate models. International Conference Learning Representations, 2022.
Xin Wang, Yujia Luo, Daniel Crankshaw, Alexey Tumanov, Fisher Yu, and Joseph E Gonzalez. Idk cascades: Fast deep learning by learning not to overthink. arXiv preprint arXiv:/706 178.00170..00885, 2017.
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph Gonzalez. Skipnet: Learning dy- namic routing in convolutional networks. In Proceedings of the European Conference on Com- puter Vision (ECCV), pp. 409-424, 2018b.
Ross Wightman. PyTorch Image Models.
https://gittuu.coommmmm
pytorch- wwwrorrch-imageeeeo www.imaaammoooo model 1s 2019.
Maciej Wotczyk, Bartosz Wojcik, Klaudia Balazy, Igor T Podolak, Jacek Tabor, Marek Smieja, and Tomasz Trzcinski. Zero time waste: Recycling predictions in early exit neural networks. Advances in Neural Information Processing Systems, 34:2516-2523 2021.
Thomas Wolf, Lysandre Debut, Victor Sanb, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: of art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in
12

---

Under review as a conference paper at ICLR 2023
Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. As- sociation for Computational Linguistics. URL lttpp:://// /wwp:://wwwwww.w. awwweb. wtg.a//oooo/tttooooyo//oooooo///
2020.emn1p-demos 6
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without creasing inference time, arXiv preprint arXiv:2203.05482, 2022.
Qizhe Xie, Minh-Thann Than-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10687-10698, 2020.
Canwen Xu and Julian McAuley. A survey dynamic neural networks for natural language pro- cessing. arXiv preprint arXiv:2202.07001, 2022.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances neural information processing systems, 32, 2019.
13

---

Under review as a conference paper at ICLR 2023
A
ADDITIONAL PARETO IMPROVEMENT COMPARISON DATA
No Ensemble
Comparison Ensemble
Mean Ensemble
ImageNet Validation Top-1 %
84
84-
82
82-
82
80
80
Max Softmax
Max Softann
Entropy
TS Enizopy
Moan Logits
Max Sollmax
TS Max Safmmax
Mean Solltan
Softmaa Margin
Softanad Margin
Moan TS Logtts
Logits Margin
Logits Margin
Mean TS Softmas:
Improvement Factor
Itppovvmentt Factor
Improvement Factor
Figure 7: 3-model cascade Pareto improvement for the accuracy-MAC trade-off. Maximum softmax confidence without ensembling is the best cascading method overall and is plotted again in the middle and right black line to have baseline for comparison. The softmax margin cascading method performs best in the lower accuracy range and mean ensembling shines at the upper end of the accuracy range.
No Eusembble
Comparison Eusemblee
Mean Ensemble
88
88
88
86
86
ImageNet Validation Top-1
84
80
Max Soltamx
78
Entropy
78
Eniropy
78
Meun Logits
Max Softmax
Max Sofamax
Moan Softmax
Softmax Margin
TS Softmax Margin
Moan TS Logits
76
Logis Margin
76
TS Logia Margin
76
Mea TS Soltmox
Improvement Factor
Iraprvvmenn Factor
Inoprvveeen Factor
Figure 8: 2-model cascade Pareto improvement for the accuracy-time trade-off. Again, maximum softmax confidence without ensembling the best cascading method of all and is plotted again in the middle and right as black line to have baseline for comparison. Notable is that mean en- sembling shows worse performance for inference speed relative the maximum softmax method. Furthermore, the mean softmax method now outperforms the mean logits method, This indicates that cascading methods which use ensembling are less reliable.
14

---

Under review as a conference paper at ICLR 2023
B
TOP-5 ACCURACY
100.0
100.0
17.5
ImageNet Validation Top-3
95.0
95.0
90.0
90.0
Preizained Models
87.5
57.5
Baseline Parato
Entropy
Max Sofımax
82.5
Sullmux Sum
K2.5
Mean Logits
40.0
10a
10
Average Inference MAC
Improvement Factor
Figure 9: For some applications, such content autocomplete recommendation, it is less im- portant that the top-1 prediction is correct and instead it matters more that the correct prediction is among the top. To investigate this scenario, MAC. 2-model cascades perform better at the lower range but worse at the higher range for top-5 accuracy in spite of the smaller accuracy differences. Here, the entropy confidence metric beats maximum softmax since takes into account more than the top-1 prediction. However, the top-5 softmax sum, which takes into account all relevant probabilities, outperforms entropy across most of the Pareto front. In an attempt to improve the cascading further, we try mean ensembling. However, neither the mean logits nor the mean softmax (not pictured) method perform well outside of narrow points at the upper limit of the accuracy range.
evaluate the trade-off between top-5 accuracy and
C CONFIDENCE METRIC FOR MEAN ENSEMBLES
89
88
ImageNet Validation Top-1
87
86
85
84
Max Softmax Mean Logits
Max Softmax Moan Softmax
83
Softmax Margin Mean Logits
Softmax Margin Moan Softmax
82
Improvement Factor
Figure 10: comparison of confidence metrics for mean ensembled 2-model cascades the high accuracy regime, where they perform best. While the difference is very small, the softmax margin confidence metric is better overall than maximum softmax for mean ensembled cascades. However, maximum softmax confidence leads better improvement in the high accuracy regime, where mean ensembles perform best, and is therefore the metric we selected.
15


Reviews:
[{"review_id": "qlnpG7saIcD", "reviewer": "Reviewer_Q4Xh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper discusses conditional execution of models from the early-exiting angle; showing that a 2-cascade of models can improve upon baseline performance of executing a single model across the entire pareto front in terms of efficiency and accuracy. The paper takes several combinations of 2 models, and picks good ones, where a threshold is learned to switch from execution from the simple model to a more complex model. The paper also evaluates several techniques for doing the thresholding/cascading. ", "summary_of_the_review": "Reading this paper knowing about conditional computing and early exiting, it does not provide any novel insights. It does reconfirm something that scholars in this area already know, but provides little new insight to the discourse. It has an interesting experimental setup, comparing cascaded ensembles of many existing networks, but this setup will likely not be optimal compared to the more common setup of a monolithic model being early exited from.\n\nAll in all, since the paper does not meaningfully contribute to my understanding of early-exiting, nor provides a new method on how to make more efficient networks, I would reject this paper. ", "strength_and_weaknesses": "Strengths:\n- The paper is well written\n- It is always good to have more results for topics like this, reaffirming what is commonly known through literature.\n\nWeaknesses\n- The paper essentially just reconfirms something that is very well known in the conditional computing literature. Early-exiting/cascading helps significantly in models to create a better efficiency trade-off. This has been shown in the past in many studies; although generally at least some part of the network is shared as not to recompute simpler features at the start of the network. Even in relatively new language and vision transformers, many variants of early-exiting have been discussed. For this specific version of early exiting, within a single model, there's already 7 papers out on this topic, showing that it helps to early exit simpler examples.\n- This paper really does nothing new on top of what is already known. Although it's good to reconfirm what is known, there is nothing novel in this paper. There are no new insights in this paper worth mentioning. \n- The setup is quite archaic, compared to the more commonly used set-up where at least some features are shared, as opposed to using an entirely new network. I don't think it's relevant to show results in this setup anymore, as it is not a very efficient setup to do  early exiting on. \n- The fact that temperature thresholding does not work very well has been discussed at length in many early-exiting papers, and things like 'patience' metrics have become more popular for doing early exiting in monolithic networks. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "RK4RqmKfmuy", "reviewer": "Reviewer_J6Fm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work explores model cascades as a way to improve the accuracy-efficiency trade-off of deep neural networks. They conduct experiments to evaluate different design choices in model cascades, such as the confidence metric, number of models in the cascade, and whether to calibrate the logits before ensembling. Experimental results on image classification and text classification are provided to support the analysis of model cascades.\n", "summary_of_the_review": "This work does not propose new techniques but rather analyze the various design choices of an existing method (model cascades). This is fine as long as the analysis results are informative. However, considering a very similar prior work [A], the contribution of this work is unclear and seems to be modest. So I vote for rejection.\n", "strength_and_weaknesses": "The results presented in this work largely make sense to me. But a major concern is that the difference between this work and previous work [A] is unclear. Many conclusions presented in this work were already verified in [A] except that [A] focuses on image classification while this work provides both image & text classification results.\n\nFor example, the first contribution “2-model cascades dominate the accuracy-compute Pareto front” was shown in Figure 7 in [A]. The comparison of different confidence metrics (entropy, margin, etc) is similar to Figure 3 in [A].\n\nTherefore, I am unsure about the new value this work could add to the existing knowledge about model cascades.\n\n[A] Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models. ICLR 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "IdYcw5BPDu", "reviewer": "Reviewer_Bbfc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper conducts empirical investigation regarding the best practice of building cascade models. Specifically, the paper investigated the impact of early-exist condition and the choice of model calibration on the efficiency-accuracy performance of the cascade, and investigated the performance of both vision and text models. The experiments reveals that the maximum softmax achieves most improvement overall, and temperature scaled calibration hurts performance.", "summary_of_the_review": "As mentioned in the weakness, the paper investigated some common obvious choices (in terms of choice of confidence metric and calibration procedure) and provided some empirically-supported conclusions. Otherwise, the cascade model setup is not too different from previous work (e.g., [Wang et al. 2022](https://openreview.net/pdf?id=MvO2t0vbs4-)). \nWhile it is useful to share empirical results, I feel the submission in its current form is sufficient as a high-quality workshop submission, but does not carry sufficient scientific significance to justify a publication at top venue like ICLR.", "strength_and_weaknesses": "Strength: \n* A comprehensive and careful study of some key design decisions of cascade models.\n* Extended evaluation to the text domain.\n\nWeakness:\n* Novelty is a bit limited: the paper investigated some common obvious choices (in terms of choice of confidence metric and calibration procedure) and provided some empirically-supported conclusions. Otherwise, the cascade model itself is not too different from previous work (e.g., [Wang et al. 2022](https://openreview.net/pdf?id=MvO2t0vbs4-)). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ft9V9sPDb18", "reviewer": "Reviewer_CNbJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors conduct a thorough investigation of cascading methods with deep neural networks, including threshold tuning, cascades of various depths, heterogenous cascades with different types of architectures and methods of ensembling predictions when early exiting is not possible. The authors demonstrate the efficacy of these techniques for both vision and text tasks.", "summary_of_the_review": "This paper is well written, well organized and thorough in their analysis of cascading methods for deep neural networks. The results are novel and potentially impactful.", "strength_and_weaknesses": "I found the paper to be clear, focused and interesting. One strength of the study is their experiment design - in particular, taking advantage of the large number of pre-trained models available through frameworks like TIMM and HuggingFace to answer questions about cascading methods convincingly. The insights from the study are also impactful, with 2-5x reductions in computation and up to 3x reductions in measured inference time.\n\nOne question I had was about the precision of the thresholds used. In each of the tables I see four digits of precision used for softmax thresholds. Is this level of precision necessary? What would the results look like if fewer digits are used?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject