Agreement-Based Cascading for Efficient Inference
Authors: Steven Kolawole
Don Dennis
Ameet Talwalkar
Virginia Smith
Venue: Accepted by TMLR ()

Abstract: Adaptive inference schemes reduce the cost of machine learning inference by assigning smaller models to easier examples, attempting to avoid invocation of larger models when possible. In this work we explore a simple, effective adaptive inference technique we term Agreement-Based Cascading (ABC). ABC builds a cascade of models of increasing size/complexity and uses agreement between ensembles of models at each level of the cascade as a basis for data-dependent routing. Although ensemble execution introduces additional expense, we show that these costs can be easily offset in practice due to large expected differences in model sizes, parallel inference execution capabilities, and accuracy benefits of ensembling. We examine ABC theoretically and empirically in terms of these parameters, showing that the approach can reliably act as a drop-in replacement for existing models and surpass the best single model it aims to replace in terms of both efficiency and accuracy. Additionally, we explore the performance of ABC relative to existing cascading methods in three common scenarios: (1) edge-to-cloud inference, where ABC reduces communication costs by up to 14x; (2) cloud-based model serving, where it achieves a 3x reduction in rental costs; and (3) inference via model API services, where ABC achieves a 2-25x reduction in average price per token/request relative to state-of-the-art LLM cascades.

Published in Transactions on Machine Learning Research (07/2025)
Agreement-Based Cascading for Efficient Inference
Steven Kolawole*
skolawol@anddrew. www. edu
Cornegie Mellon University
Don Dennis" Cornegie Mellon University
dondennii@@omau.o edu
Amect Talwalkar Cornegie Mellon University
labualkar Deciaa eduu
Virginia Smith
smithv@emaacom edu
Carnegie Mellon University
Reviewed on OpenReview: https: // openreview. net/ www//frwueeeeeeeeeee forum? id= jn9B7MMzz
Abstract
Adaptive inference schemes reduce the cost of machine learning inference by assigning smaller models to easier examples, attempting to avoid invocation of larger models when possible. In this work we explore a simple, effective adaptive inference technique we term Agreement- Based Cascading (ABC). ABC builds a cascade of models of increasing tize/complextty and uses agreement between ensembles of models at each level of the cascade as a. basis for data-dependent routing. Although ensemble execution introduces additional expense, we show that these costs can be casily offsct in practice due to large expected differences in model sizes, parallel inference execution capabilities, and accuracy benefits of ensembling, We examine ABC theoretically and empirically in terms of these parameters, showing that the approach can reliably act as drop-in replacement for existing models and surpass the best single model it aims to replace in terms of both efficiency and accuracy. Additionally, we explore the performance ABC relative to existing cascading methods in three common scenarios: (1) edge-to-cloud inference, where ABC reduces communication costs by up to 14x; (2) cloud-based model serving, where it achieves 3x reduction in rental costs; and (3) inference via model API services, where ABC achieves 2-25x reduction in average price per token/requee relative to state-of-tho-art LLM cascades.
1 Introduction
The high cost of inference associated with deploying large machine learning (ML) models presents a significant barrier to their adoption (Strubell al., 2020 Kaplan al., 2020). As models continue to increase in size, practitioners often faced with investing substantial resources in updating existing deployments settling for lower-performing alternatives. However, for many applications, it has been shown that considerable portion of the data seen during inference can be effectively evaluated using small models rather than large, state-of-the-art models (Chen et al.) 2020 Jitkrittum et al., 2023). This means that if we can identify the subsct of data samples that can be accurately evaluated by more incxpensive modcls, average inference costs can be reduced. This problem often referred to as adaptive inference, where the cost inference adapts to some notion of 'difficulty' of each example seen inference time.
A natural approach for adaptive inference is cascade over of potential models, starting from the least. expensive and moving to more expensive models based some deferral rule (Rowley et al. 1998 Viola & Jones 2001 Soo 2014). A common cascade construction is to use Pareto-efficient set of models and an
'Equal contribution.

---

Published in Transactions on Machine Learning Research (07/2025)
ABC
ABC
largest model ABC
ImageNetl K: Accuracy Scaling vs FLOPs
100
Annurayy (99)
ABC prediction
60
TorchVision HuggingFace
Privatc repo
no
conostere?
no
40
96
Open-CLiP
20
yez ABC prediction
yes
Pareta freetier
ABC, prodiciin
316M
0
1B
3B
10B
32B
100B
316B
FLOPs (log (oogssrale)
(a)
(b)
(a) Agreement-Basee Cascading (ABC): ABC introduces data-dependent routing scheme that uses agreement amongst an ensemble of models to determine whether to cascade to larger models. If the predictions of smaller ensembles do not align, the cascade moves to the next tier of larger models, continuing until agreement is reached or the largest model(s) are used. This can reduce cost by limiting the use of the largest models to cases where smaller models cannot reach consensus. (b) ABC is natural baseline for adaptive inference due to (i) the vast number of pretrained models available to ML practitioners today; (ii) the fact that even small accuracy gains often require an onder-of-maagituude increase in FLOPs, mirroring proposed scaling laws and resulting in large differences in model sizes between cascade tiers (Hestness et al.] 2017 Henighau et al.) 2020 [adaan et al.) 2023). The pink dashed line represents the Pareto-optimal frontier, showing the models with the highest accuracy for given computational budget. We show that ABC can effectively improve this frontier pontier-allowing practitioners to achieve high accuracy without incurring the full computational cost of the largest models by invoking smaller models for 'easier' samples.
Figure 1:
easy-to-compute deferral rule such as the confidence scores of the models predictions (Viola & Jones 2004
Wang et al. 2018a 2021). Recently, cascading has seen renewed interest in the ML community with the Cc.. 2024 Recent approaches consider variety of techniques to construct model cascades, including designing
advent of large language models (LLMs) and vision foundation models (e.g.. Chen et al. 2023 Gupta et al.,
novel model architectures that have built-in cascading capabilities (Cai et al.) 2019 Devvrit et al., 2023
Khare et al. 2023), or learning routing schemes/ chheess/deferral deferral rules that require data-dependent training for every task considered (Chen et al., 2023 Ding et al.) 2024). While these approaches can improve accuracy sairracc/fffiiencyy tradc-offs, they may also introduce significant computational overhead in setup and
/aaiii/ggginntnnnggggg costs.
In this work, we instead investigate a simple, training-free cascade scheme that uses the agreement among an ensemble of existing models at each cascade level as its deferral rule. We refer to this approach as Agreement- Baseeeent-Based Cascading (ABC). Intuitively, when the outputs of the ensemble at a particular cascado-level do not align, ABC triggers cascading, and inference is attempted at the next tier of (larger) models (see Figure 1. This scheme has been explored historically for specialized applications such as f'ace detection (Zuo & de With 2005 2008 Susnjak et al., 2012), where ensembles of binary face detectors focus on subregions of the face and combine their output using decision networks. However, to the best of our knowledge, our work is the first to study the use of ensemble agreement as a deferral mechanism for more general, modern machine learning workloads.
We point to few trends in ML that make ABC a particularly attractive approach for model cascading. In particular, while using an ensemble of models at each level may initially appear to increase overall inference
costs, it is natural to believe this simple bascline could excel in real-world applications due to: (1) the growing ease of obtaining pretrained models of various sizes and accuracies due to the rise of model registries (Wolf
let al. 2019) as well as flexible compression schernes (Zhu et al., 2023 Dennis et al. 2023): (2) scaling laws
that predict a large increase inference cost for every marginal increase in accuracy (Hestness et al., 2017
(Figure D); and (3) an increased ability to execute ensembles of relatively small models in parallel, with
minimal additional costs (Fern& & Givan 2003:
Kim el al. 2023
Miao et al.] 2023 (Figure
With these motivations in mind, we rigorously study ABC as bascline for adaptive inference in modern ML workloads. Overall, we make the following contributions:
We propose ABC as a training-free, deferral-based cascading approach that leverages agreement among ensembles of existing models, removing the need for additional routing networks or specialized architectures.
We theoretically characterize cases where ABC can replace existing model deployment without affecting the accuracy, and define safe deferral rules as sufficiency conditions for their existence. We further characterize the expected accuracy and inference costs in this setting.
We empirically evaluate ABC on a wide range of image and language tasks and find that ABC not only improves efficiency, but also accuracy, compared to the model that it aims to replace. We then consider
2

---

Published in Transactions on Machine Learning Research (07/)025)
the performance of ABC relative to existing cascading methods in common inference scenarios, including (1) edge-to-cloud inference where ABC reduces communication costs by up to 14x, (2) model-serving on heterogeneous GPUs, where ABC reduces rental costs by up to 3x and (3) inference using black-box access to model API services, where ABC shows up to a 25x reduction in average price per token.
2 Related Work
Adaptive inference schemes have been a topic of interest in machine learning for many years. This section discusses three main approaches to cascading and adaptive inference: score-based methods, trained routers, and dynamic networks. We highlight how ABC's approach relates to and differs from these existing methods.
2.1 Cascading using Score- based Deferrals
Traditional cascading methods often rely on simple score-based metrics for deferral decisions. We compare to (2021) as a general, representative method in this category but note that specialized instantiations of this approach have been applied across various domains in
the recent. Wisdom-of-CComiitees method of [Wang et al.
prior work, for instance, object detection (Rowley et al.
1998 Viola & Jones 2004 Wang et al.
2011 Cai
et al. 2015 Angelova et al. 2015 Streeter
2018). image classification (Wang et al. 2018a| 2021
and text
classification (Li et al.) 2021b Mamou et al.)
2022
Varshney & Baral
2023 In these 2022 Lebovitz et al.
settings, deferrals within cascading systems use metrics such as conlidence scores, entropy, and probabilities
(Gangrade et al.) 2021 Geifman & El-
2019
Narasimhan et al. 2022). While using confidence scores
of existing models is inexpensive, these scores are required to be well-calibrated, which is less common for off-the-shelf models (Guo ct al.) 2017 Enomoro & Eda 2021). Recent methods have thus considered a varioty of techniques to enhance score-based deferrals, such as explicitly adding deferrals as a prediction (Wang et al. 2018a), difficulty-wware regularization (Li et al.] 2021b), temperature scaling (Wang al.) 2023). and
calibration mechanisms (Nic et al. 2024
Recently, Jitkrittun et al. 2023 explored the specific practical setlings under which confidennce-ased deferral in cascades could suffer; these settings include: scenarios where downstream models are specialists (the error probability of the downstream model is highly non-uniform across samples), when samples are subject to label noise, and in the presence of a distribution shift between the train and test set. Notably, in all these failure modes, where confidence-based cascades tend be inadequate, our approach is likely to offer improvements, as ensembles are known help induce diversity, enable robustness noise, and mitigate issues of distribution
shift (Gontijo-Lopes et al. 2022 Sharkey 1996; Dietterich 2000 Džcroski & Zenko 2004
2.2 Cascades with Routing Procedures
Other methods avoid confidence scores entirely by training procedures independent of the model set to route instances as needed. This trend has gained prominence with the rise of black-box model API services for LLMs, where prediction scores are often unavailable. Guan et al. (2018) developed selection module trained to determine the best-fit classifiers for data instances. Yue al. (2024) (MoT LLM Cascade) used sampling and consistency checking to determine when to defer to high-cost. model. AutoMix (Madaan let al. 2023) proposed few-shot self-verification mechanism, similar to Yue et al. (0024). but also introduced a Mlrrkov-base meta-verifier for cascading in context-groe grotude-roonned tasks. FrugaIGPT (Chen et al. 2023 leveraged a cascade strategy that triages incoming queries using DistilBERT router and scoring function. HybridLLM (Ding et al., 2024) usod fine-tuned DeBERTa (He et al., 2020) to route queries to models basod on the predicted query dilficulty and the desired quality level. On the same note, OrchestraLLM (Lee el. al., 2023 using hand-labeled data to create expert model pools-selecte, selects model to query based on embedding distances between the pools' instances and the test instances. Other methods also used some form of trained router, including RouteLLM Ong et al. 2024 'Fly-swat or cannon' (Sakota et al. 2024h, and Shnitzer et al. (2023)
As we further discuss in Section 5.2.3 (where we empirically compare to FrugalGPT, AutoMix, and MoT LLM Cascade), these more sophisticated methods involve complex setups, data-dependent training, and increased computational overhead. They often require retraining routers for each new task, dataset, or model,
3

---

Published in Transactions on Machine Learning Research (07/2025)
limiting adaptability to unseen data distributions new models without incurring further costs. In contrast, ABC provides simpler, flexible, widely applicable alternative at no additional training or setup cost.
2.3 Dynamic and Adaptive Networks
Routing procedures typically train an independent routing mechanism, keeping the Pareto-set of models untouched. However, for many medium-scale applications, methods have been explored that learn Parcto-sct of models and slopping criterion logether. For example, early exit methods of Bolukbasi et al. 201 7 Huang
et al.
(2018); Shafiee et al.
(2018): Wang et al.
20188b; Hu et al. (2020): : Xin et al. (2020); Geng et al.
(2021
Zhou ct al. (2020); Liu ct al. (2020): Schuster ct al. (2022). subnetworks extraction methods (Yu
ct al. 2018 Yu & Huang 2019 Chen et al.
2021
Hou et al. 2020 Han et al. 2022: Devvrit et al. 2023).
composition-basee
methods (Suggala et al.
2020 Dennis et al. 2023 Du & Kaelbling 2024) are some of
the more recent examples. While these methods achieve impressive efficiency gains, they require training
specialized architectures from scratch with adaptive capabilities.
Our approach diverges from these methods, as it does not involve altering model architectures or retraining from scratch. Instead, ABC leverages existing models as they are, utilizing ensemble agreement as a deferral condition, which can be applied directly to these models without any need for fine-tuning or specialized training.
3 Agreement-Based Cascading (ABC)
In this section, we present a high-level overview of the Bereement-Bassee Cascading (ABC) approach, providing the essential concepts and insights needed to understand ABC 's effectiveness in the experiments (55). Readers interested in the formalizations and technical details can refer to $ for a more comprehensive treatment.
3.1 Overview of ABC Approach
As outlined in Section 1, the goal of adaptive inference is to identify data samples that can be evaluated accurately by a relatively incxpensive model. This way, we can reduce the inference cost based on whether a sample 'easy' or 'hard'. A common approach to this problem is deferral-based cascading, where we cascade over a sequence of models, starting from the least expensive and using a 'deferral rule' to determine if a higher tier model must be used. These deferral rules are typically significantly cheaper evaluate (e.g. a small neural network), and add very little additional cost top of model execution. This approach makes it so that 'simpler' cases are managed by smaller, faster models, while only the complex cases cascade up multiple tiers and to more resource-intensin models.
Agreement- Based Cascading (ABC), described in Algorithm I is one such deferral based cascading approach. ABC maintains a set of ensembles {HI...., H.g} that starts from an ensemble of inexpensive models in H1 to expensive, statc-of-tho-art models Hng- Similar other cascading techniques, given a sample x, we start inference from the lowest (cheapest) lier in the cascade and use a deferral rule, rı(x), to determine if higher tier model is needed. A core distinction between ABC and existing cascading approaches is its baree deferral rule. Instead of training small additional deferral network or post-hoc deferral rules, we use notion of agreement between models within each ensemble as a confidence measure. When a (configurable) fraction of models in a tier agree, it signals that they likely have the right answer. Conversely, if they disagree, the uncertainty triggers a deferral to the next, more powerful ensemble. Additionally, we focus on cases where we wish to maintain accuracy of the overall prediction and only improve inference cost when accuracy does not suffer. This is different from existing approaches that prioritize fixed inference budget, potentially at the cost of accuracy.
3.2 ABC in Modern ML Environments
In recent years, obtaining trained models has become much casier, thanks the rise of public and private model repositories (Wol al. 2 where a wide variety of pre-trained models are readily available. This abundance of accessible models of various sizes and performance levels makes the practical application of
4

---

Published in Transactions on Machine Learning Research (07/2025)
Algorithm Agreement-Bane Cascading (ABC)
Require:
Set of ensembles {Hi, H2, ..,,,,,, Hng) deferral rule r; for each ensemble E [mx]] as in Equation 3
or g
Require: A new inference data point
1: Current cascade level, i 1-1
2: Cascaded prediction, x- 0
3: for E [l....,,5t ng} do
4: y - (x)
if (x) = then
break
{Models in ensemble 'agree')
end if
end for return
ABC especially appealing. Unlike methods that require additional task-specific training or fine-tuning, ABC can leverage these existing models, making suitable for direct deployment as a "drop-in" replacement for high-cost models. This adaptability, combined with minimal setup and training requirements, can allow ABC to fit seamlessly into many existing ML workflows.
Strictly speaking, evaluating agreement between multiple models in an ensemble is expensive when compared to the small router models that are used in many existing approaches. However, two key aspects of modern ML workloads can help to mitigate this additional cost in practice. First, in many cases some degree of parallelization available that can reduce the impact on inference cost metrics such as latency. We use to smoothly interpolate between the fully sequential case (p = 0) to the fully parallel case, (p= as detailed $1.3 Second, in many use-cases the difference in cost between models in successive tiers of cascades is large that the impact of lower tier models the overall cost of inference is negligible, even with the added cost of constructing ensembles (see Figure 3 We use y denote the relative cost of the models - the ratio of the cost of the smaller model to the larger model.
In $5 we evaluate ABC 's performance across several real-world scenarios (resulting in varying settings of and 7), demonstrating its competitiveness and efficiency. We first show that ABC's ensemble-base deferral mechanism preserves and often improves on the accuracy of models aims to replace. We also show that when the lower tier models are small enough compared the higher tier models, ABC 's ensemble-bass deferral rule operates with negligible impact the overall cost.
We then evaluate cases in real-world workloads where such large relative costs between various tiers of ABC is natural. In edge-to-cloud setups, ABC enables substantial reductions in communication costs by processing simple tasks locally on the edge. This minimizes the need to transfer data to the cloud, reducing both latency and data transfer expenses. When serving models on heterogeneous GPU resources in the cloud, ABC significantly reduces inference costs. By selecting models based on task complexity, ABC makes efficient use of available GPU resources, optimizing for both accuracy and rental cost. In black-box API-based model deployments, where the user is billed per request or token, ABC offers substantial savings by reducing the average cost per request. By deferring to high-cost models only when necessary, ABC achieves notable economic savings compared to standard inference approaches.
4 Formalization and Theoretical Details
In this section, formalize our problem setup by first outlining the standard statistical learning framework, which serves to define the concepts of models, ensembles, the lcarning performance (c.g., accuracy) and the inference cost associated with ensembles (4..). Next, we describe deferral rule-based cascades and formulate the specific case of drop-in coscades, cascades which prioritise accuracy over inference cost savings ($4.2). We introduce safe deferral rules as sufficiency condition for constructing drop-in cascades and define our deferral rule based on agreement between models in an ensemble ($4.3). We conclude this section by formalizing the tist-time accuracy-inference cost behaviour of drop-in cascades 4.4
5

---

Published in Transactions on Machine Learning Research (07/ (225))
4.1 Problem Setup and Notation
Consider the standard statistical learning settting where X' denotes an instance space and y denotes the label or response space, Let h(z) be a model taking imputs from X' and producing outputs in y. We assume that the models are from some hypothesis class H. In this sctup, we typically characterize the learning performance of varions models using its risk with respect to some data distribution D over x y and a loss function 1, given by,
R₂(h) ((..)))DD(((()))..)))))))))))<)) v)).
As concrete example, in the classification setup, where we use the mis-classification error as loss function, the risk is given by (h) = Ely # h(x)).
Assume that each model h H. has cost of inference, denoted by C: H + R+; for instance, for cases where we are concerned about inference latency, the cost can be the latency of the model on the target hardware, Let h1 and h2 be two models with h2 being the more expensive of the two. We denote the relative cost of the ((h)) satisfying 0 < 7 <151. models by 7= ((ha))
Let Hh X + y denote an ensemble of l models from the same hypothesis class H. Similarly to before, the learning performance of various ensembles can be characterized by their risk; let R(H+*) denote the risk of ensemble H² Compared to single member model, the cost of evaluating the entire ensemble of models depends the various factors, including the degree of parallelization. Assuming that the models in an ensemble are of similar cost, say co. we model the cost of the ensemble using parallelism coefficient 0 < < as,
C(H²)= = cph
(1)
Here, when p p= 1, the ensemble suffers the same cost as a single model and corresponds to the case where models can be fully parallelized. On the other extreme, at p p= O, the cost of the ensemble of l models is kco, corresponding to no parallelization (sequential evaluation).
4.2 Deferral Based Drop-in Cascade
A deferral-based cascade consists of finite set of models and a deferral rule. Here, the idea is start with the most resource-efficient model and use the deferral rule to determine if a netter-performing model with a higher resource cost should be used for the current sample. The deferral rules themselves are designed to have negligible inference cost given the models inference outputs. For this exposition, we will restrict ourselves to cascades with only two levels, though the discussion in this section can be readily generalized to larger cascades. Let the cascade consist of an ensemble H at the lower level, and the larger model h2 at the higher level; M = { { h2]. Let r(z) denote a deforral rule;
{ defer to h2
r(x)
use H.P
In this work, we focus deployment scenarios where a decrease in model performance is critical concern. This is particularly relevant when cascades are intended to function as drop-in replacement for an existing deployment of an expensive model. This notion can be formulated as maximizing the number of calls to the smaller model while retaining the accuracy of the larger model. With a slight abuse of notation, let M,(x) denote the prediction of the cascade with M {H} h2) and deferral rule r. Then we desire that for a choice of small error budget. ę 0,
max P(r(x) = 0)
s.t. P(y + M,(x)) & Fly # hz(x)) + +
(2)
The constant function r(z) 1, which defers for every x, feasible and attains the objective value of 0. Moreover, every feasible deferral rule leads to cascade with competitive accuracy as h2 Instead of picking

---

Published in Transactions on Machine Learning Research (07/2025)
error budget &, complimentary (dual) approach is to consider fixed inference budget and aim to attain the best accuracy within the budget. In this view, drops in accuracy (compared to the existing large model) are permissible, provided the inference budget is met. Jitkrittum et al. (2023) examine this perspective, which we recommend to interested readers.
4.3 Deferral using Ensemble Agreement
Ensemble agreement is an instantiation of score based deferral rules (Jitkrittum et al. 2023). In its simplest form, we associate a score to the prediction output of an ensemble model and interpret this score as a measure of confidence in that particular prediction. We can then defer to a larger model when the confidence is below predetermined threshold. For a model Hf (z) and a scoring function s(x), the deferral rule is defined as
r(x) = T[s(x) s {
where is the indicator function. In general, such score based deferral rules can sometimes be misleading. For instance, in multi-class classification, rules that map the outputs a classifier to a probability distribution over labels can produce confidently incorrect predictions when an umperceivable amount of perturbation
is added to the data sample [Szegedy] 2013). However, such rules have been shown to be effective in practice (Wang et all 2018a; Gangrade et al. 2021 Mamou et al. 2022 Gupta et al. 2024) implying that such adversarial data is rare in many cases. Motivated by this observation, we propose the following property of scoring functions (and by extension the corresponding deferral rule).
Definition 4.1 (Safe deferral rule). Let H :X + y, k 2 1 be a classifier and let s: X + [0, 1] be a scoring function. The scoring function 8 is referved to as a safe scoring function for H if there erists [0. 1] such that, for some small c > 0,
P(s(x) 2 0, (11h(())
(x) Fv) fu)se Se.
The corresponding deferval rule r(x) = [s(x) S 0), is referved to as a safe defervul rule, and sulisfies P(r(x) = 0, + $) S..
We define safe deferral with respect to classification tasks for simplicity. The definition can be extended to other cases with appropriate choice of loss function. Intuitively, this formalizes the notion that the deferral rule can probabilistically identify a subset of X' x y for which (x)) is correct. Alternatively, we can think of the deferral rule as a one sided classifier, similar to the problem studied in Goyal et al. (2020), where the class to be identified is the subset of X' x y where (f) is correct. In general, such a safe deferral rule need not exist for a given model (x). However, in $5 we demonstrate empirically that such rules can in-fact be constructed without additional training for many real world tasks with appropriate choice of 0 (Figure 7 These rules lead to selection rates as high as 90% of the data in cases such ImageNet-1KK.
Assuming access to safe deferral rule, observe that for & 2 <> O, such rules are feasible for the program in Equation 2 (see Proposition 4.1) This implies that every cascade, M + h22 using a safe deferral rule r, is competitive with the larger model hz in terms of accuracy. We empirically evaluate these drop-in cascades in 5 Note that the optimal safe deferral rule can lead to an improvement in the accuracy of ABC in theory (see, Appendix A and we see accuracy improvements in our experiments ($5)
basee deferral rule We evaluate two flavors of deferral rules that capture the notion of agreement between models. In cases where we have direct access to the models, we directly use the outputs produced by models in the ensemble. However, in certain cases, for instance when interfacing through a third-party provider's inference API, we only have black box access to models. For such cases, we use a voting scheme between the models in an ensemble to construct our scoring function. Concretely, for an ensemble H consisting of & > 1 models, let s(x; H&) denote the average score of the majority prediction on x and let (2; H²) = Enens I[ (x) = h((z)) denote the fraction of votes received by the prediction, where,
{
1 vote(æ; H+) s A.
rv(2;0,)
(3)
otherwise.
s(x; ((); S 0z otherwise.
ra(2; 0.)
(4)
7

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Comparison on CIFAR-10
Comparison imagenet-1K
Comparison on SWAG
Comparison SST-2
06
96
Top 1901.6 1.Acc
92
964
10p LAcc
84
Top- Ace
94
263
62
J
60
WoC
Best atggle modals
10M 3GM 100M 316M
3B
4B
4B
16B
408
1008
4B
10B
168
(logsaale)
FL.Pss (log-scalie)
FLOPss (log-arala)
(log-acale)
Figure 2: Pareto curves of ABC vs. confidence-basee cascades (WoC) (Wang et al. 2021 and best single models on diverse tasks, For WoC, we tune cascade configurations across the best four confidence thresholds and generate results from their most performant cascades. ABC maintains Pareto-optinn optimal curve, which consistently oulperforms both methods in accuracy with lower FLOPs costs.
4.4 Inference Cost Savings and Competitiveness
Since we do not impose an inference cost budget on the cascade, it is possible for the cascade to incur a higher cost than simply using the larger model h2g Intuitively, if majority of the samples seen during test. time are 'hard', the cascade pays the cost of evaluating both H (x) and h2(x).
Proposition 4.1. Let M {1 h2e be two classifiers and r a deferral rule such that r is safe deferral rule for Hi according to Definition 4.1 for distribution IP over X x y. Then for every & 2 0, the agreement based cascading (ABC) classifier M,(x) is such thal,
The ABC classifier competitive with the large classifier h2 in terms of accuracy (zero-one loss),
R(M,) S R(h2) +
The ABC classifier enjoys on average inference cost of,
B[C(M,)) P(())
The proof follows directly from Definition 4.1 and basic probability (see Appendix A Safe deferral rules can be constructed with minimal cost: our threshold estimation requires only ~100 validation samples and simple voting computations, avoiding expensive router training. Thus, the cost savings we can expect with a drop-in cascade depend on three key factors: the relative cost 7, the degree of parallelization and the deferral rate P(r(x) = 1), or equivalently, the selection rate P(r(x) = 0). In particular, for every feasible deforral rule r, in the best case scenario where the cost of the smaller model is negligible (i.c., 7 = 0), the cost of inference reduces by the selection rate, P(r(x) = 0). Conversely, in the worst-case scenarios, the cost. can be (k + 1) times the cost of the larger model. Tn the next section, we demonstrate real world scenarios where the favorable interplay of these three quantities lead to significant improvements in inference cost. See Appendix C and Figure for information on selection rates for various models and datasets considered here.
5 Experiments
This section evaluates training-free Agreement-Based Cascading (ABC) approach across a variety of language and vision tasks, focusing accuracy and inference cost. First, in $5.1.1 examine ABC's accuracy-cost tradeoff against state-of-the-art models under full parallelization, setting the porallelization factor (p) to 1.0 for optimal tradeoff. Next, in $5.1.2 we show that using ensembles at lower tiers has minimal impact on inference cost when lower-tier models are at least 50x cheaper than the largest model. In such cases, the relative cost (7) satisfies 7 S 50 making ABC effective without parallelization. Finally, $5.2 explores practical scenarios where low relative costs make ABC naturally suited for real-world deployments.
Estimating Voting Threshold: ABC's deferral rule uses a configurable voting threshold, 0 (sce Equations and 4 at each cascading tier. We estimate 0 empirically a small set of unseen data; see App. B for details.
8

---

Published in Transactions on Machine Learning Research (07/2025)
To evaluate ABC, we use a range of benchmark datasets for image and language tasks, shown in Table 2 in the Appendix. Additional datasets are used in 5.2.3 to align with those explored by state-of-the-art baselines.
Datasets:
We select diverse models for both image and language lasks, summarized in Appendix's Table 3 For BERT-based models, we use the BASE and LARGE, and for image models, we tier by FLOPs count. All models are sourced from HuggingFace Zoo for inference without any additional training effort on our end. $.2.3 uses models from LLaMA 3, Gemma 2, and Qwen 2 families via the Together API. We detail this section's 's experimental setup in App. D.2
Models:
Evaluation: For the generalion tasks we consider in 5.2.3 the dalasets each consist of a fixed set of possible outputs akin to the classification tasks, and we apply our deferral rule on the final output at each tier. Certain cases don't have a fixed set of output labels (while still not being open-ended generation), like (1) GSMSK, where there is a final numeric answer at the end of the generated output to the math word problems, and (2) CoQA, where we used F1-score to capture overlaps between predictions and ground-truth answers. We do not discuss open-ended generation tasks in this work as the deferral rule we consider and the baseline methods that we compare to is not directly applicable to such cases.
5.1 When is ABC Practical?
5.1.1 When Parallelization is Cheap
We first consider the case where the inference cost of an ensemble of models is the same as the cost of a single model. This idealistic scenario could happen, for instance, in offline batch inference (Jetty et al.] 2021), when GPUs are available for parallelization, or the models are small enough that existing resources are under-utilized and an ensemble adds no additional cost. More importantly, this sctting establishes best-case accuracy and inference cost values for ABC. We consider total floating-poinl operations (FLOPs) reprosentative inference cost metric here, and discuss other metrics such as communication cost, latency, cloud rental costs in subsequent subsections.
The accuracy vs. FLOPs for ABC is shown in Fig- ure 2 As a point of comparison, we also include Besdom-of-Committaee
ImageNet1K: Cost savings Relative Cost
1.0
sequential/paralle at
(WoC) (Wang et al.) 2021), popular and representative confidence- based model cascading method. In most cases, observe that ABC is able to improve the Pareto frontiers, it usually sees 1 2 point increase in accuracy. We attribute this to a combination of (a) the improvement ensembles can have on accu- racy that is widely noted in literature (Gontijo- Lopes et al. 2022: Jiang et al., 2023) and (2) the improvement agreement- based rules can cause in cascading (Appendix A). Overall, in terms of ac- curacy, ABC either exceeds-4d or at least matches the accuracy of the best models at fixed FLOPs budget, and is a practical drop-in replacement for these models.
Frac. cost saved
0.8
p(r(x)=
0.6
Parallel (0=1.0)
Sequential (p=0.0)
0.4
p= 0.25
0.2
p= 0.5
0.75
0.0
1
1
500
200
100
20 10
5
Relative Costs (y)
Figure Fraction of inference cost saved as function of relative cost of models (7), assuming fixed selection rate P(r(x) = 0). As parallelization decreases from fully parallel (p 1) to sequential (p 0), cost of evaluating ensembles increase and the cost savings decrease. When models across tiers are of similar size (e.g., smaller model is at most 5x smaller, ? 2 }),, some paralleiization reduce costs effectively. However, for lower relative costs (e.g., smaller model is at least 50x smaller, S 1.), sequential and parallel settings achieve meaningful savings, showing ABC 's efficiency even with the added cost of using ensembles.
5.1.2 Disparity in Relative Cost High
needed for ABC to
Of course, in many real-world scenarios, the cost of evaluating ensembles is not negligible. For instance, in the edge-to-cloud inference scenario discussed in $5.2.I models in an ensemble often are ovaluated sequentially. However, even in such scenarios, if the relative cost of models across each level of the cascade, 7, is small enough, this additional cost becomes negligible compared to the overall cost.
9

---

Published in Transactions on Machine Learning Research (07 (025)
Average Latency Costs
Total GPU Usage Cost Comparison
1000
CPU Cost ($/hour)
ABC
Latency (ms)
750
2
Best Single Model
ABC
500
Best Single Model
0
CIFAR ImageNet SWAG
SST-2
Twitter
CIFAR
ImageNet
SWAG
SST-2
Twitter
Dataset
Dataset
Figure 4: (a) ABC for edge-to-clous inference: We simulate single-instance inference setup, seen in real-time applications where predictions may need be made as new data becomes available. ABC can enable small models to served at the edge without sacrificing accuracy leading to large savings in communication costs over the alternative of using only highest accuracy/ /largest model residing the cloud, the edge. (b) Total GPU usage costs of ABC using the best model. Agreement Braseeent-Based Cascading, reduced costs of GPU usage, exceeds the accuracy of the single best models in all task categories.
single small low-performing model on
In Figure 3) we demonstrate the impact of using ensembles various relative costs on the overall inference cost. As shown in the first plot, as we move away from the ideal, fully parallel setting with p = towards the sequential sctting with p = 0, the fraction of inference cost saved decreases. In fact, when the models in the cascade are of similar size for example, when 5 certain degree of parallelization is required for ABC to reduce inference cost. However, observe that when the relative costs are small enough (e.g., S 10) the need for parallelization diminishes. This is evident in Figure 3 (right); for S I the sequential execution curve (p 0) approaches the curve with full parallelization (p 1).
30:
Takeaway #1:
Although ensembling requires additional inference costs, these costs be mitigated when (1) models can be parallelized or (2) smaller models are several magnitudes cheaper than larger ones. In such scenarios, ABC can achieve substantial accuracy improvements while reducing inference costs.
5.2 Real-world Use-cases
As noted in the previous two subsections, ABC either improves on, or is at least competitive with, the single best model in terms of accuracy. Moreover, whenever the relative cost is small enough, ABC suffers negligible additional penalty for using an ensemble of models. As noted in Figure [ accuracy vs inference cost scaling for ML models already imply that the relative cost of models that only differ a few points in accuracy is small. For instance, the state-of-the-art model performance on the TmageNet-1I dataset attains about 83% top-1 accuracy with 70B FLOPs. Parcto-optimal model that achieves 63% accuracy requires only about 1B flops, and thus two-level ABC with ensembles of these two models has relative cost. of 7 = 70
In many real world scenarios, this disparity in relative cost is further amplified due to deployment considerations for instance, in an edge-to-cloud inference setting, local inter-process communication (IPC) latency is typically Iwo orders smaller than remote (cloud) IPC latency (7 & 10-2). Moreover, the ability of models in ABC to be placed on multiple, distributed devices with negligible synchronization overhead adds more opportunities for inference cost roduction. We consider three such scenarios, their respoctive cost models, and the benefits of ABC in these cases here. These include edge-to-cloud inference ($5.2.1). cloud-based model serving heterogeneous GPUs 5.2.2), and black-box inference access to model APT services (...33).
10

---

Published in Transactions on Machine Learning Research (07/ 2025)
5.2.1 Communication Cost in Edge-to Pilge-t-Goe to-Cloud Inference
An advantage of ABC is that it allows single large model to be split into multiple, potentially much smaller models, with only a simple reduce operation required to compute agreement. This allows to tune device placement at various levels of ABC to improve inference costs. One use-case where this is beneficial is edge-to- cloud inference (Forooghifar et al.) 2019;; here, inference requests are generated on user-facing edge devices like mobile phones or smart devices, which are sent over the network to a cloud service for evaluation. Given an inference request generated by a user interaction, the time-to-response or response latency in such case is dominated by communication overheads (network speed, serialization overhead, network congestion, etc.) beyond our control. By using ABC for such applications, we are able to distribute the inference load between tiny, on-device models and the cloud models, allowing us to avoid communication costs for significant. portion of requests.
To understand the effectiveness of ABC in such a scenario, we consider communication cost model previously studied in Zhu et al.] 2002 Lai et al. (2022) in a setup of edge devices (i.e., Raspberry Pis and smartphones) and cloud servers. The delay parameters adopted range from small, medium, to large [1 us, 10 ms, 100 ms, 1000 ms], where cear-instantancous local communication (< microsecond) can be expected occur with base cascade tiers performing inference on-device, and substantial network delays might oceur (> 1 second) in worst-case edge-to-cloud transmission (7 = 10-9). .e simulate this by considering two-level cascade, with the smaller level placed the edge-device. We apply the delay to the cascade exit points the edge device to capturing the time cost of transitioning between edge-to-cloud.
Our results, as shown in Figure 4 show that the flexibility that ABC affords in terms of model placement allows significant latency reductions, while providing superior accuracy compared to a single cloud model. In particular, we see that cascading in these scenarios provides an 14x reduction in communication cost for language tasks like SST-2; and for image datasets, we see 5x reduction for ImageNet-1K and a 8x reduction in CIFAR1O.
Takeaway #2:
ABC enables model placement flexibility where small models run locally and large models in the cloud. Commuication delays can make relative costs significant, resulting in ABC achieving 5-14x reductions in communication costs while maintaining superior accuracy.
5.2.2 Monetary Cost for Model Serving on Heterogeneous Hardware
Another use-case which can take advantage of the model placoment floxibility of ABC is using heterogencous
hardware for model serving on the cloud (Crago Walters 2015 Li et al. 2021a 2023 Mo et al.] 2023)
GPU/Accelerator hardware typically has disproportionately large difference in hardware costs compared
their throughput difference. For instance, based on the current pricing model offered by Lambda (Lambda 2024), a popular cloud rental platform, the rental pricing of single A100 is $1.40/hom and a V100 node is 50.06/hour (7 R 4 x 10-2), while the rated 32-bit tensor core throughput is 312 TFLOPs for A100 and 125 TFLOPS for V100. In this scenario, a simple placement strategy for a 2-level ABC that reduces inference cost may place the smaller model on V100 nodes and larger models on A100 nodes. Since the lower level models are- at least- an order of magnitude cheaper than the larger one in terms of FLOPS, this offsets the throughput loss we incur when switching from A100 to V100; and as a result, the ABC implementation incurs much lower average cost.
To concretely evaluate ABC under such a cost model, we retrieve the costs of GPU usage by hour from Lambda Cloud's pricing (sce, Table 4 in the appendix for details). For simplicity, we assume that each ensemble tier is set up on a distinet GPU in increasing order of GPU sophistication, and serves a uniform inference request rate. We also assume that the nodes are co-located and communication cost between them is negligible. We show summary in Figure | and present the detailed results in Table 5 in the Appendix. We can see at lcast. 3x reduction in inference cost in terms of 8/hour for image tasks, and a more moderate 10-30% reduction in language tasks.
11

---

Published in Transactions on Machine Learning Research (07/2025)
API-Based Inference: GSMBK
API-Based Inference: HEADLINES
0.94
0.63
0.92
Accuracy
Acouracy
0.62
V
0.90
A
ABC
A
PrugalGPT
0.61
0.88
AutoMix+T
AutoMMx+
0.86
MoT-LLM Cascade
0.60
1000
2000
3000
4000 5000 6000 7000
1000
2000
2500
3500
Total cost 1K tokens)
Total cost
1K tokens)
API-Based Inference: OVERRULING
API-Based Inference: COQA
0.96
0.52
4
0.95
*
4
0.50
Accuracy
0.94
Fl
0.48
0.93
0.46
250
500
750
1000
1250 1500
500
1000
1500
2000
2500 3000 3500
Total oast ($ 1K tokens))
'ootal cost ($ per 1K tokens)
Figure 5: Comparison of ABC against state-of-the-art cascade baselines for black-box API-based inference. The faded, hatched-patterned variants represent budget-friendly, 2-level cascade instances where do not include the costly Tier 3. Most of these methods show competitive performance, but ABC matches their accuracy at siguificantly lower costs all tasks. Note that all these methods (aside from the MoT-LLM cascade) incur additional setup costs not reflected in our plots.
Takeaway #3:
GPU rental cost differences significantly exceed throughput differences. ABC optimally places models across hardware tiers, achieving 3x cost reductions for image tasks and 10-30% savings for text tasks.
5.2.3 Monetary Cost for Black-box API-Based Inference
Finally, in the current landscape dominated by LLMs, many providers offer black-box API access to their proprictary
Tier
Model
Price
LLMS (Abdalla et al. 2023 Sun et al. 2022: Hadi et al.
LlaMA 3.1 8B-Instruct Turbo
0.18
2024j. Similarly to the large cost difference between GPU generations discussed the previous subsection, we also ob-
Tier
Gemma
0.30
LlaMA 3 8B Instruct Lite
0.10
serve large cost disparity between various generations ietrattiinnniiirrr /tiers of API calls. For example, using Together.so -ieeheer.ii-neeeeee of the lowest-cost serverless endpoint providers for LLM:] case study, we find that models within the 7B-SB range cost. 80.20 per million tokens, while LlaMA3. 1-405B costs $5.00. This translates to the larger model costing 25x more than the smaller model range (7 = 2 P- 1). If we consider GPT-4 as the gold standard, the cost of usage quickly scales to 150x of our reference smaller model's costs
LlaMA 3.1 70B Instruct Turbo
0.88
Tier 2
Gemmna 27B Instruct
0.8
Qwen 2 72B-Instruct
0.9
Tier LlaMA 3.1 405B Instruct Turbo
5.0
The cascade tiers, their models, and as- sociated costs, dollar per million tokens, for the API-based experiments; all API services are pro- vided by fogether.ai We use the tiers and models they are for ABC inference system. However, for the single-moded cascade baselines, we use the tiers' best models for their systems.
Table 1:
Since only have black-box access to these models, we cannot employ score-based deferral rule. However, we demonstrate that using ABC's voting-based based rule defined in $ is also effective such scenarios. For baseline comparison,
together.ad /ppriiiingggg September 2024.
GPTT441 106-preview costs $30 September 2024.
12

---

Published in Transactions on Machine Learning Research (07 (225))
use FrugalGPT, 2 variants of AutoMix, and MoT LLM Cascade. These state-of-the-art cascading methods are specifically designed for scenarios where we make API calls to black-box model endpoints, in contrast. to our more generally applicable ABC method. We access the models from Together.ai described in Table [ for these experiments, and consider a sctting that is advantagcous to the basclines, by selecting the best singular model from each performance tier their respective approaches.
All the baseline methods considered here are significantly more complex and involved set up than ABC; both AutoMix and FrugalGPT involve training router deferral rule at each cascade level which has to be repeated for every new task model change. Unlike AutoMix, FrugalGPT requires training DistilBERT- scorer, which would require the user's possession of GPU resources. MoT LLM Cascade generates multiple results and varies the randomness in the LLM's responses via sampling, while using in-context demonstrations and reasoning techniques (e.g., Chain-of- Thought (Wei et al.) 2022 ) to influence how the model generates answers. In contrast, ABC uses a much simpler voting-based safe deferral rule, without involving additional training or complex routing strategics. Further method-speone details for these experiments as well additional results can be found in Appendix D.2
Our results, shown in Figure demonstrate that ABC is a more reliable deferral rule, and thus, offers more favorable trade-off between accuracy and cost compared to existing wethods-despitee their more sophisticated routing mechanisms, and even without considering the additional setup costs incurred by some of the baseline methods. For instance, while FrugalGPTTs scorer struggles on harder tasks and tends to take the safer route by deferring more frequently, ABC aggressively leverages cheaper models for a siguificant. portion of inputs, reserving higher-cost models only when necessary. AutoMix, on the other hand, uses a few-shot self-verification system that is sampled & times where k is >1 (in the authors' codebase and ours, k 8); hence, the additional API calls add significantly to its cost of usage. In contrast, ABC easily maintains and often improves on accuracy while being training-fi frre,, simple approach.
Takeaway #4:
LLM API pricing creates extreme cost ratios. ABC's simple voting mechanism achieves 2-25x cost reductions compared to SOTA cascading methods without requiring complex router training.
5.3 Ablation Studies and Sensitivity Analysis
We conduct comprchensive ablation studies to validate ABC's design choices and analyze scnsitivity to key deployment parameters:
Impact Parallelization 5.1.1 and Figure S (in Appendix E.1 analyze ABC under different parallelization
scenarios. With full paralleiiaatin (p = 1), ABC achieves optimal accuracy-FL.OP trade-offs, consistently outperforming single models. Even under sequential execution (p 0), ABC maintains substantial advantages when relative costs are sufficiently disparate (e.g., 7 < 1)), demonstrating robustness to deployment constraints.
Relative Cost Sensitivity $.1.2 and Figure 3 examine how ABC depends on the relative cost y between cascade tiers. When models have similar costs (y 2 }), some parallelization may be required for cost savings. However, when smaller models are at least 50x cheaper (7 S dd), ABC provides meaningful savings even with sequential execution, validating our theoretical analysis.
Threshold Estimation Robustness Appendix B and Figure G demonstrate that agreement threshold 0 estimation is stable across different model accuracy levels (37.6% to 86.0%) and converges with minimal validation data. Using only 100 samples yields threshold estimates that remain stable when evaluated on 10x more data, confirming the effectivencss of the calibration procodure.
Safe Deferral Rule Existence Appendix C and Figure 7 characterize selection rates under different error tolerances (1%, 3%, 5%) across model accuracies and computational budgets. Higher-accuracy models achieve
13

---

Published in Transactions on Machine Learning Research (07 (07/2025)
selection rates up to 60% even with strict 1% error tolerance, empirically validating the practical existence of safe deferral rules predicted by our theory.
Cost Breakdown Analysis Tables & 5 Appendix E.2 provide detailed analysis across cascade tiers,
showing that most samples (52 to 93%) are processed at cheaper early tiers. This validates ABC's ability to concentrate expensive computation on truly difficult samples while handling the majority of requests efficiently.
Cascade Configuration Effects Figure 8 explores different cascade lengths (2-4 levels) and ensemble sizes (2-5 models per tier). The results show diminishing returns for larger ensembles, with 2-3 models per tier typically providing optimal accuracy-cost tradeoffs.
These ablations collectively demonstrate that ABC's core design principles are empirically sound, with performance gracefully degrading under suboptimal deployment conditions while maintaining substantial benefits when key assumptions (e.g., large relative cost. disparities) hold.
Takeaway #5:
Ablation studies confirm that ABC's benefits stem from the synergy of parallelization and cost disparitics. Safe deferral rules exist across diverse scttings, and threshold estimation requires minimal validation data (=100 samples) while remaining robust across model accuracies.
6 Conclusion
In this work, we introduce Agrecment-Based Cascading (ABC) as a straightforward approach for adaptive inference that utilizes existing models for constructing cascades and makes deferral decisions based on their mutual agreement. We define safe deferral rules, ensuring ABC can serve as a drop-ín replacement for models while improving accuracy.
Although using ensemble of models provide powerful deferral rule for cascading, the additional costs required to compute such an ensemble may not lead to savings in all inference scenarios. Despite this, our work shows that this simple approach is surprisingly effective given the large differences in model sizes that reach state-of-the-a
accuracy in recent ML tasks. We demonstrate improvements via a number of real-world case studies, including a study on communication costs in edge-to-cloud inference, rental costs in cloud-basod scttings, and the cost of black-box API services. Overall, our results demonstrate ABC's capacity to improve the efficiency of adaptive inference systems without the complexities associated with traditional cascade frameworks, making it compelling option for practitioners focused on reducing inference latency.
Several promising directions emerge from this work. First, extending ABC to open-ended generation lasks would significantly broaden its applicability, particularly given the growing prominence of LLM-based applications. Second, exploring more efficient ensemble methods could further enhance ABC's benefits in scenarios with limited parallelization capabilities, potentially through techniques that reduce ensemble overhead while maintaining agreement quality. Finally, investigating bias-aware agreement mechanisms construction strategies that promote diversity in decision-making patterns across demographic groups could address potential bias propagation concerns inherent in based voting systems.
Future Work.
14

---

Published in Transactions on Machine Learning Research (07 (025)
References
Mohamed Abdalla, Jan Philip Wahle, Terry Lima Ruas, Aurélie Névéol, Fanny Ducel, Saif M Mohammad, and Karen Fort. The elephant in the room: Analyzing the presence of big tech in natural language processing research. In The G1sl Annual Meeting Of The Associalion For Compulational Lánguislics, 2023.
Anelia Angelova, Alex Krizhevsky, Vincent. Vanhoucke, Abhijit S. Ogale, and Dave Ferguson. Real-time pedestrian detection with deep network cascades. volume 2, pp. 4, 2015.
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In International Conference on Machine Learning, pp. 527-536. PMLR, 2017.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-All: Train One Network and Specialize it for Efficient Deployment, April 2019.
Zhaowei Cai, Mohammad Saberian, and Nuno Vasconcelos. Learning Complexity- Aware Cascades for Deep Pedestrian Detection. pp. 3361-3369, 2015.
Lingjiao Chen, Matei Zaharia, and James Y Zou. FrugalML: How to use ML Prediction APIs more accurately and cheaply. In Advances Neurul Information Processing Systems, 2020.
Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance. arXiv preprint Xrxi::2305, 05176, 2023.
Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12270 12280, October 2021.
Kevin Clark, Minh-Thang Luong, Quoe Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint orXiv:2008.1055: 2020.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14..6 2021.
Stephen P Crago and John Paul Walters. Heterogeneous cloud computing: The way forward. Computer, 48 (01):59-61, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Don Dennis, Abhishek Shetty, Anish Prasad Sevekari, Kazuhito Koishida, and Virginia Smith. Progressive ensemble distillation: Building ensembles for efficient inference. Advances in Neural Information Processing Systems, 36, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. North American Chapter of the Association for Computational Linguislics, 2019.
Devvrit, Sncha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit Dhillon, Yulia Tsvetkov, Hannaneh Hajishirzi, Sham Kakade, Ali Farhadi, and Prateek Jain. MatFormer: Nested Transformer for Elastic Inference, October 2023.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop multiple classifier systems, pp. 1 15. Springer, 2000.
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjoe, Victor Rühle, Laks VS Lakshmanan, and Ahmed Hassan Awadallah. Hybrid Ilm: Cost-efficient and quality-aware query routing. Tn The Twelith International Conference Learning Representations, 2024.
15

---

Published in Transactions on Machine Learning Research (07/2025)
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprinl Xrv::2000. 11299, 2020.
Yilun Du and Leslie Kaelbling. Compositional Generative Modeling: A Single Model is Not All You Need, February 2024. arXiv:2402.01103 [es].
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 2rXi::2407.211 21733, 2024.
Saso Džcroski and Bernard Zenko. Is combining classifiers with stacking better than sclecting the best onc? Machine learning, 54:255 273, 2004.
Shohei Enomoro and Takeharu Eda. Learning to cascade: Confidence calibration for improving the accuracy and computational cost of cascade inference systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volune 35, pp. 7331 7339, 2021.
Alan Fern and Robert Givan. Online ensemble learning: An empirical study, Machine Learning, 53:71-109, 2003.
Farnaz Forooghifar, Amir Aminifar, and David Atienza. Resource-aware distributed epilepsy monitoring using self-awareness from edge to cloud. IEEE transactions biomedical circuits and systems, 2019.
Aditya Gangrade, Anil Kag, and Venkatesh Saligrama. Selective Classification via One-Sided Prediction. Tn Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pp. 2179 2187. PMLR, March 2021. ISSN: 2640-3498.
Yonatan Geifman and Ran El-Yaniv. SelectiveNet: A Deep Neural Network with an Integrated Reject Option. In Proceedings of the 36th International Conference on Machine Learning, pp. 2151-2111111111 2159. PMLR, May 2019. ISSN: 2640-3498.
Shijie Geng, Peng Gao, Zuohui Fu, and Yongfeng Zhang. Romebert: Robust training of multi-exit bert. arviv preprint arXiv:2101.09755, 2021.
Raphael Gontijo-Lopes, Yann Dauphin, and Ekin D. Cubuk. No One Representation to Rule Them All: Overlapping Features of Training Methods, April 2022. arXiv:2110.1225 [es].
Sachin Goyal, Aditi Raghunathan, Moksh Jain, Harsha Vardhan Simhadri, and Prateek Jain. Droce: Deep robust one-class classification. In International conference on machine learning. PMLR, 2020.
Jiaqi Guan, Yang Liu, Qiang Liu, and Jian Peng. Energy-efficient Amortized Inference with Cascaded Deep Classifiers. In Proceedings of the Twenly- Seventh International Joint Conference on Artificial Intelligence, pp. 2184-2190. Stockholm, Sweden, July 2018. International Joint Conferences on Artificial Intelligence Organization. ISBN 978-1-7992411-2-7
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330 PMLR, 06-11 Aug 2017.
Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Language model cascades: Token-lovel uncortainty and beyond. In International Conference on Learning Representations, 2024.
Muhammad Usman Hadi, Qasem Al Tashi, Abbas Shah, Rizwan Qureshi, Amgad Muneer, Muhammad Trfan, Anas Zafar, Muhammad Bilal Shaikh, Navoed Akhtar, Jia Wu, et al. Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects. Authorea Preprints, 2024.
16

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic Neural Networks: A Survey. TEEE Transactions on Paltern Analysis and Machine Intelligence, 44(11):7436- 7456, November 2022. ISSN 1939-3539.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image rocognition. Proceedings of the IEEE conference on compuler vision and paltern recognilion, pp. 770 778, 2016.
Pengcheng He, Xiaodong Liu, Jianteng Gao, and Weizhu Chen. Deberta: Decoding enhannee bert with disentangled attention. arXiv preprint arXi::20660038 03654, 2020.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprinl ar Xiv:20000. 14701, 2020.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling predictable, empiri- cally. arXiv preprint arXiv:1122.004 00409 2017.
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. DynaBERT: Dynamic BERT with Adaptive Width and Depth. In Advances in Neural Information Processing Systems, volume 33, pp. 9782-9793. Curran Associates, Inc., 2020.
Ting-Kuei Hu, Tianlong Chen, Haotao Wang, and Zhangyang Wang. Triple wins: Boosting accuracy, robustness and efficioncy together by enabling input-adaptive inference. In International Conference on Learning Represenlations, 2020.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q. Weinberger. Multi-Scale Dense Networks for Resource Efficient Image Classification, June 2018. arXiv:1703.09844
Ramesh Jetty, Indy Sawhney, and Simon Zamarin. Batch Inference at Scale with Amazon SageMaker AWS Architecture Blog, November 2021. URL htttps///vvs http::///aaaaaaaao....amm amazon hcom/bblogg///rccctttttuu. wwtlee-wwhht--h aaazom sestoon---age sagemaker/
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. L.MM-Blnner: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion, June 2023.
Wittawat Jitkrittum, Neha Gupta, Aditya K. Menon, Harikrishna Narasimhan, Ankit Rawat, and Sanjiv Kumar. When Does Confidence-Basell Cascade Deferral Suffice? Advances in Neural Informalion Processing Systems, 36:9891-999 9906, December 2023.
Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXio, abs/2001.08361, 2020.
Alind Khare, Dhruv Garg, Sukrit Kalra, Snigdha Grandhi, Ton Stoica, and Alexey Tumanov. SuperServe: Pino-Grainch Inference Serving for Unpredictable Workloads, December 2023. arXiv:2312.16738 [es].
Schoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. An Ilm compiler for parallel function calling. ArXiv, abs/2312.04511, 2023.
Alex Krichevsky and Geolfrey Hinton. Learning multiple layers of features from tiny images. 2009.
Fan Lai, Yinwei Dai, Sanjay Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha Madhyastha, and Mosharaf Chowdhury. Fedscale: Benchmarking model and system performance of federated learning at scale. In International conference on machine learning, pp. 11814 11827. PMLR, 2022.
Lambda. GPU Cloud VMs for Deep Learning Lambda, 2024. URL https: kttps://lambdallbs..cm www.://laabsccoo///.i./..ol cluud
17

---

Published in Transactions on Machine Learning Research (07/2025)
Luzian Lebovitz, Lukas Cavigelli, Michele Magno, and Lorenz K. Muller. Efficient Inference With Model Cascades. Transactions on Machine Learning Research, 2023.
Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. Orchestrallm: Efficient orchestration of language models for dialogue state tracking. arXiv preprint arXie:2311.0975 2023.
Baolin Li, Vijay Gadepally, Siddharth Samsi, Mark Veilectte, and Devesh Tiwari. Serving machine learning inference using heterogeneous hardware. In 2021 IEEE High Performance Extreme Compuling Conference (HPEC), pp. 1-8. IEEE, 2021a.
Baolin Li, Siddharth Samsi, Vijay Gadepally, and Devesh Tiwari. Kairos: Building cost-efficient machine learning inference systems with heterogeneous cloud resources. In Proceedings of the 32nd International Symposium on High- Performereo Parallel and Distributed Computing, pp. 3-16, 2023.
Lei Li, Yankai Lin, Deli Chen, Shuluai Ren, Peng Li, Jie Zhou, and Xu Sun. CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascade. In Findings of the Association for Computational Linguisties: EMNLP 2021, pp. 475-486, Punta Cana, Dominican Republic, November 2021b. Association for Computational Linguistics. doi: 00.18653/v1/2022 wttp//01...iiiigeeeelelpppr
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, and Qi Ju. Fastbert: a self-distilligg bert with adaptive inference time. arXio preprint arXiv:2004.08 022 2277 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint ar 1rXt::1907.1169, 2019.
Aman Madaan, Pranjal Aggarwal, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, Shyam Upadhyay, Mausam, and Manaal Faruqui. AutoMix: Automatically Mixing Language Models, October 2023. arXiv:2310.1296; [cs].
Jonathan Mamou, Oren Pereg, Moshe Wasserblat, and Roy Schwartz. TangoBERT: Reducing Inference Cost. by using Cascaded Architecture, April 2022. arXiv:2204.06271 [cs].
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, Tianqi Chen, and Zhihao Jia. Towards efficient generative large language model serving: A survey from algorithms to systems. ArXic, abs/ 2bss/2311.1523 2023.
Hao Mo, Ligu Zhu, Lei Shi, Songfu Tan, and Suping Wang. Hetsev: Exploiting hetcrogeneity-aware autoscaling and resource-efficiet scheduling for cost-effective nachine-learning model serving. Electronics, 12(1):240, 2023.
Harikrishna Narasimhan, Wittawat Jitkrittum, Aditya K. Menon, Ankit Rawat, and Sanjiv Kumar. Post-hoc estimators for learning to defer to an expert. Advances in Newral Information Processing Systems, 35: 29292-29304 December 2022.
Lunyiu Nie, Zhimin Ding, Erdong Hu, Christopher Jermaine, and Swarat Chaudhuri. Online cascade learning for efficient inference over streams. arXiv preprint arXix:2402.0455 04511, 2024.
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. Routellm: Learning to route Ilms with preference data. arXiv preprint 186665,, 2024.
Yulong Pei, Amarachi B. Mbakwe, Abhinav Gupta, Salwa Alamir, Huang Lin, Xiaomo Liu, and Sameena Shah. Tweetfinsent: dataset of stock sentiments on twitter. 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Tn International conference on machine learning, pp. 8748 8763. PMLR, 2021.
18

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Compulational Linguistics, 7:249-2601 2019.
H.A. Rowley, S. Baluja, and T. Kanade. Neural network- based face detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(1):23 -31), Jamuary 1998. ISSN 1939-3539. doi: 00.1109/34.655477,
Marija Sakota, Maxime Peyrard, and Robert West. Fly-swat or camnon? cost-effective language model choice via meta-modeling. In Proceedings of the 17th ACM International Conference on Web Seorch and Data Mining, pp. 606-615, 2024.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv, October 2019.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident Adaptive Language Modeling. Advances in Neural Information Processing Systems, 35: 17456 17472, December 2022.
Mohammad Saeed Shafiee, Mohammad Javad Shafiee, and Alexander Wong. Efficient inference on deep neural networks by dynamic representations and decision gates. Advances in neural information processing systems, 2018.
Amanda J. C. Sharkey. On combining artificial neural nets. Connection Science, ((3-4)229-3314 1996. doi: 10.1050/0954009961166777
Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuckai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large Language Model Routing with Benchmark Datasets, September 2023. arXiv:2309.1577 15779 [cs].
Ankur Sinha and Tanmay Khandait. Impact of news the commodity market: Dataset and results. In Advances in Information and Commuuiication: Proceedings of the 2021 Future of Information and Communication Conference (FICC), Volume 2, pp. 589 601. Springer, 2021.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631 1642, 2013.
Sander Soo. Object detection using Haar-cascade Classifier. Instilute of Computer Science, Universily of Tartu, 2(3):1 2014.
Matthew Streeter. Approximation Algorithms for Cascading Prediction Models. In Proceedings of the 35th International Conference on Machine Learning, pp. 4752 4760. PMLR, July 2018. ISSN: 2640-3498.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for modern deep learning research. In Proceedings of the AAAT conference on artificial intelligence, volume 34, pp- 13693-13696, 2020.
Arun Suggala, Bingbin Liu, and Pradeep Ravikumar. Generalized boosting. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.
2020.
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language- modellas-o-sarvice. In International Conference on Machine Learning, pp. 20841-20855. PMLR, 2022.
Teo Susnjak, Andre L. C. Barczak, and Kenneth A. Hawick. Adaptive cascade of boosted ensembles for face detection in concept drift. Neural Computing and Applications, 21:671-682, 2012.
C Szegedy. Intriguing properties of neural networks. arXiv preprint or Xiu: 2013.
19

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.0011 000018, 2024.
Necraj Varshney and Chitta Baral. Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems. In Proceedings of the 2022 Conference on Empirical Methods in Nalural Language Processing, pp. 11007-11021, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics.
P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE Computer Sociely Conference on Compuler Vision and Pottern Recognition. CVPR 2001, volume 1, December 2001. ISSN: 1063-6919.
Paul Viola and Michael J. Jones. Robust Real-Time Face Detection. International Journal of Computer Vision, 57(2))137 154, May 2004. ISSN 1573-1405. doi: 11100///.IISS.000013577...9999
Lidan Wang, Jimmy Lin, and Donald Metzler. A cascade ranking model for efficient ranked retrieval. In Proceedings of the 34th international ACM STGIR conference Research and development in Information Retrieval, pp. 105 114. ACM, 2011.
Xiaofang Wang, Dan Kondratyuk, Eric Christiansen, Kris M Kitani, Yair Movshovitz-Attias, and Elad Eban. Wisdom of commitlees: An overlooked approach to faster and more accurate models. In International Conference on Learning Representations, 2021.
Xin Wang, Yujia Luo, Daniel Crankshaw, Alexey Tumanov, Fisher Yu, and Joseph E. Gonzalez. IDK Cascades: Fast Deep Learning by Learning not to Overthink, June 2018a. arXiv:1706.00m 000:0880 [es].
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E. Gonzalez. SkipNet: Learning Dynamic Routing in Convolutional Networks. pp. 409 424, 2018b.
Yiding Wang, Kai Chen, Haishong Tan, and Kun Guo. Tabi: An efficient multi-levol inference system for large language models. In Proceedings of the Eighteenth European Conference on Compuler Systems, pp- 233 248, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24882 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, et al. Huggingface's transformers: State-of-the-art natural language processing. ar preprint ar Xio:11910 087711 2019.
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2246-2251, 2020.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1067. 2024.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoe V. Lo. Xlnet: Generalized autoregressive pretraining for language understanding. In Neural Information Processing Systems, 2019.
Jiahui Yu and Thomas S. Huang. Universally Slimmable Networks and Improved Training Techniques. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1803 1811, 2019.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable Neural Networks, Docember 2018. arXiv:1812.0892 [cs].
20

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. Large language model cascades with mixture of thought representations for cost-efficient reasoning. In The Twelfth International Conference on Learning Representations, 2024.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adversarial dataset, for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 93-104, 2018.
Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel Ho. When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings of the eighteenth international conference artificial intelligence and law, pp. 159 159-168, 2021.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 33:18330 18341, 2020.
Ligeng Zhu, Hongzhou Lin, Yao Lu, Yujun Lin, and Song Han. Delayed gradient averaging: Tolerate the communication latency for federated learning. Advances in Neural Information Processing Systems, 34: 29995-30000 2021.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. survey on model compression for large language models. arXiv preprint Xix:22008.. 07668,, 2023.
Fei Zuo and Peter H. N. de With. Fast face detection using a cascade of neural network ensembles. In Advanced Concepts for Intelligent Vision Systems Conference, 2005.
Fei Zuo and Peter H. N. de With. Cascaded face detection using neural network ensembles. EURASIP Journal on Advances in Signal Processing, 2008:1-13, 2008.
A Agreement-Basee Cascading (ABC) can Improve Accuracy
Our construction of ABC, as explained in Section 3 and 4 is based on safe deferral rules (Definition 4.1
rules such that when a data point selected at a lower cascade tier, the inference is accurate with high probability. This means that if the lower tier of the cascade has a higher accuracy compared to the largest. model, on the 'easy' data it predicts on, the overall accuracy of ABC can increase. For 'hard' samples ABC and the largest, SoTA model produces identical predictions ABC is internally deferring to this model. We formalize this notion of accuracy improvement in this section.
Consider a classification problem in the statistical learning framework, where N' is our instance space and y is the label space. Let r denote any deterministic deferral rule X' + 10. Let h22) be two classifiers h4 : X + y for i E {1,2}. We will think h2 as being the more expensive (better accuracy) model. Consider a distribution over X' V. The risk of classifier hz is defined
R(h2) P(h2(x) + 9)
Let us define a cascaded classifier M, X y such that,
Mx(x) = (x) r(x) = 0
(5)
h2(x)) r(x)
Then the risk of the cascaded classifier using rule r is,
R(M,) (M +
We wish to understand when we can replace an existing large, expensive classifier hy with a cascade such that there is no drop in accuracy. This is useful when we extremely sensitive towards accuracy drop and
21

---

Published in Transactions on Machine Learning Research (07/2025)
wish to only select models at the lower-level when certain about the classification. Let define the excess risk of using& cascade in-place of the larger model h2 as,
Rexcess (M, h22 R(M,) Rh2)).
Expanding further, we can express the excess risk as,
Rexcess Rexcess(M,
P(M, + v) ((h2(=)
(zM,(x) # -(x)= 0)+ =) 1)
-P(h2(I) #))
-P(Hi(z) /v|r(z) = 0) = 0) ((b #y/r(a)=
= 1)P(r(z) +(b(())) 1) 0)P(r(s) 0) (1))(x) r(x) P) =1 P(r(x)
-P(h2(s)
= -(P(H((z) サ r((x) = 0) - P(he(z) + r(z)= - P()()) = 0)
(6)
As can be seen, the general excess risk here depends on both models. Since our focus is on accuracy, any rule that leads to a risk that is no worse than that of the classifier h2 can be used, and we term these as admissible cascades.
Desinition A.1 (Admissible cascades). (x) defined above is an admissible cascade w.r.l the population distribution P and the reference classifier h2 if.
Remoonss (M, h2) S 0,
equivalently
((H1(z) u r())) = 0) 5 ((hg(a) # r((x) = 0).
As mentioned in Section 4 for this work, specialize to the case where whenever the smaller model is selected, it is always correct. That is, (x) + y r(w) = 0) = 0. Of course, such pair of smaller-classffie H1 and deferral rule need not exists. However, whenever they do, a cascade constructed using any h2 is admissible since,
Vh2, R(M,, h2) = -Ph2((x + y r(x) = ())((())) = 0) < 0.
This implics a universal nature of such (H, for such two tier cascades as the excess risk of the cascade does not depend on the larger classifier fı2.
Finally, the ensemble Hz(x) outperforms the SoTA model on the 'easy" samples by some strictly positive g >>0 0, such that
P(H1(z)) # |r(x)) = 0) < P(h2(x) # r(z) 0) E
Then, from Equation
Rexcess (M, h2) = (P(Hi(z) # y
= 0) P(hz(x) + (((r(()))))))))())) = 01) P(r(x) 0) ((()) (
--(P(r(=)=0) = <,
wherever the selection rate satisfies P(r(z) 0) 0. Negative excess risk implies an improvement over the
SoTA classifier were using with the overall amount of improvement depending P(r(x) 0).
Proof of Proposition 4.1
Part 1: By Definition 4.1
we have P(r(x) 0, 0, #v) <.. The risk of the cascade decomposes as:
R(M,) =
#)) P(r(x) = 1, h2(x) +v) Sc+ P((()) R(h)) < R(h2) +
(7)
(8)
Part 2: The expected cost follows from Equation 1 and conditioning on the deferral decision:
F(C(M,)) P(r(x) ())(=)))C((}))) + P(r(x) 1)(())) (K" P(r(x) 1))((h))
22

---

Published in Transactions on Machine Learning Research (07/2025)
Threshold estimate vs number of samples used on ImageNet1 K
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
Theta (0)
-X-
Acc: 37.6%
-X- Acc: 51.9%
-X-
Acc: 58.8%
0.00
0.00
0.00
10K
10K
32K
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
Acc: 65.6%
Acc: 71.6%
-X- Acc: 86.0%
0.00
0.00
0.00
1K
3K
10K
32K
1K
3K 10K
32K
1K
3K
10K
32K
Number of Samples
Figure Estimation of agreement threshold stability as a function the number samples used, across different model accuracy levels on the ImageNet-1K dataset. Each plot corresponds to model with specific accuracy, shown in the legend. The initial estimate with 100 samples, with subsequent estimates with larger and larger number of samples. The initial estimate is reasonably close within subsequent estimates with larger number of samples.
B Estimating Agreement Threshold 0
To effectively apply ABC, it's essential to configure agreement threshold 0 for the deferral rules defined in Equations 3 and ! This threshold indicates a sufficient level of confidence in the ensemble's predictions, allowing ABC to avoid deferring to a higher-cost model when the ensemble's agreement is high, thereby reducing inference costs without sacrificing accuracy. Recall from Definition 4.1 that the failure rate of a deferral rule, as function of the agreement threshold is given by,
P(s(x) 2 0, Hx(x) # v)-
In particular, safe deferral rules are those with a failure rate bounded by small 2 0 of our choice. This means that, given a distribution P over x y we can define a function p(0) as,
p(0) P(s(2)( 2 0, Hx(x) + v).
We can now define feasible thresholds, 0 as those for which the error rate p(0) S since any such 0 leads to safe deferral. In practice, we rarely have access to IP and therefore p(0). We instead use its plugin-estimator, given by
p(0) 1 Σ [((x) 2 0, Hx(2) + v).
We estimate p(0) by using small subset of samples from the validation set. Typically, we draw around 100 samples and set them aside to determine a stable threshold. Figure G shows how the estimated threshold varies with the number of samples used in the estimation process, across models with different accuracy levels. Each plot represents model of a specific accuracy, with accuracy percentages indicated in the legend. As shown, the estimate generally stabilizes even with 100 samples, suggesting that only a few validation samples needed to reliably estimate 0, making the parameter estimation process efficient and practical for wealdworld applications.
C Existence of Safe Deferral Rules and Selection Rates
This section examines the existence of safe deferral rules rules that have a very low probability of being incorrect, where data is not deferred to larger model - by evaluating selection rates. For any threshold 0,
23

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Selection rate accuracy ImageNet1K
Selection rate vs FLOPs on ImageNet1K
Max-error (%)
Max-error (%)
Data selected (%)
60
---- 1.0
Data selected (%)
60
---- 1.0
---- 3.0
-a-- 3.0
5.0
---- 5.0
40
40
20
20
0
0
20
30
40
50
60
70
80
0.5B
3B
10B
32B
100B
Accuracy (%)
FLOPs
Figure 7: Selection rate as function of accuracy (left) and FLOPs (right) for different error tolerances ImageNet- 1K. The selection rate P(r(()) 2 0) represents the fraction data handled at a lower cascade tier without deferring threshold 0. Laxer
to larger model, based samples meet the criteria for safe deferral. In contrast, stricter tolerances (e.g., 1%) result in lower selection rates. Both plots illustrate that higher-accuracy or higher-FLOPP models generally achieve higher selection rates, especially as the allowable error tolerance increases, reinforcing the stability and practicality of ABC.
tolerances (e.g., 5%) yield higher selection rates,
the selection rate P(r(x) 2 0) represents the fraction of data that deemed 'easy' enough to be processed at lower tier of the cascade. Experimentally, we observe that 0, computed in the manner described in Section does indeed adhere to the error tolerance of choice e. This means that whenever a data point selected at a lower tier, the inference is accurate with probability -- e. Stricter values of e typically result in a higher threshold for agreement and lower-sclection rate.
The plots in Figure 7 show the selection rate ImageNet-1K dataset. We explore selection rates for three error tolerances, corresponding to maximun allowable error rates of 1%, 3%, and 5%. As shown in the left plot, which compares selection rate versus accuracy of the models used in the ensemble, higher accuracy models tend to achieve higher selection rates, especially when the error tolerance relaxed (e.g., 5% maximum error). Conversely, stricter error tolerances (e.g,, 1%) lead to lower selection rates across all accuracy levels, as fewer samples meet the stringent requirements for safe deferral at lower cascade tiers. the right plot, which illustrates selection versus FLOPS,, similar trend is observed. Higher-FI.OP models, which generally more accurate, are able to safely handle larger portion of the data at lower tiers, particularly as the error tolerance increases.
different accuracy levels and FT.PP values the
Evaluation Setups
D.1 Datasets and Models
Datasets and models used in our experiments are detailed in Table 2 and Table
Table 2: Datasets used in both benchmark and black-box API experiments across various task types.
Category
Dataset
Task Type
ImageNet-Ile IK (Deng et al.
2009
Image classification
Image Tasks
CIFAR-10 Krizhevsky & Hinton
2009
Image classification
SST-2 (Socher et al.
2013
Sentiment analysis
Language Tasks
Twitter Financial News PPi et al.
2021
Sentiment analysis
SWAG [Zellers et al.
2018
Multiple-Choice QA
GSM8K Cobbe et al. 2021
Math Reasoning
COQA (Reddy et. al. 2019
Conversational QA
Black-Boo Experiments
OVERRULING (Zheng et al. 2021
Legal Reasoning
HEADLINES (Sinha & Khandait
2021
News Classificatiin
24

---

Published in Transactions on Machine Learning Research (07/2025)
Table 3: Summary of models used for both benchmark and black-box API experiments, across image and text tasks.
Category
Dataset
Tiers Used
BERT Devlin et al,
2019
RoBERTE a
Liu et al.
2019
Language Models
XLNet Yang et al.
2019
BASE, LARGE
ELECTRA (Ralford et al.
2021
ResNet (He et al.
2016
Image Models
VIT
Dosovitskiy
2020
Selection based on FLOPs
CLIP
Clark
2020
LlaMA 3.1
Dubey et al.
2024)
Black-Box Models
Gemma 2
Team et al. 2024
See Table
Qwen 2
Yang et al. 2024
D.2 Details for Back-Box API Experiments
We compare ABC FrugalGPT (Chen et al., 2023 variants of AutoMix (Madaan al.) 20230, and MoT LLM Cascade (Yue et al. 2024). Although HybridLLM (Ding al.) 2024 falls into this category of SOTA methods, it has been shown to underperform FrugalGPT and AutoMix (Madaan al. For practical comparison, we implement all methods in fully functional cascade system.
Baselines
2023
Method-Specific Details
AutoMix trains a different router for all possible cascade steps, i.e., n routers for the n cascade tiers, and this has to be repeated for every new task model replacement in the system. A threshold (AuroMix+T) or POMDP (AuroMix+P) is trained with a combination of 2 50 training samples from the same test data distribution and the initial inferences generated on the test data by the two models involved in each cascading step. After training the router, using the cascading system often involves running few-shot self-verification 8 times at a high sampling temperature (temp 1.0), using the same model that generates inference at the given cascade tier. Automix then averages the self-verification results and meta-verify it with the best parameters of the routing strategy to decide when to exit.
AutoMix:
FrugalGPT: Just like AutoMix, FrugalGPT needs train n routers for n cascade tiers, and each router needs to have sense of the data distribution and the model's predictive power. 2 500 training samples and inference generated on these samples by the tier's model are needed to train each tier/ model's scorer, a DistilBERT (Sanh et al. 2019).
MoT LLM Cascade: Yue et al. 2024 focuses on sampling and consistency checking as a means of cascade. To mcasure consistency, the weaker LLM generates multiple answers for single question by varying the randomness in the LLM's responses i.e., varying the lemperatuure of the model while using in-context demonstrations and reasoning techniques (e.g., Chain of Thought (Wei et al.) 2022) to inffuence how the model generates answers. The system compares the different sampled answers and picks the most consistent one. If the consistency score is high enough, the weaker model's answer accepted. Otherwise, the question is passed the next tier.
ABC: We use the voting-based based safe deferral rule, requiring no additional training or any complex routing strategies.
Cascade Models We use the models described in Table 1 We consider a setting that is advantageous to the baselines by selecting the best singlular model from each performance tier and cascading between them. In all cascading systems, we use all three tiers for cascade; but considering budget constraints, we also have setups where we delete Tier 3 and use only the first two tiers (2-level cascade).
We evaluate these methods on variety of (closed) generation datasets and tasks shown in Table 2 To ensure consistency in output format and easier evaluation setup, we few-shot.
Evaluation Sctup
25

---

Published in Transactions on Machine Learning Research (07 (07/2025)
prompting-speciinne specifically, 4-shot for all models across all tasks. For evaluation metrics, we use the macro F1 score for CoQA to capture overlaps between predictions and ground-truth answers, while we measure accuracy (essentially exact match) for the rest of the tasks. In terms of efficiency, we also measure the costs of using the model APIs. It is important to note that AutoMix and FrugalGPT incur extra sctup costs that we did not factor into our results. These costs and associated latency represent significant constraint, especially for scenarios requiring frequent retraining or adaptation to new tasks, distributions, and models.
As shown in Figure 5 we observe that ABC is often more cost-effective than the baselines, even with their sophisticated routing mechanisms and singular model tiers. Our analyses show that ABC's efficient deferral strategy allows a more aggrossive utilization of cheaper models for a significant portion of the input while only using expensive models when necessary. For instance, we realize upon analyzing FrugalGPT's results that the trained scorer struggles as an efficient deferral signal as the tasks get harder; hence, it is more likely to take the safer option to cascade as test sample difficulty increases. This means that ABC can be expected to be more efficient since the scaling laws ensure that the sum of the costs of using several cheaper models is still much less than the cost of using the larger model in the next lier. AutoMix, on the other hand, uses a few-shot self- verifiiciin system that is sampled > 1 times; hence, the additional API calls add significantly to its cost of usage. Considering that the verfficcin process is an integral part of the AutoMix sctup, it can be guaranteed that ABC will always be cheaper to use than AutoMix, despite using more models.
E
Benefits of ABC
E.1
Parallel vs. sequential inference execution
w/ porallel inforence execution
perallel inference excutioo
94-
94
Coscade length=2 mmodels=2
X
Cascade length=2; nmodels=3
Cascade length=2; nmodels=1
Cascade length=2; 5
92
92
Cascade length=3; nmodels=2
Accurwcy
Aneuary
Cascade length=3; m
90-
90-
Cascade length=3; nmodels-4
Cascade lmnghh=3; nmodels-
Cascade length=4; nmodels
Cascade length=4; nmodels-3
88
88
Cascade length=4; nmodels-4
Cascade length=4; nmodels=5
Top-1 models in cluster
Average FLOPs
x110
Average FLOPs
x10*
Figure 8: Impact of parallelization on ABC performance for CIFAR-10. Left: With parallel inference execution (p = 1), ABC configurations consistently outperform the best single models across different cascade lengths and ensemble sizes. Right: Even with sequential execution (p = 0), ABC maintains advantages over single models, though with reduced efficiency. The results demonstrate that while parallelization is beneficial, ABC remains effective even under sequential constraints when cost disparities are sufficient.
Based Section 5.1.1 we additionally show in Figure X the superiority of parallel inference exccution for cascading over using the best single model using CIFAR-10 a case study. We also show that in the worst-case scenarios in which every single inference is sequentially produced over each ensemble and cascade, there are still considerable savings over the largest single modols, if the scenarios assumptions are met.
E.2 Cost Benefits
Based on Section 5.2.2 Table 4 shows the GPUs' pricing across several tiers retricved from Lambda Cloud. Table 5 shows a detailed analysis of costs across all cascade tiers, associaled with the number of cascade exits at each cascade tier, to provide holistic efficiency comparisons of the aggregated cascade costs against using the best (and largest) model and ABC dominates in every mcasured metric. Typically, most cascade exits occur in the earlier (and much cheaper) tiers (as also shown in Table 5), ensuring that the more cost-intensive cascade tiers featuring are reserved for the harder test instances.
26

---

Published in Transactions on Machine Learning Research (07 (07/202) /025)
Table 4: GPU rental costs from Lambda Cloud (Lambda 2024p (September 2024) showing the substantial cost disparities between hardware generations. The 25x cost difference between H100 and V100 GPUs, combined with more modest throughput differences, creates favorable conditions for ABC's heterogeneous hardware placement strategy described in Section 5.2.2
GPU
Cost per Hour (USD)
V100
0.5
A6000
0.8
A100
1.29
H100
2.49
Table Detailed cost breakdown across cascade for each dataset, showing the fraction of samples processed at each tier, associated GPU costs, latency, and FLOPs. 'he high fraction of samples processed at cheaper early tiers (52-93%) demonstrates ABC's effectiveness at concentrating expensive computation on truly difficult samples. ABC consistently oulperforms single best models across all metrics while achieving substantial cost savings.
Best Single
Dataset
Metric
Tier
Tier 2
Tier
Tier
ABC
Model
CIFAR-10
Frac. Samples (total=10,00g)
0.73
0.09
0.08
0.10
1.00
1.00
Total GPU Cost ($ hour)
0.36
0.07
0.11
0.24
0.79
2.49
Avg. Latency (ms)
3.11
3.79
7.76
9.07
4.13
9.07
Avg. FLOPPs
5.42e6
2.32e7
1.16e8
2.47e8
3.97e7
2.48e8
ImageNet-1K
Frac. Samples (total=50,000)
0.52
0.29
0.19
1.00
1.00
Cost ($ / hour)
0.26
0.23
0.25
0.74
1.29
Avg. Latency (ms)
2.45
2.88
3.17
2.71
3.17
Avg. FLOPs
2.15e9
3.90e9
4.30e9
3.07e9
4.30e9
SWAG (MCQ)
Frac. Samples (total=20,006)
0.71
0.29
1.00
1.00
Cost ($ hour)
0.36
0.23
0.59
0.80
Avg. Latency (ms)
4.52
8.05
5.53
8.05
Avg. FLOPs
1.88e10
6.67e10
3.25e10
6.67e10
SST-2
Frac. Samples (total=872)
0.93
0.07
1.00
1.00
Cost ($) / hour)
0.46
0.06
0.52
0.80
Avg. Latency (ms)
3.88
7.22
4.13
7.22
Avg. FLOPs
5.43e9
1. 1.68e10
6.26e9
1.68c10
Twitter Fin News
Frac. Samples (total-822)
0.65
0.35
1.00
1.00
Cost. ($ / hour)
0.32
0.28
0.61
0.80
Avg. Latency (ms)
4.05
7.26
5.19
7.26
Avg. FLOPs
6.83e9
2.42e10
1.30e10
2.42e10
27


Reviews:
[{"review_id": "TImZlVY7cn", "reviewer": "Reviewer_91ij", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper proposes a generic cascading-plus-ensemble framework that requires no additional training. An agreement-based voting mechanism decides whether the models have converged, triggering an early-exit condition. The idea is intuitive, broadly applicable, and practically valuable.\n2. The authors provide a very thorough analysis of performance and usage scenarios.\n\n**Weaknesses**\n\n1. The method is not truly novel. A similar approach was already explored in \\[1], where early exit is determined by checking the consistency between the previous cascading expert and the current classification head.\n2. The experimental section on performance is rather limited. The paper mainly compares its method with WoC on classification tasks, and with FrugalGPT and AutoMix on the LLM understanding benchmarks GSM8K, HEADLINES, OVERRULING, and COQA. This raises several concerns:\n\n   * WoC is from 2021, while FrugalGPT and related baselines are from 2022–2023 and are relatively weak by today’s standards.\n   * The experiments focus solely on classification tasks. Many real-world applications cannot be framed as classification problems, which calls the generality of the approach into question.\n3. The paper includes virtually no ablation study to justify the design choices or quantify the contribution of each component, leaving the actual benefit of cascading and voting unclear.\n\n\\[1] Li, Ziyue, et al. “Towards Inference Efficient Deep Ensemble Learning.” *Proceedings of the AAAI Conference on Artificial Intelligence*, 37 (7), 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The primary contribution of this paper is an Agreement-Based Cascading strategy that merges the strengths of traditional cascading and ensemble methods. Thanks to the ensemble’s ability to run in parallel, the proposed approach delivers a performance advantage over purely sequential cascading.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In view of the weaknesses noted above, the paper should at least present a comprehensive ablation study and thorough design analysis. It would also be preferable to include stronger baselines so that readers can better assess the effectiveness of the proposed method.", "broader_impact_concerns": "There is no obvious broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cc9LNaOAca", "reviewer": "Reviewer_rnxo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths \n\n$\\bullet$ The proposed optimizations in ABC are training-free and works with off-the-shelf checkpoints. It can be employed to leverage the publicly available models (such as LLMs on hugging face hub).  \n\n\n$\\bullet$ The efficiency evaluation  distinguishes FLOPs, latency, GPU-rental $, and API-token $. This provides convincing evidence of the s broad applicability and effectiveness of optimization steps in ABC. \n\n\n$\\bullet$ Theoretical guarantees, as Safe-deferral definition gives a clear criterion for no-regret cascades and explains the observed accuracy gains. \n\n## Weaknesses  \n\n$\\bullet$ **Limited fundamental novelty:** While the application (and experimental evaluation) is thorough, ensemble agreement for cascading has been explored in prior work (as the authors acknowledge), and the core insight is relatively straightforward, limiting the conceptual and algorithmic contribution.\n\n$\\bullet$  **Ensemble overhead not fully addressed:** The authors  did not discuss  scenarios where the computational overhead of running multiple models at each tier dwarf their benefits, particularly when parallelization is limited or models are of similar scale.\n\n$\\bullet$ Threshold estimation undermines ``training-free'' claims: Despite being positioned as training-free, ABC still requires estimating agreement thresholds using validation data, which introduces hyperparameter tuning that somewhat contradicts the training-free argument (and contribution).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Agreement-Based Cascading (ABC): a training-free adaptive inference method that uses ensemble agreement as the deferral mechanism in model cascades. In particular, the proposed scheme achieves efficiency through (i) routing inputs through a size-ordered cascade of pre-trained models, (ii) using in-tier ensemble agreement as a deferral criterion, and finally (iii) it is designed to be a drop-in replacement for any high-accuracy model. \n\nThe authors have perform  extensive experimentation  on vision and NLP benchmarks, edge-to-cloud settings, heterogeneous GPU serving, and black-box LLM APIs. Real-world applicability showing significant cost reductions in three practical scenarios: edge-to-cloud inference (14$\\times$ communication cost reduction), heterogeneous cloud serving (3$\\times$ cost reduction), and API-based inference (2-25$\\times$ cost reduction).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "$\\bullet$ Dynamic early-exit and adaptive subnetwork methods (e.g., DeeBERT, DyViT) are discussed but not compared experimentally. Hence, it is not clear how ABC fares against learned adaptive networks. Authors should Include early-exit BERT/ViT and state-of-the-art adaptive layers to contextualize ABC’s gains.\n\n$\\bullet$ **Sensitivity and ablations study** How performance varies with (i) ensemble size $k$, (b) validation-set size used to fit $\\theta$ (Fig. 6 suggests stability but numbers would help), and (c) different error-tolerance budgets $\\varepsilon$. How threshold choice affects performance across different datasets and model combinations? \n\n$\\bullet$ It is not clear in the draft that how to construct ensembles at each tier in practice -- are these different model architectures, different training procedures, or simply different model sizes?\n\n\n### Minors \n\n**Missing citations** \n\n1. Bolukbasi et al., Adaptive Neural Networks for Efficient Inference, ICML 2017\n\n2. Hu et al., Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference, ICLR 2020\n\n3. Shafiee et al., Efficient Inference on Deep Neural Networks by Dynamic Representations and Decision Gates, NeurIPS 2018\n\n\nThe paper would benefit from better structural organization and more concise writing. Important technical details and results are often obscured by verbose explanations, making it challenging to extract the key takeaways efficiently. It could have been more structured formatting (bullet points, numbered lists), clearer section transitions, and moving some detailed explanations to appendices to improve the main narrative flow.", "broader_impact_concerns": "Broader Impact is not discussed in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zaTbXlDm7H", "reviewer": "Reviewer_hRB4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. ABC is elegant and does not require additional training or task-specific adaptation. This makes it broadly applicable and easy to deploy.\n2. The authors validate ABC across diverse settings, including CIFAR, ImageNet, SWAG, SST-2, CoQA, and commercial API endpoints. \n3. The paper provides formal definitions of \"safe deferral rules\" and bounds on accuracy and inference cost under ensemble agreement.\n\nWeaknesses\n\n1. The current framework and deferral rule design assume fixed output spaces (classification or extractive QA). ABC is not applicable to open-ended generation tasks, which dominate many LLM applications.\n2. While the threshold is empirically tuned on held-out data, there is limited discussion about its robustness or failure modes across datasets. It seems sensitive to threshold theta.\n3. Requiring multiple pretrained models of similar size (to form ensembles per tier) could pose a deployment challenge in memory-constrained environments or for niche tasks with limited model availability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Agreement-Based Cascading (ABC), a training-free, adaptive inference strategy that uses ensemble agreement as a deferral mechanism across model cascades. Instead of relying on confidence scores or trained routers, ABC evaluates whether predictions of lightweight model ensembles agree, and only invokes heavier models if needed. The authors demonstrate that ABC can improve both accuracy and efficiency across vision and language tasks, including in real-world setups like edge-to-cloud inference, heterogeneous GPU serving, and black-box API access.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Provide more detail or formal guarantees on how the voting threshold is chosen and whether it generalizes across tasks.\n2. Add experiments or discussion on scenarios where ABC fails to deliver cost savings (e.g., when input distributions are consistently hard).", "broader_impact_concerns": "There is no outstanding ethical issues of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4Cvl2ADqz9", "reviewer": "Reviewer_4pJr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nSimplicity and Effectiveness: ABC is a simple yet effective approach to adaptive inference.  It does not require additional training or complex routing mechanisms, making it easily applicable to existing models.    \n\nStrong Performance: The paper provides both theoretical analysis and empirical evidence to support the effectiveness of ABC.  ABC not only improves efficiency but also often enhances accuracy compared to the models it replaces.    \n\nPractical Applications: The paper demonstrates the practicality of ABC in several real-world scenarios, including edge-to-cloud inference, cloud-based model serving, and API-based inference.  These experiments highlight the potential of ABC to significantly reduce costs in various deployment settings.    \n\nThorough Evaluation: The authors evaluate ABC across a variety of language and vision tasks, using benchmark datasets and diverse models.  They also compare ABC with existing cascading methods, demonstrating its superiority.    \n\nClear Presentation: The paper is well-written and easy to follow.  The authors provide clear explanations of the proposed method, theoretical analysis, and experimental results.  The use of figures and tables effectively enhances the presentation. Further, the related work is very comprehensive giving the reader an understanding of the current landscape. \n\nWeaknesses:\n\nEnsemble Overhead: The use of ensembles at each cascade level introduces additional computational costs.  While the authors argue that these costs can be mitigated by parallelization and large differences in model sizes, the overhead could still be a concern in some scenarios.  It would be beneficial to explore strategies to reduce the ensemble overhead, such as using more efficient ensemble methods or techniques to predict ensemble agreement.   \n\nDeferral Rule Tuning: The deferral rule in ABC relies on a voting threshold that needs to be estimated empirically.  The paper mentions that this is done on a small set of unseen data, but more details on the sensitivity of the results to this threshold and methods for robust threshold selection would be valuable.    \n\nLimited Scope: The current work focuses on classification and tasks with a fixed set of possible outputs.  The authors acknowledge that their deferral rule and the baseline methods they compare to are not directly applicable to open-ended generation tasks.  Extending ABC to open-ended generation or other more complex tasks would broaden its applicability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Agreement-Based Cascading (ABC), a novel adaptive inference technique.  ABC is designed to reduce the computational cost associated with large machine learning models by using smaller models for easier examples.  The core idea is to create a cascade of models with increasing complexity and to use the agreement between an ensemble of models at each level to determine when to move to the next level.    \n\nThe authors theoretically and empirically demonstrate that ABC can match or exceed the accuracy of the largest model it aims to replace, while also improving efficiency.  They apply ABC to three practical scenarios: edge-to-cloud inference, cloud-based model serving, and inference via model API services.  Results show significant cost reductions in each scenario.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above. Overall the paper is a good read for the audience and I would recommend an acceptance to TMLR.", "broader_impact_concerns": "Usually, majority based agreement can lead to propogation of bias. It would be good to double check the impact of ABC in that line of evaluation.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)