Why Ask One When You Can Ask $k$? Learning-to-Defer to the Top-$k$ Experts
Authors: Yannis Montreuil
Axel Carlier
Lai Xing Ng
Wei Tsang Ooi
Venue: ICLR 2026 Poster (2026)
Keywords: learning to defer
selective prediction
routing
machine learning
Primary area: learning theory
TL;DR: Generalizing Learning-to-Defer: From Single-Expert Deferral to Top-$k$ Expert Selection

Abstract: Existing _Learning-to-Defer_ (L2D) frameworks are limited to _single-expert deferral_, forcing each query to rely on only one expert and preventing the use of collective expertise. We introduce the first framework for _Top-$k$ Learning-to-Defer_, which allocates queries to the $k$ most cost-effective entities. Our formulation unifies and strictly generalizes prior approaches, including the _one-stage_ and _two-stage_ regimes, _selective prediction_, and classical cascades. In particular, it recovers the usual Top-1 deferral rule as a special case while enabling principled collaboration with multiple experts when $k>1$. We further propose _Top-$k(x)$ Learning-to-Defer_, an adaptive variant that learns the optimal number of experts per query based on input difficulty, expert quality, and consultation cost. To enable practical learning, we develop a novel surrogate loss that is Bayes-consistent, $\mathcal{H}_h$-consistent in the one-stage setting, and $(\mathcal{H}_r,\mathcal{H}_g)$-consistent in the two-stage setting. Crucially, this surrogate is independent of $k$, allowing a single policy to be learned once and deployed flexibly across $k$. Experiments across both regimes show that Top-$k$ and Top-$k(x)$ deliver superior accuracy–cost trade-offs, opening a new direction for multi-expert deferral in L2D.

Published as a conference paper at ICLR 2026
WHY ASK ONE WHEN YOU CAN ASK k? LEARNING- TO-DEFER TO THE TOP-k EXPERTS
Yannis Montreuil
Axel Carlier
School Computing
Fédération ENAC ISAE-SUPAERO ONERA Université de Toulouse Toulouse, 31555, France
National University of Singapore Singapore, 117418 yannis www.reiitr....uu.mmmmmmm nus edu
Lai Xing Ng
Wei Tsang Ooi
Agency for Science, Technology and Research Institute for Infocomm Research Singapore, 138634
School of Computing
National University of Singapore Singapore, 117418
ABSTRACT
Existing Learning-to-Defer (L2D) frameworks are limited single-expert defer- ral, forcing each query to rely on only one expert and preventing the use col- lective expertise. We introduce the first framework for Top-k Learning-to- to-eeerr, which allocates queries to the l most cost-effective entities. Our formulation uni- fies and strictly generalizes prior approaches, including the one-stage and two- stage regimes, selective prediction, and classical cascades. In particular, it re- covers the usual Top-1 deferral rule a special case while enabling principled collaboration with multiple experts when k >> 1. We further propose Top-k(x) Soominggto-Deffee an adaptive variant that learns the optimal number of experts per query based on input difficulty, expert quality, and consultation cost. To en- able practical learning, we develop a novel surrogate loss that Bayes-consistent, Hen-connisteet in the one-stage setting, and (Hr, H Hoonssinttt in the two-stage setting. Crucially, this surrogate is independent of k, allowing a single policy to be learned once and deployed flexibly across k. Experiments across both regimes show that Top-k and Top-k(z) deliver superior accuracy-cost trade-offs, opening new direction for multi-expert deferral in L.2D.
INTRODUCTION
Learning-to-Dec -D-ffe (L2D) enables models to defer uncertain queries to external experts, explicitly trading off predictive accuracy and consultation cost (Madras al., 2018 Mozannar & Sontag 2020 Verma et al. (022). Classical I.2D, however, routes each query to a single expert. This design ill-suited for complex decisions that demand collective judgment. For instance, oncology, pa- tient cases are routinely reviewed by multidisciplinary tumor boards comprising radiologists, pathol- ogists, oncologists, and surgeons. Each specialist contributes a different perspective-imaginng histopathology, treatment protocols, and surgical considerations and only through aggregation can an accurate and safe recommendation be made (Jiang et al., 1999 Fatima et al., 2017). Similar multi-expert deliberation underpins fraud detection, cybersecurity. and judicial review (Dietterich) 2000). We believe this reliance on single expert constitutes fundamental limitation of existing L2D frameworks: in many high-stakes domains, deferring to only one expert is not desirable.
Motivated by these challenges, we introduce Top-k Learning-to-Defer, a unified framework that allocates each query to the k most cost-effective experts. Our formulation supports both major regimes of L2D. In the two-stage setting, all experts are trained offline, and a routing function is then trained to allocate queries either to one of the experts
fixed main predictor (Narasimhan ctt al.) 2022 Mao et al. 2023a 2024c; Montreuil et al., (2055]a) a)). contrast, the one-stage setting jointly learns the main prediction task and the allocation policy within single model, allowing

---

Published as a conference paper at ICLR 2026
both components to adapt during training (Madras et al. 2018 Mozannar & Sontag 2020). Our framework admits instantiations in both regimes, ensuring broad applicability.
We further propose Top-k(x), an adaptive extension that learns the number of experts to consult per query based on input complexity, expert competence, and consultation cost. To enable both fixed-k and adaptive deferral, design a novel surrogate loss that is Bayes-consistent, (H, Hyy K in the two-stage setting, -oncsssitt in the one-stage setting, and independent of k, allowing efficient reuse across cardinalities without retraining. Finally, we show that our framework strictly generalizes prior paradigms: selective prediction (Chow) 1970; CCortes et al. (016) and classical model cascades (Viola& Jones [0001] [Saberian & Vasconcelos 2014 Laskardds et al. 2021), with
the usual Top-1 I Bayes policy arising as special case. This situates Top-k/Top-k(a as a unifying and strictly more general framework for Learning-to-Deffer
Our main contributions are: (i) We introduce Top-k L2D, the first framework for deferral to the top-k experts, unifying both one-stage and two-stage regimes. (ii) We develop a k-independent surrogate loss with Bayes-, HA-, and (Hr, Hon-coosistency guarantees, allowing a single policy to be reused across all values of k without retraining. (iii) We show that classical model cascades are strictly subsumed as special case, and that the usual Top-I Bayes policy rule is recovered from both one-stage and two-stage formulations, as well as from selective prediction within our framework. (iv) We propose Tx)- an adaptive variant that learns the optimal number of experts per query under accuracy-cost trade-offs. (v) We provide extensive empirical results demonstrating that Top-k and Top-k(x)) consistently achieve superior aceuracy-cos trade-offs compared to prior L.21) methods.
2 RELATED WORK
Learning-to- Defer. Leamiiggto-Deffer (L2D) extends selective prediction (Chow) 1970 Bartlett
& Wegkamp 2008 Cortes et al. 2016 Geifman & El- Yaniv 2017 Cao et al.) 2022, Cortes et al., 2024] by allowing models not only to abstain on uncertain inputs but also to defer them to external experts (Madras et al.) 2018 Mozannar & Sontag 2020 Verma et al. 2022). Two main approaches have emerged. Iwo-stage frameworks, the base predictor and experts are trained offline, and a separate allocation function is Icarned to decide whether to predict or defer (Narasimhan et al. 2022 Mao et al. 2023a), with extensions to regression (Mao et al. 2024c), multi-task learning (Montreuil et al. 2025b), adversarial robustness (Montreuil et al. 2025al 2026b). online (Montreuil et al.)
2026c). and applied systems (Strong et al. 2024 Palomba et al. 2025] Montreuil et al.) (026a)) In contrast, one-stage frameworks train prediction and deferral jointly. The score- baree- formulation of Mozannar & Sontag (2020) established the first Bayes-consistent surrogate and has since become the standard, with follow-up work improving calibration (Verma et al., 2022] [Co al. 2024).
surrogate design (Charusaie et al.) [2222] Mao et al.
2024a: Wei et al. (024)) and guarantees such as H-consistencyy and realizability (Mozamnar et 2023 Mao et al. 2024b] 2025). Applications
span diverse classification tasks (Verma et al. 2022; Co et al. [2024] Keswani et al. 2021 Kerrigan
et al. 2021
Hemmer al. 2022 Benz & Rodriguez 2022 Tailor et al. 2024 Liu et al. 2024).
Top-k Classification. Top-k classification generalizes standard classification by predicting a set of top-ranked labels rather than a single class. Early hinge-based approaches (Lapin et al.) 2015 2016) were later shown to lack Bayes consistency (Yang & Koyejo (2020). and non-convex formulations raised optimization challenges (Yang& & Koyejo| 2020 Thilagar et al., 2022). More recent advances have established Bayes- and 'H-consistency for broader family of surrogates, including cross- entropy (Mao et al. (023b) and constrained losses (Cortes & Vapnik 1995). with cardinality-awar refinements providing stronger theoretical guarantees (Cortes al. 2024).
Gap. Existing L2D frameworks are restricted to single-expert deferral, a critical limitation: in high-stakes domains, robust decisions demand aggregating complementary expertise, while reliance on single expert amplifies bias and error. Crucially, no prior work enables top-k or adaptive top- k(x)) deferral in either one-stage or two-stage regimes, nor provides surrogate losses with provable consistency guarantees. We address this gap by introducing the first unified framework for Top-k and Top-k(z) L2D, supported by a k-independent surrogate loss that ensures statistically sound and cost-efficient multi-expert allocation.
2

---

Published as a conference paper at ICLR 2026
3 PRELIMINARIES
Let X be the input space and Z the output space, with training examples (a, 2) drawn i.id. from an unknown distribution D.
One-Stage L2D. In the one-stage regime (Madras et al., 2018: Mozannar & Sontag, 2020), pre- diction and deferral are optimized jointly through single model multiclass setting with label space 2 = y (1....,n) ,n), corresponding to n distinct categories. The system has access to J offline experts. In the deterministic case, each expert is mapping my : X 2. An expert may be modeled as stochastic predictor. In this case, its output M, is defined as a random variable jointly distributed with (X,, Y), and training samples include realizations viz drawn from the conditional distribution P(M, X z, Y y). All our results remain valid when experts are stochastic; the analysis extends verbatim by treating each expert's output as random variable jointly distributed with (X,Y) Y).
We treat both class labels and experts uniformly as entities. The corresponding entity set
A = .n....,n] U {n + 1,..., J):
where indices j S n correspond to predicting class j. and indices j > n correspond to deferring to expert mn-n- We define the hypothesis class of score-based classifier as Hi (h: X' x A - R). For any h HA, the induced decision rule selects h(x) arg maxje.4" hax,,, i.e., the entity in A with the highest score. If h(x) S n, the predictor outputs class h(x)) E )); otherwise, it defers expert "(x))""' The hypothesis h is learned by minimizing the risk induced by the deferral
loss (Mozannar& Sontag 2020) Verma al. [022 Cao al. [0224] Mao et al. 2024a).
Definition 3.1 (One-Stage Deferral Loss). Let E N. E V, and h E Hr be score-based classifier. The one-stage deferral loss is
((()))))))) = 11((z) # v)1(h(()) S sn + Σ
c 1(h(x) = +
1=1
2 (h, x. y) Σラ (1 Cj (z, y) D (h, x. + j), where t belongs the cross-entropy family (Mohri al. 2012 Mao et al. 2023b). The cost is defined as
11.." with surrogate def (h, "h,
9 X' x y + [0, with (z, = 0z 1 1mn5 (
7 ") + Bj. where 0j 2
0 penalizes prediction
error and By 2 is fixed consultation fee.
Two-Stage L2D. In the two-stage regime (Narasimhan et al. 2022 Mao et al. 2023a 2024c Montreuil et al. 2025b]a), a). the main predictor and experts are trained offline and remain fixed. Unlike the stage setting, where a single augmented classifier jointly performs prediction and deferral, the two-stage approach introduces a separate rejector that allocates queries among enti- ties. Formally, we consider output space Z and a main predictor g E Hg with predictions ((x) E Z, which is fully observable to the system. We also assume access to a collection of experts {m, X + 2} We treat both the main predictor and experts uniformly entities. The corresponding entity set is
2 (1, (1..... (1....,J+11 1),
where j = denotes the base predictor and j 2 2 denotes expert m We define the hypothesis class of rejectors H, {r x 2 + R). For any r E H., scores are assigned to entities, and the induced decision rule is f(z) arg 42 r(z,j). If ((x) = 1, the system outputs the base predictor's 's label ğ(z); otherwise, defers to expert m1(m)-1- The deferral loss then defindd as follows.
Definition 3.2 (Two-Stage Deferral Loss), Let E X, E Z, and r two-stage deferral loss and its convex surrogate are
a rejector. The
J+1
J+1
(2(())))
Σ (((())) j}.
$22" (r. x2
Σ j=1
8 (r, x,),
where Cj X Z R4 is defined (1 a)((0(x), + B1 with s task task-specific
penalty (c.g., RMSE, mAP, or 0-1 loss) and (z, 2) = a (x). 2) + Br for j 2 2. The term
(I, =) Σizj cxxx, aggregates the costs of all non-selected entities.
3

---

Published as a conference paper at ICLR 2026
Consistency. We restrict attention to the one-stage regime for clarity. The objective is to learn hypothesis h E Hr, that minimizes the expected deferral risk Egy (h) Ex,y y (er (h (,)),, Y)), with Bayes-optimal value E (Hr.) = inf Exse (h). Direct optimization is intractable due to discon- tinuity and non-differeniiabilit (Zhang 2002 Steinwart 2007| Awastth al.) 2022 Mozannar &
Sontag 2020, Mao et al.] 2024a). motivating the use of convex surrogates. A prominent class the comp-sum family (Mao al. 2023b), which defines cross-entropy surrogates as
4% (h, 4
Σ J'EA
where the outer function T is parameterized by
0. Specific choices recover canonical losses: (v) = log(1 + v) (logistic), (V"(v) = -- (1 + v)=- -] for u + 1, covering
sum-exponential (Weston & Watkins [998), logistic regression (Ohn Aldrich 1997). generalized
cross-entropy (Zhang & Sabuncu 2018), and MAE (Ghoss et al. 2017).
fundamental criterion for surrogate adequacy is consistency, which requires that minimizing sur- rogate excess risk also reduces true excess risk (Zhang) 2002 Bartlett et al., 2006; [Steinwart] [007]] Tewari & Bartlett (007). To formalize this. /Awasthi et al. (2022) introduced the notion of H4-- consistency bounds, which quantify consistency with respect restricted hypothesis class rather than all measurable functions. The following bound has been established in the one-stage L2D setting (Mao et al. (0244)).
Theorem 3.3 (H consiitnity bounds). Suppose the surrogate 4h is Ha.-calibrated for any distri- bution D. Then there exists a non-decre function r E - Rt, depending on u, such that for all h E Hr.
Eogg(1) eB (Ha) + Ugg (Ha) S T= ' (Eggg. (h) E 10. (Hr) + UUglo. (Hn)) Ep1
Y)] is the minimizability gap, which measures the irreducible approximation error due to the expressive limitations of Hr. When Hr is sufficiently rich (e.g., Hr, Hall), the gap vanishes, and the inequality recovers Bayes- consistency guarantees (Steinwart] 2007 Awasthi et al. 2022). Taking the limit of this bound recov- ers the same Bayes-consistency established in Mozannar Sontag (2020).
Here Ugg (Hr) &R, (Hr) Ex inn B
GENERALIZING LEARNING-TO-DEFER TO THE ToP-k EXPERTS
4.1 FROM SINGLE TO ToP-k EXPERT SELECTION
Notations. Prior Learning-to taaeettt--Diiiite tDDeeerr methods allocate each input x E X' to exactly one entity, corresponding to a top-/ decision rule (Mozannar & Sontag 2020 Verma et al., 2022 Mao al.) 2024a). Formally, this is captured by the one-stage deferral loss in Definition 3.1 or its two-stage counterpart in Definition 3.2 To unify notation across both regimes, we define the hypothesis class of decision rules as H₂ = { X x A + R). For any π E Hy. the function assigns a score π (xx to each entity j E A, and the induced selection rule
#(x) arg max JEA
In the one-stage regime, π coincides with the augmented classifier h and A = AT* while in the two-stage regime, coincides with the rejector and A 2² For clarity, will heneeforth use A without superscripts, with the understanding that it denotes the appropriate entity set for the regime under consideration.
Top-k Selection. We generalize L2D to a top-k rule, where each query may be assigned to several entities simultaneously, enabling multi-expert deferral and joint use of complementary expertise. We first formalize the top-k selection set:
Definition 4.1 (Top-k Selection Set). Let x E X' and let X x A R be decision rule that assigns a score π (x. to each entity j E A. For any 1 < k: < [A]. the top-k selection set is
T (x) {[1] [2]+-- -.. [
4

---

Published as a conference paper at ICLR 2026
where [d] denotes the index of the i-th highest-scoring entity under x(x, The ordering is non- increasing: π(2 [1] 2 π (z, [2]=)= 2 ... π (2, [
Choosing A: 1 recovers the standard top-1 rule (z) farg max (z) which corre- {arg ja h(x,))
sponds to 1 22
j)] in the one-stage setting (Mozannar & Sontag 2020
Cao et al. 2024 Mao et al. 2024a) and 1T (x) = (arg max r(x, 7) in the two-stage setting
(Narasimhan et al. [022 Mao et al. 2023a 2024c Montreuil et al. 2025b).
Remark 1. We further show in Appendix A.6 that the Top-k Selection Set subsumes classical cas-
cade approaches (Viola & Jones 2001] Saberian & Vasconcelos) 2014 Dohan et al. 2022 Jitkrittum
al., (223) strict special case, thereby unifying cascaded inference and multi-expert deferral under single framework.
Top-k True Deferral Loss. L2D losses are tailored to top-1 selection and not extend directly
to k > 1. In the one-stage case (Definition 3.1), terms such as 1 [((xx) of y) [h(2z) S n) enforce exclusivity, assuming exactly one entity is chosen. This assumption breaks in the top-k setting: the selection set (x) may simultancously include the true label y and multiple experts with heteroge- neous accuracy and cost. A naive extension, c.g. 1 E [I)( is inadequate for three reasons: (i) collapses correctness the mere inclusion of y, ignoring whether the consulted experts themselves are reliable; (ii) fails to account for the cumulative consultation costs incurred when querying several entities; and (iii) yields non-decomposable set-based indicators, which obstruct surrogate design since the accuracy-cost tradeoff is determined jointly at the set level rather than per entity. These issues motivate a reformulation of L2D losses to handle top-k deferral.
Each regime specifies an entity set A and associated functions {āz X + 3}JAA:
One-stage: A (1...., n + J), where (x = j for S n (predicting label j), and (xt)(x) = (2) for j = 1,.... (deferring expert m,).
Two-stage: A = {l..... + 1}. where a1 (x) is the base predictor prediction g(x), and (1+y(x) (x)xx) for J.. (deferring expert m,).
For any entity j E A. we define an augmented cost (x, 2) = aj (āā (z), + Bj. where aj. By 2 0, and vb is a task-specific error measure (the 0-1 loss in classification, or any non-negative loss otherwise). By construction, M4 (I, E R.,.
Definition 4.2 (Top-k True Deferral Loss), Let x E X, E Z, and (x) C A be the top-k selection set. Let (z, 2) the cost of selecting entity j for input (x, 2). The uniformized top-k true deferral loss is
M
lsc, (Ic.)
Σ
We give a detailed explanation in Appendix A.7 This loss quantifies the total cost of allocating query to i entities, thereby unifying the one-stage (Definition (3.I) and two-stage (Definition 3.2) objectives into single formulation that explicitly supports joint decision-making across multiple entities. Unlike classical top-1 deferral, which only evaluates the outcome of a single choice, the top-k loss accumulates both predictive errors and consultation costs across all selected entities.
For instance, in binary classification with y = {1,2) and two experts, the entity set is A 21, 4). where j 2 correspond to labels and j > 2 to experts. If the top-2 selection set is [1)(x)= = {3, 1}. the incurred loss is (z(2, 8) + μ1 (z, y), jointly reflecting the cost of deferring to both expert mi and predicting label 1.
Remark For l: 1, the top-k deferral loss reduces exactly the classical objectives: the one- stage loss in Definition 3.1 and the two-stage loss in Definition 3.2
4.2 SURROGATES FOR THE TOP-λ TRUE DEFERRAL LOSS
In Lemma 4.2 the top-k true deferral loss is defined via a hard ranking operator over the selection set
(x). This makes it discontinuous and non- aondifferentiable hence unsuitable for gradient- maadient-baedd op- timization. To enable practical learning, we follow standard practice in Learning-to-
(Mozan-
nar & Sontag 2020 Charusaie al. 2022
Cao al. 2024
Mao et al. 2024a Montreuil et al.
5

---

Published as a conference paper at ICLR 2026
2025b and introduce convex surrogate family grounded in the theory of calibrated surrogate losses
Zhang
2002, Bartlett et al. 2006).
Lemma 4.3 (Upper Bound on the Top-k Deferral Loss). Let x. E X, 2 2, and let S k S |A].
Let %i a convex surrogate in the cross-entropy family. Then the top-k deferral loss satisfies
Pad (T),
Σ JEA
Σ Em((x,z)
4% (T. - (IAA 1- - k)) Σ (5(2, JEA
The key observation is that the cost term EjEA My (x, does not depend on the decision rule n, since each M1 (z. 2) aj (ā) 2)+ is fixed for a given (ar, 2) in both the one-stage and two-stage regimes. Furthermore, for all V [A], have 1 (x)) S 01 (TT, x; (Lapin et al. 2017 Cortes et al. 2024). Consequently, minimizing the upper bound reduces minimizing only the first term, and the optimization becomes independent of k. This directly yields the following tight surrogate family:
We prove Lemma 4.3 in Appendix A.8
Corollary 4.4 (Surrogates for the Top-k Deferral Loss). Let x. E X, z E 2, and let π : X x A - be a decision rule. The corresponding surrogate family for the top-k deferral loss is
4 Di (T,
Σ JEA
Σ Eu(erz)
2 (T,
which is independent of
This independence is a key strength: a single decision rule can be trained once and reused for any cardinality level k eliminating the need for retraining and allowing practitioners to adjust the number of consulted experts dynamically inference time depending on budget or risk constraints. Algebraically, the surrogate in Corollary 4.4 coincides with the formulation of Mao et al. (2024c), but our derivation shows that this form arises as convex upper bound for all it. Thus, the loss expression itself remains unchanged, while our framework extends the underlying deferral objective, the decision rule, and the guarantees from top-1 to the general top-k setting.
However, convexity and boundedness alone do not suffice for statistical validity (Zhang) 2002 Bartlett et al. 2006). Crucially, the fact that our surrogate coincides algebraically with that of Mao et al. (2024c) does not imply that their guarantees transfer: their analysis establishes consistency only in the top-1 regime, leaving the multi-entity case k > unresolved. Extending consistency from k- 1 to k > is generally non-trivial, as shown in the top-k classification literature (Lapin) jet al. 2015 2016 2017 Yang & Koyejo, 2020) Cortes et al., (024), where set-valued decisions introduce fundamentally different statistical challenges. Closing this gap requires new analysis. In the next subsection, we establish that minimizing any member of the surrogate family & det, yields consistency for both one-stage and two-stage L2D, thereby guaranteeing convergence to the Bayes- optimal top-k deferral policy as the sample size grows.
4.3 THFORETICAL GUARANTERS
While recent work by Cortes et al. (2024) has established that the cross-entropy family of surrogates 6% He-consistent for the top-k classification loss lg (T)) i E [x)x)), the consis- tency of top-k deferral surrogates remains unresolved and requires dedicated theoretical analysis. Unlike standard classification, deferral introduces an additional layer of complexity: costs depend jointly predictive accuracy and consultation with heterogeneous experts, and errors propagate differently depending on whether the system predicts directly or defers. These factors fundamen- tally alter the Bayes-optimal decision rule, making existing results insufficient. Prior analyses have addressed the lo 1 case in one-stage and two-stage settings (Mozannar & Sontag 2020) Vermal al. 2022 Mao al. 2024a c) but extending consistency guarantees to <> 1 is non-trivial. Our theoretical analysis fills this gap by proving that the surrogate family h is Bayes- and class- consistent for the top-k deferral objective, thereby establishing statistical validity of learning in this more general regime.
To proceed, we impose only mild regularity conditions on the hypothesis class Hy: (i) Regularity: for any input v, the scores (xx,)) induce striet total order over all entities in A; (ii) Symmetry: the

---

Published as a conference paper at ICLR 2026
scoring rule is invariant under permutations of entity indices, i.e., relabeling entities does not affect the induced scores; (iii) Completeness: for every fixed z, the range of x(x )) is dense in R.
These assumptions standard and satisfied by common hypothesis classes, including fully connected neural networks and the space of all measurable functions H (Awasthi et al. 2022
4.3.1 OPTIMALITY OF THE ToP- SELECTION SET
A central challenge Learning-to-Dest is deciding which entities consult test time, given their heterogeneous accuracies and consultation costs. For & = 1, prior work has established that the opyesaoptimal policy selects the single entity with the lowest expected cost (Mozannar & Sontag 2020 Verma et al. [022]] Narasimhan et al. 20222 Mao et al.) 2023a; 2024a). The key question we address is how this principle extends to the richer regime Appendix A.9
1. We prove the following Lemma in
Lemma 4.5 (Bayes- (Bayes-Optine Optimal Top-k Selection). Let x E X. For each entity j A, define x=1l,(z Z)}, its Bayes-optimal expected cost as (x)
the expected cost [z(z)
inf inf gEHt, (z)x)). Then the Bayes-optimal top-k selection set is
12 arg min
E Emp(s) [11] 12 B..... [1
I.SA Mal=R JEII.
where 11,,0 denotes the index of the i-th smallest expected cost in {uz (x) : E A}. In the one-stage regime, where no base predictor class Hg is defined, we simply set 1uj (x) (x)
Lemma 4.5 shows that Bayes-optimal top-k deferral is obtained by ranking entities according to their expected cost and selecting the lowest.
Corollary 4.6 (Special cases for k = 1). The Bayes rule in Lemma 4.5 recovers prior Top-1 results:
1. One-stage L2D. For any entity j (labels j S n and experts j > T))
(z) = a (a,(z) * Y| x) + Bg,
which yields the Top-1 Bayes policy of Mozannar Sontag (2020).
2. Two-stage L2D. Let j = denote the base predictor and j 2 the experts. Then
T (x) = a1 inf Ez (e)( Z)] + 81, g
and for j 2 2. recovering the Top-l allocation in Narasimhan et al. (0022); ;: Mao et al. (2023a): Montreuil et al. (2025b).
( (x)) 2) + 33.
3. Selective prediction (reject option). We take the set of label entities and augment it with an abstain entity 1, defined by = 0 and B1 x 0, while label entities use og =
1, 3, Then
TT (c))=
( (1....... m)).
(+) = =,
yielding the Chow's 's rule (Chow 1970).
We defer the proof of this corollary and give additional details in Appendix A.10 The Top-k Bayes policy strictly generalizes all prior Top-1 results: it reduces known rules when i 1, but for k> it yields principled way to combine multiple entities under unified cost-sensitive criterion.
4.3.2 CONSISTENCY OF THE TOP-k DEFERRAL Loss SURROGATES
Having established the Bayes-optimal policy in Lemma 4.5 we now turn to the surrogate family tn Pe The central question is whether minimizing the surrogate risk guarantees convergence ward the Bayes-optimal policy for the top-k true deferral loss (Lemma 4.2). This property, known
as consistency, is crucial: without it, empirical risk minimization may converge to arbitrarily sub-
optimal policies. While consistency has been established for & 1 in both one-stage (Mozannar &
Sontag 2020; Verma et al. (022 Mao et al., (2024) and two-stage (Narasimhan et al. 2022 Mao
al. 2024c Montreuil al. 2025b), no prior results extend to the richer regime k

---

Published as a conference paper at ICLR 2026
Theorem 4.7 (Unified Consistency for Top-k Deferral). Let A denote the of entities. Assume that H is symmetric,, complete, and regular for top-k deferral, and that in the two-stage case, Hg the base predictor class. Let S == (IA) - E) (x]=,(X)], Suppose Dir is Hx Consissent for top-k classification with non-negative, non-decreasing, concave function I=
One-stage. Let Rxxx (x)) aa; XV) By. For any E Hrs
(h) Eacck (Hh) + Meaut (Hn) - kS Evice. (h) E (Ha) + Ulpurr (Hr)
S
Two-stage. Let Ex [T, (X)] a,,x,z [b) (X), 2)] + Bz. For any (r,g) E Hr x Ha.
Era (r. g) 8B & (Hr, Hgg) + (Hr, H₂g - Ex Expir,n (r) Exb int
inf Tı (X)] gEHs (H,,
Uagar (Hr)
+ SK
5
1t log(1 + v) + 1-v log(1 v) (logistic). Do(u) = 11 v1 22 (exponential), and (2(e) = 0/1 (MAE).
with T (v))
We give the proof in Appendix A.II Theorem provides the first consistency guarantees for top-k deferral across both one-stage and two-stage regimes. The bounds reveal that the excess deferral risk depends explicitly on k: consulting more entities enlarges the cost term S. At the same time, calibration of 2% ensures that minimizing surrogate risk drives the excess true risk to zero, establishing both Hr. (Hy. Hg), and Bayes-consisteney. learned policies converge to the Bayes-optimal top-k deferral rule from Lemma 4.5 as data grows.
In the two-stage regime, we assume the Bayes-optimal cost is attainable (or can be arbitrarily well approximated), i.e., there exists sequence It Hg such that Ex [T,, (X) (X)) - 0. Fur- thermore, if there exists E Hr with Ex (rx) Ejgn (H,) + Usn (Hr) + 0, then by
det Theorem 4.7 and the fact that e + KST (u/S) is nonnegative and nondecreasing on [0, oo) with r. (0) U, we obtain Elame (re. 9t) 8
Ea . (Hr, Hg) + Ueans (Hr, Hg) + 0, which shows that
the surrogate indeed minimizes its target loss.
The minimizability gap vanishes under realizability and, more generally. whenever the hypothesis class is sufficiently rich, for instance when H., Hلبll (Steinwart) 2007 Importantly, by setting
A == 1, we recover the established HA -a-connistency bounds for one-stage L2D (Mao et al. 2024a) and (H,, Hl -consistency bounds for two-stage L2D (Mao et al. 2024c 2023a| Montreuil et al, 2025b). Thus result strictly generalizes prior work, while covering the entire cross-entropy surrogate family, including log-softmax, exponential, and MAE. This unification provides the first rigorous statistical foundation for multi-expert deferral.
ToP-k(x)): ADAPTING THE NUMBER OF ENTITIES PER QUERY
While our Top-k deferral framework enables richer allocations than prior works, it still assumes uniform cardinality across all queries. In practice, input complexity varies: some instances may require only one confident decision, while others may benefit from aggregating over multiple entities. To address this heterogeneity, we propose an adaptive mechanism that selects a query- specific number of entities.
Following the principle of cardinality adaptation introduced in Top-k classification (Cortes et al,, 2024), we define cardinality function Ki X' A, parameterized by hypothesis class H4. For given input æ, the function selects the cardinality level via ko(xx) arg max k(x,, i) and returns the Top-k(x)) subset II (2)))) (x) S (x) produced by the scoring function π (I.
Definition 5.1 (Cardinality- Aware Deferral Loss). Let a E X,, and let IT (T)
(x) denote the adaptive TTo-k(<:) subset. Let denote metric, { R - R+ non-decreasing function, and À 2 0
8

---

Published as a conference paper at ICLR 2026
regularization parameter. Then, the adaptive cardinality loss is defined as
(x))
feard (z), (x), x,, - d(I ko()) (o))) + A
Σ 111t)
with surrogate (2 Kg, Even (1- Eard v, 10 (kig I, v), where
leard a normalized version of the cardinality-avv loss and 11st denotes the consultation cost of the i-th ranked entity. This surrogate is 74 conscinsst, and the proof follows the same structure as Cortes et 2024).
The term d(IT (( (a) xx, captures the predictive error of the selected set and can be instantiated using top-k accuracy, majority- emror, any other task-dependent aggregation metric (see Appendix A..13 for examples). The second component penalizes costly selections, encouraging the model to query additional entities only when the expected accuracy gain justifies the consultation cost.
Bayes-Optimal Cardinality Selection. We now characterize the behavior of the adaptive cardi- nality mechanism induced by the cardinality- aware deferral loss. For a fixed input x X and any cardinality level (1....,AAl] define the conditional risk associated with selecting the top-k entities as
(k)
= Rlleara (I), X = x] (ld(TT X x]+ X
(1)
The Bayes-optimal cardinality function is therefore given pointwise by
B (z) E arg
min kEI
Cerrrd (ls).
(2)
To understand when it beneficial query an additional entity, define the marginal difference in conditional risk
8Ce (k)) == Cecara (k) Cecara (k- 1),
S :
3 1
direct decomposition yields
8Ceaaa (k) E[d([T] Z) -d[T Z) X = x. + ((((S)) E(Sk-1))
(3)
(k)
Consequently, increasing the cardinality from & to strictly decreases the conditional risk if and only if
Cicnca (k + 1) S Creara (k)
-DD 2 A[E(S.-1) - E(St)]
(4)
Equation equation provides a transparent decision rule for adaptive querying: an additional entity is selected if and only if the expected reduction in predictive error obtained by aggregating it out- weighs its marginal consultation cost. The regularization parameter À directly controls this trade- off, interpolating between conservative policies that favor small cardinalities and accuracy-driven regimes that aggregate over larger sets when beneficial. This characterization shows that adaptive Top-k(x)) deferral is principled consequence of risk minimization rather than heuristic design choice.
EXPERIMENTS
We evaluate our proposed methods Top-k L2D and its adaptive extension Top-k(z)) L2D against state-of-the-ar one-stage (Mozannar & Sontag 2020 Mao al. 2024a) and two-stage (Narasimhan

---

Published as a conference paper at ICLR 2026
Montreuil et al. 2025b) baselines. Across all tasks, Top-k and Top-k(z) consistently outperform angle-expers deferral methods, demonstrating both improved accuracy cost trade-off's and strict generalization beyond A 1.
et al. 2022 Mao et al. 2023a 2024c;
In the main text, we report two-stage results on the California Housing dataset (Kelley Pace & Barry 1997), while deferring additional experiments for both settings CIFAR-I00 and SVHN to Appendix B.2, For the one-stage setting, we provide detailed evaluations on CIFAR-
10 (Krizheskyy 2009) and SVHN (Goodfellow et al. 2013) in Appendix [B.3 Evaluation met-
rics are formally defined in Appendices A.13 and B.1.1 We track the expected budget B(k)
Ex p and the expected number of queried entities [x]]T where k is fixed in Top-k L2D and input-dependent in Top-k(x) L2D. Algorithms are provided in Appendix A.4 with illustrations in Appendix A.5
Espeeridd Nacher Quatial Aesta
Repested Ntaslor Quiid Aguasi
Narioe quiied Agret
and baa
2nd
Trp Apd LSB Trp 130
Trp 4400
--
Tap 155
14
Trp 13t
-
Top L3F
RUSE...
Opeadd
0iee "Gee"""e"""
"rto" oidt "i1l "ide "ate " 1339
Eale "S10"" oits "110" "ie "otee"
(a) RMSEEia
(b) RMSEEg
(c) R.SEEwarg
Figure 1: Performance of Top-k and Top-k(r) L2D across varying budgets B. Each plot reports a different metrie: (a) minimum RMSE, (b) uniform average RMSE, and (c) weighted average RMSE B.I.I. Our approach outperforms the Top-1 L,2D baseline (Mao et al. 2024c).
Top-k(z) achieves a near-optimal RMSE of 6.23 with budget of 3 = 0.156 and an expected number of entities k: = 4.77, whereas Top-k requires the full budget = 0.2 and To = 6 entities to reach a comparable score (6.21). This demonstrates the ability of Top- k(=:) allocate resources more efficiently by querying only the necessary entities, in contrast Top- k, which tends to over-allocate costly or redundant ones. Additionally, our approach outperforms the Top-1 L2D baseline (Mao al. 2024c), confirming the limitations single-entity deferral.
Interpretation. In Figure la
Figures lb and Ic evaluate Top-k and Top-k(r) L2D under more restrictive metrics-RMSSI and RMSE where performance is not necessarily monotonic in the number of queried entities. In these settings, consulting too many or overly expensive entities may degrade overall performance. Top-k(ar) consistently outperforms Top-k by carefully adjusting the number consulted entities. In both cases, Top-k(z) achieves optimal performance using budget of just 3 = 0.095 a level that Top-k fails attain. For example, in Figure 1b Top-k(x)) achieves an RMSF = 8.53, compared to 10.08 for Top-k. This demonstrates that our Top-k(x) L2D selectively chooses the appropriate whei necessary-to enhance the overall system performance.
CONCLUSION
We introduced Top-k Searning-to-Dete generalization of the two-stage L2D framework that al- lows deferring queries to multiple agents, and its adaptive extension, Top-k(x) L2D, which dynam- ically selects the number of consulted agents based on input complexity, consultation costs, and the agents' underlying distributions. We established rigorous theoretical guarantees, including Bayes and (H,, Hl,) ( -oncoonist,,,,y, and showed that model cascades arise restricted special case of our framework. Through experiments on both one-stage and two-stage regimes, we demonstrated that Top-k and Top-k(x:) L.2D consistently outperforms single-agent baselines.
8 REPRODUCIBILITY STATEMENT
All code, datasets, and experimental configurations are publicly released to facilitate full repro- ducibility. Results reported as the and standard deviation four independent runs, with
10

---

Published as a conference paper at ICLR 2026
fixed set of experts. For random baseline policies, metries averaged over fifty repetitions to reduce stochastic variability. All plots include error bars indicating one standard deviation. Dataset details are provided in Appendix [.1.3] while the training procedures for both the policy and the cardinality function are described in Algorithm| and Algorithm Proofs, intermediate derivations, and explicit assumptions are included in the Appendix.
ACKNOWLEDGMENT
This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISGG2PPDD-202--0--04111- and by A*STAR, and is part of the programme DesCartes which is supported by the National Research Foundation, Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CRE- ATE) programme.
REFERENCES
Pranjal Awasthi, Anqi Mao, Mehryar Mohri, and Yutao Zhong. Multi-class h-consistency bounds. In Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088
Peter Bartlett, Michael Jordan, and Jon McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101:138-156, 02 2006. doi: 10.1198/ 01621450000000900.
Peter L. Bartlett and Marten H. Wegkamp. Classification with reject option using hinge loss. The Journal of Machine Learning Research, 1848,--840, June 2008.
Nina Corvelo Benz and Manuel Gomez Rodriguez. Counterfactual inference of second opinions. In Uncertainty in Artificial Intelligence, pp. 453-463. PMLR, 2022.
Yuzhou Cao, Tianchi Cai, Lei Feng, Lihong Gu, Jinjie Gu, Bo An, Gang Niu, and Masashi Sugiyama. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. Advances in neural information processing systems, 35:521-534 2022.
Yuzhou Cao, Hussein Mozannar, Lei Feng. Hongxin Wei, and Bo An. In defense of softmax parametrization for calibrated and consistent learning defer. In Proceedings of the 37th In- lernational Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
Mohammad- Amin Charusaie, Hussein Mozannar, David Sontag, and Samira Samadi. Sample effi- cient learning predictors that complement humans, 2022.
C. Chow. On optimum recognition enror and reject tradeoff. IEEE Transactions Information Theory, 16(1):41-46, January 1970. doi: 10.1109/TT..190.1054444
Corinna Cortes and Vladimir Naumovich Vapnik. Support networks Machine Learning, 20: 273-297, 1995. URL
Corinna Cortes, Giulia DeSalvo, and Mchryar Mohri. Learning with rejection. In Ronald Ortner, Hans Ulrich Simon, and Sandra Zilles (eds.), Algorithmic Learning Theory, pp. 67-82, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46379.7,
Corinna Cortes, Anqi Mao, Christopher Mohri, Mehryar Mohri, and Yutao Zhong. Cardinality- aware set prediction and top-SkS classification. In The Thirty-eighth Annual Conference on Neu- ral Information Processing Systems, 2024. URL https: http:://pppeeevveeee net faaet/forum2i0 /tRLhttts//ppeeeeiieeee
WAT3qu737X
Thomas G. Dietterich. Ensemble methods in machine learning. In Multiple Classifier Systems, pp- 1-15, Berlin, Heidelherg, 2000. Springer Berlin Heidelberg. ISBN 978-3-540-4501466

---

Published as a conference paper at ICLR 2026
David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A Saurous, Jascha Seh--Dccsttein, et al. Language model cascades. arXiv preprint arX1v:2207.10344 2022.
Meherwar Fatima, Maruf Pasha, et al. Survey of machine learning algorithms for disease diagnostic. Journal of Intelligent Learning Systems and Applications, 901))1, 2017.
Yonatan Geifman and Ran El- El-va..v. Selective classification for deep neural networks. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.,
2017. URL /tttps:/pprocedd.ggg neurips httppps.ccpppppe cc/paper files/paper/2017 file ////4484223555 http://8822bee11raa0000eeee66660
Aritra Ghosh, Himanshu Kumar, and P. S. Sastry. Robust loss functions under label noise for deep neural networks. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, 17, pp. 1919-1925, AAAI Press, 2017.
lan J Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, and Vinay Shet. Multi-digit number recognition from street view imagery using deep convolutional neural networks. arXiv preprint arXiv: 608222 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Patrick Hemmer, Sehastian Schellhammer, Michael Vössing, Johannes Jakubik, and Gerhard Satzger. Forming effective human-Al teams: Building machine learning models that comple- ment the capabilities of multiple experts. In Lud De Raedt (ed.), Proceedings of the Thirty-First International Joint Conference Artificial Intelligence, IJCAI-22, pp. 2478-2484, International Joint Conferences on Artificial Intelligence Organization, 2022. doi: 10.24963/iccai.22222333 URL https://doi..c.c
24 00..0..29933111// 20122/33.2222/44 Main Track.
Yulei Jiang, Robert M Nishikawa, Robert A Schmidt, Charles E Metz, Maryellen I. Giger, and Kunio Doi. Improving breast cancer diagnosis with computer-aided diagnosis. Academic radiology, (1):22-33, 1999.
Wittawat Jitkrittum, Neha Gupta, Aditya K Menon, Harikrishna Narasimhan, Ankit Rawat, and Sanjiv Kumar. When does confidence-based cascade deferral suffice? Advances in Neural Infor- mation Processing Systems, 19:9891-9906 2023.
R. Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics and Probability Let- ters, 33(3):291-297, 1997. ISSN 0167-7152. doi: http:///ha.org//0000066666666553 00140-X. URL sttps://www. scwencedi rect. wwt.///cceeecc/attttleeeeeeeeeee
S0167 8016771529600144
Gavin Kerrigan, Padhraic Smyth, and Mark Steyvers. Combining human predictions with model probabilities via confusion matrices and calibration. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 4421-4434 Curran Associates, Inc., 2021. URL htttsss:/pppoceetinss.eeee.
ec cc/paper..ep.....ee files/paper/2022/ E E11e:330411eB8555o///2222...... pap. Paper.ppl
Vijay Keswani, Matthew Lease, and Krishnaram Kenthapadi. Towards unbiased and accurate de- ferral to multiple experts. Proceedings of the 2021 AAAI/ACM Conference on A1, Ethics, and Society, AIES 21, pp. 165, New York, NY, USA, 2021. Association for Computing Machin- ery. ISBN 478344038 doi: 10.1145/34661700.3445551
URL htttps:://i..ogg///
1145/34 02. 3462566
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL https: 7/api semant i cacholas www./ooopopossi1ee...
12

---

Published as a conference paper at ICLR 2026
Maksim Lapin, Matthias Hein, and Bernt Schiele. Top-k multiclass svm. Advances in neural infor- mation processing systems, 28, 2015.
Maksim Lapin, Matthias Hein, and Bernt Schiele. Loss functions for top-k error: Analysis and insights. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 1468-1477, 2016.
Maksim Lapin, Matthias Hein, and Bert Schiele. Analysis and optimization of loss functions for multiclass, top-k, and multilabel classification. IEEE transactions on pattern analysis and machine intelligence, 40(7):1533-155x) 2017.
Stefanos Laskaridis, Alexandros Kouris, and Nicholas D. Lane. Adaptive inference through early- exit networks: Design, challenges and directions. In Proceedings of the 5th International Work- shop on Embedded and Mobile Deep Learning, EMDL' '1, pp. 1-6, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450385978. doi: 10.114573469111.34400022 114p:///..1111/3330
URL
6 1 3470012
Shuqi Liu, Yuzhou Cao, Qiaozhen Zhang, Lei Feng, and Bo An. Mitigating underfitting in learn- ing to defer with consistent losses. International Conference on Artificial Intelligence and Statistics, pp. 4816-4824. PMLR, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101, 2017.
David Madras, Toni Pitassi, and Richard Zemel. Predict responsibly: improving faimess and accu- racy by learning to defer. Advances in neural information processing systems, 31, 2018.
Anqi Mao, Christopher Mohri, Mehryar Mohri, and Yutao Zhong. Two-stage learning to defer with multiple experts. In Thirty seventhh Conference on Neural Information Processing Systems, 2023a. URL
net wttp:eewwwweeee/tteeeeeeeee forum? waw.ammnnddIIIIIHH111111
Anqi Mao, Mehryar Mohri, and Yutao Zhong. Cross-entropy loss functions: Theoretical analysis PMLR,
and applications. In International conference on Machine learning, pp. 2023b.
Anqi Mao, Mehryar Mohri, and Yutao Zhong. Principled approaches for learning to defer with multiple experts. In ISAIM, 2024a.
Anqi Mao, Mehryar Mohri, and Yutao Zhong. Realizable h-consistent and bayes-consistent loss functions for learning to defer. Advances in neural information processing systems, 37:73638- 73671, 2024b.
Anqi Mao, Mehryar Mohri, and Yutao Zhong. Regression with multi-expert deferral. Proceedings of the s1st International Conference Machine Learning, ICML'24. JMLR.org, 2024c.
Anqi Mao, Mehryar Mohri, and Yutao Zhong. Mastering multiple-expert routing: Realizable ShS- consistency and strong guarantees for learning to defer. In Forty-second International Conference on Machine Learning, 2025.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning- MIT Press, 2012.
Yannis Montreuil, Hoang Duy Dang, Maxime Meyer, Lai Xing Ng. Axel Carlier, and Wei Tsang Ooi. Online learning-to-defer with varying experts. In The 29th International Conference on Artificial Intelligence and Statistics.
Yannis Montreuil, Axel Carlier, Lai Xing Ng, and Wei Tsang Ooi. Adversarial robustness in two- stage dearning-to-defer Algorithms and guarantees. In Forty- International Conference on Machine Learning. 2025a.
Yannis Montreuil, Yeo Shu Heng, Axel Carlier, Lai Xing Ng, and Wei Tsang Ooi. A two-stage Searning-to-defe approach for multi-task learning. In Forty-second International Conference on Machine Learning, 2025b.
13

---

Published as a conference paper at ICLR 2026
Yannis Montreuil, Yeo Shu Heng, Lai Xing Ng. Axel Carlier, and Wei Tsang Ooi. Optimal query allocation extractive QA with LLMs: A learning-to-defe framework with theoretical guaran- tees. In The 29th International Conference on Artificial Intelligence and Statistics, 2026a. URL net forum? di
Yannis Montreuil, Lai Xing Ng. Axel Carlier, Wei Tsang Ooi, and Yu Letian. Adversarial robustness in one-stage Icarning-to--a to-defer. In The 29th International Conference on Artificial Intelligence and Statistics, 2026b. URL htttps://pennreview net forum? iimmmiid-ggee id=gF St
Yannis Montreuil, Letian Yu, Axel Carlier, Lai Xing Ng, and Wei Tsang Ooi. Learning to defer in non-stationary time series via switching state-space models. arXiv preprint arXiv:2601.225.68
2026c.
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org. 2020.
Hussein Mozannar, Hunter Lang, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David A. Sontag. Who should predict? exact algorithms for learning to defer to humans. In Inter- national Conference on Artificial Intelligence and Statistics, 2023. URL httpss//app..
remanticscholar.co org/ wwtp/////ooi:/:///./.... 255534412225
Harikrishna Narasimhan, Wittawat Jitkrittum, Aditya K Menon, Ankit Rawat, and Sanjiv Ku- mar. Post-hoc estimators for learning to defer to an expert. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave. K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp. 29292-29304. Curran Associates, Inc., 2022. URL httpps://proceednggg... neuripss.e...ppp. cc cctpsppr.cppppppeepeeeee. files/papee/202//// Fil.oooffecaaade7703333eeeeeeeee
pdff
R A Ohn Aldrich. Fisher and the making of maximum likelihood 1912-1922. Statistical Science, 12(3):162-179, 1997.
Filippo Palomba, Andrea Pugnana, Jose Manuel Alvarez, and Salvatore Ruggieri. A causal frame- work for evaluating deferring systems. In The 28th International Conference on Artificial Intelli- gence and Statistics, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL
https: .rg/ bbrrg/abs22103 00020
Mohammad Saberian and Nuno Vasconcelos. Boosting algorithms for detector cascade learning. Journal of Machine Learning Research, 15(74):259--605, 2014. htth:tt::
papers/v15/ http://///saaarriiiseseeaaaeaaas an 4a. html
Ingo Steinwart. How to compare different loss functions and their risks. Constructive Approxi- mation, 26:225-288 2007. URL httpps:/app..sseet www.:eesssmmtticcchhhooee www.aaarrrrrrooooooo....
16660598
Joshua Strong, Qianhui Men, and Alison Noble. Towards human-Al collaboration in healthcare: Guided deferral systems with large language models. In ICML 2024 Workshop on LLMs and Cognition, 2024. URL htll:ttts///opeerettiie http://wwwweeeooornoeeeem 4cdm:c1rg-4dg9ymmmmmmm
Dharmesh Tailor, Aditya Patra, Rajeev Verma, Putra Manggala, and Eric Nalisnick. Learning to de- fer to a population: A meta-learning approach. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li (eds.), Proceedings of The 27th International Conference on Artificial Intelligence and Statis- fics, volume 238 of Proceedings of Machine Learning Research, pp. 3475-3483 PMLR, 02-04 May 2024. URL https::///........... htttp:://prcceedingg mwlggssmm.r........ press wtt.://eeee///33///taaa//2// htmml html
Ambuj Tewari and Peter L. Bartlett. On the consistency of multiclass classsiicccccasssscitttittiooooi methods. Journal of Machine Learning Research, -036) 2007. URL
v8/tewar 07a. htm
14

---

Published as a conference paper at ICLR 2026
Anish Thilagar, Rafael Frongillo, Jessica Finocchiaro, and Emma Goodwill. Consistent polyhe- dral surrogates for top-k classification and variants. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th Inter- national Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 21329921359. PMLR, 17-23 Jul 2022. URL httpps:/pprcceeddnn .ml.. press/v162 http://ve2//hilaazaeee2 html
Rajeev Verma, Daniel Barrejon, and Eric Nalisnick. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. In Interna- tional Conference on Artificial Intelligence and Statistics, 2022. URL https://aat
semanticscholas ort./Corpua
Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume I, pp. I-1, 2001. doi: 10. N..1099CCPR..201.99955
Zixi Wei, Yuzhou Cao. and Lei Feng. Exploiting human-ai dependence for learning to defer. In Forty-first International Conference on Machine Learning. 2024.
Jason Weston and Chris Watkins. Multi-class support vector machines. Technical report, Citeseer, 1998.
Forest Yang and Sanmi Koyejo. On the consistency of top-k surrogate losses. In International Conference on Machine Learning, pp. 10727-10735. PMLR, 2020.
Tong Zhang. Statistical behavior and consistency of classification methods based on convex risk minimization. Annals of Statistics, 32, 12 2002. doi: 10. NA..21///00/10791201300
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018
15

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
2
Related Work
Preliminaries
Generalizing Learning-to-DDefer to the Top-k Experts
From Single to Top-k Expert Selection
Surrogates for the Top-k True Deferral Loss
4.3
Theoretical Guarantees
4.3.1
Optimality of the Top-k Selection Set
4.3.2
Consistency of the Top-k Deferral Loss Surrogates
5
Top= Adapting the Number of Entities per Query
Experiments
Conclusion
10
8
Reproducibility Statement
10
Appendix
18
A.1
General Notations
18
A.2
Notations for Ordered Sets
18
A.3
Useful Definition
19
A.4
Algorithm
20
A.5
Illustration of Top-k(x) and Top-k L.2D
20
A.6
Model Cascades Are Special Cases of Top-k and Top-k(xr) Selection
21
A.6.1 I
Model cascades)
22
A.6.2
Embedding fixed-k cascade
22
A.6.3
Embedding adaptive (early-exit) cascades)
22
A.6.4
Expressiveness: Model Cascades Top-k Top-k Selection
22
A.7
Proof Lemma
23
A.8
Proof Lemma
25
A.9
Proof Lemma 4.5
26
A.10 Proof Corollary
28
A.11 Proof Theorem 4.7
29
A.12 Behavior of the Cardinality-Awazr Deferral Loss
34
A. 13 Choice of the Metric d
34
A.14 Use of Large Language Models
35
16

---

Published as a conference paper at ICLR 2026
B Experiments
35
B.I Resources
35
B.I. Metrics
35
B.1.2 Training
36
B.1.3 Datasets
36
B.2
One-Stage
36
B.2.1
Results on CIFAR-I
37
B.2.2
Results on SVHN
38
B.3
Two-Stage
39
B.3.1
Results on California Housing,
40
B.3.2
Results on SVHN
40
B.3.3
Results CIFARI00
43
17

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1
GENERAL NOTATIONS
Table 1: Summary of main notation used throughout the paper.
Symbol
Description
X
Input space; x E denotes an input/query.
2
Output space: generie outcome variable. In multiclass classification, 2 y= [=......,n)) nt..
Label space in classification, y = [1.....ni]
(X,
Random variables taking values in X x Z. Training examples (æ, =) are drawn i.i.d. from D.
D
Unknown data distribution over X' x 2.
m2, my
Expert j E f1....,JJ J). with prediction map my X y (classification) or X' Z (general task).
9, 9
Base predictor in the two-stage regime, 9 X - Z.
T, 5
policy, Ā X - Z.
A
Generic entity set; A denotes either Ars or A2s depending on the regime.
0
Entity map â : X' + associated with entity E A (label or expert prediction) used in the unified top-k formulation.
(x)
Top-k selection set (Ir S A of size k, containing the & entities with the largest scores under (2.
Index of the i-th smallest expected cost (z);;;;; used describe the Bayes- optimal selection rule.
k
Unified top-k true deferral loss Edet (ll, (y 11 E [x)) (uniformized top-k deferral cost).
t
Top-k deferral surrogate based on cross-entropy family members D used for training a k-independent policy и.
def.k
ki
Cardinality function kg X' - Av used in Top-k(z); parameterized by 0 and learned with surrogate Peard.
leard
Cardinality-aware deferral loss
leard (2), ko(x),
(x))),))))))) + x Ro(a)) (1)).).
d
Metric measuring fask-specifir error between the selected entity set using an aggregation mechanism and the true outcome (used in leard).
4 Peard
Surrogate loss for adaptive cardinality, (1 ka, x, z), built from normalized variant leard-
A.2 NOTATIONS FOR ORDERED SETS
Definition A.1 (Orderings finite set). Let S (l...., N) be of cardinality N |S and
let
R, (m,w) ((,,w,, where M is measurable input space (typically M or M X x )).
f M
Descending permutation. For every fixed m E M,let
(m)) Ω - Ω
be the (tie-broken) permutation that satisfies Am, (m()())))) pm 2 ... 2 (m,))(())()))
18

---

Published as a conference paper at ICLR 2026
The element occupying the i-th largest position is denoted by
M = (m)
i= i
Ascending permutation.
Analogously, define
(m)) :2 -
such that
((())(L))) s (m..)((m)(2) s ... s (m)))(())))))
and set
[ (f).
i= i=
J + 1) and an order indicator E { f), the top-k selection Top-k Selection Set. For E (1,...,, ser
(x) = {[1],, [U) (19j).
Remark 3 (Typical instantiations). In particular:
1. Policy scores. Take Ω = A fl....,J J + 1}. M = X; and [x(1,) = <(x, Descending order (o -1) ranks agents from most least confident at retaining the query.
2. Agent-specific consultation costs. Fix Ω A but enlarge the input space M X Z and define
Se(( =), = (x);
Ascending order (o =1) lists agents from cheapest to most expensive for the specific pair (I,, 2).
A.3 USEFUL DEFINITION
Definition A.2 (H,, consssteenyy). Let Hl be hypothesis set and let (Te)e21 C. Hx. We say that the loss & &eeg is Hx -onssisstt with respect to the loss fde f.k if
Eo (Tt) - cB (Hl)) + Uo (H+)) 1000
Elarth (TT) Elaat (Hxx) ) + Ueeen (Pbx)
where E (H4) == inf Hn Easer (H₂) and Usaes (Hx)) (resp. Uesur (Hx)) denotes the mini- mizability gap associated with L (resp. (def,)).
Definition A.3 (Hl--caibrration. Let Hr a hypothesis set. We say that the loss P Hy- calibrated with respect to the loss fde if, for any > O, there exists 8 > 0 such that for all a E Hr and all x E X.
Cras (T, x) < Cin, (H,, +e
Classa (T, < CB Clay (H,, x) + ō,
where Cour (π, and Chaor (π, are the conditional risks at x, Ciase (Hr, =
infrex, Cour ()), and CB C (Hr, x) infzet, (F (T)
19

---

Published as a conference paper at ICLR 2026
A.4 ALGORITHM
Algorithm Top-k L2D Training Algorithm
Input:
Dataset {(r,, =))==r entities {āg + jeAA policy a E IT, number of epochs
EPOCH, batch size BATCH, learning rate v.
Initialization: Initialize policy parameters
for i 1 EPOCH do
Shuffle dataset {(x;,2)) 2,))/
for each mini-batch B C ((ri of size BATCH do
Extract input-output pairs (I, B.
Query entities {āz & DEA.
Compute the empirical risk minimization:
BATCH (x,))B Update parameters 0: 0 - vV Vo (T;; 0)-
Egg,
(T; (
$ de (T, 2, 2)
ai.k
(Gradient update)
end for end for Return: trained policy x.
Algorithm Cardinality Training Algorithm
Input: Dataset {(x; 2/) trained policy from Algorithm |
entities (āz X + 2hA
cardinality function Kg E Hr. number of epochs EPOCH, batch size BATCH, learning rate v.
Initialization: Initialize cardinality parameters 0.
f to EPOCH do
Shuffle dataset {(x; x,))
for each mini-batch B C {(x;, 2;)) of size BATCH do
Extract input-output pairs (I, y) B.
Query entities {āz X Z)eAA Compute the scores [x{ |A! using the trained policy и.
Sort these scores and select entries to construct the top-k entity set II (x)).
Compute the empirical risk minimization:
Esg, (kip; 0)
BATCH ((a))
Dear (II [I] Ko, x w)
Update parameters 6:
0
vVV vV Eko
(Gradient update)
end for end for Return: trained cardinality model hig.
A.5 ILLUSTRATION OF Top-k(x) AND ToP-k L2D
Learned with Corollary
Single Entity Prediction
Query rE X
Select Entity 1 (xx
n, (x)
Prediction
Figure 2: Inference step of Top- L.2D (Narasimhan et al. 2022 Mao et al. 2023a 2024c Mozannar & Sontag] 2020; Mao al.) 2024a): Given query, process it through the learned policy We select the entity with the highest score (x)) = arg max A क (I, Then, we query this entity and make the final prediction.
20

---

Published as a conference paper at ICLR 2026
Committee of k Ordered Entities
"tut (i))
Policy From Algorithm
(Vote majority, etc.)
Query E X
Scores (a
Top-k Selection (x)
Orurt (a)
Decision rule
"iit (ir)
Prediction
[ (ar)
Laae (x)
Figure 3: Inference Step of Top-k L2D: Given query first process it through the policy learned using Algorithm [ Based on this, we select fixed number k of entities to query, forming the Top-k Selection Set (x). as defined in Definition 4.1 By construction, the expected size satisfies Ex [ decision cule-such majority vote or weighted voting. The final prediction produced by this committee according to the chosen rule.
k. We then aggregate predictions from the selected top-k entities using
Policy From Algorithm
Committee of kx(x) Ordered Entities
Scores ())a)
Ppp (x)
(Vote majority, etc.)
Query
Top-k Selection Ser Ii, (z)
digg (xx)
Decision
x e
i
Cardinality Ra(n) E
(alettt (or)
Prediction
A
Cardinality function From Algorithmp
2
Aocoenn (x)
Pnann (ar)
Figure 4: Inference Step of Top-k(x) L2D: Given query x, we process it through both the policy т, trained using Algorithm ! and the cardinality function ke, trained using Algorithm 2 Based on these two functions, we construct the Top-k: Selection set. By construction, its expected size satisfies Ex [ (X)l] Ex (X)]. We then aggregate predictions from the top-ho(x:) entities using a decision rule (e.g., majority vote, weighted voting). The final prediction is produced by this committee of entities according to the chosen decision rule.
A.6 MODEL CASCADES ARE SPECIAL CASES OF TOP-k AND TOP-k(x) SELECTION
Throughout, let A be the set of entities. For A denote by àg X - Z the prediction of entity j. by π X x A + R policy score, and by
(1) [T (x) .... T (Mt (x)
the Top-k Selection Set containing the indices of the & largest scores.
21

---

Published as a conference paper at ICLR 2026
A.6.1 MODEL CASCADES
Definition A.4 (Evaluation order and thresholds). Fix permutation p ( PlAj)
PlAd) of A (the
evaluation order) and confidence thresholds 0 < P1 < U2 < ... < MAI < 1. For each entity let conf X' x A [0, 1] be a confidence measure.
Definition A.5 (Size-k k cascade allocation). For a fixed &o E fI, |A]] define the cascade set Kx(z)) = {f1..... pk).
The cascade evaluates the entities in the order p until it reaches Pk- If the confidence test conf conf 2)) 2 V4 is satisfied, the cascade allocates the set (x(z); otherwise it proceeds to the next stage (see Section A.6.3 for the adaptive case).
A.6.2 EMBEDDING A FIXED-K CASCADE
Lemma A.6 (Score construction). For a fixed k define
2 ran
j kx(x),
h ++1
T& (1.
rank Avk,) I E Kx(x),
[
where rank (j) E (1,....|B]) is the index j inside the list B ordered according to p. Then for every x E X
[1x(x) = (x)x).
Proof. Separation. Scores assigned to (x) lie in (1, while scores assigned to A K& (z) lie in [-,, [A[+T): ); hence all largest scores belong exactly to KC(xx).
-1,4)
Distinctness. Within each block, consecutive ranks differ by 1/(k+1) or 1/(IA++1), so ties cannot occur. Therefore the permutation returns precisely the indices of KCx (z) in decreasing order, and the Top-k Selection Set equals Kz)z).
M
Corollary A.7 (Cascade embedding for any fixed k), For every k E (l....,|Al] |A|} the policy T of Lemma A.6 satisfies
(x) = (x(z)) VrEN X.
Consequently, the Top-k Selection coincides exactly with the size-k cascade allocation.
Proof. Immediate from Lemma A.6
A.6.3 EMBEDDING ADAPTIVE (EARLY-EXIT) CASCADES
Let the cascade stop after data-dependent number of stages ko(x)) E (1, (1.....,4e)) |A]]. Define the cardinality function hg and reuse the score construction of Lemma with & replaced by hag (z)): Tho(z) (x.))
Lemma A.8 (Cascade embedding for adaptive cardinality). With policy Tkotz) and cardinality func- tion ko(x), the Top-k(x) Selection pipeline allocates Kiocte (x) for every input x. Therefore any adaptive (early-exit) model cascade a special case of Top-k(()) Selection.
Proof. Applying Lemma A.6 with ko(x)) yields IT ka (x) = KRo(a) The cardinality function truncates the full Top-k(a)) set to its first ho (z) elements-preeisele (x). Li
A.6.4 EXPRESSIVENES: MODEL CASCADES vs. TOP-k ToP-k(z) SELECTION
Hierarchy. Every model cascade be realised by a suitable choice of policy scores and, for the adaptive case, a cardinality function (see App. A.6 Hence
Model Cascades Top-k Selection C Top-k(ar) Selection.
& The inclusion is strict, for the reasons detailed below.
prefix fixed
learned k (a))
22

---

Published as a conference paper at ICLR 2026
Why the inclusion is strict.
1. Non-contiguous selection. A Top-k Selection Set I1x(2) may pick any subset of size (e.g. (1,2,5)) whereas a cascade always selects prefix (PI Pk) of the evaluation order.
2. Learned eardinality. In Top-k(x) Selection the cardinality function (x(x) is trained by minimizing surrogate risk; Theorem 4.7 provides consistency and ensures the optimality (2)) Classical cascades, by contrast, rely on fixed confidence thresholds with no statistical guarantee.
3. Cost-aware ordering. Lemma 4.5 shows the Bayes-optimal policy orders entities by ex- pected cost, which may vary with z. Top-k policies can realize such input-dependent or- derings by means of the policy scores π (.. Cascades, in contrast, impose a single, input-independent order p.
4. Multi-entity aggregation. After selecting k entities, Top-k Selection can aggregate their predictions (majority vote, weighted vote, averaging, etc.). A cascade, however, uses only the last entity in the prefix whose confidence test is passed. Earlier entities are effectively discarded. Thus cascades cannot implement multi-entity aggregation rules.
Separating example. Assume |A| = 3 with entities dn, 02, and consider Top-2 Selection policy defined by policy scores т (2x) such that
on some x: TΦ, 1) > (xx T22,) 2)
[2,(2) = {1,3).
some x': T(,,11) π (2x, 2), > (2),3)
II,(²) = {1,2}.
Suppose, for contradiction, that cascade with a fixed, input-independent order p realizes the same selections as Top-2. Because II2(2) = {1.3} is not prefix of any order unless 3 precedes 2, we must have p satisfying
1>p 3 > 2.
But since l2'(x) (1,2) must also be prefix of the p. must have
1>p 2 p
contradiction. Hence no fixed-order cascade can realize this Top-2 Selection Set.
Moreover, even cases where Top-k set is prefix (e.g.. (1,2)), a cascade outputs the prediction of the last confident entity in that prefix, whereas Top-A Selection may aggregate the k entities' predictions (e.g., by weighted vote). Therefore, cascades cannot, in general, implement Top-k aggregation rules.
A.7 PROOF LEMMA 4.2
Definition 4.2 (Top-k True Deferral Loss), Let x X, E Z, and [T())) S be the top-k selection set. Let (1, æ) the cost of selecting entity j for input (I, The uniformized top-k true deferral loss is
LA Σ (,, (z())),
(
1=1
Proof. We will prove this novel true deferral loss for both the one-stage and two-stage regime.
Two-Stage. In the standard L2D setting (see 3.2). the deferral loss utilizes the indicator function
(x) = j} to select the most cost-efficient entity in the system. Specifically, we have π т, [42 = + and (1, 2) (x, z). We can upper-bound the standard two-stage L.2D loss by
23

---

Published as a conference paper at ICLR 2026
employing the indicator function over the Top-k Selection Set (z) defined in 4.1
J+1
())), =
Σ
((()=)
==1
Σ 2,(z,2)1( =
(5)
J+1
$
Σ Ma(0, e mx))))
j=1
det (I).
Consider system with two experts {m, )2 and one main predictor g. Leading to A = (1, 3), and the Top-[A| Selection Set [2] = {3, 13. This indicates that expert m2 has higher confidence score than expert my and predictor g. i.., 3) 2 (x, 2) 2 x (x, 1). We evaluate d for different values of k I |A|:
For k = 1: The Selection Set is II (x) = {3}. which corresponds to the standard L2D setting where deferral is made the most confident entity (Narasimhan al., 2022 Mao al., 2023a Montreuil et al. 2025b). Thus,
2s det (x), = ps(x, =) = (s))()())) + Bs.
(6)
recovering the same result as 2e defined in 3.2
For k = 2: The Selection Set expands to [2)) |3, 2), implying that both expert m2 and expert m are queried. Therefore,
( (2, 2) + #2(2, 2),
correctly reflecting the computation of costs from the queried entities.
For k = 3: The Selection Set further extends to [T(()) = {3, 2, 1}. implying that all entities in the system are queried. Consequently,
der,3 (I)) - p2(a, 2) + u2 2) ui (a,,),
incorporating the costs from all entities in the system.
One-Stage. The standard One-Stage deferral loss introduced by Mozannar & Sootag (2020) assigns cost based whether the model predicts defers:
dl ()(() # v)i()()) S s (o(a,)))i)))=
#=1
This formulation handles two mutually exclusive cases: the model predicts a class label j E (1..... n.) and is penalized if j & y. or defers to expert my and incurs the expert-specific cost (,(,,)) v). However, this formulation relies on a hard-coded distinction between prediction and defer- ral.
To generalize and simplify the analysis, we introduce unified cost-sensitive reformulation over the entire entity set (I,..., n + J}. We define
(e,liivyi B (5 a. (m)-n # v} 33
for j S for j >>
This assigns each entity whether label expert structured cost combining prediction error and fixed usage cost. The total loss is then
D+J ((())))
(h()))))
j=l
24

---

Published as a conference paper at ICLR 2026
We now verify that this general formulation is equivalent to the original loss when the cost parame- ters are selected appropriately.
Consider binary classification example with y {1,2). two experts {m1, m2), and parameters
0j 1, 3 for all j. If the classifier h predicts label h(x) then the cost is p v) 1[1 # y/. which matches the original unit penalty for incorrect prediction. If h(z) = y, the cost becomes (2, y) [y + v) O, correctly yielding no penalty for correct prediction. If instead the classifier h defers to expert my, i.e., M(x)) n + 3, then the loss becomes 4z y) (x) * y). matching the original expert cost.
Therefore, by using x = and | = + n, follows:
J+n
(k(r))))))) = Σ 1y(2, 11(())) = =}
J+n
(9)
Σ (y(x:)))(je] j=1
C de (IIg(2),))
A.8 PROOF LEMMA 4.3
Lemma 4.3 (Upper Bound on the Top-k Deferral Loss). Let x E X, 2 E 2, and let < k: S I.A.. Let $ a convex surrogate the cross-entropy family. Then the top-k deferral loss satisfies
lag (1x()),))
Σ JEA
Σ (1(or,)) bFj
1 (T - (IAI) k:) Σ 1,(,,2) JEA
Proof. Let the entity set A and the policy Hy. For query-label pair (æ, denote the costs of allocating to an entity j by 1z( 2) 2 (j = 1, l..... (l) and the total cost by
|A
(x, = Σ (x,
i=1
Define, for each index j.
(x, Σ Mg( -2)
99
For any h E {1,..., [A]} and any size-k decision set llx (1,..., [Al) the top-k deferral loss
is
M
faet, (I)))),
Σ 2) (x)))
Because (x) and its complement IT, (x) form a disjoint partition of (1,..., [A]).
lset (T)))))))
Σ
Ctot
Σ Uÿ-
(10)
Jelle
jEll.
For every j we have μ. Crot Ej with Eú -iigg Mii whence
Σ
k= = Σ (Crat - -E,) = ((A) k) Σ Ejp
(11)
JeTL
fELT.
25

---

Published as a conference paper at ICLR 2026
with the factor |A| k being the cardinality of I[g. Substituting equation
11 into equation 10 yields
f (2)) =) = Cror
(1 k)Cuot= - ΣS5 $
(12)
JETT
A 11x1 (1A1) - k 1) Σ (z(
Σ 5=1
E
(13)
Let us inspect limit cases:
k 1. Then Thg has |A| - indices and the constant term reduces to -(IA) 1oot
expanding the sum shows f (x(x) as expected for the classical true deferral loss defined 33.3J and 3.2
2. h = [Al. The complement is empty, Σ ال & = 0 and |A| | - 1, so the formula gives (act,/AA Crors i.e. paying all deferral costs again matching intuition.
4% (T, x.j) for every As shown by [Lapin al. (2016;; Yang & Koyejo 2220)) Cortes et al. (2024)
Finally, Let ( (T, j) be a multiclass surrogate that satisfies j & (x)) <
the cross-
entropy family satisfy this condition. Because each weight E (2, 2) 2 0, we have
LA
A
laet, (I2,, 2) S
(a (1 (1AA) k- 1)
Σ
M5( z)
j=1
5=1
(14)
AA
1A41
Σ (1(2zz) (0I(T, - (IAA) k Σ
My (3(30,2)
We have shown the desired relationship.
L
A.9 PROOF LEMMA 4.5
Lemma 4.5 (Bayes- Soayeslpttimal Top-k Selection). Let x E X. For each entity j E A define the expected cost (1) E (x, 7)}, its Bayes-optimal expected cost as p infgen, (x). Then the Bayes-optimal top-k selection set is
m (z) = arg min
Σ (x) I.CA mL
[11) 12] n..... [k] 1
JEIL,
where Li denotes the index of the i-th smallest expected cost in { (z) j E A}. In the one-stage
regime, where no base predictor class Hg is defined, we simply set T7 (x) = (z)
Proof. Let's consider the Top-k Deferral Loss defined by
LA
lac (T) Em(os))EETT(())))
where (x) 2) (,) BJ in the two-stage and p (2, =) a, (á) + y) B3 in one-stage setting, is the cost associated with entity j E A. We define the expected cost as:
[x(x) Ez1x= (u,( Z)
Given the policy x : X' + A, we have the Top-k Selection Set (x) C.A.
One-stage. Here (1, a, (z)((;( * 4) + B3 and àj fixed (non-optimizable) as they labels or experts. Thus
[z(2)) = a,P(Y + (z(x)) X x=
={aypt) + (z) X = xx) +
[ajP(Y +jlX + (applY
if S n if > n
26

---

Published as a conference paper at ICLR 2026
independent of (or h here). We introduce the conditional risk (Steinwart 2007 Bartlett al. 2006) of the Top-k Deferral Loss:
Cen, (TT, Erix = (
[AI Σ (o)(t)ttie {lx(a))
Hence the Bayes (conditional) risk over policies reduces to choosing a size-k subset minimizing the sum of these expected costs:
L (++, = inf TEM,
B
Ceaer (TT,
1A|
(15)
inf E (x))
mEH,
Let [i]] denote the index of the i-th smallest expected cost, so that
Fng (z. v) S P1ar S ... S Fin+ mn (x. v).
Then the Bpyes-optimal risk is obtained by selecting the entities with the lowest expected costs:
Caa (Hr,2) 2) Ta (2,1).
i=1
Consequently, the Top-k Selection Set IT (a:) that achieves this minimum is
(x) arg min
Σ (=)x))
{ s......... [
(16)
I [x jell. (w)
meaning IT (a)) selects the & entities with the lowest optimal expected costs.
Two-Stage. Here (z(x, =) a,)( Bj and à are fixed but we have the full control of the predictor g E Hg. Thus
(x(x)) a, Ez (( 2)] Bj
a [o, 2)] 3 ag (
;=
Z)] + 83
if
independent of (or here) but not g E H. for ui. We introduce the conditional risk (Steinwart) 2007 Bartlett et al. 2006) of the Top-k Deferral Loss:
Clsta x) E2|x= [Caet (Il(())))
IA
= Σ
==1
Hence the Bayes (conditional) risk over policies reduces to choosing a size-k subset minimizing the sum of these expected costs:
Caatt (HLr Hgg inf inf Crass (T,, 2. xEH, gEH,
IA| (n)
(17)
inf SEH, jml
with (x) infg Py (x) and for j > 1. (x) (1))) Let [i]] denote the index of the i-th smallest expected cost, that
B [1
(x, 2) S B [ (I, x S T [J+1] (x,
27

---

Published as a conference paper at ICLR 2026
Then the Bayes-optimal risk is obtained by selecting the entities with the lowest expected costs:
C (H H Hg. 2)
Σ M
(z,
Consequently, the Top-k Selection Set 1 (x) that achieves this minimum is
(x) = arg min
Σ (x(() = [1I 16
(18)
TI IL( [=) (Ell(
meaning 11 (I) selects the k entities with the lowest optimal expected costs.
A.10 PROOF COROLLARY 4.6
Corollary 4.6 (Special cases for k 1). The Bayes rule in Lemma 4.5 recovers prior Top-1 results:
1. One-stage L2D. For any entity j (labels j I n and experts j > n))
n = a,)((
z) + By,
which yields the Top-1 Bayes policy of Mozannar Sontag (2020).
2. Two-stage L2D. Let j = denote the base predictor and j 2 the experts. Then
p (x) a1 inf Ez - ()((((())))) Z)] + 81, gEH,
and for j 2 2, (z) =., (1) 2)] +,3 recovering the Top-l allocation in Narasimhan et (2022) Mao al. (2023a); Montreuil et al. (2025b).
3. Selective prediction (reject option). We take the set of label entities and augment it with an abstain entiry 1, defined by To and B1 > 0, while label entities use aj
1, 1, 0. Then
( fl......H)..
(f(x)) X
yielding the Chow's rule CCoo 1970
Proof of Corollary 4.6 Set in Lemma 4.5
Then the Bayes rule selects the single index
n (x)
arg min n.( (z))
jEA
i.c., the entity with the smallest Bayes-optimized conditional expected cost at r. We verify the three specializations.
(1) One-stage L2D. In one-stage, the entities (labels or fixed experts) do not depend on any and
=(l,y))=,,1ā0..
+
(z) =1,x)), (ā,((( Y|X 2) + By.
Thus, (x) = {arg min, (z)} selects the entity with the lowest expected cost, which in the one-stage corresponds to choosing the label expert with the lowest misclassification prob- ability. This recovers exactly the Bayes-optimal Top-1 policy established prior one-stage L,D work (Mozannar & Sontag 2020 Mao al. 2024a).
(2) Two-stage L2D. Let j = 1 denote the fixed base predictor entity (x) ((z), and 2 2 denote (fixed) experts ( Then
7 (x) = inf
Ezix=z [a1v(((x),2) + +3]] a1 inf Ez|x= (V((=)) 2)] + Bi. geH,
gEH,
28

---

Published as a conference paper at ICLR 2026
while for j g g-dependence)
x (a) - (,)())) (mm))11))))))))))))0))))))))11)) 2)] 5,-
Hence 11 {arg Ji (z)( (x)) selects, among the base predictor and the experts, the single entity with the smallest Bayes-optimizes expected cost, which recovers the standard Top-1 allocation in two-stage L2D (e.g.. Narasimhan al. Mao al.) Montreuil et al.).
(3) Selective prediction (reject option). Let the action set consist of the n label entities and a reject action
Set jjj
1, 3j 0 for labels E nl, and a 0, 31 X 0. Write
(z(2) = P(Y X = x). Then
Fry PP X x) = 1p)(2)),
pf(x)= X.
Therefore
min min (1 ()()), X min max pj Pz(2), 27. SS
Equivalently, predict the most probable class j'(x) arg P)(x) if Py (x)
(i.e., Geifman &
Py (x) 1 A), and abstain otherwise. This is precisely Chow's rule (Chow 1970
DI- Yaniv 2017: Cortes et al. 2016).
A.11 PROOF THEOREM
First, we prove an intermediate Lemma.
Lemma A.9 (Consistency of Top-k Loss). sample A surrogate loss function & is said to be Ho-consistent with respect to the top-k loss (T(()())). 1{j E [x{x)} if. for any x Hm. there exists non-decreasing. and non-negative, concave function r, : SR - RRt such that:
Σ E [x)))) JEA
inf E < A
Σ JEA
inf Σ nEH. jEA
Eo to (T,
TEH, JEA
where p E AlAl denotes a probability distribution over the set and k $ |A]
Proof. Let the top-k loss be and define its conditional risk as
(<(TT))))))))) = 1 (x))
Cia (T,, x) = ExIx [I (x)x) lx(2)),
JEA
where py(x) P(Z x
The excess conditional risk
ACC, (T,, x) == Ce2 inf Ci. (TT
mEH
Assume that the following pointwise calibration inequality holds for increasing concave function Awasthi et al. 2022):
Σ & [x) inf EP,1 [x)())) S - TEH, JEA
p, Pf Σ YEA
inf Σ Py &u P0 (T, mEH,
JEA
jEA
(19)
we identify the corresponding conditional risks for a distribution p A |Al as done by Awasthi al. 2022):
Ce. (π,x) - inf Ce. (T) & kr Cag, (TT, - inf Cog, (TT (r. xEH,
(20)
mEH,
29

---

Published as a conference paper at ICLR 2026
Using the definition from as:
Awasthi al. (2022), express the expected conditional risk difference
Cae (T,, (T,, xx)
Ex (x (T, X)] = Ex
Ci. (T, inf TEH.
(21)
Ese (T) -Eey (HHx) - Ur (Hr).
Consequently, we obtain:
inf Cay, (TT, (T, Applying the expectation and by the Jensen's inequality yields:
Cra (T,, inf Ca (z, S kr Cas, (T, weg nEHr
(22)
Ex (T, X)] & Ex [Kr (ACug, (TT, x]]] Ex|ACr, (T, X)] I -r (Ex [ACig (T)
(23)
Then,
El2 (T)
EB (H+) Mr (H,)) kr (Eag. (T) Egg, (H₂) Ueg (Hm))
(24)
This result implies that the surrogate loss 1 is Hy-consistent with respect to the top-k loss lk- From Cortes al. (2024); Mao et al. (2023b). we have for the cross-entropy surrogates,
v²)
1+s log[1 + u) - log(1 vl)
u ==1
(u(2)=
(25)
(+ 1- -1
E (0, 1)
v(+
n+J
== 2.
Theorem 4.7 (Unified Consistency for Top-k Deferral), Let A denote the set of entities. Assume that Hy is symmetric, complete, and regular for top-k deferral, and that in the two-stage case, Hy is the base predictor class. Let S = - -I Eyed (x)], Suppose D is Honcssiitt for top-k classification with non-negative, non-decreasing. concave function T-1
One-stage. Let Ex (x)] (x)( xV) + By. For any h E Hh.
c Etad (h) Eand (Hh) + Ue (Hh) < 1ST Exgna (h) Eg der.
(H+r + +lay (Hrr
Two-stage. Let Ex (x, = 2x, (X) 2)] + 33. For any (r.g) E Hr x Hg. Eesu (r, g) E (H,, + (Hr, Hy) s Ex (X)
(X),
inf 1n (X)] gEHt
Expr (r) Exp der.k
(H,r) + Uspar (Hrr
+ +SST
S
with [u(w) 1t log(1 + v)+ 1-v [2(0) r/|A| (MAE). 2 log(1 - v) (logistic), To(v) = v1 22 (exponential), and
One-stage.
Proof. We begin by recalling the definition of the conditional deferral risk and its Bayes-optimal counterpart:
Cisan (T,
= (x)),
(
ch (H,, = Σ Pot (z):
(26)
30

---

Published as a conference paper at ICLR 2026
where (x) EyIx-=[ps Y) denotes the expected cost of selecting entity j. The calibration gap at imput x is defined as the difference between the incurred and optimal conditional risks:
AC (x, a)) CRas. (TT, x) Cast (H x).
(27)
To connect this quantity to surrogate risk, we use the reformulation used the Proof of Lemma 4.3 in Equation T2
n+J Σ j=1
n+J
Clas.e (T,, x) =
Σ inj
(1)
1(1 & [x(x)) - (n + 1)
Σ
py(2),
(28)
j=1
the second term is independent of the hypothesis Hg. This yields: To prepare for applying H. consistetyy result, we define normalized weights:
Enej (x)
Pj
2j=1 Ft
Σ T (()
which form probability distribution over entities j E A. Then:
ACCas (TT,
Σ (a)
Σ p,l [1)x)) inf mEH.
E, [x(())
Now, apply the Ho-cossiity guarantee of the surrogate loss $ for top-k classification (Lemma A.9), which provides:
J+J
n+
Σ p, & [x)x)) - inf xEH,
Σ E
++ inf =EH,
kr
Σ
2 T z.
Σ j=1
p, 20 (T, x.
Multiplying both sides by Eø=1 EEits (o())) we obtain:
ACisr 2
vhtt Ei=l (Eiss (4((1) 0 (TT, x) - inf EØel (E Exth (E o6 08 (T, (0)
x.
En( kT
E5 (Eiti 1,(xx)
Define the calibration gap of the surrogate as:
AC (T,
Th
& (T,
inf mE
Σ (x) t 2,))
Then,
R+ J
ACon m1, (TT, x))
ACiua <
Σ (z)
kr
n+ E)
Enjg (1(
/=1
itf
Taking expectations:
Eem (T) - EB (Hz)) - Uena (Hr)) <
n4J
Exp (T)) - (Hlx) Up (Hla)
k
Σ (x)7
In+ (Eite Ex [x(xx((x))
(29)
31

---

Published as a conference paper at ICLR 2026
Note that we have Σn+ (Eitz Ex (x(x((X))) = (IA) - -)) with Ex Ex (x, =
ayP(a,(X) Y) + Bj. leading to S = (1A1 - EjeA (x) + Y) +3):
Elea (T) - EB Leen (H+x) - -Mem (Hp)) 5
(x)
(Tt)
Egg
(Hx) - Uo (Hrr)
(30)
ST
jr.
S
Two-Stage.
Proof.
We begin by recalling the definition of the conditional deferral risk and its Bayes-optimal counterpart:
J+1 (F Ea,eo)ltie (x)),
Cac, (Hz Hg.
Σ
(x),
(31)
where (z) = Ezix=zluy(e 2) denotes the expected cost of selecting entity j. Note that the conditional risk is different because of the main predictor g. The calibration gap input is defined as the difference between the incurred and optimal conditional risks:
ACtand (T, g,
Cleak (TT, g, x) C (Hor Hgy x)
3+1
Fg(x) (l())) Σ
P
(x) 2B
(32)
mz(x)1 e (l(()))) - Σ Plo (x)
k
(x) Σ
p Wi (z)
Pot
Observing that:
Mog (x)
B 1
(x)
inf 14 (x)
(33)
gEH,
Since the only contribution of g appears through the cost term J41 we can rewrite the first term in terms of conditional risks. Importantly, the minimization is carried only over the decision rule
E Hz.
J+1
Ce g. inf Clas g, Σ (x) [x(x)) Σ (gg(
(34)
TEH,
Using the explicit formulation of the top-k deferral loss in terms of the indicator function i [x)) (Equation 12). we obtain:
n+J
Cisca (IT, 2) inf Cracs (T,,
Σ (1(x))
1 $ [l(x)))
TEH.
j=1
(35)
n+J
inf
Σ
Σ
p.(x)) 19j (x())).
meHe
j=1
Using the same change variable:
Σ Σits 1x(z)
Pj
Ex())((()))
32

---

Published as a conference paper at ICLR 2026
which form probability distribution over entities j E A. Then:
J+1
n+J
Ciasn (T,, 9, x) - inf
Crean (T, 2) =
Σ
(x)
Σ p & (l(()))
TEH
j=1
n+J
inf TE
Σ
p (1x))))))
j=1
Since the surrogate losses 0 are consistent with the top-k loss, we apply Lemma A.9
J+1
x+J
Ca (H, x)- inf TEHr
Cian. (,
S
Σ
ET(())AAT
Σ
p; & [x)
j=1
ifj
n+J
inf +
Σ
pflli $ mx(a))
j=1
Connr (T, .infxEH, Craan (T,, 9, x))
(z(()) kro
ΣØ=1 (Eist My (2)2
j=1
ifs
Earlier, we have stated:
ACrund(TT (T, 2) = Crann (TT, (T,
Clank (Hr, Hg)
J+1
Σ (5(x)1{i ]1{())3 - Σ Mus (2x)
(36)
k
+
(x)
Mi ee (x))
which is
ACten (T, 2. = Crean (T,9,22)
inf EEH
Cesan (T)
k
Plo (z)
Σ i=1
"it (at)
(37)
1=1
< Claur (T))
inf TEZ=
Clas 9T) + (7,(z)- - (x))
Then,
J+1
Co (#,
-inf AnTeE Con., (1T, x)
<
Σ itj
The (a)) ((a))))
Ej=1 (Eitz
Ø=1
(38)
(M)(T) - 11 (xx)
Taking expectations,
Eesur (T,9)= -) B Enn (Hr, Hgg - Ue (Hbr Hyg S (xx - - (X)
J+1
Exp x (TT) Ejgn. (7L=) - -Uay (Hx)
(39)
+
Ex (x(1 kT
Σ Eiig Ex(x,(X)))
Similarly,
we have Ex(r,(X)]
a,Ex, 2)] + By.
Using
Ej Ext; = (IA) - -1)E, Ex [7, (X),. leading to (IA)
1) Ejes (x 83):
33

---

Published as a conference paper at ICLR 2026
A.12 BEHAVIOR OF THE CARDINALITY-AWARE DEFERRAL LOss
For any z E A' and & E {i,...,|A|] |A]], the conditional risk of selecting the top-k experts is
Cleand (R) = E caard [Ig()), k, Z) X 2²]
k
[Id( 2) X x + x
Σ
B
The Bayes-optimal cardinality function is therefore
E[d(Ia(t)) |X xx= tYg Σ Spolun }}
kg (z) arg kE(1.....14 113
min
For 2, define
8C (k) = Clound (k) - Cleurs (k 1),
S = Σ m
simple computation gives
8Ctaara (k:) = B[d((Ta()), - -d(II (20), 1 =] +
(k)
Thus, for any & (I,..., 1}.
Cear (k: 1) S Claand (l))
5D + -[g(S=-1) - E(SL) S
(40)
that is, moving from to & + 1 strictly decreases the conditional risk and only if the marginal reduction in expected error, - (K: + 1), is at least large the marginal regularization cost, x(((S++1) - {(S))]
This shows that consulting and aggregating multiple experts is not hoc: for any fixed aggregation rule d, it is Bayes-optimal to choose k. (x) > 1 precisely when using additional experts yields a net decrease of the conditional risk Cra.. (k), i.c., when improving the final prediction quality worthy the price of consulting selected experts.
Tuning Parameters. As shown above, the parameter À controls the trade-off between consulta- tion cost and predictive reliability. Increasing makes the model more cost-sensitive, leading it to select fewer experts. Conversely, decreasing À places greater emphasis on reliability, resulting in the selection of a larger set experts when beneficial.
A.13 CHOICE OF THE METRIC d
The metric d in the cardinality- -based deferral loss governs how disagreement between the final prediction and labels is penalized, and its choice depends on application-specifc priorities. For instance, it determines how predictions from multiple entities in the k Selection Set I+((x) S A are aggregated into final decision. In all cases, ties are broken by selecting the entity with the smallest index.
Classification Metrics for Cardinality Loss. In classification, common choices include:
Top-k True Loss A binary penalty is incurred when the true label y is not present in the prediction set:
(112 = 1 & {dn ( (x)}).
Example: let 12(x) = 1} the metric will compute d (I), k.y) = 1sy $ {ar (x)))))))))
34

---

Published as a conference paper at ICLR 2026
Weighted Voting Loss. Each entity is weighted according reliability score, typically derived from a softmax over the scores ((x)) The predicted label is obtained via weighted voting:
(2, wz){āy(z) v), with wy f(x, E,, exp( at (2,
1 arg Σ
VEY
(ella(z)
The loss is defined d (z) k. 1 9)
Majority Voting Loss. All entities contribute equally, and the predicted label is chosen by
majority vote:
arg
Σ
1fā -
NEV
jelle(x)
with the corresponding loss dmmaj (II (x), k, 1 + g).
Regression Metrics for Cardinality Loss. Let feg (2, R denote base regression loss (e.g., squared error or smooth L1). Common choices include:
Minimum Cost (Best Expert) Loss. The error measured using the prediction from the best-performing entity in the Top-k Selection Set:
dmin (Ilx (x),
min (x)) jElly(a)
Weighted Average Prediction Loss. Each entity is assigned reliability weight based on softmax over scores (x.). The predicted output is a weighted average entity predictions:
exp(I((r,))
Σ wrāz(x), with wy exp((T(,,)))) (EIl,())
and the loss is computed as (wa E (re
Uniform Average Prediction Loss. Each entity in the Top-k Selection Set contributes equally, and the final prediction simple average:
i = k
Σ (z(x), (Ig((I)(z) (Ell(())
k,), = (2,
A.14 USE OF LARGE LANGUAGE MODELS
Large language models were employed exclusively as writing aids for this manuscript. In particular, used them to refine the text with respect to vocabulary choice, orthography, and grammar. All conceptual contributions, technical results, proofs, and experiments are original to the authors. The LLMMs were used to generate research ideas, mathematical derivations, experimental analyses.
B EXPERIMENTS
B. RESOURCES
All experiments were conducted internal cluster using NVIDIA A100 GPU with 40 GB of VRAM.
B. METRICS
For classification tasks, we report accuracy under three evaluation rules. The Top-k Accuracy is defined ACCop-- Ex[1 (X)], where the prediction is deemed correct if the true label y included in the outputs of the queried entities. The Weighted Voting Accuracy given by Acc Ex 11 (X)), where entity predictions are aggregated via wofthax-weighted voting.
35

---

Published as a conference paper at ICLR 2026
Finally, the Majority Voting Accuracy is defined as AcCmaj = Ex [1 dnny(X)),, where all entities in the Top-k Selection Set contribute equally.
For regression tasks, report RMSE under three aggregation strategies. The Minimum Cost RMSE is defined as RMSEmin = Ex [dmin [. corresponding to the prediction from the best-performing entity. The Weighted Average Prediction RMSE is given by RMSE Ex (dw (X))) using softmax-weighted average of predictions. The Uniform Average Prediction RMSE is computed as RMSE Ex [davg( (X)], using the unweighted mean of entity predictions.
In addition to performance, we also report two resource-sensitive metrics. The expected budget is defined as (k)) Ex Σ5 p(]t where 83 denotes the consultation cost of entity j. and [i]+ is the index of the j-th ranked entity by the policy The expected number of queried entities is given by k = Ex [x where k is fixed for Top-k L2D and varies with in the adaptive Top-k(<)) L2D Settings. Additional details are provided in Appendix
13. AA.13
B.1.2 TRAINING
We assign fixed consultation costs By to each entity. In the stage regime, class labels (j S n) incur no consultation cost (3, = 0), since predictions from the model itself are free. In the two-stage regime, we similarly set 31
the cost schedule 33 (0.05, 0.045, 0.040, 0.035, 0.03); J. with my assigned as the most expensive. This decreasing pattern reflects realistic setups where experts differ in reliability and cost. As the surrogate loss, adopt
for the base predictor g. For experts,
the multiclass log-softmax surrogate t "
(q, - log Eyca e used both for learning
ca(o.5)
the policy x E Hg and for optimizing the adaptive cardinality function kg. The adaptive function kg is trained under three evaluation protocols-Top-i Accuracy Majority Voting, and Weighted Voting (see A.13 and [ To balance accuracy and consultation cost, we perform a grid search over the regularization parameter À E (10-9 0.01, 0.05, 0.25, 0.5, 1, 1..,.... .65)), which directly shapes the learned values of kxx). Larger penalizes expensive deferral sets, encouraging smaller k. When multiple values of & achieve the same loss, ties are broken by selecting the smallest index according to a fixed ordering of entities in A.
B.1.3 DATASETS
CIFAR-10. A standard image classification benchmark with 60,000 color images of resolution 32 x 32, evenly distributed across 10 object categories (Krizhessky) 2009). Each class has 6,000 examples, with 50,000 for training and 10,000 for testing. We follow the standard split and apply dataset-specific normalization.
CIFAR-100. Identical setup to CIFAR-10 but with 100 categories, each containing 600 images.
SVHN.
The Street View House Numbers (SVHN) dataset (Goodfellow et al. 2013) is a large-scale digit classification benchmark comprising over 600,000 RGB images of size 32 x 32, extracted from real-world street scenes. We use the standard split of 73,257 training images and 26,032 test images. The dataset is released under non-commercial use license.
California Housing. The California Housing dataset (Kelley Pace & Barry 1997) is regression benchmark based on the 1990 U.S. Census (CCO). It contains 20,640 instances, each representing a geographical block in California and described by eight real-valued features (e.g.. median income, average occupancy).. The target is the median house value in each block, measured in hundreds of thousands of dollars. We standardize all features and use 80/20 train-test split.
B.2 ONE-STAGE
We compare our proposed Top-k and Top-k(æ) L2D approaches against prior work Mozannar & Sontag 2020 Mao et al., 2024a), well against random and oracle (optimal) baselines.
36

---

Published as a conference paper at ICLR 2026
B.2.1
RESULTS ON CIFAR- 10
Settings. We synthetically construct a pool of 6 experts with overlapping areas of competence. Each expert is assigned to subset of 5 target classes, where they achieve high probability of correct prediction (p 0.94). For all other (non-assigned) classes, their predictions are uniformly random [Mozannar & Sontag 20200 Verma al. 2022). This design reflects realistic setting where experts specialize in overlapping but not disjoint regions of the input space. Table 2 reports the classification accuracy of each expert the CIFAR-10 validation set.
Table 2:
Validation accuracy each expert on CIFAR-10. Each expert specializes in of 10 classes with high confidence.
Expert
2
3
4
6
Accuracy (%)
52.08
52.68
52.11
52.03
52.16
52.41
Top-k One-Stage. We train the classifier h HA using a ResNet-4 architecture (He al. 2016). following the procedure described Algorithm [ (π = h). Optimization is performed using the Adam optimizer with batch size of 2048, initial learning rate of 1 x 10-3, and 200 training epochs. The final policy h is selected based on the lowest Top-k deferral surrogate loss (Corol- We report results across various fixed values of E A corresponding to the number of queried entities at inference.
lary 4.4) on held-out validation
Top-k(()) One-Stage. Given the trained classifier h. we train cardinality function ho E H4 as de- scribed Algorithm|A.4] This function is implemented using a CLIP-based image encoder (Radford et al. 202 followed by classification head. We train & using the AdamW optimizer (Loshchilov & Hutter 2017) with a batch size of 128, learning rate x 10 10-3, weight decay of 10 10-0 and cosine learning rate scheduling over 10 epochs. To evaluate the learned deferral strategy, exper- iment with different decision rules based on various metrics d; detailed definitions and evaluation setups provided in Appendix A.13
37

---

Published as a conference paper at ICLR 2026
100
Expected Number Quesied Entities X
E
90
100
12D
(16)
Aroay (06)
96
60
50
92
00
30
Top-é L20
Top-A(s) L2D
20
Randimm Top L2D
Top-1
Top & L2D
0.00
0.00
0.10
0.15
0.20
0.25
0.06
0.10
0.15
0.20
0.25
Budget
Budget 7
(a) Comparison with Random baseline using AcC
(b) Performance under AcCop-k metric.
Expected Number Queried Entitios X
Expected Number of Quested Entities
E-d
I
E-12
F-:d
E- T-18
Togstfs) 1.00
Topek(e) 120
Top-k
Top-l
94
(00
Anto (20)
02
82
1pp-1
36p- L2U
88
0.025
0.000
0.075
0.100
0.125
0.150
0.175 0.200
0.225
0.025
0.050
0.075
0.100
0.125
0.150
0.176
0.200 0.226
Budget 7
Buiget
(c) Performance under AcCmaj metric.
(d) Performance under Acc metric.
Figure 5: Comparison of Top-k and Top-k(x) One-Stage across four accuracy metries CIFAR- 10. Top-k(z) achieves better budget-accuracy trade-offs across all settings. For clarity, only the first 12 entities shown. Results averaged over 4 independent runs. The Top-1 L.2 corresponds
to Mozannar & Sontag (2000
Mao et al. (2024a).
Performance Comparison. Figure summarizes our results for both Top-k and adaptive Top- k(x) One-Stage surrogates on CIFAR-TO. In Figure report the Top-k Accuracy function of the average consultation budget B. As expected, the Top-1 I.2D method (Mozannar & Sontag 2020) is recovered as a special case of our Top-k framework, and is strictly outperformed as k increases. More importantly, the adaptive Top-k(x) consistently dominates fixed-k strategies for given budget level across all metrics. Notably. Top-k(z) achieves its highest Majority Voting Accuracy of 95.53% at budget of 3 = 0.192, outperforming the best Top-k result of 94.7%. which requires higher budget of 3 = 0.199 (Figure 5e A similar gain observed under the Weighted
Voting metric: Top-k(r) again reaches 95.53% at 3 = 0.191, benefiting from its ability to leverage
classifier scores for soft aggregation (Figure 5d).
This performance gain arises from the ability of the learned cardinality function k(z) to select the most cost-effective subset of entities. For simple inputs, Top-k(w) conservatively queries small number of entities; for complex or ambiguous instances, it expands the deferral set to improve reliability. Additionally, we observe that increasing & indiseriminately may inflate the consultation cost and introduce potential bias in aggregation- of unreliable entities in majority voting). The Top-k(x:) mechanism mitigates this by adjusting dynamically, thereby avoiding the inefficiencies and inaccuracies that arise from over-querying.
predictions (c.g., through overdominance
B.2.2 RESULTS ON SVHN
Settings. We construct a pool of six experts, each based on ResNet-18 architecture (He et al. 2016), trained and evaluated on different subsets of the dataset. These subsets synthetically generated by selecting three classes per expert, with one class overlapping between consecutive experts to ensure partial redundancy. Each expert is trained for 20 epochs using the Adam opti-
38

---

Published as a conference paper at ICLR 2026
mizer (Kingma & Ba 2014) with learning rate of 1 x 10-3, Model selection is based the lowest validation loss computed on each expert's 's respective subset. Table reports the classification accuracy of each trained expert, evaluated on the full SVHN validation set.
Table 3: Accuracy of each expert the SVHN validation set.
Expert
2
3
5
6
Accuracy (%) 45.16
35.56
28.64
25.68
23.64
18.08
Top-k and Top-k(æ) One-Stage. We adopt the same training configuration as in the CIFAR-10 experiments, including architecture, optimization settings, and evaluation protocol.
Performance Comparison. 6 results consistent with those observed on CIFAR-10. Our Top-k One-Stage framework successfully generalizes the standard Top-1 method (Mozannar & Sontag 2020). Moreover, the adaptive Top-k(æ) variant consistently outperforms the fixed-k ap- proach across all three evaluation metrics, further confirming its effectiveness in balancing accuracy and consultation cost.
Expeecedd Number of Querted Entities F
100
Dap-1
(86
60
Artoy
40
94
Top-k Ramdom Top-é L2D
Top-k(x) L2D
20
92
10p-1 1.31
Top-k L2D
0.00
0.15
0.20
0.25
0.10
0.15 Budget 2
0.25
Budget
(a) Comparison with Random baseline using Acc
(b) Performance under AcCop-k metric.
Expected Number Quecied Entities F
Expected Number of Quecied Entities
92.0
Anaa (30)
01.5
91.5
91.0
91.0
90.5
00.5
Top-A(x)
Top-Mor)
90.0
Thy-k 12D
Top-A L20
Buiget
Budget
(c) Performance under AcCmaj metric.
(d) Performance under Accw-st metric.
Figure 6: Comparison of Top-k and Top=k(x) One-Stage across four accuracy metrics on SVHN. Top-k(<)) achieves better budget-accuracy trade-off's across all settings. For clarity, only the first entities are shown. Results are averaged over independent runs. The Top-1 L2D corresponds to
Mozannar & Sontag (2020); Mao al. (2024a).
B.3 TWO-STAGE
We compare proposed Top-k and Top-k(æ) L2D approaches against prior work (Narasimhan cet al., 2022] Mao et al., 2023a 2024c Montreuil et al. 20256), as well as against random and oracle (optimal) baselines.
39

---

Published as a conference paper at ICLR 2026
B.3.1 RESULTS ON CALIFORNIA HOUSING.
Settings. We construct a pool of regression entities (five experts and one main predictor), each trained on predefined, spatially localized subset of the California Housing dataset. To simulate domain specialization, each entity is associated with a specific geographical region of California, reflecting scenarios in which real estate professionals possess localized expertise. The training gions are partially overlapping to introduce heterogeneity and ensure that no single entity has access to all regions, thereby creating realistic setting for deferral and expert allocation.
We train each entity using multilayer perceptron (MLP) for 30 epochs with a batch size 256, learning rate of 1 x 10 10-3, 3 optimized using Adam. Model selection is based the checkpoint achieving the lowest RMSE on the entity's corresponding validation subset. We report the RMSE on the entire California validation set in Table
Table 4: RMSE X 100 of each entity on the California validation set.
Entity
1
2
3
4
5
6
RMSE x x100
21.97
15.72
31.81
16.20
27.06
40.26
Top-k L2D. We train a two-layer MLP following Algorithm The rejector is trained for 100 epochs with batch size of 256, a learning rate of x 10-4, using the Adam optimizer and a cosine learning rate scheduler. We select the checkpoint that achieves the lowest Top-k surrogate loss the validation set, yielding the final rejector r. We report Top-k L2D performance for each fixed value k E A.
(op-k(x) L2D. We train the cardinality function using the same two-layer MLP architecture, fol- lowing A Algorithm?2 2 The cardinality function is also trained for 100 epochs with a batch size of 256, learning rate of 5 10-1, using Adam and cosine scheduling. We conduct additional experiments using various instantiations of the metric d, as detailed in Section B.I.1
Performance Comparison. Figures 7
compare Top-k and Top-k(x) L.2D across multiple evalua- tion metries and budget regimes. Top-k L2D consistently outperforms random baselines and closely approaches the oracle (optimal) strategy under the RMSEmin metric, validating the benefit of using different entities (Table
In Figure 7b Top-k(æ) achieves near-optimal performance (6.23) with a budget of 3 = 0.156 and
an expected number of entities = 4.77, whereas Top-k requires the full budget 3 = 0.2 and A = 6 entities to reach a comparable score (6.21). This demonstrates the ability Top-k(x) to allocate resources more efficiently by querying only the necessary number of entities, in contrast to Top-k, which tends over-allocate costly or redundant experts. Additionally, our approach outperforms the Top-1 1 L2D baseline (Mao al. 2024c), confirming the limitations of single-entity deferral.
and R RMSF wwavg where performance is not necessarily monotonic in the number of queried entities. In these settings, consulting too many or overly expensive entities may degrade overall performance. Top-k(x) consistently outperforms Top-k by carefully adjusting the number of consulted entities. In both cases, it achieves optimal performance with budget of only 3 0.095, level that Top-k k fails to reach. For example, in Figure 7c Top-k(<)) achieves RMSE = 8.53, compared to 10.08 for Top-k. Similar trends are observed under the weighted average metric (Figure 7db, where Top-k(xr) again outperforms Top-k; suggesting that incorporating rejector-derived weights wy leads to more effective aggregation. This demonstrates that our Top-k(æ) L2D selectively chooses the appropriate entities- when necessary to enhance the overall system performance.
Figures 7c and 7 evaluate Top-k and Top-k(<r) L.2D under more restrictive
B.3.2 RESULTS ON SVHN
Settings. We construct a pool of convolutional neural networks (CNNs), each trained domly sampled, partially overlapping subset of the SVHN dataset (20%). This setup simulates realistic settings where entities are trained distinct data partitions due to privacy constraints or
40

---

Published as a conference paper at ICLR 2026
Expected Number Quezied Agents
Expecsed Number Queried Agents
E
120
+ Top L2D
120
Top
Top L2D
Top L2D
RMSEs X
RMSE x
Optinal Numbit Agaus
Optimal Namber Aguats
6
0.025
0.050
0.075
0,100
0.125
0.150
0.175
0.200
0.025
0,050
0.100
0.125
0.150
0.175
0.200
Budget
Badges
(a) Comparison with Random and Optimal Baselines: compare Top-k L.D against random and ora- cle baselines using the RMSEmin metric defined in Section Our method consistently outperforms both baselines and extends the Top-1 I L2D formula- tion introduced by Mao al. (002cc)
(b) Comparison of Top-k L2D and Top-k(2;) L2D der RMSE metric. We report the RMSEmin met- ric, along with the budget and the expected number of queried entities. Across all budgets, Top-k(æ) con- sistently outperforms Top- L2D by achieving lower error with fewer entities and reduced cost.
Exported Numbhe of Queried Agents F-a
Exported Numbher Queried Agents Z=d
T-?
Fwd
T-a
Z-6
T-?
R
T-
+ Top-k(s) L4D
Top-k(e) L2D
++ Top & L2D
+ Top L2D
RMSSEw x 100
RMSEE -aee x
12
Top-1
The- L2D
10
10.
Oytimal Number Aguss
Opeined Number Agezz
0.025
0.050
0.075
O 100
0.125
0.150
0.175
"0.000
0.025
00.000
100
0.125
0.150
0.175
0.000
Budget J
Budget
(c) Comparison of Top-k L2D and Top-k(*) L2D un- der RMSEau metric.
(d) Comparison of Top-k L2D and Top-k(z)) L2D un- der RMSE metric.
Figure 7: Results on the California dataset comparing Top-k and Top-k(x:) L2D across four evalua- tion metrics. Top-k(zz) consistently achieves superior performance across all trade-offs. The Top-1 L.2D corresponds to Narasimhan et al. (2022) ): |Mo et al. (2024c).
institutional data siloing. As result, the entities exhibit heterogeneous predictive capabilities and error patterns.
Each entity is trained for 3 epochs using the Adam optimizer (Kingma & Ba 2014). with batch size of 64 and learning rate of 1 x 10-3, Model selection is performed based on the lowest loss on each entity's respective validation subset. The table 5 below reports the classification accuracy of each trained entity, evaluated on a common held-out validation set:
Table 5: Accuracy of each entity on the SVHN validation set.
Entity
5
Accuracy (%)
63.51
55.53
61.56
62.60
66.66
64.26
Top-k L2D. We train the rejector using a ResNet-4 architecture (He et al. 2016). following Al- gorithm I The model is trained for 50 epochs with a batch size of 256 and an initial learning rate of 1 10 scheduled via cosine annealing. Optimization is performed using the Adam optimizer. We select the checkpoint that minimizes the Top-k surrogate loss on the validation set, yielding the final rejector r. We report Top-k L.2D performance for each fixed value & E A.
41

---

Published as a conference paper at ICLR 2026
Top-k(æ) L2D. We reuse the trained rejector and follow Algorithm to train cardinality cardi- nality function s S. The cardinality function is composed of CLIP-basee feature extractor (Rad-) ford et al. 2021) and lightweight classification head. It is trained for 10 epochs with batch size of 256, a learning rate of x 10-3, weight decay of 1 x 10-5, and cosine learning rate scheduling. We use the AdamW optimizer (Loshchilov & Hutter 2017) for optimization.
Exported Number Quesied Agents
80
F-1
I-
T=g
T-t
D
Top-A(x) L2D
80
+
Top-k L2D
(,00
Optinal Apanta
70
80
Amrug-
20
Top-1
Optinal Tog-k
Top L2D
Random Top- L2D
T0g-1 L.0
0.000
0.022
0.000
0.05
0,100
0.125
0.150
0.175 0.200
0.025
0.050
0.075
0.100
0.125
0.150
0.175 0.200
Budget
Budges
(a) Comparison with Random and Optimal haselines
(b) Top-k((z) vs. Top-k L.2D on ACC
using AcC
Expoecter Numbhr Queried Agests R
Exppotaal Number Queried Agents
Optinad
Optinnl Namber Ageste
71.50
11.50
71.25
Arsws (06)
71.00
71.00
20.75
1 L223
LAD
70.50
70.50
70.25
Topskis) L20 Top-k
Ropstis L20
Top-k
70.00
70.00
0.175
Budget
Budget 7
(c) Performance under AcCmj metric.
(d) Performance under Accwst metrie.
Figure 8: Comparison of Top-k and Top-k(x) L2D across four accuracy metrics on SVHN. Top- k(x) achieves better budget-accuraee trade-offs across all settings. The Top-1 L2D corresponds to Montreuil et al. 2025b).
Performance Comparison. Figure 8 compares our Top-k and Top-k(æ) L2D approaches against
prior work (Narasimhan al. 2022 Mao et al. (023a), well oracle and random baselines.
As shown in Figure Sal querying multiple entities significantly improves performance, with both of our methods surpassing the Top-1 L2D baselines (Narasimhan et al. 2022 Mao al. (023a).. Moreover, our learned deferral strategies consistently outperform the random L2D baseline, un- derscoring the effectiveness of our allocation policy in routing queries appropriate entities. In Figure 8b Top-k(r:) L2D consistently outperforms Top-k L2D, achieving better accuracy under the same budget constraints.
For more restrictive metrics, Figures 8c and 8d show that Top-k(x) achieves notably stronger per- formance, particularly in the low-budget regime. For example, in Figure &c at budget of 3 = 0.41, Top-k(æ)) attains ACCmaj = 70.81, compared to Accmai = 70.05 for Top-k: This performance gap widens further at smaller budgets. Both figures also highlight that querying too many entities may degrade accuracy due to the inclusion low low-quality predictions. In contrast, Top-k(<) identifies better trade-off, reaching up Accmaj 71.56 under majority voting and Accw-vt 71.59 with weighted voting. As the California Housing experiment, weighted voting outperforms majority voting, suggesting that leveraging rejector-derived weights improves overall decision quality.
42

---

Published as a conference paper at ICLR 2026
B.3.3 RESULTS ON CIFARI00.
entity Settings. We construct pool of entities. We train main predictor (entity using ResNet-4 (He et al. 2016) for 50 epochs, a batch size of 256, the Adam Optimizer (Kingma & Ba 2014) and select the checkpoints with the lower validation loss. We synthetically create experts with strong overlapped knowledge. We assign experts to classes for which they have the probability to be correct reaching 0.94 and uniform non-assigned classes. Typically, we assign 55 classes to each experts. We report in the Table the accuracy of each entity on the validation set.
Table 6: Accuracy of cach entity the CIFARI00 validation set.
Entity
3
4
5
6
Accuracy (%) 59.74
51.96
52.58
52.21
52.32
52.25
Top-k L2D. We train the rejector model using ResNet-4 architecture (He et al. 2016), following the procedure described in Algorithm [1 The model is optimized using Adam with a batch size of 2048, an initial learning rate of 1 x 10 and cosine annealing over 200 training epochs. We select the checkpoint that minimizes the Top-k surrogate loss on the validation set, resulting the final rejector r. We report Top-k L2D performance for each fixed value k E A.
Top-k(x) L.D. We reuse the learned rejector r and train a cardinality cardinality function s E S described in Algorithm 2 The cardinality function is composed of CLIP-based feature tractor (Radford et al. 2021) and a lightweight classification head. It is trained using the AdamW optimizer (Loshchilov & Hutter 2017) with batch size of 128, a learning rate of 10-3, weight decay of 10-h, and cosine learning rate scheduling over 15 epochs. To evaluate performance under different decision rules, we conduct experiments using multiple instantiations of the metric d; detailed definitions and evaluation protocols are provided in Section B.I.I.
Performance Comparison. Figure 9b shows that Top-k L2D outperforms random query alloca- tion, validating the benefit of learned deferral policies. As shown in Figure 9a Top-k(x) further improves performance over Top-k by adaptively selecting the number of entities per query. In Fig- ures 9c and 9d Top-k(ar) consistently yields higher accuracy across all budget levels, achieving significant gains over fixed-k strategies.
Notably, unlike in other datasets, querying additional entities in this setting does not degrade perfor- mance. This is due to the absence of low-quality entities: each entity predicts correctly with high probability (at least 94%) its assigned class subset. As result, aggregating predictions from multiple entities improves accuracy by selectively querying them.
Nevertheless, Top-k(1:) remains advantageous due to the overlap between entities and their differing consultation costs. When several entities are likely to produce correct predictions, it is preferable defer the less costly one. By exploiting this flexibility, Top- achieves large performance improvement over Top-k L2D while also reducing the overall budget.
43

---

Published as a conference paper at ICLR 2026
Espected Number of Querled Agents &
R-a
+
Top- L2D
Nrw 1 (0%)
+++ Top- 1.0
Optinal
Atcqett (86)
85
70
Tog-1 120
80
Optimal Top At L2D
60
Top-k L2D
75
Top-k
Top-1
70--2-2-
0.06
0.025
0.050
0.076
0.100
0.125
0.150
Budget
Budget
(a) Comparison with Random and Optimal baselines using ACCmp-kk
(b) Top-k(x) Top-k L2D AcCmp.k.
Expooted Number of Queried Agents F
Exppected Number of Querite Agents
Euct
E-3
F-4
5
E
E
Euitt
E-3
E
E
E
90
Top L&D
Top- LOD
Tog-k L2D
+
Top-h L2D
Optioal Number of Agents
Oyelnal Numbrree Agents
Arow (06)
85
190
65
80
80
Aore
Ti--1 1.2
75
0.05
0.10
0.15
0.20
0.06
0.30
0.15
0.26
Budget
Budigtt 77
(c) Performance under AcCmaj metric.
(d) Performance under Accwst metric.
Figure 9: Comparison of Top-k and Top-k((r) I.2D across four accuracy metrics CIFAR100. Top- k(x)) achieves better budget-accuracy trade-offs across all settings. The Top-1 L2D corresponds to Narasimhan et al. (2022): Mao et al. (2023a).
44


Reviews:
[{"review_id": "MeuLDCv0kK", "reviewer": "Reviewer_28BC", "summary": "The paper considers a setting where, given a feature, a learner makes predictions or defers to a set of experts. They formulate the problem abstractly, as learning a model for selecting k assets, where each asset can either be a label or an expert. The hypotheses selected by the learner score each of these assets, and the loss incurred by the learner is a function of the top-k highest scoring assets. The specific loss function they consider is a sum over the individual losses of each of the selected assets. The individual loss is a cost-sensitive loss combining a predictive loss for the label selected by the model (or expert), and possibly a consultation loss for deferring to an expert. \n\nThey describe how their setting generalizes previous work in what they call the 1-stage and 2-stage settings. They use ideas from the literature to bound the cost sensitive loss with a surrogate, and prove statistical consistency. They state a version of the problem where k need not be fixed. Finally, they conclude with empirical experiments that compare two methods for fixed k and adaptive k.", "questions": "Why is Lemma 4.2 stated as a Lemma, when it seems more like a definition? The proof of the Lemma does not seem to help resolve it for me. The proof seems to state that the uniform top-k loss upper bounds the standard deferral loss (which #1 seems obvious, and #2 is not in the statement of Lemma 4.2). It is also interspersed with asides about special cases, which are not proving anything.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Strength: I like that the paper unifies the 1-stage and 2-stage settings in a single framework that considers experts and labels to simply be decisions (\"assets\" in the language of the paper) that need to be scored by some model.", "weaknesses": "Weakness 1. The theoretical portion of the paper considers a loss function that sums the individual losses of each asset. As a modeling assumption, this has a number of problems. In settings where there is a good expert, or when the model is good at identifying labels without experts, the model is forced to select a number of sub-optimal assets by the setting, and is strictly penalized for doing so. The aggregation also does not allow the learner to meaningfully improve its prediction (e.g. by taking a major vote). I believe the authors mitigate this somewhat by introducing the adaptive-k setting and by considering more interesting aggregation strategies in their experiments. However, it does call into question the utility of the theory around a somewhat degenerate loss function. \n\nWeakness 2: The surrogate loss optimized by the authors is independent of k (Corollary 4.4), and recovers the top-1 solution from previous literature. Thus, the authors' algorithm learns a hypothesis for the top-1 problem, and uses the k highest scores found by that hypothesis to pick its assets. In light of Weakness #1, the most interesting aspect of this problem over top-1 is to learn a model whose scores preserve rank over suboptimal assets, which this solution expressly avoids modeling.  \n\nWeakness 3: The experiments primarily compare the paper's methods against each other. I am not convinced that there are not more adequate baselines. Cortez et al. 2024, which the authors cite, for example, seems very relevant. \n\nWeakness 4: The paper casts and re-casts the problem a number of times before finally arriving at the formulation in lines 200-208 and 226-240. I think the presentation can be improved by presenting this more succinctly. In my opinion, the observation that this generalizes prior work need not take up more than a paragraph or two.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eQ8QVN4M42", "reviewer": "Reviewer_VC3c", "summary": "This paper studies the multi-expert Learning-to-Defer (L2D) problem and extends the classical Top-1 deferral framework to a more general Top-k deferral setting, where the system may consult multiple entities (base model and experts) simultaneously. The authors formulate the Top-k deferral loss, derive the Bayes-optimal Top-k selection rule, and propose the first Bayes-consistent surrogate for Top-k L2D that is independent of k. This allows a single learned scoring function to support any fixed k, as well as an adaptive variant Top-k(x) that chooses a sample-specific number of queried entities. The proposed framework unifies prior one-stage and two-stage L2D results as a special case of k=1, and the authors further validate the approach with extensive experiments demonstrating the benefits of consulting multiple entities and of adapting k to the input.", "questions": "Please refer to the weaknesses listed above. I would be glad to revise my rating if the authors address these points in a future version of the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed Top-k formulation strictly generalizes prior Top-1 L2D methods, and the theoretical results provably recover previous consistency guarantees when k=1. This provides a clean unification of existing L2D theory under a more general and expressive decision space.\n\n2. Theoretical guarantees are established for both one-stage and two-stage regimes, which is non-trivial. The surrogate achieves Bayes-consistency in both settings, thereby offering a comprehensive and unified theoretical foundation for multi-expert deferral.\n\n3. The empirical results are thorough and convincingly support the claims. The experiments show that Top-k L2D improves over Top-1 and random querying, and that the adaptive Top-k(x) strategy yields further improvements by selecting the appropriate number of experts per input under a given budget constraint.", "weaknesses": "1. The paper does not provide theoretical justification for why consulting and aggregating multiple entities should improve final prediction quality. The theory only covers the selection stage, while the aggregation step remains ad-hoc and intuition-based, leaving the core motivation for Top-k partially unsupported from a theoretical perspective.\n\n2. The paper assumes that all experts are deterministic functions of the input $X$ (line 108). \nIn contrast, the original multi-expert deferral work [1] does not impose this constraint. \nThis assumption appears restrictive, since an expert depending only on $X$ cannot outperform the most probable label, \nwhich may lead the Bayes-optimal deferral strategy to never defer. While this setting is indeed non-trivial under the zero extra cost or $\\mathcal{H}$-consistency framework, \nextending the analysis to stochastic experts would significantly strengthen the contribution and practical relevance of this work.\n\n3. In line 171, [2] is described as being related to $\\mathcal{H}$-consistency. However, [2] mainly focuses on Bayes consistency and surrogate regret bounds.\nIt is suggested to clarify this distinction to avoid confusion and to better position the related concepts.\n\n4. Several recent advances in learning to defer [3] and closely related work on classification with rejection [4, 5] are not cited. Including these developments would provide a more complete and up-to-date literature review.\n\n[1].  Rajeev Verma, Daniel Barrejon, and Eric Nalisnick. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. AISTATS 2023.\n\n[2]. Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. ICML 2020.\n\n[3]. Zixi Wei, Yuzhou Cao, and Lei Feng. Exploiting human-ai dependence for learning to defer. ICML 2024.\n\n[4]. Yuzhou Cao, Tianchi Cai, Lei Feng, Lihong Gu, Jinjie Gu, Bo An, Gang Niu, and Masashi Sugiyama. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. NeurIPS 2022\n\n[5]. Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Theory and algorithms for learning with rejection in binary classification. Annals of Mathematics and Artificial Intelligence, 92(2), 2024, 277–315.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tqACRCHbov", "reviewer": "Reviewer_jytm", "summary": "This paper studies the Learning-to-Defer (L2D) problem, where a model may defer prediction to external experts. While existing L2D frameworks only allow deferral to a single expert, this work introduces a more general framework that supports deferring to the top-$k$ most cost-effective experts. The authors further develop an adaptive extension, Top-$k(x)$, which selects the optimal number of experts per input based on expert quality, instance difficulty, and consultation cost.\n\nTo support this generalization, the paper proposes a surrogate loss that is Bayes-consistent and class-consistent in both one-stage and two-stage L2D settings. Importantly, this surrogate is independent of $k$, meaning that a single model can be trained once and deployed for arbitrary $k$. The authors show that this framework strictly subsumes prior approaches, including selective prediction and Top-1 L2D, and they also unify cascade-style inference under the same formulation. Experimental results suggest that Top-$k$ and Top-$k(x)$ achieve better accuracy–cost trade-offs compared to existing single-expert L2D methods.", "questions": "- Can the authors comment on how the framework would behave when expert accuracy varies significantly across different regions of the input space? Is there theoretical or empirical justification that the learned top-$k$ policy will avoid consulting redundant experts?\n- How sensitive is the adaptive Top-$k(x)$ approach to the choice of regularization on consultation cost? Are there guidelines for selecting these parameters in practice?\n- In real-world settings where experts may be humans, costs and response times are not fixed. Could the authors discuss whether their framework can incorporate stochastic or dynamic consultation costs?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel and compelling extension of the L2D framework to multi-expert settings, which appears natural and highly relevant in domains requiring collective expert judgment.\n- The theoretical development is rigorous, with proofs of Bayes-consistency and class-consistency that generalize known guarantees from Top-1 to Top-$k$ and adaptive settings.\n- The unifying perspective on existing L2D, top-$k$ classification, selective prediction, and cascaded models is conceptually valuable and may stimulate further research in this direction.\n- Experimental results demonstrate noticeable improvements over prior approaches, particularly under constrained consultation budgets.", "weaknesses": "- While the theoretical contributions are strong, empirical evaluation is somewhat limited. The experiments focus on standard benchmarks, and it remains unclear how well the proposed framework performs in practical multi-expert environments with heterogeneous expert behavior or human-in-the-loop settings.\n- The implementation details and computational overhead of training with multi-expert routing are not discussed in depth. In practice, the cost of querying multiple experts and potential resource constraints may affect feasibility.\n- The method assumes that expert predictions are readily available. It would be valuable to discuss applicability when expert labels are costly or sparse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RmSeVdp8cy", "reviewer": "Reviewer_mQGW", "summary": "Learning-to-defer (L2D) deals with utilizing expert advice for queries. Existing approaches only consider using a single expert, and this paper proposes a loss that works for multiple experts and satisfies desirable consistency objectives. Further, the author shows that prior setups are special cases of the novel approach. Experiments are provided to justify the theoretical results.", "questions": "Overall, the presentation could use significant work. For instance:\n- Can $\\mathcal{Y}$ be defined at the beginning of the preliminaries? More generally, please provide a more exhaustive discussion of notation, eg. either in the preliminaries, or as a table in the appendix.\n- What is the definition of consistency, calibration? It would be good to add a section in the appendix for additional preliminaries to make all of this self-contained.\n\nAdditional:\n- Is top-k(x) also consistent (as in Theorem 4.7)?\n- In Figure 5, is there intuition why accuracy for top-k isn't monotonic increasing (in budget) whereas top-k(x) is?\n- wording in line 400 is a little unclear: minimizability gap... vanishes... if the hypothesis set is as rich as the set of all measurable functions", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper is well motivated and contextualized. For instance, it is good that the authors mention why the naive extension of the top-1 selection doesn't work (line 216).\n- Extensive experiments are provided, and description of setup is very detailed.", "weaknesses": "- Some of the notation is kind of cumbersome, eg the superscripted down and up-arrows. Also, hat seems to be used for many things: argmax (eg line 115), experts, and predictions (lines 134-135). It would be better to use something like asterisk for argmax, eg $h^* = \\arg\\max h_i$\n- It looks like the proof of Lemma A.6 is incorrect/incomplete, as eq 15 is directly assuming the implication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fd51Iyz0U0", "reviewer": "Reviewer_U2o1", "summary": "This paper proposes Top - $k$ Learning-to-Defer (L2D), a novel framework that allows a model to defer each query to a set of $k$ experts instead of a single expert. This generalizes all prior L2D approaches, which were limited to one expert (“Top-1”) deferrals. The authors formulate a unified approach covering both major paradigms of L2D: in the one-stage regime, the predictor and deferral policy are learned jointly in one model, and in the two-stage regime, experts and a main predictor are trained offline and a separate routing function learns to allocate queries. Crucially, the framework recovers the traditional Top-1 deferral as a special case while enabling principled collaboration with multiple experts when $k>1$.  In addition, the paper introduces Top-$k(x)$ L2D, an adaptive extension where the system learns to choose a query-specific number of experts (up to $|A|$, the total pool) based on factors like input difficulty, expert accuracy, and consultation cost. To support both fixed-$k$ and adaptive deferral, the authors develop a $k$-independent surrogate loss that facilitates practical training.", "questions": "- Training Strategy for $k$-Selection: Could you elaborate on how the deferral policy $\\pi(x,\\cdot)$ and the cardinality function $k_\\theta(x)$ are trained in practice? Was the $k$-adaptive model trained in an end-to-end fashion, or did you first learn a Top-$|A|$ policy and then learn the cardinality predictor (or vice versa)? \n\n- Decision Rule for Multiple Experts’ Outputs: In a deployment, once the Top-$k$ experts are consulted for a query, how is their collective response used to make a final prediction? In your experiments, what rule did you use to decide the final output or correctness from multiple predictions? For example, for CIFAR-10 with $k=2$ experts selected, did you consider the query correct if either expert (or the model) got the label right (i.e. at least one correct prediction in the set), or did you require agreement/majority? Different aggregation strategies (e.g., majority vote vs. oracle-style “at least one correct”) can yield different accuracy metrics. It would clarify your evaluation if you specify the aggregation/decision procedure for multiple expert opinions. \n\n- Sensitivity to Cost Hyperparameters ($\\alpha,\\beta,\\lambda$): The framework introduces cost parameters $\\alpha_j$ and $\\beta_j$ for each expert (measuring error penalty vs. consultation cost), as well as a regularization weight $\\lambda$ in the adaptive loss. How sensitive are your results to the choice of these values? In practice, one might not know the exact cost of consulting an expert or might have to guess a trade-off parameter. Did you perform any analysis or ablation where you vary $\\beta_j$ or $\\lambda$ to see how the deferral behavior changes? For instance, if all $\\beta_j$ are set to 0 (ignoring cost), does the policy simply always choose all experts (maximizing accuracy)? And conversely, if $\\beta_j$ are very high, does it revert to always using the model (to save cost)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Innovative Multi-Expert Deferral Framework: This is the first work to allow deferring to multiple experts in a Learning-to-Defer setting, addressing a real-world gap. The idea is original and expands the scope of L2D beyond the restrictive one-expert paradigm. By formulating Top-$k$ deferral, the paper enables more robust decision-making (aggregating complementary expertise) which is crucial for complex tasks. This multi-expert allocation capability is a direct answer to the stated limitation of prior work (that relying on only one expert can amplify bias or error). The framework cleanly unifies both one-stage and two-stage L2D, showing broad applicability across different training setups. It also strictly generalizes previous approaches: for example, the authors show that setting $k=1$ recovers the standard L2D solutions in both regimes, and even classical model cascades and selective prediction emerge as special cases of their formulation. \n\n- Theoretical Foundations: The paper provides theoretical analysis and guarantees. The introduction of a $k$-independent surrogate loss function is a technical strength. It is proven to be Bayes-consistent, meaning that as the amount of training data grows, the learned deferral policy will converge to the Bayes-optimal top-$k$ strategy (the one that minimizes true expected cost). Furthermore, the surrogate is shown to maintain H-consistency within restricted hypothesis classes providing bounds on the excess risk when using finite-capacity models. These results extend known consistency bounds from $k=1$ to the multi-expert scenario. The theoretical development is non-trivial and well-executed.\n\n- Empirical Effectiveness: The proposed methods show empirical advantages over prior approaches. Across multiple benchmarks, Top-$k$ deferral improves the trade-off between prediction accuracy and consultation cost. Notably, Top-$k(x)$ (adaptive) is very effective: it achieves comparable or better accuracy with significantly lower average cost by varying the number of experts per query. For instance, in one experiment the adaptive policy reached the same RMSE as a fixed $k$ policy while querying ~1.3 fewer experts on average (saving ~25% of the budget). This demonstrates efficient use of experts. The results also confirmed that even a fixed Top-$k$ (with an appropriate $k$) outperforms the traditional Top-1 deferral baseline in accuracy, under the same cost budget. Overall, the authors back up their claims with comprehensive experiments in both one-stage (e.g. CIFAR-10 image classification with learned deferral) and two-stage (e.g. regression with a separate main model and experts) scenarios.", "weaknesses": "- Complexity of Training & Implementation: One potential weakness is the added complexity of the proposed approach, especially the adaptive Top-$k(x)$ mechanism. The framework introduces an additional model (the cardinality function $k_\\theta(x)$) that must be learned alongside the main deferral policy. This effectively means training two interconnected components: one to score and select experts, and another to decide how many to select. The paper defers some of the training details to the appendix (Algorithms 1 and 2 outline separate procedures for the policy and the cardinality function), but it’s not entirely clear from the main text whether these components are trained jointly or sequentially, and how hyperparameters (especially the regularization $\\lambda$ in $\\ell_{\\text{card}}$) are chosen. The need to tune $\\lambda$ to balance accuracy vs. cost could require expert knowledge or cross-validation on a utility metric, which adds complexity. In practice, implementing Top-$k(x)$ may be more involved than standard L2D due to this bi-level optimization (learn policy given a fixed $k$, and learn the $k$-selection function), a point the authors could clarify further. \n\n- Evaluation Scope – Lack of Real-World Expert Data: The experiments, while thorough on synthetic and benchmark data, do not include a real-world multi-expert scenario, which slightly limits the demonstration of the framework’s practical value. All evaluations seem to use either simulated experts (e.g., classifiers with predetermined accuracy on subsets of classes) or machine models as stand-in “experts.” While this is common in L2D research (since obtaining multiple human experts’ responses can be difficult), it would have been insightful to see at least one case with actual human experts or a real ensemble to validate that Top-$k$ deferral works outside the lab setting.\n\n- Assumptions for Theoretical Guarantees: The consistency guarantees, while impressive, rely on certain assumptions that may not hold in practice. For example, the two-stage consistency assumes that the Bayes-optimal cost can be approached arbitrarily well by the model classes. Essentially this is a realizability or universal approximation assumption for the hypothesis spaces. If the chosen model architectures are misspecified or capacity-limited, there could be a minimizability gap where the learned policy cannot reach the true optimum. The authors do acknowledge this, but in practice one never knows if one’s model is rich enough. Thus, the Bayes-consistency is a bit theoretical, at finite sample sizes with imperfect models, one might not achieve the optimal deferral policy. The paper does not discuss the finite-sample performance beyond asymptotic consistency; for instance, how fast is the convergence or how the excess risk scales with $k$. Some empirical calibration of the surrogate (like comparing surrogate risk vs true deferral risk on validation data) could help validate that the loss is working as intended in finite data regimes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)