A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
Authors: Yannis Montreuil
Yeo Shu Heng
Axel Carlier
Lai Xing Ng
Wei Tsang Ooi
Venue: ICML 2025 poster (2025)
Keywords: Learning to defer
learning from abstention
learning theory
multi task
Primary area: theory->learning_theory
TL;DR: Extend Two-Stage Learning-to-Defer to multi-task problems, providing additional theoretical guarantees.

Abstract: The Two-Stage Learning-to-Defer (L2D) framework has been extensively studied for classification and, more recently, regression tasks. However, many real-world applications require solving both tasks jointly in a multi-task setting. We introduce a novel Two-Stage L2D framework for multi-task learning that integrates classification and regression through a unified deferral mechanism. Our method leverages a two-stage surrogate loss family, which we prove to be both Bayes-consistent and $(\mathcal{G}, \mathcal{R})$-consistent, ensuring convergence to the Bayes-optimal rejector. We derive explicit consistency bounds tied to the cross-entropy surrogate and the $L_1$-norm of agent-specific costs, and extend minimizability gap analysis to the multi-expert two-stage regime. We also make explicit how shared representation learning—commonly used in multi-task models—affects these consistency guarantees. Experiments on object detection and electronic health record analysis demonstrate the effectiveness of our approach and highlight the limitations of existing L2D methods in multi-task scenarios.

A Two-Stage Learning-to-Defer Approach for Multi-Tan Task Learning
Yannis Montreuil 123*
Shu Heng Yeo
Axel Carlier 455
Lai Xing Ng Wei Tsang Ooi
Abstract
learning-based model has lower confidence than the most confident expert. This deference mechanism enhances safety, which is particularly crucial in high-stakes scenarios (Mozannar & Sontag, 2020; Mozannar et al., 2023; Mao, 2025). For example, in medical diagnostics, the system utilizes patient-acquired data to deliver an initial diagnosis (Johnson et al., 2023; 2016). If the model is sufficiently confident, its diagnosis is accepted; otherwise, the decision is deferred to medical expert who provides the final diag- nosis. Such tasks, which can directly impact human lives, underscore the need to develop reliable systems (Balagu- runathan et al., 2021).
The Two-Stage Learning-to-De (L22) frame- work has been extensively studied for classifica- tion and, more recently, regression tasks. How- ever, many real- applications require solving both tasks jointly in a multi-task setting. We in- troduce a novel Two-Stage L.2D framework for multi-task learning that integrates classification and regression through a unified deferral mech- anism. Our method leverages two-stage sur- rogate loss family, which we prove to be both Bayes-consistent and (G, R)-cosiistent, ensur- ing convergence to the Bayes-optimal rejector. We derive explicit consistency bounds tied to the cross-entropy surrogate and the Lı-norm of agent-specific costs, and extend minimizability gap analysis the multi-expert two-stage regime. We also make explicit how shared representa- tion
Learning-to-Dest has been extensively studied in classifi- cation problems (Madras et al., 2018; Verma et al., 2023; Mozannar & Sontag, 2020; Mozannar al., 2023; Mao et al., 2023a) and, more recently, in regression scenarios (Mao et al., 2024h). However, many modern complex tasks involve both regression and classification components, re- quiring deferral to be applied to both components simultane- ously, as they cannot be treated independently. For instance, in object detection, a model predicts both the class of an object and its location using regressor, with these outputs being inherently interdependent (Girshick, 2015; Redmon et al., 2016; Buch et al., 2017). In practice, deferring only localization or classification is not meaningful, as decision- makers will treat these two tasks simultaneously. A failure in either component- as misclassifying the object or inaccurately estimating its position- can undermine the en- tire problem, emphasizing the importance of coordinated deferral strategies that address both components jointly.
used in multi-task mod- els-affects these consistency guarantees. Exper- iments on object detection and electronic health record analysis demonstrate the effectiveness of our approach and highlight the limitations of ex- isting L2D methods multi-task scenarios.
1. Introduction
Learning-to- Defer (L2D) integrates predictive models with human experts-or, more broadly, decision-makers--a op- timize systems requiring high reliability (Madras et al., 2018). This approach benefits from the scalability of ma- chine learning models and leverages expert knowledge to ad- dress complex queries (Hemmer et al., 2021). The Learning- to to-eefer approach defers decisions to experts when the
This potential for failure underscores the need for Learning-to- Mrriing-to--eiiie -eoeeeeer approach tailored to multi-task problems involving both classification and regression. We propose a novel framework for multi-task environments, incorporating expertise from multiple experts and the predictor-regresso model. We focus our work on the two-stage scenario, where the model is already trained offline. This setting is relevant when retraining from scratch the predictor-regressor model is either too costly or not feasible due to diverse constraints such as non-open models (Mao et al., 2023a; 2024h). We approximate the true deferral loss using a surrogate deferral loss family, based on cross-entropy, and tailored for the two- stage setting, ensuring that the loss effectively approximates the original discontinuous loss function. Our theoretical
Equual contribution 'Scoool of Computing, National Uni- versity of Singapore, Singapore "Institute for Infocomm Re- search, Agency for Science, Technology and Research, Singapore CNRS@CREATE LTD, 1 Create Way. Singapore 'IRIT, Univer- sité de Toulouse, CNRS, Toulouse INP, Toulouse, France 'IPAL, IRL2955, Singapore. Correspondence to: Yannis Montreuil <yan- nis. nis.montreuil @ @du>..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
analysis establishes that our surrogate loss is both (G, R)- consistent and Bayes-consistent. Furthermore, study and generalize results on the minimizability gap for deferral loss based on cross-entropy providing deeper insights into its optimization properties. Our contributions are as follows:
2024; Wei et al., 2024). A pivotal contribution by Mozannar et al. (2023) challenged the sufficiency of Bayes-consistency, showing that existing score-based methods may be subop- timal under realizable distributions caaurbutionsppatiulllaal when the hypothesis class is restricted. They introduced the notion of hypothesis-conas consistencs,. which strengthens theoretical align- ment between the surrogate loss and the constrained hypoth- esis space. This work sparked a broader effort to refine the theoretical foundations of L.2D using tools from surrogate risk analysis (Long & Servedio, 2013; Zhang & Agarwal, 2020; Awasthi et al., 2022; Mao et al., 2023b). Recent the- oretical advances have solidified the status of score-basce L2D. Mao et al. (2024f) established that the general score- based L.2D framework achieves H-consistency, while Mao et al. (2024g; 2025b) introduced a novel surrogate loss that guarantees realizable-consiitengg
(i) Novelty: We introduce two-stage Learning-to-Des for multi-task learning with multiple experts. Unlike previous L2D methods that focus solely on classification or regres- sion, our approach addresses situations where sole optimal agent has to be selected to jointly handle both tasks in unified framework.
(ii) Theoretical Foundation: We prove that our surrogate family is both Bayes-consistent and (g, -RCoosssstt for surrogate. We derive tight consis- tency bounds that depend on the choice of the surrogate and the L1 nn-mo of the cost, extending minimizability gap anal- ysis to the two-stage, multi-expert setting. Additionally, we establish learning bounds for the true deferral loss, show- ing that generalization improves as agents become more accurate.
any cross-entropy-baat
optimality under realizable distributions. Montreuil et al. (2025c) generalize I.2D) to deferral to the set of top-k experts. Beyond clas- sification, the L2D framework has also been extended to regression (Mao et al., 2024h), demonstrating its applicabil- ity in continuous- settings with expert deferral.
(iii) Empirical Validation: We evaluate our approach on two challenging tasks. In object detection, our method effectively captures the intrinsic interdependence between classification and regression, overcoming the limitations of existing L2D approaches. In EHR analysis, we show that current L2D methods struggle when agents have varying expertise across classification and regression- method achieves superior performance.
Two-stage Learning-to-Der scale pretrained models has motivated the development of two-stage L2D frameworks, where both the model and the expert agents trained offline. This reflects practical con- straints: most users lack the computational resources to fine-tune large models end-to-end. Narasimhan et al. (2022) were the first to formalize this setting, and Mao et al. (2023a) introduced a dedicated predictor-rejector architecture tai- lored for two-stage I.2D,, with theoretical guarantees includ- ing both Bayes- and bypothesis-consistennyy. Charusaie et al. (2022) offered a comparative analysis of one-stage (joint training) and two-stage (post hoc) L2D, highlighting trade- offs between model flexibility and sample efficiency. More recently, two-stage L2D has been successfully extended to regression (Mao et al., 2024h) and top-k expert deferral (Montreuil et al., 2025b), and has been applied to real-world tasks such as extractive question answering (Montreuil et al., 2024) and adversarial robustness (Montreuil et al., 2025a).
The emergence of large-
our
2. Related Work
Leareing-to-Defer builds upon the foundational ideas of Learning with Abstention (Chow, 2003; Bartlett & Wegkamp, 2008; Cortes et al., 2016; Geifman & El-Yaniv, 2017; Ramaswamy et al., 2018; Cao et al., 2022; Mao et al., 2024a), where a model permitted abstain from making prediction when its confidence is low. The core insight of L2D is to extend this framework from rejection to de- delegating uncertain decisions to external agents or experts whose confidence may exceed that of the model.
Despite significant progress, current two-stage L2D re- search largely addresses classification and regression in- dependently. However, many contemporary tasks involve both regression and classification components, necessitating their joint optimization. In this work, we extend two-stage L2D to joint classifier-regressor models, addressing this critical gap.
One-stage Learning-to-Dede Defer. L2D was originally intro- duced by Madras et al. (2018) 8) for binary classification, using a pass function inspired by the predictor-rejector framework of Cortes et al. (2016). In the multiclass setting, Mozan- nar & Sontag (2020) proposed a score-based formulation that leverages a log-softmax surrogate to ensure Bayes- consistency. This formulation has since been extended to wide range of classification tasks (Okati et al., 2021; Verma et al., 2023; Cao et al., 2024; 2022; Keswani et al., 2021; Ker- rigan et al., 2021; Hemmer et al., 2022; Benz & Rodriguez, 2022; Tailor et al., 2024; Liu et al., 2024; Palomba et al.,
3. Preliminaries
Multi-task scenario. We consider a multi-task setting encompassing both classification and regression problems. Let X' denote the input space, y = (l,..., n) represent the
2

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
set of n distinct classes, and T C R denote the space of real-valued targets for regression. For compactness, each data point is represented as triplet
following implication holds:
Eog (ha)) - Egg, (7) K-00
(a, y, t) E 2,
where 2 = X' x y x T. We assume the data is sampled independently and identically distributed (i.i.d.) from a distribution D over Z (Girshick, 2015; Redmon et al., 2016; Carion et al., 2020).
(2)
= Eesr (hx) El (H)
1-000 0.
This property assumes that H = Hall. condition that does not necessarily hold for restricted hypothesis classes such as Hiin or HRa.U (Long & Servedio, 2013; Awasthi et al., 2022). To address this limitation, Awasthi et al. (2022) proposed H-consistency bounds. These bounds depend on non-decreasing function r IR+ - IR+ and are expressed
We define a backbone we E W, or shared feature extractor, such that w: X' + Q. For example, can be deep net- work that takes an input x E X and produces a latent feature vector q = wxx)= E Q. Next, we define a classifier h E H. representing all possible classification heads operating on Q. Formally, h is a score function defined as h : Q x y + R. where the predicted class h(()) = arg maxyey h(g, y)- Likewise, we define a regressor f F., representing all regression heads, where f: Q - T. These components are combined into a single multi-head network y E GG where 9 = {g g(z)= = (h o w w(zz) w E W, h E H. f E F). Hence, 9 jointly produces classification and regression outputs, h(q) and f()). from the same latent representation q = w(x).
as:
Eor, (1)) Egg, (H) Mag (72 2
(3)
I(Een (h) B (H) + U1o (7)):
where the minimizability gap Ur (H) measures the dispar- ity between the best-in-class generalization error and the expected pointwise minimum error: Uto (H) = =E,, (H) E.z [insee By (fo) (h(zr), ; y)]] Nooaably, the minimizabil- ity gap vanishes when H = Hall (Steinwart, 2007; Awasthi et al., 2022; Cortes et al., 2024; Mao et al., 2024e;d:j:b.. In the asymptotic limit, inequality (3) guarantees the recovery of Bayes-consistency, aligning with the condition in (2).
Consistency in classification. In the classification setting, the goal identify a classifier h H in the specific case where wx(x) 2r such that h(x)) = arg max h(r, y)- This classifier should minimize the true error Eeg, (h), de- fined as Ee,, (h) = (aayy) [e (?xx), v)] The Bayes- optimal error is given by E (H) = infREm Exs2 (h). How- ever, directly minimizing Ex,, (h) is challenging due to the non-diffcrentiabiiiyy of the true multiclass 0-1 loss (Zhang, 2002; Steinwart, 2007; Awasthi et al., 2022: Cortes et al., 2025; Mao et al., 2025c). This motivates the introduction of the cross-entropy multiclass surrogate family, denoted by %% H. x X x y + R=, which provides a convex upper bound to the true multiclass loss t01- This family is parameterized by y 2 0 and encompasses standard surro- gate functions widely adopted in the community such as the MAE (Ghosh et al., 2017) or the log-softmax (Mohri et al., 2012).
4. Two-stage Multi-Task L2D: Theoretical Analysis
4.1. Formulating the True Deferral Loss
We extend the two-stage predictor-rejector framework, orig- inally proposed by (Narasimhan et al., 2022; Mao et al., 2023a), to the multi-task setting described in Section 3. Specifically, we consider an offline-trained model g E G. which jointly performs classification and regression. In addi- tion, we assume access to J offline-trainet experts, denoted Mj for j E (1......J) J). Each expert outputs predictions of the form my(x))
(m² m// (x)). where (x) y and mif (x) E T correspond to the classification and re- gression components, respectively. Each expert predic- tion lies in a corresponding space Mj, so that my E My. We denote the aggregated outputs of all experts as m(x) = (m) ... m=(r)) E -M- == I[ Mj. We write [-]== := JJ to denote the index set of experts, and define the set of all agents as A == (0) U [J], where agent 0 corresponds to the model g. Thus, the system con- tains |A| = 1 agents in total.
E Eyer ch(xze)) - log Ey Maw' MMaa -Maw))
- ##1
I-v
v ==1.
(1)
The corresponding surrogate error is defined as Eog, (h) B(r.y) ( (h(z). v)]. with its optimal value given by Egg, (H) = inf Eag, (h). crucial property surro- gate loss is Bayes-consistency, which guarantees that min- imizing the surrogate generalization error also minimizes the true generalization error (Zhang, 2002; Steinwart, 2007; Bartlett et al., 2006; Tewari & Bartlett, 2007; Mao et al., 2025a; Zhong. 2025). Formally, L is Bayes-consistent with respect to fo if, for any sequence (hx keeN C H. the
To allocate each decision, we introduce a rejector function R, where X A - IR. Given input E X, the rejector selects the agent j E AA that maximizes its score: r(z) = arg max r(z, r(z,) j). This mechanism induces the deferral loss, a mapping faet
+ R,, which quantifies the cost of allocating a decision to a particular agent.
3

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
Definition 4.1 (True deferral loss). Let an input x E X, for any R. we have the true deferral loss:
Although Lemma 4.2 characterizes the Bayes-optimal pol- icy under the true deferral loss ldet. this loss is non- differentiable and thus intractable for direct optimization in practice (Zhang, 2002).
(a 9, (,m,22) = Σ 5=0
2)
4.2. Surrogate Loss for Two-Stage Multi- Tassi L2D
with bounded cost ej that quantifies the penalty incurred when allocating the decision to agent j A. When the rejector r E R. predicts r(x)- = O, the decision is assigned to the multi-task task model 9, incurring base cost co defined as co(g(z), 2) = p(g(z), 2), where :(- E R measures the discrepancy between the model's output g(x) and the ground truth z. Conversely, if the rejector selects r(z) = j for some j > 0, the decision is deferred to expert j. yielding deferral cost (z(m)(z)) 2) (1)) + Bj. Here, 33 2 0 denotes the querying cost associated with invok- ing expert j, which may reflect Noomain-specifii constraints such as computational overhead, annotation effort, or time expenditure.
Introducing the surrogate. To address the optimization challenges posed by discontinuous losses (Berkson, 1944; Cortes & Vapnik, 1995), we introduce a family of con- vex surrogate losses with favorable analytical properties. Specifically, we adopt the multiclass cross-entropy surro- gates &% R X A + Rt, which upper-bounds the multiclass 0-1 loss for and facilitates gradient-based based optimization. This surrogate family is defined in Equation 1.
Building on the framework of Mao et al. (2024h), who proposed convex surrogates for deferral settings, we extend their approach to account for the interdependence between classification and regression tasks. our setting, this yields family of surrogate losses & R. x G x M x Z + R+. which incorporate the full structure of the multi-task cost.
When the classification and regression objectives are sepa- rable, the total cost can be decomposed as cj Adla cTE, where Ash x 2 0 specify the relative impor- tance of each task. A neutral setting is recovered when Asla = x = 1, ensuring a task-agnostic trade-off. If classification performance is prioritized, one can select Asla > a to favor agents with stronger classification ex- pertise.
Lemma 4.3 (Surrogate Deferral Surrogates). Let x X and let 4% be a multiclass surrogale loss. Then the surro- gate deferral loss for J + 1 agents is given by
(r.. = Σ
(r,,,,
where the aggregated cost weights are defined as Ty m(())) C) (,.))
Optimal deferral rule. In Definition 1. we introduced the true deferral loss Eeet, which quantifies the expected cost incurred when allocating predictions across the model and experts. Our goal to minimize this loss by identifying the Bayes-optimal rejector r R that minimizes the true risk. To formalize this objective, we analyze the pointwise Bayes rejector B (xx), which minimizes the conditional risk Cr The corresponding population risk is given by Eege (g. r) = Ez (x (g, r, ") The following lemma characterizes the optimal decision rule at each input x E X.
The surrogate deferral loss Pher combines the individual surrogate losses 46 (r, x, i) for each agent E A, weighted by the corresponding aggregated cost Tj. Intuitively, To quantifies the total cost of deferring to any expert instead of using the model, while T, for j > reflects the total cost incurred by selecting expert j instead of any other agent, including the model and other experts.
This construction preserves task generality and only requires that the base surrogate b% admit an R-consistency bound. The modular formulation of the cost functions ej allows this surrogate to flexibly accommodate diverse multi-task settings.
Lemma 4.2 (Pointwise Bayes Rejector). Given an input t E X and data distribution D, the rejection rule that minimizes the conditional risk Cla, associated with the true deferral loss lag is:
0 if inf Enf Eysta[coo] I min Ey, lcy] jE(M]
Consistency of the surrogate losses. In Lemma 4.3, we established that the proposed surrogate losses form con- vex upper bound on the true deferral loss laet- However, it remains to determine whether this surrogate family pro- vides reliable approximation of the true loss in terms of optimal decision-making In particular, it is not immediate that the pointwise minimizer of the surrogate loss, r" (x), aligns with the Bayes-optimal rejector (z) that minimizes laet- To address this, we study the relationship between
gEg otherwise,
The proof is provided in Appendix B. Lemma 4.2 shows that the optimal rejector E R assigns the decision to the model E G whenever its expected cost is lower than that of any expert. Otherwise, the rejector defers to the expert with the minimal expected deferral cost.
4

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
the surrogate and true risks by analyzing their respective excess risks. Specifically, we compare the surrogate cess risk, Exy,
offline, it reasonable to assume that the ex-exesss risk also
vanishes: Eng (9x) - E (Gan) +Mgg (Gan) K= O. Combin- ing the two convergence results and invoking the properties of I", we conclude that
(G, R), to the true excess risk, Ee r) - E (g. R). Understanding this discrepancy is crucial for establishing the (G, R)- connsittenyy of the surro- gate loss family, a topic extensively studied in prior work on multiclass surrogate theory (Steinwart, 2007; Zhang, 2002: Bartlett et al., 2006; Awasthi et al., 2022).
del (g, Egg, sst
Eg rk) Etax (Gall, Rall) + (ernrGaun Rall) k O.
Hence, the following corollary holds:
Corollary 4.5 (Bayes-consistenccy of the deferral surrogate
Leveraging consistency bounds developed in (Awasthi et al., 2022; Mao et al., 2024c), we present Theorem 4.4, which proves that the surrogate deferral loss family Djer is indeed (G. R)- consiisent.
losses). Under the conditions of Theorem 4.4, and assuming (g, R) = (Gull, Rutt) and (xo(g)) E. (Gal) k O, the surrogate deferral loss family 4 is Bayes-consistent with respect to the true deferral loss Ede.. Specifically, minimiz- ing the surrogate deferral excess risk ensures convergence of the true deferral excess risk. Formally, for sequences {re)KEN C RR and ( C GG we have:
Theorem 4.4 ((G, Ri-consistencyy bounds). Let g E 9 be a multi-task model. Suppose there exists a non-decreasing function I" F + R.,. parameterized by v 2 O, such that the R R-consistency bound holds for any distribution D:
(rx) - -E k-00, 0
Egg (r) -Ea, (R) + May (R) 2
Eehy rk) EB (Gat: Raul k k+000 O.
T (Et. (r) - -Et (R) + Ue (R))
then for any (g.r) E &G* x R, any distribution D, and any E X,
This result confirms that, as k + co, the surrogate losses &% attain asymptotic Bayes optimality for both the rejector r and the offline-trained multi-task model g. Thus, the surrogate family faithfully approximates the true deferral loss in the limit. Moreover, the pointwise surrogate-optimal rejector r (a)) converges to a close approximation of the Bayes- rejector ra (xx), thereby inducing deferral decisions consistent with the characterization in Lemma 4.2 (Bartlett et al., 2006).
E) - 1G R) + Ui R, S
T" (Eoz (r) -E,y (R) Us: is. (R) + Egg(g) - E.0 + Urg (9),
where the expected aggregated cost vector is given by T (By,ttzl [Tu]...... B,, [T]]). and I" (u) T with I ("())= = =T... (u). In the case of the log- sof-sofmax sur- rogate (v 1). the transformation is given by T==1 (u) 14th log(1 1 + u) + 1- log(1 (1 n). 2
Analysis of the minimizability gap. In As shown by Awasthi et al. (2022), the minimizability gap does not van- ish in general. Understanding the conditions under which it arises, quantifying its magnitude, and identifying effective mitigation strategies are crucial for ensuring that surrogate- based optimization aligns with the true task-specific objec- tives.
The proof of Theorem 4.4, along with generalizations to any V 2 0, is provided in Appendix C. This result yields refined consistency guarantees for the surrogate deferral loss, im- proving upon the bounds established by Mao et al. (2024h). The bounds are explicitly tailored to the cross-entropy sur- rogate family and parameterized by 1/, allowing for precise control over the surrogate's approximation behavior. Cru- cially, the tightness of the bound depends on the aggregated deferral costs, and is scaled by the Inrm || which quantifies the cumulative cost discrepancy across agents.
We provide a novel and strong characterization of the mini- mizability gap in the two-stage setting with multiple experts, extending the results of Mao et al. (2024i), who analyzed the gap in the context of learning with abstention (constant cost) for single expert and specific distribution.
Theorem 4.6 (Characterization of Minimizability Gaps). Assume R is symmetric and complete. Then, for the cross- entropy multiclass surrogates 4% and any distribution D, the following holds for v 2 0:
Moreover, we show that the surrogate deferral losses are (G, R)- conniistet whenever the underlying multiclass sur- rogate family D6 is R-consistent. Under the assumption that R- Rau and g = Gan the minimizability gaps van- ish, as established by Steinwart (2007). As a result, mini- mizing the surrogate deferral excess risk while accounting for the minimizability gap yields Eœy (rx) - E (Rai) + at
H
for =
|
|
==2
C. (R,x)= by
| 1|
- ||
V (1,2)
Mog (Ral) k-00 O. Since the multi-task model g is trained
k=
T
|
<> 2,
5

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
where: (By,q [ol Bni,,,,,,,,, [r. the aggregated
Define
costs are Tj= = -Kn-o C& lezj, and H denotes the Shannon
Emiul (w) == inf Eco (9w, h''S"
entropy. The minimizability gap is defined as Mazg(R)
E4p (R) Ex
[ (R, x))
A beads (W. h f) Eco (Bw, Emia
Drepr (wp)(w))= ;= Emin (u) - E (9).
The quantity Aeade measures head sub-optimality given the extracted representation from the encoder fixed at a particular iteration, while Arepr captures how far w lies from Payes-optime shared representation.
We provide the proof in Appendix D. Theorem 4.6 charac- terizes the minimizability gap Uzg (R)
(R) for cross-entropy multiclass surrogates over symmetric and complete hypoth- esis sets R. The gap depends on v 2 O, and its behavior varies across different surrogates. Specifically, for = 1, the gap is proportional the Shannon entropy of the nor- malized expected cost vector | which increases with entropy, reflecting higher uncertainty in the misclassification distribution. For
Lemma 4.8 (Non-negativity of the gaps), For all (w, h, f) W x H x F, have heads h, f) 2 O. For every w E W, we have Arepr 2 0.
the gap simplifies to the difference between the -1 normm and L -nom of F. where smaller gap indicates concentrated misclassifications thus reducing uncertainty. For E (1, the gap balances the entropy- based sensitivity at v= = and the margin- based sensitivity at V 2. As v + 1+, the gap emphasizes agents with higher misclassification counts, while as v + 2-, it shifts towards aggregate misclassification counts. For v < 1, where p 2ーツ E 1). the gap becomes more sensitive to miselassification distribution, increasing when errors are dispersed. For v > 2, with p < O, reciprocal weighting re- duces sensitivity to dominant errors, potentially decreasing the gap but at the risk of underemphasizing critical misclas- sifications.
Proof. Fix By definition of the infimum, Emin(w) infar, f Eco (1 I Eco (Sw,h.f) for any heads (h, f), hence Arrade (w. f) 2 0. For the representation gap, note that E (G) = inf , f Eeu (Awe, S Emin (w), so Arepr (w) = Emin (w) - - (g) 2 O. Both inequalities hold with equality when (w, h, f) is -oytemlp
Proposition 4.9 (Excess-risk decomposition). For every (w, h. f).
Eeo (gw.h.s) E (9) Aheade (w. f) + Arepr(w).
Proof. Add and subtract Emin
Proposition 4.10 (Cost of Enforcing a Shared Encoder).
Suppose two independent heads act directly on the raw input x: Isep. (x) (I(x), Let E Exp.co
In the special case of learning with abstention and single expert (J 1), assigning costs To 1 and TJ = recovers the minimizability gap introduced by Mao et al. (2024i). Thus, our formulation generalizes the minimizabil- ity gap to settings with multiple experts, non-constant costs, and arbitrary distributions D.
infn, Eep (9rep and define
AMTL E. (G)
cB _xppp00
Hence AMTL- 0 indicates that forcing shared encod- ing is beneficial, whereas AMTI. > 0 points to potential penalty relative to two stand-alone models.
4.3. Encoder-Aware Bounds
In this section, we show that our approach is theoretically aligned with multi-task learning using shared representa- tions. Let W denote a class of representation functions (encoders), H a class of classification heads, and J class of regression heads. For any (w, f) E W H x F. the multi-task predictor is defined as (x) = (h wxxx; Io w(x)). where the shared representation w((x) is passed to both task-specific heads. The true risk defined as Eeg (g) = Eg-D (x)(g(x), (y. t))], where 2 (x. E X' x y x T. The Bayes risk over a class G is given by cgg (G) = infgeg (g).
Combining definitions,
En, (gw.n.))) - (g) = [E.. EB sep.ro AMTI.
we can link these relationships with the main Theorem 4.4 that states that for any (g.r) E & x R. Setting g = Jw.h. and invoking Proposition 4.9 yields the encoder-aware con- sistency bound:
Corollary 4.11 (Encoder-aware (9, R) consistency). For any (w, h, e H Fx x R.
Ela (Bw.h.. c (9, R) + Ue (9, R) S F" (E (r) E. (R) + Umy (R)) + Aboada h, ()) + Drope (w) + Ueg (9).
Proposition 4.7 (Head and representation gaps.). Fix w E W and let G = {Sw J f : w E W, W' E H, f' F).
6

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
Corollary (4.11) decomposes the end-to-end excess de- ferral risk into three orthogonal sources: (i) the rejector optimisation error, (ii) the head sub-optimality, and (iii) the representation gap.
probability 1 - for ā E [0, that the following bound holds at the optimum:
log 1/8
(he f,r) S Een (h, + 2 (Cae) +
2K
The bound suggests a two-stage pipeline: (i) learn or select high-capacity representations to minimise Arpp as well as best heads for this representation, then (ii) optimise the rejector to tighten the remaining terms. The pipeline exactly mirrors proposed L.2 solution. Such decoupling is particularly attractive when [7] is large and feature sharing is essential for sample efficiency.
with
J Rx (Cdes) S 1 (X))) + >> 2(mmg.
2
j=1
max L (mg. t) + +2) (R),
4.4. Generalization Bound
with A(m}) v) 1 D(m3) # y) exp (- f D # y)) +
We aim to quantify the generalization capability of our system, considering both the complexity of the hypothe- sis space and the quality of the participating agents. To this end, we define the empirical optimal rejector 7 as the minimizer of the empirical generalization error:
RR RKiin /0)/2
We prove Theorem 4.13 in Appendix F. The terms Rx (H) and Rx(FF) denote the Rademacher complexities of the hypothesis class H and function class F, respectively, indi- cating that the generalization bounds depend on the com- plexity of the pre-trained model. The term M(m)) v) cap- tures the impact of each expert's classification enror on the learning bound. It includes an exponentially decay- ing factor. D(m #v) exp
K
TB arg min PER
(ae m, 2k);
(4)
K
k=1
where lae denotes the true deferral loss function. To char- acterize the system's 's generalization ability, we utilize the Rademacher complexity, which measures the expressive richness of a hypothesis class by evaluating its capacity to fit random noise (Bartlett & Mendelson, 2003; Mohri et al., 2012). The proof of Lemma 4.12 is provided in Appendix E.
[K((m which decreases
rapidly as the sample size K grows as the expert's error rate D(m) & v) declines (Mozannar & Sontag, 2020). This reflects the intuition that more accurate experts contribute less to the bound, improving overall generalization. Finally, the last term suggests that the generalization properties of our true deferral loss depend on the expert's regression performance.
Lemma 4.12. Let L1 be family of functions mapping X to (0, 11, and let L2 be a family of functions mapping X to 10. Define f = (h12 l E L1,12 E L2). Then, the empirical Rademacher complexity of for any sample S of size KI is bounded by:
5. Experiments
In this section, we present the performance improvements achieved by the proposed Learning-to- to-nine surrogate in a multi-task context. Specifically, we demonstrate that our ap- proach excels in object detection, a task where classification and regression components are inherently intertwined and cannot be delegated to separate agents, and where existing L2D methods encounter significant limitations. Further- more, we evaluate our approach on an Electronic Health Record task, jointly predicting mortality (classification) and length of stay (regression), comparing our results with Mao et al. (2023a; 2024h).
Rs(C) & Rs(L1) + (L2)
For simplicity, we assume costs cu(g(z),2) fo (h(z),)) + (xz()(x)),) and ())0(m)(4)))) co(m,(z), We assume the regression loss beg to be non-negative, bounded by L. and Lipschitz. Furthermore, we assume that mł. is drawn from the conditional distribution of the random variable Md givenparameters {X = 24, Y }, and that mf, is drawn from the conditional distribution of Mff given {X = T } We define the family of deferral loss functions as Laer (faet= x R x M 2 - TR. Under these
For each experiment, we report the mcan and standard devi- ation across four independent trials to account for variability in the results. All training and evaluation were conducted on an NVIDIA H100 GPU. We give our training algorithm in Appendix A. Additional figures and details are provided in Appendix G. To ensure reproducibility, we have made our implementation publicly available.
assumptions, we derive the generalization bounds for the binary setting as follows:
Theorem 4.13 (Learning bounds of the deferral loss). For any expert M,, any distribution 'D over Z, we have with
7

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
5.1. Object Detection Task
0.55
We evaluate our approach using the Pascal VOC dataset (Everingham et al., 2010). multi-object detection bench- mark. This is the first time such a multi-task problem has been explored within the L2D framework as previous L2D approaches require the classification and regression compo- nent to be independent (Mao et al., 2023a; 2024h).
0.5
Model
MAP
Expert 1
0.45
Expert 2
0.4
Allocaton Razio
Model
0.5
Expert
Expent2
Dataset and Metrics: The PASCAL Visual Object Classes (VOC) dataset (Everingham al., 2010) serves as widely recognized benchmark in computer vision for evaluating object detection models. It consists of annotated images spanning 20 object categories, showcasing diverse scenes with varying scales, occlusions, and lighting condi- tions. To assess object detection performance, we report the mean Average Precision (mAP), standard metrie in the field. Additionally, in the context of L2D, we report the allocation metric (All.,, which represents the ratio of allocated queries per agent.
0.2
0.4
92
Figure I. Performance comparison across different cost values 32 on Pascal VOC (Everingham et al., 2010). The table reports the mean Average Precision (mAP) and the allocation ratio for the model and two experts with mean and variance. We report these results in Appendix Table
and Expert 2 correctly predict the target (while the model does not). In such cases, Expert I is preferred due to its lower cost 31 < .32. Conversely, for instances x E X where Expert is accurate and Expert (along with the model) is incorrect, the system continues to select Expert 2, as f3 remains relatively low. For B2 0.2, the increasing cost differential between the experts shifts the balance in favor of Expert 1, enabling the system to achieve strong performance while minimizing overall costs.
Agents setting: We trained three distinct Faster R-CNN models (Ren al., 2016) to serve our agents, differen- tiated by their computational complexities. The smallest, characterized by GFLOPS = 12.2, represents our model E G with g = {g g(x)= = (/ o w(z), w(x)) w E W. h E H. f F}. The medium -izuud denoted as Expert 1, has a computational cost of GFL.OPS = 134.4, while the largest, Expert 2. operates at GFLOPS = 280.3. To account for the difference in complexity between Experts I and 2, we define the ratio R& 280.3/ 280.3/134.4 134.4 and set the query Rc.
This demonstrates that our approach effectively allocates queries among agents, thereby enhancing the overall per- formance of the system, even when the classification and regression tasks are interdependent.
cost for Expert 1 as 31 = R2/ This parameterization reflects the relative computational costs of querying experts. We define the agent costs as co(g(z),2) 2) = mAP(g(z), and C]e[J] (x), 2) mAP(m, (z)) 2). We report the per- formance metries of the agents alongside additional training details in Appendix G.I.
5.2. EHR Task
We compare our novel approach against existing two-stage L2D methods (Mao et al., 2023a; 2024h). Unlike the first experiment object detection (Subsection 5.1), where clas- sification and regression tasks are interdependent, this eval- uation focuses on second scenario where the two tasks can be treated independently.
Rejector: The rejector is trained using smaller version of the Faster R-CNN model (Ren et al., 2016). Training is per- formed for 200 epochs using the Adam optimizer (Kingma & Ba, 2017) with a learning rate of 0.001 and batch size of 64. The checkpoint achieving the lowest empirical risk on the validation set is selected for evaluation.
The Medical Information Mart for Intensive Care IV (MIMIC-IV) dataset (Johnson et al., 2023) is a comprehensive collection of de-identified health-related data patients admitted critical care units. For our analysis, we focus on two tasks: mortality prediction and length-of- stay prediction, corresponding classification and regres- sion tasks, respectively. To evaluate performance, report accuracy (Acc) for the mortality prediction task, which quan- tifies classification performance, and Smooth LI loss (sLI) for the length-of offytay prediction task, which measures the deviation between the predicted and actual values. Addi- tionally, we report the allocation metric (AlI.) for L.2D to
Dataset and Metrics:
In Figure 1, we observe that for lower cost val- ues, specifically when 31 < 0.15, the system consistently avoids selecting Expert This outcome arises because the cost difference between 31 and 32 is negligible, making it more advantageous to defer to Expert 2 (the most accurate expert), where the modest cost increase is offset by superior outcomes. When 3g = 0.15, however, it becomes optimal to defer to both experts and model at the same time. In particular, there exist instances x E X where both Expert
Results:
8

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
capture query allocation behavior.
2023a) and regression-focusee rejector (Mao et al., 2024h).
The results highlight the limitations of task rejectors and the advantages of our balanced approach.
Agents setting: We consider two experts, M, and M2.
acting specialized agents, aligning with the category al- location described in (Mozannar & Sontag, 2020; Verma et al., 2023; Verma & Nalisnick, 2022; Cao et al., 2024). The dataset is partitioned into Z = 6 clusters using the K-means algorithm (Lloyd. 1982). where Z is selected via the Elbow method (Thorndike, 1953). The clusters are de- noted as {Ci, C2 ...,, Cz). Each cluster represents sub- set of data instances grouped by feature similarity, enabling features- specific specialization by the experts. The experts are assumed to specialize in distinct subsets of clusters based on the task. For classification, Mj correctly predicts the out- comes for clusters Ca = {Ci, C22 C.), while M2 handles clusters C-M2 = {C1, Cs, C.}. Notably, cluster C1 is shared between the two experts, reflecting practical scenarios where domain knowledge overlaps. For regression tasks, M1 is accurate on clusters Cyty = [C1, C3. Cz}. while M2 spe- cializes in clusters CM2 = {C1, Ca. C&}. Here too, overlap is modeled, with cluster Ci being common to both experts and classification cetssfficcaionneggresstet regression task. Note that the category assignments do not follow any specific rule.
Rejector
Acc (9)
sL1
All. Model
All. Expert
All. Expent
Mao et al. (2023a)
71.3
1:45 4: 03
.00 .02
.01 2: 001
.9 d: 00
Mao (20241) Ours
50.7 + 8
1.18 4.05
38 01
33 4: 02
.25 4:
70.0 -5
1.29 .02
.66 .01
.124
Table I. Performance comparison of different two-stage L2D. The table reports accuracy (Acc), smooth LI loss (sLI), and allocation rates (All.) to the model and experts with mean and variance.
The classification-focused rejector achieves the highest clas- sification accuracy at 71.3% but struggles with regression, as reflected by its high smooth L.1 loss of 1.45. On the other hand, the regression-focused rejector achieves the best regression performance with an sLl loss of 1.18 but per- forms poorly in classification with an accuracy of 50. 50.7%. In contrast, our method balances performance across tasks, achieving classification accuracy of 70.0% and an sL1 loss of 1 1.28. Moreover, it significantly reduces reliance on experts, allocating 66% of queries to the model compared to 60% for Mao et al. (2023a) and 38% for Mao et al. (2024h). Expert involvement is minimized, with only 12% and 22% of queries allocated to Experts and 2, respectively.
ola
We assume that each expert produces correct predictions for the clusters they are assigned (Verma et al., 2023; Mozan- nar & Sontag, 2020). Conversely, for clusters outside their expertise. predictions are assumed to be incorrect. In such cases, for length-of-stay predictions, the outcomes are mod- eled using a uniform probability distribution to reflect uncer- tainty. The detailed performance evaluation of these agents is provided in Appendix G.2.
Since the experts possess distinct knowledge for the two tasks (CM, and CM, for M1). independently deferring clas- sification and regression may lead to suboptimal perfor- mance. In contrast, our approach models deferral decisions dependently, considering the interplay between the two com- ponents to achieve better overall results.
The model utilizes two compact transformer architectures (Vaswani et al., 2017) for addressing both classification and regression tasks, formally defined as g = {g : g(x) = (M(x), f())) | h E H. f E F). The agent's 's costs are spec- ified as cu(g(z), 2) X (h(z), v) + lo tt) and (Je(s) (m, 2)) = co(m)(r)) z) 3y. Consistent with prior works (Mozannar & Sontag, 2020; Verma al.,
6. Conclusion
We introduced a Two-Stage Searning-to-Deffe framework for multi-task problems, extending existing approaches to jointly handle classification and regression. We pro- posed a two-stage surrogate loss family that is both (G, R)- consistent and Bayes-consistent for any cross- pross-entropyy-aat surrogate. Additionally, we derived tight consistency bounds linked cross-entropy losses and the Lı-norm of aggregated costs. We further established novel minimizabil- ity gap for the two-stage setting, generalizing prior results to Learning-tor to--eeerr with multiple experts. Finally, we showed that our learning bounds improve with a richer hy- pothesis space and more confident experts.
2023; Mao et al., 2023a; 2024h), we set B3 = O.
The two-stage L2D rejectors are trained using a small transformer model (Vaswani et al., 2017) as the en- coder, following the approach outlined by Yang et al. (2023), with a classification head for query allocation. Training is performed over 100 epochs with a learning rate of 0.003, warm-up period of 0. a cosine learning rate scheduler, the Adam optimizer (Kingma& & Ba, 2017), and a batch size of 1024 for all baselines. The checkpoint with the lowest empirical risk on the validation set is selected for evaluation.
Rejectors:
We validated our framework on two challenging tasks: (i) object detection, where classification and regression inherently
seceddepennee-byyoonee the scope of existing L2D methods; and (ii) electronic health record analysis, where we demonstrated that current L2D approaches can be suboptimal even when classification and regression tasks are independent.
Results: Table 1 compares the performance of our pro- posed Learning-too to-ffer (Ours) approach with two exist- ing methods: a cewiifiiaatio-oocuseee rejector (Mao et al.,

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
Acknowledgment
the IEEE Conference on Com- puder Vision and Pattern Recognition, pp. 2911-2920, 2017.
posals. In Proceedings the
This research is supported by the National Research Founda- tion, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2022 Ph0--022-00-00-1-- and by A A*STAR, and is part of the programme DesCartes which is supported by the National Research Foundation, Prime Minister's s Of- fice, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.
Cao, Y., Cai, T., Feng. L.,, Gu, L., Gu, An, B., Niu, G., and Sugiyama, M. Generalizing consistent multi- class classification with rejection be compatible with arbitrary losses. In Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387110
Impact Statement
This paper advances the theoretical and practical understand- ing of machine learning, contributing to the development of more effective models and methods. While our research does not present any immediate or significant ethical con- cerns, we recognize the potential for indirect societal im- pacts.
Cao, Y., Mozannar, H., Feng, L.., Wei, H., and An, B. In defense of softmax parametrization for calibrated and con- sistent learning to defer. In Proceedings of the 37th Inter- national Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. In Computer Vision - ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I, pp. 213-229, Berlin, Heidelberg, 2020. Springer-Veriiee ISBN 978-3-030-58451-11 doi: 10. 10.1007799833-000--5445---3 13. URL https:tt////eo /doi. org/10. 1000/11000//0000 110-0000007770-0---------9--9
References
Awasthi, P., Mao, A., Mohri, M., and Zhong, Y. Multi-class h-consistency bounds. In Proceedings of the 36th Inter- national Conference on Neural Information Processing Systems, NIPS 22,, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Balagurunathan, Y., Mitchell, R., and El Naqa, I. Require- ments and reliability of AI in the medical context. Physica Medica, 83:72-78, 2021.
Charusaie, M.-A., Mozannar, H., Sontag, D., and Samadi, S. Sample efficient learning of predictors that comple- ment humans. In International Conference on Machine Learning, pp. 2972-3005. PMLR, 2022.
Bartlett, P.. Jordan, M., and McAuliffe, J. Convexity, classification, and risk bounds. Journal of the Ameri- can Statistical Association, 101:138-156, 2006. doi: 10.1 10.1198/01621455500000099999
Chow, C. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory, 16(1):41-46 2003.
Bartlett, P. L. and Mendelson, S. Rademacher and Gaussian complexities: Risk bounds and structural results. J. Mach. Learn. Res., 3(6ull):463- 482, March 2003. ISSN 1532- 4435.
Cortes, C. and Vapnik, V. N. Support-vector networks. Machine Learning, 20:273-297, 1995. URL https: / //api. saapiissmmaticsccceee wttpsschhhaaaa.rrooooooooooo org/CorppsII: 52874011.
Bartlett, P. L. and Wegkamp, M. H. Classification with reject option using hinge loss. Journal of Machine Learning Research, 9(8), 2008
Cortes, C., DeSalvo, G., and Mohri, M. Learning with rejec- tion. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, pp. 67-82. Springer, 2016.
Benz, N. L. C. and Rodriguez, M. G. Counterfactual in- ference of second opinions. In Uncertainty in Artificial Intelligence, pp. 453-463. PMLR, 2022.
Cortes, Mao, A., Mohri, C., Mohri, M., and Zhong, Y. Cardinality-awar set prediction and top-k classification. Advances neural information processing systems, 37: 18265-18309, 2024.
Berkson, J. Application of the logistic function to bio-assay. Journal of the American Statistical Association, 39:357-365, 1944. URL https: //api. www.nticschollsee sem.atticcccola....rr/oooo...oo. org/CorpusID: 122893121.
Cortes, C., Mao, A., Mohri, M., and Zhong, Y. Balancing the scales: A theoretical and algorithmic framework for learn- ing from imbalanced data. In Forty-second International Conference on Machine Learning. 2025. URL https: / /pppnrevv ew net forum? d=gscscNNi
Buch, S., Escorcia, V., Shen, C., Ghanem, B., and Car- los Niebles, J. SST: Single-stream temporal action pro-
10

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
Everingham, M., Gool, L., Williams, C. K., Winn, J., and Zisserman, A. 'he pascal visual object classes (voc) challenge. Int. J. Comput. Vision, 88(2): 303-338, June 2010. ISSN 0920-5691. doi: 10.1007/ sl $11263-000-027544 URL https: http//dd..oo/// //opi.or//1....///////// 1007/s11263 009 0275-4.
volume 34, pp. 4421-4434. Curran Associates, Inc.,
2021. URL httpss:/pproeeeddngg neurips.
cc/ hc//appee_.iilee/pppe/////////// 2332911ee877550177a111ad22222223 pdf.
Keswani, V., Lease, M., and Kenthapadi, K. Towards unbiased and accurate deferral to multiple experts. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES '21, pp. 154-165, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384735. URL
Geifman, Y. and Elnvv,,v, R. Selective classification for deep neural networks. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:/ neurs. cc cc/ fi 4a8 pdf.
doi: 10.1145/
3461702.3462516. 1145/3461702. 3462516...
Kingma, D. P. and Ba, J. Adam: A method for stochastic op- timization, 2017. URL https: /atpaaaaxxv.....aa.aaaa
Ghosh, A., Kumar, H., and Sastry, P. S. Robust loss functions under label noise for deep neural networks. ArXiv, abs/1712.09482, 2017. URL https://apl. semanticscholar: org prtg//orpusD::655772
1412.6980.
Liu, S., Cao. Y., Zhang, Q.. Feng, L., and An, B. Mitigating underfitting in learning to defer with consistent losses. In International Conference on Artificial Intelligence and Statistics, pp. 4816-4824 PMLR, 2024.
Girshick, R. Fast R-CNN. arXiv preprint arXiv: 2015.
Lloyd, S. Least squares quantization in PCM. IEEE Trans- actions on Information Theory, 28(2):129-337, 1982. doi: 10.1
Hemmer, P., Schemmer, M.. Vössing, M., and Küihl, N.
Human-Al Al complementarity in hybrid intelligence sys- tems: structured literature review. PACIS, pp. 78, 2021.
Long, P. and Servedio, R. Consistency versus realizable h-consistency for multiclass classification. In Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th In- ternational Conference on Machine Learning, number in Proceedings of Machine Learning Research, pp. 801- 809, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL. ht www.tttt////cc pttp://ep/oeeedeeeoeooooooooooe ings ml wwl..press/vee v longl3. hhml.
Hemmer, P., Schellhammer, S., Vössing, M., Jakubik, J., and Satzger, G. Forming effective human-Al AI teams: Building machine learning models that complement the capabilities of multiple experts. In Raedt, L. D. (ed.), Proceedings of the Thirty-First International Joint Conference on Artifi- cial Intelligence, IJCAI-22, pp. 2478-2484. International Joint Conferences on Artificial Intelligence Organization, 7 2022. doi: http:9/////ccai...222333// URL https: Track.
Madras, D., Pitassi, T., and Zemel, R. Predict responsibly: Improving fairness and accuracy by learning to defer. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 6150-6166 2018
/itp//////jjaaaaa . Main
Johnson, A. E., Pollard, T. J., Shen, I.., Lehman, L.-w. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., An- thony Celi, L., and Mark, R. G. MIMIC-III, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Mao, A. Theory and Algorithms for Learning with Multi- Class Abstention and Multi- Aaltri-Expert Deferral. PhD thesis, New York University, 2025.
Johnson, A. E., Bulgarelli, L.,, Shen, I.., Gayles, A., Sham- mout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B., et al. MIMIC-IV, a freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
Mao, A., Mohri, C., Mohri, M.. and Zhong, Y. Two-stage learning to defer with multiple experts. In Thirty- Conference on Neural Information Processing Systems, 2023a. URL https: /wtpr/eieeeeeeeeeeeeeeeeeee net/ forum? (d=GIlsHOT45)
Kerrigan, G., Smyth, P., and Steyvers, M. Combining human predictions with model probabilities via confusion matrices and calibration. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems,
Mao, A., Mohri, M., and Zhong, Y. Cross-entropy loss func- tions: Theoretical analysis and applications. In Proceed- ings of the 40th International Conference on Machine Learning, ICMI''3. 233, 2023b.

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
Mao, A., Mohri, M., and Zhong, Y. Predictor-rejector multi- class abstention: Theoretical analysis and algorithms. In International Conference on Algorithmic Learning Theory, pp. 922-867, PMLR, 2024a.
Mao, A., Mohri, M., and Zhong, Y. Mastering multiple-expen routing: Realizable ShS and strong guarantees for learning to defer. In Forty- second International Conference on Machine Learning, 2025b, URI. htt.tttp///// http:///ppeeeeveeeeeeeeeeeeeeeee net/ forum? Ad=2k1xjR61sd.
Mao, A., Mohri, M., and Zhong, Y. h-consistency guarantees for regression. In Salakhutdinov, R., Kolter, Z. Heller, K., Weller, A., Oliver, N., Scar- lett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp- 34712-34731 PMLR, 21-27 Jul 2024b. URL https: http:////oceeegeeeeeeee mlr. press/v235 http////mmmm2.hhmmm/
Mao, A., Mohri, M., and Zhong, Y. Principled algorithms for optimizing generalized metrics in binary classifica- tion. In Forty-second International Conference on Ma- chine Learming, 2025c. URL ttttps://openreeeee
net/
Mohri, M., Rostamizadeh, A., and Talwalkar, A. Founda- tions of Machine Learning. The MIT Press, 2012. ISBN 026201 825X.
Mao, A., Mohri, M., and Zhong. Y. Enhanced h--conistenccy bounds. arXiv preprint arXiv:2407.13722, 2024c.
Montreuil, Y., Yeo, S. H., Carlier, A., Ng. L. X., and Ooi, W. T. Optimal query allocation in extractive qa with llms: A learning-to-dele framework with theoretical guaran- tees, 2024.
Mao, A., Mohri, M., and Zhong, Y. H-consistency guaran- tees for regression. Proceedings of Machine Learning Research, 235:34712-34737, 2024d.
Mao, A., Mohri, M., and Zhong, Y. Multi-label learning with stronger consistency guarantees. Advances in neural information processing systems, 37:2378-2400 2024e.
Montreuil, Y., Carlier, A., Ng, L. X., and Ooi, W. T. Ad- versarial robustness in two-stage learning-to-defe Al- gorithms and guarantees. In Forty-second International Conference on Machine Learning, 2025a. URL https: / //openreview.. htteet/forrrmmm.dhh333d///000
Mao, A., Mohri, M., and Zhong, Y. Principled approaches for learning to defer with multiple experts. Interna- tional Workshop on Combinatorial Image Analysis, pp. 107-135. Springer, 2024f.
Montreuil, Y., Carlier, A., Ng, L. X., and Ooi, W. T. Why ask one when you can ask k?? two-stage saaming-to-defer to the top-k experts, 2025b.
Mao, A., Mohri, M., and Zhong, Y. Realizable $h$- consistent and bayes-consistent loss functions for learn- ing to defer. In The Tiirth- Annual Con- ference on Neural Information Processing Systems, 2024g. URL http:::///peeeeeeeewe wwt//foruuurr id=Oco2XakUUK.
Montreuil, Y., Carlier, A., Ng. L. X., and Ooi, W. One- stage top-k learning-to-defer: Score-based surrogates with theoretical guarantees, 2025c.
Mozannar, H. and Sontag, D. Consistent estimators for learn- ing to defer to an expert. In Proceedings of the 37th In- ternational Conference on Machine Learning, ICML'20, 2020.
Mao, A., Mohri, M., and Zhong, Y. Regression with multi- expert deferral. Forty-first International Conference on Machine Learning. 2024h.
Mozannar, H., Lang, H., Wei, D., Sattigeri, P., Das, S., and Sontag, D. A. Who should predict? exact algo- rithms for learning to defer to humans. In Interna- tional Conference on Artificial Intelligence and Statistics, 2023. URL https: /ppi. semanticscholas org/ Co:
Mao, A., Mohri, M., and Zhong, Y. Theoretically grounded loss functions and algorithms for score-based multi-class abstention. In International Conference on Artificial In- telligence and Statistics, pp. 47611. PMLR, 2024i.
Mao, A., Mohri, M., and Zhong, Y. A universal growth rate for learning with smooth surrogate losses. Advances in neural information processing systems, 37:41670-41708, 2024j.
Narasimhan, H., Jitkrittum, W., Menon, A. K., Rawat, A., and Kumar, S. Post-hoc estimators for learning to defer to an expert. Advances in Neural Information Processing Systems, 35:29292-29304, 2022.
Mao, A., Mohri, M., and Zhong. Y. Enhanced $h$- consistency bounds. In 36th International Conference on Algorithmic Learning Theory, 2025a. URL ht tttp:: //openreview ner/forum? dd=ggVGGF JMJJ..
Okati, N., De, A., and Rodriguez, M. Differentiable learning under triage. Advances in Neural Information Processing Systems, 34:9140-915 2021.
12

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
Palomba, F., Pugnana, A., Álvarez, J. M., and Ruggieri, S. A causal framework for evaluating deferring systems. CoRR, 2024.
Verma, R. and Nalisnick, E. Calibrated learning defer with one-vs-all classifiers. In International Conference Machine Learning, pp. 22184-22202. PMLR, 2022.
Ramaswamy, G., Tewari, A., and Agarwal, S. Consistent algorithms for multiclass classification with an abstain option. Electronic Journal of Staristies, 12(1):530 554, 2018. doi: 10.1214/17-EIS1388 URL htt.t:/t///// /doi. org/10 100.1111117111110 /17--.551388
Verma, R. Barrejon, D., and Nalisnick, Learning to defer multiple experts: Consistent surrogate losses, confi- dence calibration, and conformal ensembles. In Interna- tional Conference on Artificial Intelligence and Statistics, 2023. URL https: api. org Perg/CorpuaTD: 25333748..
Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. You only look once: Unified, real-time object detection. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 779-788, 2016.
Wei, Z.., Cao, Y., and Feng, L. Exploiting human-ai depen- dence for learning to defer. In Forty-first International Conference on Machine Learning, 2024.
Yang. C., Wu, Z., Jiang, P., Lin, Z., Gao, J., Danek, B., and Sun, J. PyHealth: A deep learning toolkit for healthcare predictive modeling. In Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discov- ery and Data Mining (KDD) 2023, 2023. URL https: / //github. com/ sunnsunnaaa sunlabuiuc/ByHeee.
Ren, S., He, K., Girshick, R., and Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks, 2016. URL pestps///arxin org/ /bs/ 1506. 01497.
Steinwart, How to compare different loss functions and their risks. Constructive Approximation, 26:225-287, 287, 2007. URL https:/ /ttp///apiia/ semanticscholee 16660598.
Zhang, M. and Agarwal, S. Bayes consistency vs. h-consistency: The interplay between surrogate loss func- tions and the scoring function class. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 16927-16936. Curran Associates, Inc., 2020, URL htwps: https://proceedd.g.. neurips.. cc/ cc/paaperrfftles/pppp///0000 oa pdf.
Tailor, D., Patra, A., Verma, R., Manggala, P., and Nal- isnick, E. Learning defer population: A meta-learning approach. In Dasgupta, S., Mandt, S., and Li, Y. (eds.), Proceedings of The 27th Interna- tional Conference on Artificial Intellignnce and Statis- tics, volume 238 of Proceedings of Machine Learn- ing Research, pp. 3475-3483, PMLR, 02-04 May 2024. URL StRh:tts:: /ttpp/eeeoocc..seeeeeeee mlr press/ v238/tailor244 html.
Zhang, T. Statistical behavior and consistency of classifica- tion methods based on convex risk minimization. Annals of Statistics, 32, 12 2002. doi: 10.1214/as/100911211
Tewari, A. and Bartlett, P. L. On the consistency of multi- class classification methods. Journal of Machine Learn- ing Research, 8336):1007-1025, 2007. URL http: //jmlr. org/ wttpppppp.ptttaariiiipp
Zhong, Y. Fundamental Novel Consistency Theory: H-
Consistency Bounds. New York University, 2025.
Thomdike, R. L. Who belongs in the family? Psy- chometrika, 18:267-276, 1953. URL https:
//api. www.ntiisccoolii.t.o org/CorpusID: 120467216.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, u. and Polosukhin, Attention is you need. Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan. S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL
neurips. ce/paper pptpper_filee/aaperr//0/////// 3 Paper. pdf.
13

---

A Two-Stage Learning-to-Defer Approach for Mask Learning
A. Algorithm
Algorithm Two-Stage Leafing for Multi-Task Learning Algorithm
Input: Dataset { Vk, | //mm multi-task model g G, experts E M., rejector R. number of epochs EPOCH, batch size B, learning rate 7/-
Initialization: Initialize rejector parameters 0.
for i = to EPOCH do
Shuffle dataset ((rk, Mk 1+))==1
for each mini-batch
((rk, yk, k=) of size B do
Extract input-output pairs z (z, y, E B.
Query model (xx) and experts m(x). Evaluate costs co (xx) =) and ()>)(m(a))
{Agents pre-trained and fixed) {Compute task-specific costs)
Compute rejector prediction r(z) = arg maxjEA j)
{Rejector decision}
Compute surrogate deferral empirical risk Eog.
Exss B E. EB (g. m. m,
(Empirical risk computation)
Update parameters 0 using gradient descent:
0 - 8
Parameterr update}
end for end for Return: trained rejector model r*.
We will prove key lemmas and theorems stated in our main paper.
B. Proof of Lemma 4.2
We aim to prove Lemma 4.2, which establishes the optimal deferral decision by minimizing the conditional risk.
By definition, the Bayes-optimal rejector B (xr) minimizes the conditional risk Cis given by:
Cisa (g, r, = (eveeeelaaer( m, =)].
(5)
Expanding the expectation, obtain:
Claa (g. r, x;) = Ey
Σ m())
(6)
Using the linearity of expectation, this simplifies to:
Ciar Cen (gr,,r))
Σ Bysq (e,0()),m,(x)))) (
(7)
Since we seek the rejector that minimizes the expected loss, the Bayes-conditional risk is given by:
inf gEG,rER (G,
Eg. m,,,,))
(8)
Rewriting this expression, obtain:
C (g, R, .)) inf Ey inf co(g()), + (t)
(9)
rER
gEg
1=1
This leads to the following minimization problem:
cB R, = min
inf Ey,t (co(g(x), 2)] min Byato (2(m,(x)) je(M]
(10)
oeg
14

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
To simplify notation, we define:
inf (y.cl=lco(a()) x)} E yy (m, (x).,))) 2)},
j =0 0, otherwise.
(11)
Thus, the cayesiconii risk simplifies to:
Cl (9, R
min S. jEA
(12)
Since the rejector selects the decision with the lowest expected cost, the optimal rejector is given by:
0, if inf (=,) min (1))))) =)],
(r)
gEG
jEll]
(13)
J.
otherwise.
This completes the proof.
C. Proof Theorem 4.4
Before proving the desired Theorem 4.4, will use the following Lemma C.1 (Awasthi al., 2022; Mao et al., 2024h): Lemma C.1 (R-consistency bound). Assume that the following R-consistency bounds holds for E R, and any distribution
Eigs (r) Eto (R) + Ukn (R) I (E (r) Eøg, (R) + Log (R))
for p E (Po... PJ) A and X, get
J inf Σ rER j=0
Σ
Ps P,
P S S
inf FER
Σ pj p (r, x, j=0
1=0
j=0
Theorem 4.4 c)-ossstene bounds). Let g E 9 be multi-task model. Suppose there exists non-decreasing function I + R,, parameterized by 0, such that the R-consistency bound holds for any distribution D:
Eog, (r) E4g, (R) Uog (R)
I (Er. (r) Egs (R) + Ui (R)).
then any (g,r) E G x R. any distribution D. and any X,
Eg BB (g, R) + Uea (9. R) S
F" (Eag (r) Egg ( + Uay (R)
+ Ecg - (g) Uan (g),
where the expected aggregated cost vector is given by T = (Ey.4 [To]. (y,.(alTal), and (u) = I" T
with (u) (w). In the case of the log-soffmud surrogate (v = 1), the transformation is given by Tw=l (u) 1t log(1 + u) - log(1 - -u).
2
Proof. Let denote cost for j E A (O......
infgeg (co 2)] jf 0 [e =)]
otherwise
Using the change of variables and the Bayes-conditional risk introduced the proof of Lemma 4.2 Appendix B, we have:
CB (g, x) min 2 jEA
J
(14)
Ciar (g. r,)) Σ
Ey,c (g(x)) ()((z))m(x))))..).) 1m))=
1=0
15

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
We follow suit for our surrogate &ert and derive its conditional risk and optimal conditional risk.
Co = -By,tta
Σ
x,, +
co(g((), + (m,, Bir (r,
j=1
C = inf Bygit 10
Σ ==1
(r 0) + [loo((()); + Σ (1(m( (r, (())
rER
Let us define the function (1))))) 2) = minzelJi (z), where (2) denotes the model's output and 23 represents the corresponding cost function. Using this definition, the calibration gap is formulated as ACtar == Clae - C less where Céas
represents the original calibration term and Ch denotes the baseline calibration term. By construction, the calibration gap satisfies ACr. 2 0, leveraging the risks derived in the preceding analysis.
ACes Ey ((((()) +
Σ ()) 33) [m()==s) 33 j=1
=x(m(x),)) -
min min
jeA
Let us consider ACtes A₁ + A2, such that:
A y/ (r())=0((()))) + Σ (,)== (p(m) +B)]] =(maa))
(15)
A2 = (v(m(z)) min jEA
min (())
By considering the properties of min, we also get the following incquality:
u(m()), min (g(x)) Exsselco(((s)) a)] inf Bo..ilcooo(()),)) =)]
(16)
jEA
9Eg
implying,
AC A =) -()((())))
(17)
now select distribution for our rejector. We first define Vj E A.
Po (m)
and
Pję[J] = (o(g(z),2) + Ejp (m,
JE T (g(x) m (z))
which can also be written as:
T Pj =
(18)
7
Injecting the new distribution, we obtain the following:
AC ||
(
pj 1r
(r,
inf Σ p, (r,,,,)))) FER
(19)
1=0
1=0
16

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
Now consider the first and last term of ACtar. Following the intermediate step for Lemma 4.3, have:
A = =o
+
E (uttles((mm())))))llr0 =x(m(x),
- +
FER [o =)11
inf
Σ
E (m
(2)) +
Σ (((((()) + Σ C)(mm)( ((())
inf
Σ
(z), + (o(o({),,) 2)
Σ
(my(m) | (1(0)))))))))
FER
==1
1F8
Then, applying a change of variables to introduce || we get:
|
+ |F Σ
P)-
inf [|
|| Σ
Py 1o077
rER
j=1
j=1
|
P()))))
inf
|i
Σ f=0
Pj [{)))t)
FR
1=0
We now apply Lemma C.I introduce r,
Σ Pi ( - inf
L
pý ( ST
Σ
p. (r,x,))
inf FER
Σ pj th (r, (r,,,
rER
==
j=0
3=0
j=0
Σ 7, (()))
inf FER 1=0
Σ Ty (m)) ST
Σ
7y 2h (r, rER
inf
Σ T, D (r, (r,
(20)
T
|
j=0
j=0
j=0
ACosr
ACCm $ T
|
We reintroduce the coefficient A2 such that:
ACous
ACter < || [|]]]]]]]]]
A2
|
ACo
ACtm < |
+ (y.c)e/o(((())) 2)] inf E
(c)())()(((()))))) 2)] (upper bounding with Eq 16)
|
geg
Mao et al. (2023b) introduced tight bound for the comp-sum surrogates family. It follows for 2 0 the inverse transformation (u) = (-):
(1+0) 2x +11-)) 2n
3-6
2¹ 1-
v vE (0
1 I-
14:1
log[1 + w) + 1-t logl - v]
==1
(*())=
(0- 1ln I
(
(1+t) 20
+(1-r)
2
2-
2-v
-1
v (1,2)
(w- 10=1 F
[2, +20)
17

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
We note (u)= || ( |illi ). By applying Jensen's Inequality and taking expectation both sides, get
(g r) - EEea (G, (G.R) + (G, R) r (E E (R) U8 (R) + (g) E., (g) Heg (9)
D. Proof Theorem 4.6
Theorem 4.6 (Characterizatioon of Minimizability Gaps). Assume R. is symmetric and complete. Then, for the cross-entropy multiclass surrogates Di and any distribution D. the following holds for 0:
|
for
T
|
|
== 2
Ca (R,
T
11
V E (1,2)
Ek=
v > 2,
where F = [Bo,st/[Tol..... F. the aggregated costs are Tj = -k= Ck and H denotes the Shannon entropy. (R)
k=
The minimizability gap is defined as (R) = Egg (R) - Ex Ch.. (R,
(((i))) Proof. We define the softmax distribution 8j ΣyeA erlanf where sj E [0,1]. Let = F)(9(z)) m(x), 2) with
Ty E Rt and denote the expected value as r = By.t/ [T]. We now derive the conditional risk for given 0:
Cha
Σ
E. 1l (Ty] Po1 (r,
1=0
Σラ T
E (Epee 1-v
v # 1 -1 -1]
(21)
Σラ=o F, log
E e
==1
I- Σラ= Tj 89 -1 1]
#
Σ/ F, log(s,)
v ==1
For v= = we can write the following conditional risk:
J
Caar (r, x)) = Σ Ta (((()))) - -log Σ cr(z))))
(22)
1=0
JEEA
Then,
aC
(1,z) = =
(23)
ar(z,
1=
At the optimum, we have:
F,
s²(,,)
(24)
Σ₁= 7,
Then, it follows:
CaP==1 (R,
Σ T, log
F,
(25)
Σy Tj¹
1=0
18

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
As the softmax parametrization is distribution s* A write this conditional in terms of entropy with == (Tyheed:
(R,
Tk
83
log(s})
J
7
(26)
H
E Tj
k=0
=
|
(as E IR.
For # 1,2: The softmax parametrization can be written as constraint Σÿ 8j 1 aadd sj 2 0. Consider the objective
(
Fo 83 1}
(27)
We aim to find s" (6i.....5) that minimizes (27) subject to Σラ = Introduce a Lagrange multiplier À for the normalization Σド= $j = 1. The Lagrangian is:
[(s, A) =
Σ T
1| + (1 Σ 8)) 1=
(28)
1=0
We take partial derivatives with respect to s;:
0C
Fi (v 1))8 1-2 - x = 0.
(29)
3s;
1 -
Since 1-1 -1, we get
1-v
T 1-2
0
8,
v-2
a
for some a >
(30)
7i
Hence
8,
a 2-2
(31)
Summing s; over {i oJ....J} and setting the total to 1 yields:
w- 1.
(32)
T=U
Let
Q
(33)
k= ( 4-2
k=0
Therefore, for each i,
T
: =
(34)
Tk
This { is valid probability distribution. Let
A =
Tk
2-v
(35)
km0
19

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
Then the optimum distribution is
(36)
A
Recall
J
2(s) =
1
3
-1 1
(37)
V j=0
At 85,, we have
w-1
2-ル
7j 2-1
(8))
(38)
A4-1
Hence
j
J
Σ ==
Ti (5g) r-1 = A
Σ [j
2-v
A
1
Σ
T
2-v
A4-1 =
A
2
(39)
j=0
1=0
Substituting back,
J
C* (R,
bas
(R, =
1 V
Tk
2- v
Σ
(40)
j=0
We can express this conditional risk with a valid ('zer) norm as long as v E (1,
|
|
-
(41)
For 2: Since Σi=o T,- = S, we have
J
J
C (r,
Σ T3 [1-()(r)]
T, -T,-
Σ j=0
Tj (r()..
(42)
1=0
j=
Hence
FE Ch (r,x)
inf
S
sup
Σ
Fy sr()
(43)
ER
j=0
Therefore, minimizing ( x) is equivalent to maximizing
F(r) = s.)()))
(44)
1=
Its partial derivative w.r.t. ra is the standard softmax derivative:
0s; Ōr;
8i (1 -t),
if=
83 (ōis - -si)
(45)
83 8i;
otherwise.
Hence, for each i,
OF
J
Osj
J
Σ Ti Ør:
= 7, (1 10--s) 81) +
Σ
(46)
3r; 1=0
j=0
Factor out s;
0F
si
81) (1))))
(1())
Σ T,
8i
Ti
T, 8
(47)
Ör;
iti
==
20

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
because T5
Ty8s
T184. Define P(r) Σÿ= Ty (58((r) Then:
OF
T, F(r))-
(48)
Ōr.
Setting Sr,
0 for each implies
[T, - F(r)) 0, Vi.
(49)
Thus, for each index
8, = 0 or 7: F(r).
(50)
To maximize F(r), notice that:
If Ti+ is strictly the largest among all Tx, then the maximum is approached by making 8;- R so F(r) T;., In the softmax parameterization, this occurs in the limit +o0 and Tı -00 for k * i"
If there is tie for the largest 7;, we can put mass on those coordinates that share the maximum value. In any case, the supremum is max; Tx.
Hence
sup F(r) = max T/-
(51)
rER
0S15J
Because 2 -F(r.
inf
Cu=2 C (r)
sup F(r) rE
Σ F, j=0
max T, = -| - -||
(52)
rE
tEA
Hence the global minimum of C || limit as one coordinate rı- goes to +00 and all others go -00. No finite r yields an exactly one-hot (r) but the limit is enough to achieve the infimum arbitrarily closely.
||
the "softmax" parameterization, this only approached in the
It follows for F (Fyhia and 2 0:
H
|||||
V
|||||
|
==
inf CV (r, FER
(53)
|
(1.2)
2- 2-v
J E
Tx
2-v
-|7|11
otherwise
Building on this, we can infer the minimizability gap:
Us (R) EE,n(R) - Ey Enlinf
inf Ch Co (r, x) neR.
(54)
E. Proof Lemma 4.12
Lemma 4.12. Let La be family of functions mapping X to (0, 1), and let La be a family of functions mapping X' (0, 1}. Define L = (412 l E Li,12 E C2). Then, the empirical Rademacher complexity of C for any sample S of size K is bounded by:
Rs(C) s Rs(Lr) + Rt)
21

---

A Two-Stage Learning-to-Defer Approach for Multi-Tass Task Learning
Proof. We define the function & follows:
L1 C2 1 + l2
L1 (11 + l2 1)+
-
(55)
Here, 4 Li and 12 E L2 The function & is I I-Lipschitz as we have + (t 1)+ for t = 4 + 12. Furthermore, given that & is surjective and I-Lipschitz, by Talagrand's lemma (Mohri et al., 2012), we have:
Rs(v(LL L2)) Rs(Li + L2) S Rs + Rs((2)
(56)
This inequality shows that the Rademacher complexity of the sum of the losses is bounded by the sum of their individual complexities.
F. Proof Theorem 4.13
Theorem 4.13 (Learning bounds of the deferral loss). For any expert Mj. any distribution D over Z, we have with probability 1 - 5 for 8 E [0, 1/2], that the following bound holds at the optimun:
log 1/8
Eesy (h, f.r) S Eea (h, f.r) + 28 +
2K
with
Rx (Car) S 2 Bx(()) + Rx( + Σ Amm3. v)
+
Σ max Eegg (m/, 1) + +2) 2 Rx (R),
with A(m3. v) (mm, # y) exp (- K xo D(m)) + v)) + R RxDim3 (1))
Proof. We are interested in finding the generalization of (g.r) E L:
K Rs(C) = I ssup Σ
ou
laaala r, DR. yu. bz, ml)1
K
gEt k=1
1 Ee /sup SER k=1
Σ TN
K
1=0
K
K
<
Eeg
sup
JRcO (())==))))
Eg sup FER
Tkey [kc))
(By the subadditivity of sup)
K
gEL.
K
==1
Let's consider j =
1
K
E
sup G
03co-P(z4)=0)
J&co
- E,
sup DEC
Th Tk(Ln(s))z +
K
K
E
sup
σh 1 Ny l(())==
sup Σ rn (e)
K
gec k=1
k=1
MR (H) + Rx (R) + (F) + Rx (Ri) Rx(R
(using Lemma 4.12)
2
Rx (7t) + (F) + (2xx(R)
2
(57)
22

---

A Two-Stage Learning-to-Defer Approach for Mask Learning
Let's consider j > 0:
K
1
B
2 K
Eg sup PER
oh mm #v (r(2)=s)
K
rc
R
km
(58)
K
Eg
sup TER km
The onl bn) |e(e))=)]
j=1
Using barning-bounds for single expert in classification (Mozannar & Sontag, 2020), we have:
K
D(m +D)
cxp (- KDDmm' + v)
sup K
E
ok my [r(ww==
RKD(m)
(59)
PER
2
8
Applying it our case:
D(m) + v)
KD(m)) + y)
K
E E.
sup rER
o #v ty-(())) $
exp
RKD(m)
(60)
For the last term,
K
E sup
Σ FER
a bn) (((e)=s) (r)
$
Σ
max Crog maax (ms, 1) (R)) (R)
(61)
K
j=1
j=1
Then, leads to:
RK (Laer) S 1 [H) + Rx((F) Σ M(m3 y) +
Σ max log (m2, 1) + +2) 2) R (R)
D(m 4v)
KD(m, #2)
with 2(m/) y)
exp
+ R 4u)
G. Experiments
G.1. PascalVOC Experiment
Since an image may contain multiple objects, our deferral rule is applied at the level of the entire image æ E X, ensuring that the approach remains consistent with real-world scenarios.
Model
M
M2
mAP
39.5
43.3
52.8
Table Agent accuracies on CIFAR- validation set. Since training validation sets are pre-determined this dataset, the agents' knowledge remains fixed throughout the evaluation.
Cost Bg
mAP (%)
Model Allocution (%)
Expert Allocation (%)
Expert Allocation (%)
0.01
$2.8 = 0.0
0.0 + 0.0
0.0 - 0.0
100.0 + 0.0
0.05
52,5 : 0.1
7.3 0.8
0.0 0.0
92.7 1 0.3
0.1
49.1 : 0.6
48.0 0.7
0.0
$2.0 4 0.2
0.15
44,2 : 0.4
68.1 0.3
19.7 1 0.4
12,2 - 0.1
0.2
42.0 = 0.2
77.5 1:5=00 0.2
22,5 - 0.5
0.0 0.0
0.3
40.1 = 0.2
98.1 = 0.0
1,9 = 0.1
0.0 - 0.0
0.5
39,5 = 0.0
100.0 + 0.0
0.0 + 0.0
0.0 - 0.0
Table 3. Detailed results across different cost values 8g. Errors represent the standard deviation over multiple runs.
23

---

A Two-Stage Learning-to-Defer Approach for Multi-Task Learning
G.2. MIMIC-IV Experiments
MIMIC-IV (Johnson et al., 2023) is large collection of de-identified health-related data covering over forty thousand patients who stayed in critical care units. This dataset includes wide variety of information, such as demographic details, vital signs, laboratory test results, medications, and procedures. For analysis, focus specifically on features related to procedures, which correspond to medical procedures performed during hospital visits, and diagnoses received by the patients.
Using these features, we address two predictive tasks; (I) a classification task to predict whether a patient will die during their next hospital visit based on clinical information from the current visit, and (2) regression task to estimate the length of stay for the current hospital visit based on the same clinical information.
A key challenge in this task is the severe class imbalance, particularly in predicting mortality. To mitigate this issue, sub-sample the negative mortality class, retaining a balanced dataset with K = 5995 samples, comprising 48 48.2% positive mortality cases and 51.8% negative mortality cases. Our model is trained on 80% of this dataset, while the remaining 20% is held out for validation. To ensure consistency in the results, we fixed the training and validation partitions.
Model
M
M2
Accuracy Smooth L1
60.0
39.7
46.2
1.45
2.31
1.92
Table 4. Performance of the agents the MIMIC-IV dataset, evaluated in terms of accuracy and Smooth LI loss. We fixed the trainiog/validationn set such that agents' knowledge remains fixed throughout the evaluation.
24


Reviews:
[{"review_id": "0wXrtmlmZU", "reviewer": "Reviewer_oatj", "summary": "In this paper, the authors analyze the learning-to-differ (L2D) problem in the two-staged multi-task (classification and regression) setting. The paper introduces the pointwise Bayes rejector for the mult-task deferral and introduces a surrogate differal loss that is Bayes consistent. The paper further provides generalization guarantees for the learned rejector, which provide insights into the conditions of the problem setting that can lead to better generalization. Numerical experiments are provided to validate the proposed method, and a comparison of the proposed multi-task rejector with prior single-task rejectors shows that the proposed method can have a balanced trade-off between the tasks compared to prior methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and thoughtful feedback. We are glad that they found our theoretical contributions novel, the paper well-motivated, and the analysis of the two-stage multi-task Learning-to-Defer setting valuable to the literature. Please find some clarification below:\n\n> It is unclear how the coefficients $\\lambda^{cla}$ and $\\lambda^{reg}$ [...] balanced performance described in the discussion.\n\n> Given the possibly different scales of classification and regression losses, more discussion seems to be needed regarding designing a balanced/meaningful rejector in this setting.\n\nWe agree that the choice of the coefficients $\\lambda^{cla}$ and $\\lambda^{reg}$ is crucial for achieving balanced and meaningful performance. In our experiments, we considered both classification and regression tasks equally important; thus, we set $\\lambda^{cla} = \\lambda^{reg} = 1$. This choice represents a balanced and task-agnostic baseline, ensuring no implicit bias toward either component. In practical scenarios, these coefficients should indeed be tuned according to task-specific priorities and performance requirements. **For instance, if the classification is more important in this setting, we should set $\\lambda^{cla}>\\lambda^{reg}$ to prioritize agent with more proficiency on the classification task**. We will explicitly discuss this and highlight the importance of tuning $\\lambda^{cla}$ and $\\lambda^{reg}$ in the revised manuscript.\n\n> It is unclear how the rejection is implemented for the prior methods [...] or only the classification task?\n\nIn the EHR task, we treat classification and regression as independent but simultaneously allocated tasks. Specifically, when the classification rejector (as described in [1]) is triggered, we allocate both the classification and regression queries jointly to the selected agent. Similarly, when the regression rejector is triggered, it also allocates both tasks together [2]. We will clarify this joint-allocation implementation explicitly in the revised manuscript. \n\n>The definition of the deferral function class (column 1 line 319) to be a mapping to $[0,1]$ seems unrealistic.\n\n> It seems like in proof of Lemma 4.2, $\\mathcal{C}\\_{\\ell\\_{def}}^B$ should not depend on $g,r$.\n\nThank you for pointing this out. You are correct—these are typos and do not affect the correctness of the proofs. We will fix them in the revised manuscript.\n\n> $\\overline{\\tau}$ is not defined in Theorem 4.4, and it is not clear where $T^{1-\\nu}$ appear in the Theorem.\n\nIndeed, $\\overline{\\tau}\\_j$ represents the expected aggregated cost of agent $j \\in \\mathcal{A}$, formally defined as $\\overline{\\tau}\\_j = \\mathbb{E}_\\{y,t|x}[\\tau\\_j]$. Consequently, we define the vector $\\overline{\\boldsymbol{\\tau}} = (\\overline{\\tau}\\_0, \\overline{\\tau}\\_1, \\dots, \\overline{\\tau}\\_J)$ and $||\\boldsymbol{\\overline{\\tau}}||\\_1$ its L1 norm.   \n \nThe transformation function $\\mathcal{T}^{\\nu}(u)$ is introduced to determine the function $\\Gamma^{\\nu}(u)$ corresponding to a given multiclass surrogate loss $\\Phi^\\nu\\_{01}$. **We explicitly show that the consistency bounds of the deferral loss (Lemma 4.3) inherently depends on the consistency bounds of the multiclass surrogate loss** $\\Phi^\\nu_{01}$ from the comp-sum family (Equation 1). For example, choosing the log-softmax surrogate corresponds to setting $\\nu = 1$. In this case, the relevant transformation function is explicitly given by $\\mathcal{T}^1(u)=\\frac{1+u}{2} \\log[1+u] + \\frac{1-u}{2} \\log[1-u]$ [8]. Then, by taking the inverse of this transformation, we obtain the function $\\Gamma^\\nu(u)=(\\mathcal{T}^\\nu)^{-1}(u)$ leading to a bound with the function  $\\overline{\\Gamma}^\\nu(u)=||\\overline{\\boldsymbol{\\tau}}||\\_1 \\Gamma^\\nu(\\frac{u}{||\\overline{\\boldsymbol{\\tau}}||\\_1})$ depending on both the surrogate transformation $\\mathcal{T}^{\\nu}(u)$ and the L1 norm $||\\overline{\\boldsymbol{\\tau}}||\\_1$. \n    \nWe will explicitly include these definitions and clarifications in the revised manuscript.\n\n> The clusters $C_{cla}^{M_1}, C_{cla}^{M_2}$ defined in Experiments in 5.2 seem not distinct as described.\n\nWe confirm that clusters $C\\_{cla}^{M_1}$ and $C\\_{cla}^{M_2}$ were explicitly constructed to represent distinct regions of expertise, although they do indeed exhibit some overlap. Upon review, we acknowledge a minor inconsistency in the original manuscript regarding their definitions (but not on experiments). The correct cluster assignments are $C^{M\\_1}\\_{cla} = \\lbrace C\\_1, C\\_2, C\\_4\\rbrace$ and $C^{M\\_2}\\_{cla} = \\lbrace C\\_1, C\\_5, C\\_6 \\rbrace$. We will correct this in the revised manuscript.\n\n### References \n[1] Mao, et al. (2023). Two-Stage Learning to Defer with Multiple Experts. NeurIPS23\n\n[2] Mao, et al. (2024). Regression with multi-expert deferral. NeurIPS24\n\n[8] Mao et al. (2023). Cross-entropy loss functions: theoretical analysis and applications. ICML23", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Did not check all the proofs, minor typos in the proof of Lemma 4.2 (mentioned in Other Comments Or Suggestions).", "experimental_designs_or_analyses": "* It is unclear how the coefficients $\\lambda^\\text{cla}, \\lambda^\\text{reg}$ used in the agents' cost were determined. These coefficients seem crucial in attaining the balanced performance described in the discussion.\n* It is unclear how the rejection is implemented for the prior methods in the experiments in Section 5.2 in the multi-task setting. For example, when the classification rejector is triggered, does the model consult for the expertise on both classification and regression tasks from the expert, or only the classification task?", "relation_to_broader_scientific_literature": "This paper extends the literature in the area of L2D, by providing theoretical insights into L2D in the two-staged multi-task setting.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths\n\n* The paper provides theoretical insights into L2D in the two-staged multi-task setting, which seems novel in the related literature.\n* The paper is mostly self-contained and easy to follow\n* The problem is well-motivated, and the theoretical insights are sufficiently discussed.\n\nWeaknesses\n\n* The definition of the deferral function class $L_{def}$ (column 1 line 319) to be a mapping to [0,1] seems unrealistic, given that the $\\ell_{def}$ is made up of $c_0$, which in turn is assumed to be a summation of $\\ell_{01}$ and $\\ell_{reg}$. Since $\\ell_{reg}$ is not necessarily bounded between 0 and 1, it is unclear why the definition of $L_{def}$ is correct.\n* Given the possibly different scales of classification and regression losses, more discussion seems to be needed regarding designing a balanced/meaningful rejector in this setting.", "other_comments_or_suggestions": "* It seems like in proof of Lemma 4.2, $C^B_{\\ell_{\\text{def}}}$ should not depend on $g, r$.\n* $\\bar{\\tau}$ is not defined in Theorem 4.4, and it is not clear where $T^{-1, \\nu}, T^{\\nu=1}$ appear in the Theorem.\n* The clusters $C_{cla}^{M1}, C_{cla}^{M2}$ defined in Experiments in 5.2 seem not distinct as described.", "questions_for_authors": "* Please refer to the questions/concerns raised in previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4gDFPmw1zs", "reviewer": "Reviewer_cfAF", "summary": "The paper developed a Two-Stage Learning-to-Defer framework for multi-task problems, enabling joint classification and regression. This framework features a novel two-stage surrogate loss family that is both $(\\mathcal{G}, \\mathcal{R})$-consistent and Bayes-consistent for cross-entropy-based surrogates. The authors derived tight consistency bounds and established minimizability gaps, extending prior work on Learning-to-Defer. Their learning bounds improve with richer hypothesis spaces and more confident experts. The authors validated the approach on object detection and electronic health record analysis, demonstrating its superiority over existing methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful reading and thoughtful comments. We are pleased that they found our framework well-presented and the theoretical analysis rigorous.  In the following, we clarify the technical challenges that arise in the multi-task setting.\n\nFirst, we emphasize that **our paper provides theoretical novelty**. Our framework is generalized to accommodate any positive cost $c\\_j$, enabling a unified treatment of classification, regression, and multi-task problems within Learning-to-Defer. We derived the Bayes-optimal rejector and established consistency guarantees for multi-task L2D. In particular, we prove consistency bounds that hold for any surrogate in the comp-sum family—parameterized by $\\nu$ as defined in Equation 1—which encompasses commonly used surrogates such as the logistic, MAE, and exponential losses. These bounds are both tighter and more interpretable than those in [1, 2, 3], accounting for both the parameter $\\nu$ and the $L\\_1$ norm of the aggregated cost vector $\\boldsymbol{\\tau}$ (see discussion with reviewer @oatj). Additionally, we present a novel analysis of minimizability gaps in Theorem 4.6, demonstrating that the optimal conditional risk critically depends on both the norm of the aggregated costs $\\boldsymbol{\\tau}$ and the choice of the multiclass surrogate $\\Phi\\_{01}^\\nu$ (Equation 1). Notably, our results hold without any assumptions on the underlying distribution $\\mathcal{D}$—in contrast to [4]—and apply to any surrogate $\\Phi\\_{01}^\\nu$ within the comp-sum family. Furthermore, we provide new generalization bounds specifically tailored to this setting.\n\nWhile classification can conceptually be viewed as a special case of regression under the zero-one loss, such a simplification overlooks important theoretical distinctions that arise when analyzing consistency and optimality in deferral decisions. To highlight this, we give some additional analysis: \n\nLet $g(x) = (h \\circ w(x), f \\circ w(x))$ be any multi‐head model where $w \\in \\mathcal{W}$ is a shared representation and $h \\in \\mathcal{H}$, $f \\in \\mathcal{F}$ are heads for classification and regression, respectively. Define $L(w,h,f) = \\mathbb{E}\\_{y,t \\mid x}[c\\_0(h(w(x)),f(w(x)),z)]$ as the conditional risk for the triple $(w,h,f)$. Let $L^\\ast = \\inf\\_{(w,h,f)} L(w,h,f)$ be the optimal multi‐head conditional risk in the hypothesis class $\\mathcal{G} = \\lbrace x \\mapsto (h(w(x)), f(w(x))) \\mid w \\in \\mathcal{W}, h \\in \\mathcal{H}, f \\in \\mathcal{F} \\rbrace$. We then write $L(w,h,f) - L^\\ast$ to denote the conditional excess risk of $(w,h,f)$ above the best multi‐head. Observe that for a chosen triple $(w,h,f)$, we can decompose:\n$$L(w,h,f) - L^\\ast = \\bigl[L(w,h,f) - \\inf\\_{h',f'} L(w,h',f')\\bigr] + \\bigl[\\inf\\_{h',f'} L(w,h',f') - L^\\ast\\bigr],$$\n\nwith $h'$ and $f'$ acting directly on $x$. Define the heads gap $\\Delta\\_{\\mathrm{heads}}(w,h,f) = L(w,h,f) - \\inf\\_{h',f'} L(w,h',f')$, which measures how well the specific heads $(h,f)$ perform given a fixed representation $w$. Then define the representation gap $\\Delta\\_{\\mathrm{repr}}(w) = \\inf\\_{h',f'} L(w,h',f') - L^\\ast$, which captures how close $w$ is to the best possible representation for both tasks. From this notation, we rewrite\n\n$$L(w,h,f) - L^\\ast = \\Delta\\_{\\mathrm{heads}}(w,h,f) + \\Delta\\_{\\mathrm{repr}}(w).$$ \n\nWe next introduce a multi‐task comparison by defining a separate‐training conditional risk. Let $L\\_{\\mathrm{sep}}(h',f') = \\mathbb{E}\\_{y,t \\mid x}[c\\_0((h'(x),f'(x)),z)]$ be the conditional risk of using completely separate models $h'$ and $f'$ without any shared representation. Let $L\\_{\\mathrm{sep}}^\\ast = \\inf\\_{h',f'} L\\_{\\mathrm{sep}}(h',f')$. We measure the quality of forcing a single shared $w$ via\n\n$$\\Delta\\_{\\mathrm{MTL}} = L^\\ast - L\\_{\\mathrm{sep}}^\\ast.$$ \n\nIf $\\Delta\\_{\\mathrm{MTL}} < 0$, then joint training (i.e. a shared $w$) is strictly better than separate solutions; if $\\Delta\\_{\\mathrm{MTL}} > 0$, it is worse. Finally, we can link $\\Delta\\_{\\mathrm{MTL}}$ to the multi‐head conditional excess risk by noting \n\n$$L(w,h,f) - L^\\ast = \\bigl[L(w,h,f) - L\\_{\\mathrm{sep}}^\\ast\\bigr] - \\Delta\\_{\\mathrm{MTL}}.$$ \n\nHence, if $\\Delta\\_{\\mathrm{MTL}} < 0$, it follows that $L(w,h,f) - L^\\ast < L(w,h,f) - L\\_{\\mathrm{sep}}^\\ast$, meaning the multi‐head optimum is easier to approach than the separate‐training optimum. Conversely, if $\\Delta\\_{\\mathrm{MTL}} > 0$, we get $L(w,h,f) - L^\\ast > L(w,h,f) - L\\_{\\mathrm{sep}}^\\ast$, so it is harder to reach the best multi‐head risk than to reach the best separate‐training solution.\n\nFollowing the discussion with Reviewer @oXmZ, we will also incorporate a discussion on how our approach can be adapted to account for coverage constraints. \n\nWe hope this clarifies your question. We will add those novel analysis and revise the manuscript to make this point clearer in the final version.\n\nPlease refer to the discussion with reviewer @oXmZ for references.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "Section 2 provides a sound discussion.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "Strengths: The paper is well-written, clearly presented, and provides a detailed discussion of prior work, making it accessible to non-experts.\n\nWeaknesses: The results, while sound, seem to offer incremental progress rather than significant novelty compared to existing work. Furthermore, I question whether classification can be viewed as a specific instance of regression, with the zero-one loss as the evaluation metric. If so, would the proposed framework be adequately addressed by existing regression-based deferral frameworks?", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "1. Is it possible to view classification as a specific instance of regression, where the zero-one loss serves as the evaluation metric? If this perspective is valid, would the current framework be adequately addressed by the regression-based deferral framework presented in previous research?\n\n2. What specific challenges arise when adapting the learning-to-defer framework for multi-task learning scenarios?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qw2D3mcAeb", "reviewer": "Reviewer_oXmZ", "summary": "The paper presents a new application of learning to defer to the context of multitask, where the task's target consists of both a regression and a classification task. The authors provide a theoretical analysis of two-stage L2D, showing that the proposed surrogate loss is both Bayes-consistent and $\\mathcal{G}, \\mathcal{R}$ consistent.\nEmpirical results showcase the effectiveness of the approach.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their constructive feedback. We appreciate their positive assessment of the strength of our theoretical and empirical contributions.\n\nBelow, we address the potential connection to coverage constraints and provide further details on specific aspects of our experimental design.\n\n> The analyses provided are sufficiently sound, with enough details to reproduce results. Overall, the main concern I have is regarding the choice of subsampling the dataset for MIMIC IV, as detailed in Appendix G.2. Other options could be available, e.g., weighting the loss between classes differently, and the authors could have commented a bit more this aspect.\n\n> Could the authors comment on my concern for the choice of subsampling the dataset? I guess adding a weighted loss is a safe option, but I do not want to overlook consistency.\n\nThank you for raising this point. Indeed, alternative heuristics such as class-weighted losses could have been employed. However, our primary objective was not necessarily to optimize overall performance metrics, but rather to highlight the relative improvement and effectiveness of our approach. Nevertheless, your suggestion of using weighted losses is valid and aligns with our theoretical framework, as our consistency proofs hold for any positive cost $c_j$. A promising direction for future experiments would involve adapting agent-specific cost functions in a cost-sensitive manner to address class imbalance explicitly. \n\n> The essential literature is correctly discussed, the only reference I think must be necessarily added is the work by Okati et al., 2021, which considers a formulation with explicit constraints in terms of coverage.\n\nThis is correct, we will add this paper in the related work.\n\n> Have the authors considered how to directly model the coverage for experts, as considered for instance in [Okati et al.,2021; Mozannar et al., 2023; Palomba et al., 2025]?\n\nThis is an excellent question. In the single-expert setting considered in [5,6,7], modeling coverage explicitly is relatively straightforward. Consider a rejector $r:\\mathcal{X}\\rightarrow\\mathbb{R}$, where $r(x)\\geq 0$ indicates deferring to the expert, and $r(x)<0$ indicates no deferral. To directly model coverage, one can introduce a coverage level $k \\in \\mathbb{R}^+$ corresponding to the $\\overline{k}^{\\,\\text{th}}$ percentile of the rejector scores $r(x)$ computed over a validation set. This yields an adjusted decision rule: $1\\_{r(x)\\geq k}$, instead of $1\\_{r(x)\\geq 0}$, resulting in an expected coverage of $\\mathbb{E}[1\\_{r(x)\\geq k}] = 1-\\overline{k}$.\n\nHowever, in the multi-expert setting, extending this idea is nontrivial because deferral decisions involve complex allocations across multiple experts. This complexity likely explains why [5,6,7] primarily focus on single-expert scenarios. **One possible direction to handle coverage explicitly in multi-expert settings is to consider margin-based rejectors**. Let $\\mathcal{A}= \\lbrace 0,\\dots,J \\rbrace$ denote the set of $J$ experts plus a main model, and define a rejector $r:\\mathcal{X}\\rightarrow\\mathcal{A}$. We then define the margin-based score as\n\n$$\\rho\\_r(x,j)=r(x,j)-\\max\\_{j'\\neq j} r(x,j'),$$\n\nwhere a positive margin $\\rho\\_r(x,j)\\geq 0$ leads to deferring to expert $j$. Coverage can be incorporated by setting thresholds $\\tilde{k}\\_j$ separately for each expert $j$, corresponding to specific percentiles of the margin $r(x,j)-r(x,0)$. The decision rule then becomes $1\\_{\\rho\\_r(x,j)\\geq \\tilde{k}\\_j}$. **Importantly, this approach naturally reduces to the single-expert coverage** definition used in [5,6,7], since deferral occurs when $\\rho(x,1)\\geq 0$, i.e., $r(x,1)\\geq r(x,0)+\\tilde{k}\\_1$.\n    \nWe hope this aligns with your question and would be glad to discuss it further. We will include this clarification and an expanded discussion in the revised manuscript.\n\n### References\n[1] Mao, et al. (2023). Two-Stage Learning to Defer with Multiple Experts. NeurIPS23\n\n[2] Mao, et al. (2024). Regression with multi-expert deferral. NeurIPS24\n\n[3] Narasimhan et al. (2022) Post-hoc estimators for learning to defer to an expert. NeurIPS22\n\n[4] Mao et al. (2024) Theoretically grounded loss functions and algorithms for score-based multi-class abstention. AISTATS24\n\n[5] Okati et al. (2021). Differentiable learning under triage. NeurIPS21\n\n[6] Mozannar et al. (2023, April). Who should predict? exact algorithms for learning to defer to humans. AISTATS23\n\n[7] Palomba et al. (2025). A Causal Framework for Evaluating Deferring Systems. AISTATS25", "claims_and_evidence": "I think the paper is convincing enough in terms of contribution and impact of the approach in real-life scenarios.\nThe theoretical analysis correctly supports the paper's claims.", "methods_and_evaluation_criteria": "The proposed datasets and evaluation criteria make sense. The authors consider two datasets, one that is new in the setting of learning to defer, and one used also in previous works [Mao et al., 2023a, 2024e]. Hence, the empirical evaluation makes sense overall.", "theoretical_claims": "The proofs seem correct, but I might have missed some details.", "experimental_designs_or_analyses": "The analyses provided are sufficiently sound, with enough details to reproduce results. Overall, the main concern I have is regarding the choice of subsampling the dataset for MIMIC IV, as detailed in Appendix G.2. Other options could be available, e.g., weighting the loss between classes differently, and the authors could have commented a bit more this aspect.", "relation_to_broader_scientific_literature": "The paper is correctly positioned in the literature, with most works in learning to defer correctly referenced.\nMoreover, the work extends existing literature,  considering cases where regression and classification occur simultaneously. I think this is a valuable contribution with sound proof and analysis.", "essential_references_not_discussed": "The essential literature is correctly discussed, the only reference I think must be necessarily added is the work by Okati et al., 2021, which considers a formulation with explicit constraints in terms of coverage.\n\n[Okati et al., 2021] - Okati, N., De, A., & Rodriguez, M. (2021). Differentiable learning under triage. NeurIPS 2021", "other_strengths_and_weaknesses": "The paper is clearly written, with convincing motivations for the proposed approach.", "other_comments_or_suggestions": "I provide here for completeness a few recent articles, which could enlarge the current related work section (as they expand different aspects of Learning to Defer):\n\n- In [Wei et al., 2024], the authors provide a refinement of Bayes consistency, called Bayes-dependent consistency;\n- In [Palomba et al., 2025], the authors bridge causal inference and learning to defer for improved evaluation of such systems;\n- In [Strong et al., 2025], the authors present an application of L2D for healthcare using LLMs\n\nFinally, according to ICML guidelines, the caption for tables should be placed above. Please consider adjusting the ones in the paper.\n\n\n[Wei et al., 2024] - Wei, Z., Cao, Y., & Feng, L. (2024). Exploiting human-AI dependence for learning to defer. ICML '24\n\n[Palomba et al., 2025] - Palomba, F., Pugnana, A., Alvarez, J., Ruggieri, S. (2025). A Causal Framework for Evaluating Deferring Systems. AISTATS '25\n\n[Strong et al., 2025] - Strong, J., Men, Q., & Noble, A. (2025). Towards Human-AI Collaboration in Healthcare: Guided Deferral Systems with Large Language Models. AAAI '25", "questions_for_authors": "I think the paper is sound enough, with good motivation and adequate evaluation.\nHence, I am prone to suggesting acceptance of the paper.\n\nI have a couple of questions for the authors:\n\n- Have the authors considered how to directly model the coverage for experts, as considered for instance in [Okati et al.,2021; Mozannar et al., 2023; Palomba et al., 2025]?\n- Could the authors comment on my concern for the choice of subsampling the dataset? I guess adding a weighted loss is a safe option, but I do not want to overlook consistency concerns.\n\n### Update After Rebuttal\n\nI am satisfied with the answers from the authors. I keep my positive score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pJw3gKMX2T", "reviewer": "Reviewer_phu8", "summary": "This paper presents a two-stage learning-to-defer (L2D) approach for the multi-task setting involving both classification and regression. The authors provide theoretical justification in the form of consistency bounds, as well as empirical justification in the form of validation on (multi-task) object detection and EHR analysis.\n\n## Update after rebuttal\nAs stated in my rebuttal comment, I thank the authors for the clarifications and will maintain my recommendation of Weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful and constructive feedback. We are pleased that they found the paper well-written, the empirical setup appropriate, and the contributions clearly positioned in relation to prior work.\n\nBelow, we provide clarifications on several points raised in the review.\n\n> Empirical results may be difficult to interpret for non-experts in L2D.\n\nTo clarify, we conducted two distinct sets of experiments. In the object detection task, we considered three models of varying complexity (a lightweight Faster R-CNN, a medium-sized Faster R-CNN, and a large Faster R-CNN). To reflect their varying complexity and computational cost, we assigned different consultation scores to each agent: $\\beta_0=0$, $\\beta_1=\\beta_1$, and $\\beta_2=R_G\\beta_1$ with $R_G$ being a ratio of GLOPs. This setup explicitly illustrates the trade-off between computational efficiency and prediction correctness, as the largest the model is, the better is its performance. Figure 1 shows that, for higher consultation costs $\\beta_1$, our approach allocates queries mostly to the main lightweight model (cost $\\beta_0$) or the first expert (cost $\\beta_1$). Conversely, when $\\beta_1$ is lower, the rejector strategically allocates queries across both experts (expert 1 and expert 2). \n\nIn the EHR task, baseline approaches fail to achieve strong performance because they base allocation decisions solely on individual tasks (classification or regression separately). In contrast, our method jointly considers both tasks, resulting in more balanced and effective query allocation. We will clearly emphasize these interpretations in the revised manuscript to improve clarity.\n \n> References to tables/figures are unusual (ex: “Figure 5.1” [L372] and “Table 5.2” [L435]). These should refer to the tables/figures themselves, not the section.\n\nThank you for pointing this out.  We have taken note and will update this in the revised manuscript. \n\n>  When describing a medical use case of L2D, the authors write, “If the model is sufficiently confident [...] I was expecting some sort of human-in-the-loop evaluation where low-confidence cases were deferred to an actual human expert. Is my understanding correct, and could the authors provide some way to make this distinction clearer for non-experts? \n\nThank you for raising this important point. To clarify, the medical use-case mentioned in the Introduction is intended solely as an illustrative example of deference rather than a description of the concrete mechanism used in our empirical experiments. In our experiments, “deference” refers to the process of automatically selecting predictive agents (e.g models, humans) based solely on their predictions and associated costs.\n        \nA key strength is its agent-agnostic nature: it does not rely on any specific internal structure, training paradigm, or decision process of the agents. As long as we have access to an agent’s predictions, our framework can learn a rejector $r\\in\\mathcal{R}$ that, for each input $x\\in\\mathcal{X}$, computes confidence scores and allocates the query to the agent deemed most reliable. Importantly, during inference, we only query the selected expert.\n\nIn our empirical implementation, we have used automated black-box models or synthetic distribution as the experts. However, the framework is fully flexible and can incorporate human experts or any other decision-making system, as long as their predictions are available. We will update the manuscript to explicitly clarify this distinction, thereby reducing any potential confusion.\n                \n\n> In the object detection setting, why do the authors not compare performance to other approaches? Also, can the authors explain why this approach is practically useful if it never outperforms the largest model (Expert 2)?\n\n The primary reason we did not include comparisons with other object detection approaches is that our paper focuses explicitly on evaluating the effectiveness of the allocation mechanism within the Learning-to-Defer framework, rather than competing directly with state-of-the-art detection methods. Importantly, our method remains practically valuable even when it does not surpass the performance of the largest expert model (Expert 2). This is because Expert 2, while accurate, is typically too computationally expensive or slow to deploy on every query in realistic, resource-constrained environments. Our L2D approach provides a principled solution to this critical challenge by strategically routing queries to less computationally intensive models whenever appropriate, significantly reducing resource usage without severely impacting accuracy. Furthermore, when expert models have specialized strengths or operate on slightly different distributions—as demonstrated clearly in our EHR experiments—our method can exploit these discrepancies to effectively improve overall system performance beyond what any single agent could achieve alone.", "claims_and_evidence": "Regarding theoretical claims, I will have to defer to domain experts to verify accuracy. Empirical claims appear sound, though it is somewhat difficult to interpret these results without being intimately familiar with L2D literature and evaluation.", "methods_and_evaluation_criteria": "The two validation datasets were appropriate for this unique setting.", "theoretical_claims": "I did not verify proofs due to time constraints.", "experimental_designs_or_analyses": "Experimental design of empirical experiments seems appropriate – performance metrics were appropriate, and results (+ variability) were presented over 4 trials. Source code was provided in order to reproduce experimental results.", "relation_to_broader_scientific_literature": "The authors clearly lay out how, despite recent progress in L2D, previous two-stage L2D approaches do not address multi-task “classifier-regressor models”, which are relatively common in more complex tasks such as object detection.", "essential_references_not_discussed": "If there are key references missing, then I am not aware of them nor qualified to propose them.", "other_strengths_and_weaknesses": "*Strengths*\n- The paper is very well-written and carefully organized to logically walk the reader through each section.\n- The related work is thorough, with unique contributions of this study clearly laid out relative to prior work.\n\n*Weaknesses*\n- Empirical results may be difficult to interpret for non-experts in L2D.", "other_comments_or_suggestions": "References to tables/figures are unusual (ex: “Figure 5.1” [L372] and “Table 5.2” [L435]). These should refer to the tables/figures themselves, not the section.", "questions_for_authors": "I want to be transparent that this is outside of my area of expertise, so I may have elementary questions regarding the background of L2D and interpretation of results.\n\n1.\tWhen describing a medical use case of L2D, the authors write, “If the model is sufficiently confident, its diagnosis is accepted; otherwise, the decision is deferred to a medical expert who provides the final diagnosis.” I take it this is an illustrative example of the concept of deference rather than a concrete description of L2D as manifested in empirical experiments, correct? For example, in the empirical experiments, it seems that “deference” refers to accepting/rejecting (weighting) the predictions of task experts rather than deferring to, say, a human expert. Perhaps this is an obvious point to researchers like yourselves, but based on this description in the Introduction, I was expecting some sort of human-in-the-loop evaluation where low-confidence cases were deferred to an actual human expert. Is my understanding correct, and could the authors provide some way to make this distinction clearer for non-experts? \n2.\tIn the object detection setting, why do the authors not compare performance to other approaches? Also, can the authors explain why this approach is practically useful if it never outperforms the largest model (Expert 2)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper examines two-stage learning to defer under the multi-task setting, meaning that there are supervised losses for both classification and regression, providing a theoretical analysis of the consistency of the proposed loss as well as empirical validation on the Pascal VOC and MIMIC datasets.  The reviewers generally had positive opinions of the work (1 x accept, 2 x weak accept, 1 x weak reject).   The reviewer who leaned toward rejection's primary criticism is that the extension to multi-task learning had marginal novelty.  The authors provided a detailed rebuttal, nicely explaining the paper's theoretical contributions, which I found convincing.

Small comment: I noticed two errors in the years of the citations---one was that Verma et al. should be AIStats 2023, not 2022.  I can't remember the other one
Decision: Accept (poster)