Identity-Free Deferral For Unseen Experts
Authors: Joshua Strong
Pramit Saha
Yasin Ibrahim
Cheng Ouyang
Alison Noble
Venue: ICLR 2026 Poster (2026)
Keywords: learning to defer
healthcare
medical
human-AI collaboration
deferral
uncertainty
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Learning to Defer (L2D) improves AI reliability in decision-critical environments by training AI to either make its own prediction or defer the decision to a human expert. A key challenge is adapting to unseen experts at test time, whose competence can differ from the training population. Current methods for this task, however, can falter when unseen experts are out-of-distribution (OOD) relative to the training population. We identify a core architectural flaw as the cause: they learn identity-conditioned policies by processing class-indexed signals in fixed coordinates, creating shortcuts that violate the problem's inherent permutation symmetry. We introduce Identity-Free Deferral (IFD), an architecture that enforces this symmetry by construction. From a few-shot context, IFD builds a query-independent Bayesian competence profile for each expert. It then supplies the deferral rejector with a low-dimensional, role-indexed state containing only structural information, such as the model's confidence in its top-ranked class and the expert's estimated skill for that same role, which obscures absolute class identities. We train IFD using an uncertainty-aware, context-only objective that removes the need for expensive query-time expert labels. We formally prove the permutation invariance of our approach, contrasting it with the generic non-invariance of standard population encoders. Experiments on medical imaging benchmarks and ImageNet-16H with real human annotators show that IFD consistently improves generalisation to unseen experts, with gains in OOD settings, all while using fewer annotations than alternative methods.

Published as a conference paper at ICLR 2026
IDENTITY-FREE DEFERRAL FOR UNSEEN EXPERTS
Joshua Strong, Pramit Saha, Yasin Ibrahim, Cheng Ouyang, J. Alison Noble Department of Engineering Science, University of Oxford, United Kingdom
ABSTRACT
Learning to Defer (L2D) improves Al reliability in decision-critical environments by training AI either make its own prediction defer the decision human expert. A key challenge is adapting to unseen experts at test time, whose com- petence can differ from the training population. Current methods for this task, however, can falter when unseen experts are out-of-distribution (OOD) relative to the training population. We identify core architectural flaw as the cause: they learn
policies by processing class-indexed signals in fixed co- ordinates, creating shorteuts that violate the problem's 's inherent permutation sym- metry. We introduce Identity-Frres Deferral (IFD), an architecture that enforces this symmetry by construction. From few-shot context, IFD builds query- independent Bayesian competence profile for each expert. It then supplies the deferral rejector with low-dimensional, role-indexed state containing only struc- tural information, such the model's confidence in its top-ranked class and the expert's 's estimated skill for that same role, which obscures absolute class identi- ties. We train IFD using uncertainty-awan context-only objective that removes the need for expensive query-time expert labels. We formally prove the permuta- tion invariance of our approach, contrasting with the generic non-invariance of standard population encoders. Experiments on medical imaging benchmarks and ImageNet-166 with real human annotators show that IFD consistently improves generalisation to unseen experts, with gains in OOD settings, all while using fewer annotations than alternative methods.
INTRODUCTION
In high--itakee domains such as healthcare, autonomous systems must decide not only what to predict but also when defer a human. Learning Defer (L2D) formalises this by training classifier to predict labels and a rejector to decide whether to accept the Al's prediction or defer to an expert, directly optimising end-to-end performance of the resulting human-Al collaborative system (Madras) al. 2018). This design promises safer AI deployment, but real-world environments pose a key challenge: the experts available at test time are often unseen (i.e., not among the experts observed during training). Hospitals rotate staff, trainees progress, and practice varies across sites. As such, robust deferral system must adapt to such turnover.
Existing population aopulation-adapiivee extensions of L2D tackle this challenge by meta-learning expert rep- resentations from small context past predictions (Tailor et al. 2024). Conditioning the rejector on these embeddings enables transfer unseen experts drawn from the same distribution (i.e., in- distribution (ID)). However, in practice, systems must also handle out-of-distribution (OOD) unseen experts whose competence profiles are systematically shifted-for example, the same specialisation pattern under a different class indexing or altered strengths across sites. Here, existing methods can falter. We diagnose the root cause: architectural dleetity-conditionne deferral. By processing class-indexed signals in fixed coordinates (c.g., per-class embeddings). population encoders expose label identities. This allows the rejector learn shortcuts such as "if index strong, defer." which may work for the training label indexing but break under simple class relabellings (i.e., re- naming/reindexing classes should not change the deferral decision). As result, such policies can misgeneralise under reindexings common in OOD settings.
Our approach. We propose Identity-Free Deferral (IFD), which enforces the correct inductive bias architecturally. IFD eliminates all absolute identity channels before the rejector sees any input.
*Correspondence joshua.stronseeee.....oo stwongeeng.g..ooooo ox, wx. ac.uk

---

Published as a conference paper at ICLR 2026
From a few-shot context, it builds Bayesian competence profile for each class. IFD then supplies the rejector with a role-indexed state: values read at structural roles such as the model's top class or the expert's estimated best class, plus symmetric aggnegates. This low-dimensional interface ensures that deferral decisions depend only on transferable structure, never on class IDs. Training uses an uncertainty-aware context-only surrogate objective, which naturally downweights uncertain profiles and eliminates the need for query-time expert labels.
Why this transfers. By aligning the architecture with the permutation symmetry of class rela- bellings, IFD blocks ddentity-conditionee shortcuts. Its query-independent profiles reduce variance small context sizes and scale gracefully as more context is available. The resulting policy focuses on the structural question- who is better for this case?-rather than memorising identities.
In summary, our contributions are:
Identity-Free Deferral (IFD): role-indexed rejector interface that enforces invariance to class relabelling by construction. We formally prove this property and show not guaranteed by standard population encoders.
Context-only, uncertainty-aware training: a data-efficient objective for adapting to new experts without query-time expert labels, enabled by explicit Bayesian competence profiles that naturally support uncertainty quantification and the incorporation of prior knowledge. Empirical findings: across medical and general imaging benchmarking datasets, IFD improves generalisation to unseen experts, with the largest gains under OOD expert shifts.
2
RELATED WORK
An extended survey on related work appears App-
Learning to Defer (L2D). L.D augments selective predictionn/reeecton (Chow] 1957 1970 Cortes
et al.) 2016) by training a classifier and rejector that can defer to an expert, improving end- to-end outcomes over prediction-only systems (Madras al. 2018 Mozannar & Sontag 202 Follow-ups study multiple experts, limited expert signals, cost/triage constraints, and two-stage training Verma et al. 2023] Hemmer et al., 2023 Alves et al. 2024, Mao et al. [023 Montreuil et al., 2025b). with applications in healthcare, fraud, and content moderation (Joshi et al. 2021
Strong et al. 2025 Alves et al., 2023 Lykouris Weng 2024).
Adapting to population of experts. The state-of-the-art approach for handling unseen experts is L2D-Pop (Tailor et al., 2024). which meta-learns an expert representation from few-shot context to condition the rejector. While effective for ID experts, its high-capacity encoders process class information in fixed coordinates, allowing the model to learn the identity-conditione shortcuts we identify key failure mode. These architectures lack sufficient inductive bias for permutation symmetry, which hinders generalisation to OOD experts.
Our work, Identity-Frre Free Deferral (IFD), addresses this architectural gap directly. Instead of learning an unstructured latent embedding, IFD constructs an explicit, guery-independent Bayesian profile of expert's s competence. Crucially, it communicates this information the rejector through -in-dimesiional, role-indexed state that is provably invariant to class relabelling. This design is complemented by a context-only, uncertainty-aware loss that removes the need for expensive query- time labels, further improving data efficiency.
3 BACKGROUND & PROBLEM SETUP
3.1 STANDARD (SINGLE EXPERT) L2D
In a classification task, we predict a target Y y [1....,K] K} from inputs X E X, where (X,Y) ~ P. The L2D framework assumes an expert E provides predictions M E M = y Given a dataset D {x,, Vi, J. the objective is to learn predictor Y X + y (1 (1 denotes deferral) by jointly training a classifier h X' + ) and a rejector r X' - {0. 1). predicts the target; decides to use h(x)) (r(z) 0) defer (r(z)
1). For this subsection only, the expert is fixed (i.e., a single expert Eg supplies all responses M at both train and test time - no other experts enter leave the system) so write r(x:) = r(æ, Eo) for brevity.
2

---

Published as a conference paper at ICLR 2026
Optimisation. Assuming costs (æ, y, h(x)) for classification and lesp(xt, y, m) for deferral, the system-level L2D loss is:
(h, r) = B(x.w)P, P() [c. y, h(z)) = 0} + tep y. [(((z) = 11] (1)
This non-convex loss hard to optimise directly: Mozannar Sontag (202T) proposed a consistent surrogate for Eq. ( derived from the Bayes minimiser of the 0-1 system loss. The 0-1 loss is:
(o-1hh, r) = (r..)) mm (<)(()) * ))) I{((x) = 0} + 1m + [)(() = 1H
Minimising Lo yields the Bayes-optimal classifier (xx) and rejector r* (z):
ht (z) ;= arg max P(Y r). r" (x) := P(Y=M M x, Ea) 2 max P(Y x) }. (2)
yEy
yEy
With score functions gy (x) for h(z) arg max, By (x) and I1 (x) for r(x) 1 max gr (x) 5 gi (x)). consistent softmax-bases based surrogate is:
LCE(9h..... ,K,, (.....K.... m)) ;= log plu xx) 1m y} log (og((1] x),
where ply/ 1-) explyx ())) /ee//Eyraa Eyreu exp(ay
This loss encourages accurate classification (via 9y) and deferral (via g.) when the expert correct
(m y). At inference, prediction is deferred if g. (x) 2 maXyey (y(r). Standard L2D assumes single, fixed expert. As result, the learned rejector tailored to that expert's behaviour; substituting different (unseen) expert at test time is unsupported here. This limitation motivates population- adaptive L2D, described next.
3.2 POPULATION-ADAPTIV L2D
The standard approach the problem of fixed fixed-expert coupling is adapt the rejector to the identity
of the test-time expert E using small context set DE = {(x; ",m,m,mmm...... expert's past behaviour. Here, the test-time expert varies, so we henceforth denote the rejector and its respective score function conditioned on E as r(z, E) and 91 (2, E), respectively.
-, m,. summarising that
Tailor et 20244) propose learning a set encoder ub which maps context DE to a fixed-d dimen- sional latent expert representation. There are two variants:
Query-independent (QI):
+ IRd, with (1 F.) 91 -1, (5),))))),
(4)
Query-conditioned (QC):
s (x))
= (iz D ERA, with 91 (z. E) = O1 (a,, ab (x)).
(5)
The QI variant uses a Conditional Neural Process (Gamelo et al. 2018). the QC variant an Attentive
Neural Process Kim et al. 2019b). Note that both encode the context: [<(xx]) e(y e(m, B.C.) ).), where e(-) is hoe-hot/eer label embedding and je a feature encoder.
Bayes rule. Averaging the system 0-1 loss over the expert population E ~ Q yields
20, r) = (x,,))>,, E~ px [ E) 0} [1((()) + y} + Ur((, E) = 1m # vHl-
The classifier is unchanged, and the rejector compares the expert-conditioned correctness to the model's best class:
ht (z) = arg max P(Y=y I( xr E) yEy
max P(Y=y) xz) (6)
vEY
Surrogate and training, Let
exp(gy(x))
for Eo x(11 exp (x, 1b)
ply 1z, v))
Z(x, vb)
Z(z, v))
where Z(x, v)) == exp(91 v)} E (xployzz)) A consistent surrogate follows Eq. 33
CC9 (g1..... ........9999.. 9K1 x, x, m, v) - log xz 1)) 11m=y) p(L) v) Inference is analogous: predict (xx) arg max, By (x: and defer iff 91 (I, v) 2 max, 9y (2.)
Multi-expert extension. Given available set of experts (E,) each j and defer to the expert with the largest deferral margin. See Remark 4.] for the explicit rule and its relation the standard K-JJ-wy surrogate (Verma et al. 2023).
evaluate Ie (x. Ey) for
3

---

Published as a conference paper at ICLR 2026
3.3 PERMUTATION SYMMETRY, IDENTITY LEAKAGE, AND UNSEEN VS. OOD EXPERTS
In this subsection we make explicit the permutation symmetry that the population problem enjoys and explain why architectural violations of that symmetry predict poor transfer to OOD experts. Proving (non-)invariance under coheren! relabellings (CR) is an architectural, distribution-free prop- erty: reveals whether the hypothesis class permits dependence on absolute label identities. This serves as a clean proxy for vulnerability to the expert-only reindexings used in our OOD stress tests. The CR non-invariance result establishes the possibility of deenity-condiitioned shortcuts un- der standard parameterisations; empirical failures then arise from standard ERM and optimisation dynamics, which select such shortcuts whenever they reduce empirical risk.
The core symmetry: coherent relabelling. For any class-indexed vector (un) and permu- tation π E K define the permuted vector v" by (v"), == (g))())- coherent relabelling applies the same permutation every class-indexed object (model outputs, labels, expert profiles, etc.). If the original data mechanism is P(Y X), the relabelled mechanism is
pd (Y = X X-x)= = PT (v) X-z).
Write the population Bayes action from 6 as
(o) E; P. F) (h (x:; P), r' (x,
Proposition 3.1 (Classifier equivariance and rejector invariance under coherent relabelling). For any π E GK, we have:
1. Classifier equivariance: (1" P*) (1;"
2. Rejector invariance: (z( P (
Proof sketch. Classifier equivariance is immediate: applying simply permutes the label space. For the rejector, all quantities that determine the Bayes decision (e.g. model top probability and the ex- pert's 's classwise correctness terms used in the expected expert accuracy) are permuted coherently and therefore remain identical as real numbers. Hence the binary defer/predict decision is unchanged.
ID-unseen vs. OOD-unseen experts and the expert-only stress test. An expert is ID-unseen if is new at test time but drawn from the same population as the training experts; an OOD-unseen expert has competence pattern outside this population. To model simple, interpretable OOD shift we consider expert-only permutations: the task and model posterior remain fixed, but the expert competence vector is reindexed.
Let n (x) = P(ME=y X=xr, F) be the expert's class-conditional correctness at v, and let 17y (x) = P(Y=y X=x). Define the expected expert correctness
((x = Σ (1)( n (xx)
An expert-only permutation yields (11 E. (. (x) = - (z) (z) giving
((), Σ my(z) (f = Σ (mt)))))
As the permutation alters how expert competence is reweighted by model probabilities, ((x, E") can differ from q(z,, E) and may change the Bayes decision (as is to be expected).
Proposition 3.2 (Expert-only permutations can change the Bayes action). If q(x, Ex) 4 g(x, E) and the deferral margin is nondegenerate at (xx, E), then r* (I., E²) + r" (x., E).
(z, Proof sketch. Let (x, E) == q(x, E) max, (z). The Bayes rule defers iff (2, Any permutation that flips the sign of this margin (achievable by aligning large e with small or large T=(=) appropriately) flips the Bayes defer/predict decision when the margin is nondegenerate.
0.
Where architectures can fail: identity channels. Architectures that expose absolute class identi- ties (e.g. one- mne-hoos/untied embeddings/ccorddnntt -eeeintte-seciicccccceeeeeeeccee pea-class channels) allow rejectors to form rules that depend on fixed coordinates of class vectors. These label-identing channels enable index- tied shortcuts such as "defer when coordinate j is strong," which interpolate well in-distribution but brittle under expert-only permutations.
4

---

Published as a conference paper at ICLR 2026
Theorem 3.1 (Generic non non-invariance of standard L2D-Pop parameterisations). Under standard untied ianss-dddeedd parameterisations (explicit one-hots or untied embeddings feeding per-class channels), both the QI and QC population encoders generically fail to be invariant to coherent relabellings: there exist (I, E,, π) such that g.1 (I, E") # g1 F) See App. E.2 K.3 for formal statements and proofs.
Proof sketch. QI and QC encoders consume class-indexed inputs in fixed coordinates and parame- terise each coordinate with untied weights. As result, the network can attach semantic meaning to absolute indices (e.g. treating coordinate
"the expert's strong class"). Under CR x, the expert's strongest class moves from coordinate j to π (i), but the encoder's weights tied to coor- dinate j do not move with it. Thus the latent representation typically changes in way that does not correspond to the relabelling. Formally, Appendix E.2 E.3 show that, for full-measure set of parameters, there exist (x, E, π with Ii (z, E²T) ) # 91 [T, E). Achieving invariance would require non-generic weight-tying -yiig patterns that these architectures do not impose.
Corollary 3.1 (Architectural vulnerability to expert-only permutations). If an encoder/rejector is non-invariant to coherent relabellings as in Thm. [3.] then there exist inputs x, expert profiles E, and expert-only permutations for which the rejector output changes when only the expert profile permuted. In short: architectural non-im invariaaace the problem's 's symmetry implies vulnerability to expert-only permutations.
Remarks. Non-invariance is an existential architectural diagnosis: it shows that absolute-index shortcuts are representable. Whether training actually selects such shorteuts depends on the data distribution and the optimiser but is routinely observed in practice. Conversely, architectures that enforce CR-invariance eliminate these shortcuts by construction.
Lay summary. The population Bayes rule respects coherent relabelling (Prop. 3.1 Standard L.2D- Pop architectures break this symmetry via identity-indexet channels and are therefore generically non-invariant (Thm. 3.I), implying vulnerability to expert-only permutations (Cor. [.1].. This mo- tivates our OOD stress tests and, in $4 our Identity-Free Deferral architecture, which enforces the relevant symmetry by construction.
METHOD: IDENTITY-FREE DEFERRAL (IFD)
Motivated by the symmetry analysis in $3.3 propose Identity-Free Deferral (IFD): deferral mechanism that enforces the problem's permutation symmetry architecturally. IFD removes all label/expert identity channels before the rejector sees any input, so the rejector can depend only on structural relations between the model's beliefs and an expert's competence. Consequently, the learned policy insensitive to coherent relabellings (matching the Bayes-rule equivariance), avoids ddentity-conditionn shortcuts characteristic of standard population encoders, and transfers to both ID- and OOD-unseen experts whose competence profiles are permuted or shifted relative to training.
Overview. From a few-shot context, form query- -ndee-inde Bayesian competence profile per class (c.g.. Beta-Binomial posterior mcan/variance) plus a Lower Confidence Bound (LCB) weight encoding uncertainty/priors ($4.1). We then build identity-free, role-based state with permutation-ing invariant rejector ($4.2) and map it to a deferral logit via a small MLP. Finally, we train with an uncertainty cross-entropy that uses only these profiles and LCBs-no query-time expert labels-and accommodates prior knowledge
We follow the notation of $3)
4.1 FROM CONTEXT TO AN EXPLICIT BAYESIAN EXPERT PROFILE
The purpose of this subsection is to describe how, from a few-shot context of an expert's past pre- dictions, we compute an explicit Bayesian per-class competence profile. Importantly, because the role-indexed state depends only on values read at structural roles (c.g., model-top and expert-top) and on commutationinvvaiian aggregates, removes all label-identity channels. This enforces the coherent-relabelling symmetry identified in $3.3 and blocks the generic non-invariance mechanisms that standard population encoders permit.
"GGnerically" means an open (full-measure) set of parameter values; achieving invariance would require explicit weight tying or symmetry constraints.
5

---

Published as a conference paper at ICLR 2026
We model per-class correctness via Beta-Binone conjugacy. For each class y E V let n T(u = y, m E,C
>. My, = y) t = E.
= =y}. With prior 0 ~ Beta(ay, Py), the posterior is:
o D Beta(ay + +,y By n -
Two important properties of the posterior are the classwise mean and variance: E = E DEL, (a Var[0r D We identify the estimated strongest class TE
"best arg max, 1
Note that we approximate P(ME - y IY - y, E) rather than P(ME = improve data efficiency and reduce spurious instance-level couplings. In high-stakes domains such as medical imaging, inter-expert variation is often aligned with class-level specialisation. The class- level abstraction yields (i) closed-form posteriors and uncertainty; (ii) state that transfers across experts; and (iii) a natural handle for prior knowledge ($4.4). Because our method's s deferral logic depends knowing the expert's s strongest class, must ensure this class can be identified reliably from a finite context, as Proposition 4.1 guarantees.
E) to
Proposition 4.1 (Peak-clase identification). Let an arbitrary expert have true but unknown per-class accuracies 02 E 10, 1] (we drop the E' notation here). Let kbest arg maxg On be the unique class of maximal expertise. As the number of context samples ny + 0C for each class k. the posterior mean estimates μ* converge almost surely Gs, and the selected expertise class arg max Me converges almost surely to kbeas. Moreover, if each class is observed at least n. times, where
(n(2K]
2
with Ascc == 0 Bk
max OR.
2
Aax 2
le Rieot
then with probability at least 1 5, the posterior means satisfy Mkaee > 1h for all k of kbest; that is, the correct expertise class is identified. Proof: See App. D.1
4.2 IDENTITY-FREE, ROLE-INDEXED STATE AND REJECTOR
In this subsection we show how the Bayesian per-class profile of and the classifier's 's scores are converted into compact, role-indexed state (values read at structural roles like the model-top and expert-top plus symmetric aggregates). By exposing only these identity-free features to the rejector, we block label-identity channels and guarantee the rejector's decisions are permutation-invarre and transferable across unseen experts.
Let the classifier produce p(z) = softmax(g(x)) with coordinates px(z) and (x) e arg maxk (z()). From the expert's context DE, fix JF 2 1 per-class posterior functionals F == {fi) (e.g., mean, variance, MAP, quantiles/LLCBs, log-odds, updated hyperparameters) Define
(v) 10 DE),
$ = ==1 Imn.k, E KJJ
Choose ranking functional u E F. For any v E K let (()) 2 (2) 2 v(K) denote order statistics (with ties broken by index).
Identity-free state. The IFD rejector never sees absolute class IDs. Instead, it receives small vector 2(x, E) Rd made only of (i) values read at roles-indices picked by the data, not by name-an (ii) permutation-invarian aggregates, The two roles use are the predictor's 's top class Keop (x) and the expert's top class from context R.pest- Define 2(x,, F) RT(p(x), &F), where RI returns a fixed set of features such as (x)
(I), oF (x)) PRR PR (x), uhe " the indicator lf (z)= the gaps (x)mp()) (x:) P(a) (ar) and "(i - uoy and symmetric aggregates of p (e.g., entropy). As entries are either read at roles or are symmetric aggregates, 2(x, E) does not depend on absolute class identities.
Our instantiated feature set (used in all experiments). We take /=2 posterior functionals (per- class mean and variance) and rank by mean ub u. Our 6-dimensional state is
{x, E) = (Prnnpte) (2), PRe (x), beas
(eop(()) Roop (o))
(o RE ) ER.
(8)
model confidences roles
expert profile at model-top expert profile expert-top

---

Published as a conference paper at ICLR 2026
Rejector. An MLP maps the state to the deferral logit: 91 E) = (((((z,)) no RR R.
Permutation invariance (why this blocks identity leakage). If a permutation π E GK reindexes all class-indexed quantities coherently, (p"), Px-+(y) and (Ib..T) )2. (Φ "())s then the roles move together: (r") = x (ktop(()) and be (TL.") T (T best (TE)). Values read at roles and symmetric aggregates are therefore unchanged, giving:
Proposition 4.2 (Permutation invariance of the role-indexed state). For any E 6x. RI(p" (x), 4 F..))) = Rl(p(z), D²). Equivalently, z(z, E) is invariant to coherent relabellings.
Corollary 4.1 (Rejector invariance). For any parameters & and any x E Gk. 91 (z:)" = (o(RI(p" {Bix)) - (RT(RR +)) 99. (x, E.
Intuition. Under coherent relabelling ($3.3) the roles (model-top. expert-top) move together and the aggregates are symmetric. Hence 2x, E) and thus 91 gi-are are unchanged. Full proofs in App. E.4
The following Remark details the multi-expeet inference procedure of IFD:
Remark 4.1 (Multi-expert inference with budget sweep). Let the available experts be {Ez}}=1 De- fine the model best-class logit s(x) == maxtey Ih (x(r) and expert deferral logits tz = 91 (z, Ej). The expert-specific deferral margins are
(5(x) == tz(x) s(x), (*))) = maxje(1)45(0
(x)x) = arg max ja|j] (z
For any threshold T E I, the decision rule is
((x) = defer to (x)) arg max (x(,), otherwise.
(a) 2T,
(9)
The default operating point used by standard L2D corresponds to T == 0. Our evaluation varies the deferral budget [0, 1] by sweeping т: for a target d, set T (d) to the -qqantilee of {a*(x;)) on the evaluation set, i.e., defer exactly the top d fraction by A This "winner-takes routing
matches the K K-Jolo mulli-expert L2D view with g ((x) = (z).
4.3 UNCERTAINTY-AWAN TRAINING WITH MINIMAL ANNOTATIONS
This subsection describes the context-only training objective that supervises the rejector using the Bayesian per-class profile and formulates deferral supervision as an uncertainty-weighted term.
Here, reuse the (K+1)-way wwyy softmax notation p(- instead plug our identity-fred state into the embedding by setting virD( F)) = 2(x, F.).
æ, 1)), p(1) x, v) Z(x, 1b) from $3.2 but
For labelled query (ar,)) and sampled expert E, avoid query-time expert labels. We supervise deferral only when the expert's peak class matches y, and we weight it by a lower-confidence bound y = [uy o y (a 2 0):
CUB (I. E) logp(y) VIFD) L. I4 4 log p( [| vIFD).
(10)
This objective is novel in two respects. First, requires no query-time expert labels, as supervi- sion is derived entirely from the context-derived profile (h My cant annotation bottleneck. Second, it incorporates risk-sensitive weighting via a lower confidence bound (L that reduces supervision from uncertain expert profiles to prevent over-deferral. Prior population-adappie adaptive methods rely on query-time labels and lack this uncertainty-awe the effectiveness of which we validate in an ablation study (App. At training time we aggregate per-expert losses. For a minibatch B and an optional per-batch subset Eg C E, we optimise
E . which eliminates a signifi-
mechanism,
Diran
B Σ (Z.D)E
Σ EEED
CIE (x,, y. E),
[E3]
with CID from 10 At inference, we score each available F. and apply the decision rule (9) As by can alter the proportion cases deferred, benchmarking against IFD should take L = and evaluate against sweep of all deferral thresholds.
Asymptotic analysis and the IFD deferral rule. We analyse the behaviour of the surrogate loss 10) the context size grows (n oo). While the induced classifier h² (xx) converges to

---

Published as a conference paper at ICLR 2026
the Bayes-optimal predictor, the induced rejector follows distinct and interpretable rule (proof in App. D
(x, E) = I{max,ey P(Y=yla) s P(Y=kbost xx)
0 }.
(11)
where bbest the expert's true best class and 0h their accuracy on it. This differs from the stan- About
dard 0-1 Bayes rule (Eq. (6). which compares model confidence to the expert's expected accuracy across all classes (i.e., by P(Y=y x:) 0.)) The IFD rule instead focuses on the expert's peak competence. This direct consequence of the loss design, which strategically targets deferral supervision using only context-derived information.
4.4 PRIOR KNOWLEDGE AND ROBUSTNESS TO MISSPECIFICATION
The Beta prior used in constructing the Bayesian expert profile ( provides way incorpo- rate prior knowledge about an expert's competence. If an expert supplies self-assessed per-class accuracy ay with confidence of E [0, 1] and global strength s 2 2.
agy 1+ by ay (s 2), By = 1 (1-
(12)
Low confidence (e 0) recovers the uniform prior; high confidence increases prior influence. As context accumulates, posteriors override potentially miscalibrated priors:
Proposition 4.3 (Overcoming prior misspecification). Let the true accuracy be g and the prior mean Hprior with strength Sprior = ay + Sy. For any > 0, E (0, 1), it suffices that
Sprior (1pprior - 0 - e/22) 21n(2/0) )
E ny
max
e/2
to ensure 1n
< with probability at least 6.
Proof is deferred to App. D.I Practitioners can obtain prior confidence values experts using
structured expert judgment protocols such Cooke's Classical Model (Colson & Cooke 2018).
5 EXPERIMENTS
We evaluate Identity-Free Deferral (IFD) on simulated and real human experts. Our primary re- search questions (RQs) are:
RQ1: How does IFD compare to state-of-the-ar methods in robustness to unseen experts and varying expert diversity?
RQ2: How robust is IFD compared to L2D-Pop under input distribution shift?
Additional RQs: RQ3: How does performance scale with increasing context size for IFD versus baselines?, and RQ4: What is the effect of incorporating prior knowledge under limited context?, appear in App. and App. G.4 respectively. We first detail datasets, metrics, expert simulation, and baselines, then present results per RQ. Full implementation details are in App.
Datasets. We use three medical datasets HAM1 0000 (Abhishek al., 2025 Tschandl et al. 2018) (dermatology). Blood Cells (Acevedo et al., (020)) (microscopy), and Liver tumours (Bilic et al. 2023) Saddiolggy--panniing realistic, high-stakes modalities, plus ImageNet- Steyvers al. 2022) with real human annotations.
Blood
Evaluation under variable deferral budgets. The default operating point gu (x, F) 2 max; ga (x) hides trade-offs across deferral budgets. We therefore the budgeted multi-expert inference from Remark 4.1 rank by the combined margin A Axxx), set r(d) to the (1 (1-d)-quantil) of (A* (A*));)) on the current test split, and defer those with A*xx) (dd) (routed to Ex)())); otherwise predict with the model. We summarise performance with:
AURSAC: area under the system accuracy (correctness of the composite model- expert system: expert accuracy on deferred cases and model accuracy on non-deferred cases) SysAcc(d) curve. AURDAC: area under the expert accuracy deferred cases ExpAcc(d) curve.
8

---

Published as a conference paper at ICLR 2026
AURSAC CESSAC(duiin dmmxx) Jdma (ds dd
Sdmm ExpAcc(d) dd dmax
AURDAC (dmia dmax))
dmax dani
dmii
for any [dmin; dmax] - [0, 1), where integrals are approximated numerically.
Experts. We evaluate on (i) real human annotators (ImageNet-16H) and (ii) a simulation that extends TTilor et al. (2024) with instance-dependent accuracy and correlated error types. Instance difficulty is tied to class prototypicality; we consider two archetypes: Variable Specialists (accuracy varies sharply with difficulty) and Stable Specialists (flatter profiles). As this framework explicitly models instance-level correctness, it theoretically favours query- corry--ondi population encoders (L2D-Pop), making our comparisons conservative for IFD. Full details are in App.
Baselines. We compare IFD against:
L2D-Pop (Tillor et al. 2024 y. SOTA appulation-adapiiv L2D with both cuery-con (QC)
and query-in indrreeddeeeddnn (QI) context encoders.
Multi-Experr L2D (Verma et al.] 2023): A fixed-expert baseline (ID only; not applicable to unseen experts). This baseline exists measure the improvement of population-adaptie methods.
5.1 RQ1: PERFORMANCE COMPARISON ACROSS EXPERT DIVERSITY
Table 1: Performance of IFD, L2D-Pop (QI, QC variants), and Multi- Expriit L2D on healthcare datasets with realistic simulated experts and on ImageNet- 16H with real human expert annotations (and scaling image noise). Results are -tdd AURSAC and AURDAC (abbreviated as SAC and DAC, respectively) across 10 sampled cohorts. ASAC reports IFD's improvement over the next best method. Values in bold denote column- best performance.
Expert/
Expert Dist.
IFD
L.D-Pop (QC)
L2D-Pop (QI)
Multi-Exp. L,D
Dataset
Nolse Level
(dan)
Telercaa
Telorcd M00
ASAC
SACT
DACT
SACT
DACT
SACT
DACT
SACT
DACT
HAM10000
Stable
86+.01
.88t.03
866+.02
85+.02
.86+.01
87+.02
88+
84+.01
3/4 IDIOOD Experts
O0D
+.02
86+.01
86tt.0
84+.02
83+.02
84+.01
83+.01
N/A
N/A
N 10S Cntx Preds
Viriable
+.02
S0+.03
78+
78+.03
.61+003
Dermatolopical
00D
+004
.77..04
69s.99
73
612.03
732.03
60s.05
N/A
N/A
4/4 [DIOOD Exports
R1 ooo Cella
Stable
00D
+.01
88:.01
85s.03
88a
87+:01
81+.02
88+
80s.01
831.01
N/A
.81.011
N/A
N25 = Catx Preds
Variable
ID
+:03
81s.01
.73..03
.7
62+.03
.77+.01
.59..02
.75+.01
.68.01
Microscopic
00D
+.06
.80..01
.69.00
.74
.54..03
.74a.01
.52a.02
N/A
N/A
Live: tumouus
Stable
ID
+.01
.881.01
.81..02
87a.01
762.01
861.01
.772.01
.86
.66.02
5/6 IDIOD Experts
00D
+.01
.88..01
.11..02
874
.6t.02
86ct 01
.772.01
N/A
N/A
165 Preds
Variable
ID
+.08
.77+.01
.6t..02
.6t 01
442.02
.6tt.01
.4+.02
.67+001
.3t.01
Radiological Img.
.76a.01
.88
.681.000
N/A
ImageNet-I2 611
Noise
00D
+.02
+:01
.76e.01
.762.01
.91..02
.73a.02
.724002
.1s.01
.75+.01
.74..01
.924.02
89s.01
N/A
.0+.02
N/A
2/2 ID/OOD Experts Ng 240 Cntx Preds
Noise 110
+:03
.691.01
801
00D
+.03
.71..02
.82+.02
.67+02
.80e.02
.68+.01
.0+.03
N/A
N/A
Benchmarking
Noise 125
ID
+.02
.64..02
.692.04
.59+.01
.64+.03
.62.00
.66+.03
.54.03
.63..02
00D
+.03
.662.02
.74t.04
.61+.02
.67.002
.634.01
.688.02
N/A
N/A
We compare IFD and baselines on medical datasets with realistic simulated experts, and on ImageNet-16H 16H with real annotators. Table reports AURSAC(O, and AURDAC(0, 1) (abbre- viated as SAC and DAC, respectively) for ID and OOD experts.
Findings. IFD consistently matches or exceeds alternatives. For Stable experts, all methods are close, with IFD best or tied on SAC/DAC both ID and OOD. The largest margins appear with Vari- able experts, where diversity and instance-dependence make adaptation harder: IFD improves both overall system accuracy (SAC) and the quality of deferred cases (DAC). Notably, L2D-Pop (QC) and L2D-PPp (QI) exhibit sharper OOD drop-off. Beyond accuracy, IFD is amnotation-efficien.. L2D-Pop requires E x N query annotations, whereas IFD uses only context labels Eg E. with N > E (c.g., Blood Cells: = NE =12),, this yields substantial savings.
5.2 RQ2: ROBUSTNESS UNDER INPUT DISTRIBUTION SHIFT
To test robustness to input shift, we use ImageNet-1$5 (Steyvers et al. 2022). which provides
human annotations for 16 classes at four degradation levels (80, 95, 110, 125). Annotators

---

Published as a conference paper at ICLR 2026
highly homogeneous (avg. pairwise correlation 0.93), apply Selective Annotation Sampling (App. C.4 to induce specialisation while preserving human error patterns. We define four groups (Animals. Transport, Household, Tools/Appliances), draw two ID experts from Animals/Houschold and two OOD experts from Transport/Toole (App. C.5 The backbone is pretrained on 10k Ima- geNet images disjoint from ImageNet-11 6H. All methods train the system ID experts at noise 80 and are evaluated on ID/OOD experts across all other levels.
Findings. As input images are increasingly corrupted, IFD maintains strong performance and con- sistently outperforms all baselines for both ID and OOD experts, with gains most pronounced at higher noise (T. This robustness follows from representation design: both L2D-Pop (QC) and L2D-Pop (QI) build latent expert representations by embedding the (potentially distorted) context images, so corruption propagates into the rejector, whereas IFD uses an explicit, query-independent expert profile that avoids re-embedding noisy inputs and yields more stable routing.
CONCLUSION AND LIMITATIONS
We exposed structural failure in population-adaptive L2D: identity-conditioned shortcuts that vi- olate permutation symmetry and make rejectors brittle to unseen/OOD experts. identity-Free De- ferral (IFD) removes identity channels by design- -uiingssn&& a query-independert Bayesian per-class profile, a low-dimensional role-indexed state, and an uncertainty-aware context-only loss -ielding architectural permutation invariance, smaller effective hypothesis class, and an interpretable peak- competence rule. Across medical imaging benchmarks and ImageNet- 16H under input shift, IFD matches exceeds I.2D--opp, with the largest gains OOD-unseen experts and in scarce-label regimes; it eliminates query-time expert labels and improves monotonically with additional context.
Limitations. IFD summarises each expert with query-independent, per-class correctness profile. This brings data efficiency and symmetry, but does not capture within-class, instance-level variation in competence, so for highly case-sensitive experts the class-level summary may misestimate the benefit of deferral on particular instances. We accounted for this in our experiments via instance- dependent expert prediction generation (and with real annotator variability on ImageNet-16H)) de- spite this stronger stress test, IFD still outperformed query-conditioned baselines, suggesting its explicit profiles support robust routing even without per-instance expert modelling.
We emphasise that our formal invariance results target coherent relabellings (one permutation ap- plied consistently to all class-indexed objects). Real OOD shifts can be weaker or more complex (e.g. expert-only reindexings, partial remappings that affect a subset of classes, or incoherent shifts in which expert and model posteriors drift independently). While our proofs do not directly treat these weaker transforms, the architectural diagnosis (identity leakage) together with Cor. [..1 ex- plains the practical failure mode: any learned dependence on absolute class coordinates makes the rejector sensitive which coordinate carries given competence value, so permuting or perturbing coordinates can change the defer/predict decision.
Future work. Two complementary paths can reduce this vulnerability: (i) architectural remedies that enforce permutation-awarr structure (weight-tying, permutation-equivariaatt encoders, or role- indexed interfaces like IFD) to eliminate absolute-inder channels: and (ii) training remedies such as within-episode randomised label permutations. targeted remaaiinr augmentations, or regu- larisers that penalise sensitivity coordinate swaps. Together these directions aim turn the ar- chitectural certificate of risk into concrete, deployable robustness; a systematic empirical/theoretical study is left future work.
ACKNOWLEDGEMENTS
We sincerely thank the anonymous reviewers for their thoughtful feedback and constructive criticism of the manuscript. This work was supported by the Engineering and Physical Sciences Research Council. JS and YI are funded by the EPSRC Centre for Doctoral Training in Health Data Science [EP/S02428X/1]] AN, CO and PS acknowledge EPSRC Turing AI Fellowship: Ultra Sound Multi- Modal Video-based Human-Machinn Collaboration [EP/X040186/)
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Kumar Abhishek, Aditi Jain, and Ghassan Hamarneh. Investigating the quality of DermaMNIST and Fitzpatrick17k dermatological image datasets. 12(1):196, February 2025. ISSN 2052- 4463. doi: 10. 10.11000444197--05--033222 URL BRLhhtps:// www www.atttee...mmrraataeeeeaaeeeee
s 025597-025-2 04382-3
Andrea Acevedo, Anna Merino, Santiago Alférez, Ángel Molina, Laura Boldú, and José Rodellar. A dataset of microscopic peripheral blood cell images for development of automatic recognition systems. Data in Brief, 30:105474, 2020. ISSN 2352-3409. doi: http:://bo..rgg/1..0111/// 2020. 105474. URL httpt: httpp/:/ttww...tiiiee..o.ee. http::/wwwwwcccccceiiccccccc rect. com com./sciennce/articll//...
$235234092030360
Jean V Alves, Diogo Leitão, Sérgio Jesus, Marco OP Sampaio, Pedro Saleiro, Mário AT Figueiredo, and Pedro Bizarro. Fifar: A fraud detection dataset for learning to defer. arXiv preprint 1rXiv:2312.132/8 2023.
Jean V. Alves, Diogo Leitão, Sérgio Jesus, Marco O. P. Sampaio, Javier Liébana, Pedro Saleiro, Mário A. T. Figueiredo, and Pedro Bizarro. Cost-sensitive learning to defer to multiple experts with workload constraints, 2024. URL
Anastasios N Angelopoulos and Stephen Bates. A gentle introduction conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 2021.
Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, Fabian Lohöfer, Julian Walter Holch, Wieland Sommer, Felix Hofmann, Alexandre Hostettler, Naama Lev-Cohain, Michal Drozdzal, Michal Marianne Amitai, Refael Vivanti, Jacob Sosna, Ivan Ezhov, Anjany Sekuboyina, Fernando Navarro, Florian Kofler, Johannes C. Paetzold, Suprosanna Shit, Xiaobin Hu, Jana Lipková, Markus Rempfler, Marie Piraud, Jan Kirschke, Benedikt Wiestler, Zhi- heng Zhang. Christian Hülsemeyer. Marcel Beetz, Florian Ettlinger. Michela Antonelli, Woong Bac, Míriam Bellver, Lei Bi. Hao Chen, Grzegorz Chlebus, Erik B. Dam, Qi Dou, Chi-Wing Fu, Bogdan Georgescu, Xavier Giró Nieto, Felix Gruen, Xu Han, Pheng-Ann Heng. Jürgen Hesser, Jan Hendrik Moltz, Christian Igel, Fabian Isensee, Paul Jäger, Fucang Jia, Krishna Chaitanya Kaluva, Mahendra Khened, Ildoo Kim, Jae-Hun Kim, Sungwoong Kim, Simon Kohl, Tomasz Konopezynski, Avinash Kori, Ganapathy Krishnamurthi, Fan Li, Hongchao Li, Junbo Li, Xi- aomeng Li, John Lowengrub, Jun Ma, Klaus Maier-Hein, Kevis-Kokitsi Maninis, Hans Meine, Dorit Merhof, Akshay Pai, Mathias Perslev, Jens Petersen, Jordi Pont-Tuset, Jin Qi, Xiaojuan Qi, Oliver Rippel, Karsten Roth, Ignacio Sarasua, Andrea Schenk, Zengming Shen. Jordi Torres, Christian Wachinger, Chunliang Wang. Leon Weninger, Jianrong Wu, Daguang Xu, Xiaoping Yang, Simon Chun-Ho Yu, Yading Yuan, Miao Yue, Liping Zhang. Jorge Cardoso, Spyridon Bakas, Rickmer Braren, Volker Heinemann, Christopher Pal, An Tang, Samuel Kadoury, Lue Soler, Bram van Ginneken, Hayit Greenspan, Leo Joskowicz, and Bjoemn Menze. The liver tumor segmentation benchmark (lits). Medical Image Analysis, 84:6080,, 2023. ISSN 1361-8415, doi: https://doi.org/10.. 1wtp://0...00110.........m 2022. 102680. URL htttps://mae www.:wwwweiinnnneeeeeeeee.. com/ sst.enee/arriccee//1///333331333
Benjamin Bloem-Reddy and Yee Whye Teh. Probabilistic symmerrees and invariant neural networks. Journal of Machine Learning Research, 21(90):1-61, 2020.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021.
C. Chow. On optimum recognition emror and reject tradeoff. IEEE Transactions on Information Theory, 16(1):41-46, 1970. doi: 10.1 10544005
C. K. Chow. An optimum character recognition system using decision functions. IRE Transactions Electronic Computers, EC-6(4):247-254, 1957. doi: 00.1109/EC..955..2220000
Abigail R Colson and Roger M Cooke. Expert elicitation: using the classical model to validate experts' judgments. Review Environmental Economics and Policy, 2018.

---

Published as a conference paper at ICLR 2026
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, pp. 67-82. Springer, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks, 2017. URL
org
Marta Garnelo, Dan Rosenbaum, Chris J. Maddison, Tiago Ramalho, David Saxton, Murray Shana- han, Yee Whye Teh, Danilo J. Rezende, and S. M. Ali Eslami. Conditional neural processes, 2018. URL org. 807.01613
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference
confeerree on machine learning, pp. 1321-1330. PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015. htttps://///// wwtpr/iiii.oo.vveeoooooeooeeeeee abs 1512. 03385
Patrick Hemmer, Lukas Thede, Michael Vössing, Johannes Jakubik, and Niklas Kühl. Learning to defer with limited expert predictions, 2023. URL
07306
Muhammad Abdullah Jamal, Guo-Jun Qi, and Mubarak Shah. Task-agnostic meta-learning for few-shot learning, 2018. URL jRttps: wwwp:///......///....o.. /ttp:p/://aaaal.oooo//////1...
07722
Shalmali Joshi, Sonali Parbhoo, and Finale Doshi-Velez, Learning-to-defer for sequential medical decision-making under uncertainty. arXiv preprint arXiv:2109.06312, 2021.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garelo,, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations, 2019a. UR//nttps::/opeereevieee
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes, 2019b. URL
abs/1901. 05761
Minyoung Kim and Timothy Hospedales. hierarchical bayesian model for few-shot meta learning. In The Twelfth International Conference Learning Representations, 2024. URL https:
/ //openreview. wwwprrevieww..tt/oooeeeeeeeee
Thodoris Lykouris and Wentao Weng. Learning to defer in content moderation: The human-ai interplay, 2024. URL https::/ /tpps://arxvv..o www.swwwwww.gaee... 2tt.//aa//2222222323
David Madras, Toniann Pitassi, and Richard Zemel. Predict responsibly: Improving fairness and accuracy by learning defer, 2018.
http:///gaaaaaaaaaaaa/// 06664
Anqi Mao, Christopher Mohri, Mehryar Mohri, and Yutao Zhong. Two-stage learning to defer with
multiple experts. In Thirty-sevemth Conference on Neural Information Processing Systems, 2023. URL
net. www.:nntew.oortttoooooooo id= sHOT4b2
Anqi Mao, Mehryar Mohri, and Yutao Zhong. Principled approaches for learning to defer with multiple experts, 2024. htthttts:/// /ttp/viiivvv.////////ee//aaaalll org/
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning, 2019. URL
Yannis Montreuil, Shu Heng Yeo, Axel Carlier, Lai Xing Ng. and Wei Tsang Ooi. A two-stage learning-to-deferr approach for multi-task learning, arXiv preprint arX2v:2410.15729, 2024.
Yannis Montreuil, Axel Carlier, Lai Xing Ng, and Wei Tsang Ooi. Adversarial robustness in two-
stage dearning-to-defer Algorithms and guarantees. arXiv preprint arXiv:2502.010027 2025a.
Yannis Montreuil, Axel Carlier, Lai Xing Ng, and Wei Tsang Ooi. Why ask one when you can ask k? two-stage camming-to-defee to a set of experts, 2025b. URL https: /ttpss//arrreee arxiv. org/ abs/ 2504 12988
12

---

Published as a conference paper at ICLR 2026
Yannis Montreuil, Axel Carlier, Lai Xing Ng, and Wei Tsang Ooi. One-stage top-k learning- defere Score-based surrogates with theoretical guarantees. arXiv preprint 2x0x:25005.00160) 2025c.
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert, 2021.
URL https://aaxix: org/ 20s/2006.00000 01862
Cuong C Nguyen, Thanh- Toan Do, and Gustavo Carneiro. Probabilistic learning to defer: Handling missing expert annotations and controlling workload distribution. In The Thirteenth International Conference on Learning Representations, 2025.
Nastaran Okati, Abir De, and Manuel Rodriguez. Differentiable learning under triage. Advances in Neural Information Processing Systems, 34:9140-9111 9151, 2021.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks, 2019. URL
18C1. 04381
Mark Steyvers, Heliodoro Tejeda, Gavin Kerrigan, and Padhraic Smyth. Bayesian modeling of human-ai complementarity. Proceedings of the National Academy of Sciences, 119(11): e2111547119, 2022.
Joshua Strong, Qianhui Men, and J Alison Noble. Trustworthy and practical AI for healtheare: A guided deferral system with large language models. Proc. Conf. AAAI Artif. Intell., 39(27): 28413-28421, April 2025.
Dharmesh Tailor, Aditya Patra, Rajeev Verma, Putra Manggala, and Eric Nalisnick. Learning to defer population: meta-learning approach, 2024. URL 2403.02 02683]
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks, 2020. URL
11946
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5(1), August 2018. ISSN 2052-4463. doi: 10. 1..100/sdata.2018.... URL http:/ /ttp///xx./// doi org/ 10. 1038 sdata. 2018.161
Rajeev Verma, Daniel Barrejón, and Eric Nalisnick. Learning to defer to multiple experts: Con- sistent surrogate losses, confidence calibration, and conformal ensembles, 2023. http::
/arziiv.rrg/abs/2221 16953
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world. Springer, 2005.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander Smola. Deep sets, 2018.
wttp://///...b/.....//// 06114
Zheng Zhang, Wenjie Ai, Kevin Wells, David Rosewarne, Thanh-Tomm Do, and Gustavo Cameiro. Learning to complement and to defer to multiple users. European Conference on Computer Vision, pp. 144-162. Springer, 2025.
13

---

Published as a conference paper at ICLR 2026
CONTENTS OF APPENDIX
Extended Literature Review
15
Instance-Dependent Expert Simulation Protocol
15
B
Motivation for a More Realistic Simulation
16
B.2
Methodology
16
B.2.1
Quantifying Instance prototypicality
16
B.2.2
Instance-Dependent Accuracy modelling
17
B 2.3
Stochastic and Correlated Prediction Generation
17
B.3
Expert Archetypes for Evaluation
18
Evaluation using Real-World Human Annotations
18
Motivation and Challenges
Analysis and Quantification of Annotator Homogeneity
19
Preparation of the Annotation Pool (Filtering)
19
Semi-Symthetic Expert Generation: Selective Annotation Sampling
19
C.5
Experimental Designs for OOD generalisation
20
C.6
Verification of Synthesis and the Constraint Eifect
20
General Proofs
D.1 Proof of Proposition 4.1
21
D.2 Proof of Proposition
22
D.3 Theoretical Characterisations of the IFD Deferra Rule
25
Permutation Symmetry, Invariance, and generalisation of Identity-Free Deferral
27
E.
Setup. Notation, and Problem Formulation
27
Why L2D-Pop Fails on OOD Experts: Query-Independent Variant
29
E.3
Why L2D-Pop Fails on OOD Experts: Query-Conditioned Variant
30
ID: Architecture and Invariance Proof
30
E.5
Plain-Language Takeaways
32
Experimental Details
32
Additional Experiments
32
G.1
Toy Example: Identity Leakage vs. Invariance
32
Ablation: Uncertainty-Aware LCB Weighting
.eggaa..........................
G.3
RQ3: Performance Scaling with Limited Context Data
34
G.4
Robustness Prior Misspeciffication
35
Datasets
36
14

---

Published as a conference paper at ICLR 2026
A EXTENDED LITERATURE REVIEW
Selective prediction and rejection. Classical selective prediction (i.e., rejection learning) trains a
predictor that may abstain when confidence is low (Chow 1957 1970 Cortes al. 2016). Beyond calibrated abstention (Guo et al., 2017), modern work studies accuracy-coverage trade-off's, and
cist-controlling prediction sets (Vovk et al. 2005; Angelopoulos & Bates 2021
These threads
motivate deferral policies that reason about who should act.
Learning to Defer (L2D). L2D augments selective prediction by comparing the model's 's prediction
to (costed) expert's decision, optimising system-level loss (Madras et al. [2018] MMzannarr & Son-
tag 2021). Extensions cover multiple experts (Verma et al., 2023] Mao et al. (002)), triage/budget
constraints (Okaii et al. 2021). partial or noisy expert signals (Hemmer et al. 2023), top-k setups
Montreuil et al. (2025c bb. and two-stage training that treats the classifier distinct modules where
the classifier is frozen (Mao et al. 2023; Montreuil et al. (2024). Applications span healthcare (Joshi
ct al. 2021 |0 Strong et al. 2025). fraud, moderation and safety (Alves et al. 2023 Lykouris & Weng
2024
Montreuil al.) (025a).
Population adaptation and unseen experts. When the test-time expert is unseen, L2D-Pop meta- learns expert representation from few-shot context and conditions the deferral rejector it (Tai- lor et al.] 2024). Query- indeeededdde encoders summarise the context; query-conditioned variants cross-attend with the current input. Related meta-learning formulations (neural processes and amor-
tised Bayesian adaptation) provide conpextttottprreseentttion mappings (Kim et al.] 2019a Kim & unseen experts but differs in representation: instead of learning holistic latent, we construct an explicit Bayesian, per-class com- petence profile and expose only role-indexed, identity-free statistics to the rejector. This architectural choice enforces permutation symmetry and mitigates deenity-conditionee shortcuts.
Hospedales 2024). Our approach shares the jnjeetivee-daptiin
Limited expert labels and annotation efficiency. practical barrier to L,2 is acquiring query- time expert responses. Recent work infers missing labels with EM or latent-variable models (Nguyen et al., 2025), learns expert embeddings to synthesise predictions (Hemmer et al. (202), or introduces cost-sensitive formulations to manage expert workload (Alves et 20244. Our method avoids query-time labels altogether: we train the rejector with a context-only, imcertaint--awar sur- gegate-using lower-confidence bounds from the expert profile (64.3)-while while retaining the ability to inject priors (4.44) To our knowledge, this combination (i) explicit Bayesian expert summarics, (ii) identity-free gating, and (iii) context-only supervision is novel in I.21).
Human-AI complements beyond deferral. Another line complements rather than defers to hu- mans e.g., routing to humans with decision support, combining predictions through learned aggregation, allocation, or explanation mechanisms (Zhang et al. (2025). These works differ in objective (joint performance, trust, workload) and interface (advice vs. hand-off); our focus is explicit hand-off under class-aware competence.
Agnostic and permutation-aware learning. IFD echoes agnostic/meta-leaamee that targets robust- ness to task/domain variation (c.g., MAML, TAML, and agnostic federated learning) (Finn et al., 2017 Jamal et al. 2018 Mohri al. 2019). Technically, our contribution architectural: we en- force invariance to coherent class re relabellings by operating on role-indexed statistics and symmetric rejectors. This mirrors recent interest in symmetry, invariance and equivariance as inductive biases for OOD generalisation (Bloem- & Teh 2020 Bronstein al. 2021). We show that standard L22DPop parameterisations are genericully non-invariant.
Positioning, Compared to 1.2D-Pop. our explicit profile: (i) reduces variance small context sizes, (ii) removes identity channels by construction, (iii) naturally incorporates informative priors, and (iv) scales with additional context at test time. Compared to limited-label L2D variants, eliminate query-time labels while preserving adaptation to unseen experts. Compared to complement-style collaboration, target the Bayes comparison between the model's confidence and the expert's competence and implement it with identity-free features.
B INSTANCE-DEPENDEE EXPERT SIMULATION PROTOCOL
To enhance the fidelity of our evaluation and address the limitations of expert simulations that as- sume uniform intra-class accuracy, we developed more realistic, instance-dependent expert simu-
15

---

Published as a conference paper at ICLR 2026
lation protocol than that utilised in Tailor et al. (2024). This protocol models expert's accuracy as a function of instance difficulty, allowing for a more granular and challenging assessment of deferral strategies.
B.1 MOTIVATION FOR A MORE REALISTIC SIMULATION
Learning to Defer (L2D) systems are most valuable when they can adapt to the nuanced capabilities of human experts. Prior work, including the baseline simulation protocol from Tailor et al.) (2024). often models expert performance at the class level. For instance, an expert might be assigned fixed accuracy of 95% their specialty class and lower accuracy all other classes.
This abstraction, while useful for studying generalisation to OOD expert specialisations, overlooks a critical aspect of real-world expertise: performance might not be uniform within a class. An expert radiologist, for example, will find it trivial to identify a classic, textbook example of a tumour (a "prototypical" case) but may struggle with an atypical presentation of the same disease.
By assuming fixed per-class accuracy, simpler simulations cannot test deferral system's 's ability to distinguish between easy and hard instances for an expert. To address this. we introduce simulation framework where an expert's ss accuracy for any given sample is direct function of that sample's prototypicality within its true class. This allows us to model experts with more complex and realistic behaviours.
more challenging testbed for our method. important to note that this instance-dependent simulation protocol creates more challenging evaluation landscape specifically for our proposed method, IFD, while theoretically favouring the baseline, L2D-Pop. The L2D-Pop framework is designed to learn holistic, latent representation of an expert that implicitly models the complex function mapping an instance to a prediction (P(M) Y, X)). Our new simulation generates data that explicitly follows such instance-dependent function. In contrast, IFD makes a principled abstrac- tion, modelling performance at the class level (P(M)Y) for robustness and generalisability. There- fore, by evaluating IFD in setting where the ground-truth expert behaviour is instance-dependent, we are stress-testing our method against a baseline whose core assumptions are better aligned with the simulation's 's design. Strong performance under these conditions would provide compelling evi- dence for the robustness of our approach.
B.2 METHODOLOGY
Our simulation methodology consists of three main stages: (1) Quantifying instance prototypicality using a learned feature space, (2) modelling sitaace-deeendder expert accuracy using this proto- typicality score, and (3) Generating stochastic, correlated expert predictions based on the modelled accuracy.
B.2.1 QUANTIFYING INSTANCE PROTOTYPICALITY
We define the prototypicality of an instance as its proximity to the central tendency of its class in a learned feature space. To achieve this, first train standard classifier the task dataset serve as a feature extractor.
Let FF be the feature extractor (i.e., the penultimate layer of trained classifier). For each image x; in the training dataset Dtrain, we extract its feature vector z; = F(x;).
1. Feature Extraction:
2. Class Centroid Calculation: For each class k {1...., ......,K K}, we compute the mean feature vector, or centroid Cs., from all training samples belonging to that class:
C₁
Σ
F(x,)
(13)
Perain, X;ED
where D is the subset of training data with label k.
3. Prototypicality Score: The prototypicality p(x;) of instance x; with true label y A is defined by its normalised distance to its class centroid. First, we find the maximum distance dmaz, k between any training instance of class ls and its centroid:
dmax.k
max
[|F(xx) c
(14)
XDErann
16

---

Published as a conference paper at ICLR 2026
The prototypicality score p(x,) [0, 1] is then calculated as:
p(x,) min (1. [[F(x;) c#
(15)
dman,k te
score of p(x:) = 1 indicates a highly prototypical (easy) instance, while p(x;) 22 O indicates a highly atypical (hard) instance.
B.2.2 INSTANCE-DEPENDERN ACCURACY MODELLING
We model an expert's instance-specife accuracy using sigmoid function, which takes the proto- typicality score as input. This allows us define expert behaviour with two intuitive parameters. Let A(x; F)) be the accuracy of expert E on instance We model this as:
A(x,EE = aa p(x,) + 8) 1 1- (a--(xp(x)+8)
(16)
where:
a controls the steepness of the curve. A high a models an expert whose performance is highly sensitive to instance difficulty (i.e., large gap between performance easy hard cases). A low a models an expert with more consistent performance.
3 controls the horizontal shift the curve, effectively setting the baseline accuracy for most atypical cases (p 0).
To make these parameters more intuitive, we derive them from two target accuracies: acCensy (the desired accuracy for the most prototypical cases, p 1) and acChart (the desired accuracy for the most atypical cases, p = 0). Using the inverse sigmoid (logit) function, we solve for a and f::
-1-1 -)
3 = logit (acChard) = log
(17)
aCChard
Q logit (aCCeaCCC))))))))
B
(18)
B.2.3 STOCHASTIC AND CORRELATED PREDICTION GENERATION
The final simulation process for an instance x; with true label yr, evaluated against population of experts, is follows:
1. Compute the instance's prototypicality score p(x:).
2. Generate single instance-specifee random number u, (O, 1). This value represents latent solvability factor for the instance and will be used for all experts.
3. For each expert E the population:
(a) Determine if the instance belongs to the expert's specialty class (y/ kg).
(b) Select the appropriate parameters: (aspect Sspec) if y/
kg.
(O specy Snon-spec) otherwise.
(c) Calculate the expert's instance-specific accuracy A A(x: using the selected parame- ters and equation (4).
(d) uy A(x; E)) the expert's prediction is correct (m, = 31).
(e) Otherwise, the expert's 's prediction incorrect. The specific incorrect class, mi is then drawn from a shared, instance-specific error distribution.
Correlating the Type of Error. A simple simulation might draw an incorrect prediction uniformly from the set of wrong answers. To create more realistic error patterns, where experts tend to agree on the same incorrect class for confusing instance, we model the eтrо distribution. When an expert fails to predict correctly (i.c., u4 2 (x)),,) we make the probability of predicting an incorrect class j inversely proportional to the feature space distance to that class's centroid. This ensures that an instance is most likely to be confused with visually or semantically similar class. The incorrect prediction m, is drawn from the distribution:
e -x11((x))-511
P(m; jlinncorect, x:) softmax((11F(()) -2il2)
(19)
EiEr((N)) -1117(x)--
17

---

Published as a conference paper at ICLR 2026
for all incorrect classes 3 (v6). Since this distribution depends only on the instance x; and the class centroids, all experts who make an error on this instance will sample their incorrect prediction from the same distribution, thus correlating their specific mistakes.
Reproducibility. The instance-specifie variable u, is generated deterministically as a function of the instance index and global seed, so that all experts evaluating the same instance share the same correctness draw while results remain reproducible.
Sampling from the shared error distribution. When u; 2 A(x; the incorrect class is sampled independently for each expert from the same, instance-specific distribution in Eq. (7), with the true class masked (probability zero). This induces correlated types across experts without forcing identical wrong labels. If one wishes to enforce identical wrong labels, a single shared draw from the distribution can be used for all experts on that instance.
B.3 EXPERT ARCHETYPES FOR EVALUATION
To rigorously test the capabilities of deferral system, we simulate populations composed of distinct expert archetypes, as summarised in Table These archetypes are defined by their performance on their specialty class versus non-specialty classes, and critically, how sensitive their performance is instance difficulty (i.e., the gap between aCCeasy and acChard).
Parameters defining the expert archetypes used in the simulations. Archetypes are defined by the target accuracy on the easiest (acCensy. = 1) and hardest (aCChard, = 0) instances, which determine the sigmoid parameters a and via Equations (5) and (6).
Table 2:
Archetype
Context
aCCeasy
aCChard
Specialty 0.99
0.50
4.595
0.000
Variable Specialist
Non-Specialty
0.80
0.20
2.773
-1.386
Specialty 0.99
0.85
2.860
1.735
Stable Specialist
Non- con-Specialty
0.85
0.70
0.886
0.847
The Variable Specialist represents an expert whose accuracy drops significantly when faced with atypical examples. In contrast, the Stable Specialist maintains high accuracy even on difficult instances, representing more consistently reliable expert.
C EVALUATION USING WOALD HUMAN ANNOTATIONS
A critical requirement for validating L.2D systems, particularly those intended for high-stakes do- mains, is the assessment of performance using real-world human annotations. While simulated experts (Appendix B) allow for controlled stress-testing, benchmarking against actual human be- haviour ensures robustness to the nuances inherent in human decision-making. We utilise the ImageNet-16it 16H (Steyvers et al. 2022) dataset, which provides multiple human annotations per im- age.
C.I MOTIVATION AND CHALLENGES
The core contribution of IFD is its robust generalisation OOD experts those whose specialisa- tion profiles differ significantly from experts encountered during training. Evaluating this capability necessitates datasets containing diverse, specialised expert behaviours. However, standard multi- annotator datasets, often collected via crowd-sourcing for general recognition tasks, frequently ex- hibit high homogeneity in annotator behaviour. This homogeneity precludes their direct use in evaluating OOD specialisation generalisation, there is insufficient diversity define distinct ID and OOD specialisation profiles.
To address this limitation while adhering to the requirement for real-world data, employ semi- synthetic evaluation strategy. This strategy introduces the necessary specialisation structure for OOD testing while strictly utilising the real, instance-dependent error patterns present in the datasets.
18

---

Published as a conference paper at ICLR 2026
C.2 ANALYSIS AND QUANTIFICATION OF ANNOTATOR HOMOGENEITY
We first empirically assessed the diversity of specialisation within the raw datasets to justify the semi-synthetic approach.
Methodology. We define the "behavioural fingerprint" of an annotator as their normalised confu- sion matrix. This matrix captures P(Chosen Label|True Label), representing both their classwise accuracy and their specific patterns of mistakes. The analysis proceeds as follows:
1. Fingerprint Calculation:
The confusion matrix for each individual annotator calculated and normalised by the true class (rows).
2. Vectorisation: The K X K normalised matrix is flattened into K vector.
3. Homogeneity Quantification: We calculate the average pairwise correlation between all annotator vectors.
4. Characterisation: K-Means clustering is applied to the vectors to identify distinct be- havioural subgroups within the population.
Findings. The analysis confirmed extreme homogeneity in specialisation profiles, with average pairwise correlation 0.926 for ImageNet- 16H. Clustering analysis revealed that the primary axis of variation among annotators overall skill level (e.g., separating high-accuracy annotators from low-accuracy or noisy annotators), rather than distinct areas of specialisation.
Implications. The high homogeneity empirically demonstrates that these datasets cannot be used "as-is" to evaluate generalisation across diverse specialisation profiles, validating the necessity of the semi-synthetic approach detailed below.
C.3 PREPARATION OF THE ANNOTATION POOL (FILTERING)
To ensure that synthesised expert behaviours are representative of competent experts rather than noise or inattention, the raw annotation pool was filtered prior to synthesis. Annotators identified in the clustering analysis (Section 2 belonging low-skill clusters (e.g.. those performing near random chance) were excluded. This filtering step ensures that when synthesised expert makes a mistake, the error drawn from a plausible mistake made by competent human annotator, maintaining the realism of the evaluation.
C.4
EXPERT GENERATION: SELECTIVE ANNOTATION SAMPLING
We employ a methodology termed Selective Annotation Sampling to synthesise specialised ex- perts. The defining principle of this approach is that does not introduce artificial errors. 100% of the annotations used in the evaluation are actual human responses provided for the specific image being considered.
Algo Algorithm Description. The synthesis process involves the following steps:
1. Define Target Profiles: We define the desired specialisation profiles (c.g., ID and OOD groups) and the target accuracies for specialty and non-specialty classes (e.g.. Anigh and Atou).
2. Organise the Annotation Pool:
For every image X, in the dataset, the available human annotations (from the filtered pool) are partitioned into two sets:
Correct Pool: Annotations matching the true label y:-
Incorrect.Pocke Annotations not matching yt-
3. Synthesise Expert Annotations: For a synthesised expert Ezyoth evaluating image X;:
(a) Determine the target accuracy Atarget based on the true label and the expert's de- fined profile.
(b) Stochastically determine the desired outcome:
Should Be-Correct
Bernoulli(
(c) Selective Sampling with Constraints:
19

---

Published as a conference paper at ICLR 2026
If the desired outcome is Correct: Attempt to sample from Correct.Pool If Correct.Pool is empty (a difficult image where all available humans erred), the expert must sample from Incorrect Pool.
If the desired outcome is Incorrect: Attempt to sample from Incorrect_Pool. Incorrect Pool empty (an easy image where all available humans were correct), the expert must sample from Correct.Pooli
This methodology generates experts with distinct specialisation profiles while ensuring that all errors authentic human mistakes, thereby preserving the crucial instance-dependent error patterns of the original data.
C.5 EXPERIMENTAL DESIGNS FOR OOD GENERALISATION
Using Selective Annotation Sampling, we construct several experiments test OOD generalisation on ImageNet-16H.
IMAGENET- 16H DESIGN: OOD GENERALISATION UNDER DISTRIBUTION SHIFT
ImageNet-16H contains 16 classes evaluated across four distinct levels of Gaussian noise added to the images, indexed 80 (lowest noise), 95, 110, and 125 (highest noise). This structure allows us to evaluate OOD generalisation concurrently with robustness to input distribution shift. We utilised Ahigh 0.99 and Alow 0.40.
Expert Group Definitions. We partitioned the 16 classes into four specialisation groups (indices correspond to the experimental mapping):
Group (Animals): Indices [1, 3, 4, 10, 14] (dog, bird, bear, elephant, cat).
Group 2 (Transport): Indices [5, 8, 9, 12, 13] (airplane, car, boat, truck, bicycle).
Group (Household); Indices [0, 6, 7, 11] (chair, clock, bottle, knife).
Group (Tools/Appliancee): Indices [2, 6, 15] (oven, keyboard, knife).
Note the overlap of "knife' (Index 6) between Group 3 and Group 4, reflecting potential overlaps in real-world expertise.
ID/OOD Split and Training Protocol. We utilised multi-group ID/OOD split:
ID Cohort (Training): Comprised of 4 synthesised experts: 2 specialised in Group (Animals) and 2 specialised in Group 3 (Household).
Comprised of 4 synthesised experts: specialised in Group (Transport) and specialised Group (Tools/Appliances)
OOD Cohort (Testing):
To evaluate robustness input distribution shift:
1. Classifier Pretraining:
The classifier backbone was pretrained on 10,000 standard Ima- geNet images disjoint from the ImageNet-16H dataset.
2. L2D Training:
The deferral policies were trained using annotations from the ID Cohort, exclusively on images with the lowest noise level (80).
3. Evaluation (Distribution Shift): The trained systems were evaluated on both the ID and OOD cohorts across all three noise levels (95, 110, 125) without retraining or fine-tuning on the higher noise levels.
C.6 VERIFICATION OF SYNTHESIS AND THE CONSTRAINT EFFECT
To verify the synthesis process, calculate the average realised confusion matrix for each synthe- sised cohort and compare the realised classwise accuracy to the target accuracy.
The Constraint Effect. During verification, it is frequently observed that the Realised Accuracy higher than the Target Accuracy particularly when the target is low (c.g., realised 0.75 when the target 0.60). This phenomenon, termed the "Constraint Effect," arises directly from the
20

---

Published as a conference paper at ICLR 2026
constraint that only existing human annotations can be utilised (Step 3c in Section C.4 4
If the algorithm aims for an incorrect answer on an image where all filtered human annotators were correct (Incorrect Pool is empty), it is forced to select the correct answer. This effect confirms the fidelity of the simulation to the real data, as it accurately reflects the underlying difficulty of the images for the human pool and avoids introducing artificial emors trivially easy instances.
D GENERAL PROOFS
Theorem D.1 (Hoeffding's Inequality). Let Zi...., .....,,, Zn be random independent, identically dis- tributed random variables, such that 0 s Z; S 1. Then,
Z: 1 Σ
Pr
E|Z]
u 2exp(-2ne
i=1
Theorem D.2 (Boole's Inequality). For any countable collection of events A1, A2,..., An in probability space,
Pr
U
A <
P(A)).
1=1
D.1 PROOF OF PROPOSITION 4.1
Proposition 4.1 (Peak-class identification). an arbitrary expert have true but unknown per-class accuracies 02 E 0, 1] (we drop the E notation here). Let Kbeut arg maxg On be the unique class of maximal expertise. As the number of context samples n. + 0G for each class k, the posterior mean estimates μ* converge almost surely Og, and the selected expertise class arg max Ma converges almost surely to Rbeat. Moreover, if each class is observed at least n. times, where
In (n(
11 2
with Aacc = Bx max 0x-
2 2(20))
then with probubility least - 8, the posterior means satisfy Wkaan > Ps for all ls of khesti that is, the correct expertise class is identified.
Proof. Step 1: Posterior convergence For each class k, the expert's true accuracy 0g E [0, 1] modelled with Beta prior:
O₂ ~ (eta((ss,
After observing ng samples for class k. which to are correct, the posterior becomes:
0x De Beta(ay + tv. Bx + + tx),
with posterior mean:
Okk
14
ag
Bx n2
By the Law of Large Numbers, the empirical accuracy Òs = I//1 converges almost surely to the true accuracy 0g as ni oo. Since the prior's influence vanishes in the limit, the posterior mean μ4 also converges almost surely to dg.
Step Consistent identification of the expert class Let kbest arg max; 0g be the unique best class. Since μlg a.=,, Ōz. we have: arg maxx 14g a.=,, khea. That is, the class selected by maximizing the posterior mean converges almost surely to the correct class.
Step Finite-sample guarantee using Hoeffding's inequality We apply Hoeffding's inequality (Thm. D.I.:
P ((
0x 2 s 2exp( (
To ensure correct identification, we require:
0
0
Ascc
Vk,
2
21

---

Published as a conference paper at ICLR 2026
where Aaco Okre
ht 8g. This guarantees:
Ok.
4k o Aoest-
Setting e = Dace 2 and assuming
2 for all k, Hoeffding gives:
P 3K s.t. Ōg. 02
Aac
S 2K exp
-2
Aacc
2
2
We want this probability I 8, so solve:
-2 (
Aacc
In
2K
2K exp
2 7
T1
2
Auee
Thus, with this many samples per class, we have Mkan pt for all + kbea with probability 1-5. 8.
D.2 PROOF OF PROPOSITION 4.3
Lemma D.1 (Recovery Rate from Misspecified Priors). Let 0 be the true accuracy of expert E for class ki and suppose potentially misspecified prior Beta( (a ,eeioo) is used. After observing ny context samples for class l resulting th correct predictions, the absolute error of the posterior mean estimate ml is bounded by:
1ng
0 S
Sprior
Ipprior 0g1 +
n
E)
00
Sprior + my
Sprior +
E
y
where Sprior
a prior
Oprior
Hpprior
oftirr
and ((E) Sprie
Furthermore, as ny - oo, the ny
posterior mean converges to the true accuracy: My + OF.
Proof. The posterior mean by after observing th successes in ny trials, given the prior (et . gprior ), is:
prior t a
prior
Ppeio + n
Using the definitions Sprior = 0&
prior
aprior
and ((()) = -t n we can rewrite the posterior mean:
prior
n m6(
k
My
Sprior + n
prior
n g
Sprior + n
S Sprior n
Recalling Mpprior = az
prior Spricess we have of Sprioe/ Substituting this gives:
nly (()
2y F SpriorMprior
Sprior + n
Sprior + n
E (())) ny 0x
Sprior
Sprior + n. Mprior +
Sprior + n
Let Uprior tn and Wdata a weighted average of the prior mean and the empirical accuracy:
Spece
Spacttnn Note that Wprior + Wdata
1. Thus, the posterior mean
Pyy (lpprioo//priic Winn ((E)
22

---

Published as a conference paper at ICLR 2026
Now consider the absolute error I 0% Since Wprior + Wdtaa 1, can write 0 peior +
Wdata )0
Myy 0 I(
pririor/prioo + Wdata 0(E)) - (Wpriir + W 00
Uiprioo (prrior
k + Wdata
(0% (0)=) 0
Applying the triangle inequality (a + bl S |a| + bbD:
l 0
S prrror (Aprior k + aa (0
Wprior Mpprior
0 + Wanaa 10 (
0%
(since Wprior, Wdata 2 0)
Substituting back the definitions of Wprior and Wdata yields the error bound:
Sprioe
l -
Ppprior 0
+
(
Sprior + By
Sprior
ny
This proves the first part of the proposition (i.e., bounds of the absolute error of the posterior mean estimate).
For the convergence analysis, consider the limit as ny
0o:
Sprioe
lim
Pprior
lim my +oc Sprior + ne
0
"
E
n
lim
Wdta
00
+00 Sprior + "
By the Law of Large Numbers (specifically, the Strong Law for Bernoulli trials), the empirical accuracy converges almost surely to the true accuracy:
0
E
OE
as ny - 0c
E
y
Therefore, taking the limit of the posterior mean:
Uprior/prior + Waata 0 ((
lim n +cc
Hy
lim n -200
lim
Uprior
Hprior +
lim
Widuaa
lim
n +00
n +00
no 1000000
(0) Pprior -1)-0
0
Thus, the posterior mean converges the true accuracy regardless of the initial prior specification.
Proposition 4.3 (Overcoming prior misspecification). Let E be the true accuracy of expert for class l. with prior Beta(aepi sprrior having mean Hprior and strength Sprior af + pprior For any O and 5 E (0, 1), the number of context samples ny needed to ensure the posterior mean my satisfies mn 0 c with probability at least 8 bounded by:
SSpiio Sprioo(PPepe ((ppior
0 e/2) 2 ll(2285) e c/2
E nl
max
22
The first term accounts for overcoming prior bias (relevant if Pprrio accounts for statistical uncertainty via Hoeffding's 's inequality.
0
e/2) and the second
Proof. From Lemma
D.I. have the bound:
Sprior
n Srior
08
<
n (Mprior 0
0
Sprior
n
Torm I: Prior Bias
Term 2: Data Estimation Error
23

---

Published as a conference paper at ICLR 2026
Our objective is find n such that 1 f with probability at least 1 - 8. A sufficient condition is to ensure that each term the right-hand side is bounded by e/2.
Bounding Term (Prior Bias): We require:
Sprior Mprior G
Sprior n 2
If Uprior - 0f S e/2, the condition trivially satisfied for the term's contribution for large enough "y (as the fraction Sence
0). Assume |pprior
0 > e/2. Multiplying both sides by (Sprior +
t
my F) (which is positive):
Sprior Mprior 0
< (S (Spror prior + myy
Rearranging solve for ng:
Sprior Mprior
0%
6
Ppior + E
2 n
2
| (Mprior
0 S prior
2 y
E
S ptior (1
0g e/2)
2 n F
Since we assumed |pprior
021 > e/2, the term in brackets is positive. Dividing by c/2:
> Sprive (1
0 c/2)
E ny
c/2
Let Nprior (c) Spowe (lptyter 0 e/2)
We need n 2 Mprioeee) (considering integer nzf").
c/2
Bounding Term (Data Estimation Error): than e/2:
We require the contribution from this term to be less
n
((())
0
Sprice + n
2
Since
n
E for n > 0, Sprior 2 2, sufficient condition is ensure:
Spice t
10(5) - -
2
We want this inequality to hold with high probability. We use Hoeffding's 's inequality for the empir- ical mean of ny independent Bernoulli trials (8())) approximating the true mean (0f) ):
P(O E) 0 2 1) I 2e 2n
We want the probability of the undesired event (1)) E)) 0 N c/2) to be at most ó. Setting 1 = c/2:
P( o 2 c/2) S 2e 2n (c/2)²
We require this probability to be less than equal 8:
2e 2n 5 (r/ < 8
Solving for n
2n 2n (c/2)²
8
<
2
8
2n 3 (e/2)² & In
2
8
2
2n (e/2)) 2 -In In
In
2
8
1n(2/0)
2 2n (2/5)
ny >
2(c/2)2
24

---

Published as a conference paper at ICLR 2026
Let (e, 6) 2ln(2/6) We need ny Nda
Combining Conditions: To ensure that lug 0 < e with probability at least 8, we need to satisfy both the condition derived from bounding Term 1 (if applicable) and the condition derived from bounding Term 2 (probabilistically) Therefore, we require ny be large enough for both:
Ny 2 max (Nypiir()) (e,a(c,6))
Substituting the expressions for N Nprior (t) and (emale, 6) gives the result stated in the proposition. The first term is considered 0 if |prrir 0 I e/2.
D.3 THEORETICAL CHARACTERISATIONS OF THE IFD DEFERRAL RULE
We analyse the asymptotic behaviour of the IFD loss function (Eq. (TO).. We adopt the no- tation from $4.1 where 0 denotes the true underlying per-class accuracy of expert E, and KE = arg max, By is the true best class.
Proposition 4.3 (LCB Convergence). Under the Sita-Binomial model of $4.1 with prior Beta(ay, 3y), for each class y E V, as the context count n n + oc, the posterior mean ue con- verges almost surely to the true expert accuracy 0 and its variance (a,))² - O. Consequently, for any fixed 2 0, the LCB weight Ly converges almost surely to the true accuracy:
I y = [ a oy a.s. A
Proof. From $4.1 the posterior distribution for the accuracy parameter given context DE is Beta(ay + by By + m - ) The posterior mean is:
E 2 + E ag By + +
y
By the Strong Law of Large Numbers (SLLN), the empirical accuracy ( /ng "*, OF as ny + cc. We rewrite the posterior mean by dividing the numerator and denominator by ny
(a./n + (t, /n (ay + By)/ng +
As n - co, the terms involving the fixed prior parameters vanish (assuming finite ag By). Thus, the posterior mean converges almost surely: uy as D+00 The posterior variance is:
o- OF
(ay + +1 n 1
30 (a, + By + 12)2(11 (ay + By + n +1)
E + co, the numerator is (((gg)) (since 1 /n converges), while the denominator is (((n)))) Thus, the variance is ((1/n and converges to Consequently, of as, 0.
As my
Therefore, F [n a J. By the continuous mapping theorem, I. 25, [0E a. 0-0 BE (since y E 10,
Theorem D.1 (Asymptotic IFD Rule Characterisation). Let RrD((g) be the population risk asso- ciated with the IFD loss LIFD (Eq. (1O)). As the context counts my - co for all classes y, any minimiser g" of RIrD induces the following decision rule fxx F):
if max P(Y=y) X=x) S P(Y X=x) 0 arg max (x) otherwise.
1,
(x =
where k arg max, 05 is the expert's true best class (assumed unique). Furthermore, the induced classifier h² h*(x) arg max,ey gy (x) is the Bayes-optimal multiclass predictor, h((x)) arg max, P(Y=y) X=r).
25

---

Published as a conference paper at ICLR 2026
Proof. We aim to minimise the population risk (g) B F) C (X, E)). We proceed by minimising the conditional risk RED( (g E) Byjx== [IED (z. Y, E)] pointwise for each (1r E).
We assume P(Y E) P x))
1. Deriving the Limiting Conditional Risk. The IFD loss (Eq. I0 is:
CB (t y, F) Po E) l To = y) logpi (z, E).
Let (x) - P(Y y|X = x.). The conditional risk is:
R E) = Σ my (2) (-log Par (z, E) - L I = 5/) log P1 E))
y'Ey
We consider the asymptotic limit where ny - oc for all y. In this limit:
From Proposition 4.3
the LCB weight converges a.s. to the true accuracy, L g
From Proposition
4.1 (Peak-class identification), since 2y + 0g (by Prop. 4.3 , the estimated
best class best arg max, b converges a.s. the best class kgest arg max, 0g.
Substituting these limits, the term I. = y'} converges to Bg I( y'}. The limiting conditional risk L(glar, E) is:
L(gla, E)
Σ (x) (- log (2, By I(k
11) log Pi (x,
N'EY
=En y'EY
my (x][- log Py E]] log PI (x, Σ
To Tg (z) I1 y/}
v'EV
The second sum simplifies because the indicator is non-zero only when y' = bestt Thus, the sum becomes T
(z)
The limiting conditional risk is:
beat
bart
L) =-
(x) lagglogPyss (x, 13) (mk (x) N'EY
Σ
0E k
log Pi E).
Deriving Optimal Model Probabilities. The risk L(glx, E) is weighted cross-entropy loss for the (K + 1) categories y (L( The weights are W, (2) for E V, and wi = m (z)0f.. The total weight is W = Eg "y + w1 proportional these weights (see e.g., Mozannar & Sontag (021)):
wi. The minimising probabilities (= E) are
Wy
(x(x)
(, E)
for y E V,
W
1 + MhE (x)
RE
(x)) 05
p1 (Jr, E)
I ThE (x)
OE
Mbeat
3. Induced Classifier is Bayes-Optimal. The classifier h" h"(x)) maximises (z), which (due to softmax monotonicity) is equivalent to maximizing (z, E). Since the denominator W is constant for a fixed (Ir E),, maximizing (2(2, E) is equivalent to maximizing (z).. Therefore, A (x) = arg max (x) = arg max,ey P(Y = =y|X- = r), the Bayes-optimal classifier.
4. Deferral Rule. The system defers if g" (a,, E) maxyey 9g (x). which is equivalent to pi (x, F)) 2 max PI F). Substituting the optimal probabilities and canceling the positive denominator W:
T
(x) E
max (x)
yEy
Substituting back (x) = P(Y = y/X x) yields the deferral condition stated in the theorem.
Remark D.1 (Interpretation of the Asymptotic IFD Rule). Theorem D.I shows that IFD asymp- totically implements the Bayes classifier h² (z). However, its deferral rule differs from the stan- dard L2D Bayes-opie Bayes-optimal rule derived from the 0-1 loss (Eq. (6). The standard rule defers if max, P(Y y| < P(Y M|x, F.)
26

---

Published as a conference paper at ICLR 2026
Assuming the class-conditional expert correctness model (i.e., P(M=y E) 8 and relevant conditional independencies, the standard rule compares model confidence to the expert ss expected accuracy:
max P(Y=y X=r) S P(Y-y) X=x) | Ey (04)
31
The IFD rule, in contrast, defers if:
max
xx=x)
0E bast
This distinct rule arises from the structure of the LIFD surrogate (Eg. 10
The deferral term
rewards deferral only when the true class y aligns with the expert's 's peak class knest (asymptotically
k ). This encourages deferral based on the expert's ' peak competence, rather than their expected correctness across all likely classes.
E PERMUTATION SYMMETRY, INVARIANCE, AND GENERALISATION OF IDENTITY-FREE DEFERRAL
E.1 SETUP, NOTATION, AND PROBLEM FORMULATION
Labels, inputs, experts. We consider K-class classification problem with label set y [1. K}, inputs X E X. and target Y E V. Let (X,Y) ~ P. For any I, denote the poste- rior
(x) = P(Y = y X = x), nT() (x), ................................ 7x , (x)).
An expert E provides a discrete prediction M e. Each expert E has context set
D {( vf m ))
E.C y m
summarising past behaviour. An encoder up may map this context to a representation; we will write the deferral logit as 9 (ar, E).
Classifier and logits. Let By (2), y E V, be the class logits used by the classifier h,
h(x)) arg max (2) (with any fixed measurable tie- if needed).
Rejector (deferral rejector) and (K+1) softmax. The rejector produces a single deferral logit 91 (x, E) R. Define the joint logits and normaliser
g(x, E) = (g. 9K (x), Ie (x, E)).
= ex + Σ exx(gg(x)))
yEy
The (K+1)-way probabilities are
explgg(x)} E)),, 2(112,F) Z(x, E)
explgi E)}
pY = yy
2(z, E)
At inference, the system predicts h(z) and defers according the rejector
r(x, E" Il max (x) S 91 91 E3
yEy
the overall predictor is (a, F) = h(x) if r(x E) 0 and Yá, F) =1 E) = Query-conditioned expert correctness and qι E). Define the expert's query-conditioned cor- rectness
q(x, E) P(M = Y X = x, E).
Define the class-conditional correctness profile at query x,
()(x) = P(ME = X x Y y. F.) [0.1].
27

---

Published as a conference paper at ICLR 2026
so that by the law of total probability
k Σ y=1
(1,, E)
P(Y = x=
= n. E) Myy ( (x))
When additionally Y T E X, we have g(z, = - my (2) My (2).
1 system loss and Bayes choice. Under the 0-1 system loss, the pointwise risks at (xz, E) are
Reredic( (z) max P(Y x and yEY
Rdster (z, E) - q(z E).
Equivalently, using h(x) arg maxy By (z).
(x)) 1- ())(o)) (x).
Define the (system) deferral margin
(sys(a, E) (xr, E)- max P(Y y X = =x ((x,E) ())))())
DEY
The Bayes action to defer iff Asys (I, E)) 0:
(x, E) = I P(Y = M x = x, E) 2
max P(Y = X x)) yEy
A regret proxy for a given rejector. Given logits
(91, 9K: 91), define the induced action
defer,,
91 (a, E) 2 max, (x),
(2r, F.)
predict,
otherwise.
and the pointwise regret
reg(ū: x. F.) max (0, sas (ar, F.) 1(ag predict) max(0, Ls,s(E, F.) 1/ag defer).
This is zero iff the rejector matches the Bayes decision at (x, E).
Coherent relabelling (permutation symmetry). Let k be the permutation group on (1,..., K} and define its action on any class-indexee quantity v { by
(v y = (2))
A coherent relabelling applies π to all class-indexed objects:
(M, + (M(M),
(" X=x) = P(Y=T-((1) X=x),
and to the context
D { T(y ((m E,C M)
No
Because the event {M-Y} is invariant under (M,Y) + (T(M), we have P(Y=M X=z, E) unchanged, and max, P(Y=y X=r) unchanged (the arg max simply relabels).
OOD experts via identity shifts. We model an "expert with the same competence structure under label identity" by permuting class-indexes expert profiles while leaving the task fixed. For class-conditional correctness, this amounts to
K ((x,E)) = P(Y-ylX-xx. E") " (x)
Ed Hy (x) = M (x))())())
E (I)
E
Under (A1) V 1 E X, this reduces to q(2, E) = (z))))) and g(z, E²) E, Th (2) (1))))))()(()) Expert-only permutations can therefore change the Bayes action when they alter q(ø, E) relative to max, P(Y y v tests used later.
X = z). This motivates permutation-basee OOD stress
28

---

Published as a conference paper at ICLR 2026
E.2 WHY L2D-POP FAILS ON OOD EXPERTS: QUERY-INDEPENN VARIANT
Architecture. The query-independent (QI) variant encodes the expert's context via a Deep Sets Zaheer al. (2018) mean:
N
1
Σ
e(vi); e(m, F.C ));
No ==1
where (p()) is fixed feature map and e(+) is either one-hot or learned class embedding deferral logit is
The
91 (x, E) = rol (D)())))) p(z)1).
Pop-ind
Set-permutation vs. coherent relabelling invariance. Deep Sets confers invariance to set per- mutations: reordering the elements of Dē leaves vi unchanged. The symmetry relevant to label identity is coherent relabelling (:3.) applying the same permutation æ Gg to all class-indexed objects in each token, m)) i (v), (m)). Deep Sets does not guarantee invariance to such label relabellings.
Lemma E.1 (Generic non-invariance of token MLP with class-specific columns). Let the token be tyy m) = [x(xx); e(y); e(m)] E Rd=2K (with one-hot e(-); the caamed-embedding case is covered in foomote 2). Suppose the first affine layer of is
z = Wa ()(()) + ely + Wm e(m) + b, W,,Wn, Wm E HR not all columns of Wy are identical and not all columns of Wm m are identical, then generically (i.e., for an open of downstream parameters) there exist (x. y, m, R) such that >(p((=)); Pre((y;; P=)(m)) # >(pp(< e;y)); e(m)]).
Proposition 4.3 (QI encoder is generically non-invariant under coherent relabelling). Let
(DE
Σ (x0 J..)EDD
{(p(>0) e(y); e(m))):).
N
If there exist E 6K and a token (x .y, m) such that T(pp(x)) Prely): P. () * ((()=))))());((m))) then there exist one-element context D = 1(x M, m) for which
DD E." :)) * (D;
In particular, for standard L2D-Pop with untied class embeddings and an unconstrained MLP % this non-invariance holds generically.
Corollary E.1 (QI Rejector inherits con-invariance generically). I& is non-constant, then generi- cally there exist (x, E) and a permutation π such that
91
Pop-ind
(x, E²) + 91.
Pop-ind (x. E).
Concrete example (K=2). Let e(1) = (1, e(2) = 10 and suppose y is linear in the class channels:
)((((0); e(y); e(m))) = Ae(y) + B Be(m) + u ((x())
with A, B E Rdx2 and u E Rdim Under the swap π (1 2), the token output shifts by (A+B) (e(2) e()). which is nonzero except measure-zero parameter set. The claims follow by Prop. 4.3 and Cor.
Consequences for OOD risk. Let IT be any distribution over permutations of labels and let l be 1-Lipschitz in 91 (classifier logits fixed). Then LOOD(O1) LID(g_) < [ 91 -- (x,
and the inner expectation is strictly positive generically.
generalisation viewpoint. Let Hoi be the induced hypothesis class. Because HQI contains identity- conditioned functions, it effectively contains K! distinct "cases" for single structural pattern (one per permutation). With finite ID coverage of permutations, ERM can overfit to the seen identities. permutation-innariann rejector collapses these K! cases one.
Forr learned embeddings, write e(y) F. cg with F. E Kx and e, the yth canonical basis vector. All arguments below apply verhatim the effective linear map multiplying e(y)-
29

---

Published as a conference paper at ICLR 2026
E.3 WHY L2D-POP FAILS ON OOD EXPERTS: NUERY-CONDITIONEE VARIANT
The query-conditioned (QC) variant builds a query from the current input and cross-attends the expert's context:
Architecture.
((x) = We (<x),
h; - >(<(a;); e(vi); e(m,, 1); == p(x;), oi= = -Wv his
ag (a) softmax, k;/ /d).
5(x) FFN(LN(((()) +Da (z)
with 91 Pop-cond E) = ra(l<(>); s (x))). As above, yp a fixed feature map and e(-) denotes class-ID embedding (one-hot or learned).
Set-permutation vs. coherent label relabelling. Without positional encodings, the cross-attention is invariant to context index permutations (reordering the items of D The OOD symmetry is coherent relabelling ($ (uf" .m m...) (T())), ), π ((m (m..))) for single 6 applied to all tokens. Attention alone does not enforce invariance to such relabellings.
Proposition 4.3 (QC cross-attention is generically non-invann -nv-invrriant under coherent relabelling). Let
DE be obtained by applying E 6 to all class-indexed fields of DE- Assume the token map not Sg-equivariant (Lemma E. Then, for open set of parameters (Wy, FFN, LN, and for some (r,E, T)
B (x) + (x))
and
91
Pop-coud (x, E.) #1
Pop-cond (x, E))
Amplified shorteut via queryfidentity interaction. Compared QI, QC learn bilinear, index-aligned features that explicitly tie the current query class to the same absolute index in the context. A single head can implement
K
Pop-cond
(x,, E)
Σ j=1
a
(,))
b,(D
"evidence for class 3
with untied coefficients aj and class-indexed context statistics Φj. strengthening identity condition- ing.
Conerete counterexample (index alignment, K-3). Suppose Y produces vector whose jihh coordinate encodes "specialisation evidence for class j. Choose We, WK so that ((x) ke is large when oi resembles x for the model's top class ktop and choose Wv, FFN, rg so that 91 (a, E) ss دj= 1 (jj (z(x)) jevidence for class j]. If an OOD expert E' presents the same special- isation pattern index क Tj),, the values do not transform by P., the weights a; remain fixed, and I1 fails to implement the coherent relabelling.
Consequences for OOD risk (comparison with QI. With the same Lipschitz and permutation averaging setup as in E.2
(OoD(g1) - CDD(g S E() Ex 91 (X, E*) 9 (X, E)]
and the inner expectation is strictly positive generically (Prop. 4.3) The bilinear index alignment expands Hgc. typically increasing the OOD penalty relative QI.
Remark (positional encodings). If positional encodings over the context sequence used, even set-index permutation invariance lost; this does not mitigate-and can exacerbate-the label-relabelling non-invariance above.
E.4
IFD: ARCHITECTURE AND INVARIANCE PROOF
Setup and notation. Let g(z) = (x),..... 9K be the classifier logits and p(x)
softmax(g(x)) the class-probabilitty vector with coordinates pa (x). From the expert's context DE. fix J > pla-slass posterior functionals F {f and define
(f() 10 DE),
te [ b- j=n... RRKX
30

---

Published as a conference paper at ICLR 2026
Choose ranking functional F and let u E R denote its per-class values, uy = u(8 D We use a fixed, permutation-equivaraann tie-breaker shared across the paper. Write keep (x) arg max (x) and TE E arg maxy up- For л E 6x, the induced action on any class-indexed vector v is (v") = (p) and on the functional matrix is
(8 = ( (6")a=(>>>
BT = P=
which also yields (u = - 'w)'
Role indexing and identity-free state. The identity-free state 2(x, E) Exed-dimension vector built by reading values roles and aggregating symmetric statistics, never exposing absolute class IDs:
Id is any
E) = R(((()), $²).
(20)
E Concretely, RI(-) may include: (i) role values such as Pkoup (x) (x)p(()) (x) Pi (I), W.
any
best
other 1/ evaluated these roles; (ii) role comparisons such (z) 1. gaps Phiop(z) (z) P(2) (x) and u(i) " and (iii) symmetric aggregates of p (e.g., entropy Hp)) or of rows of E (e.g.. norms), as long as they are permutation invariant. A concrete 6-dimensional choice used in experiments (with J=2, fi mean and variance, mean) is
2(x, E) (z)))))) Pige
E (x), (x) (o (o)
1e bass
(o )²). bast
(21)
The IFD rejector applies small 1LP
0 (2, = ro(2(0, E)),
(22)
and the (K+1)-way softmax over V {1) defines the deferral probability
pi (x, F.) softmax(gg1 (2), gk (2), IFD (z. F.)
Permutation action and role equivariance. Under coherent relabelling π 6x acting all quantities, (p*),, )² = fс-*(z) and ( FT )u.j = ( - By definition arg max
and the fixed tie breaker, the role indices transform equivariantly:
krop((") = nl (P))
T EBT) = n (o (T))).
(23)
Consequently, values read at roles are preserved; e.g., (p"))(((n)(())) = Pkiup (p) and u FSt (Kkoop()))
u (p)"
Proof of Proposition 4.2 (permutation invariance of the role-indexed state). We must show that is invariant under coherent relabelling and, hence, so is glFD by composition with Tá. Each coordinate z belongs to one of three classes:
or any ff these roles): by 23 the index moves to =T-) while the value at that index is unchanged when the underlying vec- tor/matrix is permuted coherently. Hence every role value is preserved.
Role values (e.g.. (p)) (o) PE RE
Symmetric aggregates (e.g., (p), norms or fixed-order statistics with a label-invariant tie-breaker): these are permutation-invarite by construction.
Role comparisons (e.g., II[ (p)= gaps between role values): since both roles transform by the same x, equalities and differences are preserved.
Therefore,
RI(p (z), F.v) RI(p(z), F) for all Gk.
and invariance of the rejector follows:
IFD 91 (x, E*) ro(Rl(p", &B.E)) ro(RI(p, 2B)) IFD E).
31

---

Published as a conference paper at ICLR 2026
Corollary (rejector invariance). For any parameters and any GK. g (z,
gl (x, E). Hence the IFD deferral decision is invariant to coherent relabelling.
Scope and caveats. (1) The invariance proven here is with respect to coherent relabellings acting on all class-indexed quantities (p, 2.). It does not assert invariance under expert-only permutations with fixed classifier/posterior; such permutations can change the Bayes action (Prop. 3.2 (2) The role selection uses arg max, which is con-iifeerentable only on measure- zero tie sets; the fixed tie-breaker makes & measurable and the rejector trainable almost everywhere. (3) Invariance holds irrespective of calibration quality; it is enforced architecturalyy by the construction of
Why this proof matters for OOD experts. OOD experts often instantiate the same competence structure under different absolute class names. Because IFD collapses all K! relabellings of struc- ture to the same z(,, E), single learned rule covers all permutations. In contrast, population encoders with identity channels must effectively relearn across permutations unless explicitly sym- metrized; see the non non- -nnairiieeece results for L2D-Pop in App. E.2 E.3
E.5 PLAIN-LANGUAGE TAKEAWAYS
is coherent relabelling: if consistently rename classes everywhere (model outputs, expert profile. labels), the underlying decision problem is the same. IFD respects this by construction; generic L2D-Pop encoders do not.
The symmetry that matters
E.33 y its encoders expose label-identity channels. That lets the rejector learn shortcuts like "index 3 is special," more subtly "defer when the model currently favours index and the context also looks strong at that same absolute index." These identity-keyed rules break when the same competence structure shows up under different indices OOD. In other words, the hypothesis class is not closed under per- mutations, so the learned function can change when you just rename classes.
Why L2D-Pop fails OOD E2
Why IFD succeeds OOD ( (E.4): it first turns per-class signals into a small set of role- indexed values 2 (e.g., "model's top probability," "expert's competence the model's top class," and symmetric aggregates like an inner product), then drops the indices. We showed z (and hence the rejector) is invariant to coherent relabelling, so single learned rule auto- matically covers all K! ways the same structure can be labelled.
F EXPERIMENTAL DETAILS
Computational Resources All experiments in this paper were run using an NVIDIA GeForce RTX 4090 GPU, AMD Ryzen 9 7950X 16-Core Processor and 64GB DDR5 RAM.M
Classifier Models The classifier architecture varies by dataset: ResNer-34 (He et al. 2015) for HAMI0000, and pretrained EfficientNet BO (Tan & Le 2020 for Liver tumourr, and Mo- bileNetV2 (Sndlerr et al. 2019) for Blood Ce
Rejector Model For all experimental models, we keep the architecture of the rejector constant. For this, we utilise 6-layer MLP with hidden dimensionality of 256.
Training Details Details on number of ID/OOD experts, context set sizes N and n are provided in Table D We utilise training and validation batch sizes of 128 for all datasets. We utilise the Adam optimiser (?) for all experiments, with a learning rate of 0.001 for Liver tumours and HAM10000, and 0.0005 for Blood Cells. All models are trained until convergence, measured early stopping on the validation set measuring AURSAC (patience 50). We utilise standard = hyperparameter for the JCCB-weightee loss for all experiments.
G ADDITIONAL EXPERIMENTS
G.1 TOY EXAMPLE: IDENTITY LEAKAGE VS. INVARIANCE
To illustrate the identity-conditonn failure mode discussed design simple 2D, 3-class (K=3) toy experiment.
33.3 and the robustness IFD, we
32

---

Published as a conference paper at ICLR 2026
DATA AND EXPERT SETUP
Data. We generate a 2D, 3-class dataset (N=1000) composed of three concentric cir- cles:
Class
Class forms the inner circle, Class the middle annulus, and Class 2 the outer amnu- lus (Figure [T. Moderate base noise (0.7) and additive Gaussian noise (o=0.5) create strong overlap between neighbouring classes. A small MLP classifier trained on this data and used as fixed feature extractor for all models.
Fosture
Experimental controls. Both pipelines (L.2D- Pop and IFD) use the same pretrained MLP feature extractor, identical training procedure, random seeds, and context-set sampling. The only difference lies in the rejector architecture and how expert context is summarised.
Expert competence. During training, both models observe context from single Figure 1:
Toy dataset. Each colour corresponds to Class 0 specialist: high accuracy (90%) on class (three concentric rings). Class 0, and low accuracy (50%) on Classes and 2. At time, evaluate under experts:
ID Expert:
another Class 0 specialist (in-distribution).
OOD Expert: Class 2 specialist with the same competence profile permuted (high on Class 2, low on 0 and 1). This is an expert-only permutation the task is fixed, but the expert's 's strengths are reindexed.
Hypothesis. Because 1.2D-Pop encodes absolute label identities, it is expected to learn an identity- conditioned rule (e.g., "defer on Class 0")), and therefore fail adapt when the expert's specialty moves to Class 2. IFD, which removes label-identity channels and summarises per-class competence symmetrically, should instead learn a structural rule (e.g., "defer on whichever class the expert best at") and transfer correctly under the expert-only permutation.
RESULTS AND ANALYSIS
Figure visualises the deferral patterns determined by 9- (1( v) maxy By (x). Points deferred the expert are labelled by their true class; non-deferred points are shown as faint background dots.
Interpretation. L2D-Pop's deferral pattern largely fixed: it continues defer heavily on Classes 0-1 even after the expert's competence shifts to Class 2. This demonstrates an identity- conditioned shortcut consistent with Thm. 3.1 IFD, in contrast, changes its deferral behaviour automatically-der defer decisions migrate from Class 0 (ID) to Class 2 (riD>sshownng the expected invariance predicted by Prop. 4.2
G.2 ABLATION: UNCERTAINTY-AWA. LCB WEIGHTING
Setup. To isolate the effect of the uncertainty-aware LCB weighting the IFD loss (Eq, (10) ablated the deferral weighting three datasets: HAM1 CO00, Blood Cells, and Liver tumours (as seen in $5) We compare four variants, averaged over five seeds, and only report the AURSAC and AURDAC across the full deferral budget:
1. Full (ours): - = [ y. with a = 1.0.
Mean: LE 1y (equivalent a 0).
3. Binary: L = (1y )
Classifier: no deferral (for reference).
All other details (data preprocessing, model backbones, training schedules, and context construc- tion) follow the main experiments. We aggregate system performance across experts (no per-expert reporting here).
33

---

Published as a conference paper at ICLR 2026
Expert Proficiency: Class 0
Expert Proficiency: Class 2 A
L.2D-Pop (QC)
Feature 2
IFD (ours)
Feature 2
Feature
Feature
Deferred (class 0)
Deferred (class 1)
Deferred (class 2)
Not deferred (class 0) Not deferred (class
Not deferred (cluss
Figure 2: Deferral behaviour on the toy test set. Top:
L2D-Pop: Bottom: IFD. Left: ID Expert (Class 0 specialist). Right: OOD Expert (Class 2 specialist, an expert-only permutation). L2D-Pop fails to adapt, whereas IFD correctly shifts deferrals to the new expert's specialty. Deferred points determined by the default L.2D deferral operation I1 (z, eb) 2 max, (1)).
Findings. Across datasets, Full is best statistically tied with the best LCB variant. It yields the strongest AURSAC on HAM1 00C0 and Liver tumours, while Mean edges out Full by negligible margin Blood Cells and in AURDAC Liver tumours; these differences are well within one standard deviation. All LCB-based methods substantially outperform the non- deferring baseline AURSAC.
Using LCB weights with a=1.0 consistently matches or exceeds simpler alternatives on AURSAC(O, 1.0), justifying Full default for uncertainty-aware deferral training.
G.3 RQ3: PERFORMANCE SCALING WITH LIMITED CONTEXT DATA
We study how IFD and baselines exploit additional expert context at test time. During training, for each expert E we compute the profile from a randomly drawn size-n subset of context DE resampled each epoch, which regularises the rejector and avoids overfitting to any single context realisation. On Blood Cells (Variable), we train with fixed small n and evaluate as the context pool No increases (15++120)).
34

---

Published as a conference paper at ICLR 2026
Table 3:
Ablation of the LCB trust signal (deferral budget mean + sd over seeds; best among LCB variants in bold.
[0, 1D.
Values are
Full
Mean
Binary
Classifier
Dataset
Metric
Mean
SD
Mean
SD
Mean
SD
Mean
SD
AURSAC f
0.668 + 0.029
0.669 = 0.027
0.665 0.665 0.028
0.595 + 0.025
Blood Cells
AURDAC 1
0.425 0.044
0.428 0.040
0.427 0.044
0.335 1 0.040
AURSAC f
0.641 0.042
0.639 = 0.040
0.638 + 0.044
0.603 + 0.014
HAM10000
AURDAC
0.349 0.076
0.346 0.069
0.343 + 0.083
0.350 + 0.035
AURSAC1 f
0.004
0.632 = 0.021
0.628 + 0.027
0.562 0.018
Liver tumours
AURDAC t
0.340 0.031
0.344 0.045
0.338 + 0.050
0.264 + 0.031
AURDAC for Classifier not meaningful because never defers.
< 25
m =50
0OD Eapert ID Experts
0.75
0.70
AURDAC
0.60
0.55
0.70
0.65
0.55
15
30
60
120
240
480
960
15
30
120
240
480
960
15
30
60
120
240
480
960
context examples (NE. log scale)
IFD (ours)
L2D-Pop (QC)
L2D-Pop
Figure 3: Scaling with context. Blood Cells, Variable Specialists. Models trained with small per-expert subsamples n E (10, 25, 50) and evaluated as the available test-time context Ng grows. IFD benefits from more context; 1.2D>Pop remains flat.
Figure demonstrates that IFD consistently turns additional test-time context into better system accuracy and higher-quality deferrals, with gains emerging early and tapering at very large Ng. In contrast, L2D-Pop (QI) is largely insensitive to more context, and L2D-Pop (QC) tends to plateau or degrade as N grows. These patterns hold across all training budgets (n E {10, 25, 50)) and are most pronounced for OOD experts, reflecting that IFD's identity-free Bayesian profile nat- urally scales with context while fixed-dimensional, meta-trained encoders struggle to exploit large pools.
Findings:
G.4 ROBUSTNESS TO PRIOR MISSPECIFICATTIO
We evaluate fixed, pre-trained IFD model the HAM1 0000 dataset without any retraining, All experiments vary only the evaluation- priors and context; the classifier and rejector parameters remain fixed. We consider the Stable and Realistic experts defined in Section B For each class y. a Beta prior Beta(ag, 3y) is parameterised by mean My and strength S, via Gg = My By (1 S. We evaluate:
Accurate Prior (AP): My 05 (ground-truth profile).
Uninformative Prior (UP): Beta(1, (i.e., S 2).
Adversarial Prior (AdvP): "inverted" specialist (high competence on an incorrect class).
Overconfident Generalist (AdvP-OCG): My 0.90 for all y,
We sweep strong prior strengths $ E (10, 30, 100} for AP, AdvP, and AdvP-OCG.
Context Protocol We vary the number of context points per class nc (0, 5, 10, 20, 40, 80, 160, 320, 640} (total context Nø Knc), For each ne > 0, we sample context outcomes i.i.d. from the ground-trute touth expert profile dº. Posteriors are updated by conjugate Beta-BBionee updates per class.
35

---

Published as a conference paper at ICLR 2026
Evaluation Given the posterior means and variances, the IFD rejector computes deferral prob- abilitics for each expert on held-out test queries. We report the MAE of posterior means:
R Σ 1
-
Aggregation For each combination of expert archetype, prior scenario, strength S, and ne, we perform R 20 independent repeats (new context draws), and report mean standard error across repeats. Plots use x-axis n and summarise MAE.
Poweever Moon MAC Sunkle Spociliie S-10
Ponurtio Mon MAE
Steble, S-10
Mous MAE Specialis, S
Socuuo AP
Sereato .0
Add
AND OCG
Add OCC CP
AM
CP
X20
Postetic Variable Specialist 5-10
Postaior Maac
Variable Specialis, 5-30
Portctoor More MAE Varialle Specialio, S-100
Scardo
Socarie
Auh
AAP-0CO
Topottir Noet
1D
17
250 -20
150 330
<8
<5
Figure 4: Adversarial priors experiment results. MAE vs context per class for Stable and Variable Specialists. Curves are near-identical; misspecified priors are overcome as context grows: R 30 points (S = 10). 2 320 (S = 30), = 640 (S 100).
Findings: Figure 4 displays the results. The trajectories are near-identical for Stable and Variable Specialists, indicating that sensitivity to prior misspecification is largely independent of the expert's polarity. MAE decreases monotonically with added context, and the context required to overcome misspecified prior scales with prior strength S: for S=10, convergence occurs within 30 context points; for S=30, around ~ 320; and for 5=100, about ~ 640. These thresholds mark when the posterior effectively overrides the prior, as expected under Bayesian updating.
H DATASETS
We used three public medical imaging datasets to develop and test our methods. Each dataset is described below, noting its contents and license.
HAM10000 Dataset The HAM10000 dataset (Tschandl et al., 2018) contains 10,015 dermato- scopic images of common pigmented skin lesions. was created to provide a larger, more var- ied resource for training systems to identify skin conditions. Images were collected over twenty years from Vienna, Austria, and Queensland, Australia. The dataset covers seven diagnostic types: melanocytic nevi, melanoma, benign keratosis-like lesions, basal cell carcinoma, actinic keratoses, vascular lesions, and dermatofibroma. Over half the diagnoses were confirmed by histopathology, with the rest checked by follow-up, expert review, or confocal microscopy. The dataset is available via the Harvard Dataverse) The original paper states the license is Creative Commons Attribution 4.0 International (CC BY 4.0).
The Blood Cells dataset (Acevedo et al. 2020), has 17,092 microscope images of single normal blood cells. It was made to help build and test systems for recognising blood cells, particularly using deep learning. The images are 360x363 pixel JPGs, taken with Cella DM96 machine at the Hospital Clinic of Barcelona. Expert pathologists labelled the cells. The dataset groups cells into eight types: neutrophils, cosinophils, basophils, lymphocytes, monocytes, immature granulocytes, erythroblasts, and platelets. Immature granulocytes are grouped together because individually identifying them is less clinically important and difficult even for experts. This
Blood Cells Dataset
36

---

Published as a conference paper at ICLR 2026
grouping focuses clearer categories for model training. The dataset is on Mendeley Data and uses the Creative Commons Attribution 4.0 International (CC BY 4.0) license.
Liver tumours (LiTS) Dataset: Liver tumours The Liver tumour Segmentation (LiTS) dataset (Bilic et al. 2023) (referenced as "Liver tumours" in this paper) is a resource for testing liver and liver tumour segmentation methods. It includes 131 CT volumes for training and 70 for testing. Data came from seven hospitals and research centres, offering varied clinical cases. The dataset includes different tumour types, sizes, looks, and contrast levels, making it tough, realistic test. The LiTS dataset and its online testing system are on CodaLab It uses the Creative Commons Attribution-Nonde CannCommmrial-ooDeriviei 4.0 International (CC BY-NC-ND 4.0) license.
37


Reviews:
[{"review_id": "tyBCvvwG1H", "reviewer": "Reviewer_R4fz", "summary": "The paper proposes Identity-Free Deferral (IFD) for Learning-to-Defer with population experts. The authors argue that common “population” encoders leak class/expert identity and are not invariant to coherent relabelings of classes, which can hurt transfer to new experts. Their approach build a small, permutation-invariant, role-indexed state from a query-independent competence profile estimated from a few context examples. The rejector compares the model’s confidence to an expert’s peak-class accuracy. Several experiments are made on different medical dataset, results are compared with the exist population L2D approach as well as a standard one.", "questions": "I have a couple of questions: \n\n1. Your current surrogate loss appears largely inspired by [5]. Is there a specific reason why you compare primarily with the formulation from [1] rather than directly with that of [5]?\n\n2. The base surrogate loss employed in the paper has previously been shown to produce *miscalibrated* probability estimates [2] and to be *non-realizable* under standard assumptions [3, 4]. Is there a particular reason you chose this formulation instead of those from [2] or [3], which are known to yield calibrated or realizable solutions?\n\n3. In Table 3, you argue that **Full** consistently matches or exceeds other variants, but the difference between **Full** and **Mean** appears very small overall. Do you have an explanation for this? Is there a specific trade-off between $\\mu_y^E$ and $\\sigma_y^E$?\n\n4. Out of curiosity, how does your approach perform in the standard Learning-to-Defer setting (with fixed experts), compared for instance to [1]?\n\n5. Is there a reason why the current formulation is restricted to single expert?\n\n\n------\n\n[1] Rajeev Verma, Daniel Barrejon, and Eric Nalisnick. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. \n\n[2] Yuzhou Cao, Hussein Mozannar, Lei Feng, Hongxin Wei, and Bo An. In defense of softmax parametrization for calibrated and consistent learning to defer.\n\n[3] Anqi Mao, Mehryar Mohri, and Yutao Zhong. Realizable h-consistent and bayes-consistent loss\nfunctions for learning to defer.\n\n[4] Hussein Mozannar, Hunter Lang, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David A. Sontag. Who should predict? exact algorithms for learning to defer to humans.\n\n[5] Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The work is sound and I understand the motivation on the symmetry. \n- Reducing the number of predictions required by the expert is good. \n- The method is quite practical and scalable.", "weaknesses": "-  I had a lot of trouble following the paper. The presentation could be substantially clarified: the structure feels dense, and it is often unclear how the different components are connected. A significant rewrite or reorganization might be needed to make the flow easier to follow.\n\n- Some background and notation are imprecise or undefined. For example, $r(x)$ later becomes $r(x, E)$ without a clear definition, $\\Delta$ is introduced but never defined, and LCB appears before Lower Confidence Bound is spelled out.\n\n- Several small presentation issues and typos remain:\n    - The 'QI' column is labeled [1].\n    - Typo at line 10: 'delerejector',  line 1698: 'aggrerejector', parentheses line 229.  \n\n- The base surrogate loss employed in the paper has previously been shown to yield miscalibrated probability estimates [2] and to be non-realizable [3, 4].\n\n- Only for a single expert.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6MrYjIhsUC", "reviewer": "Reviewer_z5Ji", "summary": "This paper introduces Identity-Free Deferral (IFD), a new architecture for L2D systems that improves generalization to unseen human experts, especially those that are OOD to the training population. In standard L2D frameworks, a model learns when to predict and when to defer to an expert. Recent population-adaptive extensions condition this decision on learned expert embeddings, yet these architectures often rely on class-indexed signals that leak label identities and thus violate the permutation symmetry inherent in the problem. As a result, they tend to overfit to specific expert–class alignments and fail to transfer to unseen or re-indexed experts.\n\nIFD addresses this issue by enforcing permutation invariance architecturally. Instead of embedding expert identities directly, it constructs a Bayesian, query-independent competence profile for each expert using a few-shot context of past predictions. The deferral rejector then operates on a compact, role-indexed state (containing structural information such as the model’s confidence in its top-ranked class and the expert’s estimated competence for that same role), thus eliminating all absolute class-identity channels. The model is trained with a novel uncertainty-aware, context-only loss that removes the need for query-time expert labels, weighting supervision by a lower confidence bound on expert reliability.\n\nThe authors provide formal proofs of permutation invariance, contrasting IFD with the non-invariance of standard population encoders, and show theoretically that the resulting policy focuses on structural relations between model and expert rather than identity-conditioned shortcuts. Empirical results across multiple medical imaging datasets and ImageNet-16H (with real human annotators) demonstrate that IFD consistently outperforms L2D-Pop and other baselines, especially for OOD experts and under input distribution shift. Moreover, IFD achieves these gains with substantially fewer expert annotations.", "questions": "1) Could the authors include or describe a simple empirical toy example demonstrating this failure mode and how IFD avoids it?\n2) The invariance proof assumes coherent relabelings (joint permutations of model, labels, and expert), while real OOD scenarios often involve expert-only shifts. Could the authors clarify how IFD behaves in such cases and whether approximate invariance can still be expected?\n3) IFD models expert competence at the class level, ignoring within-class instance variation. How sensitive is performance to this assumption, and do the authors plan to explore instance-aware but still identity-free extensions?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper tackles a highly relevant and practically important problem in L2D: ensuring reliable deferral to unseen or OOD experts. This setting reflects real-world scenarios such as hospital staff turnover, making the problem both timely and significant for AI deployment.\n\nIn terms of originality, the work offers a clear conceptual advance by identifying identity-conditioned shortcuts as a core limitation of existing L2D approaches and proposing an architectural solution (IFD) that enforces permutation symmetry by design. This symmetry-based perspective introduces a novel and principled inductive bias for expert adaptation.\n\nThe quality of the work is strong. The theoretical analysis is sound and well-motivated, including formal proofs of invariance and clear links to the Bayes-optimal rule. The accompanying uncertainty-aware, context-only training objective is practical, removing the need for costly query-time expert labels.\n\nThe experimental evaluation is thorough, spanning multiple datasets, both simulated and real experts, and a wide range of conditions such as in-distribution and OOD settings, input shifts, and ablations.", "weaknesses": "1) Although the theoretical analysis convincingly demonstrates that standard population-adaptive L2D architectures suffer from identity-conditioned shortcuts and that IFD enforces permutation invariance by design, the paper would benefit from a direct empirical illustration of these shortcuts. A simple toy experiment (e.g. showing how a conventional L2D-Pop model fails under systematic class reindexing while IFD remains stable) would make the argument more concrete and visually intuitive.\n\n2) While the theoretical results appear sound, their scope could be clarified more explicitly. The proven invariance strictly holds under coherent relabelings (when both the model and expert are permuted consistently), whereas in practice the paper’s OOD setup often involves expert-only permutations where this assumption may not fully apply. Stating this distinction in the main text and briefly discussing potential extensions to partial or incoherent relabelings would improve conceptual precision.\n\n3) The class-level abstraction of expert competence, though central to the method’s efficiency and invariance, inherently limits sensitivity to within-class, instance-specific expertise differences. While the authors acknowledge this in their discussion, a small-scale experiment or ablation (e.g., comparing performance when experts vary mainly within a class) would clarify how much this simplification affects performance.\n\n4) There are minor presentation issues that should be addressed to improve polish and readability: the word “delerejector” in the abstract appears to be a typo, and Section 4.2 repeats the parenthetical phrase “(why this blocks identity leakage)\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)