Towards Scalable Oversight via Partitioned Human Supervision
Authors: Ren Yin
Takashi Ishida
Masashi Sugiyama
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
scalable oversight
weak supervision
agentic systems
Primary area: foundation or frontier models, including LLMs

Abstract: As artificial intelligence (AI) systems approach and surpass expert human performance across a broad range of tasks, obtaining high-quality human supervision for evaluation and training becomes increasingly challenging.
Our focus is on tasks that require deep knowledge and skills of multiple domains, where this bottleneck is severe.
Unfortunately, even the best human experts are knowledgeable only in a single narrow area, and will not be able to evaluate the correctness of advanced AI systems on such superhuman tasks.
However, based on their narrow expertise, humans may provide a weak signal, i.e., a *complementary label* indicating an option that is incorrect. For example, a cardiologist could state that ''this is not related to any cardiovascular disease,'' even if they cannot identify the true disease.
Based on this weak signal, we propose a scalable oversight framework that enables us to evaluate frontier AI systems without the need to prepare the ground truth.
We derive an *unbiased* estimator of top-1 accuracy from complementary labels and quantify how many complementary labels are needed to match the variance of ordinary labels. We further introduce two estimators to combine scarce ordinary labels with abundant complementary labels.
We provide finite-sample deviation guarantees for both complementary-only and the mixed estimators.
Empirically, we show that we can evaluate the output of large language models without the ground truth, if we have complementary labels.
We further show that we can train an AI system with such weak signals: we show how we can design an agentic AI system automatically that can improve itself with this partitioned human supervision. Our code is available at https://github.com/R-Yin-217/Towards-Scalable-Oversight-via-Partitioned-Human-Supervision.

Published as a conference paper at ICLR 2026
TOWARDS SCALABLE OVERSIGHT VIA PARTITIONED HUMAN SUPERVISION
Ren Yin Takashi Ishida Masashi Sugiyama
'he University of Tokyo 2RIKEN
ABSTRACT
As artificial intelligence (Al) systems approach and surpass expert human perfor- mance across broad range of tasks, obtaining high-quality human supervision for evaluation and training becomes increasingly challenging. Our focus is on tasks that require deep knowledge and skills of multiple domains, where this bottleneck is severe. Unfortunately, even the best human experts are knowledgeable only in a single narrow area, and will not be able evaluate the correctness of advanced Al systems on such superhuman tasks. However, based on their narrow expertise, hu- mans may provide weak signal, i.e., a complementary label indicating an option that is incorrect. For example, a cardiologist could state that "this is not related to any cardiovaseular disease," " even if they cannot identify the true disease. Based on this weak signal, we propose scalable oversight framework that enables us to evaluate frontier AI systems without the need to prepare the ground truth. We derive unbiased estimator of top-1 accuracy from complementary labels and quantify how many complementary labels are needed to match the variance of ordinary labels. We further introduce two estimators to combine scarce ordinary labels with abundant complementary labels. We provide finite-sample deviation guarantees for both complementary-only and the mixed estimators. Empirically, we show that we can evaluate the output of large language models without the ground truth, if we have complementary labels. We further show that we can train an Al system with such weak signals: we show how we can design an agentic Al system automatically that can improve itself with this partitioned human su- pervision. Our code is available at Q Towards Scalable Oversight via Partitioned Human Supervision
INTRODUCTION
As foundation models and artificial intelligence (AI) systems (OpenAl 2025a Anthropic, 2025 DeepSeek 2025 Midd 2025) approach and in some areas surpass expert human performance, supervision itself becomes key bottleneck. Current alignment pipelines such supervised fine- tuning (SFT), reinforcement learning from human feedback (RLHF; Ziegler et al. (2019): (Stiennon
et al. 2020); Quyang et al. (2022)), or reinforcement learning from verifiable rewards (RLVR; Shao et al. (2024) Yu et al. (2025): Lambert et al. (2025)) presuppose that humans can reliably provide supervision design verifiers for training and evaluation. Yet for the superalignment regime (Bow-) man et al. 2022). future models will tackle problems whose solutions are too technical or too cross- disciplinary for any single human verify comprehensively. When cannot produce ground truth or prepare automated verifiers, how should we evaluate and train such systems?
We begin from an observation about the expertise in high-skill domains in the human society (Wuchty et al.) 2007). As tasks grow in difficulty, human experts specialize ever more narrowly: cardiology vs. oncology medicine, sector-speciff analysts in finance, or sub-subfields in science. Specialists can often make local determinations with high precision, e.g., cardiologist might say, "this is not a cardiovascular disease." " while an oncologist might point out, "this pattern is never seen in the field of oncology." While such judgments may fail to positively certify correctness, they readily certify that certain options are wrong. In other words, even when positive ground truth is out of reach. human experts can often provide reliable complementary labels (Ishida et al. 2017 Yu jet al. 2018; Wang et al. 2024a 2025): labels indicating one or more options that are definitely the incorrect.

---

Published as a conference paper at ICLR 2026
Tasks Beyond
Human Supervision Laver
ALSystem
Human Capabilities
this task domain or expertise
Experts
Dvlluate/ Train
A
Σ
Query
P Parwing
Menory
a Cardfonano der
Complementary Label
Figure I: Superhuman tasks are routed to a human expert who can only give weak signal that they are not capable of solving the task. We show that this weak signal, i.e., complementary label, can be used evaluate/train an Al system.
This paper develops a scalable oversight (Amodei et al. 2016) framework that treats such parti- tioned human supervision as a first-class superviston primitive. Our starting point is multi-choice evaluation setting where ordinary (true) labels may be scarce or unavailable, but complementary labels arise naturally from partitioned expertise and are abundant. We show that one can utilize these complementary labels to propose an unbiased estimator of top-1 accuracy for an AI system under test, and combine them with scarce ordinary labels for refinement of the estimator. The esti- mators allow us to (i) evaluate frontier models without preparing any ground truth, and (ii) use these evaluations as learning signals to design agentic systems on tasks beyond expert human abilities.
Our work complements many other active lines of research on scalable oversight. For example, Seeak-to-strong generalization (Burns et al.] 2024) studies whether strong models trained on weak (noisy) supervisors exceed those supervisors, possibly by eliciting knowledge that the strong model already possesses. Easy-to-hard generalization (Sun et al.) (024) trains evaluators on easier problems and applies them to supervise harder ones, by leveraging the assumption that evaluation is easier than generation. Debate (Khan et al.] (024) elicits truth via adversarial arguments that a judge selects among. Reinforcement learning from Al feedback Constitutional Al (Bai et al.) 2022) replaces human preferences with Al-generated feedback. Recursive task decomposition (Wu al. 202T) breaks the problem into leaf subtasks that humans can directly label or compare, for tasks that are easily decomposable. We address new, orthogonal axis: when human experts cannot supply correct answers for hard, cross-domain instances, can we use their partitioned "this option is wrong" judgments to obtain performance estimates and useful training signals?
We formalize a labeling protocol in which each instance is routed to randomly chosen domain spe- cialist responsible for exactly one option (class). The specialist either confirms that option (yielding an ordinary label) or confidently rejects it (yielding a complementary label). Under this uniform wrong-index design, we derive an unbiased estimator of accuracy from complementary labels alone via a simple linear correction, and characterize its variance. We then introduce two mixture esti- mators that combine ordinary and complementary labels: (i) an wwverse-varinnee-weighe (IVW) estimator with a practical plug-in form, and (ii) a closed-form maximum-likelihodd estimator (ML).
We empirically validate the framework in three settings. First, we perform statistical validation on many popular large language model (LLM) benchmarks, which confirms that we do not need the ground truth evaluate the performance of AI systems experimentally, such as LLMs. We focus on real-world evaluation on tasks that cannot be solved by a single expert human with only narrow expertise. As a proof-of-concept, we use finance and medical benchmarks demonstrate that partitioned feedback from sector analysts or highly-speciaalzzed doctors enable accurate model evaluation without the ability to solve the task alone. Finally, we show that this weak signal can be used as training signal for an AI system. More specifically, we use complementary labels for agentic training: we replace ordinary accuracy with our estimator as the fitness signal inside agent search pipelines, and show that this improves downstream performance, demonstrating a pathway to training when only complementary feedback is available.
Our main contributions are as follows: 1) We introduce scalable oversight via partitioned human su- pervision, a protocol that exploits real-world specialization to collect complementary labels at scale for superhuman tasks. 2) We derive unbiased complementary-labe estimator for top-1 accuracy, analyze its variance, and show how many complementary labels are needed to match the variance of the estimator obtained from ordinary labels. We further propose two mixture estimators together
2

---

Published as a conference paper at ICLR 2026
with lintee bounds and practical plug-in implementations. 3) We demonstrate empirically that these estimators enable both evaluation and training without the full ground truth.
METHODOLOGY
In this section, we explain our problem setup and notations and explain our proposed method.
Setup and Notations We consider a multiple-choice evaluation task with K options per item (e.g., K for choices A, B, C, and D). Formally, each item is represented by an input æ X, where X' denotes the input domain (e.g., the space of question texts). Each item has a correct label y {I..... K), representing the index of the correct option after any shuffling. We call this the ordinary label. Let (X, Y) denote random pair drawn from an identical joint distribution D over X' (1....,k) K).
Equivalently, we can regard to first sample item X, and then a human expert provides label Y based on the underlying conditional distribution (Y)))). An Al system (e.g., a machine learning classifier, a single LLM, or an LLM agent/workflow) produces a prediction Y = f(X) E f1...., K}. For an item with an ordinary label y. define the indicator Z = 1(Y = Y} E {0, 13. The (population) top- Pr(Z Z 1) = (X) Y). where Pr(-) denotes probability with respect
accuracy is then
to the joint distribution D.
We assume the complementary label is drawn uniformly at random among the non-y classes (Ishida et al. 2017): :
1
p(
K-1 *
That is, for each item, the revealed wrong index is sampled uniformly from the K - non-Y classes. For an item with complementary label Y * Y. define W = IςÝ & +Y} E 13. The assumption in Eq. [ enforced by the following data collection protocol. We assume that we have a set of questions { J.Y and K annotators, where each annotator is human expert in the field of one of the classes. For each question z,, choose an annotator randomly (let's say an annotator in charge of the k-th class is chosen), and ask "Is the answer of question z; class A?", If the answer is yes, save this in the ordinary-labe set as an item with (ordinary) class & label. If the answer is no, we item with (complementary) class & label. Under this
save this in the complementary-labe set protocol, the uniform sampling assumption in Eq. [ holds by construction (Appendix [T]. Since each expert queried with equal probability and only performs binary compatibility check for their designated class, the potential for systematic annotation bias is substantially reduced.
Finally, we will have two disjoint evaluation sets, drawn from the same distribution and evaluated by the same system: 1) An ordinary-labed set of size no, yielding realizations 21,..., Zn, and total cor- rect count S, Σy= Zj. 2) complementary-ubbe set of size nee yielding realizations wi....y Wn. and total "weakly-correct" count S. =En
Se. wy- The total number of samples is N= = no + ne.
2.1 ESTIMATOR FROM COMPLEMENTARY LABELS
It is straightforward to estimate the system accuracy by using ordinary labels as Aora Se If we want use complementary labels estimate A, a naive approach would be count the proportion of "avoiding the complementary label," i.e., = Σi w; However, q is not an unbiased estimator of the accuracy. Thus, we propose the following complementary-labe estimator: Corollary 1. Under the assumption in Eq. 7 the estimator
Acomp = (K 1)g (K 2) is unbiased for A, where 2 3 is the number of choices.
We show the proof in Appendix A This linear correction is precisely the 0-1 loss specialization of the risk-rewrite identity of Eqs. (8) and (10) in Ishida et al. (2019); replacing expectations by sample means yields our unbiased estimator. A full derivation is provided in Appendix
'eere "weealy-correet" means that the model prediction is not equal to the complementary label. For example, when K avoiding the complementary label still leaves a 2 chance of being wrong.
3

---

Published as a conference paper at ICLR 2026
Even if we have unbiased estimator with only complementary labels, the variance is expected to become larger compared with the case when we use the same number of ordinary labels. This is because complementary labels are "weaker" than ordinary labels in the sense that K different complementary labels equivalent to ordinary label. We first confirm this as follows. Since W is obtain
Bernoulli with mean q = 1+*-2
K-1
(K -
(A+ K - 2) A) q(1 (1-9) ne
1)2
-
Var Var(Acomp) (K 1)2 Var
W.
12
Tee
This expression depends the unknown A. In practice, plug-in estimator is used. For the ordinary-labbe dataset, the number of correct predictions satisfies S. ~ :in A), where Bin(n. denotes the binomial distribution with parameters n (number of independent trials) and p (success probability in each trial). Thus, [[] = A, and Var(S./no) = A(1 - A) where E[-] and Var(.) denote expectation and variance, respectively. taken with respect the underlying population distribution D over (z, y). Comparing this with Eq. (3) the variance ratio is
satisfiee
Var (comp)
(A + K 2) no
(4)
Var(S./mo)
A Te
Hence, when Ro. the complementary estimator has larger variance for all K 2 3, as expected. Next, it would be interesting derive how much more complementary labels we will need to prepare to have the same amount of variance as the ordinary label case. To match the variance of the ordinary estimator, the size of the complementary dataset must be
( 1+
K 2 A
The =
No.
(5)
That is, the larger A is, the fewer additional complementary labels are needed. Since A is unknown, small ordinary-labeled bound A and infer the required ne, motivating the com- bined estimator in 2.2
2.2 COMBINING ORDINARY AND COMPLEMENTARY LABELS
We now combine information from both datasets. Recall Aord S. and Acomp 1K-1) -(K- 2) are both unbiased under the independent and identically distributed (i.i.d.) sampling assumption.
Inverse Variance Weighted Estimator (IVW) Consider the linear combination Amix = w Aora
(1 - w) Acomp where w E [0, 1]. Because the two components are independent and unbiased (disjoint samples), Amix is unbiased with variance
Var (ar(A)ix) 2 A(1-A)
+ (1 w)2((-11 q(1 9), q
4+K-2 K-
nhe
Minimizing it w.r.t. w yields
(K-1)2 g(1 - 9)
t6*
(6)
A(1 A)
(K 1)²
q(1 - -Q)
A(1-A) (K-1)²
Ne 9(1 9(1-0)
The corresponding minimum variance equals Varmin (Auix)
no
Since A is
All A(1-A)
(K 1K-1)" 9(1
mhe
unknown, plug-in estimators: Var(Aued)
Aoea (1- Aord) and Var (Acomp)
(K-1)² q(1
ne
no
i), - Sa Then
Var Acomp)
where Årvw w Anni + (1 w) Acomp
Var (Aord) Var(Acomp)
Its (plug-in) variance is the harmonic mean:
4

---

Published as a conference paper at ICLR 2026
Var(Awww) Var(Aora) Var (Acomp)) (aaloompp///((aaaaaaa (Var(Aord) + Var (ar(Acom))) Substituting q A+K-2 into Eq. (6) gives
w* (A + K 2) no
(8)
Ane + (A+ K n)
and replacing A by a pilot Á (e.g. = Aora). yields a closed-form estimator of w". If we prepare separate set other than ne no samples to estimate the weight, of Amixi not (as in our experiments), we will show
maintain the unbiasedness 2.3 that we have consistent estimator.
Closed-Foo Form Maximum-Likehine in A given by S. Bin A). The complementary set induces likelihood term S. Bin (me: q) with q 1t K- Assuming independence between the two datasets, the joint log-likelihood is K-1
Estimator (ML) The ordinary set induces likelihood term
((A) = So log A + (n. S.) log A) + Se log q + (ne - Sc)log log(1 - q)-
Setting 0e/0A- yields the score equation S n.-3.
1-A + A+k
n- 0. which simplifies
I-A
to quadratic form in Let To = no - S. Tc = nee Sc, a = N, B = (K 2To - Te) + (K
3)5. Se. 7 (K 2) Then the ML is the unique root in [0. 1]:
AML -3 + v B2 4cry
(9)
2a
1/2
A large-sample standard error is se(ANI) 2 served or expected information.
Ro
based on ob-
AMe AMEL)
[K-
If ne = 0. Eq. reduces to Aord- If no = 0, reduces to Acomp. For = 4, one always has i E 12 when A E [0, 1]. which bounds the caaplemmnntryycompoeee variance. Moreover, a one-step Newton update around the truth shows that the IVW estimator with optimal fixed weights coincides with the ML estimator up to the first order and both are statistically efficient.
2.3 THEORETICAL ANALYSIS
We analyze the deviation of the complementary-laael estimator Acomp under the uniform wrong- index assumption Eq. [1 Although Aoomp unbiased, variance can be non-negligible at small sample sizes. We present two finite-sample deviation bounds: distribution-free Hoeffd- ing bound (Boucheron et al.] (2003) and a data-dependent Bernstein bound (Maurer Pontil 2009) leveraging empirical variance. We then extend these fixed-weight mixture estimator.
2.3. BOUNDS FOR Ācomp
We derive a unified bound for Acomp by taking the minimum of the Hoeffding and empirical Bern- stein inequalities. This formulation is both variance-free (via Hoeffding) and variance-adaptive (via Bernstein). The proof is shown Appendix E
Theorem 2. With probability at least - 8,
(K - min {
Aromp A S
log(2/6)
24(1-g) me
log &
7 log(1/8) +
2
3 (n
The first branch (Hoeffding) is simple and variance-free but conservative. The second branch (em- pirical Bernstein) adapts to the empirical variance q(l - q) and is typically tighter when g is close to or 1. Both bounds require only i.i.d. sampling of (W;) under Eq. (1) and hold for any ne-
2.3.2 BOUNDS FOR Amix
Consider the mixture estimator with (possibly data- weight w [0, 1]: Amix = Aord + (1-w) w) Acomp where Aord n. Σx= Zj. and 2j E {0, 1}. Since the ordinary and complementary sets are disjoint, we have Amix A = w( Aord A) + (1 w) A). With a union bound and Hoeffding inequality, we can derive the following proposition.
A detailed proof is provided in Appendix
5

---

Published as a conference paper at ICLR 2026
Proposition 3. For any split å = 60 + de, with do, O, with probability at least 8,
w
(2/62/
2no
+ (1 - w)((K 1)
l2g(2/8.)
2np
Amix Al S
min
w
2 Aard(1- Aordl
log b, + [m
7 (o/(4/)))
th
+ + (1- w) (V
4(1-9) Be
4 log
((K-1) (og(4/6.)
8.
3 [mc
A symmetric choice 8, 6c 8/2 yields the simpler logs log(4/6) and log respectively.
The union- boound approach provides guarantees that hold for any weight w E [0, 1], including data- dependent plug-in choices such as #tvw. This makes it directly applicable in practice, although the constants are typically looser than those obtained from the moment generating function based Bernstein bound (See Theorem
We now state Bernstein-type PAC bound for the mixed estimator that complements the union- its proof is deferred Appendix
bound inequalities established in Proposition 3
Theorem 4. With probability at least - 8, the following holds for Amix:
Amix A
v
2v log
+ log 3.
(10)
where 2 A(1-A) (1 ) (K 1K-1) g(1-9) and max nd
(1-w)(K-))
ne
no ne
This bound tightens Hoeffding's inequality by incorporating both the variance term and the (range- driven) linear term c. valid for any fixed choice the weight E [0, 1]. for instance if w set constant or estimated on independent pilot set. If instead is estimated from the same evaluation data (c.g. the plug-in WTyw), the fixgh-weight assumption is violated; in that case one must either revert to the union-bound bounds (Proposition 3). apply standard remedies such as sample splitting (Chernozhukov et al., (0018) WWager & Athey 2018) a finite grid with a union bound (Boucheron et al. [013] and further discussion, see Appendix G
& Ben-David 2014). For practical plug-in forms
3 EXPERIMENTS
We evaluate our unbiased and mixed estimators in three settings: (I) Statistical validation on standard multiple-choice benchmarks, where we verify unbiasedness and variance properties (see Eqs. (2). 4 [6):: ) (II) Practical evaluation on two real-world classification benchmarks: the Japanese financial dataset EDINET-Bench (Sugiura et al.) 2026), where each option corresponds to a professional industry domain and annotators provide weak complementary feedback; and the En- glish Medical Abstracts dataset (Schopf al. 2023). which covers multiple disease categories with natural clinical text. (III) Agentic training in agent- based workflows, where we replace ordinary accuracy with our proposed estimator as the fitness function and examine the resulting agent perfor- mance. For agent scaffolding, we use ADAS (Hu et al. (025) and AFlow (Zhang et al. (025b). For details of the labeling protocol, please refer to Appendix
3.1 STATISTICAL VALIDATION OF COMPLEMENTARY AND MIXED ESTIMATORS
Statistical validation is conducted on four benchmarks. Since oracle labels available, comple- mentary labels are synthetically constructed following Appendix C We report three types of evalu- ation metrics. (A) Unbiasedness and variance are examined by subsampling 300 ordinary and 300 complementary labels (120 for GPQA due to dataset size). Accuracy is estimated using Eq. ( and variability is quantified through standard deviations, reported Ā+ with & denoting the estimated (B) The relative efficiency of complementary labels is assessed by computing the number of complementary samples required to achieve the same variance as n. ordinary labels, according to Eq. 4.. where the full ordinary dataset is used to approximate the ground -rutt aceu- racy A. (C) To mimic practical annotation protocols (Appendix Cp. assume that each ordinary
standard deviation.

---

Published as a conference paper at ICLR 2026
label can be replaced by (K- 1) complementary labels, where K is the number of answer options. This setting is used to evaluate the mixed estimators in Eq. and Eq. (9)
Each dataset is evaluated all five metries using gpt (OpenAl 2025b), with three in- dependent runs averaged; aggregated outcomes are summarized in Table For the MATH-MC benchmark, use GPT-4. 1 -nano instead of GPT-5-nano. This choice is due to (i) output- format constraints in CPT-5-nano that confiic with our final-answer extraction protocol, and (ii) the need avoid ceiling effects: in our pilot runs, GPT 5PT-5-mas 5-nano attains near-ceiling accuracy on MATH-MC, which compresses the observable gaps among estimators.
Table 1: Performance of estimators across benchmarks. Values reported mean accuracy +: standard deviation across three random seeds; the average of per-run standard deviations is additionally shown in paren- theses. A brief description of the estimators is provided in the legend helow Detailed estimator setups and confidence-ineed interval analyses are provided in Appendix ! The last column reports the macro-an averrge across datasets (counting Math and Math-COT separately). For each benchmark, the estimator whose mean accuracy closest to the Ord-Eval reference is highlighted bold. Column deviations from the oracle (A) are reported in Appendix D
Estimator
MMLUU-Pr
MedQA-USMME
GPQA
MATHI
MATH(CoT)
Avernge
Ord
78.33 (2)2.88)
92.89 1.35 (2:148)
64.17
(=4.39)
47.56 3,91 (12.88)
(1:2.77
Comp n,
77.00 1: 12.49 (1.7.95
92.67 : 1.53 (12667)
59.17
3.82 (2942)
48.44 1 10.78 (L7.72)
80.44 4: 2.78 (1:.98)
71.54
Comp-Va Var
75.67
2: 2.15 (12.51)
90.61 : 1.43 (2:16)
63.67
2:0:1 (24.22)
41.10
(12.3)
81.35 : 0.29 (12.28)
70.18
IVW-0.5
77.89
(.1)8))
91.61
1.11 (1.11)
65.28
1,34
43.44
3,95 (#2.52)
83.56
1.17 (s:158)
72.36
IVW
77.97
1,58 (+1.79)
91.86
(+1.1)
65.14
1.38 (-3.0)
44,87
182 (+2.37)
83.86
0 (+1156)
72.74
ML
77.94
= 1.58 (1:1.79)
91.65
1.08 (11118)
65.11 = 1,38 (=228)
44.75
3.79 (12.36)
83.65
+ 1.0
72.62
Ord-Eval
77.97
92.66
59.52
44.21
83.89
Legend:
Ord ordinary-label estimator;; Comp-nt, complementary labels with the same sample size as ordinary: Comp- complementary labels with variance matched sample size; (WW.055- = fixed 0.581.5 weighting of urdinary and complementary estimates; IVW = inverse variance weighted combination piug-in weights; MI. = joint maximum- likelihodd estimator using both label types; Ord.Eval Eval oraele accuracy the fall datasct.
Table summarizes the results across benchmarks, together with a reference evaluation using all available ordinary labels (Ord-Eval). Importantly, higher accuracy in isolation does not imply better estimator; the desideratum is to approximate the Ord-Eval reference as closely and reliably as possible under scarce supervision. With finite ordinary labels and (K 1)-times complementary labels (where K is the number of choices), Ord and Comp-n. can occasionally yield point estimates that are numerically closer to Ord-Eval (as seen in MedQA and GPQA). However, their standard errors are substantially wider, indicating poor reliability and limited reproducibility. For Comp- Var, increasing the number of complementary labels reduces variance, consistent with Eq. (3) and Eq. { We report both across-seed variability and the average within-run standard deviation, the latter more directly reflecting estimator variance. Ord-Eval itself should not be regarded as the ground-truth esppralyye e on GPQA, where dataset size is small and uncertainty remains substantial but rather as an empirical reference.
Mixture estimators (IVW and MI.) re- duce the gap to the Ord-Eval ref- erence compared with both ordinary- only and complementary-only estimators across most benchmarks. Among mix- ture estimators, IVW consistently im- proves over the equal-weighted baseline in within-run standard deviation (IVW-0055, confirming that inverse-variance weight- ing provides the optimal linear combina- tion of ordinary and complementary la- bels. ML further achieves narrower inter- vals owing to its joint likelihood formula- tion. However, ML requires more precise pilot estimates of accuracy and may thus be less robust in practice, whereas IVW remains stable by directly incorporating variance esti- mates. This distinction is supported by the within-run standard deviations, which indicate that IVW achieves competitive efficiency with simpler computation.
Ortre if Bond
Optral
Samples 1x3001
Samples 300
(a) Optimal vs. number of (b) Corresponding variance samples.
on a log scale.
Figure 2: Comparison of optimal weighting and baseline (w 0.5). Optimal weighting consistently achieves lower variance compared to the equal-weighted baseline, demon- strating more reliable estimation with limited samples.
7

---

Published as a conference paper at ICLR 2026
Table 2: Mean acc. + standard devi- ation across seeds, w/ the average of per-run standard deviations. Bold: Lowest variance across 3 runs. See App-Ofor details.
Estimator
EDINET
EDINET (Extended)
Medical Abstract
Ord
5.59 (+7.13)
2,46 (
70.38 0.62 (=0.6)
Comp-(K-1)
10.75 + (+1664))
26.60 +.000:3.44 3.45 (45.86)
68.79 + 0.71 (+1.13)
IVW-0.5
15.05 5.18
2.21 (4:327)
69.58 0.57 (=0.74)
IVW
17.95
4,88 (466533
22.64 + 4.27 (4:2.59)
69.71 0.56 (-0.73)
MI.
17.93
4.87 (1.6.29)
22.61
2.28 (12.26)
69.70 0.56 (1.0.4)
Overall, while Comp-no may occasionally appear closest to Ord-Eval in mean accuracy, its insta- bility makes it unreliable for reproducible evaluation. IVW and ML instead strike the best balance between bias and variance, yielding stable and reproducible estimates close to the Ord-Eval refer- across datasets. The empirical closeness of the two methods further validates theoretical analysis, suggesting that mixture estimators can serve as a practical foundation for scalable oversight in settings where ground-truth labels are scarce infeasible.
Ablation Study on IVW. Recall that the optimal coefficient for IVW is given in Eq. (8. When 1)m,. (where K denotes the number of choices), the expression simplifies to the coefficient lies in the range S
setting n. (K
A+K-2
11 (K-1XK-2] AK+K-2 Since
AR+K-2
In particular, when A close l-as suggested by the Ord-Eval reference- optimal weight approaches 1. explaining why in Table the IVW estimator often performs similarly to the ordinary baseline.
(K-IXK-2)
AK+K-2
To further examine whether the closed-form coefficient yields the optimal trade-off, we conduct an ablation MedQA-USMLE by varying the number of complementary labels from ne no to ne = 20n, (with n. = 300), Results are averaged over three random seeds. As shown Figure 2 when n. & 3n, (the setting used in Table D. the optimal weight is close to the fixed baseline w = 0.5. However, for nee < no or ne > 6n., the optimal variance is consistently lower than the baseline. This behavior is intuitive:
the number of complementary labels exceeds the variance- matching threshold in Eq. (4), the complementary estimator alone has smaller variance, and the closed-form IVW solution naturally shifts weight away from ordinary labels. By contrast, the fixed baseline cannot adapt, leading to consistently higher variance.
3.2 BENCHMARK EVALUATION IN REAL- WORLD TASKS
To demonstrate the practicality of our method in world scenarios, we conduct proof-of-concepp experiments for estimation and leurning under partitioned human supervision, as directly collecting superhuman tasks would prohibitively costly. Following the data collection protocol Algo- rithm [ for each domain we randomly select expert and ask whether given object belongs to their field. Applied to the benchmarks, complementary labels are generated from the oracle labels by splitting each instance into 1 ordinary label and K complementary labels, corresponding to the candidate classes.
We evaluate two benchmarks. First, we use EDINET-Bench (Sugiura et al. 2026), Japanese financial report classification dataset, where each industry corresponds to a professional domain, and we adopt the prompting setup of (Sugiura et al. (2026). Since the original dataset contains only 496 samples (comparable to GPQA), we extend it following their released code, resulting in 3,269 samples; we refer this as EDINET-Bench Extended. To our knowledge, there are no public medical benchmarks in which each class explicitly corresponds to medical specialization. We therefore adopt the Medical Abstracts dataset (Schopf et al. 2023), which consists of paper abstracts labeled by disease categories, simulating the task of narrow domain experts classifying diseases.
Both EDINET-Bente Extended and Medical Abstracts are imbalanced across classes. Nevertheless, our assumption in Eq. [1 is that complementary labels are sampled uniformly at random, indepen- dent of the original label distribution. We follow this procedure and empirically verify that class imbalance does not affect the validity of our estimator.
Results As shown in Table both IVW and ML consistently achieve the lowest average per- standard deviation, confirming the variance reduction effect observed in $ [..]] In contrast, the Comp-(K-I estimator exhibits much larger variance on EDINET-Bennn which aligns with Eq. 4 with K = 16 industries and relatively low accuracy, the variance is amplified.
8

---

Published as a conference paper at ICLR 2026
Math-MC
Medicall Abstract
Figure 3: Accuracy across dif- ferent benchmarks. Across GPQA, Math-MC, and Mcd- ical Abstract, agentic sys- tems guided by weak comple- mentary signals (ADAS and AFlow) consistently outper- form manually designed base- lines.
100
Chair of Thought
Se1 Zarine
Step back Abstraction Quellyytrenetty
Sode Axegrmmt
XDNS
LLM Datelee
Comparing EDINET-Bench and its extended version, we observe that increasing the number of samples substantially reduces the fluctuations of the estimators, supporting our earlier finding in $ [311 that small-scale benchmarks such as GPQA can lead to unstable estimates. Finally, on the Medical Abstracts dataset, IVW and ML again achieve the most stable performance, demonstrating that the complementary-label estimators generalize across domains and languages.
3.3 AGENT SEARCH WITH ESTIMATOR FEEDBACK
Finally, we show that weak human supervision can be leveraged not only for evaluation but also as training signal for AI systems. We evaluate agentic methods under complementary-labed sig- nals on three benchmarks. GPQA serves as the most challenging setting our study (Table [ Math-MC focuses on reasoning ability; and Medical Abstracts best captures our practical scenario. Following Hu al., (2025), we include CoT (Wei et al. 2022). CoT-SC (Wang et al.) [023)) Self- Refine (Madaan et al. (023), LLM Debate (Du et al. 22224) Step-back Abstraction (Zheng et al.)
2024). Quality-Diversity (Lu et al. 2025), and Role Assignment (Xu et al.) (025)) as manually de- signed baselines, and compare against ADAS (Hu al. (225) and AFlow (Zhang al., 2025b).
We define the accuracy computed by q as complementary accuracy and apply Eq. (2) to obtain the transformed complementary accuracy. For ADAS and AFlow, follow the original protocol (see Appendix[] J computing both complementary and transformed complementary accuracy on the validation set and reporting the better-performing variant.
Figure 3 compares ADAS and AFlow against strong baselines on GPQA, Math-MC, and Medical Abstracts. Both agentic systems consistently outperform manually designed workflows once weak complementtry--lbel signals are incorporated, demonstrating that weak human feedback can pro- vide learning signal. While the choice of estimator (raw vs. transformed complementary accuracy) leads to different outcomes depending on task difficulty, the overall pattern remains robust: weak signals suffice to improve performance beyond established baselines, with additional variance anal- ysis provided in Appendix K] To further illustrate the resulting agentic system's 's behavior, we present a case study in Appendix M These findings reinforce the central theme of our study: humans' weak labels can provide reliable and scalable supervisory channel for steering agentic Al systems.
RELATED WORK
Scalable oversight In superhuman regimes where Al systems surpass human abilities, humans longer provide reliable supervision to evaluate and train them. Recent scalable-oversight work explores strategies that let limited (human) supervisors steer stronger models. Among the many scalable oversight protocols (such as weak-to-strong generalization (Burns et al. (024). easy- to-hard generalization (Sun et al. 2024), debate (Khan et al. 2024), and RLAIF (Bai et al. 2022 our approach may seem similar to iterated amplification/IDA (Christiano et al., 2018) and recursive task decomposition (Wu et al.) 202 at glance. The difference is that our approach can be used for tasks that are not easily decomposable, as we are not decomposing the task, but rather relying on expertise-wing wise partitioned human supervision to enable the usage of complementary labels.
Weak supervision Weakly-supervised learning (Sugiyama al. 2022 studies learning with weaker form of supervision instead of ground-trut truth labels. Complementary labels (Ishida et al. Yu et al. 2018. Wang al. 2024a 2025). in particular, provide "not-this-class" in-
2017 2019

---

Published as a conference paper at ICLR 2026
formation, with the traditional motivation of reducing annotation cost. Our perspective differs: leverage weak supervision not merely for cheaper labeling, but as a principled route to scalable over- sight in the superhuman regime, where no single human expert can provide verification end-to-end.
Agentic systems LLMs have inspired wide range of system architectures (Wei al., 2022 Wang
iet al. 2023; Madaan et al. 2023 Shinn et al. 2023). in particular by incorporating multiple LLMs
into collaborative frameworks (Du et al. [024; Liu et al., 2024 Chen al. 2024 Wu et al. 2024
Wang et al. 2024c). Early studies relied on manually designed agentic workflows, which demanded substantial manual design and domain expertise. More recent work has shifted toward automated agent discovery, in which LLMs themselves are used to generate, refine, and optimize workflow structures (Hu et al.
2025 Zhang et al. 2025a 2026; 2025b), but required ground-truth supervision to construct fitness score. We investigate whether automated agentic system design can proceed
even with weaker training signals, i.e., complementary labels.
DISCUSSION & LIMITATIONS
The assumption of uniformly querying an index plays a key role in obtaining unbiased complementary labels, and in our labeling protocol this assumption is guaran- teed for the practical scenario we consider. However, to make the whole system more robust, we should also consider possible issues that may arise in practice, such as biased answers from experts, uncertain answers (which can be reframed as a type of noise), and overlaps between expert fields. For these cases, a wide range of complementary-late plementary labels (Yu et al. 2018 Wang et al. 2024a), noisy complementary labels (Ishiguro et al.) 2022), and multiple complementary labels (Feng et al. 2020). We believe that incorporating these formulations would further enhance the robustness of our framework.
Assumptions and extensions
variants has been studied, such as biased com-
Beyond the assumptions on the labels themselves, we also need to consider the structural assump- tion that the task is posed in a multiple-choiee format. The multiple-choice format still has wide application, and in scenarios beyond the standard multiple choice setting, some open-ended ques- tions can be transformed into structured multiple-choice forms to allow the use of our framework. For example, in debugging, modern systems strongly rely on cooperation among different sectors. The end-to-end service may look open-ended, but it can often be separated into different sectors or a hierarchical structure. In particular, when the root cause is unknown, we can ask engineers responsible for different parts of the product to provide weak signals, for example, confirming that the bug does not originate from their component. This kind of knowledge fragmentation provides opportunities apply our framework.
Limitations Although our framework has shown its potential for addressing the expertise problem in evaluating foundation models, we would still like to discuss its current limitations. First, the framework assumes the availability of complementary labels. When even complementary labels are scarce or inapplicable, for example, in open-ended problems that are hard to decompose into multiple-choice pipeline, complementary labels may be ill-defined, and the formulation may need to be revised. Second, when feedback from humans is no longer available and the boundary of the ground truth becomes ambiguous, neither complementary labels nor ordinary labels can be reliably defined. In such cases, new measures of accuracy would be required to revise the framework. We view these current limitations as interesting directions for future research.
6 CONCLUSION
We introduced scalable oversight via partitioned human supervision, a protocol that utilizes the weak human supervision for tasks beyond any single expert's ability. We derived an unbiased estimator of performance metric from complementary labels, analyzed its variance, and provided finite-sample guarantees. We further proposed two practical mixture estimators that combine scarce ordinary labels with abundant complementary labels. Empirically, we show that these estimators allow us estimate the performance of AI systems without requiring full ground truth. They also serve as effective training signals which we demonstrate with automatically designing agentic work- flows.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
RY was supported by the RIKEN-AIP Undergraduate Research Program, TI was supported by JST ASPIRE Grant Number JPMIAP25B1, and MS supported by JST ASPIRE Grant Number JP- MJAP2405.
ETHICS STATEMENT
Our work motivated by AI safety: we aim to improve oversight of advanced Al systems by enabling evaluation and training even on tasks beyond any single human expert's 's competence. This approach is intended as a step toward improving the reliability and alignment of future models with human interests, which believe is positive contribution to society.
Currently, we do not foresee immediate direct negative social impacts from the research itself. How- recognize that alignment and oversight techniques can be double-edged sword: they dual-use the sense that they could be repurposed by malicious actors to optimize systems toward harmful objectives (Machine Learning Street Talk (025), and even when used defensively, they can be vulnerable to strategic behavior that produces false sense of safety (Hubinger et al. [2024] Greenblatt al., (004).. These risks misuse and vulnerabilities.
unique our approach, but we caution against such
REPRODUCIBILITY STATEMENT
Code and scripts to reproduce our experiments are available in our public GitHub repository: Towards Scalable Oversight via Partitioned Human Supervision
For ADAS (Hu et al. 2025) and AFlow (Zhang et al., 2025b), we follow their official implemen- tations; see Appendix for additional details. Because experiments use the OpenAl API, exact replication may vary over time if models are updated or retired. We therefore provide full imple- mentation details, dataset references, and evaluation scripts support reproducibility closely as possible.
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete Problems in AI Safety. arXiv preprint 1rXiv:1606.06365,, 2016
Anthropic. Introducing Claude 4: Claude Opus 4 and Claude Sonnet 4. Blog post the An- thropic website, May 22 2025. URL Accessed 2025-09-24.
com/ nwws//wwwweeeeeeeeee clw.de-aaeeeeeeeeeea 4
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldic, Azalia Mirhoscini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li. Eli Tran- Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Con- erly, Tom Henighan, Tristan Hume. Samuel R. Bowman, Zac Hatfield -Ddddss Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from Al feedback. arXiv preprint a2 2.00:733, 2022.
Stéphane Boucheron, Gábor Lugosi, and Olivier Bousquet. Concentration Inequalities. In Summer school on machine learning, pp. 208 240. Springer, 2003.
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities A Nonasymp- totic Theory of Independence. Oxford University Press, 2013.
Samuel R. Bowman, Jecyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilè Lukošiūté, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron

---

Published as a conference paper at ICLR 2026
McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran- Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Liane Lovitt, Nelson Elhage, Nicholas Schiefer, Nicholas Joseph. Noemí Mercado, Nova DasSarma, Robin Larson, Sam McCandlish, Sandipan Kundu, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Ben Mann, and Jared Kaplan. Mcasuring progress on scalable oversight for large language models, arXiv preprint (.X.x:2221..35542 2022.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbren- ner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeffrey Wu. Weak-to-Stre -tronn Generalization: Eliciting strong capabilities with weak supervision. In Proceed- ings of the 41st International Conference on Machine Learning, 2024.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian. Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. AgentVerse: Facilitating multi-agent collaboration and exploring emergent behaviors. In Proceedings of the Twelfth International Conference on Learning Representations, 2024.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural pa- rameters, The Econometrics Journal, 21(1):C1-C68, 2018.
Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint /t/0:/8///0857. 2018.
Google DeepMind. Introducing Gemini 2.5 Pro and Gemini 2.5 Flash. Blog post on the Google DeepMind website, May 14 2025. URL hhttps::///eeppe wwwglee gemini/ Accessed 2025-09-24.
DeepSeek. Deepseek R1 updates: Build R1-0528. Model release notes the DeepSeek platform, May 28 2025. URL
deepseek. com) Accessed 2025-09-24.
Yilun Du Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving Factuality and Reasoning in Language Models through Multiagent Debate. In Proceedings of the 41 International Conference on Machine Learning, 2024.
Lei Feng, Takuo Kaneko, Bo Han, Gang Niu, Bo An, and Masashi Sugiyama. Learning with multiple complementary labels. In International conference on machine learning, 2020.
Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, Sören Mindermano, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, and Evan Hubinger. Alignment faking in large language models. arXiv preprint ar arXiv:2412.14093, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Erie Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Ad- vances in Neural Information Processing Systems, 2021.
Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. In Proceedings of the Thirteenth International Conference on Learning Representations, 2025.
Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong. Monte MacDiarmid, Tam- era Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Ka- mal Ndousse, Kshitij Sachan, Michael Sellitto, Mrinank Sharma, Nova DasSarma, Roger Grosse, Shauna Kravec, Yuntao Bai, Zachary Witten, Marina Favaro, Jan Brauner, Holden Karnofsky, Paul Christiano, Samuel R. Bowman, Logan Graham, Jared Kaplan, Sören Mindermann, Ryan Greenblatt, Buck Shlegeris, Nicholas Schiefer, and Ethan Perez. Sleeper Agents: Training decep- tive Ims that persist through safety training. arXiv preprint arXiv:2401.085 055666, 2024.
12

---

Published as a conference paper at ICLR 2026
Takashi Ishida, Gang Niu, Weihua Hu, and Masashi Sugiyama. Learning from complementary labels. In Advances in Neural Information Processing Systems, 2017.
Takashi Ishida, Gang Niu, Aditya Menon, and Masashi Sugiyama. Complementary- learning for arbitrary losses and models. In Proceedings of the 36th International Conference Machine Learning, 2019.
Hiroki Ishiguro, Takashi Ishida, and Masashi Sugiyama. Learning from noisy complementary labels with robust loss functions. IEICE TRANSACTIONS on Information and Systems, 2022.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang. and Peter Szolovits. What dis- ease does this patient have? large- scale open domain question answering dataset from medical exams. Applied Sciences, 1114), 2021.
Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Ed- ward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. Debating with More Persuasive LLMs Leads to More Truthful Answers. In Proceedings of the 41st International Conference on Machine Learning, 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord. Christo- pher Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers open language model post-training. Proceedings of the Second Conference on Language Modeling, 2025.
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. A dynamic LLM-poweree agent net- work for aaientrie agent collaboration. In Proceedings of the First Conference on Language Modeling, 2024.
Cong Lu. Shengran Hu, and Jeff Clune. Intelligent Go-Explore: Standing on the shoulders of giant foundation models. In Proceedings of the Thirteenth International Conference on Learning Representations, 2025.
Machine Learning Street Talk. Don't invent faster horses Prof. Jeff Clune. YouTube video, January 2025. URL pettps://wwtt wwtp://wwwttb.e..oooooooo 09-24.
yout www.tuuu..coo/w/taaaaeoeeeom www.::atthh.......aaa.a.
Accessed: 2025-
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann. Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Relf-Refine: Iterative refinement with self-feed feedbeee In Advances in Neural Information Processing Systems, 2023.
Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample-variance penal- ization. In Proceedings of the 22nd Conference on Learning Theory, 2009.
OpenAl. Introducing GPT-5. Blog post the OpenAl website, August 7 2025a. URL https: Accessed 2025-09-24.
OpenAl. Introducing GPT-5 for developers: GPT-5, GPT-5-mini, and GPT-5-nano. Ope- nAI Blog/ / API documentation. August 7 2025b. URL /tttps://opents www com/index/ index/ Stwpodduiing--pt-----a-
opers Includes the GPT-5 nano model; developer API pricing and performance trade-off'.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. Ad- vances in Neural Information Processing Systems, 2022.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. In Proceedings of the First Conference on Language Modeling, 2024.
13

---

Published as a conference paper at ICLR 2026
Tim Schopf. Daniel Braun, and Florian Matthes. Evaluating unsupervised text classification: Zero- shot and similarity-bar -based approaches. In Proceedings of the 6th International Conference on Natural Language Processing and Information Retrieval, 2023.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory algo-
rithms. Cambridge university press, 2014.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of math-
ematical reasoning in open language models. arXiv preprint arXiv:2402.033000 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Re- flexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, 2023.
Nisan Stienmon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, 2020.
Issa Sugiura, Takashi Ishida, Taro Makino, Chieko Tazuke, Takanori Nakagawa, Kosuke Nakago, and David Ha. EDINET-Bench: Evaluating LLMs on complex financial tasks using Japanese financial statements. In Proceedings of the Eleventh International Conference on Learning Rep- resentations, 2026.
Masashi Sugiyama, Han Bao, Takashi Ishida, Nan Lu, and Tomoya Sakai. Machine learning from weak supervision: An empirical risk minimization approach. MIT Press, 2022.
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to-Hard Generalization: Scalable alignment beyond human supervision. Advances in Neural Information Processing Systems, 2024.
Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228-1242 2018.
Hsiu-Hsuan Wang, Mai Tan Ha, Nai-Xuan Ye, Wei-I Lin, and Hsuan-Tien Lin. CLImage: Human- annotated datasets for complementary-labe learning. Transactions on Machine Learning Re- search, 2025.
Wei Wang, Takashi Ishida, Yu-Jie Zhang, Gang Niu, and Masashi Sugiyama. Learning with com- plementary labels revisited: The glleted-completely wwwpleeeey-a---adddoooooo at-raoomm setting is more practical. In Pro- ceedings of the 41st International Conference on Machine Learning. 2024a.
Xuezhi Wang, Jason Wei, Dale Schuurmans. Quoc V Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency improves chain of thought reasoning in language models. In Proceedings of the Eleventh International Conference on Learning Representations, 2023.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task lan- guage understanding benchmark. In Advances in Neural Information Processing Systems, 2024b.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-person self-collaboration. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2024c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia. Ed Chi, Quoc V Le, and Denny Zhou Sfain-off-Toughhe prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon. Ryan Lowe, Jan Leike, and Paul Chris- tiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109, 10862, 2021.
14

---

Published as a conference paper at ICLR 2026
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi-agent conversations. In Proceedings of the First Conference on Language Modeling, 2024.
Stefan Wuchty, Benjamin F Jones, and Brian Uzzi. The increasing dominance of teams in production of knowledge. Science, 10665
2007.
Benfeng Xu, An Yang, Junyang Lin, Quan Wang. Chang Zhou, Yongdong Zhang, and Zhendong Mao. ExpertPrompting: Instructing large language models to be distinguished experts. arXiv preprint arXiv:2305. 14688, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yue Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Ru Zhang, Wang Zhang, Hang Zhu Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou. Jingjing Liu, Wei- Ying Ma, Ya- Fa-Qin Zhang, Lin Yan, Yonghui Wu, and Mingxuan Wang. DAPO: An open-source LLM reinforcement learning system scale. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with biased complementary labels. Proceedings of the European conference on computer vision, 2018
Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, LEI BAI, and Xiang Wang. Multi-agent architecture search via agentic supernet. In Proceedings of the 42nd International Conference on Machine Learning, 2025a.
Jenny Zhang, Shengran Hu, Cong Lu, Robert Tjarko Lange, and Jeff Clune. Darwin Gödel Machine: Open-ended evolution of self-improving agents. In The Fourteenth International Conference on Learning Representations, 2026.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong-Hut Hui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong. Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating agentic workflow generation. In Proceedings of the Thirteenth Interna- tional Conference on Learning Representations, 2025b.
Ziyin Zhang, Lizhen Xu, Zhaokun Jiang, Hongkun Hao, and Rui Wang. MultipleChooice questions are efficient and robust LLM evaluators. arXiv preprint arXiv: 19966, 2024.
Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoe V Le, and Denny Zhou. Take step back: Evoking reasoning via abstraction in large language models. In Proceedings of the Twelfth International Conference on Learning Representations, 2024.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1wX.::99999.08933 2019.
15

---

Published as a conference paper at ICLR 2026
A PROOF OF COROLLARY 1
Proof. For a single item, under Eq.
B = Pr(Y = Y) Pr(W = 1 Y) y + Pr(y *
*Y) = =Pr(Y = Y) 1 Pr( * *Y) (1 - A 1+K-
= P +
K -
If consider q ]. Hence, Acomp (K 1)4- (K- - 2) is an unbiased estimator of A.
ni E Wi empirical version of E[W], this is unbiased estimator of
B CONNECTION BETWEEN THE COMPLEMENTARY-LABE ESTIMATOR AND COMPLEMENTARY LOSS
Notation. Let (X, D with Y E [K] (I,.... K}. Given a model g. denote its prediction
by Y g(X). For any class E [K], the loss is E(k, Y). In complementary-labe learning (Ishida
al. 2019), one observes Y * Y sampled uniformly from [K] \ (Y}. The complementary loss is
defined (Eq. (10) in (Ishida et al. 2019) as
K
((,, = (K-i))(k, +
Σ j=1
Specialization to 0-1 loss.
For the 0-1 loss E(k, 8) = 116 + kj. we compute
K
= - 1) * k} + Σ 1| イン
j=1
-K - 1) k + (K 1)
-(K-1)I($=
(11)
Risk under complementary labels.
The standard risk is
((x,))((XX))) = Pr(y YY =
where = Pr(Y Y) is the accuracy. By the risk-rewrite identity of Ishida al. 2019) (Eq. (8)),
we also have
R= Ex,, V)] -K-1) Pr(p=1)
Equivalence our estimator.
Let W Tイ #Y. Then Pr(y F) = E(W), so
1 1 (K 1) - E(W]).
Rearranging yields
A= (K 1) [| (K 2).
This is exactly the linear correction in Eq. 2 Replacing E[W] by its empirical mean q
Dhe W/ gives the unbiased estimator Acomp (K 1)4 (K 2) of Theorem
ne
C DATA LABELING AND ESTIMATION PROTOCOL.
In multiple-choice evaluation, the effective "class" is the index position of an option rather than its textual content. To satisfy the uniformity condition in Eq. (T). the labeling process proceeds as follows. For each item, all options first shuffled uniformly at random. After shuffling, the correct index y is recorded. A complementary label is then created by sampling a wrong index ğ uniformly from the K remaining positions. The final published datum is (z, shuffled options, 7). When
16

---

Published as a conference paper at ICLR 2026
the ground truth is known, this procedure can be automated: is only used internally for generating y. and the released dataset never exposes y. When labels are collected from human raters, the annotation interface
enforce the same randomization in the backend to guarantee uniformity. Any deviation, such as asking raters to pick the "most plausible wrong option," would violate the assumption behind [
The same protocol can be extended to cases where the ground truth not available in advance. A complementary label is first generated in the same way, and then the annotator is asked verify whether it happens to be correct. This verification yields a mixture of ordinary and complementary labels, which forms the mixed dataset studied in $2.2
The end-to-end evaluation pipeline, including both ordinary and complcmentary sets and the as- sociated estimators, is summarized in Algorithm ! Two estimation strategies are supported: closed-form maximum-likelihon estimator and an varissevvrraane weighted (IVW) estimator.
Algorithm 1 Collecting complementary labels and estimating accuracy (fixed K)
1: Input: MCQ items; evaluation model f; sizes Roy
2: for each item do
3: Shuffle options uniformly; record correct index
4: if' item E ordinary set then
5:
Evaluate obtain prediction ÿ; record
1(5 y)
6:
else
& complementary set
7:
Sample ý uniformly from f1..... K} {u}: publish (x. options, g).
8:
Evaluate to obtain prediction ÿ; record 11ÿ + 7)
9: end if
10: end for
Compute S. =,=ı, 2z. Se Ei Wi.
12: Option A (ML): return AML via Eq. (9) and confidence interval from observed information.
Option B (IVW): compute Aord, Acomp. and W, then return Årvw with plug-in confidence interval.
Two assumptions are critical for the validity of these estimators. First, the wrong index in cach complementary label must be sampled uniformly; systematic biases, such consistently preferring hard negatives, lead to biased estimates of Acomp- Second, the ordinary and complementary sets must be drawn from the same distribution of items. If domain shift or subject imbalance exists, the combined estimator reflects accuracy on mixture distribution. In such cases, stratified or subject- wise weighting may be preferable.
D ONE-STEP NEWTON UPDATE AND IVW EQUIVALENCE
Setup and notation. Let A E (0. 1) denote the (unknown) accuracy for K-way multiple-choice task (K 2 3). We observe two independent samples: (i) ordinary labels S, Bin(n,, with empirical accuracy Aord = So/nos no: (ii) complementary labels obtained by uniformly sampling a wrong option per item, that S. Bin with
A -
(A)
T=Se/na Acnmp (K - 1)7 (K - 2).
K 1
We consider pilot Ao (e.g, Aord Acomp) and write qu 9(Ao).
Newton update and plug-in simplification. The joint log-likelihood!! is
((A) = S. log A + (n, S.) log(11 - A) + So log a + (ne - Sc) log(1 (1 q).
Differentiating gives the score functions
(A(A) no - AA)
U.(A) = ne ne(AAomp - -A)
A(l A)
(K 1)2 q(1 gg)
17

---

Published as a conference paper at ICLR 2026
The observed information terms Tobs (A)
no S.
Tobs(
1
S.
Ne
A +
+
(1 - A)2
(K - 1)2
g²
(1 q)
one-step Newton update around Ao is
(Aa) A = Ao U'(A))
Uo(Ao) Uc()o) Ao + Tobe (An) + Iobe (Ao)
When set Ag Aord, the ordinary part simplifies exactly: since S, no Ao. have (A O and Zobb (Ao)) Poo[Ao(11 plug in
(1 - AAo] = (Ao)), the Fisher information. For the complementary part, that S. neg holds identically. This yields
Iobe Ie (A0) (K 1) q
n
lo(e),
Uc(Ao)lg=ã Lo(@) (Acomp Ag)-
Substituting back into the Newton update gives
Lo( (Acomp - Ao) Io(A)) + (c(g)
A ~ Au +
Closed-form IVW rule. nation of the two empirical estimators:
Rearranging shows that the practical one-step update is a convex combi-
Aww Io(Aard) Aord + Ic( Acomp
Io(Aor)) + Ic(
Variance-information identities and final form. For the two constituent estimators, the vari- ance-information identities
1
1 Var(Acomp) Te
Var To¹
with the plug-in in Fisher information
lo (Aorr) =
Too
Le(g)
71
Aord(1 Aord)
(K - 1)2 222 @(1 3)
Therefore the IWW weight can be written either in information form, w in variance form (using I. = 1/Var(A..)
Io To or, equivalently,
Var (ar(Acom))
w" To + Var(Aord) + Var(Acomp)
Plugging these into the convex combination yields exactly results in 22. w²) Acomp-
Årww = w* Annd + (1 (1==")
In summary, the IVW estimator is one-step Newton approximation to the joint ML: it first-order equivalent and asymptotically efficient, but may differ in finite samples because the weights are fixed by plug-in in information rather than optimized jointly. While the ML theoretically optimal, the IVW rule is simple linear combination that often delivers practically comparable performance and is easier to apply in practice.
E PROOF OF THEOREM 2
Proof. Hoeffding part. For i.i.d. w; E [0, 1], Hoeffding gives
Pr(lá Tej é) < 2 2expl 22 2).
18

---

Published as a conference paper at ICLR 2026
Since Acomp (K - - (K -2) and A (K 1)Eğ (K - 22,, we have
Acomp A (K 1) 1 -l
Empirical Bernstein part. Apply the two-sided empirical Bernstein inequality (Maurer & Pontil 2009 to i EE wa with wy E [0, 1]:
2 9) (vg(4/6)
7 (9g(4/))
Eq
+ (ne 1)
Tee
Multiplying both sides by (K 1) and using Aoomp - (A) = (K - 1) ((l)) yields the second branch in Theorem
F PROOF OF THEOREM 4
Proof. Recall 2j Lú; m3 1,-1,-.. 1,..., No. are i..d. Bernoulli(A) with E[aj] A; and
Wi I(0: + ū.). = 1,.... ne. are i..dd Bernouli(g) with = (A + K 2)/(K - 1) under
Eq. II). The two groups are independent.
Define the rescaled variables
x
w2 2j.
X-() == (1 (I-w)((-)) Wi;
no
and collect them as X X1....y Xn with n1 no Re. Then the centered sum
no
5
> [ Ex(") +
[ EX!
Amix A.
(12)
We verify the conditions of Theorem 2.10 in Boucheron et al. (2003):
(i) Independence) holds by construction.
(i) Boundedness). For all j.
w0 x -E- S
x EX! 5 (I-WXK-1)
my
Set
max
(1-wX(K1))
(13)
mo
ne
(iii) Second moment). Summing variances,
The V Var(X Σ Var(X((o) = A(1-A)
+ (1 (1-w)) ww- 2K) 9(1-0)
(14)
no
(iv) Higher moments). Since X - IEX,| S c, for all integers q 2 3, - -X, S 2 - EX,)²; summing and taking expectations yields >, [[()) S 8 which dominated by the requirement &f 2- in (Boucheron et al.
2003 Thm. 2.10).
Therefore, Theorem 2.10 applies to in Eq.
12 and (using the symmetric lower tail plus union bound) gives Pr |S] /2ut + ct 2e Setting log(2/6) yields Eq. (10).
G PRACTICAL PLUG-IN FORMS OF THE BERNSTEIN BOUND
The quantities A and 9 unknown: replace them by their empirical counterparts. Define Omix 20 + (1 - w) with q = Se/ne- Substituting into Eq. (10 yields the computable bound A - A S / émix log + e log g
6. Avral1 A G. (K-1)*
no
where c is as in Eq. 13 For clarity, equivalent compact statement is
Amiix - A| S
2 2og & [w2 + (1 - -w) + log 3 max
m [1 [K
(15)
19

---

Published as a conference paper at ICLR 2026
The variance component is minimized w*
to practical w is defined as drvw
6.6.
coinciding with the IVW weight in Eq. 77. At this choice, the dominant (variance) term equals
V 2 3
Pate.
to
Defining Amin = futtle using the fully plug-in version gives
|mii - Al - 2 log &h Temin + log &
max Wryw (1 Wi-wwwWKK
(16)
Conceptually, the mgf-based Bernstein bound (Eq. (T0) tightens Hoeffding's inequality by adding both a variance term v and a range-driven linear term c. As n, or ne grows, the right-hand side vanishes: the variance part decays as O(1/no) + O(1/ne), while the linear term scales as O( max (w/no: (1 w)(K 1)/ne))- The plug-in IVW weight 7hvw to converges to w "e: in particular,
+ 0 when n. + oo with n, fixed (the complementary arm domi- nates), and w² - when oc with ne fixed (the ordinary arm dominates).
Formally, Eq. 10 is valid for any fixed choice of the weight w E [0. 1] (e.g., a constant choice or an estimate obtained from independent pilot split). If instea w is chosen adaptively from the same evaluation data (e.g., the plug-in @Ivw), the fixed-weight assumption is violated and the bound is no longer guaranteed. such cases one can (i) revert to the union-bound inequalities of the main text, which hold for arbitrary (possibly data-dependent) w via triangle inequality: (ii) restore the fixed-w condition through sample splitting or cross- fooss-fitting (Chernozhukov
al. 2018 Wager & Athey 2018); or (iii) restrict w to a finite grid G C [0. 1| and apply union bound with ó adjusted by [G| (Boucheron et al. 2013; Shalev-Shwartz & Ben-David 2014).
H BENCHMARK DETAILS
From MMLU-Pro WWagg et al. 2024b), we select only the 10-option questions, yielding 9,795 examples. MedQA-USMLAA (Jin et al. 2021) and GPQA (Rein et al. (024) are both four-option multiple-choice benchmarks; for GPQA we adopt the extended version for broader coverage. For MATH-MC, following the mondidate-generatiin protocol of of et al. (2024), we construct five- option questions from the original MATH dataset (Hendrycks et al.J 2021), discarding problems with fewer than four distractors, which yields 11,751 examples. All duplicates removed to ensure consistency.
I ESTIMATOR CONFIGURATIONS
For completeness, we summarize the exact configurations of all estimators used in Table !
Ord Ordinary-labed estimator based on n. = 300 samples per dataset (n, = 120 for GPQA due to dataset size). Accuracy is computed directly from the proportion of correct ordinary labels.
Comp-no Complemenntrry-lahee estimator with the same number of samples as the ordinary case, i.e. no (n, 120 for GPQA). This setting highlights the loss of efficiency when only complementary labels are available.
Comp-(K-1) Complementary-label estimator with sample size fixed (K 1) times the ordinary case, i.e. n. = (K- - -1)m,- Following the protocol in Appendix C each instance with oracle label y converted into (K- 1) complementary "No" labels {g y {v}}. and the ordinary label is discarded: thus no ordinary labels are present in this setting.
Comp-Var Var Complementary-latee estimator with sample size nee chosen such that its variance matches that of no ordinary labels, following Eq. (4). This represents the idealized regime where complementary labels are sufficiently abundant to offset their lower per-sample informa- tion.
IVW-0.5 A baseline inverse-variant weighted estimator with fixed equal weight 0.5 between ordinary and complementary accuracies. This serves as a naive combination strategy with- out using variance information.
20

---

Published as a conference paper at ICLR 2026
IVW Inverse-variance weighted estimator with closed-form plug-in weights (Eq. 6 combining ordinary and complementary estimates optimally under the variance- intunncciooooaiiion identities.
ML Maximum-likelihowe estimator obtained by solving the joint likelihood equation using both ordinary and complementary samples. Asymptotically equivalent to IVW but involving nonlinear update (Appendix D)
Ord-Eval Oracle reference: accuracy evaluated directly on the entire dataset using ordinary labels. This is shown only for comparison and is excluded from macro-averages
Notes on datasets. Math denotes Math-MC evaluated with GPT-4. UPT-4.1-nans instead of GPT-5-nano, due to reasoning-output constraints. Math (CoT) denotes the same Math-MC dataset evaluated under chain-of-thought prompting (applied only to Math; model=GPT-4.1-11
J IMPLEMENTATION DETAILS ON ADAS AND AFLOW
To ensure fair comparison, we constructed validation and test splits for each benchmark, following the numbers and ratios used in Hu et al. (002)). Specifically, we sampled 120 validation and 800 test examples for Math-MC and Medical Abstract, and 88 validation and 458 test examples for GPQA. For AFlow, we set the number of validation runs to 3 and used the same validation set as ADAS, rather than the separate validation subset reported in Zhang et al. (2025b).
For code and prompts, we largely follow the official implementations of al. (2025) and Zhang et al. (2025b), with two minor modifications for reproducibility in our setting.
The set of experts was adapted to better align with our benchmarks (e.g., replacing domain- specific experts in LLM Debate with ones relevant to GPQA and Medical Abstract). All other prompt components remain unchanged. We uploaded the full text of the adapted prompts and scripts the supplementary material.
ADAS.
AFlow. Since the original AFlow does not directly correspond to the multiple-choice format, we modified the answer extraction and normalization functions, and used the following prompt:
Answer the following multiple chcice question.
(problem[' Question']]
(A) (problem['A']]
(B) (problem('R) [BB']
(C) (problem[ ('c') []
(D) (problem (DD'
At the very end, output the final answer the format:
## X
Where is exactly one of A, B, C, or D.
Do not add anything after this line. The last line of your output must only contain 'HI x', If you write anything after the line 'FEE x', your answer will be marked as incorrect.
K ESTIMATOR VARIANCE ANALYSIS
Figure reports ADAS and AFlow under the two estimators- -aw complementary accuracy (g) and its linear transform earh selected at the iteration that maximizes validation accuracy. The results show clear regime dependence:
GPQA (the hardest benchmark), is more reliable (e.g.. ADAS drops from 47.2 with q to 42.4 with trans, while AFlow remains unchanged at 47.2). On Math-MC, ADAS still prefers q (88.1 vs. 84.2), whereas AFlow shows only slight gain with trans (86.8 vs. 86.5). On Medical Abstract, where complementary accuracy is relatively high, the transform becomes beneficial: ADAS improves from 61.6 to 67.8, and AFlow from 68.5 to 68.9.
21

---

Published as a conference paper at ICLR 2026
GPQA
Math-MC
Medical Abstract
93.:
85.5
00.5
68
68.9
CB.
00
02
20
73
10
70
58-
ADAS (trans)
ADAS (c)
Aflow (trans)
Afllow lql
Estimator sensitivity under weak signals. Detailed comparison of raw complementary accuracy (q) and its linear transform (trans) across GPQA, Math-MC, and Medical Abstract. The figure illustrates how the two estimators behave differently depending on task difficulty, with variance amplification explaining when the transform helps hurts.
Figure 4:
This behavior is explained by variance amplification: according to Eq. (3). the variance of the trans- formed estimator scales by (K 1)² compared to q. When signals are weak (e.g., GPQA). variance amplification exacerbates noise and harms performance.
settings with stronger signals (Math- MC, Medical Abstract), the effect is less detrimental, and the transformed estimator occasionally yields modest gains. A systematic study of this trade-off is left for future work.
DEVIATION FROM THE ORD-EVAL REFERENCE
For completeness, Table reports the absolute deviation (A) between each estimator and the Ord- Eval oracle on each benchmark. Column-wise minima are shown in bold. These values show that mixture estimators (IVW and ML) reduce the gap to the Ord-Eval reference compared with both ordinary-only and complementary-only estimators across most benchmarks, indicating that they provide more robust estimates. Ord and Comp-n, can occasionally produce point estimates that are numerically closer to Ord-Eval (as in MedQA and GPQA). However, as discussed in Section 3.1 their standard errors substantially larger, suggesting poorer reliability and limited reproducibility. This further supports the robustness of the mixture estimators.
Table 3: Absolute deviation (A; in percentage points) from the Ord-Eval oracle. Column-wist minima
shown bold.
Estimator
MMLU-Pro
MedQA-USMLE
GPQA
MATH
MATH(CoT)
Avg A
Ord
0.36
0.23
4.65
3.35
1.00
1.92
Comp-no
0.97
0.01
0.35
4.23
3.45
1.80
Comp- Var
2.30
2.05
4.15
3.11
2.54
2.83
IVW-0.5
0.08
1.05
5.76
0.77
0.33
1.60
IVW
0.00
0.80
5.62
0.66
0.03
1.42
ML
0.03
1.01
5.59
0.54
0.24
1.48
M CASE STUDY: BEHAVIOR OF AGENTS TRAINED WITH PARTIAL FEEDBACK
In this section, visualize and analyze the internal mechanisms of the agentic systems produced by ADAS and AFlow on the Medical Abstracts dataset. This case study illustrates how agentic workflows learned under partial feedback acquire coherent, structured decision processes. Our goal to show that, even with weak supervision, these systems diverge meaningfully and improve from fixed workflows.
22

---

Published as a conference paper at ICLR 2026
Medical Abstract
Principle Extractor
List of Medical Principles
Medical Classifier
Final Answer
(a) Principled Medical Abstract Classifier (ADAS)
Medical Abatract
Review / self-check
Reasoning Output
times
Generate Answer
Candidate
Candidate 2
Candidate
Ensemble
Final Answer
(b) Review-Generale Generaat-EEnsmbble Workflow (Aflow)
Figure Workflows of the best ADAS and AFlow agents learned under partial-feedback training. Com- plementary label rewards lead ADAS to adopt principle-ettration-guuie classifier, while AFlow learns a wwiew-generate ensemble pipeline.
Figure 5) presents the best pessorreffrminggggg agentic workflows discovered by ADAS and AFlow. The ADAS system, which we refer to as the Principled Medical Abstract Classifier, and the AFlow workflow, which name chosen for clarity, both adopt multi-stage decision pipelines rather than directly producing single-shot answer, as done by baseline methods.
In Figure a)), ADAS performs consistent two-stage decomposition. Instead of mapping an ab- stract directly to disease label, the agent first produces a numbered list of key medical principles or mechanistic factors extracted from the text. These principles are then passed which uses them as evidence for the final prediction. Figure 5jb) shows that AFlow adopts a multi- candidate reasoning pipeline. The agent first generates a structured review of the abstract using task-specific prompt. Conditioned on this shared reasoning trace, it then produces several candi- date answers. A final ensemble operator aggregates these candidates and selects the most consistent prediction.
separate classifier,
Such modular, review-based, and multi-candidate workflows do not appear established baselines, which rely on fixed or single-pass reasoning. The emergence of these structures demonstrates that complementary label supervision can shape the agent's 's decision process, not merely its accuracy. Partial feedback encourages the agent construct interpretable intermediate representations and use them as evidence, even without full supervision. This confirms that complementary label training provides behavior shaping signal that yields more structured and interpretable workflows.
23

---

Published as a conference paper at ICLR 2026
N LARGE LANGUAGE MODELS USAGE STATEMENT
LLMs assisted us formulating our ideas, working on extensions and analysis, and polishing the writing. All of the LLM outputs were subsequently verified, revised and modified by the authors. Any errors or inappropriate statements remain the responsibility of the authors.
24


Reviews:
[{"review_id": "rP6fLCSuhv", "reviewer": "Reviewer_t5Fc", "summary": "The paper studies how do we evaluate AI systems when direct verification is difficult (as the abilities of an AI may exceed that of any single human). The paper relies on the notion that ruling out an incorrect answer is easier than verifying that the answer is correct. It prposes a partitioned supervision protocol, where the human experts indicate when a particular answer is definitely incorrect rather than providing a correct answer (or actual verification). Using these complementary labels, with the additional assumption that these complementary labels are drawn uniformly at random from, the incorrect options, the paper proves that it is possible to construct an unbiased estimator for model top-1 accuracy using these labels, and variance estimators are also studied. Experimental results are convincing (but some questions remain).", "questions": "1. I had bit difficulty understanding the paper due to its notation. I believe, using $X$ and $Y$ for random variables and small letters $x, y$ for their realisations could help.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I think the theoretical framing of complementary labels as sufficient to estimate model performance is clean and clearly motivated. I like that the paper focuses on the oversight bottleneck, which is timely.\n2. The derivation of the estimator and the variance analysis are correct to me. The decomposition and reasoning about how the uniform query mechanism leads to unbiased estimation is neatly done. I also found the combination with ordinary labels to reduce variance to be intuitive.\n3. The experimental evaluation is thorough within the context of multiple-choice settings. In that sense, the paper is rigorous in validating the correctness of the statistical estimator.", "weaknesses": "1. While I follow the theoretical contribution, I am less convinced about the real-world applicability for the kind of motivation the paper has. The experiments are exclusively on closed-form multiple-choice tasks. However, the motivating context in the paper is open-ended and superhuman domains. In that sense, there is a bit of a mismatch: the paper does not really demonstrate that the method scales to the intended setting.\n2. I also think the paper assumes too much familiarity with the labeling protocol. The uniform selection of an index to query is crucial to guaranteeing unbiased complementary labels, but this assumption is not deeply discussed. In realistic scenarios, domain experts may not always be queried in such a uniform way.\n3. The paper claims agentic training benefits using this estimator as a reward signal, but the empirical improvements there are small and relatively difficult to interpret. It would help to clarify what practical training improvements this enables and whether there are examples where the method meaningfully changes agent behavior.\n\nOverall, I think the paper makes a meaningful and conceptually clean theoretical contribution. The estimator is sound and the controlled experiments support the theory. However, I think the paper could improve on motivating and validating how this method translates to settings beyond multiple-choice evaluation. I would benefit from stronger discussion on how the uniform querying assumption interacts with real-world annotation workflows.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HoXKUOi2VU", "reviewer": "Reviewer_2oJj", "summary": "This paper proposes a scalable oversight framework to evaluate and train advanced AI systems using \"complementary labels,\" which are weak signals from experts identifying incorrect options rather than the correct ground truth. The authors derive estimators to measure accuracy from these weak signals and show how to combine them with scarce ordinary labels. Empirically, the results demonstrate this method can successfully evaluate LLMs without ground truth and can even be used to train an agentic AI system to perform better.", "questions": "1) Can the authors comment about the limitations of their approach?\n2) Assuming full ground truth is scarce, what is an estimation of the number of weak supervision required to get the same performances?\n3) What the authors think is the reach of the different learning tasks that a LLMs can learn by using weak supervisions only, or weak supervisions + a few full ground truth supervisions?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- This work studies a very relevant topic that is fundamental for the enhancement of current AI systems, especially for their training and evaluation w.r.t. humans.\n- The paper provides mathematical details, motivation and theoretical analysis of its proposed framework. In this way, the contributions are concretely based on mathematical principles and characterization.\n- The result demonstrating that weak human feedback can provide useful learning signal is very promising.", "weaknesses": "- Top-1 accuracy can be scarce or too limited to evaluate the performance of a foundation model.\n- The presentation is a bit difficult to follows as it is very technical. However, I don't consider this a proper weakness. \n- If I understand correctly, the number of complementary labels needed for a satisfactory training can be very large.\n- I didn't understand if it is a typo, but several paragraph titles are coloured, which is quite uncommon.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L97vPsntYH", "reviewer": "Reviewer_KfQX", "summary": "This work aims to improve LLM evaluations in settings where we have limited access to expert annotations by using complementary labels (derived by asking experts to evaluate a model output as either confidently in or out of their expert class). These complementary labels are weaker than ordinary labels as they provide less information and have high variance as compared to ordinary labels, but this work formalizes the impact that these labels can have in evaluation by introducing 2 estimators for combining ordinary and complementary labels (IVW and ML), deriving associated Hoeffding/Bernstein bounds for the estimators, extensive empirical evaluation over traditional and more real-world datasets, and agentic training using the estimator as a fitness function. This work is answering a timely question of how do we leverage humans in LLM evaluations, especially in cases where human annotation is scarce, which is becoming more likely as LLM capabilities increase.", "questions": "1. Please provide extended explanations for why MathMC results in Figure 3 are not as strong as on the other benchmarks. What is it about this benchmark that makes the complementary labels less effective?\n\n2. In Table 1, it is not clear what the estimator methods are. I see that the methods are outlined in the appendix, but there needs to be some discussion/legend present in the main text as well. Further, the presentation of the table generally can be improved.  Can you highlight how much mixture helps beyond just regular usage of fewer ordinary labels? You also could just report the delta directly between the oracle instance and the different methods.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work regarding how we can leverage human data in nontraditional ways via complementary labels is original and timely to the current dearth of adequate evaluation paradigms for LLMs. The paper leverages high-quality evaluation paradigms to support their claims, approaching the problem both empirically (across multiple benchmarks, tasks, and domains) and theoretically (unbiased estimator derivation, bounding the variance of the estimators, quantifying how many complementary labels equate to a singular ordinary label) and deriving strong and consistent results. The work is generally clear, although I discuss within the weakness/questions the parts that would benefit from additional detailing and motivation. The work also has potential for significance in the field of scalable oversight and evaluations, as more instances arise where obtaining explicit human feedback from a desired expert may be infeasible but alternative, weaker forms of feedback may be more scalable and readily available.", "weaknesses": "One of my biggest qualms with the work is the general motivation framework. “...future models will tackle problems whose solutions are too technical or too crossdisciplinary for any single human to verify comprehensively. When we cannot produce ground truth or prepare automated verifiers, how should we evaluate and train such systems?” In theory there is value to this question and motivation, but how do we ensure that “not cardiology” is an informative label? Specifically, in this toy example, why is cardiology feedback available but the true specialist is not? Additional discussion of why we may expect this paradigm in medicine and in what instances (Is this a diagnosis problem? If it is, would we ever truly have a multiple choice set where we cannot define the true label but we know the complementary label? Then how did we get the set of choices?) . As such, if we are entering a paradigm beyond human expert understanding, is multiple choice the best setting in which to be evaluating this set-up? And if not, how would this system generalize to more open-ended evaluation tasks? It would be helpful to discuss these limitations in the conclusion as potential future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g6hlVoaUSp", "reviewer": "Reviewer_oJ9E", "summary": "This paper addresses the growing challenge of evaluating and supervising AI systems that exceed human expert capabilities. The authors propose Partitioned Human Supervision, a scalable oversight framework in which multiple experts each provide complementary labels, indicating which candidate answers are incorrect, rather than full ground-truth annotations. The paper derives unbiased estimators for model accuracy using only these partial signals, and further introduces mixture estimators that combine complementary and ordinary labels. Experiments in financial and medical domains demonstrate that such supervision can reliably estimate model performance and even serve as a weak training signal when full supervision is infeasible.", "questions": "- The framework can involve experts who reliably provide negative feedback. How would it handle cases where experts are uncertain or only partially knowledgeable? Does the estimator remain valid under probabilistic or noisy complementary signals?\n- Does partitioned supervision assume disjoint expert domains? If experts’ competencies overlap, how are conflicting or redundant complementary labels managed, and what effect might this have on estimator consistency?\n- The method is tested on structured tasks. Could it extend to open-ended or generative settings by redefining complementary feedback (e.g., flagging logically inconsistent outputs)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper tackles a timely and important problem: how to scale human oversight when AI systems operate beyond individual expertise. The framing of ``partitioned’’ expertise is both realistic and conceptually strong.\n- The derivation of unbiased estimators for accuracy under complementary labeling is clear and mathematically grounded. The analysis of variance and the proposed IVW and MLE-based estimators are rigorous and interpretable.\n- The approach can significantly reduce supervision cost by replacing full labeling with weaker but cheaper expert judgments. This has meaningful implications for high-expertise domains like medicine or law.\n- The authors evaluate their method on domain-specific benchmarks, showing that it remains reliable even when each expert only rules out incorrect answers. The results support the method’s robustness and real-world applicability.", "weaknesses": "- The framework suits multi-choice or structured decision problems but may not extend naturally to open-ended or generative tasks where correctness is ill-defined.\n- The method assumes experts can accurately identify incorrect options. In practice, experts may err or have partial uncertainty, which could bias the estimators. The paper could better analyze the effect of noisy or correlated human feedback.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)