ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge
Authors: Zhilin Wang
Jaehun Jung
Ximing Lu
Shizhe Diao
Ellie Evans
Jiaqi Zeng
Pavlo Molchanov
Yejin Choi
Jan Kautz
Yi Dong
Venue: ICLR 2026 Poster (2026)
Keywords: expert-annotated
professional knowledge
llm judge
rubric evaluation
Primary area: datasets and benchmarks
TL;DR: We curate a PhD/MBA-level human-annotated rubrics dataset across Physics, Chemistry, Finance and Consulting with >7000 criterion-response pairs and introduce methods to mitigate bias and high cost of evaluation to make it fair and accessible to all.

Abstract: Evaluating progress in large language models (LLMs) is often constrained by the challenge of verifying responses, limiting assessments to tasks like mathematics, programming, and short-form question-answering. However, many real-world applications require evaluating LLMs in processing professional documents, synthesizing information, and generating comprehensive reports in response to user queries. We introduce ProfBench: a set of over 7000 response-criterion pairs as evaluated by human-experts with professional knowledge across Physics PhD, Chemistry PhD, Finance MBA and Consulting MBA.  We build robust and affordable LLM-Judges to evaluate ProfBench rubrics, by mitigating self-enhancement bias and reducing the cost of evaluation by 2-3 orders of magnitude, to make it fair and accessible to the broader community. Our findings reveal that ProfBench poses significant challenges even for state-of-the-art LLMs, with top-performing models like GPT-5-high achieving only 65.9\% overall performance. Furthermore, we identify notable performance disparities between proprietary and open-weight models and provide insights into the role that extended thinking plays in addressing complex, professional-domain tasks.

Published as a conference paper at ICLR 2026
PROFBENCH: MULTI-DOMAIN RUBRICS REQUIRING PROFESSIONAL KNOWLEDGE TO ANSWER AND JUDGE
Zhilin Wang, Jachun Jung, Ximing Lu, Shizhe Diao,
Ellie Evans, Jiaqi Zeng, Pavlo Molchanov, Yejin Choi, Jan Kautz, Yi Dong NVIDIA
[zhilinw, yidong) env @nv aiiidiiii...mo com
ABSTRACT
Evaluating progress in large language models (LLMs) is often constrained by the challenge of verifying responses, limiting assessments to tasks like mathematics, programming, and short-form question-answering However, many real-world applications require evaluating LLMs in processing professional documents, syn- thesizing information, and generating comprchensive reports in response to user queries. We introduce ProfBench: set of over 7000 response-criterio pairs evaluated by human-experts with professional knowledge across Physics PhD, Chemistry PhD, Finance MBA and Consulting MBA. We build robust and afford- able LLM-Judges to evaluate ProfBench rubrics, by mitigating self- self-enhancement bias and reducing the cost of evaluation by 2-3 orders of magnitude, to make it fair and accessible to the broader community. Our findings reveal that ProfBench poses significant challenges even for state-of-the-zn LLMs, with top-performing models like GPT-5-high achieving only 65.9% overall performance. Furthermore, we identify notable performance disparities between proprietary and open- weight models and provide insights into the role that extended thinking plays in addressing complex, professional-donaaeee tasks.
Table 1: Comparison of ProfBench with select rubric-based benchmarks. Rationales for the classifi-
cation into good (V) and poor (X) on all aspects are discussed in the Introduction.
Dataset
Diverse Domains
Professional Knowledge Human Written Rubrics Fair Grading
PaperBench (Starace 2025)
HnalthBench (Arora et al., 2025)
DeepResearch--e Beach RACE al., 2025)
ProfBench (Ours)
INTRODUCTION
For many problems, verifying the correctness of a solution be much simpler than coming up with the solution. Take for instance, Sudoku verifying a solution is trivial for anyone with a basic understanding of the rules (no repeated number in a row, column or small square) but the hardest initial positions can take experts a long time to solve. This insight inspired many to use Reinforcement Learning with Verified Rewards (RLVR; Lambert al. 2025; DeepSeck-Al al. 2025) by verifying LLM responses to tasks of such nature. However, there are only a limited set of tasks for which verified rewards can be easily constructed. One common domain is competition math, which have unique correct answers such as AIME 25 (White et al., 2025). Other common uses lay in competitive programming problems where generated code has to pass existing unit tests such as LiveCodeBench (Jain et al., 2024) writing with precise instruction following (e.g., checking if particular word appears 5 times using a python script) in IFBench (Pyatkin et al., 2025). In other domains such as scientific question answers, RLVR is restricted to settings with unique correct answer (Multiple- Choice Question or Short Answer Span), seen in popular evaluations like MMLU-Pro ( (Wang et al., 2024), GPQA (Rein al., 2023) and HLE (Phan et al., 2025).

---

Published as a conference paper at ICLR 2026
ProfBench Finunce MIRA Example
You are helpeng me assess the potemtal for a new business unit within a major investment bank. This unit is entirely deducated to innovative finance for healthcare, as well as social imput and environmenial challenges the plobal level. One example we are studying to usess this oppoctunity is how the Global Allianee Vaccination and Immunization (GAVI) been able to raise mosey capital markets through the Insernational Finance Facility for Immunizazion (IFFl)).
I would like you help answer the following questions:
HHo was IFFl able to mase money on cogical markess for vaccunation and immunization compaigns?
How did UFFmm apply secuniization?
'hhat was facters that made possible IFFIm to mise money on cogital markets?
-What risks and challanges does IF have to address and overcoms??
How would you issess whether 1FI his been effective and successful at missigg funding for GAVI, and overall, has been a success for their operations
and health goals?
funding other areas global bealth, other mayer social environmental challenges? yes, idensify short pipeline of 5 organizationstopicc that could take a samilar appronch and use innovative finance to raise funding to advance their goals. Provide a cobust contest, starting with defining what Gavi and IFFls are, how they work, and how they ure both related. Discuss, in detail, the tochniccal aspects 1+1lm works, deseriee what makes possible and effective.
Can IFFm viewed blucgrint for inverting
technical discussion should descrihe how
and provide
detailed everview how raisel money until now (hhi should include simmary of post teanaeees shewing what warkets raised and what investors subscribed). Also, discuss what elemests made IFFM possible, as well what risks and challenges IFHI faces. This should help you form a robust and documented view what makes IFFlm success (or not), and how it could act a Reganding
(ar not for other initiatives.
am looking for the style of detailed investment
discnosion. You can tables bullers, but the memo should moutly sext, through your reasoning. don't wast the outpot to be moatly
and you should
bullets.
Extraction Kubric Sample: States that breach of IFII''s liquidit policy could negurively impact IH+Mm's rating peofile.
Rocioning Ruhric Sample: States that vaccines are one of the most suoceseful and cost-sffecrive healtt investments the woeld.
Style Rulvic Sample (amatker tack): Present Bindings clearly allow for effective
Deep Research-Read Finance Business Example
What the invesmment philosophies Duan Yorgping, Wancs Ruffier, and Charlic Munger?
Example from ProfBench (Finance MBA) is substantially more challenging and detailed compared with Deep Research- Bench Finance & Business (Du al., 2025). More examples in $A.
Figure I:
The ease of verification should not limit the type of tasks we can train models to successfully complete. While some of these tests might predict how well PhD candidates (or other early-stage domain-experts, understand their field, people do not graduate from PhD programs (or become true-cxperts) simply based on how well they can pass these "cxam-style" questions. Instead, they do so based on the merits of their original contributions in ways that are valuable to others- for instance, coming up with a technique to synthesize cancer treatment medication. To expand the set of tasks which can be evaluated with RLVR, there is a pressing need to cover challenging tasks with real-world value into format can be verified. Grading problems using rubrics represent a promising path forward (Gunjal et al., 2025), with the central idea being to decompose a complex problem into several criteria that a good response needs to fulfill. A trivial example is that when asking an LLM for food recommendation for a Tues dinner with a vegan friend in New York City, there can be many good answers but they must fulfill a. open on Tues night; b. have vegan options; c. in New York City.
Recent works (Starace et al., 2025; Arora et al., 2025) show that humans are often tasked with coming up with these criteria while LLM-judges are used to determine if a response fulfills the criteria (given both cost and efficacy considerations). Specifically, Starace et al. (2025) demonstrate that this can be done for specialized domains requiring professional knowledge health in evaluating aatient-physiciin
conversations while Arora et al. (2025) shows its usefulness for machine learning in reproducing ICML papers. However, there currently is no robust publicly available benchmark that makes use of rubric fulfillment across diverse professional domains beyond health and machine learning. DeepResearch- Bench RACE (Du et al., 2025) claims to curate PhD-level tasks across multiple domains, but many examples such as What are the investment philosophies of Duan Yongping. Warren Buffett, and Charlie Munger? or How did Netflix manage to successfully adapt One Hundred Years of Solitude, a notoriously difficult book to bring to the screen? can be answered by an educated generalist (i.e., college graduate or equivalent experience) with few straightforward internet searches. It also falls short due to synthetically generated criteria that were not verified by expert humans- meaning that it's unclear if such criteria are robust anchors for scoring. For instance, there is a pervasive bias towards Gemini-2.55Pro since the criteria and the reference "high-quality" answer are both generated by this model. This results in Gemini-2.5-Pa being rated as by far the best snfforming-rraching more than 48.5 out of a maximum of 50 on each of the axes, meaning 997.
To support the community with robust rubrics for real-world, professional-level tasks, we propose Pro/Bench-a rubric-guided benchmark for professional-grace LLMs, curated by expert human pro- fessionals. ProfBench covers tasks across four domains Chemistry, Physics, Consulting. Finance- annotated by human experts with either PhD degree (Chemistry and Physics), MBA (Consulting and Finance) or equivalent experience and are incumbent professionals in their respective domains. We summarize our main contributions below, which can also serve inspire future directions in SH:
2

---

Published as a conference paper at ICLR 2026
Top Top-level Categories
Reasoning Sub Calegories
Style Sub-Categories
17.4%
Percmaage (96)
g 3
Style
3.0%
62.9%
Reasoning
60
19.2%
Extraction (Recall)
34.1%
Logioa
Deserattation Woaseniigg
Clarts Sectaaliiy
Conismen
Formatiin
Sone
Figure 2:
Distribution of rubrics by category. Reasoning dominates (62.9%), with most on logical validity and correctness. Extraction accounts for 34.1 1 1%, emphasizing accurate retrieval of information (w/o meaningful subcategories). Style is minor (3.0%), with focus on formatting and clarity.
1. We introduce ProfBench, the first benchmark with expert- eppertccrcaed rubrics across multiple pro- fessional domains including over 7000 response-criterion pairs involving Scientific Research and Business tasks. This benchmark is challenging requiring PhD/MBA-level knowledge with the strongest GPT-5-High model reaching only 65.9% performance.
2. We measure the performance of over 40 models in terms of their ability to generate high-quality responses to these tasks as well as to evaluate whether such responses fulfill various rubrics that experts determine good response require LLM-Judges. We analyze various trends across open/closed-souuee source models, reasoning/instruct models and model size.
3. We propose methods reduce the bias of LLM-Judges in favoring responses from specific providers as well as the cost of running the benchmark, in order to improve its accessibility. Our benchmark can be run with no more than 1% bias across 3 models from different providers and $12 (using 03 model), which is 2-3 orders of magnitude cheaper than existing rubric-based evaluation.
2 PROFBENCH OVERVIEW
ProfBench contains 7,347 human-written response-criterion pairs across 80 distinct tasks equally divided among 4 domains (Chemistry PhD, Physics PhD, Finance MBA and Consulting MBA). This is comparable to the 8,316 response-criterion pairs used in PaperBench (Starace et al., 2025) across 20 tasks as well as HealthBench (Arora et al., 2025) with 8,053 response-criterion pairs across (Du et al., 2025) has 100 tasks (each with one reference
hundreds of tasks. DeepResearch-Bende response, and no human-written rubric) across 22 domains, equally split between Chinese and English, meaning there is only an average of 2.3 English tasks per domain. Rubric-based evaluations requiring annotation by human-experts tend to be small as each task is time-consuming and recruiting for such professionals (e.g., Physics PhD holders with industry experience) is difficult. We show example in Fig. 1, a breakdown of rubrics by category in Fig. 2, and further descriptive statistics in $D.
3 DATA COLLECTION
Annotator Recruitment Annotators with PhD, MBA or equivalent work experience are recruited and managed by our vendor. Prior to their inclusion into the project, we check that they pass test on their domain expertise and their understanding of the annotation task (e.g. performing under ambiguity and attention to detail). Each annotator is expected to work through an entire task including Prompt Ideation, Rubric Creation and Response Annotation as annotators are asked to create tasks that capture their personal expertise. Annotators are supported by or more reviewers (with substantial task creation expertise in their domain) to review (several back-and-forth rounds possible if needed) and approve each stage. before moving to the next. Each annotator spends around 10 to 20 hours on each task. To ensure diversity of prompts, each annotator is allowed to contribute no more than tasks. We disallow the use of L.LMs at any stage of the annotation process. Across the 80 tasks, 38 annotators from 8 countries were involved. 44.7 % of annotators hold a PhD, 18.4% 8..4% hold MBA and others hold related degrees such as Bachelor of Science in Finance, Bachelor of Commerce or Bachelor of Business Administration along with work experience. Annotators have an average of 5.24 years of experience following graduation of their highest degree. Annotators are paid well-above minimum-wer wage following local standards, with hourly pay often exceeding full-time employment hourly pay in these professional fields to attract the most qualified annotators. Further details on annotator recruitment in SC.
3

---

Published as a conference paper at ICLR 2026
Prompt Curation Our goal was to curate tasks that would be difficult even for the frontier LLMs at time of our collection in July 2025 (e.g. OpenAl o3; xAI Grok4; DeepSeek R1-0528). More specifically, we asked for tasks that annotators might ask their (junior) colleagues help on, resulting in multi-page report. Therefore, these questions often contain multiple related sub-questions as seen in the Finance MBA example in Fig 1. All prompts (and expected generations) must be written in English and only involve the text modality. We allow the annotator to use Internet search to identify documents that would support finding evidence for answers but disallow the use of proprietary documents (i.e. not findable on public internet). To ensure that prompts are created purposefully, we ask annotators to provide brief rationale for how they come up with each prompt.
Rubric Creation For each task, we ask annotators to create around 15 to 60 criteria used to score the response of models. Each criteria should be independently usable to grade responses and collectively capture all aspects of the response quality. Each criterion contains a description of the criterion as well as a justification of the criterion (to encourage thoughtful creation). In addition, annotators are asked to assign an importance and one or more criteria types. The reviewer then provides overall feedback on the set of criteria as well as recommendation keep or improve each criteria, 41.4% are marked as needing improvement at various stages, indicating the high quality standard that we set for when reviewing this data.
criterion (details in SI). Among annotator-wrinn
Response Annotation Our vendor generates responses with 3 models: OpenAl o3, Grok4 and DeepSeek R1-0528. These models represent the models that we believe to perform best on the tasks we collected at the start of annotations (July 2025), including both proprietary and open-weight models. The annotator scores each of the three responses on each criteria with either Yes or No, alongside brief justification. Further details in $E
4 BENCHMARKING MODELS AS LLM-JUDGES
Task Formulation For each criterion, we provide the LLM-Judge the response, the criterion and ask whether it fulfills the criterion in a binary fashion. In this sense. the task is formulated similar to a two-class Natural Language Inference/Recognizing Textual Entailment task (Bowman al., 2015) with two possible answers (Entailment or Contradiction). Following PaperBench (Starace et al., 2025), we do not provide the original task prompt to the L.LM-jugee as the criteria are designed be used independently of the prompt, and further providing the prompt might confuse the LLM-Judge. Prompt templates are in SB.
4.1 EVALUATION
Agreement with Human Annotations To evaluate L.M-Judges, we Macro-F1 based on the ground-truth human- binary) as used by HealthBench (Arora et al., 2025) and PaperBench (Starace et al., 2025).
and the model preeeepred criterion-fulillmm (both
Bias However, another important aspect to consider is fairness to various model responses since LLMs are known have self-enhancement bias (Zheng et al., 2023), meaning that they allocate higher scores to their own responses or responses from the same model family. We formulate a bias- index by first calculating the bias for each model using N Σ emodel duuman where N is total number of criteria and 6 the fulfilmmtt predicted by model or labeled by human. Then, calculate the min and max bias across three models (o3, Grok4 and R1-0528) before calculating the bias-index by taking the difference between max-bias minus the min-bias. A low bias-index means that LLM-Judge does not overly-reward/penalizz any model's 's responses, relative to human annotated ground- ground-truths and other models. Overall performance is Macro-F1 minus Bias-Index.
We calculate the upper-bound cost of running the full LLM-Judge evaluation, using the number of input and output tokens multiplied by their public cost (OpenRouter, 2025), excluding discounts due to caching and other methods. In practice, caching alone will reduce the cost low as one-tenth as stated cost for judges whose costs are dominated by input tokens. Cost is useful when comparing two judges with similar performance, the cheaper judge can be accessible for more.
Cost
4

---

Published as a conference paper at ICLR 2026
Inference Setup Following HealthBench (Arora et al., 2025). use GPT-4.1 as judge of response-criterion fulfillment in early experiments to identify an optimal prompt template used across all models subsequently among possible templates (See ŞB). Our exploration supports generating only I completion token (Yes or No) for non-reasoning L.MMs and up to 32,000 tokens for reasoning L.MSs (and expect the post-thinking-tran non-reasoning LLMs will be substantially cheaper and faster than reasoning LLMs (approximately two to three orders of magnitude in practice). Following best practices (Yang et al., 2025; DeepSeek- AI et al., 2025), we set temperature to 0.6 / top-p 0.95 for reasoning LLMs (when it can be set) and temperature 0/ top-p 0 (i.e. greedy decoding) for non-reasoning LLMs. Our experiments with GPT-4.1 Judge also suggests that both Macro-Fl and Bias-Index are highly consistent, differing no more than 0.2% across three independent runs therefore we only run with each judge once to save cost. Note that for judges, we do not require web search or file upload as human annotators were explicitly asked to create criteria that can be independently used to grade responses. For experimental purpose, we only use half of the dataset remaining half as the private dataset in order to mitigate test contamination (Han et al., 2025).
response be either Yes or No). This formulation means that
plan for this to be the public dataset, while we keep the
Human Validation We carry out a validation experiment on a portion of the dataset (1 127 response- criterion pairs). Specifically, we ask two other annotators with the same expertise (e.g. Chemistry PhD) to re-annotate the response-criterion fulfillment. Based such annotations, we find inter- annotator agreement to be Fleiss' x = 0.912, suggesting excellent agreement among annotators.
4.2
RESULTS
Are Closed Source Models better than Open Weight ones? Across the board, the best LLM- Judge models in Tab. 2 are proprietary models. GPT-4.1 is the best model that outputs only 1 token per task at 75.4%. while Gemini-2.5-Prr takes the crown for Reasoning LLM Judges at 78.2% (with its Flash sibling only 0.1% behind). However, open- weights are often not far behind. In the non-thinking category, Kimi-K2-071 11 is only 0.2% behind while OPT-OSS--20B--1 within the reasoning category. Such a small gap is especially impressive given the cost difference. Kimi-K2 costs mere 7.16% of GPT-4.1 while GPT-OSS-120BB ($0.50 vs $41.46). This means that using open-weight models can allow experiments with evaluating model responses to be much cheaper and hence accessible to many more. Such accessibility allows more researchers from a wider background to iterate with methods to train subric-followe behavior.
is 1.5% behind
is only 1 1.21 % of Gemini-2.5
Does Model Size Matter? Within the same model family, larger models generally perform better than smaller models- - but the improvement plateaus after a certain size. For instance, the improvement between GPT-4.1-nano and GPT-4.1 was large at +20.8% while the further improvement to GPT-4.1 was much smaller at +0.5%. A similar trend can be observed with the Llama-3. 1/3.2 series going from 1B to 405B. However, the jump from 1.1a-33117093 to 3.3-70B (+3.4%) much larger than the jump from 1.ma-33.1275 of improvements in post-training recipe beyond model size alone. This is also supported by the minuscule 0. 2.11 1% model performance gains between Remini-2.5-Flashh- and Gemini-2.5 (non-thinking) as well as Gemini-2.5-Flash (thinking) and Gemini-2.5-PPo (thinking).
to llama-3.1 --05BB (+0.9%), indicating the importance
To Think or Not to Think? When comparing the same model (e.g. 1ominn-2.55-Flssee Claude- Sonnet-4, DeepSeek- V3.1) that have thinking enabled disabled, enabling thinking typically leads to some improvements, although the magnitude of the gain varies: 0.7% for Claude-Sonnet-4 to 4.8% for Gemini-2.55FFlash. This is further supported by the OpenAl GPT-5 series, for which going from minimal reasoning to low reasoning substantially increases performance by 4.4% to 18.6%, with a larger gain for the smallest GPT-5-nano. In contrast, the gain between low, medium and high reasoning effort is not consistent, with 04-mini, GPT-5-nano and gpt-oss-20b/ 1 20b showing the best performance at low reasoning effort. A possible explanation is that increasing reasoning effort generally increases alignment with human annotations but also increases bias to specific models as observed consistently across all models (except for o3). This could be a result of greater self- enhancement bias with more thinking, as the bias towards OpenAl o3 responses generally increases.
Furthermore, thinking with greater effort seems to lead to largest improvement in Physics, Chemistry and criteria related to Style (e.g. Rounds all stock prices and equity values to two decimal places). These criteria could be harder to judge without extending thinking, since they often entail multi-step reasoning to ensure that the response fulfills the criteria, instead of simple answer-matching.
5

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation of LuMgJudges. Higher is better for Macro-F1 and Overall while closer to 0 is
better for Bias Index. Sensitivity and Qualitative analyses of LLM-Judges are in SJ.
Agreement with Human Amaotadias (1)
Bin Index
Ovenal MPI -BI
Tokem Out
Modkl
Plys Chee
Fir Crench Surret
Resos
All
o3
R1-05
Gnadd" Mac-Min
Instrat Julge Oviget tolee
501
09.3
71.0
50.0
19
76.4
65.8
76.3
1.5
4.5
50
03
15.4
1619
11:31
07.3
80.6
76-T
09.8
76.8
-03
-03
161.9
Teeglattamnn 5-3leb
87.9
71.
79.6
70.4
67.7
26-5
-43
-3.1
13.4
1.57
81.6
683
68.2
80.5
17.9
75.0
71.0
76-4
-13
20
06
3.1
73.3
1770
0.67
06.9
72-8
T5.1
29.7
-6.1
3000
QecerQres3
70.6
77.3
308
2007
76.5
4.7
MoonsaeAAAKan Kans lestac 1905
54.5
69.0
07.55
$1.9
19.3
75.5
05.9
27.0
1.5
6.3
5.2
3.3
74.7
1023
0.51
83.
09.5
82.3
50.3
76.
05.4
72.6
1.1
6.1
4.7
15.2
1635
0.81
Deppack DeDyyee
V3.034
68.0
76.6
1.5
asstaallanss Lammrmn can--tber Mbve:
66.
59.3
57.
16.7
61.3
58.5
99-1
58.5
-26.
-MLD
55.5
1533
asstiallrm-- Semmeno opar-hib-cl
72.
63.1
70.2
72.1
1117
64.1
22.5
-157
-122
-139
nesallara---mmn 4-monencs- 12-1220 monect
662
23.4
16.4
1014
619
22.4
443
-165
-83
nezvllara--a dacces- Tib tb-eee-ndrnct sntrect
01.
694
71.5
15.6
18.1
895
628
118
-88
-622
-66
6399
1385
0164
masullara 14bb--pannt
69.1
6:6
81.1
125
6-2
710
11.
51
98
nesallara- Li-tt
66-7
72.6
16.0
130
642
15
nesullann- A. mettitt
262
09.3
70.2
71.0
16.5
71.5
61.
732
460
6.1
-13
1001
63.1
1025
0.00
mssullarn- ---baanttntt
63.8
66.1
86.6
34.
83
59.3
Reasoning 1245 judge
M0.9
76.7
7911
16.
1.0
-1.5
16.0
1618
10.34
09.2
67.9
(0.0
T6.1
573
77.
.0.
-1.
1.3
76.7
1619
757
17.06
1
(A
76.6
74.8
0.3
-1.
11.18
77.0
84.5
69.2
70.4
82.8
75.
24.1
27.2
15.3
1619
4.88
76.7
BL.
760
81.7
64.0
69.1
15.9
725
588
738
4..
-111
1615
1.46
67.6
264
93.5
67.
68.6
17.5
T6.9
73.5
330
15.4
2.4
0.5
1.9
1.5
13.6
1619
141
68.8
35.3
60.9
63.0
62.1
343
632
-187
-19.6
26.9
833
55.0
1615
0.2
OperAinn3
68.2
81.1
76.1
75.3
71.9
2.0
764
21.04
99.3
69.1
68.9
$1.0
76.4
76.9
262
3.0
0.5
33
76.0
1613
207
17.05
88.9
69.3
70.3
81.9
79.7
76.
76.7
76.7
3.8
2.6
76.4
1615
14.01
68.9
70.5
$1.5
76.8
76.5
76.
4.5
10:93
58.
69.
70.
81.6
T8.9
76.
2611
28.
1.0
2.
2.6
15.5
1613
224
4.0
88.6
70.1
70
81.0
78.8
76.
241
26.5
3.4
76.5
161
7.0
kAltyok-4
70
777
$2.0
75.
55.
4.5
177
75.7
66.1
60
77.6
77.
12.1
66.
260
-11.
-107
5.1
70:0
1985
8in
63.64
GoogleGGena 5-2m
87
70.1
71.9
$2.6
81.3
77.
26.3
26.
31
78.2
11.46
(Taiiing)
GeealeUermm 25-F1ab-Lii Lite (Thicking)
83.7
67.0
72.2
$1.9
75.9
79.1
77.5
-1.3
-2.6
34.3
1739
1S70
2.05
Open eeight
Tigh
D9.1
65.7
68
M1T
77.6
76.5
77.7
77.9
1/3
-0.3
0.9
71-4
1079
161
11:6
87.2
05.1
M119
76.1
26.7
79.8
1.6
1683
Sig
00.4
65.9
69.
MIK
T8.9
76.
303
76.8
1.6
-14
0.5
3.0
75.4
1683
639
0188
67.4
205
79.9
76.0
29.
21.2
96
0.85
phyoralttrraanilltt maystt for others
68.9
22.2
79.7
76.9
905
267
-85
-1.
Desptidd-AAt Aithoopee V.5 (Thinknng)
86.5
09.3
70.
M1.3
18.9
75.6
72.1
77.5
322
3.1
26
B0
16.6
1387
887
2.9
Despliad Altpeeee
79.6
69:1
71.6
1199
64.1
722
-1.6
Quearieent- 2oatt-----ttaa A.28- (binkingshoor
679
09.0
80.4
343
213
-30
1.84
Best Performing Judge Overall In selecting the optimal judge for evaluating report-generation models, we consider both the overall score (which consider alignment with humans and freedom from bias) and the cost of running the LLM Judge, which influences the accessibility of the benchmark. Therefore, we decided to use GPT-OSS- 120B as the judge since it does well on both. We also noticed that the high reasoning effort version does better on Physics, Chemistry and Style-related criteria, while the low reasoning effort version does better on others. Therefore, we decided to alter the reasoning effort based on the domain/criterion type encountered, inspired by Jung et al. (2025) to balance quality and cost. The resulting judge performs as well as the best proprietary model 2.emini-2.5-PPrt at 78.2% Overall while costing only 1.68% of Gemini-2.5-PPr 2ro's 's cost. This means ProfBench-Judge only costs $0.70 vs. $1320 for PaperBench JudgeEval (Starace et al., 2025).
5 BENCHMARKING MODELS AS EPPORT-GENERATORS
Task Formulation We formalize the task given a prompt alongside grounding documents, generate a response that addresses the task prompt. The use of grounding documents is inspired by how human professionals commonly work when tackling real-world tasks, which is formulation

---

Published as a conference paper at ICLR 2026
that has not been applied in popular benchmarks such as HLE, GPQA and MMLU-Pro. Based on the generated report, we use the pest-performin GPT-OSS- 120B Judge (high reasoning effort for wwwess/hheemiityyyylleeehhhe
criteria and low for all others) from $4.2 to grade the response each criterion. Inspired by HealthBench (Arora et al., 2025) and PaperBench (Starace et al., 2025), calculate the funilmmtf rate of each response, weighted by their criterion importance (I for additional, 2 for minor, 3 for major and for critical). To validate this scoring schema, we calculate the predicted performance of the three models with numan-annotated responses (o3, Grok4 and R R1-0528). We find that this schema is capable of scoring each model with only a 0.7 to 1.3% gap between judge- audge-preediteed and human-annotar annotated performance across models, Further details in $E.
Inference Setup Our inference setup largely follows that of $4.1. As we expect long generations, we generate up to 32,000 tokens for non-reasoning LLMs (max possible sequence length for GPT-4.1) and 64,000 for reasoning LLMs (max possible sequence length for Claude 4 Sonnet and Gemini 2.5 Pro). To estimate the performance variance, we repeat the generation 16 times and report the average alongside the standard deviation, following Bercovich et (2025). Further details in $E.
5.1
RESULTS
Table 3: Evaluation of LLMs as report-generators. Higher is better for Score and Accuracy.
Criterion Resoning
Langth Chars.
Tokens
Madei
Phydes
Carniory
Prnnnr
Canesting
Dunill
ParnetRocal
Style
1x
Cur
Retsoning LLM
Cloed Reasniit
OpenAAGGPsS Digh:
403 (2.6)
20.11.11
(3.7(2.5)
900 12.00
65.91111
66.4
66-2
(2.3
5451
23758
14583
112.34
0peAAiGPP55-mm Dighh
$1. (2.85
636.63.)
51A--3-4
(3.13
01 71
16.711.0 (1
00.101.0
06.3 (0
9678
20859
19038
21:39
OpenAirrr---ua (light
OpenAlia
699-2.5)
61.31.3)
4156
19445
4709
45:02
OpenAliok-an misi
41.9
58.5122)
$4.7
(2.7)
$8.2
55.8
$8.3
6LD (5.6)
3856
31679
4363
35.31
Geegiarrinn-22 S-1to
16.12.00
440(1-5)
(1.5
80.01 91
614 (2.1)
98.10.4
16.6
7449
6006
1940
15.95
GorgaGGmini S-lladd (Thinking
450/2.0
61.6
669
52.11
16.12.8) 81
C3.0
(0
13947
130.0
20:43
GrogleGeniti
(Thikking)
317/2.1
(2:9)
05.012.5
31
54.015.5)
x 0005
(3.3
53.11.55
51.915221
(7.70
5350
Anllenpollinnk-ooneee (Thnking)
439
30.3 (2.3)
71.4
$5.80090
13.8
S48.033)
65.8
3856
51044
5015
164.39
Open weiel- Reseering
OjenAlgg--s::31 30h
41.4
(3.5
(2.5
66.0
11
11
01.2
16.2
5331
11600
4705
1175
Daspood AbDepSree-V3 V3. (Thnningg
(3.0
(3.3
6 (3.3
538(13) 21
51.12.40 13.9
$360.3
(0.7
5359
1135%
7486
5.27
QweaQuenn- -11---5---------8 Thiking 2507
(2.1
(2.4
42.3 (2-5)
67.3 12.4)
$14
$LA
61.9 00.00
6046
12842
9256
2.47
Queakpeen-3ee A3B Trinking
618
44.11.44
(1.4
42.3
4757
12339
InstrestNon.noo Rearoning LLM
Cloed Source lasrect
0geAAIGFF44
51.0:3.4)
(1
(1.5
50.7
53.7
6921
9.52
OpenAiGPPE-4 1-aaa
245..2.1
40.8 (3.3)
33.4/2.9)
56.2
39 39.5 m
36.912.00
38.4
$3.5 (5.90
6350
35581
1965
2.78
48.613.0
40 (2:7)
56.302.0
08.8 12.4)
16.113 31
11.1 m
65.7
11613
6080
1036
10:67
194
49:0
44.0-346
63. 13.35
42 31
4..00.33
480 06 71
34167
4006
1747
2.55
402
(2.00
69.6
53.5 m1
55.3 (2.2)
SL. (1.9)
06.30
4058
51016
111.37
27.7
Open-weigtt Instroc
Orennhern 200-:a---50---22----0-0 2000
45.61244
(3.5
(5.7
68.
542
51.0117)
(1.
66.2005
13450
4244
MoonbenAAKinii Aani--e X2 Ausree 2905
40452.55
12.9
65.9 45.5-2.)
51.3
(1.
(5.9)
4117
11462
1565
3.36
DaspSeek ADDeepSeck V3.1
(2.0
(3.0
63.1 12.4)
53.5 40
52 11
53. 01.0
561 (4.30
7150
11331
3057
2.07
Metallana---nn munciic
35..2211
33.513.5
34./22-5)
53.5 12.50
11 43
30.32220 (2-70
C6.5
(5.9
4223
16806
1191
1.56
matallbma---cco extal
23.411.5
33.4
50L3
35.1 (1.7
33.33
42.3
3612
16675
1039
Inforence
OpenAAGPES
high nonceing
49:112.6
(2.1
(37.3.4)
10.0
68911
66.4 51
(1.3
5151
2775%
14583
112.34
masoring
(1.9
47.41.33
604(1.5)
73.7 (1.5
62.
60.912.0)
5328
4960
49.50
miniral aonoring
5184224
46.
11.9
59..11.0 91
58.2 11
7384
23986
1381
11.45
high wathoekry
49.0
71.1 (3-1)
656-2-11
76.7 33
11 4)
66.012.4
06.70.75
62 (6.3)
7133
29652
10984
67:04
mediun verbosity ideiia)))
49.9/2.0
(344
(3.6
75.0 (2.7)
63.2 30
(2.9
(3.8
62.5 (5.0)
5388
23775
9911
$2.45
welbosiyy
429
(2.9)
76.
(2.2
609
02.4 (099
(5.50
3752
23033
5899
Overall Top-performing Model Overall, GPT-5 achieves the best performance in Tab. 3 at 65.9%.
confirming the benchmark's 's challenging nature compared to popular benchmarks like AIME 25 (GPT-5 reaches 94.6%). GPQA-Diamond (GPT-5 reaches 87. 87.0%) and SWEBench Verified (GPT-5 reaches 72.4%) (OpenAI, 2025). The result shows that ProtBench is approximately as challenging as HealthBench, where GPT-5 reaches 67.2%, despite covering many diverse domains and cheaper evaluation cost (e.g., g. one round of HealthBench evaluation takes up to $300 with o3 while the same model on ProfBench only costs $48). Among the domains, Physics is most challenging (49.3%), followed by Finance (63.8%) Chemistry (70.6%) and Consulting (80.0%). We further analyze performance across domains in SK and the best performing model at each price-point in $F.
7

---

Published as a conference paper at ICLR 2026
Are Closed-source Models Better than Open-weight Models? In general, the top- oop-erformmng models tend to be proprictary models such as GPT-5 (65.9%), o3 (61.4%) and Gemini 2.5 Pro (60.3%). The top weight models are GPT-OSS-1200 performing 54.9% and DeepSeek V3.1 (Thinking) at 53.8%. The performance gap between closed-source and open-weight models is small for domains like Physics (<I %)) moderate for Chemistry and Consulting (9.2% and 9.6%) and particularly large for Finance (15.0%). This might be result of open-weight models having more in-domain training data and potentially over-emphasizing benchmarks relating to Code (e.g. LiveCodeBench) and Math (e.g. AIME 25) which are similar in problem-solving approach to Physics. On the other hand, much less attention has been placed on measuring and improving model performance in Chemistry, Consulting and Finance. We believe ProfBench can facilitate measurement and thereby catalyze progress model capabilities within these domains, especially among open-weight models.
Does Model Size Matter? Similar to LLM-Judge performance, larger models tend to perform better within each model family. However, size alone exhibits diminishing return across the model families-for for instance, GPT-5-mini shows a 10.2% improvement over GPT-5-nano, but GPT-5 only shows a 5.66 gain over GPT-5-mini. A similar trend is also observed with the Gemini-2.5 family (Thinking), as well as open-weight model families such as Llama 4 and Qwen3-Instruct22507 This suggests that improvement in ProfBench necessitates not only model scaling, but possibly further innovations in training techniques and data curation.
To Think or Not to Think? Using the same model for which thinking can be either enabled or disabled, turning on the thinking feature slightly improves overall performance (between 0.3 to 2.3%) as exemplified by Gemini-2... 2.5--Fashh (and its Lite sibling), Claude-Sonnet-- and DeepSeek V3.1. Similarly, increasing reasoning effort of GPT-5 from minimal to high gradually increases overall performance by 4.8%. However, when inferring with separate models of identical size trained for instruction following and thinking respectively, thinking does not necessarily confer an advantage. For instance,
30B-A3B-Thimme 2500 scores 44.6% while Qwen3-30B- -0----3-B-3B----- 2-0------Instttt-22222 Instruct-2507 records 49, 3% in overall performance, as similar to their larger 235B cousins. This might be because the instruct version generates much longer response (1 117 characters on average) compared to the thinking version (4757 characters on average), which we investigate below.
Is there any Advantage for Longer Response Lengths? Intuitively, longer responses tend to cover more content and hence increase the chance of satisfying more criteria. This explains the reason behind the poor performance of Claude-3.5 3.5de--3--Hiikee Haiiu (27.6 %), which generates only 1784 characters on average, less than half of the next most concise model. However, we also find that beyond a certain threshold, longer responses do not warrant better performance. For instance, o3 scores 61.4% with only an average of 4158 characters while GPT-5-nano scores 50. 5% despite having more than twice the average response length (9796 characters). To better understand the effects that response verbosity plays in influencing ProßBench performance, we experiment with changing the verbosity flag on GPT-5. With low verbosity setting, the overall score dropped by only 3.0% compared to medium verbosity, even though the average response length dropped by 30.7%. Similarly, with high verbosity, the score increases by 1.1% compared to medium verbosity even though the average response length increased by 32.4%. This suggests that while response length does influence performance, its effect is minimal typically within the standard deviation of the two verbosity settings.
Can Inference Cost be Further Reduced without Affecting Robustness? With o3, running ProfBench with 16 responses per task costs $48, which is substantially cheaper than Hench or PaperBench at $300 and $8000 respectively. Here, we further explore if ProfBench can be even more accessible by cutting down the cost, without compromising the robustness of the evaluation. To address this, we first observe that the performance variance differs significantly across the 40 task evaluated (Fig. fo- example, the score of Gemini-2.5-Flaatt on Task Chem-9 ranges from 11.2 to 63.8, whereas its score on Task Chem-4 remains stable, ranging from 82.0 to 96.1. This suggests that allocating a fixed number of samples uniformly across all tasks may be suboptimal; instead, the overall variance can be reduced without additional cost by generating more samples for high variance tasks and fewer for low variance tasks. We formulate this as an optimal allocation problem in integer programming, where fixed budget (i.e., number of generations) is distributed across tasks to minimize the variance of the overall performance. Given the small scale of the problem, we solve it efficiently using dynamic programming. For formal description of the problem and comparison
8

---

Published as a conference paper at ICLR 2026
Optimal Samples for each Task ID
2.2
Score Distribution for each Task ID
0.8
So0r diritucoo
42
4.3
Phyne
Phond
rind
creand
crenee
Carod
Hovd
Phondd
Cousia
Caoood
Coro
cremm
Figure 3: Score Distribution and Optimal Samples for various tasks. Each blue box represents 25th, 50th and 75th percentile and whiskers represent the worst and best score out of 16 samples for a task by 2..mini-..5-Flssl (Thinking). Tasks with lower variance can be estimated with fewer samples (shown as height of orange bar), reducing inference cost w/o sacrificing estimation robustness.
with baseline methods, see $G. This allows us allocate average of responses per tasks (and cut the inference cost to 25% of the original) without compromising robustness of our estimations.
Table 4: Ablation of LLM as report- generators. Higher is better for Score and Accuracy.
Score Firance
Accuracy by Criterion Type
Length Chars
Tokems Our
Model
Phys
Chem
Consthing
Ovemil
EstretReeolll
Reseeiing
Style
In
5
OpenAlin
LLM only
399((1.) (1.9)
(3.5
(2.3
(1.5 (2.2
(3.8
(1.8
63.1 (6.6)
4084
467
5383
28.16
+ soarch capability
40.9 (2.6)
(5.8
(3.9(
(1.9(1))
56.1 (1:2)
(1.1
57.(11))
608 (5.4)
4458
243370
5046
940.68
+ grounding dacuments
(2.5
61.5
(2.9
(1.(
61.3)
6..9
61.5(1.3)
(5.0
4158
18445
4709
47.72
OpenAlot mini
+ -LM only
(2.9 (1.)
(3.1)
(2.6
463 (1.2)
37.1 (2.3)
(1.4)
(3.4
3232
467
4293
12,42
4 scarch cagability
(2.6)
(3.7
(2.0
(2.2
53.3 (1.7)
(2.4
(1.7
60.4 (5.6)
4774
127403
6409
107.74
* grounding docuseeuss
45.5 (1.9)
(2.2)
(2.5
(2.7
58.2 (1.00
55.5 (1.7)
(1.8
61.01 (5.6)
3896
31679
4763
35.71
6 ABLATION: How IMPORTANT ARE GROUNDING DOCUMENTS?
Task Formulation Recent benchmarks such as Humanity's 's Last Exam report that model perfor- mance can be significantly improved via web search (xAI, 2025; OpenAl, 2025). To understand the effect of web search and the grounding documents in each task, we conduct two ablations-(1) prompting without the documents, and (2) prompting without the documents but allowing the models to retrieve relevant documents through web search (as all documents have publicly accessible urls).
Results With both o3 and 04-mini, removing grounding documents greatly reduces the performance (9.4 to 11.9 %), suggesting that explicit reference to the documents is crucial for the generation quality; the effect is particularly pronounced in information extraction recall (17.1 to 18.7%). We notice that without grounding prompts in such documents, models commonly respond with clarifying requests/questions (e.g., Please supply the RETT's Q1 25 NOI, total assets, total liabilities, shares outstanding and 3-month ADV... ...), especially for finance and consulting. Adding search capabilities recovers some performance (4.1 to 7.0%), indicating that web search can identify some relevant documents. However, the large amount of input tokens (0.12 0.24 million per task, or 4 to 12x as much as the original grounding documents) suggests that a large quantity of document tokens might be included in prompt context, highlighting retrieval precision as a potential area for improvement.
CONCLUSION
We present ProfBench, the first benchmark with expert-curated rubrics across diverse professional domains including Physics PhD, Chemistry PhD, Finance MBA and Consulting MBA. ProfBench addresses core limitation of existing benchmarks by moving beyond exam-style tasks with short answers, enabling systematic evaluation on open-ended problems with real- real-world value. ProfBench is also fair and accessible with reduced L..M-Judge bias and substantially lowered evaluation costs.
9

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
All data collection carried out on this project was performed by our vendor, following internal reviews on ethical and legal standards prior to the start of the project. All individuals engaged in the project were qualified for their respective roles and provided services under fair and appropriate working conditions. Compensation for all personnel was set or above locally applicable standards, and no practices involving coercion, overwork, or unfair treatment were employed. Prior to the inclusion into the project, annotators sign an agreement not to infringe on any third party's intellectual property or other proprietary rights, indemnifying our vendor if any such claims were brought up by third parties.
REPRODUCIBILITY STATEMENT
Procedures for data collection has been extensively documented in $3, C and D. Evaluation details are in $4.1, and B.
REFERENCES
Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quinonero-Candela. Foivos Tsimpousaas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. Healthbench: Evaluating large language models towards improved human health, 2025. URL https: / arxiv. orooorgg///oooo absg///b////a / 2505. 08775.
Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran EI- Eaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp. Ehud Karpas, Ran Zilberstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang. Tugrul Konuk, Gerald Shen, Ameya Sunil Mahabaleshwarkan Bilal Kartal, Yoshi Suhara, Olivier Delalleau, Zijia Chen, Zhilin Wang. David Mosallanezhad, Adi Renduchintala, Haifeng Qian, Dima Rekesh, Fei Jia, Somshubra Majumdar, Vahid Noroozi, Wasi Uddin Ahmad, Sean Narenthiran, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Siddhartha Jain, Igor Gitman, Ivan Moshkov, Wei Du, Shubham Toshniwal, George Armstrong, Branislav Kisacanin, Matvei Novikov, Daria Gitman, Evelina Bakhturina, Jane Polak Scowcroft, John Kamalu, Dan Su, Kezhi Kong, Markus Kliegl, Rabeeh Karimi, Ying Lin, Sanjeev Satheesh, Jupinder Parmar, Pritam Gundecha, Brandon Norick, Joseph Jennings, Shrimai Prabhumoye. Syeda Nahida Akter, Mostofa Patwary, Abhinav Khattar, Deepak Narayanan, Roger Waleffe, Jimmy Zhang, Bor- BiinY Su, Guyue Huang, Terry Kong, Parth Chadha, Sahil Jain, Christine Harvey, Elad Segal, Jining Huang, Sergey Kashirsky, Robert McQueen, Izzy Putterman, George Lam, Arun Venkatesan, Sherry Wu, Vinh Nguyen, Manoj Kilaru, Andrew Wang. Anna Warno, Abhilash Somasamudramath, Sandip Bhaskar, Maka Dong, Nave Assaf, Shahar Mor, Omer Ullman Argov, Scot Junkin, Oleksandr Romanenko, Pedro Larroy, Monika Katariya, Marco Rovinelli, Viji Balas, Nicholas Edelman, Anahita Bhiwandiwalla, Muthu Subramaniam, Smita Ithape. Karthik Ramamoorthy, Yuting Wu, Suguna Varshini Velury, Omri Almog, Joyjit Daw, Denys Fridman, Erick Galinkin, Michael Evans, Katherine Luna, Leon Derczynski, Nikki Pope, Eileen Long, Seth Schneider, Guillermo Siman, Tomasz Grzegorzek, Pablo Ribalta, Monika Katariya, Joey Conway, Trisha Saar, Ann Guan, Krzysztof Pawelec, Shyamala Prayaga, Oleksii Kuchaiev, Boris Ginsburg, Oluwatobi Olabiyi, Kari Briski, Jonathan Cohen, Bryan Catanzaro, Jonah Alben, Yonatan Geifman, and Eric Chung. Llama-nemotron: Efficient reasoning models, 2025. URL httpt:://///..... /tttps::/arkee
org/ aasg//b//// /tabs/2500c 2505. 00949.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. large annotated corpus for learning natural language inference, 2015. URL https:: wttpaawwww...ggga//araa//aaa 1508. 05326.
DeepSeek-AL, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen. Dongjie Ji. Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen. Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian L Liang, Jin Chen, Kai Dong,
10

---

Published as a conference paper at ICLR 2026
Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong 7hang, Ruizhe Pan, Runji Wang. R. Chen, R. L.. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye. Tao Yun, Tian Pei. Tianyu Sun, Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang. Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL https:://///. //aaaaiiiiv//aaa///aa org/ abs/ 250.. .11001.11 12948.
Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents, 2025. URL https: htttps://axxiv.ortt abs/ 2506. 11763.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback, 2023.
Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Re, Adam Chilton, Aditya Narayana, Alex Chohlas- Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John J Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li. Legalhench: A collaboratively built benchmark for measuring legal reasoning in large language models. In Thirty-sevnnth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL wwt.tttp/////c /ttp//eeeeeeeeeeeeeeeeeeeeeeeeee weteeeewwweeeeeeeeeeeeeeeeeeeeee /amee/foruuee forum? Md=:aspF XFRCC.
Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains, 2025. URL pttttps:///rxiie org/abs/2507. 17746.
Ziwen Han, Meher Mankikar, Julian Michael, and Zifan Wang. Search-ting data contamination, 2025. URL httpt:
www..org//abbo hatg//b///// 2500 13180.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. URL httpt::t////atttii /ttp///aavvv///////////////ae 07974.
httttp:://axiivoorr//bbbsts/ crg/abs /2403.
JazzCore. Pdlkit. https: http::///ppp..rrr/ppr//ttt///e/p
2025.
Jachun Jung, Facze Brahman, and Yejin Choi. Trust escalate: LLM judges with provable guarantees for human agreement. In The Thirteenth International Conference on Learning Representations. 2025. URL https::////. openreview net. forum? www.rrmmmmuuunneeeee d=UHH
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca
11

---

Published as a conference paper at ICLR 2026
Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing fromtiers in open language model post-training, 2025. URL https: hwtpt::ttpptiii///eeee// h/tpiv..r////v/////// abs/2411.15t 151244
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The Arena-Hard pipeline. https: //1 ///msys..rm org/bogg///o.. /b ollog//// oollog//22222 201//224--0000 04-004--19 19 arena rrena-hardd/ April 2024.
Ximing Lu, Melanie Sclar, Skyler Hallinan, Niloofar Mireshghallah, Jiacheng Liu, Seungju Han, Allyson Ettinger. Liwei Jiang, Khyathi Chandu, Nouha Dziri, and Yejin Choi. Al as humanity's s salieri: Quantifying linguistic creativity of language models via systematic attribution of machine text against web text. In The Thirteenth International Conference on Learning Representations, 2025. URL https: : http:////ppeet....oooo/////////a -ati--ppt---- 2025.
OpenAl. Introducing
OpenRouter. Openrouter. httpppss/ggeeerooteerrr///eeeeee
2025.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang. Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy. Oliver Zhang, Mantas Mazeika, Dmitry Dodonov, Tung Nguyen, Jaeho Lee, Daron Anderson, Mikhail Doroshenko, Alun Cennyth Stokes, Mobeen Mahmood, Oleksandr Pokutnyi, Oleg Iskra, Jessica P. Wang, John-Clark Levin, Mstyslav Kazakov, Fiona Feng, Steven Y. Feng, Haoran Zhao. Michael Yu, Varun Gangal, Chelsea Zou, Zihan Wang, Serguei Popov, Robert Gerbicz, Geoff Galgon, Johannes Schmitr, Will Yeadon, Yongki Lee, Scott Sauers, Alvaro Sanchez, Fabian Giska, Mare Roth, Søren Rüs. Saiteja Utpala, Noah Burns, Gashaw M. Goshu, Mohinder Maheshbhai Naiya, Chidozie Agu, Zachary Giboney, Antrell Cheatom, Francesco Fournier-Facio. Sarah-Jane Crowson, Lennart Finke, Zerui Cheng, Jennifer Zampese, Ryan G. Hoerr, Mark Nandor, Hyunwoo Park, Tim Gehrunger, Jiaqi Cai, Ben McCarty, Alexis C Garretson, Edwin Taylor, Damien Sileo, Qiuyu Ren, Usman Qazi, Lianghui Li, Jungbae Nam, John B. Wydallis, Pavel Arkhipov, Jack Wei Lun Shi, Aras Bacho, Chris G. Willcocks, Hangrui Cao, Sumeet Motwani, Emily de Oliveira Santos, Johannes Veith, Edward Vendrow, Doru Cojoc, Kengo Zenitani, Joshua Robinson, Longke Tang. Yuqi Li, Joshua Vendrow, Natanael Wildner Fraga, Vladyslav Kuchkin, Andrey Pupasov Maksimov, Pierre Marion, Denis Efremov, Jayson Lynch, Kaiqu Liang, Aleksandar Mikov, Andrew Gritsevskiy, Julien Guillod, Gözdenur Demir, Dakotah Martinez, Ben Pageler, Kevin Zhou, Saeed Soori, Ori Press, Henry Tang, Paolo Rissone, Sean R. Green, Lina Brüssel, Moon Twayana, Aymeric Dieuleveut, Joseph Marvin Imperial, Ameya Prabhu, Jinzhou Yang, Nick Crispino, Arun Rao, Dimitri Zvonkine, Gabriel Loiseau, Mikhail Kalinin, Marco Lukas, Ciprian Manolescu, Nate Stambaugh, Subrata Mishra, Tad Hogg, Carlo Bosio, Brian Coppola, Julian Salazar, Jachyeok Jin, Rafael Sayous, Stefan Ivanov, Philippe Schwaller, Shaipranesh Senthilkuma, Andres M Bran, Andres Algaba, Kelsey Van den Houte, Lynn Van Der Sypt, Brecht Verbeken, David Noever, Alexei Kopylov, Benjamin Myklebust, Bikun Li, Lisa Schut, Evgenii Zheltonozhskii, Qiaochu Yuan, Derek Lim, Richard Stanley, Tong Yang, John Maar, Julian Wykowski, Martí Oller, Anmol Sahu, Cesare Giulio Ardito. Yuzheng Hu, Ariel Ghislain Kemogne Kamdoum, Alvin Jin, Tobias Garcia Vilchis, Yuexuan Zu, Martin Lackner, James Koppel, Gongbo Sun, Daniil S. Antonenko, Steffi Chern, Bingchen Zhao Pierrot Arsene, Joseph M Cavanagh, Daofeng Li, Jiawei Shen, Donato Crisostomi, Wenjin Zhang, Ali Dehghan, Sergey Ivanov, David Perrella, Nurdin Kaparov, Allen Zang. Ilia Sucholutsky, Arina Kharlamova, Daniil Orel, Vladislav Poritski, Shalev Ben-Da Zachary Berger, Parker Whitfill, Michael Foster, Daniel Munro, Linh Ho, Shankar Sivarajan, Dan Bar Hava, Aleksey Kuchkin, David Holmes, Alexandra Rodriguez-Romert Frank Sommerhage. Anji Zhang, Richard Moat, Keith Schneider, Zakayo Kazibwe, Don Clarke, Dae Hyun Kim, Felipe Meneguitti Dias, Sara Fish, Veit Elser, Tobias Kreiman, Victor Efren Guadarrama Vilchis, Immo Klose, Uijwala Anantheswaran, Adam Zweiger, Kaivalya Rawal, Jeffery Li, Jeremy Nguyen, Nicolas Daans, Haline Heidinger, Maksim Radionov, Václav Rozhoö, Vincent Ginis, Christian Stump, Niv Cohen, Rafal Pošwiata, Josef Tkadlec, Alan Goldfarb, Chenguang Wang, Piotr Padlewski. Stanislaw Barzowski, Kyle Mont- gomery, Ryan Stendall, Jamie Tucker-Foltz, Jack Stade, T. Ryan Rogers, Tom Goertzen, Declan Grabb, Abhishek Shukla, Alan Givré, John Amold Ambay, Archan Sen, Muhammad Fayez Aziz, Mark H Inlow, Hao He, Ling Zhang, Younesse Kaddar, Ivar Ängquist, Yanxu Chen, Harrison K Wang, Kalyan Ramakrishnan, Elliott Thornley, Antonio Terpin, Hailey Schoelkopf, Eric Zheng,
12

---

Published as a conference paper at ICLR 2026
Avishy Carmi, Ethan D. L. Brown, Kelin Zhu, Max Bartolo, Richard Wheeler, Martin Stehberger, Peter Bradshaw, JP Heimonen, Kaustubh Sridhar, Ido Akov, Jennifer Sandlin, Yury Makarychev, Joanna Tam, Hieu Hoang, David M. Cunningham, Vladimir Goryachev, Demosthenes Patramanis, Michael Krause, Andrew Redenti, David Aldous, Jesyin Lai, Shannon Coleman, Jiangnan Xu, Sangwon Lee, Ilias Magoulas, Sandy Zhao, Ning Tang. Michael K. Cohen, Orr Paradise, Jan Hen- drik Kirchner, Maksym Ovchynnikov, Jason O. Matos, Adithya Shenoy, Michael Wang, Yuzhou Nie, Anna Sztyber-Belley, Paolo Faraboschi, Robin Riblet, Jonathan Crozier, Shiv Halasyamani, Shreyas Verma, Prashant Joshi, Eli Meril, Ziqiao Ma, Jérémy Andréoletti, Raghav Singhal, Jacob Platnick, Volodymyr Nevirkovets, Luke Basler, Alexander Ivanov, Seri Khoury, Nils Gustafsson, Marco Piccardo. Hamid Mostaghimi, Qijia Chen, Virendra Singh, Tran Quoe Khánh, Paul Rosu, Hannah Szlyk, Zachary Brown, Himanshu Narayan, Aline Menezes, Jonathan Roberts, William Alley, Kunyang Sun, Arkil Patel, Max Lamparth, Anka Reuel, I. Linwei Xin, Hanmeng Xu,, Jacob Loader, Freddie Martin, Zixuan Wang. Andrea Achilleos, Thomas Preu, Tomek Korbak, Ida Bosio, Fereshteh Kazemi, Ziye Chen, Biró Bálint, Eve J. Y. Lo, Jiaqi Wang. Maria Inés S. Nunes, Jeremiah Milbauer, M Saiful Bari, Zihao Wang, Behzad Ansarinejad, Yewen Sun, Stephane Durand, Hossam Elgnainy, Guillaume Douville, Daniel Tordera, George Balabanian, Hew Wolff, Lynna Kvistad, Hsiaoyun Milliron, Ahmad Sakor, Murat Eron, Andrew Favre D. O., Shailesh Shah, Xiaoxiang Zhou, Firuz Kamalov, Sherwin Abdoli, Tim Santens, Shaul Barkan, Allison Tee, Robin Zhang, Alessandro Tomasiello, G. Bruno De Luca, Shi-Zhuo Looi, Vinh- -haa Le, Noam Kolt, Jiayi Pan, Emma Rodman, Jacob Drori, Carl J Fossum, Niklas Muennighoff, Milind Jagota, Ronak Pradeep. Honglu Fan, Jonathan Eicher, Michael Chen, Kushal Thaman. William Merrill, Moritz Firsching, Carter Harris, Stefan Ciobåcā, Jason Gross, Rohan Pandey, Ilya Gusev, Adam Jones, Shashank Agnihotri, Pavel Zhelnov, Mohammadreza Mofayezi, Alexander Piperski, David K. Zhang, Kos- tiantyn Dobarskyi, Roman Leventov, Ignat Soroko, Joshua Duersch, Vage Taamazyan, Andrew Ho, Wenjie Ma. William Held, Ruicheng Xian, Armel Randy Zebaze, Mohanad Mohamed, Julian Noah Leser, Michelle X Yuan, Laila Yacar, Johannes Lengler, Katarzyna Olszewska, Claudio Di Fratta, Edson Oliveira, Joseph W. Jackson, Andy Zou, Muthu Chidambaram, Timothy Manik, Hector Haffenden, Dashiell Stander, Ali Dasouqi, Alexander Shen, Bita Golshani, David Stap, Egor Kretov, Mikalai Uzhou, Alina Borisovna Zhidkovskaya, Nick Winter, Miguel Orbegozo Rodriguez, Robert Lauff, Dustin Wehr, Colin Tang, Zaki Hossain, Shaun Phillips, Fortuna Samuele, Fredrik Ekström, Angela Hammon, Oam Patel, Faraz Farhidi, George Medley, Forough Mohammadzadeh. Madellene Peñaflor, Haile Kassahun, Alena Friedrich, Rayner Hernandez Perez, Daniel Pyda, Taom Sakal, Omkar Dhamane, Ali Khajegili Mirabadi, Eric Hallman, Kenchi Okutsu, Mike Battaglia, Mohammad Maghsoudimehrrbann, Alon Amit. Dave Hulbert, Roberto Pereira, Simon Weber, Handoko, Anton Peristyy, Stephen Malina, Mustafa Mehkary, Rami Aly, Frank Reidegeld, Anna-Katharina Dick, Cary Friday, Mukhwinder Singh, Hassan Shapourian, Wanyoung Kim, Mar- iana Costa, Hubeyb Gurdogan, Harsh Kumar, Chiara Ceconello, Chao Zhuang. Haon Park, Micah Carroll, Andrew R. Tawfeek, Stefan Steinerberger, Daattavya Aggarwal, Michael Kirchhof, Linjie Dai, Evan Kim, Johan Ferret, Jainam Shah, Yuzhou Wang. Minghao Yan, Krzysztof Burdzy, Lixin Zhang, Antonio Franca, Diana T. Pham, Kang Yong Loh, Joshua Robinson, Abram Jackson, Paolo Giordano, Philipp Petersen, Adrian Cosma, Jesus Colino, Colin White, Jacob Votava, Vladimir Vinnikov, Ethan Delaney, Petr Spelda, Vit Stritecky, Syed M. Shahid, Jean-Christophe Mourrat, Lavr Vetoshkin, Koen Sponselee, Renas Bacho, Zheng-Xin Yong, Florencia de la Rosa, Nathan Cho, Xiuyu Li, Guillaume Malod, Orion Weller, Guglielmo Albani, Leon Lang. Julien Laurendeau, Dmitry Kazakov, Fatimah Adesanya, Julien Portier, Lawrence Hollom, Victor Souza, Yuchen Anna Zhou, Julien Degorre, Yiğit Yalm, Gbenga Daniel Obikoya, Rai, Filippo Bigi M. C. Boscá, Oleg Shumar, Kaniuar Bacho, Gabriel Recchia, Mara Popescu, Nikita Shulga, Ngefor Mildred Tanwie, Thomas C. H. Lux, Ben Rank, Colin Ni, Matthew Brooks, Alesia Yakimchyk, Huanxu, Liu, Stefano Cavalleri, Olle Häggström, Emil Verkama, Joshua Newbould, Hans Gundlach, Leonor Brito-Santana, Brian Amaro, Vivek Vajipey, Rynaa Grover, Ting Wang. Yosi Kratish, Wen-Ding Li, Sivakanth Gopi, Andrea Caciolai, Christian Schroeder de Witt, Pablo Hernández-Cámara. Emanuele Rodolà, Jules Robins, Dominic Williamson, Vincent Cheng, Brad Raynor, Hao Qi, Ben Segev, Jingxuan Fan, Sarah Martinson, Erik Y. Wang, Kaylie Hausknecht, Michael P. Brenner, Mao Mao, Christoph Demian, Peyman Kassani, Xinyu Zhang, David Avagian, Eshawn Jessica Scipio, Alon Ragoler, Justin Tan, Blake Sims, Rebeka Plecnik, Aaron Kirtland, Omer Faruk Bodur, D. P. Shinde, Yan Carlos Leyva Labrador, Zahra Adoul, Mohamed Zekry, Ali Karakoc, Tania C. B. Santos, Samir Shamseldeen, Loukmane Karim, Anna Liakhovitskaia, Nate Resman, Nicholas Farina, Juan Carlos Gonzalez, Gabe Maayan, Earth Anderson, Rodrigo De Oliveira Pena, Elizabeth Kelley, Hodjat Mariji, Rasoul Pouriamanesh, Wentao Wu, Ross Finocchio, Ismail
13

---

Published as a conference paper at ICLR 2026
Alarab, Joshua Cole, Danyelle Ferreira, Bryan Johnson, Mohammad Safdari, Liangti Dai, Siriphan Arthormthurasuk, Isaac C. McAlister, Alejandro José Moyano, Alexey Pronin, Jing Fan, Angel Ramirez-Trinidad, Yana Malysheva, Daphiny Pottmaier, Omid Taheri, Stanley Stepanic, Samuel Perry. Luke Askew, Raúl Adrián Huerta Rodríguez, Ali M. R. Minissi, Ricardo Lorena, Krishna- murthy Iyer, Arshad Anil Fasiludeen, Ronald Clark, Josh Ducey, Matheus Piza, Maja Somrak, Eric Vergo, Juchang Qin, Benjámin Borbás, Eric Chu, Jack Lindsey, Antoine Jallon, I. M. J. McInnis, Evan Chen, Avi Semler, Luk Gloor, Tej Shah, Mare Carauleanu, Pascal Lauer, Tran Duc Huy, Hossein Shahrtash, Emilien Duc. Lukas Lewark, Assaf Brown, Samuel Albanie, Brian Weber, Warren S. Vaz, Pierre Clavier, Yiyang Fan, Gabriel Poesia Reis e Silva, Long, Lian, Mareus Abramovitch, Xi Jiang, Sandra Mendoza, Murat Islam, Juan Gonzalez, Vasilios Mavroudis, Justin Xu, Pawan Kumar, Laxman Prasad Goswami, Daniel Bugas, Nasser Heydari, Ferenc Jeanplong, Thorben Jansen, Antonella Pinto, Archimedes Apronti, Abdallah Galal, Ng Ze-An, Ankit Singh, Tong Jiang. Joan of Are Xavier, Kanu Priya Agarwal, Mohammed Berkani, Gang Zhang. Zhchang Du, Benedito Alves de Oliveira Junior, Dmitry Malishev, Nicolas Remy, Taylor D. Hartman, Tim Tarver, Stephen Mensah, Gautier Abou Loume, Wiktor Morak, Farzad Habibi, Sarah Hoback, Will Cai, Javier Gimenez, Roselynn Grace Montecillo, Jakub Lucki, Russell Campbell, Asankhaya Sharma, Khalida Meer, Shreen Gul, Daniel Espinosa Gonzalez, Xavier Alapont, Alex Hoover, Gun- jan Chhablani, Freddie Vargus, Arunim Agarwal, Yibo Jiang. Deepakkumar Patil, David Outevsky, Kevin Joseph Scaria, Rajat Maheshwari, Abdelkader Dendane, Priti Shukla, Ashley Cartwright, Sergei Bogdanov, Niels Mündler. Sören Möller, Luca Amaboldi, Kunvar Thaman, Muhammad Re- han Siddiqi. Prajvi Saxena, Himanshu Gupta, Tony Fruhauff. Glen Sherman, Mátyás Vincze, Siranut Usawasutsakorn, Dylan Ler, Anil Radhakrishnan, Innocent Enyekwe, Sk Md Salauddin, Jiang Muzhen, Aleksandr Maksapetyan, Vivien Rossbach, Chris Harjadi, Mohsen Bahaloohoreh, Claire Sparrow, Jasdeep Sidhu, Sam Ali, Song Bian, John Lai, Eric Singer, Justine Leon Uro, Greg Bateman, Mohamed Sayed, Ahmed Menshawy, Darling Duclosel, Dario Bezzi, Yashaswini Jain, Ashley Aaron, Murat Tiryakioglu, Sheeshram Siddh, Keith Krenek, Imad Ali Shah, Jun Jin, Scott Creighton, Denis Peskoff, Zienab EL-Wasif, Ragavendran P V. Michael Richmond, Joseph McGowan, Tejal Patwardhan, Hao- Yu Sun, Ting Sun, Nikola Zubić, Samuele Sala, Stephen Ebert, Jean Kaddour, Manuel Schottdorf, Dianzhuo Wang, Gerol Petruzella, Alex Meiburg, Tilen Medved, Ali HlSheikh, S Ashwin Hebbar, Lorenzo Vaquero, Xianjun Yang, Jason Poulos, Vilém Zouhar, Sergey Bogdanik, Mingfang Zhang, Jorge Sanz-Ros, David Anugraha, Yinwei Dai, Anh N. Nhu, Xue Wang, Ali Anil Demircali, Zhibai Jia, Yuyin Zhou, Juncheng Wu, Mike He, Nitin Chandok, Aarush Sinha, Gaoxiang Luo, Long Le, Mickaël Noyé, Michal Peretkiewicz, Ioannis Pantidis, Tianbo Qi, Soham Sachin Purohit, Letitia Parcalabescu, Thai-Hoa Nguyen, Genta Indra Winata, Edoardo M. Ponti, Hanchen Li, Kaustubh Dhole, Jongee Park, Dario Abbondanza, Yuanli Wang. Anupam Nayak, Diogo M. Caetano, Antonio A. W. L. Wong, Maria del Rio-Chanona, Dániel Kondor, Pieter Francois, Edd Chalstrey, Jakob Zsambok, Dan Hoyer, Jenny Reddish, Jakob Hauser, Francisco-Javier Rodrigo- Ginés, Suchandra Datta, Maxwell Shepherd, Thom Kamphuis. Qizheng Zhang, Hyunjun Kim, Ruiji Sun, Jianzhu Yao, Franck Dernoncourt, Satyapriya Krishna, Sina Rismanchian, Bonan Pu, Francesco Pinto, Yingheng Wang. Kumar Shridhar, Kalon J. Overholt, Glib Briaa, Hieu Nguyen, David, Soler Bartomeu, Tony CY Pang, Adam Wecker, Yifan Xiong, Fanfei Li, Lukas S. Huber, Joshua Jaeger, Romano De Maddalena, Xing Han Lå, Yuhui Zhang, Claas Beger. Patrick Tser Jern Kon, Sean Li, Vivek Sanker, Ming Yin. Yihao Liang, Xinlu Zhang, Ankit Agrawal, Li S. Yifei, Zechen Zhang. Mu Cai, Yasin Sonmez, Costin Cozianu, Changhao Li, Alex Slen, Shoubin Yu, Hyun Kyu Park, Gabriele Sarti, Marcin Briański, Alessandro Stolfo, Truong An Nguyen, Mike Zhang, Yotam Perlitz. Jose Hernandez-Oralle Runjia Li, Amin Sha- bani, Felix Juefei-Xu, Shikhar Dhingra, Orr Zohar, My Chiffon Nguyen, Alexander Pondaven, Abdurrahim Yilmaz, Xuandong Zhao, Chuanyang Jin, Muyan Jiang, Stefan Todoran, Xinyao Han, Jules Kreuer, Brian Rabern, Anna Plassart, Martino Maggetti, Luther Yap. Robert Geirhos, Jonathon Kean, Dingsu Wang. Sina Mollaei, Chenkai Sun, Yifan Yin, Shiqi Wang, Rui Li, Yaowen Chang. Anjiang Wei, Alice Bizeul, Xiaohan Wang, Alexandre Oliveira Arrais, Kushin Mukherjee, Jorge Haamorro-Padiall Jiachen Liu, Xingyu Qu, Junyi Guan, Adam Bouyamourn, Shuyu Wu, Martyna Plomecka, Junda Chen, Mengze Tang, Jiaqi Deng, Shreyas Subramanian, Haocheng Xi, Haoxuan Chen, Weizhi Zhang, Yinuo Ren, Haoqin Tu, Sejong Kim, Yushun Chen, Sara Vera Marjanović, Junwoo Ha, Grzegorz Luezyna, Jeff J. Ma, Zewen Shen, Dawn Song, Cedegao E. Zhang, Zhun Wang, Gaël Gendron, Yunze Xiao, Leo Smucker, Erica Weng, Kwok Hao Lee, Zhe Ye, Stefano Ermon, Ignacio D. Lopez-Miguel, Theo Knights, Anthony Gitter, Namkyu Park, Boyi Wei, Hongzheng Chen, Kunal Pai, Ahmed Elkhanany, Han Lin, Philipp D. Siedler, Jichao Fang, Ritwik Mishra, Károly Ssolnai-Fehéee Xilin Jiang, Shadab Khan, Jun Yuan, Rishab Kumar
14

---

Published as a conference paper at ICLR 2026
Jain, Xi Lin, Mike Peterson, Zhe Wang. Aditya Malusare, Maosen Tang, Isha Gupta, Ivan Fosin, Timothy Kang, Barbara Dworakowska, Kazuki Matsumoto, Guangyao Zheng, Gerben Sewuster, Jorge Pretel Villanueva, Ivan Rannev, Igor Cheryyassyy, Jiale Chen, Deepayan Banik, Ben Racz, Wenchao Dong, Jianxin Wang. Laila Bashmal, Duarte V. Gonçalves, Wei Hu, Kaushik Bar, Ondrej Bohdal, Athary Singh Patlan, Shehzaad Dhuliawala, Caroline Geirhos, Julien Wist, Yuval Kansal, Bingsen Chen, Kutay Tire, Atak Talay Yücel, Brandon Christof, Veerupaksh Singla, Zijian Song, Sanxing Chen, Jiaxin Ge, Kaustubh Ponkshe, Isaac Park, Tianneng Shi, Martin Q. Ma, Joshua Mak, Sherwin Lai, Antoine Moulin, Zhuo Cheng, Zhanda Zhu, Ziyi Zhang, Vaidehi Patil, Ketan Jha, Qiutong Men, Jiaxuan Wu, Tianchi Zhang, Bruno Hebling Vieira, Alham Fikri Aji, Jae- Won Chung, Mohammed Mahfoud, Ha Thi Hoang, Mare Sperzel, Wei Hao, Kristof Meding, Sihan Xu, Vassilis Kostakos, Davide Manini, Yueying Liu, Christopher Toukmaji, Jay Paek, Eunmi Yu, Arif Engin Demircali, Zhiyi Sun, Ivan Dewerpe, Hongsen Qin, Roman Pflugfelder, James Bailey, Johnathan Morris, Ville Heilala, Sybille Rosset, Zishun Yu, Peter E. Chen, Woongycong Yco, Eeshaan Jain, Ryan Yang, Sreekar Chigurupati, Julia Chemyyvsky, Sai Prajwal Reddy, Subhashini Venugopalan, Hunar Batra, Core Francisco Park, Hieu Tran, Guilherme Maximiano, Genghan Zhang, Yizhuo Liang, Hu Shiyu, Rongwu Xu, Rui Pan, Siddharth Suresh, Ziqi Liu, Samaksh Gu- lati, Songyang Zhang, Peter Turchin, Christopher W. Bartlett, Christopher R. Scotese, Phuong M. Cao, Aakaash Nattanmai, Gordon McKellips, Anish Cheraku, Asim Suhail, Ethan Luo, Marvin Deng, Jason Luo, Ashley Zhang, Kavin Jindel, Jay Pack, Kasper Halevy, Allen Baranov, Michael Liu, Advaith Avadhanam, David Zhang, Vincent Cheng, Brad Ma, Evan Fu. Liam Do, Joshua Lass, Hubert Yang. Surya Sunkari, Vishruth Bharath, Violet Ai, James Leung, Rishit Agrawal, Alan Zhou, Kevin Chen, Tejas Kalpathi, Ziqi Xu, Gavin Wang. Tyler Xiao, Erik Maung, Sam Lee, Ryan Yang, Roy Yue, Ben Zhao, Julia Yoon, Sunny Sun, Aryan Singh, Ethan Luo,, Clark Peng. Tyler Osbey, Taozhi Wang, Daryl Echeazu, Hubert Yang. Timothy Wu, Spandan Patel, Vidhi Kulkarni, Vijaykaarti Sundarapandiyan, Ashley Zhang, Andrew Le, Zafir Nasim, Srikar Yalam, Ritesh Kasamsetty, Soham Samal, Hubert Yang, David Sun, Nihar Shah, Abhijeet Saha, Alex Zhang, Leon Nguyen, Laasya Nagumalli, Kaixin Wang, Alan Zhou, Aidan Wu, Jason Luo, Anwith Telluri, Summer Yue, Alexandr Wang, and Dan Hendrycks. Humanity's s last exam, 2025. URL https: / /tt///axvvvv//// org/ alos 2501 14449
Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following, 2025.
URL https:://// / /tt///aaxaaa aaa/iviiiivv///vv///aaa//aaaaava org/ abs /200s/2250..00 02833.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof qa benchmark. 2023. URL htw.s:t/p//// /tte:///xxxxii aawiiiiiiivvv...vv http//awxv...orgrbaaaaaa
2trrg//ab//311..100000 12022..
SearXNG. Searxng. https: htttps://githbb..oo cttnn//eaxxgggg searxng, 2025.
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai's s ability to replicate ai research, 2025. URL https: /tss://arae /ttp///axvvvv//////////ee//..av. org/ abss 2504 01848.
Yubo Wang. Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: more robust and challenging multi-task language understanding benchmark, 2024. URL https: arxiv. org/ abs 22abb//2400 22..2440..000 01574.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain. Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. Livebench: A challenging, contamination-free LLM benchmark. In The Thirteenth International Conference on Learning Representations, 2025.
xAI. Grok 4. https::///// htt.:/aa/ieww///////////se 4, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin
15

---

Published as a conference paper at ICLR 2026
Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang. Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025, URL https: /aaaaaxxvv..... org/ abs/ 0tlos//505.09385 09388.
Lianmin Zheng, Wi Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Xiyuan Zhou, Xinlei Wang, Yirui He, Yang Wu, Ruixi Zou, Yuheng Cheng, Yulu Xie, Wenxuan Liu, Huan Zhao, Yan Xu, Jinjin Gu, and Junhua Zhao. Engibench: A benchmark for evaluating large language models engineering problem solving, 2025. URL https: : / //axxiv...v///////////////a/// org/ / abs/ / 2509.17611
16

---

Published as a conference paper at ICLR 2026
A EXAMPLES
ProfBench Chemistry PhD Example
Acid-base reactions in commercial applications such as electroplating often involve multiple coupled equilibria that must be regulated in real time to maintain desired ratios of concen- trations in a mixture. Help me with the following calculations for the titration of a 100 mL mixture of acetic acid (0.5 M) and formic acid (0.1 M) with 0.5 M NaOH: 1) Calculate the volume of NaOH titrant required to reach the point where the two conjugate bases have equal concentrations.
2) Calculate the concentrations of the acids and their conjugate bases at the point referenced in part 1.
3) Calculate the concentration of hydronium ions and the pH of the analyte at the point referenced in part
4) Calculate the volume of NaOH titrant required to reach the point where the pH of the
analyte 7.0.
5) Calculate the concentrations of the acids and their conjugate bases at the point referenced part 4.
6) Calculate the volume of NaOH titrant required to neutralize both acids.
7) Calculate the concentrations of the acids and their conjugate bases at the point referenced
in part 6.
8) Calculate the concentration of hydronium ion and the pH of the analyte at the point referenced in part 6.
Example Extraction Rubric: Determines the volume of NaOH titrant required to reach the point where the pH of the analyte 7.0 0. 1938 # 0.001 L
Example Reasoning Rubric: Determines the pH of the analyte at the point at which both acids are neutralized as 9.05=0.05.
Example Style Rubric (another task): The molecular weight is rounded to I decimal place.
ProfBench Physics PhD Example
In a variant of a KSVZ-style axion model, five heavy vector-like fermions are introduced, each with different Seccei--uinn charges for left- and right-handed components (Xz, XR). Their Standard Model gauge quantum numbers and PQ charges are:
1. Q : (3.2,+ X = +1, X 0
2. Q2 : (6. 3) X + X
3, X₂ =11, Xa ===0 +1
4. :.,2:
X2 = = XR=0
5. 5.Q;: 1,0,1,-1]
+1,
Here Q is the electric charge Q TT= where T3 is the third component of weak isospin and Y is the hypercharge. and we work entirely in Standard Model hypercharge normalization. T(r) denotes the SU(3) Dynkin index and d(r)) the representation dimension. For the SU(3) representation 10, take ((10) = 15
Using the chiral PQ charge difference (X,, - Xn) as the weight for each Weyl fermion, compute the ratio F/N (electromagnetic to color anomaly coefficients) and give your answer in lowest-term fractional form.
Example Reasoning Rubric: Sums the individual color anomaly contributions obtain the total color anomaly N N₁ + N₂ - N₂ + N4 + N₂ 4
Example Extraction Rubric: Calculates wis = 1 7 3
1 uuinn Xr.5 = and X 2
Example Style Rubric: Calculates the ratio E/N = (35)/(55)
232
195
17

---

Published as a conference paper at ICLR 2026
ProfBench Consulting MBA Example
ABC Education (the "Client") delivers premium after-school STEM in English for ages 6-11 and is evaluating partner-led Hong Kong entry. The entry would involve its renting rooms from international schools and using part-time Native English Teachers (NETs).
Formatting: (i) British English, (ii) No tables, (iii) Every HKD figure must be prefixed HKS, (iv) Show intermediate steps when calculating quantitative answers rather than outputting formulas, and (v) Round HKD only at the end of each sub-task (i.e., do not round figures for intermediate calculations, unless explicitly stated).
Task 1: Competitor dynamics (<=100 words) ABC Education's 's main competitors (i) Big Bang Academy, (ii) Blueinno, and (iii) ESF Glenealy School. For each competitor, state the following: 1. Pricing: Output each competitor's class length class count package price in the following format: (e.g.. "4 classes 60 min per class HK$4 HK,64,30"". Compute HK$/h (round the end). If length is missing, write "Cannot compute (no per-hour figure)". 2. Delivery: Pick one (i) On-Campus Partner (classes hosted at partner school premises), (ii) Learning Center (dedicated provider-run teaching location), or (iii) Hybrid: Kit Video (take- home kit plus guided videos). 3. Pedagogy: Pick one (i) -eater--enteeed (teacher leads instruction; students follow), (ii) Project-Bass Based (students build projects to learn concepts), or (ii)) Inquiry-Based (students investigate questions; teacher facilitates).
Task Hourly economies (<=160 words) The Client aims to run a series of courses. They wish to understand the hourly economics of their prospective endeavor in Hong Kong. For the next questions, assume the following about the Client's 's program: Tuition price per class hour is HK$3990; capacity 8 students per class. As first step, calculate paid seats assuming they are equal to 85% of capacity; the remaining 15% will be free to offer scholarships and incentives to drive demand. Apply 15% sibling discount to 25% of the paid seats. - Apply a 2% leakage margin of safety to the post-discount tuition. This will result in Net Tuition. Venue rental fee: The Client has a global venue rental partner who is offering HK$500 per class hour for venue rental, but the Client does not wish for the venue rental cost to be higher than 22% of Net Tuition (which is standard for the countries which the Client operates). As such, choose the higher of HK$500
22% of Net Tuition for the venue rental cost. NET labor: Assume HK$480 per class hour teaching plus half an hour of prep at HK$240 per class hour. Processing Fee: 5% of Net Tuition plus HK$5 multiplied by the paid seat count. Now solve the following: 4. Compute Net Tuition, Direct costs (venue + NET labor + processing), and contribution margin per class-hour. 5. State whether 22% of Net Tuition exceeds HK$500. 6. Monthly profit: Determine the monthly profit if the Client ran two classes per week across four weeks. 7. Identify whether ABC Education's 's tuition fee is either (i) higher than all competitors, (ii) in between competitors, or (iii) lower than all competitors. Task 3: Venue rental fit (<=100>0 100 words) Harrow and DSC have approached the Client offering venue rental services. The Client wishes to understand what they are offering and whether they should partner with either school as opposed to their existing global venue rental partner. 8. Venue rental offers: Harrow is offering weekend windows at their standard base rental rate for the first hour, then HK$150/h thereafter, + 110% weekend surcharge. DSC is offering weekday windows at their standard base rental rate for two hours, + HK$500 tech, + HK$200 cleaning per booking. Calculate the total venue rental cost per school, assuming the Client is looking for two-hour windows. 9, Comparison vs. global venue rental partner: Compare each school's 's offered rate vs. the global venue rental partner's 's rate. State which provider the Client should partner with (i.c., Harrow, DSC, or the existing global venue rental partner). Example Reasoning Rubric: Calculates the processing fee by multiplying net tuition by 5% plus paid seats multiplied by HKS5.
Example Extraction Rubric: Classify ESF Glenealy pedagogy as Inquiry-Bass -Based
Example Style Rubric: Keep Task within 100 words total.
18

---

Published as a conference paper at ICLR 2026
ProfBench Finance MBA Example
You are helping me assess the potential for new business unit within a major investment bank. This unit is entirely dedicated to innovative finance for healthcare, as well as social impact and environmental challenges the global level. One example we are studying to assess this opportunity is how the Global Alliance for Vaccination and Immunization (GAVI) has been able raise money on capital markets through the International Finance Facility for Immunization (IFFIm).
I would like you to help answer the following questions:
How IFFIm able raise money capital markets for vaccination and immunization campaigns?
How did IFFIm apply securitization?
What were the factors that made it possible for IFFIm to raise money on capital markets?
What risks and challenges does IFFIm have to address and overcome?
How would you assess whether IFFIm has been effective and successful at raising funding for GAVI, and overall, has it been success for their operations and health goals?
Can IFFIm be viewed as a blueprint for investing in and funding other areas of global health, or other major social or environmental challenges? If yes, identify short pipeline of organizations/toiic that could take similar approach and use innovative finance to raise funding to advance their goals.
Provide robust context, starting with defining what Gavi and IFFIm how they work, and how they are both related. Discuss, in detail, the technical aspects of how IFFIm works, and describe what makes it possible and effective. The technical discussion should describe how it works, and provide a detailed overview of how it raised money until now (this should include a summary of past issuances, showing what markets raised and what investors subscribed). Also, discuss what elements made IFFIm possible, as well as what risks and challenges IFFIm faces. This should help you form robust and documented view what makes IFFIm success (or not), and how it could act as blueprint (or not) for other initiatives. Regarding format,
looking for the style of detailed investment memo discussion. You can use tables and bullets, but the memo should be mostly text, and you should walk me through your reasoning. I don't want the output to mostly tables and bullets.
Extraction Rubric Sample: States that breach of IFFIm's 's liquidity policy could negatively impact IFFIm's rating profile.
Reasoning Rubric Sample: States that vaccines are one of the most successful and cost- effective health investments in the world.
Style Rubric Sample (another task): Present findings clearly to allow for effective use.
B PROMPT TEMPLATES
Main prompt template Response: <response>
Evaluate whether the response above satisfies this criterion: <eriterion description>, Only answer Yes or No.
Alternative prompt template Response: <response>
Evaluate whether the response above satisfies this criterion:
Only answer True or False.
Alternative prompt template 2
Response: <response>
Evaluate whether the response above satisfies this criterion: <criterion.co www.riippnsscriiiioo...o Only answer 1 or 0.
Further Discussion Prompt Templates We found the main prompt work substantially better than the alternative (by 2.2%) or alternative 2 (by 4.5%). We believe this is because Yes/No are more commonly used in natural language compared to others. We also attempted more elaborate prompt templates (e.g. informing the LLM that they are an expert in grading responses, giving specific instructions to analyze the entire solution from every angle, providing reasoning process).
19

---

Published as a conference paper at ICLR 2026
However, did not observe any gains and had substantially increased token expenses. Therefore, we used our main prompt template, while noting that reasoning LLMs can still do reasoning within their thinking traces without explicit prompting.
C ANNOTATOR RECRUITMENT
Annotator Countries
1. United States: 20
2. United Kingdom:
3. Canada: 4
4. India:
5. Australia:
6. Spain: 1
7. Greece: 1
8. France:
Consulting MBA: We generally require annotators to have had 2 years of work experience at McKinsey, Boston Consulting Group, Bain & Company, Deloitte, PricewaterhouseCooper Ernst and Young or KPMG. Alternatively, they could have years of experience at another consulting firm. These work experience includes those prior to the completion of their highest degree (i.e. MBA). Candidates are interviewed based on their past experience, their professional ability to reason from first principles, communicate a professional manner, break down projects into logical components, and create well-supported recommendations among others.
We generally require annotators have 2 years of experience select bank including JPMorgan, Bank of America, CitiGroup, Wells Fargo, Goldman Sachs, Morgan Stanley, PNC, Truist, TD Bank and other banks with similar selectivity in regions outside of North America. These work experience includes those prior to the completion of their highest degree (i.e. MBA). Candidates are then interviewed on their previous experiences projects, deals, investments, credits, or portfolio decisions including their individual contributions, decisions made, and alternatives considered while ensuring applicants discuss key risks, explain context, and suggest mitigations.
Finance MBA:
Chemistry and Physics PhD: Physics or Chemistry at Global Top 100 university while taking into account other factors such the program selectivity. Candidates are then evaluated based interviews that test how deeply they understand their field, their ability to synthesize new information related to their niche, and clear communication of findings.
We generally require annotators have completed PhD relating to
D FURTHER DESCRIPTIVE STATISTICS
Prompts ProfBench contains 80 unique prompts, with 20 from each domain. Prompts are generally around 2 to paragraphs long with an average of 2052.4 characters (std of 997.4, min of 696, max of 5339). As shown in the examples (see ŠA), prompts typically contain multiple related sub-questions, making them substantially more specific to construct evaluation criteria for and therefore minimizing the accidental penalization of responses that approach the task in alternative ways. Chemistry and Physics PhD tasks tend to be shorter averaging 1617.4 (std of 925.6) and 1799.2 2 (std of 622.9) characters respectively while Consulting and Finance MBA tasks tend be longer 2360.8 (std of 717.2) and 2432.2 (std of 1314.3) characters respectively.
Responses are generated with OpenAl o3, DeepSeek R1-0528 and Grok4 models. They are similar in length, with o3 being the most terse at 4816.7 characters (std of 2590.5), followed by R1-0528 at 5187.7 (std of 2504.2) 2) and Grok4 is most verbose at 5997.2 (std of 3789.3) characters. Across the different models, responses to Physics PhD are generally most succinct 3839.5 characters (std of 2319.0). followed by Finance MBA at 4924.8 characters (std of 3383.0), 0). Chemistry PhD at 5609.3 characters (std of 2934.7) and finally Consulting MBA at 6961.8 0.8 characters (std of 2620.0).
20

---

Published as a conference paper at ICLR 2026
Table 5: Rubric taxonomy with sub-categories, and representative examples. Reasoning dominates with the majority of checks on logical validity and causal correctness, while Extraction emphasizes faithful and detailed retrieval, and Style focuses primarily on formatting and clarity.
Category
Concise Description
Representative Example
Extraction (Recall)
Evaluates whether the model rerrieves right information with correct coverage, granularity and fairhfulness.
"Identifies RTX as one four companies with the largest DoD obligations to the US goveroment 2022."
Reasoning
Logical Validity
Assesses whether reasoning steps are logi- cally sound, consistent, and free contra-
"Reasons that Anduril could have higher gross margins due to business model because software hos higher margins than hardware scale."
dictions.
Cassal/Mathenaiial Conrectness
Applies correct formulas, computations, and causal inferences.
"Calculates 2020 202-20244 defense spend CAGR by dividing 2024 spending by 2020 spending, raising to the power of 1/4, then subtracting "Shows steps for intermediate calculations when arriiing at quan- titative answers nuther quoting formalas." "Notes that strong donor support. robust risk framework, and proven impact are necessary to replicate the IFFlm model."
Completeness of Reasoning Generalizatin & Abstraction
Shows intermediate steps or justifications
rather than jumping to conclusion. Combines information or draws higher- level insights beyond literal recall.
Style Clarity & Readability
Organizes response clearly and makes it euy to follow.
"Displays answers and includes a concise summary tying together key findings: price change, EPS impact, valuation change." "Keep Task 2 within 160 words total."
Conciseness
Avoids verbosity and keeps the response tight and relevant.
Formatting
Correctly follows requested structure, nota- tion, and units, grammnr and spelling.
"Quotes all percentages to one decimal place."
Tone & Appropriatenes
Matches the expected tone (formal, nentral, explanatory).
"Presents analysis in strictured consulting framework formar."
Rubrics Each task comes with 15 to 59 individually-graddde criteria, with a mean of 30.6 and std of 9.9. Chemistry has the fewest criteria at average of 26.0 followed by Consulting (30.5), Finance (32.9) and Physics (33.0). In terms of rubric weight, around half (49.8%) fall under Major, with roughly quarter Critical (23.4%) and Minor (23.9%) and tiny fraction Additional (2.9%). Our analysis shows that in terms of criterion types, Reasoning dominates, covering 62.9 % of all items. This category primarily assesses whether the model's reasoning process is sound, complete, and coherent. Extraction (Recall) accounts for 14.1% of items, focusing whether the model retrieves the correct information with sufficient coverage and granularity. Style makes up the remaining 3.0%, evaluating how well responses are communicated in terms of clarity, structure, formatting, and tone. This suggests that ProfBench emphasizes on extracting and reasoning with professional knowledge, and less on the stylistic presentation of the response. To provide further insight, divide Reasoning ----------22--nstseee
and Style into sub-categories by collaborating interactively with Specifically, we prompted the model to come up with initial sub-category candidates based on set of randomly selected criteria within the Reasoning and Style categories. Then, the research team vetted the proposed candidates to make sure that the sub-categories are non-overlapping and at a suitable granularity. Once the research team is confident about the subcategories, we prompt the model to individually classify each criterion based on the concise definitions Tab. 5. Our results are in Fig. 2, with a representative sample for each sub-category in Tab. 5 Extraction (Recall) is kept as a single category since its core criterion is straightforward. Within Reasoning, most checks target logical validity (over 70%). with smaller but notable portion addressing mathematical or causal correctness (about 24%). ). Within Style, the majority of items focus on formatting (over 70%), followed by clarity and readability (about 19%)
Response Annotation Overall, o3 fulfills 51.6% of criteria as annotated by humans, while Grok4 fulfills 47.4% and R1-0528 fulfills 45.2%, indicating a relatively small gap between SOTA closed- source and open-source LLMs. Analyzing across domains, the three model responses on average fulfills only 39.1 1 % of criteria in Finance and 40.3% in Physics, suggesting that they are harder domains for SOTA LLMs. Conversely, Consulting criteria are fulfilled 56.5% and Chemistry criteria are fulfilled 59.4% of times, suggesting that they contain relatively easier tasks for SOTA LLMs.
Grounding Documents During our initial collection, annotators are requested to identify CSV or PDF documents from the public internet, which can support answering the task. However, this resulted in some documents being as long as 838 pages, with 200-400 page documents being relatively common. Each file has an average of 42.3 pages (std of 86.5), with the sum of files across each task
21

---

Published as a conference paper at ICLR 2026
reaching 141.7 pages (std of 193.5). After running some initial experiments, we found that such documents often cannot fit into the limited context windows of popular LLMs. For instance, OpenAI o3/o4-mini and Anthropic Claude-Sonnet 4 only supported up to 200k context tokens while many open-weights models (e.g. OPPT-oss-100B or DeepSeek V3.1) only support 128K context tokens. To ensure that ProfBench is compatible with these L.Mss, ask annotators to perform an additional step to truncate these documents by identify the most relevant information: Each pdf file is no longer than 20 pages and each csv file has less than 100 rows and 10 columns. Post-truncation, this was reduced to 7.37 pages per file (std of 3.10, max of 15) and 24.69 pages per task (std of 19.28, max of 85, min of 3). This truncation step will make tasks substantially easier but we believe this is a necessary step in order to make benchmarking many current models feusible. feasibbe. As the average context length increase in the future, we can use the original documents directly, which will serve as a better, realistic use of long context evaluation in the future.
In addition, some LLM providers such as OpenAl restrict the max size of each file to 10 MB, with total of no more than 32MB across all files for a request. Therefore, we implement additional requirements for annotators to restrict each file to 10 MB, up to 10 files per task and 30MB across all files for task. We also noticed that many popular providers' API (e.g. OpenAI and Google) can natively process PDF documents but not CSV documents, therefore requiring them to be passed as either plain text or processed through a code interpreter. Given that there were only 4 CSV files in total across ProfBench, we decided that the best workaround was to convert CSV documents into PDFs showing the same data, using pandas and pdfkit (JazzCore, 2025). These resulting PDFs contain table headers on every page, allowing rapid identification of relevant information. Each task contains an average of 3.35 files (std of 2.71, max of 10, min of 1). The size of each file is 0.813 MB (std of 1.196, max of 9.165, min of 0.017). The average size of all files in each task is 2.723 MB (std of 2.92, max of 12.555, min of 0.085).
E INFERENCE SETUP
Response Annotation during Data Collection Temperature were set at 0.2 (except o3, which doesn't allow setting temperature and uses medium effort), with access to a web search tool. o3 and Grok use a native search tool while DeepSeek model uses SearXNG (2025) and document upload support (pdf documents were uploaded natively while csv files were uploaded as plain text). Note that responses were generated with the un-truncated documents at this stage.
Report Generation Google models use default temperature of I as we observe that temperature 0 induces highly repetitive generations and Kimi-K2-091 uses the recommended inference temperature of 0.6, similar reasoning models. Note that these are slightly different from Response Annotation during Data Collection inference setup, since those were run by our vendor. Despite the relative affordability of the judge, we incur approximately $3.50 for judging criterion-fulfillment alone (outside of response generation). We use the native PDF documents processing capabilities for OpenAI and Google models, and use OpenRouter's default document processing for other models. For models using OpenRouter document processing, cost does not include approximately $1 charged by OpenRouter for PDF processing (when using file annotations 'cache' file processing).
Difference between LLM-Judge performance and Human Grading Performing of three models R1-0528- pred 46.8 vs human 46.1; Grok4 pred 50.5 vs human 51.8; o3 pred 53.5 vs human 52.7. Note these score are not comparable to those in $5 as they were generated by vendor as discussed above.
Judge Sensitivity to Response Generation Model We selected the three representative models (03, Grok4 and R1 -0528) intentionally as they represent the top-performing systems at the time of data collection (July 2025), spanning diverse providers and paradigms: two proprietary (o3 from OpenAl and Grok4 from xAI) and one open-weight (Rl 10528 from DeepSeek). Ideally, sensitivity analysis would involve re-am re-annotating responses from alternative reference sets with human experts, but this is resource-intensive due to the need for PhD/MBA-leve judgments on thousands of new response-criterion pairs. Nevertheless, our existing analyses (including sust-submission checks) demonstrate that the evaluations robust and unlikely to change materially with different references:
22

---

Published as a conference paper at ICLR 2026
As detailed in Section 4.2 and Table 2, top LLM-Judges (e.g.. Gemini-2.5- 78.2%, gpt-oss- 120b at 78.2%) exhibit extremely low bias-index values (less than 0.5% across these three reference models), meaning they align closely with human annotations regardless of the response's 's origin. This low spread suggests that good judges are impartial to the specific mix of proprietarylopen-veer weight styles in set. The impartiality of the LLMM judges in ProfBench is in large part contributed by the nature of the evaluation to decompose the LLM judgement into I 15-60 rubric criteria per prompt that we independently score, then weighted together. Our approach results in substantially lower bias (in both relative terms < 0.5% and absolute terms across 3 models < 1.0% for our official gpt-oss- 120B judge) compared to alternative approaches of applying LLM judges (e-g. picking preference among two responses (Dubois et al., 2023; Li et al., 2024; Zheng et al., 2023) or giving single scalar score for each response (Zheng et al., 2023)) where biases can be as high as 30 75%.
To directly test the null hypothesis that judge evaluations do not depend on the specific reference model identities, we conduct a permutation test (10,000 iterations) on judge evaluation. Specifically, we fix the judge predictions and human labels, randomly shuffle the model labels assigned to responses, and recompute the Bias Index under each shuffled labeling. For our final judge model, gpt-oss-120b, the observed Bias Index falls well within the null distribution, with two-sided p-value 0.56. This confirms there's 's no evidence that the evaluation depends on the specific reference model; the low bias and high agreement we report are genuine properties of the judges, rather than the artifacts of our particular reference set.
Ablation
Cost excludes search query costs, which is is estimated be around $6.40.
23

---

Published as a conference paper at ICLR 2026
F OPTIMAL PERFORMANCE ON PROFBENCH AT EACH PRICE-POINT
70
63.4
gpt-S 60.3
60
gemini-2 22222-11aa 5-flash 56.8
57.
56
58
$5.8
gptrasa
66a.5
53.8
gemini-2. S-pro
53191,4
gpl-ore-20b
51.3
qwend
52.7
ProfBench
8
46.6
39
56.8
36.4
27.6
100
Cost (in $)
Figure 4: Optimal ProiBench performance at each price-point. OpenAl models are the Pareto Frontier at each price-point, likely because first-mover advantages in rubric-style data such as HealthBench (Arora al., 2025) and PaperBench (Starace et al., 2025). Gemini-2.5 and Qwen3- Instruct-2507 models are close to the Pareto Frontier.
G FORMULATING OPTIMAL ALLOCATION PROBLEM
As exemplified in Fig. 3, we observe that the variance of model performance differs significantly by task. Motivated by this finding, we propose to reduce the variance of the estimated model performance by dynamically allocating different number of responses each task ideally allocating more for tasks with high variance, and less for tasks with low variance.
More formally, let n, denote the number generations allocated for task i. Our goal is minimize the variance of the overall performance S across N tasks, while satisfying the total number of generations Σ, n, to match fixed budget B. First, the overall performance is defined as:
S Σ 8;
where 8; the mean score over nı responses we generate for Assuming i.i.d. sampling of the responses and the independence between N tasks, compute the variance of S as:
0²( = N
m
where is the variance associated with task i, which we estimate by generating 16 rollouts per prompt across 4 representative models and aggregating their variance. Now, minimizing the variance o²( is equivalent minimizing E. u subject to: n.
N Σ
n, E
Z n.
f Vi
Given the small size of the budget and number tasks, we directly solve the objective via dynamic programming. The results of this allocation shown in Fig. 5. We follow the allocation scheme to
24

---

Published as a conference paper at ICLR 2026
sample responses from 4 representative models Gemini-2.5-Pro Gemini-2.55 (Thinking), o3, and o4-mini and report their average standard deviation of overall performance.
As expected, the optimal allocation consistently achieves smaller variance compared to heuristic baselines, and we set B to be 160 (i.e. average n, = 4) with N = 40, reducing down the stan- dard deviation to only 50% of the naive point estimate.
1.39
Standard Doviation
Our results, illustrated in Fig. 5, compare ap- proaches: uniform allocation (generating a fixed number of responses per task) with mean and median-based aggregation of the scores, and op- timal allocation, as well as a heuristic baseline: min max exclusion (uniform allocation followed by exclusion of min max score responses from each task during aggregation). Overall, we find that generating multiple response per task can ef- fectively reduce standard deviation, substantially below the level of naive point estimate (budget- = 1). Furthermore, optimal allocation consistently reduce the performance variance compared to the alternative methods, consistently across all budget scales.
Uniform allocation (mean)
allocation (median)
Max exclusion
Optimal allocation
2
8
Budgel (Average à of samples per ta04))
Figure 5: Standard deviation of overall perfor- mance using multiple samples per task. Optimal allocation of samples consistently reduce the vari- ance across all budget levels.
H FUTURE DIRECTIONS
Tool Use The main tool we focus in Section 5 is the document-ingestion tool, which allows LLMs to utilize information from PDF grounding documents that human expert annotations identify. We also conducted smaller-scale experiments with other tools such as the search tool in Section
6. We decided to de- ce-prioritize further explorations on the search tool because the implementations of LLM search tool use are currently non-standardized. For instance, proprietary providers (e.g.
OpenAI, Google and Anthropic) typically have their first-party search tool while open-weight models have depend third-party search tools (e.g. Exa AT). Such differences mean that when model using search-tool is measured, it becomes challenging to understand whether model performance can be accounted for by search-tool capabilities, model capabilities or the integration of search-tools with models. In addition, we also considered and made initial attempts with other tools (e.g. code interpreter environment- which integrates spreadsheet tool and calculator tool), since we initially had a small portion of documents in CSV format. However, at the time of our experiments, only OpenAl had code interpreter tool integrated into publicly accessible API. Using such tools makes it challenging to fairly compare these models with vast number of other models that do not use these tools. As the ecosystem for various tools matures, we certainly plan to incorporate further evaluations using such tools.
Multimodality We purposefully designed ProfBench to be text-only (i.e. no image or audio) in order support recent popular open-weight models since many of them text-only models (e.g. DeepSeek V3.1 / Kimi K2/ / GPT-OSS). This ensures the open-source community can readily build upon these models and utilize ProfBench measure and improve model capabilities relevant to LLM-judge and long report generation. As the open- weight model ecosystem develops with more models supporting multi-modee capabilities, followup works can build upon the foundations laid by ProfBench.
Other Domains We select the domains of Chemistry Research PhD, Physics Research PhD, Consulting MBA and Finance MBA based on three factors: Whether other works have covered such domains previously, b. Real-world usefulness of these domains and c. Whether we could recruit suitable annotators to robustly annotate for these fields.
25

---

Published as a conference paper at ICLR 2026
For the first factor, focused on domains that have not been previously worked by earlier works (Arora et al., 2025; Starace et al., 2025; Guha et al., 2023; Zhou et al., 2025) in order to show fresh capabilities in new settings, which we believe to be more helpful to the community.
For the second factor, we wanted to measure LLM capabilities that real-world users care about. For instance, many existing advanced Chemistry and Physies evaluations (e.g. GPQA (Rein et al., 2023) and HLE (Phan et al., 2025)) mostly measure exam-style problems that can be answered in Multiple-choice or short span format. For many real-world physical scientists, answering these exam questions is less helpful in their work compared to assisting them in their research planning, as covered in ProfBench. In addition, we want focus on some enterprise use-cases and we found that capabilities relevant to Consulting MBA and Finance MBA are relevant to many white-collar professions- meaning that measuring L.M capabilities in these domains can proxy their usefulness for many real-world professional users of AL.
For the third factor, we were constrained to annotators that we could feasibly recruit for to do these tasks within our research timeframe and budget. Given these constraints, we opted to focus on the set of 4 domains, each with a useful number of tasks (20 tasks domain, each taking 10-20 hours of expert annotations) rather than covering a large set of domains but with a small number of tasks in each (e.g. DeepResearch Bench (Du et al., 2025) covers 22 domains but only has an average of 2.3 tasks in each domain in English). We do not have near-term plans to expand the number of domains for newly-annotated tasks although it is possible in the medium or long term. In the short term, we recommend researchers to utilize ProfBench along other benchmarks (Arora al., 2025; Starace et al., 2025; Guha et al., 2023; Zhou et al., 2025) where appropriate for their research goals.
Measuring creativity in Professional Tasks We believe that aspects such as originality and creativity are hard objectively measure (even for humans, they tend to be very subjective). We specifically ask annotators to phrase prompts in ways that do not necessarily require creative answers (e-g. making calculations correct in multi-step chemical equation). As models gradually become performant in objective tasks in ProfBench, we agree that the next step of evaluation might involve measuring creativity of such professional tasks (e.g. finding creative chemical experimental approaches or structuring novel financial deals), which can build upon ProfBench and prior works on measuring creativity (Lu et al., 2025).
Risk of Saturation We believe that the current difficulty (50% +/- 15% for most models) is useful for many model trainers to get effective signals and improve. To reach this target difficulty, we focus on the easiest setup for report generation (Section 5), where models are given grounding documents that were truncated by human experts (to max. 20 pages each) to make it easy for LLMs to locate where the relevant information challenging and avoid saturation. One approach is to use the original documents (up to 800+ pages long each) to better test their long context processing capabilities as many models get more than 128k context window size.
As models get better, we can adapt the benchmark to make it more
Furthermore, we can even remove the grounding documents altogether and force models to either search for relevant documents or answer prompts through parametric knowledge alone. An ablation study that we conducted in Section 6 shows that such settings cause model performance to drop by 4.9% to 11.9%. This means that once the current version of ProfBench becomes too easy for the strongest LLMs, we can easily flip a switch to make ProfBench substantially more challenging and future-promt proof. For instance, the top performing LLM only reaches 52% when forced to rely on parametric knowledge.
I REVIEW PROCESS
Review refers to the process in which the prompt and rubrics were iteratively improved. Below is the detailed description accompanied by a running example from Physies PhD) pool::
1. Annotator A submits an initial prompt: Consider five-level "quadpod" atom with four long-lived states [2), [3), [4) and one excited state (e). In the rotating-wave frame (h 1);
(2,,113
26

---

Published as a conference paper at ICLR 2026
2. Reviewer B recommends Major prompt edits: The prompt is mathematically precise, defines all relevant symbols, and fixes a gauge, making the problem well-posed and non open-ended. It clearly states the task of computing the 3 x 3 non-Abelian Berry holonomy for a specified loop. For improved clarity and rubric alignment, consider: Stating explicitly that the holonomy is to be reported in the ordered dark basis (Id1), ) Idz), (da)) evaluated at (a, 7) = (0, 0). Noting that G is interpreted modulo 2 due to the periodic nature of 7. Indicating whether the expected answer should be a fully simplified closed-form matrix in terms of f, Φ, X, G, or if an ordered product form is acceptable. Optionally mentioning that the resulting holonomy lies in SU(3) for this loop.
3. Annotator A makes changes based on Reviewer B's feedback : Thank you for the detailed feedback! I have incorporated all the changes suggested by the reviewer. have also changed scaffoldings and split multistep tasks into separate tasks. Hopefully the quality of the prompt has been improved.
4. Reviewer B approves prompt
5. Annotator A submits an initial rubrics States the explicit orthonormal degenerate dark-state basis
6. Reviewer B recommends minor rubric edits: Overall, the rubric covers the full logi- cal progression from defining the dark-state basis and gauge potentials to computing the holonomy. However, several criteria reference variables, definitions, or results from earlier criteria without reintroducing them. Each criterion should be self- cofffcontaiee,, including all necessary definitions and expressions, so the model does not need to look back. I included my revisions for each section and checked them as well.
7. Annotator A makes changes to rubries States the explicit orthonormal degenerate dark-state basis
8. Reviewer B approves rubrics
9. Annotator A annotates 3 model responses based on rubrics
10. Reviewer B recommends minor changes to model response annotations
11. Annotator A updates annotation 3 model responses based on rubrics
12. Reviewer B approves response annotations
Annotator A is single annotator who developed the prompt, rubrics and annotated model responses against the rubrics, as supported by Reviewer B. Note that Reviewer B is typically one reviewer but can at times be different reviewers at each stage if the original reviewer is unavailable since each task takes 10-20 hours across the three stages. Steps 3 to 4/7 to 8 11 to 12 can be repeated a few times until the reviewer is satisfied. During the rubric review process, both the annotator and reviewer are instructed to check that the rubrics fulfil a number of requirements (e.g. each can be graded in binary fashion, should not overlap with other criteria and must collectively cover all important aspects of good response). There is a longer list of requirements that we initially planned to include into the Appendix. However, some of this information constitutes confidential business-sensitive information for our data vendor and hence we were unable to include it into the submission.
SENSITIVITY AND QUALITATIVE ANALYSES OF LLM JUDGES
Sensitivity Analysis The ideal way to stress-test any bias metric is to compute it over a large and diverse set of reference models. In practice, however, computing the Bias-Index (or any similar fairness metric that relies on human expert ground-truth) inherently constrained by the availability of model-generated responses but also the expert-annotated judgements for all model generations.
In ProfBench, expert annotators provided binary fulfillment judgments for responses generated by the three models that, at the time of data collection (July 2025), represented the strongest proprictary and open-weight systems on these professional tasks (OpenAI o3, xAI Grok4, and DeepSeek R1- 0528). Obtaining comparable high-quality human expert judgments for many additional models would require re-rumning the full annotation pipeline on thousands new generations, which is prohibitively expensive and time-consuming given the PhD/MBA-level expertise required.
27

---

Published as a conference paper at ICLR 2026
Nevertheless, believe the current three-model setup provides sufficiently meaningful and interpretable signal of bias for the following reasons:
The three reference systems come from three distinct providers/orgamizztions with different training data, objectives, and output styles (OpenAI, xAI, DeepSeek), providing reasonable coverage of the proprietary vs. open-weight divide that matters most for faimess concerns in the community. As shown in Table 2, the top-performing judges (gpt-oss-122) 120b and gemini-2.5 differences of less than 1%, suggesting no significant preference for any single provider.
exhibit max-min
That said, we further propose complementing Bias index with two additional metrics:
Standard deviation of biases: the standard deviation of biases collected across models. This is less sensitive to an extreme outlier than max- min while penalizing systematic spread.
Provider-level faireess gap: We compute the gap between average bias toward the open-weight model (RI) versus the average bias toward the two proprietary models (o3, Grok4).
Table 6: Comparison of Standard deviation of biases and Provider-level fairness gap
Model
Standard Deviation of Biases Provider-level Fairness Gap
gpt-4.1
0.37%
0,65%
gemini-2.5-fRe
0.12%
0.25%
gpt-5
0.51 %
0.30%
03
0.65 %
0.89%
gpt-oss- 120b
0.22%
0.15%
deepseek-v3.1
0,29%
0.45%
The summary of results with the new metrics are shown in Tab. 6. The top judges (as identified in our original analysis) achieve consistently small biases in these new setups, with gpt-oss- 120b achieving 0.22% in standard deviation of biases and 0. 0.15% in fairness gap. We believe these results strengthen the fairness analysis while remaining fully grounded in the high-quality expert annotations.
Performance across categories The overall FI score only captures misalignment between the LLM and human experts. In addition to this, we also include the category-specifiie F1 (Extraction, Reasoning and Style) in Table 2. Across these categories, models generally perform better Extraction, followed by Reasoning and finally Style. This suggests that many LLM-Judges are weaker in judging whether responses fulfil stylistic requirements (e.g. writing investment memo in specific format; returning chemistry calculations to the requested precision) since they are sometimes more nuances and could involve checking different parts of the response (e.g. all concentration values to two decimal places) rather than Extraction tasks that are generally more focused on specific segment within the response.
Qualitative Analysis We also did qualitative analysis of some reasoning traces, where the human annotators disagree with the optimal LLM judge (GPT-OSS-120B, high reasoning effort for Chemistry/Pyysie//tyyl. low otherwise).
Here are some patterns we observed that explains the shortcomings of the LLM--udge:
1. LLM Judge sometimes does not recognize the equivalence of certain statements.
Criterion:
Identifies the mechanism of the ylide formation SN2 substitution.
Explanation: In this case, the Bimolecular (i.e., two reactants) Nucleophilic substitu- tion (SN2) reaction produces both the ylide and the phosphonium salt, but the LLM judge wrongly focuses only on the phosphonium salt.
Reasoning Trace: The user asks: "Evaluate whether the response above satisfies this criterion: Identifies the mechanism of the ylide formation as SN2 substitution. Only answer Yes or No." We need to examine the provided response (the synthesis plan) and see if it identifies the mechanism of the ylide formation as SN2 substitution.
The ylide formation step is the formation of the ylide from the phosphonium salt and n- BuLi. The ylide formation is actually deprotonation of the phosphonium salt by n-Bul.i, which is not SN2 substitution but base deprotonation.
28

---

Published as a conference paper at ICLR 2026
The response includes step "Preparation of Isoprpyylripheey//hhophhosr Bromide," which is an SN2 substitution (PPhy attacks alkyl bromide). However, the question specifically asks about the ylide formation.
Step 3: "Preparation of hww.ay/lripphee/yhopphooiii Bromide: Mechanism:
SN2 attack by P on alkyl halide." This is formation of the phosphonium salt, not the ylide.
Step 4: "Wittig Reaction to Form Dienophile: Mechanism: (a) Deprotonation by does not state that the
n-BuLi forms ylide. (b) Ylide attacks carbonyl...
ylide formation is SN2 substitution; it is deprotonation.
Thus, the response does not identify the mechanism of ylide formation SN2 substitu- tion. It identifies the mechanism of phosphonium salt formation as SN2, not the ylide formation.
Answer: No.
2. LLM judge sometimes mistakes implied statements for explicit statements.
Criterion:
Defines the theoretical moles of hydrazine as
n
R4 Number of electrons
Explanation:
This was not explicitly mentioned in the response but utilized as an assumption in subsequent calculations.
Reasoning Trace: We need to parse the question. The response includes calculation:
"The theoretical production rate assumes 100% Faradaic efficiency. Total moles of electrons:
Q
0.23768
2.4633 10 mol.
F
96485
Theoretical moles of hydrazine:
NNHH. n
2.4633
10-6
6.1583 10-7 mol.
"
This effectively defines the theoretical moles of hydrazine as n(N2H4) = na- /4, satisfying the criterion.
Answer: Yes.
3. LLM judge sometimes fails to integrate information from different parts of the re- sponse.
Proposes a substitution reaction with isopropanol and phosphorus tribro- mide in diethyl ether.
Criterion:
Explanation: The response contains two parts: (a) "All steps are designed to be carried under standard laboratory conditions. with diethyl ether as the primary solvent" and (b) "Preparation of isopropyl bromide (iPrBr): Mix isopropanol (27 0.45 mol) with phosphorus tribromide (33 g. 0.12 mol) a round-bottom flask." The LLM judge missed the first part.
Reasoning Trace: The response proposes a substitution reaction between isopropanol and PBr;, but does not specify diethyl ether as the solvent.
The criterion requires the reaction be specifically in diethyl ether.
Answer: No.
Overall
While these shortcomings highlight that LLM-Judges are not perfect oracles, the errors tend to be systematic rather than random and therefore may have less effect on relative comparisons between models opposed the absolute value. As noted in lines 331 to 332, we only 0.7 to 1.3% performance across 3 models that were annotated
gap between sudge-predicted and by our L.LM judge and professional human annotators.
29

---

Published as a conference paper at ICLR 2026
K
REPORT-GENERATIN PERFORMANCE ACROSS VARIOUS DOMAINS
Our design philosophy for ProfBench was to faithfully reflect the authentic difficulty and nature of real-world professional work in each field, rather than to artificially equalize difficulty through task selection, rubric pruning, or post-hoc reweighting. Forced equalization would risk several undesirable side effects: (i) discarding the hardest (and often most diagnostically valuable) tasks in a domain, (ii) inflating scores in inherently more complex scientific domains by relaxing rubrics, or (iii) introducing arbitrary weighting schemes that lack objective grounding and reduce transparency.
Henceforth we did not make any specific intervention on the collected data, except that we match the number of tasks in each domain to be equal. At the same time, we report the comprehensive domain- wise / reasoning type-wise performance of all models, given that the overall score like any other aggregation methods inevitably distills useful information about subcategory-Ievel performances. Our design principle directly aligns with subcategories in prior benchmarks that focus professional tasks, such as conversation theme-wiss wise analysis in HealthBench (Arora et al., 2025) and machine- learning problem domains in PaperBench (Starace et al., 2025).
Nonetheless, we have additionally examined whether the observed absolute difficulty gaps actually distort the ranking or relative capability assessment of models when scores aggregated. Two sets of additional analyses (conducted post-submission) show that our aggregation scheme does not harm the validity of cross-model comparisons: For each domain, normalize all reasoning model scores by dividing with the best model score in that domain (i.e., setting the best model score = 1.0). We then compute the deviation of these normalized scores separately for each domain. In addition, also analyze the rank correlation between the performance in each domain and the overall aggregated score.
Table 7: Standard deviation and Spearman's rank correlation across domains.
Domain
Physics
Chemistry
Finance
Consulting
Standard Deviation
0.11
0.10
0.11
0.09
Spearman's 's Rank Correlation
0.85
0.79
0.91
0.89
The resulting variances are consistently similar to each other across all 4 domains, which indicates that regardless of absolute difficulty, the performance hierarchy and gaps between strong and weak models remain stable across all four domains. In all domains, the correlation marks close to beyond 0.8, indicating very strong correlation. This directly evidences that the overall aggregated score faithfully reflects model capabilities in all domains, rather than systematically penalizing a specific subset. Taken together, these results show that our domain-based categorization and aggregation is diagnostically valid.
30


Reviews:
[{"review_id": "TtGB0FtGOm", "reviewer": "Reviewer_2ui6", "summary": "The paper introduces ProfBench, a rubric-guided benchmark of 80 expert-curated tasks spanning four domains, Chemistry, Physics, Finance, and Consulting, with more than 7000 human-written response-criterion pairs. Domain experts create the tasks as well as weighted task-specific rubrics and label three “seed” model responses (o3, Grok4, R1-0528) per criterion as Yes/No, forming the ground truth for calibrating an LLM judge. The quality of the judge is measured by a metric that takes into account also the bias towards the same model family. Each task has the structure of report-writing with grounding documents, and removing the documents leads to worsening performance. The best model reaches an average score of 65.9%, with lowest performance on Physics (49.3%) and best performance on Consulting (80%).", "questions": "- What was the rationale for choosing the four domains? Do you plan to add others?\n- Are you planning on performing evaluations with models that can use tools that are relevant for the benchmark tasks?\n- The LLM judge evaluation depends on three seed models. I wonder whether the judge evaluation can change when picking different models. Do you have any thoughts on that?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The tasks are realistic, complex, with grounding documents and created by domain experts, with reviewer feedback.\n- The LLM judges are evaluated in a clear way that takes also into account the bias towards the same model family.\n- Separate re-annotations show high inter-annotator agreement. Moreover, the LLM judge is highly reliable, with a tiny difference with human-annotated scores.", "weaknesses": "- GPT-5 already achieves a high score on Consulting (~80%), while Physics lags at ~49%. This suggests that one domain might saturate sooner than the other.\n- The set-up is text-only, even if tool use might be helpful for some tasks, e.g. calculators, spreadsheets, code etc.\n- Despite current difficulty, the text-only format may offer limited room for improvement as models become more capable.\n- Domain coverage is narrow, covering only two science and two business domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Tq5VE7L8E", "reviewer": "Reviewer_yfAJ", "summary": "The paper proposes a benchmark including response-criterion pairs to evaluate professional knowledge across multiple fields. It introduces an efficient LLM-as-judge evaluation framework that mitigates self-enhancement bias. The authors evaluate current LLM performance on both criterion fulfillment classification and response generation for these challenging tasks.", "questions": "1. While using LLM to judge the criterion-fulfillment, is there any way to extract more information from the LLM performance for further failure mode analysis? Reasoning models are allowed to generate some inference steps prior to binary predictions. How might these reasoning traces be utilized to analyze misalignment between LLM predictions and human annotations?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The benchmark covers multiple scientific domains and evaluates knowledge storage and complex reasoning capabilities. Its expert-designed criteria facilitate precise, granular assessment of LLM performance on challenging tasks.\n2. The paper assesses a wide range of LLMs to provide comprehensive performance benchmarks. The experimental design encompasses comparisons across model accessibility (open-source and closed-source), scale, and reasoning capabilities.\n3. The high-quality annotators group and reliable rubric creation pipeline guarantee the dataset quality.", "weaknesses": "1. In Section 4, the LLM-as-judge is used as a binary classifier, with performance evaluated by F1 score. The target LLM is used to identify whether the provided criterion fulfills all the requirements to check the quality of the response. However, for such complex tasks, the F1 score only captures misalignment between the LLM and human experts. It does not reveal the LLM's internal understanding of the task or identify specific weaknesses.\n2. In the rubric creation process from Section 3, the criteria creation and review stages are not described in detail. Both stages appear heavily dependent on annotator judgment, and it remains unclear how each proposed criterion contributes to the granular assessment of response quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "urC43VYNJ1", "reviewer": "Reviewer_knFu", "summary": "The paper introduces a new benchmark (ProfBench) to test LLMs on PhD/MBA-level tasks that require specialized knowledge. Unlike prior benchmarks focusing on quickly verifiable problems, ProfBench focuses on open-ended tasks like writing reports that require expertise. The paper’s contributions include: (1) establishing a multi-domain, expert-annotated rubric benchmark; (2) assessing LLMs both as report generators and as judges; (3) and proposing techniques for LLM-based grading that aims to reduce evaluation cost and bias. The experiments demonstrate that LLM judges can grade responses with reasonable agreement to human experts.", "questions": "- Is there a risk of saturation with the best model scoring over 65%? Is the benchmark future-proof? Should there be a “hard” subset of prompts that are more adversarial in nature?\n\n-  Could the authors clarify their rationale for selecting the four domains for ProfBench? For example, was the health or legal domains excluded due to difficulty in obtaining annotations or because existing benchmarks like HealthBench or LegalBench already cover this ground? If the goal is to provide a broad measure of professional knowledge, adding other professional domains seems valuable.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A major strength is the exploration of LLMs as automated judges. Building on work in rubric-evaluation, the authors propose a framework to have LLMs determine if a given response satisfies each expert criterion. The framework aims to reduce self-enhancement bias (i.e., where LLM judges would favor responses from the same model or provider), as well as API costs.\n- ProfBench benchmarks LLMs as report generators in scenarios that mirror actual professional workflows, requiring multi-step reasoning and synthesizing information from multiple reference documents.\n- Ablation experiments demonstrate the importance of reference documents for model performance.", "weaknesses": "- The benchmark’s scoring relies on LLM judges, and while the authors do measure agreement with human annotators, it’s shown that the best judge isn’t near perfect (<80% Macro-F1 overall). There’re risks that LLM-judges might miss nuanced criteria fulfillment or penalize creative answers. The paper doesn’t deeply discuss failure modes of the LLM-judge.\n- ProfBench covers only four domains, leaving out several important domains of professional reasoning — notably, the legal, health, and engineering domains. \n- Relatedly, the paper omits some relevant benchmarks involving professional tasks in the legal and engineering domains, which are not covered by the benchmark:\n\n      Chilton, A., Guha, N., Nyarko, J., Ho, D., Ré, C., Narayana, A., ... & Peters, A. (2023). LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. Advances in Neural Information Processing Systems, 36.\n\n      Zhou, X., Wang, X., He, Y., Wu, Y., Zou, R., Cheng, Y., ... & Zhao, J. (2025). EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving. arXiv preprint arXiv:2509.17677.\n\n- The paper does not discuss any strategy for preventing test data leakage or overfitting. In contrast, some prior benchmarks like MMLU-Pro introduced hard, out-of-distribution questions to stay ahead of models.\n\n- The paper’s evaluation doesn’t include an overall quality judgment beyond summing criteria. While rubric scoring is objective, it might not capture important aspects of professional work such (e.g., originality or creativity) that are hard to enumerate. This is a philosophical weakness of rubric-based evaluation in general, and acknowledging this potential limitation would improve the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hGwtPC3kDF", "reviewer": "Reviewer_pJFi", "summary": "This paper introduces ProfBench, a high-quality evaluation benchmark spanning multiple professional domains. It contains 80 tasks and more than 7,000 human-constructed response–criterion pairs, developed entirely by domain experts with PhD or MBA degrees and without any LLM assistance, ensuring authenticity and professional rigor. The authors further propose a rubric-based LLM-as-a-Judge evaluation paradigm and design three metrics to comprehensively assess assessment consistency, fairness, and efficiency. A systematic evaluation of over 40 open-source and closed-source models investigates the influence of different reasoning mechanisms, model sizes, and response lengths, and presents strategies to reduce evaluation cost and bias. Experimental results show that even the strongest closed-source model (GPT-5-high) reaches only 65.9% on this benchmark, highlighting the professional difficulty and challenge posed by the tasks.", "questions": "1. You mentioned that the annotation process underwent multiple rounds of review and reported a relatively high consistency metric. When discrepancies arise among annotators, is there a fixed adjudication process or arbitration mechanism in place? If yes, could you supplement typical cases in the appendix to help readers understand how to handle criteria with strong subjectivity?\n2. Is this metric stable when the number or type of reference models changes? Have sensitivity analyses been conducted, or has the consistency of calculating the bias-index in a larger model pool been tested?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "In the data annotation phase, the paper adopts a rigorous expert participation mechanism. A total of 38 professionals with PhD or MBA backgrounds were recruited to design tasks and formulate scoring criteria. Multiple rounds of review and consistency verification were conducted to ensure the reliability of annotations. This process guarantees the high quality of the dataset in terms of knowledge depth and annotation accuracy.\n\nThe paper systematically compares over 40 types of mainstream models, covering dimensions such as closed-source vs. open-source, different sizes, and \"thinking\" settings. It also analyzes the relationships between model performance, bias, output length, and reasoning costs. The overall experiments are comprehensive, and the conclusions are credible.\n\nDuring the evaluation process, the authors systematically studied the performance and cost differences of different judge models, and proposed an optimal sample allocation method and a low-cost evaluation scheme. This strategy can significantly reduce evaluation costs while maintaining high consistency, which provides valuable insights for large-scale evaluation practices.\n\nProfBench covers four professional domains: Physics, Chemistry, Finance, and Consulting. With diverse task types, it can comprehensively reflect the generation and judgment capabilities of large language models in professional scenarios.", "weaknesses": "1. The paper describes the annotation process and consistency metrics in detail, but lacks qualitative demonstration of controversial samples and explanations of the adjudication mechanism. Given the subjectivity of rubric-based evaluation, it is recommended that the authors supplement several typical cases to illustrate the judgment differences among different annotators and the final adjudication process, thereby enhancing interpretability and transparency.\n2. The current definition of the Bias-Index relies on a limited set of reference models, and directly subtracting the Bias-Index from the Macro-F1 score to obtain the \"Overall\" indicator may lead to issues of dimension inconsistency and sensitivity in multi-model scenarios. It is suggested that the authors verify the robustness of this indicator on a larger model set and supplement other fairness metrics.\n3. The paper’s results show obvious variations in task difficulty across different domains, yet the authors do not explain whether task balancing or weighting was performed. When comparing across domains, comprehensive scores may be affected by the distribution of domain samples, thereby impacting overall fairness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)