ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists

## Abstract

Abstract This paper introduces ExpertLongBench , an expert-level benchmark containing 11 tasks from 9 domains that reflect realistic expert workflows and applications.
Beyond question answering, the application-driven tasks in ExpertLongBench demand long-form outputs that can exceed 5,000 tokens and strict adherence to domain-specific requirements.
Notably, each task in ExpertLongBench includes a rubric, designed or validated by domain experts, to specify task requirements and guide output evaluation.
Furthermore, we propose Clear , an evaluation framework that supports accurate evaluation of long-form model outputs in our benchmark.
To achieve fine-grained, expert-aligned evaluation, Clear derives checklists from both model outputs and references by extracting information corresponding to items in the task-specific rubric.
Checklist items of model outputs are then compared with corresponding items of reference outputs to assess their correctness, enabling grounded evaluation.
We benchmark 13 popular large language models (LLMs) and analyze components in Clear , showing that
(1) existing LLMs, with the top performer Gemini-2.5-Pro achieving only a 33.4
F1 score, require significant improvement for expert-level tasks;
(2) models can generate content corresponding to the required aspects, but far from correct;
and
(3) accurate checklist extraction and comparison in Clear can be achieved by open-weight models for more scalable, reproducible, and low-cost usage.

## 1 Introduction

Large language models (LLMs) have been integrated into applications that demand domain-specific expertise, such as student tutoring (Sonkar et al., 2024), legal case summarization (Siino et al., 2025), and medical diagnosis (McDuff et al., 2025).
These expert-level tasks pose greater challenges because they require specialized knowledge and strict adherence to domain standards, where human typically acquires these through advanced education and professional training.
Furthermore, many real-world expert-level tasks require understanding lengthy inputs (Wang and Brorsson, 2025) and generating complex and nuanced long-form outputs (Shen et al., 2022; Cascella et al., 2023).

Existing expert-level benchmarks such as MMLU (Hendrycks et al., 2020) and GPQA (Rein et al., 2024) prioritize ease of evaluation of multiple-choice or short-form answers at the expense of alignment with realistic expert applications.
Though ExpertQA (Malaviya et al., 2023) assesses longer-form, domain-specific outputs, it remains centered on QA scenarios rather than end-to-end expert workflows.
In addition to task scope, a key limitation is the lack of evaluation methods tailored to the specific requirements of each task.
WildBench (Lin et al., 2024) creates checklists to evaluate model responses to user queries, yet their model-generated checklists fall short in capturing domain-specific requirements. Finally, a broader challenge is the lack of references in expert-level benchmarks with long-form responses, leading to ungrounded evaluations and missing recall estimation.

To address these gaps, this paper first introduces ExpertLongBench, a multi-disciplinary, expert-level benchmark with tasks that require long-form outputs.
ExpertLongBench comprises 11 tasks across 9 domains,
with 1050 samples in total ([Tab.˜1](https://arxiv.org/html/2506.01241v3#S2.T1)).
Our benchmark enables more comprehensive assessment of a model’s problem-solving capabilities by including domain-specific end-to-end applications, such as drafting legal briefs or clinical notes, with maximum input and output lengths exceeding 200K and 5K tokens, significantly longer than existing datasets (see comparison in Appendix [H](https://arxiv.org/html/2506.01241v3#A8)).
The completion of these tasks is time-consuming for experts.
For example, it takes over 10 hours for a proficient legal practitioner to summarize a complex legal case, as they need to follow the docket and review tens to hundreds of court filings (Shen et al., 2022).
Moreover, ExpertLongBench provides expert-written references per task to support grounded evaluation.

Figure: Figure 1: Pipeline of Clear. The example shown is from task T1: multi-document legal case summarization. The checklist mapper takes as input the model output (or human-written reference) and extracts checklist items according to the rubric. Checklists of the model output and the reference are compared at the item level, and the results are subsequently aggregated into the final scores.
Refer to caption: x2.png

To ensure more accurate evaluation of model performance on expert-level tasks, we further propose Clear—CheckList-based Expert-level Assessment with Rubric
([Fig.˜1](https://arxiv.org/html/2506.01241v3#S1.F1)).
We collaborate with domain experts to develop and validate a detailed evaluation rubric for each task.
Unlike subjective criteria, such as “usefulness” (Malaviya et al., 2023) and “helpful information” (Lin et al., 2024), our rubric specifies the essential elements required in the output for each application, enabling fine-grained evaluation that closely align with domain-specific requirements.
For example, a well-written summary of legal case documents intended for lawyers must accurately identify the cause of action, the relevant statutory or constitutional basis, and the remedy sought.
Based on the rubric, Clear breaks down model responses and references into itemized checklists.
For each item, our framework compares the corresponding information extracted from the model’s output against that of the reference, thereby ensuring a grounded and objective checklist-based assessment.
(^1^11The benchmark leaderboard and links to code and public data, along with the necessary evaluation resources: [https://huggingface.co/spaces/launch/ExpertLongBench](https://huggingface.co/spaces/launch/ExpertLongBench).)

We evaluate the performance of 13 frontier LLMs on ExpertLongBench, including both open-weight and proprietary model families. Our findings are:
$\bullet$ The best model achieves an average F1 score of 33.4 across all tasks, revealing that the end-to-end expert-level tasks in ExpertLongBench present significant challenges for existing LLMs.
$\bullet$ Despite a low overall output quality, models can generate content that matches over 67% of the aspects required in the checklist, suggesting a risk of models producing content that appears expert-aligned but is incorrect and thus potentially misleading.
$\bullet$ Clear can be configured with open-weight models for cost-effective and scalable benchmarking, as evidenced by a Pearson correlation of 0.88 between the scores assigned by Qwen2.5-72B and GPT-4o for checklist-based evaluation.
To ensure ExpertLongBench remains relevant and reliable as LLM capabilities evolve, we commit to continuous development with an interdisciplinary community, incorporating new tasks, high-quality data, and domain-specific evaluation insights.

## 2 Related Work

##### Evaluating LLMs on Expert-Level Knowledge.

Previous studies have assessed domain-specific knowledge using multiple-choice questions (e.g., MMLU (Hendrycks et al., 2020), GPQA (Rein et al., 2024)) or short-answer questions (e.g., AGIEval (Zhong et al., 2023), SciEval (Sun et al., 2024), HLE (Phan et al., 2025), OlympiadBench (He et al., 2024)).
Their scope remains constrained to exam-style questions, failing to reflect the long-form nature of realistic domain-specific tasks.
ExpertQA curates open-ended questions written by domain experts (Malaviya et al., 2023).
However, the answers average only about 100 words and target information seeking expert knowledge rather than the full scope of an end-to-end workflow.
Though DOLOMITES (Malaviya et al., 2025) and ResearchQA (Yifei et al., 2025) target domain-specific long-form generation tasks, they only cover methodological writing and research question answering problems.

ExpertLongBench fills these gaps by introducing diverse, expert-level, long-form tasks that align with complete workflows and features lengthy inputs that represent practical uses;
and providing rubric-based evaluation for fine-grained, domain-specific assessment. Further discussions of differences from existing benchmarks are in Appendix [H](https://arxiv.org/html/2506.01241v3#A8).

**Table 1: Benchmark statistics. ^∗: rubric is developed by experts; otherwise, it is created by refining and expanding upon established evaluation protocols; ^†: task data is held privately. For each task, we report whether the task data is newly created (✓) or adapted from previous work; the average number of checklist items in each sample (*#Rubric*); and the average length of the input (*#Input*) and human reference (*#Reference*). Several of these tasks feature significantly longer inputs and references compared to existing domain-specific datasets (see Appendix [H](https://arxiv.org/html/2506.01241v3#A8)).**
| Task | Description | New | Sample num | #Rubric | #Input | #Reference |
| --- | --- | --- | --- | --- | --- | --- |
| T1LegalMDS ^* | Multi-Document case Summarization | ✓ | 100 | 26 | 113,745 | 1,778 |
| T2LegalSFG ^*^† | Statement of Fact Generation | ✓ | 100 | 41 | 187,302 | 5,155 |
| T3MaterialSEG ^* | Synthesis Explanation Generation | ✓ | 50 | 6 | 199 | 125 |
| T4EduPAE | Pedagogical Alignment Evaluation | ✓ | 100 | 7 | 277 | 60 |
| T5EduFG ^*† | Feedback Generation | ✓ | 100 | 19 | 666 | 139 |
| T6HealthCNG | Clinical Note Generation |  | 100 | 29 | 1,304 | 479 |
| T7ChemMDG ^* | Molecule Description Generation |  | 100 | 6 | 112 | 197 |
| T8BioPDG ^* | Protein Description Generation |  | 100 | 5 | 111 | 274 |
| T9MedicalDR ^† | Diagnosis & Reasoning |  | 100 | 11 | 1,335 | 429 |
| T10FinanceESG ^† | ESG report generation | ✓ | 100 | 27 | 69,297 | 280 |
| T11CyberRDG | Risk Description Generation |  | 100 | 6 | 555 | 157 |

##### Evaluating Long-Form Generations.

For granular evaluation of a generated text against a reference, recent research explores fact decomposition-based methods, extracting atomic facts from both the texts for comparison using NLI models or LLMs (Min et al., 2023; Kamoi et al., 2023).
Nevertheless, the fact decomposition process lacks task-specific considerations for fact granularity, which can result in inconsistent evaluations (Hu et al., 2025).

Another line of work on “LLM-as-a-judge” prompts LLMs to assess generated outputs against general, high-level criteria such as coherence and relevance (Liu et al., 2023).
To enhance task specificity, checklist-based evaluations are incorporated into WildBench (Lin et al., 2024) and BiGGenBench (Kim et al., 2024), which evaluate outputs against instance-specific checklists derived from task requirements.
Beyond these benchmarks, recent checklist-based methods, e.g., TICK (Cook et al., 2024), CheckEval (Lee et al., 2024a), RocketEval (Wei et al., 2025), and LLM-Rubric (Hashemi et al., 2024), adopt structured rubrics to guide LLM judges.
However, these approaches are either not tailored to domain-specific requirements or are not grounded in references, leading to subjective assessments and hindering the accurate evaluation of content coverage and relevance.

Our evaluation framework, Clear, builds on these directions by extending fact decomposition and checklist-based evaluation.
We derive task-specific checklists from an expert-designed rubric, thereby adapting prior checklist-based methods to better support expert-level tasks.
For each checklist item, we extract relevant facts from the model outputs and/or references and compare them to compute item-level scores, enabling fine-grained and objective evaluations.

## 3 Benchmark Construction with Multi-disciplinary Expert Tasks

##### Overview.

We introduce ExpertLongBench, a multi-disciplinary benchmark designed to evaluate the capabilities of LLMs on real-world expert-level tasks that require long-form inputs and outputs.
ExpertLongBench features a fine-grained evaluation rubric tailored to reflect the rigorous standards and nuanced requirements of expert domains.
The benchmark covers 11 expert-level tasks as shown in [Tab.˜1](https://arxiv.org/html/2506.01241v3#S2.T1),
including six tasks with newly collected data featuring unique challenges such as super-long input.
Notably, rubric design is complex and highly time-intensive; for instance, creating the rubric for T1LegalMDS required over 10 hours of expert effort. Details regarding the experts involved in rubric design and verification are provided in Appendix [A](https://arxiv.org/html/2506.01241v3#A1).
The selection of these tasks adhere to specific criteria: (1) it is possible to create well-defined rubric for consistent and objective evaluation; (2) it requires domain expertise for both task resolution and assessment; and (3) it is grounded in real-world expert workflows.
Additional details are discussed in Appendices [A](https://arxiv.org/html/2506.01241v3#A1) and [B](https://arxiv.org/html/2506.01241v3#A2). Generally, each sample in ExpertLongBench includes the following key elements:

- •
Task Input: The context of the task.
For example, the inputs for T2LegalSFG consist of an average of 14 long transcripts used to generate a Statement of Fact.
- •
Human-written Reference: Each task includes human-authored references in natural language.
Details can be referred to Appendix [B](https://arxiv.org/html/2506.01241v3#A2).
- •
Checklist-mapped Reference: A checklist-based, fine-grained evaluation rubric is designed for each task by collaboration with domain experts and referencing established resources. A checklist-mapped reference is constructed for every sample, as detailed in §[3](https://arxiv.org/html/2506.01241v3#S3.SS0.SSS0.Px3).

[Tab.˜1](https://arxiv.org/html/2506.01241v3#S2.T1) summarizes key statistics of our benchmark.
In total, it contains 1,050 samples, with an average input length and human reference length of 36,204 and 851 tokens, respectively.
For each task, we describe the task definition, significance, data acquisition and preprocessing, representative examples, evaluation rubric, and the construction of checklist-mapped references in Appendix [B](https://arxiv.org/html/2506.01241v3#A2).

While public evaluation sets enable transparency and facilitate model development, they are also prone to contamination and overfitting.
We design our benchmark with both a public and private subset as shown in [Tab.˜1](https://arxiv.org/html/2506.01241v3#S2.T1). The public set supports open experimentation, while the private set contains sensitive data used for fair evaluation to ensure robustness and confidentiality.(^2^22We provide a submission channel on the benchmark website that allows researchers to evaluate models on the private subset.)
The public set is shared under the CC BY-NC-SA 4.0 license(^3^33[https://creativecommons.org/licenses/by-nc-sa/4.0/](https://creativecommons.org/licenses/by-nc-sa/4.0/)), while the private set remains confidential. Details are provided in Appendix [I](https://arxiv.org/html/2506.01241v3#A9).

##### Task Sources.

ExpertLongBench draws data from two primary sources: 6 newly curated task data and 5 adapted existing task data. The first source includes tasks newly developed by the authors to address gaps in existing resources and better capture real-world expert challenges. Tasks are selected by discussing with domain researchers and practitioners, who also contributed to rubric design.
In parallel, we also look into existing datasets and corpora that fit the goal of accessing the capabilities of LLMs on expert-level tasks.
To adapt these for our benchmark, we carefully select representative and challenging samples
with details and examples shown in Appendix [B](https://arxiv.org/html/2506.01241v3#A2). Moreover, we design a checklist-based rubric for the tasks and create checklist-mapped references.

##### Expert-guided Rubric Design.

For each task, we design a checklist-based fine-grained evaluation rubric that is applicable to all data points within the same task to assess model performance.
Our rubric design follows two complementary approaches: 1)
Expert-guided design: we collaborate closely with domain experts(^4^44Experts involved in designing or verifying the rubrics possess relevant academic or professional expertise, with details provided in Appendix [A](https://arxiv.org/html/2506.01241v3#A1).) to co-design evaluation criteria that reflect professional standards and practical requirements and needs.
2) Protocol-refinement design: for tasks with established evaluation protocols guided by experts, we design fine-grained criteria by refining and expanding upon existing standards, ensuring more granular and systematic assessment of model performance.
An example rubric showcasing a subset of its checklist items is illustrated in [Fig.˜1](https://arxiv.org/html/2506.01241v3#S1.F1). In Appendix [B](https://arxiv.org/html/2506.01241v3#A2), we show all checklist items for each task.
Notably, these rubrics are written by experts, ensuring high-quality criteria that existing LLMs cannot yet replicate (refer to Appendix [I](https://arxiv.org/html/2506.01241v3#A9)), pointing to future research on automatically generating high-quality checklists.
Additional examples along with detailed rubric designs for each task are in the Evaluation Rubric section for each task in Appendix [B](https://arxiv.org/html/2506.01241v3#A2).

##### Sample Selection.

We establish criteria for further refining the sample pool in tasks with an initial sample size exceeding 100, ultimately selecting 100 representative samples for each task. Our selection criteria focus on two key aspects: diversity and difficulty.
For diversity, the select samples encompass a broad range of variations.
For difficulty, we identify key factors influencing difficulty for each task. For instance, we determine factors such as document length, number of appeals, number of complaints, and number of dockets for T1LegalMDS.
We applied these criteria across most tasks.
Details are provided in the Appendix [B](https://arxiv.org/html/2506.01241v3#A2).

##### Checklist-mapped Reference Creation.

To evaluate models’ capability for long-form generation on expert-level tasks according to the designed fine-grained rubric, we construct checklist-mapped references by extracting the content corresponding to each checklist item from the reference using GPT-4o. Exceptions include T3MaterialSEG and T10FinanceESG, where the checklist-based references are constructed during the data collection process (T3) or the references are already well-structured and can be processed into a checklist format (T10).
Example checklist-mapped references for T1LegalMDS are shown in [Fig.˜1](https://arxiv.org/html/2506.01241v3#S1.F1) and
the checklist-mapped reference subsections in Appendix [B](https://arxiv.org/html/2506.01241v3#A2).

Specifically, we adopt a role-playing strategy by prompting GPT-4o to extract all relevant information for each checklist item as comprehensively as possible from the reference. If no such information is present, the model is instructed to return “N/A” (prompts are in checklist-mapped reference subsections in Appendix [B](https://arxiv.org/html/2506.01241v3#A2)).
Throughout this paper, we will use the following notation: the checklist for a given task is denoted as $\{c_{i}\}_{i=1}^{n}$, where $n$ is the number of checklist items associated with the specific task. We design instructions for each checklist item and use GPT-4o to extract the corresponding information.

We then use both human evaluation and LLM-as-a-judge evaluation to ensure the quality of the extraction.
Evaluation results show that the mapped references achieve over 90% faithfulness and coverage on two tasks.
Detailed information is provided in §[4.2](https://arxiv.org/html/2506.01241v3#S4.SS2). For other tasks, human inspection was employed to ensure the mapping quality. Additional details can be found in Appendix [B](https://arxiv.org/html/2506.01241v3#A2).

## 4 Checklist-based Performance Assessment using Clear

In this section, we introduce Clear for expert-aligned evaluation of model performance. As shown in [Fig.˜1](https://arxiv.org/html/2506.01241v3#S1.F1), given a model output, our framework maps its information to items in the checklist derived from the expert-design rubric, and assesses the quality of the model output by comparing the checklist-mapped model output against the checklist-mapped human reference (§[4.1](https://arxiv.org/html/2506.01241v3#S4.SS1)).
We quantitatively justify key design choices in Clear, including the selection of the checklist-mapper and the judge for checklist comparison (§[4.2](https://arxiv.org/html/2506.01241v3#S4.SS2)).

### 4.1 Evaluation Process

##### Generating Checklist-Mapped Model Responses.

We follow the same procedure described in §[3](https://arxiv.org/html/2506.01241v3#S3.SS0.SSS0.Px2) to extract checklists from model responses.
However, instead of using GPT-4o, which can be significantly more expensive while extracting checklists from all models, we opt for the open-weight model Qwen2.5-72B as the checklist mapper considering its availability of model weights and decent extraction performance, which will be validated in §[4.2](https://arxiv.org/html/2506.01241v3#S4.SS2).

##### Assessing Response Quality using Checklists.

To evaluate checklist-mapped responses, we assess the degree to which the checklist-mapped model response aligns with that of the reference. To this end, we use GPT-4o within an LLM-as-a-judge paradigm (Gu et al., 2024; Li et al., 2024a, b), adapting the reference-based scoring methodology (Verga et al., 2024) to evaluate the semantic alignment between each checklist item in a model response and the corresponding information in the reference for every sample. For instance, consider a checklist item $c$ with corresponding information in the model response and reference denoted as $H(c)$ and $R(c)$, respectively. The LLM judge assigns a binary score to each checklist item by evaluating whether the semantic content of $R(c)$ is contained within $H(c)$. In other words, we assign a binary score of 1 only when all the information conveyed by $R(c)$ is also present in $H(c)$. The prompt for the judge is in Appendix [C](https://arxiv.org/html/2506.01241v3#A3).
We use GPT-4o as the judge, as it saves cost and shows high agreement with candidate judge models, as discussed in §[4.2](https://arxiv.org/html/2506.01241v3#S4.SS2).

After performing item-level assessment for a checklist-mapped model response, we define its
checklist precision (checklist recall) as the fraction of checklist items whose model response (reference) is semantically contained within the reference (model response) and accuracy as the fraction of checklist items whose model response and reference mutually contain each other.
We obtain the task-level performance by averaging the sample-level metrics.

### 4.2 Evaluation Component Validation

Model Selection for Checklist Mapper.
To enable cost-effective and reproducible checklist mapping, we primarily consider open-weight models—Llama-3.3-70B-Instruct, Mistral-Large-Instruct, and Qwen2.5-72B —and evaluate them using the reference checklists extracted by GPT-4o.
We, first, validate the quality of these reference checklists through human and automated evaluations on tasks T1 and T6, confirming over 90% faithfulness and coverage. We selected these two tasks due to their challenging long contexts and extended output requirements involving over $25$ checklist items. To identify a suitable open-weight mapper, we evaluated tasks T1, T6, T7, and T8 which spans diverse domains and configurations of input / output length. Qwen2.5-72B achieved the highest average performance with an average F1-score of 90.1, demonstrating its applicability for accurate checklist mapping.(^5^55To investigate more cost-efficient options for checklist mapping, we also evaluate smaller models from the same family as Qwen2.5-72B, as detailed in Appendix [E](https://arxiv.org/html/2506.01241v3#A5). While these models achieve reasonable performance, they still fall notably short of Qwen2.5-72B. Consequently, we continue to use Qwen2.5-72B as the checklist mapper to ensure high-quality and accurate mappings.)
See Appendix [E](https://arxiv.org/html/2506.01241v3#A5) for details on above analyses.

##### Model Selection for Checklist Evaluation.

To support the choice of solely using GPT-4o for evaluating the checklist, we measured the alignment between the annotations produced by GPT-4o and Gemini-2.0-Flash on the aforementioned data used for examining checklist mapping by open-weight models.
The observed Cohen’s Kappa scores were $0.81$ for T1, $0.87$ for T6, $0.89$ for T7, and $0.85$ for T8, indicating a near-perfect level of inter-annotator agreement. These high agreement scores validate the choice of using GPT-4o annotations exclusively for the final evaluations.

## 5 Experiments

We evaluate 13 models from 3 open-weight families and 3 proprietary families on ExpertLongBench.
For open-weight models, we use their instruction fine-tuned and RLHF variants.
Model details are provided in Appendix [G](https://arxiv.org/html/2506.01241v3#A7).

**Table 2: Evaluating LLMs on ExpertLongBench (scaled to 0–100) using F1 scores. Models are sorted by average performance and the best performing model on each task is bolded. Model ranking is indicated by the color of the cell, with green (best) to white (worst).**
| Model | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Gemini-2.5-Pro | 25.4 | 10.0 | 16.9 | 50.2 | 47.9 | 44.0 | 50.4 | 14.9 | 43.2 | 21.1 | 43.3 | 33.4 |
| GPT-5 | 27.2 | 10.3 | 5.6 | 47.2 | 56.5 | 54.7 | 49.8 | 15.3 | 18.7 | 13.9 | 41.5 | 31.0 |
| Gemini-2.0-Flash | 13.1 | 7.9 | 19.5 | 49.3 | 30.2 | 20.8 | 35.6 | 9.0 | 46.0 | 36.7 | 26.5 | 26.8 |
| GPT-4o | 13.2 | 6.2 | 15.2 | 56.9 | 29.9 | 25.3 | 34.5 | 10.1 | 35.2 | 35.7 | 29.2 | 26.5 |
| GPT-4o-mini | 16.5 | 5.9 | 15.2 | 49.2 | 25.0 | 25.5 | 33.9 | 10.3 | 29.5 | 42.2 | 34.2 | 26.1 |
| Llama-3.3-70B-Instruct | 12.1 | 4.9 | 16.8 | 49.5 | 15.9 | 20.2 | 33.4 | 8.3 | 32.8 | 42.6 | 33.8 | 24.6 |
| Mistral-Large-Instruct | 9.3 | 4.0 | 17.9 | 51.8 | 19.2 | 24.1 | 33.6 | 9.0 | 19.5 | 39.4 | 36.2 | 24.0 |
| Qwen2.5-72B | 12.3 | 4.1 | 17.4 | 51.0 | 10.8 | 21.2 | 32.6 | 9.3 | 35.7 | 33.3 | 33.9 | 23.8 |
| Mistral-Nemo-Instruct | 4.5 | 1.5 | 16.1 | 40.2 | 27.6 | 24.9 | 33.8 | 8.5 | 23.3 | 50.8 | 30.1 | 23.8 |
| Llama3.1-8B-Instruct | 9.5 | 3.3 | 20.7 | 48.2 | 18.4 | 23.9 | 32.4 | 6.2 | 25.1 | 40.4 | 29.1 | 23.4 |
| Claude-3.7-Sonnet | 11.5 | 0.9 | 18.1 | 30.4 | 35.0 | 26.1 | 36.1 | 9.2 | 21.4 | 33.1 | 33.5 | 23.2 |
| Qwen2.5-7B | 10.7 | 4.0 | 16.5 | 42.5 | 12.7 | 22.0 | 34.8 | 8.7 | 15.1 | 35.0 | 23.8 | 20.5 |
| Claude-3.5-Haiku | 2.8 | 1.1 | 22.1 | 30.3 | 9.7 | 10.9 | 33.4 | 9.2 | 18.1 | 40.8 | 34.1 | 19.3 |

[Tab.˜2](https://arxiv.org/html/2506.01241v3#S5.T2) presents the performance of the evaluated models, measured with the F1 score. The models are sorted by their average F1 score across all tasks. Accuracy, precision, and recall are reported in
[Tab.˜45](https://arxiv.org/html/2506.01241v3#A4.T45),
[Tab.˜46](https://arxiv.org/html/2506.01241v3#A4.T46), and [Tab.˜47](https://arxiv.org/html/2506.01241v3#A4.T47).
The tasks in ExpertLongBench pose significant challenges to existing LLMs.
Notably, Gemini-2.5-Pro, the top-performing model in our evaluation, achieves an average F1 score of only 33.4, underscoring the difficulty of the benchmark.
This also indicates substantial room for improvement, highlighting the benchmark’s longevity and its continued value as a challenging resource for evaluating future LLMs.
Among all tasks, T2 proves to be the most challenging, with all models scoring an F1 below 11. In addition to the complex nature of legal argumentation, T2 involves longer inputs and necessitates longer outputs, thereby further testing the long-context capabilities of the models.

##### Scaling does not consistently improve performance across tasks.

We observe that within the same model family, the larger model outperforms their smaller counterpart in terms of average performance.
However, this superiority is not consistent across all individual tasks.
For example, on T4, Mistral-Large-Instruct ranks 2nd while Mistral-Nemo-Instruct ranks 11th. Conversely, on task T10, Mistral-Large-Instruct ranks 6th, whereas Mistral-Nemo-Instruct achieves the best performance across all models.

Figure: Figure 2: F1 score vs. coverage of checklist items (i.e., the percentage of checklist items that are covered in the generation regardless their correctness).
Refer to caption: x3.png

In fact, no single model consistently outperforms its smaller variant from the same family across all tasks.
This inconsistency may suggest that the pre-training curricula of existing LLMs might not uniformly emphasize or adequately represent the domain-specific tasks,
especially when scaling for larger models,
potentially leading to an imbalance in their capabilities across different tasks.

##### Proprietary models are not always superior.

Proprietary models, which are expected to be larger and more capable of handling diverse tasks, are not always better than open-weight models.
Specifically, Claude is less designed for expert-level workflows. Compared with other families, Claude-3.7-Sonnet only outperforms Qwen2.5-7B, and Claude-3.5-Haiku yields the lowest overall score.
Among open-weight models, Qwen generally demonstrates the weakest performance.
Both Qwen models have a 32K token context length, shorter than other models which offer 128K or longer.
This discrepancy in context length leads to a significant performance disparity for Qwen on T10, which requires processing an average of 64K input tokens, where the performance of Qwen2.5-72B trails the leading model by 17.5.

##### High coverage of checklist items does not correlate with quality.

We further investigate the percentage of checklist items covered in the generated text, irrespective of correctness, and its correlation with output quality, as measured by the F1 score.
[Fig.˜2](https://arxiv.org/html/2506.01241v3#S5.F2) illustrates a overall negative correlation between checklist item coverage and the F1 score, aggregated across all experimented models and tasks.
Models frequently obtaining high coverage scores while concurrently exhibiting low F1 scores, suggesting their ability to produce content that seemingly adheres to domain-specific standards but is incorrect.
Such behavior presents a risk, as the output might mislead users into perceiving the generated content as reliable due to its apparent completeness Further main results and detailed outcomes for each task can be found in Appendix [D](https://arxiv.org/html/2506.01241v3#A4).

## 6 Towards Reproducible and Low-Cost Evaluation

While GPT-4o demonstrates strong performance in evaluating checklist-aligned model responses (Xu et al., 2024; Posner and Saran, 2025; Seßler et al., 2025; Jo et al., 2024; Eriksen et al., 2024), there are three primary concerns associated with the use of closed models: (a) Performing large-scale experiments are costly, (b) Their deployment on external servers poses privacy risks in sensitive domains, and (c) Unannounced updates or version changes that undermine reproducibility across studies.

Figure: Figure 3: Correlation of different model combinations with GPT-4o judgments averaged over all the tasks.
Refer to caption: x4.png

Therefore, the primary objective of this section is to investigate whether the judge component in our evaluation pipeline can be substituted with open-weight alternatives to promote a more accessible and reproducible research environment. To this end, we evaluate various open-weight models and their combinations as judge models by measuring how well their accuracy scores correlate with those from GPT-4o, which serves as the ground truth due to its strong domain-level performance. For model combinations involving multiple judge models, we explore two pooling strategies (Wang et al., 2022; Verga et al., 2024) to compute an aggregate score for each sample: (a) Mean Pooling, where the final score is the average of the scores assigned by each constituent judge; and (b) Majority Pooling, where the final score corresponds to the mode of the individual scores. In cases where multiple modes exist, we resolve the tie by averaging their values.

The results presented in [Tab.˜51](https://arxiv.org/html/2506.01241v3#A5.T51)
show the correlation among the computed scores of various model variants.
In this table, the category labeled Small combines scores from Llama3.1-8B-Instruct, Mistral-Nemo-Instruct, and Mistral-Large-Instruct. Similarly, the category Large includes Llama-3.3-70B-Instruct, Mistral-Large-Instruct, and Qwen2.5-72B. The category All-small encompasses models from Small and GPT-4o-mini.

##### Performance of single-model judges.

From [Fig.˜3](https://arxiv.org/html/2506.01241v3#S6.F3), we find that most judge models exhibit a high degree of correlation with GPT-4o, with Qwen2.5-72B yielding the highest average correlation among all single judges.
This indicates that a single model can reliably assess checklists while significantly maintaining computational overhead to a low value.

Figure: Figure 4: Regression analysis between the average performance along each task and the average correlation of judgments with GPT-4o assignments
Refer to caption: x5.png

##### Combining models results in better correlation than its individual counterparts.

Additionally, combining open-source models enhances alignment with GPT-4o.
For instance, while Llama3.1-8B-Instruct, Mistral-Nemo-Instruct, and Qwen2.5-7B individually achieve average correlations of 0.65, 0.80, and 0.78, respectively, majority pooling their outputs raises the correlation to 0.82.
Moreover, in most model combinations—excluding large-open-source-models—majority pooling consistently outperforms mean pooling in terms of correlation.

##### Connection between task complexity and judgment correlation with GPT-4o .

A regression analysis of the relationship between task complexity and average judgment correlation with GPT-4o (see [Fig.˜4](https://arxiv.org/html/2506.01241v3#S6.F4)) reveals a strong correlation and a moderately high $R^{2}$ value, indicating that the regression model explains a substantial portion of the variance.
This can be used to decide the scale of model for evaluating a particular task. For instance, smaller models like Qwen2.5-7B and Mistral-Nemo-Instruct align well with GPT-4o for less complex tasks. For more challenging tasks such as T1 and T2, larger models like Qwen2.5-72B are recommended. Since majority pooling is more effective for estimating binary variables (Verga et al., 2024), and our checklist scoring relies on binary labels, it leads to more accurate overall score assignment.

## 7 Skill Decomposition Analysis

We provide a detailed skill-level and difficulty-level analysis of model performance on ExpertLongBench for a better understanding of where current models excel and where they fall short.
Our analysis is grounded in a fine-grained examination of each task’s checklist items, identifying the specific skills required to fulfill each item at corresponding difficulty levels. Detailed explanations of the skills, difficulty levels, and item-level skill and difficulty mapping are provided in Appendix [F](https://arxiv.org/html/2506.01241v3#A6).

We examine model performance across varying knowledge levels including Below College, College, and Graduate, and reasoning difficulty including Low (Knowledge Memorization), Medium (Knowledge Understanding), High (Knowledge Applying), Very High (Knowledge Creating) (Krathwohl, 2002; Yu et al., 2024).
We present our key findings as follows.

Figure: Figure 5: Model performance across various levels of reasoning complexity.
Refer to caption: x6.png

##### Knowledge complexity and reasoning difficulty are major barriers.

The consistent performance drop at the Graduate-level knowledge as shown in [Fig.˜11](https://arxiv.org/html/2506.01241v3#A6.F11)
highlights a substantial challenge for current models in mastering expert-level knowledge. The clear performance decline with higher reasoning difficulty as shown in [Fig.˜5](https://arxiv.org/html/2506.01241v3#S7.F5) emphasizes the importance of assessing models beyond knowledge memorization and understanding, focusing on their ability to perform complex reasoning.
Many models perform worse on low reasoning difficulty level (knowledge memorization) than medium reasoning difficulty level (knowledge understanding) because the expert-level knowledge in ExpertLongBench may be long-tail knowledge that rarely appears during training, limiting memorization performance, while the model can still leverage its reasoning and generalization abilities to approximate understanding (Kandpal et al., 2023).
Notably, ExpertLongBench is specifically designed to assess more advanced knowledge processing skills.

##### Importance of diverse difficulty levels in benchmarking.

Models show divergent strengths. For instance, Claude-3.7-Sonnet ranks first in the low reasoning level (knowledge memorization) but ranks relatively low in the very high reasoning level (knowledge creation) ([Fig.˜5](https://arxiv.org/html/2506.01241v3#S7.F5)). These results emphasize the value of benchmark with diverse difficulty levels, which captures performance differences across varying knowledge and reasoning levels, revealing model strengths and weaknesses that might be overlooked in general benchmarks.
Notably, ExpertLongBench is meticulously designed to provide a comprehensive evaluation across diverse difficulty levels, and we recommend the community to leverage ExpertLongBench for future model evaluation.
Analysis results on more general skills and additional details are in Appendix [F](https://arxiv.org/html/2506.01241v3#A6).

## 8 Conclusion

We introduce ExpertLongBench, a multi-domain expert-level benchmark. ExpertLongBench comprises 1050 samples, spanning 11 tasks across 9 distinct domains.
These tasks, originating from realistic expert applications rather than typical question-answering scenarios, demand long-form outputs that adhere to domain-specific standards.
Each task is accompanied by an expert-designed and validated rubric, which serve as evaluation guidelines. To evaluate long-form model outputs on these expert tasks, we design Clear, an evaluation framework that maps model outputs and references into checklists derived from the rubric and subsequently performs an item-by-item comparison of these checklists.
This process yields a fine-grained, grounded evaluation that aligns with domain-specific requirements.
We conduct experiments with 13 LLMs, revealing that ExpertLongBench pose a significant challenge for current LLMs.
While these models can generate content that superficially matches required aspects, they frequently lack accuracy.
Further analysis of Clear demonstrates the feasibility of substituting proprietary models with open-weight alternatives for the roles of checklist mapper and evaluator, facilitating low-cost, reproducible and scalable benchmarking.

## Acknowledgments

This project is partially supported by the NVIDIA Academic Grant Program. This work is supported in part by the National Science Foundation through grants 2046016 and 2302564. Shuyang Cao is supported by a Bloomberg Data Science Ph.D. Fellowship.
We would like to thank the following domain experts for engaging in thoughtful discussions and sharing their perspectives during the development of this work. Their input helped inform our understanding across a range of specialized fields. For discussions related to ESG, we thank Daniel Cremin, Lauren Yeung, Rumi Mahmood, and Umar Ashfaq from MSCI Inc., as well as Mohammed Ombadi. For health-related topics, we thank Sharon Kardia. We are also grateful to Sabina Tomkins and Derek Van Berkel for discussions on environment and sustainability, Charles Brooks for conversations on chemistry, and Wenhao Sun for insights related to material science. We appreciate the time and perspectives shared by Patrick Barry in the legal domain. For education, we thank Xu Wang, Anne Gere, Xinyi Lu, and Jason Godfrey for their helpful discussions and references.
We would also like to thank Aditya Tambe, Evan Wang, and Kaelyn Lin for their assistance on the project. Finally, we appreciate the discussions and suggestions from Michigan LAUNCH Lab members.

## References

- Sonkar et al. [2024]
Shashank Sonkar, Kangqi Ni, Sapana Chaudhary, and Richard G Baraniuk.
Pedagogical alignment of large language models.
*arXiv preprint arXiv:2402.05000*, 2024.
- Siino et al. [2025]
Marco Siino, Mariana Falco, Daniele Croce, and Paolo Rosso.
Exploring llms applications in law: A literature review on current legal nlp approaches.
*IEEE Access*, 2025.
- McDuff et al. [2025]
Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, Le Hou, Yong Cheng, Yun Liu, S. Sara Mahdavi, Sushant Prakash, Anupam Pathak, Christopher Semturs, Shwetak Patel, Dale R. Webster, Ewa Dominowska, Juraj Gottweis, Joelle Barral, Katherine Chou, Greg S. Corrado, Yossi Matias, Jake Sunshine, Alan Karthikesalingam, and Vivek Natarajan.
Towards accurate differential diagnosis with large language models.
*Nature*, April 2025.
ISSN 1476-4687.
doi: 10.1038/s41586-025-08869-4.
URL [https://doi.org/10.1038/s41586-025-08869-4](https://doi.org/10.1038/s41586-025-08869-4).
- Wang and Brorsson [2025]
Xinlin Wang and Mats Brorsson.
Can large language model analyze financial statements well?
In Chung-Chi Chen, Antonio Moreno-Sandoval, Jimin Huang, Qianqian Xie, Sophia Ananiadou, and Hsin-Hsi Chen, editors, *Proceedings of the Joint Workshop of the 9th Financial Technology and Natural Language Processing (FinNLP), the 6th Financial Narrative Processing (FNP), and the 1st Workshop on Large Language Models for Finance and Legal (LLMFinLegal)*, pages 196–206, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics.
URL [https://aclanthology.org/2025.finnlp-1.19/](https://aclanthology.org/2025.finnlp-1.19/).
- Shen et al. [2022]
Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlanger, and Doug Downey.
Multi-lexsum: Real-world summaries of civil rights lawsuits at multiple granularities.
*Advances in Neural Information Processing Systems*, 35:13158–13173, 2022.
- Cascella et al. [2023]
Marco Cascella, Jonathan Montomoli, Valentina Bellini, and Elena Bignami.
Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios.
*Journal of medical systems*, 47(1):33, 2023.
- Hendrycks et al. [2020]
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
Measuring massive multitask language understanding.
*arXiv preprint arXiv:2009.03300*, 2020.
- Rein et al. [2024]
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman.
Gpqa: A graduate-level google-proof q&a benchmark.
In *First Conference on Language Modeling*, 2024.
- Malaviya et al. [2023]
Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth.
Expertqa: Expert-curated questions and attributed answers.
*arXiv preprint arXiv:2309.07852*, 2023.
- Lin et al. [2024]
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi.
Wildbench: Benchmarking llms with challenging tasks from real users in the wild.
*arXiv preprint arXiv:2406.04770*, 2024.
- Zhong et al. [2023]
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan.
Agieval: A human-centric benchmark for evaluating foundation models.
*arXiv preprint arXiv:2304.06364*, 2023.
- Sun et al. [2024]
Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu.
Scieval: A multi-level large language model evaluation benchmark for scientific research, 2024.
URL [https://arxiv.org/abs/2308.13149](https://arxiv.org/abs/2308.13149).
- Phan et al. [2025]
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al.
Humanity’s last exam.
*arXiv preprint arXiv:2501.14249*, 2025.
- He et al. [2024]
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun.
OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 3828–3850, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.acl-long.211.
URL [https://aclanthology.org/2024.acl-long.211/](https://aclanthology.org/2024.acl-long.211/).
- Malaviya et al. [2025]
Chaitanya Malaviya, Priyanka Agrawal, Kuzman Ganchev, Pranesh Srinivasan, Fantine Huot, Jonathan Berant, Mark Yatskar, Dipanjan Das, Mirella Lapata, and Chris Alberti.
Dolomites: Domain-specific long-form methodical tasks.
*Transactions of the Association for Computational Linguistics*, 13:1–29, 2025.
- Yifei et al. [2025]
Li S Yifei, Allen Chang, Chaitanya Malaviya, and Mark Yatskar.
Researchqa: Evaluating scholarly question answering at scale across 75 fields with survey-mined questions and rubrics.
*arXiv preprint arXiv:2509.00496*, 2025.
- Min et al. [2023]
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi.
FActScore: Fine-grained atomic evaluation of factual precision in long form text generation.
In Houda Bouamor, Juan Pino, and Kalika Bali, editors, *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 12076–12100, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.741.
URL [https://aclanthology.org/2023.emnlp-main.741/](https://aclanthology.org/2023.emnlp-main.741/).
- Kamoi et al. [2023]
Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett.
WiCE: Real-world entailment for claims in Wikipedia.
In Houda Bouamor, Juan Pino, and Kalika Bali, editors, *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 7561–7583, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.470.
URL [https://aclanthology.org/2023.emnlp-main.470/](https://aclanthology.org/2023.emnlp-main.470/).
- Hu et al. [2025]
Qisheng Hu, Quanyu Long, and Wenya Wang.
Decomposition dilemmas: Does claim decomposition boost or burden fact-checking performance?, 2025.
URL [https://arxiv.org/abs/2411.02400](https://arxiv.org/abs/2411.02400).
- Liu et al. [2023]
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.
G-eval: Nlg evaluation using gpt-4 with better human alignment.
*arXiv preprint arXiv:2303.16634*, 2023.
- Kim et al. [2024]
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, et al.
The biggen bench: A principled benchmark for fine-grained evaluation of language models with language models.
*arXiv preprint arXiv:2406.05761*, 2024.
- Cook et al. [2024]
Jonathan Cook, Tim Rocktäschel, Jakob Foerster, Dennis Aumiller, and Alex Wang.
Ticking all the boxes: Generated checklists improve llm evaluation and generation.
*arXiv preprint arXiv:2410.03608*, 2024.
- Lee et al. [2024a]
Yukyung Lee, Joonghoon Kim, Jaehee Kim, Hyowon Cho, Jaewook Kang, Pilsung Kang, and Najoung Kim.
Checkeval: A reliable llm-as-a-judge framework for evaluating text generation using checklists.
*arXiv preprint arXiv:2403.18771*, 2024a.
- Wei et al. [2025]
Tianjun Wei, Wei Wen, Ruizhi Qiao, Xing Sun, and Jianghong Ma.
Rocketeval: Efficient automated llm evaluation via grading checklist.
*arXiv preprint arXiv:2503.05142*, 2025.
- Hashemi et al. [2024]
Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie.
Llm-rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 13806–13834, 2024.
- Gu et al. [2024]
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al.
A survey on llm-as-a-judge.
*arXiv preprint arXiv:2411.15594*, 2024.
- Li et al. [2024a]
Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu.
Llms-as-judges: a comprehensive survey on llm-based evaluation methods.
*arXiv preprint arXiv:2412.05579*, 2024a.
- Li et al. [2024b]
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu.
From generation to judgment: Opportunities and challenges of llm-as-a-judge.
*arXiv preprint arXiv: 2411.16594*, 2024b.
- Verga et al. [2024]
Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis.
Replacing judges with juries: Evaluating llm generations with a panel of diverse models.
*arXiv preprint arXiv:2404.18796*, 2024.
- Xu et al. [2024]
Simin Xu, Xiaowei Huang, Chung Kwan Lo, Gaowei Chen, and Morris Siu-yung Jong.
Evaluating the performance of chatgpt and gpt-4o in coding classroom discourse data: A study of synchronous online mathematics instruction.
*Computers and Education: Artificial Intelligence*, 7:100325, 2024.
- Posner and Saran [2025]
Eric A Posner and Shivam Saran.
Judge ai: Assessing large language models in judicial decision-making.
*University of Chicago Coase-Sandor Institute for Law & Economics Research Paper*, (2503), 2025.
- Seßler et al. [2025]
Kathrin Seßler, Maurice Fürstenberg, Babette Bühler, and Enkelejda Kasneci.
Can ai grade your essays? a comparative analysis of large language models and teacher ratings in multidimensional essay scoring.
In *Proceedings of the 15th International Learning Analytics and Knowledge Conference*, pages 462–472, 2025.
- Jo et al. [2024]
Eunbeen Jo, Sanghoun Song, Jong-Ho Kim, Subin Lim, Ju Hyeon Kim, Jung-Joon Cha, Young-Min Kim, Hyung Joon Joo, et al.
Assessing gpt-4’s performance in delivering medical advice: comparative analysis with human experts.
*JMIR Medical Education*, 10(1):e51282, 2024.
- Eriksen et al. [2024]
Alexander V Eriksen, Sören Möller, and Jesper Ryg.
Use of gpt-4 to diagnose complex clinical cases, 2024.
- Wang et al. [2022]
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.
Self-consistency improves chain of thought reasoning in language models.
*arXiv preprint arXiv:2203.11171*, 2022.
- Krathwohl [2002]
David R Krathwohl.
A revision of bloom’s taxonomy: An overview.
*Theory into practice*, 41(4):212–218, 2002.
- Yu et al. [2024]
Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, et al.
Kola: Carefully benchmarking world knowledge of large language models.
In *ICLR*, 2024.
- Kandpal et al. [2023]
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel.
Large language models struggle to learn long-tail knowledge.
In *International Conference on Machine Learning*, pages 15696–15707. PMLR, 2023.
- Lin [2004]
Chin-Yew Lin.
Rouge: A package for automatic evaluation of summaries.
In *Text summarization branches out*, pages 74–81, 2004.
- Zhang et al. [2019]
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.
Bertscore: Evaluating text generation with bert.
*arXiv preprint arXiv:1904.09675*, 2019.
- LSD Law [n.d.]
LSD Law.
Statement of facts, n.d.
URL [https://www.lsd.law/define/statement-of-facts](https://www.lsd.law/define/statement-of-facts).
Accessed: 2025-05-19.
- TypeLaw [2021]
TypeLaw.
Infographic: The anatomy of a legal brief, 2021.
URL [https://www.typelaw.com/blog/infographic-the-anatomy-of-a-legal-brief/](https://www.typelaw.com/blog/infographic-the-anatomy-of-a-legal-brief/).
Accessed: 2025-05-20.
- Sun and David [2025]
Wenhao Sun and Nicholas David.
A critical reflection on attempts to machine-learn materials synthesis insights from text-mined literature recipes.
*Faraday Discussions*, 256:614–638, 2025.
- Curtarolo et al. [2013]
Stefano Curtarolo, Gus LW Hart, Marco Buongiorno Nardelli, Natalio Mingo, Stefano Sanvito, and Ohad Levy.
The high-throughput highway to computational materials design.
*Nature materials*, 12(3):191–201, 2013.
- Kononova et al. [2019]
Olga Kononova, Haoyan Huo, Tanjin He, Ziqin Rong, Tiago Botari, Wenhao Sun, Vahe Tshitoyan, and Gerbrand Ceder.
Text-mined dataset of inorganic materials synthesis recipes.
*Scientific data*, 6(1):203, 2019.
- Jain et al. [2013]
Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al.
Commentary: The materials project: A materials genome approach to accelerating materials innovation.
*APL materials*, 1(1), 2013.
- Paul [1995]
Richard Paul.
Critical thinking: How to prepare students for a rapidly changing world.
*(No Title)*, 1995.
- Chi [2009]
Michelene TH Chi.
Active-constructive-interactive: A conceptual framework for differentiating learning activities.
*Topics in cognitive science*, 1(1):73–105, 2009.
- Hattie and Timperley [2007]
John Hattie and Helen Timperley.
The power of feedback.
*Review of educational research*, 77(1):81–112, 2007.
- Gao et al. [2025]
Ge Gao, Amelia Leon, Andrea Jetten, Jasmine Turner, Husni Almoubayyed, Stephen Fancsali, and Emma Brunskill.
Predicting long-term student outcomes from short-term edtech log data.
In *Proceedings of the 15th International Learning Analytics and Knowledge Conference*, pages 631–641, 2025.
- Li et al. [2023a]
Qingyao Li, Lingyue Fu, Weiming Zhang, Xianyu Chen, Jingwei Yu, Wei Xia, Weinan Zhang, Ruiming Tang, and Yong Yu.
Adapting large language models for education: Foundational capabilities, potentials, and challenges.
*arXiv preprint arXiv:2401.08664*, 2023a.
- Razafinirina et al. [2024]
Mahefa Abel Razafinirina, William Germain Dimbisoa, and Thomas Mahatody.
Pedagogical alignment of large language models (llm) for personalized learning: a survey, trends and challenges.
*Journal of Intelligent Learning Systems and Applications*, 16(4):448–480, 2024.
- Sonkar et al. [2023]
Shashank Sonkar, Naiming Liu, Debshila Basu Mallick, and Richard G Baraniuk.
Class: A design framework for building intelligent tutoring systems based on learning science principles.
*arXiv preprint arXiv:2305.13272*, 2023.
- Stone [1998]
C Addison Stone.
The metaphor of scaffolding: Its utility for the field of learning disabilities.
*Journal of learning disabilities*, 31(4):344–364, 1998.
- Kluger and DeNisi [1996]
Avraham N Kluger and Angelo DeNisi.
The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory.
*Psychological bulletin*, 119(2):254, 1996.
- Ferris [1997]
Dana R Ferris.
The influence of teacher commentary on student revision.
*TESOL quarterly*, 31(2):315–339, 1997.
- Hayes et al. [1987]
John R Hayes, Linda Flower, Karen A Schriver, James Stratman, Linda Carey, et al.
Cognitive processes in revision.
*Advances in applied psycholinguistics*, 2:176–240, 1987.
- Matsumura et al. [2002]
Lindsay Clare Matsumura, G Genevieve Patthey-Chavez, Rosa Valdés, and Helen Garnier.
Teacher feedback, writing assignment quality, and third-grade students’ revision in lower-and higher-achieving urban schools.
*The Elementary School Journal*, 103(1):3–25, 2002.
- Bitchener et al. [2005]
John Bitchener, Stuart Young, and Denise Cameron.
The effect of different types of corrective feedback on esl student writing.
*Journal of second language writing*, 14(3):191–205, 2005.
- Sugita [2006]
Yoshihito Sugita.
The impact of teachers’ comment types on students’ revision.
*ELT journal*, 60(1):34–41, 2006.
- Westmacott [2017]
Anne Westmacott.
Direct vs. indirect written corrective feedback: Student perceptions.
*Íkala, revista de lenguaje y cultura*, 22(1):17–32, 2017.
- Bloxham et al. [2011]
Sue Bloxham, Peter Boyd, and Susan Orr.
Mark my words: the role of assessment criteria in uk higher education grading practices.
*Studies in Higher Education*, 36(6):655–670, 2011.
- Jeong [2015]
Heejeong Jeong.
Rubrics in the classroom: do teachers really follow them?
*Language Testing in Asia*, 5:1–14, 2015.
- Broad [2003]
Bob Broad.
*What we really value: Beyond rubrics in teaching and assessing writing*.
University Press of Colorado, 2003.
- Nair et al. [2024]
Inderjeet Nair, Jiaye Tan, Xiaotian Su, Anne Gere, Xu Wang, and Lu Wang.
Closing the loop: Learning to generate writing feedback via language model simulated student revisions.
*arXiv preprint arXiv:2410.08058*, 2024.
- Contributors [2023]
Wikipedia Contributors.
Soap note, 2023.
[https://en.wikipedia.org/wiki/SOAP_note](https://en.wikipedia.org/wiki/SOAP_note).
- Podder et al. [2023]
V. Podder, V. Lew, and S. Ghassemzadeh.
Soap notes, 2023.
URL [https://www.ncbi.nlm.nih.gov/books/NBK482263/](https://www.ncbi.nlm.nih.gov/books/NBK482263/).
Updated 2023 Aug 28. In: StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing; 2025 Jan–.
- Arndt et al. [2017]
Brian G Arndt, John W Beasley, Michael D Watkinson, Jonathan L Temte, Wen-Jan Tuan, Christine A Sinsky, and Valerie J Gilchrist.
Tethered to the ehr: Primary care physician workload assessment using ehr event log data and time-motion observations.
*Annals of Family Medicine*, 15(5):419–426, 2017.
- Yim et al. [2023]
Ww Yim, Y Fu, Asma Ben Abacha, et al.
Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation.
*Scientific Data*, 10(1):586, 2023.
doi: 10.1038/s41597-023-02487-3.
- Weininger [1988]
David Weininger.
Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules.
*Journal of Chemical Information and Computer Sciences*, 28(1):31–36, 1988.
doi: 10.1021/ci00057a005.
- Edwards et al. [2021]
Carl Edwards, ChengXiang Zhai, and Heng Ji.
Text2mol: Cross-modal molecule retrieval with natural language queries.
In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 595–607, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.
doi: 10.18653/v1/2021.emnlp-main.47.
URL [https://aclanthology.org/2021.emnlp-main.47](https://aclanthology.org/2021.emnlp-main.47).
- Edwards et al. [2022]
Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji.
Translation between molecules and natural language, 2022.
URL [https://arxiv.org/abs/2204.11817](https://arxiv.org/abs/2204.11817).
- Rives et al. [2021]
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Dustin Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al.
Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences.
*Proceedings of the National Academy of Sciences*, 118(15):e2016239118, 2021.
doi: 10.1073/pnas.2016239118.
- Li et al. [2023b]
Zhen Li, Yuxuan Wang, Yifan Wang, Yuxuan Zhang, Yixue Li, and Yang Zhang.
Annopro: a strategy for protein function annotation based on multi-scale representations and deep learning.
*Genome Biology*, 24(1):1–17, 2023b.
doi: 10.1186/s13059-024-03166-1.
- Feng et al. [2024]
Kehua Feng, Keyan Ding, Weijie Wang, Xiang Zhuang, Zeyuan Wang, Ming Qin, Yu Zhao, Jianhua Yao, Qiang Zhang, and Huajun Chen.
Sciknoweval: Evaluating multi-level scientific knowledge of large language models, 2024.
URL [https://arxiv.org/abs/2406.09098](https://arxiv.org/abs/2406.09098).
- Murzin et al. [1995]
Alexey G. Murzin, Steven E. Brenner, Tim Hubbard, and Cyrus Chothia.
Scop: A structural classification of proteins database for the investigation of sequences and structures.
*Journal of Molecular Biology*, 247(4):536–540, 1995.
ISSN 0022-2836.
doi: https://doi.org/10.1016/S0022-2836(05)80134-2.
URL [https://www.sciencedirect.com/science/article/pii/S0022283605801342](https://www.sciencedirect.com/science/article/pii/S0022283605801342).
- Berman et al. [2000]
Helen M. Berman, John Westbrook, Zukang Feng, Gary Gilliland, T. N. Bhat, Helge Weissig, Ilya N. Shindyalov, and Philip E. Bourne.
The protein data bank.
*Nucleic Acids Research*, 28(1):235–242, 01 2000.
ISSN 0305-1048.
doi: 10.1093/nar/28.1.235.
URL [https://doi.org/10.1093/nar/28.1.235](https://doi.org/10.1093/nar/28.1.235).
- Szklarczyk et al. [2023]
Damian Szklarczyk, Rebecca Kirsch, Magdalini Koutrouli, Katerina C Nastou, Farhad Mehryary, Rachid Hachilif, Annika L Gable, Tao Fang, Nadezhda T Doncheva, Sampo Pyysalo, et al.
The string database in 2023: protein–protein association networks and functional enrichment analyses for any sequenced genome of interest.
*Nucleic Acids Research*, 51(D1):D638–D646, 2023.
doi: 10.1093/nar/gkac1000.
- Oughtred et al. [2021]
Rose Oughtred, Jenna Rust, Cecilia Chang, Bobby-Joe Breitkreutz, Chris Stark, Anjali Willems, Lorrie Boucher, Gabrielle Leung, Natalia Kolas, Frank Zhang, et al.
The biogrid database: A comprehensive biomedical resource of curated protein, genetic, and chemical interactions.
*Protein Science*, 30(1):187–200, 2021.
doi: 10.1002/pro.3978.
- Wang et al. [2024]
Bowen Wang, Jiuyang Chang, Yiming Qian, Guoxin Chen, Junhao Chen, Zhouqiang Jiang, Jiahao Zhang, Yuta Nakashima, and Hajime Nagahara.
Direct: Diagnostic reasoning for clinical notes via large language models.
In *Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track*, 2024.
URL [https://proceedings.neurips.cc/paper_files/paper/2024/file/892850bf793e03b5c410dfd9425b94c8-Paper-Datasets_and_Benchmarks_Track.pdf](https://proceedings.neurips.cc/paper_files/paper/2024/file/892850bf793e03b5c410dfd9425b94c8-Paper-Datasets_and_Benchmarks_Track.pdf).
- Yuan et al. [2024]
Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, and Gongshen Liu.
R-judge: Benchmarking safety risk awareness for llm agents.
*arXiv preprint arXiv:2401.10019*, 2024.
URL [https://arxiv.org/abs/2401.10019](https://arxiv.org/abs/2401.10019).
- Li et al. [2025]
Ang Li, Yin Zhou, Vethavikashini Chithrra Raghuram, Tom Goldstein, and Micah Goldblum.
Commercial llm agents are already vulnerable to simple yet dangerous attacks, 2025.
URL [https://arxiv.org/abs/2502.08586](https://arxiv.org/abs/2502.08586).
- Andriushchenko et al. [2024]
Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, Eric Winsor, Jerome Wynne, Yarin Gal, and Xander Davies.
Agentharm: A benchmark for measuring harmfulness of llm agents.
*arXiv preprint arXiv:2410.09024*, 2024.
URL [https://arxiv.org/abs/2410.09024](https://arxiv.org/abs/2410.09024).
- Efron and Tibshirani [1994]
Bradley Efron and Robert J Tibshirani.
*An introduction to the bootstrap*.
Chapman and Hall/CRC, 1994.
- Altman and Bland [2005]
Douglas G Altman and J Martin Bland.
Standard deviations and standard errors.
*Bmj*, 331(7521):903, 2005.
- Bayat et al. [2024]
Farima Fatahi Bayat, Lechen Zhang, Sheza Munir, and Lu Wang.
Factbench: A dynamic benchmark for in-the-wild language model factuality evaluation.
*arXiv preprint arXiv:2410.22257*, 2024.
- Yang et al. [2024]
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al.
Qwen2. 5 technical report.
*arXiv preprint arXiv:2412.15115*, 2024.
- Grattafiori et al. [2024]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Oltramari et al. [2020]
Alessandro Oltramari, Cory Henson, Ruwan Wickramarachchi, Don Brutzman, and Richard Markeloff.
Hybrid ai for context understanding.
[https://us2ts.org/2020/program-hybrid-ai](https://us2ts.org/2020/program-hybrid-ai), March 2020.
Session at the 3rd U.S. Semantic Technologies Symposium (US2TS 2020), Raleigh, NC.
- Wang and Ruhe [2007]
Yingxu Wang and Guenther Ruhe.
The cognitive process of decision making.
*IJCINI*, 1:73–85, 04 2007.
doi: 10.4018/jcini.2007040105.
- Styles et al. [2024]
Olly Styles, Sam Miller, Patricio Cerda-Mardini, Tanaya Guha, Victor Sanchez, and Bertie Vidgen.
Workbench: a benchmark dataset for agents in a realistic workplace setting, 2024.
URL [https://arxiv.org/abs/2405.00823](https://arxiv.org/abs/2405.00823).
- Bettencourt [2009]
Luís M. A. Bettencourt.
The rules of information aggregation and emergence of collective intelligent behavior.
*Topics in Cognitive Science*, 1(4):598–620, 2009.
doi: https://doi.org/10.1111/j.1756-8765.2009.01047.x.
URL [https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1756-8765.2009.01047.x](https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1756-8765.2009.01047.x).
- Li et al. [2024c]
Bo Li, Irina Sigler, and Yuan (Emily) Xue.
Evaluating large language models: Principles, approaches, and applications.
[https://neurips.cc/media/neurips-2024/Slides/99524.pdf](https://neurips.cc/media/neurips-2024/Slides/99524.pdf), December 2024c.
NeurIPS 2024 Tutorial.
- Lee et al. [2024b]
Hyunji Lee, Sejune Joo, Chaeeun Kim, Joel Jang, Doyoung Kim, Kyoung-Woon On, and Minjoon Seo.
How well do large language models truly ground?, 2024b.
URL [https://arxiv.org/abs/2311.09069](https://arxiv.org/abs/2311.09069).
- Brooks [2022]
J. Brooks.
The art of problem solving and its translation into practice.
*BDJ in Practice*, 35(9):21–23, 2022.
doi: 10.1038/s41404-022-1714-y.
URL [https://doi.org/10.1038/s41404-022-1714-y](https://doi.org/10.1038/s41404-022-1714-y).
- Kwon et al. [2023]
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In *Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles*, 2023.

## Appendix A Benchmark Description

##### Task Selection Considerations.

We follow a set of rigorous task selection criteria designed to meet our objective to accessing LLMs’ capability to solve expert-level tasks:

- 1.
Human reference outputs: We only include tasks for which reliable human reference outputs are either directly available or can be accurately derived.
- 2.
Long-form outputs: We focus on tasks with open-ended and long-form outputs, avoiding simple information extraction or classification tasks that lack substantial natural language output.
- 3.
Objectivity: All selected tasks must be objective, allowing multiple annotators to reasonably agree on the correctness of a response.
- 4.
Domain expertise requirement: Tasks must also require domain expertise, both for task completion and for evaluating the outputs.
- 5.
Real-world importance and impact: We emphasize tasks with clear importance and real-world impact, particularly those that reduce expert workload or improve efficiency, accessibility, or scalability in professional workflows.
- 6.
Targeting real-world users: We ensure all tasks are designed to target realistic problem-solving scenarios.

For tasks adapted from existing datasets, we apply two additional filters:

- 1.
Accessibility: by selecting only publicly available datasets
- 2.
Recency: giving preference to datasets released recently to minimize potential contamination from model training data.

##### Key Elements of Samples.

Each sample in ExpertLongBench includes the following key elements:

- •
Task input: The context of the task. For example, the inputs for T2LegalSFG consist of relevant transcripts used to generate a Statement of Fact, with each sample on average having 14 documents totaling 70k input tokens.
- •
Human reference: Each task includes human-authored references in natural language. Exceptions include T3MaterialSEG and T9MedicalDR, for which checklist-based references are constructed during the data collection process.
- •
Checklist-mapped reference: A checklist-based, fine-grained evaluation rubric is designed for each task through collaboration with domain experts and by referencing established resources. A corresponding checklist-mapped reference is constructed for every sample, as detailed in §[3](https://arxiv.org/html/2506.01241v3#S3.SS0.SSS0.Px3) in the main paper.

##### Checklist-mapped Reference Creation.

To ensure the quality of our LLM-based checklist-mapped references, we adopt the following two strategies:
For tasks where mapping references to checklists is straightforward (e.g., T1, T2, T5, T6, T7, T8, T10), we randomly select two tasks (T1, T6) and conducted human evaluation and LLM-as-judges evaluation for ensuring the quality. Details are shown in §[4.2](https://arxiv.org/html/2506.01241v3#S4.SS2) in the main paper.
For other tasks (e.g., T4, T11), we perform manual quality assurance on the model’s extracted checklist-mapped references for all samples. This process involves human reviewers inspecting and correcting any extraction errors.
The remaining tasks (e.g., T3, T9) directly produce the checklist-mapped reference and no prompting is needed.

##### Experts Involved in Rubric Design and Verification.

We detail the experts involved in rubric design and verification as follows:

- •
T1LegalMDS: We collaborated with a JD student at an R1 university in the US who created the checklist based on an established instruction for writing high-quality legal summaries.
The JD student was compensated through supervisor funding support.
- •
T2LegalSFG: This task involved a different set of experts from T1—two JD students and one legal professor from an R1 university in the US. The JD students were also compensated via supervisor funding.
- •
T3MaterialSEG: The checklist was developed with input from a senior PhD student at an R1 university in the US specializing in material science, with extensive academic training and multiple publications, supporting the representativeness of the checklist.
- •
T4EduPAE: We adopted the checklist from an existing, validated publication, which was confirmed by a professor in learning sciences at an R1 university in the US with a strong publication record.
- •
T5EduFG: The checklist was adapted from instructional materials developed and taught over multiple years in the Econ 101 course at an R1 university in the US by a senior instructor.
- •
T7ChemMDG: The checklist was developed in collaboration with a PhD student at an R1 university in the US who holds a master’s degree and has extensive academic training along with multiple publications.
- •
T8BioPDG: Two graduate students studying biology at an R1 university in the US collaboratively created the rubric, and they reached consensus.
- •
T10FinanceESG: We consulted professionals at a leading ESG rating institution. The checklist guidelines were taken and adapted from their ESG rating methodology document and the underlying key metrics.
- •
T6HealthCNG, T9MedicalDR and T11CyberRDG: Checklists for these tasks were generated by the authors based on careful review of prior work and established relevant guidelines.

## Appendix B Task Description

### B.1 T1LegalMDS

#### B.1.1 Task Definition

Legal case summarization involves generating concise, coherent, and informative summaries from multiple legal documents pertaining to a single case.
These documents can include a wide range of materials such as complaints, appeals, court opinions, motions, filings, judgments, and other related legal records.
The objective is to summarize critical aspects of a case into a summary that is accessible to a broad audience, including legal professionals such as policymakers, advocates, researchers, educators, and students, as well as the general public.
When using LLMs to conduct the legal case summarization task, the input is typically a collection of relevant legal documents associated with a specific case and the output is a case summary that encapsulates the essential information from the original documents.

#### B.1.2 Task Significance

Legal case summarization is a foundational task for improving access to justice, supporting legal research, and facilitating informed decision-making across the legal system. Many real-world cases involve large volumes of interrelated documents. These include complaints, motions, rulings, and settlement agreements, which must be synthesized into concise, coherent summaries.
Automatic summarization addresses this issue by condensing lengthy texts into concise summaries, enabling users to quickly grasp essential details without reading the entire document [Shen et al., 2022]. This automation enhances accessibility by providing clear summaries for lawyers, scholars, and the general public, improves efficiency by assisting legal professionals in quickly understanding case details, and supports research by highlighting crucial aspects of cases. By reducing the time and effort required to comprehend extensive legal materials from legal experts, automatically generated summaries streamlines legal workflows and aids in informed decision-making. This challenge becomes even more pronounced in domains where litigation is prolonged, procedurally complex, and socially impactful.

One such domain is civil rights litigation, which has played a pivotal role in reforming public institutions and shaping policy in areas such as education, policing, incarceration, and disability rights. These cases often span years and generate hundreds of filings, yet most remain inaccessible to the broader public, scholars, and even practitioners. The Civil Rights Litigation Clearinghouse(^6^66[https://clearinghouse.net/](https://clearinghouse.net/)), a leading repository in this space, curates extensive documentation from landmark and ongoing civil rights cases—many of which have never resulted in published judicial opinions. As a result, key legal developments are often buried in unstructured documents that are difficult to access and interpret without expert knowledge. Multi-document case summarization helps bridge the accessibility gap. It enables non-experts to understand legal developments, allows legal professionals to quickly grasp the structure and outcome of complex cases, supports advocacy and policymaking by surfacing systemic patterns, and reduces the burden of reviewing lengthy legal documents, giving researchers easier access to representative case content. We collect data from Clearinghouse, with further details provided in [Section˜B.1.3](https://arxiv.org/html/2506.01241v3#A2.SS1.SSS3).

A related work, Multi-LexSum [Shen et al., 2022], also investigates the task of multi-document legal case summarization in civil rights litigation domain using data from Clearinghouse. However, their study relies on automatic evaluation metrics such as ROUGE-{1,2,L} [Lin, 2004] and BERTScore [Zhang et al., 2019].
In contrast, our work constructs a new dataset by collecting cases data from the Clearinghouse and introducing clear selection criteria with legal researchers grounded in both diversity and difficulty, where diversity refers to the range of case topics and difficulty is based on structural factors such as such as the number of documents, length, and complexity of legal proceedings.
Furthermore, we develop a fine-grained, checklist-based evaluation rubric in collaboration with a legal researcher to ensure comprehensive coverage of key case elements. To assess model performance more reliably, we design a structured evaluation pipeline leveraging LLMs as evaluators, enabling nuanced analysis beyond surface-level lexical similarity.

#### B.1.3 Data Acquisition and Preprocessing

We collected multi-document legal case summarization data with 1,393 legal case samples from a widely recognized legal repository, Clearinghouse.
We obtained the case data through the official Clearinghouse API(^7^77[https://clearinghouse.net/api](https://clearinghouse.net/api)), which provides structured access to metadata and associated documents for civil rights litigation cases. For documents available in PDF format, we used the ocrmypdf(^8^88[https://github.com/ocrmypdf/OCRmyPDF](https://github.com/ocrmypdf/OCRmyPDF)) tool to extract text from scanned legal filings, ensuring complete retrieval of document content.
Each sample includes multiple case documents. Alongside these inputs, the dataset provides expert-crafted, gold-standard summaries that distill the key facts, legal principles, and outcomes of each case. These summaries are produced by trained legal experts—including lawyers and law students—who follow detailed annotation guidelines. To further ensure consistency and accuracy, each summary undergoes an additional round of expert review.
We then selected a small, high-quality and representative subset from the large-scale dataset to evaluate the ability of LLMs in generating legal summaries.

For representative data selection, we filter the data based on the following standards to select a high-quality dataset.

- •
Diversity: We obtain the topic for each case on the Clearinghouse website and select cases spanning as many distinct topics as possible. The selected data covers diverse case types such as “Equal Employment”, “Prison Conditions”, “Election/Voting Rights” which contain the specific topic the legal case is dealing with.
- •
Difficulty: We identify several factors influencing difficulty after discussing with legal researchers. The factors are as follows:
–
Length of the documents
–
Number of appeals: Appeals are processes that the litigant asks for the higher court review of the lower court’s decision, we obtain the number of appeals for each case from the Clearinghouse website.
–
Number of complaints: Complaints are documents that start the lawsuit. We obtain the number of complaints for each case from the Clearinghouse website.
–
Number of dockets: Dockets are logs containing a comprehensive history of the case in chronological order. We obtain the number of dockets for each case from the Clearinghouse website.

After this process, we select 454 samples with a high difficulty level.

To further ensure the quality of the gold-standard case summary, we discuss with legal researchers and identify the required checklist items that must be included in the summary. We then use GPT-4o as a judge to assess whether the human-written reference covers the information and select the cases with highest-quality human summaries, determined by the highest coverage of checklist items. The specific prompt used for accessing the reliability of human-written summary is
shown in [Tab.˜3](https://arxiv.org/html/2506.01241v3#A2.T3). Based on the judgment results, we finally select 100 samples with high-quality and long human reference summaries.

Table: Table 3: T1LegalMDS - Prompt for checking the quality of human-written references.

#### B.1.4 Illustrative Example

An illustrative example is shown in [Tab.˜4](https://arxiv.org/html/2506.01241v3#A2.T4).

The sample input for T1 consists of multiple legal case documents from a class-action lawsuit concerning COVID-19 conditions in immigration detention facilities. These include court filings such as status reports, motions, court orders, and objections, all focused on health and safety concerns at the Mesa Verde Detention Facility and the Yuba County Jail during the pandemic. Due to the length of the original documents, we provide only brief descriptions of each one’s purpose and topic. The human reference output is an expert-curated summary that concisely captures the key legal developments and arguments in the case. It outlines the plaintiffs’ claims regarding unsafe detention conditions, the legal basis for the lawsuit, major court decisions (e.g., temporary restraining orders and preliminary injunctions), and procedural milestones such as appeals and mediation efforts.

Additionally, [Tab.˜5](https://arxiv.org/html/2506.01241v3#A2.T5) presents the model prompt. The prompt was carefully constructed to be high-level and general instead of providing the exact checklist items. The inputs for this prompt are the documents shown in [Tab.˜4](https://arxiv.org/html/2506.01241v3#A2.T4).

Table: Table 4: T1LegalMDS - A sample of case documents and the corresponding case summary.

Table: Table 5: T1LegalMDS - Model prompt.

#### B.1.5 Evaluation Rubric

We closely collaborated with legal researchers to design a checklist-based evaluation rubric to assess the helpfulness of legal case summaries.
The process of designing, refining, and finalizing the rubric takes roughly 11 hours.
This rubric comprises of 26 checklist items that capture key aspects to consider during the evaluation process. Some items are marked as “if applicable”, meaning they should only be considered when the case pertains to the corresponding checklist criteria. This terminology will be used in other task checklists as well. The detailed checklist items are listed as follows:

- 1.
Filing Date
- 2.
Class Action or Individual Plaintiffs? (if applicable): If there are class action plaintiffs the summary should say it’s a class action; if there are individual plaintiffs it can just describe the plaintiffs. For example, use specific terms like “The city” or “The parents” rather than general terms like “The defendant” or “The plaintiffs.”
- 3.
Cause of Action: e.g., a statute (e.g., 42 USC 1983) or a case (e.g., Ex Parte Young)
- 4.
Statutory or Constitutional Basis for the Case: A case can either be based on a statute or a provision of the Constitution—i.e., a case will either claim that someone violated a statute, or violated the Constitution. For cases that have a constitutional basis, the summary should refer to the clause of the Constitution that was allegedly violated, as well as the amendment if applicable. For example it would say “the plaintiffs alleged violations of the Fourteenth Amendment’s Equal Protection Clause,” or “the plaintiffs alleged violations of the Commerce Clause.”
- 5.
Remedy Sought: e.g., declaratory judgment
- 6.
Who are the parties (description, not name)?
- 7.
Type of Counsel: type of counsel contains private, legal services, ACLU, etc.
- 8.
Consolidated Cases Noted (if applicable)
- 9.
Related Cases listed by their case code number (if applicable)
- 10.
Note important filings (if applicable): Note important filings including motions for temporary restraining orders or preliminary injunctions, motions to dismiss, motions for summary judgment, etc.
- 11.
All reported opinions cited with shortened Bluebook citation (if applicable): For example, the summary could write “2020 WL 4218003” after the paragraph in which it discusses that opinion. The summary does not need to include the case name, court, or date unless helpful—such as when the summary cites an opinion from a different case.
- 12.
First and Last Name of Judge: Form: Judge John Smith. Find judge’s first names at http://www.fjc.gov/public/home.nsf/hisj
- 13.
Significant Terms of Decrees (if applicable): Significant terms means the substance of the decree or settlement. In a decree, the judge orders the defendants to do something; in a settlement, the defendants agree to do something. The significant terms would be what the defendants are ordered/agree to do.
- 14.
Dates of All Decrees (if applicable)
- 15.
How long decrees will last (if applicable)
- 16.
Significant Terms of Settlement (if applicable)
- 17.
Date of settlement (if applicable)
- 18.
How long settlement will last (if applicable)
- 19.
Whether the settlement is court-enforced or not
(if applicable)
- 20.
Was there a monitor? Note the name of the monitor (if applicable)
- 21.
Monitor’s Reports (if applicable): A monitor’s report explains whether a defendant is complying with a court order, so people want to know which terms of the order are being complied with. For example, from this case: In February 2016, the monitor filed her first semi-annual report. The report stated that the payment to the plaintiffs and plaintiffs counsel was made; the requirement of hiring ADA Coordinators was nearly compliant; videophone installation is apparently compliant; free access to videophone, provision of qualified interpreters for unscheduled medical emergencies, and provision of qualified interpreters for disciplinary hearings were unclear; informational materials were partially compliant; the routine and situational reporting were difficult or partially noncompliant; and training was noncompliant.
- 22.
Appeal (if applicable)
- 23.
Trials (if applicable)
- 24.
Court rulings on any of the important filings (if applicable): This category corresponds with the “important filings” category—so whenever an important filing is mentioned, people also want to know what the ruling on that filing was (if there is one)—e.g., whether the judge granted or denied a motion to dismiss. Generally these filings would be:
Motions to dismiss,
Motions for summary judgment,
Motions for a preliminary injunction or temporary restraining order,
Motions for class certification,
Motions for attorneys’ fees,
Amended complaints–these won’t have rulings, so they should be in the “important filings” category but not the “rulings on to important filings” category, statements of interest–similar to above, there won’t be rulings on these.
- 25.
Factual basis of case: Refers to the facts or evidence upon which the case is built. These facts are essential in the legal process and are used to support legal claims or decisions. It typically includes:
1. Details of the relevant events—For example, what happened, when it happened, where it happened, and who was involved.
2. Evidence – Physical evidence, documentary records, witness testimonies, etc., that support these facts.
3. Background information—Context or explanatory facts that provide additional understanding.
In legal proceedings, the factual basis is crucial for determining the outcome of a case, as the judge or jury makes decisions based on the facts and the applicable legal principles.
- 26.
Disputes over settlement enforcement (if applicable)

#### B.1.6 Checklist-mapped Reference

We created checklist-mapped references for the human reference and the model output based on our 26-item checklist.

For the human reference, we used GPT-4o to extract answers to each checklist item individually. Each prompt targeted a single item and followed a structured format. To improve contextual grounding, we used role-playing to frame the model as an assistant to a lawyer. If no related information was mentioned in the legal case summary, the model should return “N/A” to reduce the possibility of hallucination. An example is shown in [Tab.˜6](https://arxiv.org/html/2506.01241v3#A2.T6).

For the model output, we use Qwen2.5-72B to create the checklist-mapped reference. Our decision to use a different model is supported in §[4.1](https://arxiv.org/html/2506.01241v3#S4.SS1). To reduce cost while maintaining quality, we grouped the checklist items rather than prompting for them individually. We divide the 26 items into five groups, each with four to six checklist items, based on the average expected length of the model’s response. Each model output prompt used the same template, role-playing format as the human reference prompt. Due to space constraints, we will release the full prompt set in our public GitHub repository.

The checklist-mapped reference of the example in [Tab.˜4](https://arxiv.org/html/2506.01241v3#A2.T4) is presented in [Tab.˜7](https://arxiv.org/html/2506.01241v3#A2.T7). To assess the quality of the checklist-mapped reference, we selected an additional 30 difficult and diverse samples, different from the main dataset, and conducted human verification to examine the faithfulness of the model response. Details of this experiment can be found in [Section˜E.1](https://arxiv.org/html/2506.01241v3#A5.SS1).

Table: Table 6: T1LegalMDS - Prompt for extracting checklist-mapped reference.

Table: Table 7: T1LegalMDS - Checklist-mapped reference.

### B.2 T2LegalSFG

#### B.2.1 Task Definition

A Statement of Fact (SOF) is a document used in legal cases to persuade the judge of a certain viewpoint. It outlines the events and circumstances leading up to a legal dispute in an objective manner, while also incorporating persuasive language when appropriate [LSD Law, n.d.]. The objective is to generate a comprehensive account of the case, based on courtroom transcripts, that accurately reflects the its development. The input includes transcripts from multiple proceedings, such as preliminary trials, adjudication hearings, and termination trials. The output should present all relevant facts, direct quotations, and procedural details with proper attribution and clear alignment to legal standards. It must stand alone as a complete summary, without requiring reference to the original transcripts.

#### B.2.2 Task Significance

SOFs play an crucial role in improving clarity, consistency, and accessibility in the documentation of court proceedings. They are a part of a legal brief, formal documents submitted to a court that outline the relevant facts, applicable laws, and legal arguments of a case. Our samples are specific to in child welfare and termination of parental rights cases. These cases often involve complex and emotional content that must be translated into reliable and grounded summaries.

Traditionally, drafting such briefs is a time-intensive and expert-level task. Attorneys must review hundreds of pages of transcripts to extract and organize key information into create an coherent narrative. This process often takes 20 to 40 hours or more, even for seasoned professionals [TypeLaw, 2021]. Manual note-taking adds further difficulty - often leading to inconsistent phrasing, missed information or subjective interpretation. It also requires frequent references to transcripts, which not only slows progress, but increases the risk of delays and oversights in case processing.

Automating this process with LLMs addresses these challenges by generating detailed and structured briefs. They are especially crucial in child welfare systems, where timely and accurate information directly impacts case decisions and the well-being of children. In this task, transcripts are drawn from real juvenile court proceedings provided by our collaborators, and outputs are compared against a detailed rubric developed with experts to ensure both completeness and legal relevance.

#### B.2.3 Data Acquisition and Preprocessing

Domain experts provided us with 113 samples of transcripts in Word document or PDF form. We first processed all documents using a custom script to extract clean, structured text from each file, regardless of format. The script handles .docx, .doc, and .pdf files and includes fallback methods to ensure robustness across formatting inconsistencies.

For .pdf files, we first attempted direct text extraction using PyPDF2(^9^99[https://pypi.org/project/PyPDF2/](https://pypi.org/project/PyPDF2/)). If the PDF lacked embedded text (e.g., scanned images), we applied OCR using OCRmyPDF(^10^1010[https://github.com/ocrmypdf/OCRmyPDF](https://github.com/ocrmypdf/OCRmyPDF)) with deskewing, noise cleaning, and compression optimizations. This was the library recommended to us by our collaborators.

For .docx files, we used the python-docx library(^11^1111[https://pypi.org/project/python-docx/](https://pypi.org/project/python-docx/)) to extract paragraph-level content. For older .doc files, we employed a fallback approach using textract(^12^1212[https://pypi.org/project/textract/](https://pypi.org/project/textract/)), antiword(^13^1313[http://www.winfield.demon.nl/](http://www.winfield.demon.nl/)), or conversion to .docx using LibreOffice in headless mode(^14^1414[https://www.libreoffice.org/](https://www.libreoffice.org/)).

Each line of extracted text was then cleaned to remove extraneous whitespace and empty lines, ensuring that only content-relevant information was retained. However, page numbers were retained in the final text as the model prompt asked the LLM to generate SOFs that cited the transcript’s page numbers. Cleaned output was saved in plain .txt files, one per document. For each case, all .txt files were then concatenated into a single input string in the format: *transcript 1 name: transcript 1 content, transcript 2 name: transcript 2 content … *. This structured format served as the sample input.

To filter 113 transcript samples, we selected the 100 with the longest human-written reference SOFs, consistent with the reasoning in §3. On average, the 100 samples included 14 transcript documents, each 5K+ tokens in length, and paired with a human reference SOF also exceeding 5K tokens. This filtering ensured that our dataset contained the most information-dense and well-documented cases.

#### B.2.4 Illustrative Example

Due to the proprietary nature of the data, we cannot provide a specific example. However, in [Tab.˜8](https://arxiv.org/html/2506.01241v3#A2.T8), we provide the model prompt for obtaining model outputs.

Table: Table 8: T2LegalSFG - Model prompt.

#### B.2.5 Evaluation Rubric

We worked with three domain experts and drafted through three iterations of the checklist through discussion. It takes approximately 8 hours to design, revise, and finalize the rubric. The experts designed the checklist items and helped categorize each checklist item into general levels and provided brief definitions of some levels. It is important to note that some checklist items are conditional and depend on the responses to previous items. For instance, Items 26 and 27 correspond to different outcomes based on the response to Item 25. If the parent pled to adjudication, Item 26 should be completed and Item 27 will be “N/A”. If there was an adjudication trial instead, Item 27 should be used and Item 26 will be “N/A”. The detailed checklist items are listed below:

- •
Initial incident
1.
When DHHS first made contact
2.
What caused DHHS to become involved
3.
Discrepancies between DHHS’ account and the parent’s account
- •
Background situation
–
We want to know any facts about the parent and children that can help the court understand their situation more favorably. Examples can include a parent having an unstable home environment growing up, a parent having an abusive partner, a child having disabilities, etc.
4.
Challenges the parent faced when they were young
5.
Problems the parent currently faces
6.
Child’s special needs
- •
How the child is doing
7.
How the child(ren) is doing
8.
Child(ren)’s development
9.
Child(ren)’s interactions with the parent
10.
Child(ren) talked about their feelings or experience
- •
Impact of each court event
11.
Decision of judge
12.
New actions the parent or agency has to do
13.
Actions the parent or agency does not have to do anymore
- •
Details of service plan
–
In these cases, the court typically orders the parent to complete some requirements so they can get custody of their child back. Examples of “services” can include parenting classes, therapy, drug screens, etc.
14.
Reasonable efforts to reunify
15.
If the agency was not ordered to make reasonable efforts to reunify,what aggravating circumstances were present
16.
Services the court required the parent to complete
17.
For each service, what the service means
18.
For each service, what objectives the parents have to meet
- •
Parent progress on service plan
19.
How the parent complied with the services
20.
How the parent did not follow through with the services
21.
Parent’s compliance or noncompliance with the service plan
- •
Agency role in service plan
–
The child welfare agency might help the parent with compliance with things like transportation help and therapy referral. But the government can also hurt the parent with things like delays and gaps in communication.
22.
How the agency tried to help the parent get the services they need
23.
How the agency or government ignored a parent or undermined their efforts
- •
Adjudication hearing
24.
When the adjudication hearing was held
25.
An adjudication trial or the parent pled to adjudication
26.
If the parent pled to adjudication, the specific allegations the parent pled to
27.
If there was an adjudication trial, what statutory grounds for adjudication were found
- •
Permanency planning hearings
28.
When each permanency planning hearing was held
29.
What permanency plan was established
30.
Court’s reasons for the chosen permanency plan
31.
Whether DHHS was ordered to initiate termination of parental rights
32.
If the child has been in foster care for more than 15 of the last 22 months, and the court did not order DHHS to initiate termination proceedings, the court’s reasoning
- •
Termination of parental rights hearings
33.
Statutory grounds DHHS sought termination
34.
Statutory grounds the court found that termination was proper
35.
Evidence offered to show statutory grounds for termination were not met
36.
Evidence offered to show termination was in the child’s best interests
37.
Evidence offered to show termination was not in the child’s best interests
38.
If the child was placed with a relative, whether any parties discussed that placement weighing against termination
39.
Evidence regarding what kind of permanency the child will have if the parent’s rights are terminated
40.
Alternatives to termination of parental rights
- •
Last sentence
41.
Ends with the final lower-court ruling

#### B.2.6 Checklist-mapped Reference

The creation of the checklist-mapped reference is for T2 is similar to T1’s process in described in [Section˜B.1.6](https://arxiv.org/html/2506.01241v3#A2.SS1.SSS6).

However, unlike in T1, the checklist items used to construct the model output checklist-mapped reference are grouped differently. We divide the 41 items into nine groups based on: (1) the average expected length of the model’s response, and (2) the topical category of the item (e.g., Details of service plan, Impact of each court event, etc.). For instance, we grouped checklist items 1–6 together, as they all fall under the categories *Initial incident* and *Background situation*. This grouping strategy helps the model process large inputs more effectively by allowing it to focus on fewer sections at a time. The prompt structure for extracting an individual checklist item is illustrated in [Tab.˜9](https://arxiv.org/html/2506.01241v3#A2.T9).

Due to the proprietary nature of the data, we cannot provide a specific example for the checklist-mapped reference.

Table: Table 9: T2LegalSFG - Prompt for extracting checklist-mapped reference.

### B.3 T3MaterialSEG

#### B.3.1 Task Definition

Materials science research involves synthesizing new materials and understanding their properties.
A synthesis recipe includes a series of steps for creating a target material, specifying the precursors and synthesis conditions used.
In the task of synthesis explanation generation, the objective is to provide justifications for key decisions made in the recipe, considering factors such as the precursors’ structural motifs, reactivity, thermodynamic stability, and more.

#### B.3.2 Task Significance

A material can often be synthesized through multiple methods, and the choice of precursors and synthesis conditions can significantly affect the properties and yields of the resulting material [Sun and David, 2025].
Understanding the reasoning behind these choices is essential for materials scientists aiming to optimize synthesis protocols and develop new materials with desirable properties through more efficient processes.
Additionally, materials discovery efforts, especially those aided by high-throughput computations [Curtarolo et al., 2013, Kononova et al., 2019], can be accelerated by prioritizing synthesis recipes that are grounded in the most promising mechanistic insights.
Per our discussion with materials science researchers, writing the explanations for a synthesis recipe from scratch would take a well-trained materials science PhD student 1-2 hours.

#### B.3.3 Data Acquisition and Preprocessing

Despite the availability of large-scale synthesis recipe datasets [Kononova et al., 2019, Jain et al., 2013], these curated recipes are not accompanied by explanations.
Even for well-trained materials science PhDs, writing explanations for synthesis recipes from scratch is challenging and time-consuming.
In contrast, given statements about the synthesis recipe, undergraduate-level chemistry students are able to distinguish between explanations and non-explanations.
Therefore, we adopt a semi-automated approach to collect synthesis recipes along with corresponding explanations from existing literature.

We begin by consulting materials science researchers(^15^1515They are PhD candidates in materials science.) to identify the key aspects that explanations should address.
Since papers in the field do not always explicitly state the rationale behind synthesis choices, we begin by identifying papers that are more likely to include such explanations.
To do this, we extract synthesis recipes and potential explanations using Llama-3.3-70B-Instruct with carefully designed instructions.(^16^1616We also experimented with in-context learning, but found that Llama-3.3-70B-Instruct often copied from the in-context examples rather than extracting content from the target paper.)
We then rank the papers based on the number of aspects covered.
Starting from the top-ranked papers, we use GPT-4o to further extract synthesis recipes and corresponding explanations.
Based on GPT-4o’s outputs, annotators with undergraduate-level chemistry knowledge verify the extracted recipes and collect accurate explanations.
When explanations are absent from the paper, annotators are asked to discard the paper and move on to the next one, until the target number (50) of samples is collected.

We retrieve papers containing the keyword “solid-state synthesis” from Science(^17^1717[https://www.science.org](https://www.science.org)) and Wiley(^18^1818[https://onlinelibrary.wiley.com](https://onlinelibrary.wiley.com)), as the aspects of interest are more relevant to solid-state synthesis.
The text of the paper is extracted directly from the HTML version of the paper.
During the automated extraction step, we generate five different outputs using GPT-4o for each paper to increase the recall of possible explanations.
To improve the faithfulness of the extraction, we also prompt the model to provide the source sentences corresponding to each explanation.
The prompt is shown in [Tab.˜10](https://arxiv.org/html/2506.01241v3#A2.T10), which is based on the rubric designed in [Section˜B.3.5](https://arxiv.org/html/2506.01241v3#A2.SS3.SSS5).
Feedback from the annotators indicates that some explanations extracted by the LLMs describe properties of the synthesized materials (e.g., crystal structure, conductivity) rather than justifying the synthesis steps themselves.

Table: Table 10: T3MaterialSEG - Prompt for automatically extracting synthesis recipes and explanations.

#### B.3.4 Illustrative Example

An illustrative example is shown in [Tab.˜11](https://arxiv.org/html/2506.01241v3#A2.T11). All samples for T3 are publicly available.

The sample input is a synthesis recipe for a complex oxide thin film. The recipe describes the starting precursors, key thermal steps, and atmospheric conditions used during synthesis. This format mirrors the language and structure typically found in materials science literature and experimental protocols. The human reference output provides explanations for why each major synthesis step and condition was chosen. It should help researchers understand the scientific rationale behind the recipe, not just its procedural steps.

Additionally, [Tab.˜12](https://arxiv.org/html/2506.01241v3#A2.T12) presents the model prompt.

Table: Table 11: T3MaterialSEG - A sample of a synthesis recipe and its corresponding explanations.

Table: Table 12: T3MaterialSEG - Model prompt.

#### B.3.5 Evaluation Rubric

We work with a PhD student in material science, whose research focuses on solid-state synthesis.
They provide us with five papers(^19^1919[doi.org/10.1002/ejic.202100901](doi.org/10.1002/ejic.202100901), [doi.org/10.1002/kin.21467](doi.org/10.1002/kin.21467), [doi.org/10.1002/smll.202206248](doi.org/10.1002/smll.202206248), [doi.org/10.1126/sciadv.adj5431](doi.org/10.1126/sciadv.adj5431), [doi.org/10.1126/sciadv.adp3309](doi.org/10.1126/sciadv.adp3309)) that contain well-written explanations for synthesis recipes.
We prompt GPT-4o, Claude-3.7-Sonnet, and Gemini-2.0-Flash to summarize the aspects of the explanations covered in these papers.
The PhD student then reviews the aspects summarized by the three models, revises them, and adds additional key aspects that are not covered by the models.
The final rubric includes six items:

- •
Selection of Precursors
1.
Structural Considerations: Justify precursor selection by explaining how the precursor’s structural motifs (e.g., coordination environments, lattice arrangement) influence the target phase formation.
2.
Handling Precursor Reactivity: Justify precursor selection by explaining the impact of precursor reactivity on phase evolution.
3.
Physical and Chemical Properties of Precursors: Justify precursor selection by addressing how precursor properties (e.g., particle size, morphology) influences reaction kinetics and product morphology.
- •
Synthesis Conditions
4.
Temperature and Heating Method: Justify the choice of synthesis temperature and heating method (e.g., based on thermodynamic considerations, reaction kinetics, heat transfer efficiency, or side reactions).
5.
Atmosphere: Justify the choice of synthesis atmosphere environment (e.g., based on thermodynamic considerations, reaction kinetics, or side reactions).
6.
Duration: Justify the choice of synthesis duration (e.g., based on reaction kinetics, phase transformation rates, or side reactions).

#### B.3.6 Checklist-mapped Reference

Synthesis explanation generation does not involve the checklist mapping process, as the annotation process directly produces the reference checklist.
An example checklist-mapped reference is shown in the illustrative example in [Tab.˜11](https://arxiv.org/html/2506.01241v3#A2.T11).

### B.4 T4EduPAE

#### B.4.1 Task Definition

Assessing pedagogical alignment involves evaluating how well a large language model (LLM) can replicate effective teaching strategies, such as offering scaffolded guidance without directly revealing the answer [Paul, 1995, Chi, 2009, Hattie and Timperley, 2007, Sonkar et al., 2024]. One possible approach is to evaluate LLMs through real-world interactions with students. However, this poses challenges, as student responses can vary unpredictably across different models [Gao et al., 2025]. Additionally, using the same group of students to test multiple LLMs introduces complications: once students are exposed to a topic in one session, their familiarity with the material can influence their performance in subsequent sessions, making it difficult to isolate the effectiveness of each system. Conversely, using different students for the same set of topics may lead to non-standardized assessments due to variations in their prior knowledge and skill levels. The objective is to assess whether LLMs can provide pedagogically sound feedback that supports student learning without directly giving away answers.

To provide a repeatable and standardized framework for evaluating the pedagogical abilities of different LLMs, we propose a task focused on assessing the effectiveness and pedagogical alignment of feedback within a tutor-student dialogue. Each task instance presents a multi-turn conversation in which a student attempts to solve a biology problem with the guidance of a tutor. The tutor employs a step-by-step problem-solving strategy, breaking the larger problem into smaller sub-problems and promoting active learning by guiding the student through each segment [Chi, 2009]. This process is sequential: the student addresses one sub-problem at a time before moving on to the next. The conversational context includes the full dialogue history up to the current sub-problem, concluding with the student’s latest response.

For each instance, the evaluated LLM is required to carry out the following: (1) assess the accuracy of the student’s response using three coarse-grained categories—accurate, partially accurate, and inaccurate; (2) determine whether the student is experiencing difficulty with the current sub-problem by detecting repeated errors across multiple attempts; and (3) provide constructive, pedagogically sound feedback that both diagnoses the underlying misconceptions and supports the student in resolving them and (4) initiate a transition to the next sub-problem, if applicable, once the student’s response is deemed satisfactory. When the student has made only a single error on a sub-problem, the feedback should be indirect and refrain from revealing the correct answer. Conversely, in cases of repeated mistakes, the feedback may include direct identification of the error and explicit corrective guidance, even if that entails disclosing the correct solution. This pedagogical plan is provided to each of the models being evaluated and we assess their ability in adhering to these instructions.

#### B.4.2 Task Significance

Given their remarkable capabilities across a wide range of language processing and knowledge-intensive reasoning tasks, LLMs hold great promise as powerful tools for delivering effective learning experiences through intelligent tutoring systems. However, their application in educational contexts faces two key limitations.

First, LLMs may produce inaccurate reasoning [Li et al., 2023a] or rely on incorrect information [Razafinirina et al., 2024] when generating feedback, which can hinder rather than support student learning. Second, while LLMs are generally optimized to be helpful and harmless, the operational definition of “helpfulness” may not align with pedagogical goals. Specifically, a fundamental strategy in conceptual learning is to guide students through problem-solving processes without directly revealing the answer. In contrast, LLMs are often trained to prioritize immediate assistance, which may conflict with the educational objective of fostering deep understanding through indirect guidance.

To address these concerns, our task aims to evaluate several critical dimensions of an LLM’s pedagogical capacity:
(1) Its effectiveness in adhering to an explicit pedagogical plan;
(2) Its ability to apply domain-specific knowledge to accurately interpret student responses and determine appropriate next steps;
(3) Its capacity to comprehend the conversational context and identify the current sub-problem being addressed;
(4) Its skill in offering feedback that accurately identifies and corrects errors; and
(5) Its ability to formulate feedback that encourages active learning through indirect guidance rather than direct answers.

#### B.4.3 Data Acquisition and Preprocessing

We first provide a background on the source of the data that will be used for building this task. We used the multi-turn student-tutor conversations simulated using GPT-4o by [Sonkar et al., 2023]. These dialogues are grounded in the socio-constructivist model of learning [Stone, 1998], where the tutor adopts a supportive and encouraging tone while offering feedback through Socratic questioning and indirect hints, rather than direct instruction. An illustrative example of such a conversation is provided in [Tab.˜15](https://arxiv.org/html/2506.01241v3#A2.T15). Each simulated interaction features a student working through a complex biology problem that requires multi-step reasoning and problem decomposition, with the tutor guiding the student through the process, making the dataset well-suited for our evaluation. Human annotators validated each simulated example to ensure factual accuracy and alignment with the socio-constructivist learning model.

The original authors used evaluation criteria that measured the accuracy of information (factual correctness), the relevance of the feedback to the current sub-problem and student errors (relevancy), the thoroughness of feedback in addressing all aspects of the sub-problem (completeness), and the impact of feedback on maintaining student interest (motivation).

In contrast, we concentrate solely on the non-affective components of the feedback, as prior studies have shown minimal influence of positive language and praise on student outcomes [Kluger and DeNisi, 1996, Ferris, 1997]. Our primary objective is to develop evaluation criteria that assess the effectiveness of feedback in promoting conceptual understanding. Studies have consistently found that feedback elements which identify problems and errors and propose solutions are the most beneficial for student learning [Hayes et al., 1987, Matsumura et al., 2002, Bitchener et al., 2005, Sugita, 2006]. Accordingly, our task is designed to evaluate an LLM’s ability to generate feedback that provides accurate problem diagnosis and rectification. While indirect feedback—such as hints or questions that avoid explicitly identifying the mistake or providing the answer—can foster meta-cognitive and conceptual learning, its effectiveness may diminish when students repeatedly struggle with the same sub-problem [Westmacott, 2017]. Accordingly, our evaluation also emphasizes the LLM’s ability to generate contextually appropriate feedback based on the nature of the student’s mistakes for a given sub-problem. In cases where the student makes repeated errors on the same sub-problem, the LLM should accurately recognize this pattern and respond with explicit error identification and corrective guidance. In this section, we explain how to get fine-grained information from the seed data collected by [Sonkar et al., 2023].

In the next subsection, we provide a brief overview of the relevant information contained in the seed data. In the following subsection, we describe how to extract relevant elements from this data to support evaluation based on our proposed criteria. Finally, we explain our procedure to sample a representative data covering diverse scenarios from this dataset.

##### Information Elements in Seed Data:

Consider a single response of the tutor from the example shown in [Tab.˜15](https://arxiv.org/html/2506.01241v3#A2.T15). We focus on the second response from the tutor and provide fine-grained information associated with it in [Tab.˜13](https://arxiv.org/html/2506.01241v3#A2.T13).

Table: Table 13: T4EduPAE - A sample of the information elements annotated by the authors.

The Evaluation of the Student Response field assesses the accuracy of the student’s most recent answer. Although the original seed data includes over eight categories for this entry, we focus on the following three: (a) Incorrect response, (b) Correct response, and (c) Partially correct response.

The Subproblem field indicates which sub-problem the feedback addresses. If the student’s response is correct, the feedback typically prompts them to proceed to a new sub-problem, and the Subproblem field reflects this next target. Conversely, if the response is incorrect or partially correct, the feedback remains focused on the current sub-problem, and the Subproblem entry corresponds to that same step.

##### Processing Data for Evaluating along our desired criteria:

To assess effectiveness in error identification and rectification, we extract these components from the feedback generated by the simulated tutor. This extraction is carried out using GPT-4o for automated annotation, with subsequent human verification to ensure the accuracy of the identified issues and corresponding corrective suggestions. Since the task does not require deep domain-specific knowledge, the manual verification was performed by one of the authors with strong proficiency in English. An illustrative example of the extracted elements from a single data point is provided in [Tab.˜14](https://arxiv.org/html/2506.01241v3#A2.T14).

Table: Table 14: T4EduPAE - A sample of the information elements extracted using GPT-4o.

To annotate whether a student has made repeated mistakes on a specific subproblem, we use the Evaluation of the Student Response and Subproblem fields to infer this state. Specifically, if the Subproblem value remains the same across two consecutive tutor responses and both responses evaluate the student’s answers as incorrect or partially correct, it indicates that the student is repeatedly struggling with the same subproblem.

Data Sampling: We sampled a total of $109$ conversation instances, ensuring that each context corresponds to a distinct problem. To study the impact of dialogue length, we selected approximately equal numbers of examples for conversation lengths in the set $\{5,7,\ldots,13\}$. Each length is an odd number, as conversations are structured to begin with a student query and end with a student response.

Furthermore, for most conversation lengths, we balanced the samples across the following five outcome categories: (a) ends with an incorrect response without a reattempt, (b) ends with an incorrect response after a reattempt, (c) ends with a correct response, (d) ends with a partially correct response without a reattempt, and (e) ends with a partially correct response after a reattempt.

#### B.4.4 Illustrative Example

An illustrative example is shown in [Tab.˜15](https://arxiv.org/html/2506.01241v3#A2.T15). All samples for T4 are publicly available.

The sample input includes a multi-turn tutor-student dialogue in which the student is asked to identify features that help mammals survive in energy-scarce environments. The conversation is broken into subproblems to promote step-by-step reasoning. In this example, the student suggests “a larger body size” as an energy-conserving trait. The tutor must evaluate this response, determine its correctness, and provide pedagogically appropriate feedback that encourages deeper thinking without simply revealing the answer. The human reference demonstrates how to strike a balance between affirming partial correctness and guiding the student toward improved reasoning.

The model prompt is shown in [Tab.˜16](https://arxiv.org/html/2506.01241v3#A2.T16).

Table: Table 15: T4EduPAE - A sample of a tutor-student conversation and its corresponding pedagogical aligned response.

Table: Table 16: T4EduPAE – Model prompt.

#### B.4.5 Evaluation Rubric

The detailed evaluation rubric is provided below:

- 1.
Evaluation of the student response: Evaluating the response generated by the LLM in accurately identifying whether the student response is correct, incorrect or partially correct.
- 2.
Repeated mistake: Whether the LLM generated response accurately identifies if the student’s reattempt along a problem in incorrect / partially correct.
- 3.
Error Identification: Whether the error identified by the LLM generated response is consistent with that of the reference if present. For cases where student response is correct, the LLM must not generate “N/A”.
- 4.
Error Rectification: Whether the elements that describe the rectification of the error is consistent with that of the reference if present. For cases where student response is correct, the LLM must not generate “N/A”.
- 5.
Error Identification Type: Whether the feedback involves direct error identification when the student makes multiple inaccurate / partially accurate attempts along a sub-problem and involve indirect problem identification otherwise
- 6.
Error Rectification Type: Whether the feedback involves direct rectification of the errors when the student makes multiple inaccurate / partially accurate attempts along a sub-problem and involve indirect problem rectification otherwise
- 7.
Sub-question Transition: Whether the next sub-question in the LLM generated response is consistent with that of the reference. If all the sub-problems are solved or if the student makes an error along current one, then the feedback must not involve this aspect.

#### B.4.6 Checklist-mapped Reference

A prompt for generating the checklist-mapped reference was unnecessary, as it was produced directly from the model output.

Instead, to evaluate the quality of the generated output according to the rubric
, we extracted comparable information from the reference tutor data. [Section˜B.4.3](https://arxiv.org/html/2506.01241v3#A2.SS4.SSS3) already describes how to identify elements that describe a problem and corresponding solutions. Additionally, it provides a method for determining whether the student is struggling with the current sub-problem—specifically by checking for repeated errors across attempts. This Repeated mistake attribute is used to assign appropriate values to the Error Identification type and Error Rectification type. These types are set to Indirect when the student is attempting the sub-problem for the first time. If the Repeated mistake flag is active, both types are instead set to Direct. Notably, these properties are only assigned when the student’s response contains an error. For the Sub-question Transition property, we directly use the value provided in the original dataset, which is only set when the student gives a correct answer but the overall problem remains unsolved. An example of the checklist-mapped human reference from [Tab.˜15](https://arxiv.org/html/2506.01241v3#A2.T15) is shown in [Tab.˜17](https://arxiv.org/html/2506.01241v3#A2.T17).

Table: Table 17: T4EduPAE - Checklist-mapped reference.

### B.5 T5EduFG

#### B.5.1 Task Definition

The objective of this task is to evaluate whether an LLM can assess student responses with the same discernment as a trained instructor or expert. In this setup, we consider an assignment designed by the course instructor, whose response takes the form of a long-form essay. Although the instructor provides guidelines that students are expected to follow, these are intentionally not exhaustive—ensuring that students are not simply spoon-fed a detailed checklist of requirements. Instead, students are expected to adhere to additional implicit expectations informed by the course material covered thus far. While instructors are skilled at evaluating responses based on these unstated criteria, this task aims to determine whether an LLM can perform such evaluations effectively.

More concretely, the assignment includes both public and private requirements. While the instructor is aware of both, the LLM is given only the public requirements and the course syllabus, and must use these to also evaluate the response against the private requirements.

#### B.5.2 Task Significance

Our motivation to assess LLMs’ ability to evaluate student responses beyond student-facing rubric stems from the practical insight that instructors often do not rely solely on a rubric Instead, they apply holistic judgments that incorporate broader, often implicit, expectations [Bloxham et al., 2011, Jeong, 2015]. This is particularly evident in writing assignments, where public rubrics tend to be too general to support nuanced evaluation. Instructors frequently augment them with private, instance-specific criteria to enable more accurate and context-sensitive scoring [Broad, 2003]. Building on these observations, we seek to determine whether LLMs can similarly move beyond surface-level guidelines to produce evaluations that align with instructor judgment.

To study this, we use responses to an assignment, where the student-facing (public) rubric is subsumed by the private rubric used by the instructor. This setup allows us to test the LLM’s capacity to evaluate in a manner consistent with expert grading practices, including criteria not explicitly stated.

#### B.5.3 Data Acquisition

##### Data Description:

For this task, we use data from the Economics 101 course at an R1 university in the US [Nair et al., 2024]. The assignment presents a scenario in which “an increase in the minimum wage in San Francisco could lead to increased adoption of automation.” To counter this, two policy options are proposed: (a) a tax on automation and (b) a ban on automation. Students are asked to write a persuasive letter outlining the economic implications of the wage increase and to argue against one of the proposed policies using concepts and tools covered in the course. The student-facing rubric is shown in [Tab.˜18](https://arxiv.org/html/2506.01241v3#A2.T18) - it is clearly evident that the student-facing rubric is high-level and cannot be used as a criteria for robustly assigning the scores.

Table: Table 18: T5EduFG - Public/student-facing rubrics for the ECON 101 Assignment.

In contrast, the instructor also uses a set of private evaluation rubrics to enable more fine-grained and robust assessment of the essays. Examples include:

- •
Identifying the shift from a non-binding to a binding price floor
- •
Recognizing key concepts such as price floor/minimum wage, binding vs. non-binding constraints, substitutes, and labor vs. automation
- •
Explaining how an increase in the minimum wage could reduce the supply of final goods/services due to higher input costs or firm closures

In total, there are $20$ such private rubric items used to guide a more nuanced evaluation of the essay content.

Each student in the course submits an essay-based assignment, which is then evaluated by the instructor or teaching assistant using the private set of rubrics to guide their feedback. Although the feedback may be unstructured, it can be parsed into a structured format by mapping it to Yes/No responses for each item in the private rubric list.

##### Feedback Parsing:

For a given unstructured feedback, we use GPT-4o to extract information along each rubric item as shown before. For instance, the prompt to extract above items would be:

- •
Does the feedback indicate that the following statement is missing in the essay: Identification of the change from nonbinding to binding price floor?
- •
Does the feedback identifies that the following concepts are missing from the essay: Price floor/min wage, binding/non-binding, substitutes, labor/automation
- •
Does the feedback indicate that the essay contains an error along the following aspect: The increase of minimum wage will lead to a decrease in supply for final goods/services due to an increase in input prices or firms going out of business?

An affirmative response to a prompt implies that the corresponding rubric item is not properly addressed in the essay. This forms the checklist-mapped reference for each feedback.

##### Sampling:

To sample 100 data points—including student responses, instructor feedback, and the corresponding checklist-mapped references—we selected those with the longest instructor feedback, based on word count. This is because essays with longer instructor feedback typically contain more errors, requiring more careful analysis to identify and address them.

#### B.5.4 Illustrative Example

Due to the proprietary nature of the data, we cannot provide a specific example. However, an example of instructor feedback is provided in [Tab.˜19](https://arxiv.org/html/2506.01241v3#A2.T19) for a student written essay and the model prompt is shown in [Tab.˜20](https://arxiv.org/html/2506.01241v3#A2.T20):

Table: Table 19: T5EduFG - An example of instructor feedback.

Table: Table 20: T5EduFG – Model prompt.

#### B.5.5 Evaluation Rubric

In this section, we present the rubric used for evaluation. Each rubric item is annotated with the type of issue it addresses. Specifically, (missing) indicates whether the feedback correctly identifies the absence of a corresponding statement, while (error) assesses whether the feedback accurately detects an error in the corresponding element. To measure whether a certain mistake has been accurately detected, we check whether the checklist-mapped response entry matches with the reference one.

- 1.
Identifies the change from nonbinding to binding price floor (missing): Does the feedback accurately indicate that the following statement is missing in the essay: Identification of the change from nonbinding to binding price floor?
- 2.
Identifies the concepts of price floor/min wage, binding/non-binding, substitutes, labor/automation (missing): Does the feedback accurately identify that the following concepts are missing from the essay: Price floor/min wage, binding/non-binding, substitutes, labor/automation?
- 3.
Mentions that the increase of minimum wage causes a decrease in the demand for automation (error): Does the feedback accurately mention that the essay contains erroneous information such as: the increase of minimum wage causes a decrease in the demand for automation?
- 4.
Defines the concepts of price floor/min wage, binding/non-binding, substitutes, labor/automation (missing): Does the feedback accurately identify that the definitions of the following concepts are missing from the essay: Price floor/min wage, binding/non-binding, substitutes, labor/automation?
- 5.
(Only when the student is writing against the ban) States that a ban is essentially a [quota of zero] (missing): Does the feedback accurately indicate that the following statement is missing in the essay: A ban is essentially a [quota of zero]?
- 6.
Specify that minimum wage is a price floor (missing): Does the feedback accurately indicate that the following statement is missing in the essay: Minimum wage is a price floor?
- 7.
Treating the labor market/final good market as a homogeneous entity without distinguishing different workers or labor-made/automation-made goods (error): Does the feedback accurately identify that different workers or labor-made/automation-made goods are distinguished, making the labor market/final good market a non-homogeneous entity?
- 8.
State that the firm has lower costs of production after the minimum wage increase (missing): Does the feedback accurately indicate that the following statement is missing in the essay: The firm has lower costs of production after the minimum wage increase?
- 9.
Mentions that automation and labor are substitutes in consumption (missing): Does the feedback accurately indicate that the following statement is missing in the essay: Automation and labor are substitutes in consumption?
- 10.
Explains that the increase of minimum wage will lead to a decrease in supply for final goods/services due to an increase in input prices or firms going out of business (error): Does the feedback accurately indicate that the essay contains an error along the following aspect: The increase of minimum wage will lead to a decrease in supply for final goods/services due to an increase in input prices or firms going out of business?
- 11.
Explains that the increase of minimum wage will lead to a decrease in supply for final goods/services due to an increase in input prices or firms going out of business (missing): Does the feedback accurately indicate that the essay lacks an explanation along the following aspect: The increase of minimum wage will lead to a decrease in supply for final goods/services due to an increase in input prices or firms going out of business?
- 12.
State that with tax or ban on automation, the demand for labor increases (missing): Does the feedback accurately indicate that the following statement is missing in the essay: Tax or ban on automation results in increased demand for labor?
- 13.
State that with tax or ban on automation, the supply of the final goods/service decrease (missing): Does the feedback accurately indicate that the following statement is missing in the essay: Tax or ban on automation results in decreased supply of the final goods/service?
- 14.
State that with the binding minimum wage, the quantity demanded for the labor will decrease (missing): Does the feedback accurately indicate that the following statement is missing in the essay: The binding minimum wage decreases the quantity demanded for the labor?
- 15.
Provide reason for the shift when identifies a curve shift or change in demand/supply (missing): Does the feedback accurately indicate that the reason for the shift is missing when identifying a curve shift or change in demand/supply?
- 16.
Explain conceptually instead of solely rely on shifting the supply and demand curve (missing): Does the feedback accurately indicate that the essay lacks conceptual explanation and solely relies on shifting the supply and demand curve?
- 17.
Identify the change in quantity supplied/demanded instead of supply/demand (missing): Does the feedback accurately indicate that the essay does not identify the change in quantity supplied/demanded instead of supply/demand?
- 18.
Propose a solution on their own to the problem of the increasing minimum wage (missing): Does the feedback accurately indicate that the solution to the problem of increasing minimum wage is not proposed in the essay?
- 19.
Does not explain concepts the readers are expected to know from the prompt (i.e. demand, supply, consumer surplus, producer surplus, and efficiency). (error): Does the feedback accurately indicate that the essay explains concepts already known to the readers such as demand, supply, consumer surplus, producer surplus, and efficiency?
- 20.
Address all the three markets and the inter-market effects at play (missing): Does the feedback accurately identify that the essay fails to address all the three markets and the inter-market effects at play?

For each item, “Yes” indicates that the essay has a problem along the corresponding element as indicated by the feedback. For each response generated by the LLM, we parse it into a similar format and measure the extent of alignment between the answers along each checklist item.

#### B.5.6 Checklist-mapped Reference

A prompt for creating the checklist-mapped reference is not necessary in this task, as all model responses are binary (“Yes” or “No”) and easily verifiable. An example of the checklist-mapped reference of the instructor feedback in [Tab.˜19](https://arxiv.org/html/2506.01241v3#A2.T19) is presented in [Tab.˜21](https://arxiv.org/html/2506.01241v3#A2.T21).

Table: Table 21: T5EduFG - Checklist-mapped reference.

### B.6 T6HealthCNG

#### B.6.1 Task Definition

Clinical note generation is the task of producing well-structured, accurate, and comprehensive clinical notes based on patient-doctor dialogue during a clinical encounter [Contributors, 2023]. In this task, we focus on generating SOAP notes, as they are the most common way to document medical interactions Podder et al. [2023]. SOAP notes consist of four key sections: Subjective (patient-reported symptoms), Objective (clinician-observed data), Assessment (diagnoses or impressions), and Plan (treatment or follow-up steps). The objective is to generate accurate, structured, and useful SOAP notes from unstructured patient and doctor conversations. Automating this process would greatly reduce the clinician workload and enhance the quality of medical documentation. In using LLMs to perform this task, we input a the transcript of a conversation between a healthcare professional and patient. The output is a SOAP note that includes the key information from the conversation. An example note is shown in [Tab.˜22](https://arxiv.org/html/2506.01241v3#A2.T22).

#### B.6.2 Task Significance

Clinical documentation plays an important role in effective, high-quality care by supporting accurate diagnoses and standardizing communication among healthcare providers. However, the generation of structured clinical notes is time-consuming and prone to human error. Studies show that physicians can spend over an hour documenting a single clinical visit and nearly two hours on electronic health record (EHR) tasks for every hour of direct patient care Arndt et al. [2017]. Given the demanding schedules of healthcare professionals, this burden increases the risk of misremembered or inaccurately entered information. Automating this process with large language models (LLMs) can improve consistency and streamline workflows, ultimately reducing patient wait times and enhancing the quality of care.

#### B.6.3 Data Acquisition

We use the ACI-Bench dataset, a benchmark for clinical note generation with 207 annotated clinical encounters. Each sample includes a transcript of a patient-doctor interaction and its corresponding SOAP structured note written by the doctor [Yim et al., 2023]. Transcripts were created using three methods: (1) a virtual assistant, in which the doctor had to use explicit terms (e.g. “Hey Dragon show me the diabetes lab”) during the visit; (2) a virtual scribe, automated or otherwise, which assisted in note creation without distracting in-person doctor-patient interactions; and (3) an ambient clinical intelligence (ACI), which created the transcript without interrupting natural conversation flow between the patient and physician.

From these 207 samples, we selected a final subset of 100 on difficulty and diversity consistent with the rationale in §3. Difficulty was measured using the length of the human reference. Longer SOAP notes will contain more information about the patient, treatment, and analysis, making diagnosis a more difficult task. Additionally, ACI-Bench was originally collected in a diverse manner. Not only were transcripts created in a variety of methods, the content of the SOAP note itself was also diverse. For example, some notes contained *Chief Complaints*, but others did not; other notes had longer *History of Present Illness* sections, while others included symptom context in the *Assessment* or *Plan* sections.

#### B.6.4 Illustrative Example

An illustrative example is shown in [Tab.˜22](https://arxiv.org/html/2506.01241v3#A2.T22). All samples for T6 are publicly available.

The sample input consists of a multi-turn conversation capturing the doctor’s history-taking process and the patient’s responses about symptoms, routines, and relevant background. In this example, the patient is a teenage girl accompanied by her mother—presents for an acne evaluation. The human reference is a structured clinical note in SOAP format. The note reflects standard clinical reasoning, summarizes key details, and serves as a reference for future clinical visits.

The model prompt shown in [Tab.˜23](https://arxiv.org/html/2506.01241v3#A2.T23) was taken directly from the ACI-Bench benchmark Yim et al. [2023]. Although the prompt does not explicitly specify the four SOAP sections, the paper explains that it was strongly adapted from the SOAP note format: the History of Present Illness aligns with the Subjective section, the Exam and Results correspond to the Objective section, and the Assessment and Plan are combined into the final section. This structure also justifies our use of checklist items that incorporate SOAP elements in [Section˜B.6.5](https://arxiv.org/html/2506.01241v3#A2.SS6.SSS5).

Table: Table 22: T6HealthCNG - A sample of a transcript of a conversation between a patient and doctor and its corresponding SOAP note.

Table: Table 23: T6HealthCNG - Model prompt.

#### B.6.5 Evaluation Rubric

In T6, we did not consult domain experts as in previous tasks, but created our rubric independently through extensive research. Our rubric items and definitions were verified across multiple reputable online sources [Contributors, 2023, Podder et al., 2023] and are still expert-level. The rubric comprises 29 checklist items that captures the key information present in a SOAP note. The detailed checklist items are listed as follows:

- •
Subjective (S) Section
1.
Reason for Patient Office Visit or Hospitalization: The primary reason for the patient’s visit. Identifies the most pressing issue if multiple complaints are present. Uses concise and medically appropriate language.
2.
Patient’s Age
3.
Patient’s Sex
4.
Patient’s Reason for the Visit
5.
Onset (if applicable): When the complaint started.
6.
Location: The exact location the Chief Complaint happened.
7.
Duration: How long the complaint has persisted.
8.
Character: How the patient describes the Chief Complaint.
9.
Alleviating & Aggravating Factors: What makes the issue better or worse.
10.
Radiation: Whether symptoms move or stay in one spot.
11.
Temporal Factor: Whether or not the Chief Complaint is worse (or better) at a certain time of the day.
12.
Severity: The rating from the patient about the Chief Complaint using a scale of 1 to 10, 1 being the least pain, 10 being the worst pain.
13.
Relevant Medical History: Any relevant medical history, including past diagnoses, surgeries, and hospitalizations.
14.
Surgical History (if applicable)
15.
Family History
16.
Home and Environment: The patient’s living situation, relationships with family/roommates, and sense of safety/stability at home.
17.
Education: The patient’s current level of schooling, academic performance, or school engagement.
18.
Employment: The patient’s job status, job satisfaction, work hours, or financial independence.
19.
Eating Habits: The patient’s diet quality, body image concerns, or disordered eating patterns.
20.
Activities: The patient’s hobbies, friends, online/social media use, or after-school/work activities.
21.
Drugs: Use of alcohol, tobacco, marijuana, or other substances.
22.
Sexuality: The patient’s sexual activity, orientation, or gender identity.
23.
Suicide/Depression: The patient’s mood, self-harm, suicidal ideation, or prior mental health diagnoses or treatments.
24.
Review of Systems (ROS): Describes an inventory of the body systems to identify signs and/or symptoms which the patient may be experiencing. The body systems must fall into one of the 14 systems: Constitutional symptoms (i.e. fever, weight loss, vital signs); Eyes; Ears, nose, mouth, throat; Cardiovascular; Respiratory; Gastrointestinal; Genitourinary; Musculoskeletal; Integumentary; Neurological; Psychiatric; Endocrine; Hematologic/Lymphatic; and Allergic/Immunologic. Document both positive and pertinent negatives for each system reviewed.
- •
Objective (O) Section
25.
Vital Signs: BP, HR, RR, Temp, SpO2, weight, height (if relevant).
26.
Physical Examination Findings: The basic systems of cardiac and respiratory, affected systems, possible involvement of other systems, pertinent normal findings and abnormalities.
27.
Other Objective Data (if applicable): Results from laboratory and other diagnostic tests already completed.
- •
Assessment (A) Section
28.
Diagnosis & Clinical Impression: Provides a tentative diagnosis, assessment of patients status based on subjective and objective findings, a list of other possible diagnoses usually in order of most likely to least likely. The assessment will also include possible and likely etiologies of the patient’s problem. It is the patient’s progress since the last visit, and overall progress towards the patient’s goal from the physician’s perspective.
- •
Plan (P) Section
29.
Treatment & Management Plan: States what the health care provider will do to treat the patient’s concerns—such as ordering further labs, radiological work up, referrals given, procedures performed, medications given and education provided. The plan will also include goals of therapy and patient-specific drug and disease-state monitoring parameters. This should address each item of the differential diagnosis. For patients who have multiple health problems that are addressed in the SOAP note, a plan is developed for each problem and is numbered accordingly based on severity and urgency for therapy. A note of what was discussed or advised with the patient as well as timings for further review or follow-up are generally included. This part is often grouped together with Assessment.

#### B.6.6 Checklist-mapped Reference

The checklist-mapped references for both the human reference and model output were constructed with a similar approach to that used in T1 (see [Section˜B.1.6](https://arxiv.org/html/2506.01241v3#A2.SS1.SSS6)). However, in T6, we grouped 29 items into five groups based on (1) the expected length of the model’s response and (2) the SOAP category the item belonged to (e.g., *Subjective*, *Objective*, etc.). This is to help the model focus on distinct clinical sections and break down complex notes into meaningful, manageable parts. The prompt for extracting an individual checklist item is presented in [Tab.˜24](https://arxiv.org/html/2506.01241v3#A2.T24). A checklist-mapped reference of the sample in [Tab.˜22](https://arxiv.org/html/2506.01241v3#A2.T22) is shown in [Tab.˜25](https://arxiv.org/html/2506.01241v3#A2.T25).

To assess the quality of the checklist-mapped reference, we selected an additional 30 difficult and diverse samples, different from the main dataset, and conducted human verification to examine the faithfulness of the model response. Details of this experiment can be found in [Section˜E.1](https://arxiv.org/html/2506.01241v3#A5.SS1).

Table: Table 24: T6HealthCNG - Prompt for extracting checklist-mapped reference.

Table: Table 25: T6HealthCNG - Checklist-mapped reference.

### B.7 T7ChemMDG

#### B.7.1 Task Definition

Molecule description generation involves creating accurate and structured natural language descriptions of molecular structures based on their SMILES (Simplified Molecular Input Line Entry System) representations [Weininger, 1988]. SMILES encodes molecules as linear strings that represent atoms, bonds, rings, and branching patterns, and serves as a textual representation of molecular graphs. The objective is to translate these symbolic sequences into natural language descriptions that capture key structural and chemical features of the molecule. When using LLMs to complete this task, the input consists of a SMILES string (e.g., “CC(=O)OC1=CC=CC=C1C(=O)O”). The output is a molecule description a professional might read. An example is shown in [Tab.˜26](https://arxiv.org/html/2506.01241v3#A2.T26).

#### B.7.2 Task Significance

Chemical databases contain tens of millions of molecules, each represented by complex SMILES strings that are not easily interpretable by humans. Experts often find it time-consuming and challenging to infer the functional class or properties of a molecule solely from its SMILES representation, especially for more complex structures. Automating the translation of SMILES into structured, ontological, and natural language descriptions can significantly enhance the accessibility and usability of chemical data. This advancement supports key fields such as drug discovery, materials science, and chemical education by enabling quicker understanding and communication of molecular information [Edwards et al., 2021].

For example, models such as MolT5 have been specifically developed to complete this task. MolT5 leverages transformer-based architectures to improve the quality and informativeness of molecular descriptions, facilitating a deeper understanding of molecular structures and their properties [Edwards et al., 2022]. Such models not only accelerate the discovery phase but also improve collaboration across multidisciplinary research teams.

#### B.7.3 Data Acquisition and Preprocessing

We use the ChEBI-20 dataset originally collected for the Text2Mol task [Edwards et al., 2021]. This dataset was created by collecting compound annotations from the ChEBI database(^20^2020[https://www.ebi.ac.uk/chebi/](https://www.ebi.ac.uk/chebi/)), which were scraped from PubChem(^21^2121[https://pubchem.ncbi.nlm.nih.gov/](https://pubchem.ncbi.nlm.nih.gov/)). Descriptions shorter than 20 words were excluded to ensure sufficient detail. The resulting dataset contains 33,010 pairs of SMILES strings and their corresponding textual descriptions. Both ChEBI and PubChem are specialized, domain-specific chemical resources commonly utilized by chemists, aligning well with the expert-level knowledge demanded by this task.

To specifically evaluate the capability of LLMs in generating detailed molecule descriptions, we selected a small subset from ChEBI-20 based on difficulty. In T7, difficulty is approximated by the length of the human-written reference, with longer descriptions generally indicating more complex or detailed explanations. We selected 100 samples whose reference descriptions exceed 500 characters to focus on more challenging examples.

#### B.7.4 Illustrative Example

An illustrative example is shown in [Tab.˜26](https://arxiv.org/html/2506.01241v3#A2.T26). All samples for T7 are publicly available.

The sample input consists of a SMILES string representing the chemical structure. The human reference is a detailed description that captures both the molecular structure and pharmacological function of the compound. It uses precise chemical nomenclature and is clear and concise.

The model prompt is presented in [Tab.˜27](https://arxiv.org/html/2506.01241v3#A2.T27). This prompt was carefully designed in collaboration with domain experts to ensure both generality and relevance across a broad range of chemical compounds. To align with the ChEBI dataset, which stands for “Chemical Entities of Biological Interest,” we incorporated the phrase “Chemicals of Biological significance” into the prompt. However, we deliberately avoided including the acronym “ChEBI” itself to prevent the model from relying on any dataset-specific shortcuts.

Table: Table 26: T7ChemMDG - A sample of a SMILES string and its corresponding molecule description.

Table: Table 27: T7ChemMDG – Model prompt.

#### B.7.5 Evaluation Rubric

Furthermore, we develop a fine-grained, checklist-based evaluation rubric in collaboration with a chemistry domain expert to ensure comprehensive coverage of key information in a molecule description. The expert not only advised us on the selection of rubric items, but also annotated a subset of human-written reference descriptions to identify which specific elements were explicitly or implicitly mentioned. These annotations informed the basis for the design of our checklist and guided the formulation of model prompts, ensuring that generated descriptions capture the same level of completeness and domain accuracy as the human references.

The final rubric comprises of six checklist items that capture key information typically found in professional molecule descriptions. The last sentence of each definition (starting with “Usually, but not always,”) is included to support the construction of the checklist-mapped reference described in [Section˜B.7.6](https://arxiv.org/html/2506.01241v3#A2.SS7.SSS6). The detailed checklist items are listed as follows:

- 1.
Structure: The chemical composition of the molecule. Usually, but not always, this description begins with “The molecule is”.
- 2.
Biological Function and Applications: The molecule’s functions in living organisms, including its interactions, effects on biological processes, and applications in the pharmacological domain. Usually, but not always, this description begins with “It has a role as”.
- 3.
Chemical compound classifications: The specific group(s) of atoms that are used to categorize chemical compounds based on their structural characteristics, functional groups, or biological origin. Usually, but not always, this description begins with “It is a member of”.
- 4.
Conjugate base (if applicable): The species that remains after an acid donates a proton (H^+) in a chemical reaction. Usually, but not always, this description begins with “It is a conjugate base of”.
- 5.
Conjugate acid (if applicable): The species that remains after an base accepts a proton (H^+) in a chemical reaction. Usually, but not always, this description begins with “It is a conjugate acid of”.
- 6.
Origin (if applicable): The molecule or chemical compound that has been extracted or obtained from a specific natural resource. Usually, but not always, this description begins with “It is isolated from”.

#### B.7.6 Checklist-mapped Reference

In line with the method described in T1, we generated checklist-mapped references for the human reference and model outputs with our evaluation rubric. However, with only six checklist items, we were able to create one group with one model extraction prompt. The prompt for extracting an individual checklist item is presented in [Tab.˜28](https://arxiv.org/html/2506.01241v3#A2.T28) and the model output extraction prompt will be available in our public GitHub repository. The checklist-mapped reference for the sample in [Tab.˜26](https://arxiv.org/html/2506.01241v3#A2.T26) is shown in [Tab.˜29](https://arxiv.org/html/2506.01241v3#A2.T29).

Table: Table 28: T7ChemMDG - Prompt for extracting checklist-mapped reference.

Table: Table 29: T7ChemMDG - Checklist-mapped reference.

### B.8 Task 8: Biology - Protein Description Generation

#### B.8.1 Task Definition

Protein description generation, also known as protein captioning, refers to creating an accurate, informative, and useful description of a protein given its amino acid sequence. These sequences are composed of letters representing individual amino acids and are the standard linear representations of protein primary structure. The objective is to generate a natural language description that captures essential biological characteristics of the protein, such as its function, cellular location, family or domain classification, and any relevant structural or catalytic properties. An example of a sequence-description pair is shown in [Tab.˜30](https://arxiv.org/html/2506.01241v3#A2.T30). In using LLMs to complete this task, the input is an amino acid sequence (e.g., MKWVTFISLLFLFSSAYSRGVFRRDTH…) and the output is a protein description [Rives et al., 2021].

#### B.8.2 Task Significance

Protein sequences are long, complex, and typically require expert analysis and database lookups (e.g., UniProt(^22^2222[https://www.uniprot.org](https://www.uniprot.org)), PDB(^23^2323[https://www.rcsb.org/](https://www.rcsb.org/))) to determine their biological functions. Automating the generation of descriptions helps make protein information more accessible and supports tasks like genome annotation, database curation, and bioinformatics research. Recent tools such as AnnoPRO Li et al. [2023b] apply deep learning to predict protein function from sequences, reducing the need for manual curation and accelerating biological discovery [Li et al., 2023b].

#### B.8.3 Data Acquisition and Preprocessing

We collected protein sequences and their corresponding descriptions from the SciKnowEval dataset, which sources its protein and caption entries from the UniProtKB database(^24^2424[https://www.uniprot.org/help/uniprotkb](https://www.uniprot.org/help/uniprotkb)) —a comprehensive, curated resource for protein sequence and functional information [Feng et al., 2024]. Reference descriptions are written by experts, curated by UniProt, and serve as the human reference for this task. From this dataset, we filtered a subset of 100 high-quality samples for our task. These samples were selected on length, all of which are over 900 characters.

#### B.8.4 Illustrative Example

An illustrative example is shown in [Tab.˜30](https://arxiv.org/html/2506.01241v3#A2.T30) All samples for T8 are publicly available.

The sample input is an amino acid string. The human reference captures key biological attributes, including functional roles (e.g., antimicrobial activity), cellular localization (extracellular), molecular interactions (lipopolysaccharide binding), and structural features (e.g., $\beta$-strands with supporting PDB annotations). This example demonstrates the level of specificity and scientific grounding expected in generated protein descriptions.

Considering contextual grounding, we adopt a role-based prompting strategy. The model is instructed to assume the role of a protein researcher tasked with writing descriptions for protein sequences and encouraged to use domain-specific terminology with the model prompt shown in [Tab.˜31](https://arxiv.org/html/2506.01241v3#A2.T31).

Table: Table 30: T8BioPDG- A sample of an protein in its amino acid sequence and its corresponding description.

Table: Table 31: T8BioPDG- Model prompt.

#### B.8.5 Evaluation Rubric

We collaborated with two graduate students from an R1 university at the US studying biology to create the evaluation rubric. We first presented an initial draft of the rubric, created using online sources, and fine-tuned it with the experts’ assistance across three separate meetings until both experts agreed with the rubric. The experts’ also provided us with sources that explained their reasoning for including or removing certain items in the rubric. The rubric comprises five checklist items that captures the key information present in a protein description. The detailed checklist items are listed as follows:

- 1.
Domains/Motifs: Functional regions (e.g., kinase domain, zinc finger) [Murzin et al., 1995, Berman et al., 2000].

- •
Functional characteristics
2.
Functional Role: How a molecule contributes to biological systems, including the processes it is involved in (e.g., metabolism, cellular functions) and the specific role it plays (e.g., enzymatic, structural, or signaling) [Szklarczyk et al., 2023, Oughtred et al., 2021].
3.
Cellular Localization: "Cellular location (e.g., mitochondrial matrix, cell membrane).
4.
Gene Ontology: Identify key GO terms.

- 5.
Interactions: The physical and functional associations within a cell or organism, including but not limited to protein partners, ligands/substrates, and cofactors [Szklarczyk et al., 2023, Oughtred et al., 2021].

#### B.8.6 Checklist-mapped Reference

Consistent with the approach in T1, we generated checklist-mapped references for both human and model outputs. Because T8 has only five checklist items, a single model output extraction prompt was sufficient. The item-level extraction prompt in [Tab.˜32](https://arxiv.org/html/2506.01241v3#A2.T32) and the model output extraction prompt will be available in our public GitHub repository. The checklist-mapped reference for the example in [Tab.˜30](https://arxiv.org/html/2506.01241v3#A2.T30) is shown in [Tab.˜33](https://arxiv.org/html/2506.01241v3#A2.T33).

Table: Table 32: T8BioPDG- Prompt for extracting checklist-mapped reference.

Table: Table 33: T8BioPDG- Checklist-mapped reference.

### B.9 T9MedicalDR

#### B.9.1 Task Definition

The objective of T9 is to assess a model’s ability to infer a Primary Discharge Diagnosis (PDD) from parts of a SOAP note. The input consists of selected sections of a SOAP note: the Chief Complaint, History of Present Illness (HPI), Past Medical History, Family History, Physical Exam, and Pertinent Results. Any explicit mentions of the diagnosis are manually removed. Given this input, the model must output the correct diagnosis and provide reasoning based on textual evidence.

#### B.9.2 Task Significance

This task addresses a challenge in clinical decision-making: deriving accurate diagnoses from patient information. Errors in diagnosis can lead to misdiagnosis, delayed treatment, or inappropriate care, with potentially severe consequences for patient outcomes. By enabling LLMs models to support diagnostic inference in a structured and interpretable way, this task contributes to improving medical safety and clinical decision support systems.

#### B.9.3 Data Acquisition and Preprocessing

**Table 34: T9MedicalDR - Distribution of diagnoses.**
| Diagnosis | # Cases |
| --- | --- |
| Atrial Fibrillation | 6 |
| Adrenal Insufficiency | 7 |
| Hypertension | 6 |
| Alzheimer | 6 |
| Pneumonia | 6 |
| Stroke | 6 |
| Gastro-oesophageal Reflux Disease | 5 |
| Epilepsy | 6 |
| Hyperlipidemia | 2 |
| Asthma | 6 |
| Heart Failure | 6 |
| Peptic Ulcer Disease | 6 |
| Diabetes | 6 |
| Pulmonary Embolism | 6 |
| Migraine | 4 |
| Thyroid Disease | 5 |
| Tuberculosis | 5 |
| Aortic Dissection | 6 |

The DiReCT dataset consists of 511 de-identified discharge summaries Wang et al. [2024]. From each note, specific sections relevant to the diagnostic process were extracted: chief complaint, history of present illness, past medical history, family history, physical exam, and pertinent results. To ensure high quality data, reasoning annotations were performed by nine licensed clinical physicians and verified for accuracy and completeness by three senior medical experts. Each diagnosis is one of 25 disease categories across five high-level clinical domains.

DiReCT originally contained information that our task has filtered out. The human reference is structured as follows: {o: [z, r, d]}. o is the extracted observation from raw text, z is the rationale to explain why an observation can be related to a diagnosis d, r is the section (from one of input1-6) of the clinical note where o is extracted, and finally, d is the name of the diagnosis. To calculate the *#Rubric* items of the human reference in Tab.˜1 of the main paper, we concatenated this dict into a string and found its token length.

We then selected a smaller, high-quality, and representative subset from the dataset with the criteria of diversity and difficulty described in §3.
To maintain diversity, we aimed to sample equally from all 25 final diagnosis. This promotes a balanced distribution that supports generalizable evaluation across multiple clinical domains. We first selected six samples per diagnosis type, resulting in an initial pool of 150 samples. Next, we prioritized cases with longer and more complex reasoning chains, as these are more likely to challenge model capabilities in clinical inference. From the 150 samples, we identified 100 with the longest human-generated reference outputs, reflecting the depth of clinical reasoning involved. Finally, we manually verified that each sample met both diversity and difficulty thresholds. [Tab.˜34](https://arxiv.org/html/2506.01241v3#A2.T34) shows the diagnosis distribution of our final 100 samples across 18 diagnoses.

#### B.9.4 Illustrative Example

Due to the proprietary nature of the data, we cannot provide a specific example. However, [Tab.˜35](https://arxiv.org/html/2506.01241v3#A2.T35) displays the model prompt. The model is instructed to diagnose the patient and provide supporting evidence in a structured dict_reasoning format. To ensure alignment with human references and enable verification, we constrained the model to select from a predefined list of possible diagnoses. This was necessary because, during early testing, the model often generated overly specific diagnoses— "Hypertension in the setting of atrial fibrillation" instead of "Hypertension"—which could not be matched against the simpler human reference.

Table: Table 35: T9MedicalDR - Model prompt.

#### B.9.5 Evaluation Rubric

In T9, we did not consult domain experts as in previous tasks, but created our rubric independently. For this task, we introduce the concept of global versus instance-level checklist items.

- •
Global checklist items are evaluated for every sample, regardless of the specific content of the human reference. For example, the item Diagnosis is global—it is always present in the human reference and model output and therefore always evaluated.
- •
Instance-level checklist items are conditional: they are only evaluated when the relevant content appears in the human reference. For example, if the reference mentions the observation “elevated heart rate,” this becomes the key for an {Evidence: Reasoning} checklist item. The corresponding Reasoning value might be “An elevated heart rate is a common symptom of an infection.” These items are included only when such content is explicitly present in the human reference.

Thus, the number of checklist items in the final rubric varies based on the length and detail of the human reference, but comprises of three general items:

- 1.
Diagnosis (global)
- 2.
Each Evidence (instance-level): An observation that is textually present in the input
- 3.
Each Reasoning (instance-level): The explanation in why the Evidence supports to the Diagnosis.

#### B.9.6 Checklist-mapped Reference

Due to the proprietary nature of the data, we cannot provide a specific example for the checklist-mapped reference. A prompt for generating the checklist-mapped reference was also unnecessary, as it was produced directly from the model output.

### B.10 T10FinanceESG

#### B.10.1 Task Definition

ESG reports detail a company’s environmental, social, and governance practices and performance.
These documents are typically long and complex, addressing a broad spectrum of key ESG issues such as carbon emissions, labor practices, and board diversity.
The objective of ESG report summarization is to distill this information into concise summaries that emphasize the most pertinent aspects of a company’s ESG performance for investors and stakeholders.
For each company, the model is given both an ESG report from a third-party rating agency and the company’s self-published ESG report as input for generating the summary.

#### B.10.2 Task Significance

Environmental, social, and governance considerations are increasingly recognized as critical components of long-term corporate performance.
In response to global sustainability initiatives—such as the United Nations Sustainable Development Goals(^25^2525[https://unglobalcompact.org/](https://unglobalcompact.org/)) and the Paris Agreement(^26^2626[https://www.un.org/en/climatechange/paris-agreement](https://www.un.org/en/climatechange/paris-agreement))—governments, institutions, and consumers have placed growing emphasis on corporate ESG practices.
These factors now influence consumer behavior, regulatory compliance, and capital allocation.
As a result, investors are integrating ESG performance into portfolio design and risk assessment.

Given this context, ESG disclosures have become critical tools for evaluating a company’s long-term value and social responsibility.
However, ESG reports are often comprehensive and cover different aspects of ESG, not all of which are equally important for every company.
Automated summarization that prioritizes the most relevant key issues enables more efficient consumption of ESG information.
This task supports more informed decision-making by helping stakeholders focus on the ESG factors most relevant to a company’s financial, operational, and reputational outcomes.

#### B.10.3 Data Acquisition and Preprocessing

We utilize ESG reports published by MSCI(^27^2727[https://www.msci.com](https://www.msci.com)), a leading ESG rating agency.
MSCI’s ESG research covers over 10,000 companies across a wide range of sectors and regions.
For each company, MSCI provides a comprehensive ESG report that outlines the company’s performance on ESG key issues, which vary by sector.(^28^2828[https://www.msci.com/our-solutions/esg-investing/esg-industry-materiality-map](https://www.msci.com/our-solutions/esg-investing/esg-industry-materiality-map))
In addition to performance data, the reports include textual summaries of the most relevant ESG issues and provide an overall ESG rating.

To ensure a diverse sample, we select 20 sectors with the highest number of companies rated by MSCI and choose 5 companies from each sector.
These companies are selected to represent a range of ESG ratings (AAA, AA, A, BBB, BB, B).
For each selected company, we collect both the most recent MSCI ESG report and the self-published ESG report available on the company’s website.
Both types of reports are typically in PDF format and are converted to text using PyMuPDF4LLM.

From the MSCI reports, we extract only the sections that describe the company’s performance on key issues, discarding content such as summaries.
For company-issued reports, which may include additional content such as financial data, we extract only the pages that contain ESG-related content.

Reference summaries are obtained directly from MSCI’s web interface rather than the PDF reports to ensure accurate extraction.

#### B.10.4 Illustrative Example

Due to the proprietary nature of the data, we cannot provide a specific example.
However, in [Tab.˜36](https://arxiv.org/html/2506.01241v3#A2.T36), we provide the model prompt used in obtaining model outputs.

Table: Table 36: T10FinanceESG - Model prompt.

#### B.10.5 Evaluation Rubric

The ESG report summaries discuss the company’s performance on ESG key issues.
While the ESG report of a company details its performance on ESG key issues that are important to the corresponding sector, a well-written summary should cover the most significant key issues relevant to the company and give a precise overview of the company’s performance on these issues.
We build the evaluation rubrics based on MSCI’s ESG scoring framework, which includes 33 key issues across 10 themes.(^29^2929[https://www.msci.com/documents/1296102/34424357/MSCI+ESG+Ratings+Methodology.pdf](https://www.msci.com/documents/1296102/34424357/MSCI+ESG+Ratings+Methodology.pdf))
The discussion of key issues is sector-specific,(^30^3030[https://www.msci.com/our-solutions/esg-investing/esg-industry-materiality-map](https://www.msci.com/our-solutions/esg-investing/esg-industry-materiality-map)) and therefore in this task we have different rubrics for different data samples, as shown in [Tab.˜37](https://arxiv.org/html/2506.01241v3#A2.T37).

**Table 37: Selected sectors and their corresponding key issues.**
| Sector | Key Issues |
| --- | --- |
| Banks | Access to Finance; Consumer Financial Protection; Corporate Behavior; Corporate Governance; Financing Environmental Impact; Human Capital Development; Privacy & Data Security |
| Biotechnology | Access to Health Care; Corporate Behavior; Corporate Governance; Human Capital Development; Product Safety & Quality; Toxic Emissions & Waste |
| Construction & Engineering | Corporate Behavior; Corporate Governance; Health & Safety; Human Capital Development; Opportunities in Clean Tech |
| Diversified Financials | Access to Finance; Carbon Emissions; Corporate Behavior; Corporate Governance; Human Capital Development |
| Electronic Equipment, Instruments & Components | Chemical Safety; Controversial Sourcing; Corporate Behavior; Corporate Governance; Labor Management; Opportunities in Clean Tech |
| Food Products | Corporate Behavior; Corporate Governance; Opportunities in Nutrition & Health; Packaging Material & Waste; Product Carbon Footprint; Product Safety & Quality; Raw Material Sourcing; Water Stress |
| Health Care Equipment & Supplies | Carbon Emissions; Corporate Behavior; Corporate Governance; Human Capital Development; Product Safety & Quality |
| Health Care Providers & Services | Carbon Emissions; Corporate Behavior; Corporate Governance; Labor Management; Privacy & Data Security; Product Safety & Quality |
| Industrial Machinery | Corporate Behavior; Corporate Governance; Labor Management; Opportunities in Clean Tech; Toxic Emissions & Waste |
| Media & Entertainment | Carbon Emissions; Corporate Behavior; Corporate Governance; Labor Management; Privacy & Data Security |
| Metals and Mining - Non-Precious Metals | Biodiversity & Land Use; Carbon Emissions; Community Relations; Corporate Behavior; Corporate Governance; Health & Safety; Labor Management; Toxic Emissions & Waste; Water Stress |
| Pharmaceuticals | Access to Health Care; Corporate Behavior; Corporate Governance; Human Capital Development; Product Safety & Quality; Toxic Emissions & Waste |
| Real Estate Development & Diversified Activities | Corporate Behavior; Corporate Governance; Health & Safety; Opportunities in Green Building; Product Safety & Quality |
| Real Estate Management & Services | Corporate Behavior; Corporate Governance; Human Capital Development; Opportunities in Green Building |
| Retail - Consumer Discretionary | Chemical Safety; Corporate Behavior; Corporate Governance; Labor Management; Privacy & Data Security; Product Carbon Footprint; Raw Material Sourcing |
| Semiconductors & Semiconductor Equipment | Controversial Sourcing; Corporate Behavior; Corporate Governance; Human Capital Development; Opportunities in Clean Tech; Water Stress |
| Software & Services | Carbon Emissions; Corporate Behavior; Corporate Governance; Human Capital Development; Opportunities in Clean Tech; Privacy & Data Security |
| Specialty Chemicals | Carbon Emissions; Chemical Safety; Corporate Behavior; Corporate Governance; Opportunities in Clean Tech; Toxic Emissions & Waste; Water Stress |
| Telecommunication Services | Carbon Emissions; Corporate Behavior; Corporate Governance; Labor Management; Privacy & Data Security |
| Utilities | Carbon Emissions; Corporate Behavior; Corporate Governance; Human Capital Development; Opportunities in Renewable Energy; Toxic Emissions & Waste |

The company’s performance on each key issue is broken down into several underlying indicators.
For example, one of the indicators for environment-related issues is the extent to which the company has established programs and initiatives to address the issue.
We examine the ESG report summaries for each sector and observe that the granularity of the discussion of key issues usually shares the same level of detail as the underlying indicators.
Thus, we take these indicators as the items to be included in the rubrics, and we obtain the descriptions of these indicators based on the raw data MSCI uses to assess the company’s performance on these indicators.

Due to the proprietary nature of the data, in the paper, we cannot provide the list of rubrics items and their descriptions.

#### B.10.6 Checklist-mapped Reference

For each sample, to map the human-written summary into the reference checklist, we prompt the checklist mapper with the instruction that contains all the rubric items and their descriptions for the sector the company belongs to.
The items are not separately extracted, as it is easier for the checklist mapper to distinguish between items that originate from the same key issue when they are presented together.
We show the prompt for checklist mapping in [Tab.˜38](https://arxiv.org/html/2506.01241v3#A2.T38).

Due to the proprietary nature of the data, we cannot provide a specific example for the checklist-mapped reference.

Table: Table 38: T10FinanceESG - Prompt for extracting checklist-mapped reference. {major_key_issue} includes the key issues for the sector the company belongs to, and {sub_key_issue} includes the underlying indicators for each key issue.

### B.11 T11CyberRDG

#### B.11.1 Task Description

This task focuses on assessing the security LLM agents in generating structured risk descriptions from execution traces involving interactions between the agent, a user, and the environment. Each interaction is analyzed to determine whether it leads to a risky outcome and, if so, why. The objective is to produce a binary safety label (0 if the execution trace is safe, and 1 if it is dangerous) and a detailed explanation of the risk. The explanation should contain all parts of a MTO (Motivation, Trigger, Outcome) schema [Yuan et al., 2024] and specify any trigger or attack tools. Definitions of these terms can be found in [Section˜B.11.5](https://arxiv.org/html/2506.01241v3#A2.SS11.SSS5).

#### B.11.2 Task Significance

LLM agents are increasingly integrated into real-world applications such as virtual assistants, customer support bots, and IoT controllers due to their autonomy and adaptability. However, their deployment in unsupervised, complex environments introduces significant security concerns. These include financial losses from unauthorized transactions, data breaches through inadvertent exposure of sensitive information, and physical harm resulting from misinterpreted instructions to real-world devices [Li et al., 2025].

To address these risks, a systematic approach to security assessment is essential. This involves not only identifying unsafe execution traces but also providing explanations for their dangers. Understanding the motivations, triggers, and outcomes of such traces is crucial for auditing LLM-agent behavior and enhancing system robustness. By doing so, researchers can develop safer defaults, improve alignment, and establish a foundation for regulatory oversight.

#### B.11.3 Data Acquisition and Preprocessing

We created a dataset of 100 samples drawn from two execution trace corpora: AgentHarm and R-Judge. Specifically, we selected 19 samples from AgentHarm and 81 from R-Judge, based on the following filtering criteria described in §3:

- •
Diversity: Both datasets include execution traces spanning multiple domains such as fraud, cybercrime, and harassment [Andriushchenko et al., 2024]. We aimed to capture a broad range of scenarios where execution traces could pose safety risks.
- •
Difficulty: Longer risk descriptions tend to correspond with extended multi-turn interactions involving the user, agent, and environment. These cases are more challenging due to the increased cognitive load required for annotation and analysis.

R-Judge contains 569 annotated multi-turn agent interactions covering 27 key risk scenarios across five application categories and ten risk types [Yuan et al., 2024]. From this set, we filtered 114 samples with risk descriptions exceeding 600 characters from three subdomains: Program, Finance, and Application. We then selected the longest 23, 29, and 29 samples from each respective domain, resulting in 81 final R-Judge samples.

Risk descriptions in R-Judge follow the MTO schema and were annotated by six domain-informed experts (1 PhD candidate, 3 master’s students, and 2 senior undergraduates) with degrees in AI, computer science, and cybersecurity. However, R-Judge lacks trigger and attack tool labels. Thus, we used GPT-4o to extract these elements, followed by manual verification to ensure the accuracy of the final annotations.

AgentHarm consists of 440 explicitly malicious agent tasks spanning harm categories such as fraud, cybercrime, harassment [Andriushchenko et al., 2024], etc. While R-Judge focuses on indirect risks from tool use or ambiguous instructions, this dataset targets the risks of instructions that entail direct harm. We included 19 samples from AgentHarm in our collection, capturing a range of behaviors such as outright refusal, attempted compliance with tool failure, and initial compliance followed by rejection during execution.

The trajectories in this dataset were annotated with safety labels and the corresponding functions and tools invoked. We used this information as input to GPT-4o to generate responses in MTO format and manually verified the accuracy of the annotations.

#### B.11.4 Illustrative Example

An illustrative example is shown in [Tab.˜39](https://arxiv.org/html/2506.01241v3#A2.T39) All samples for T11 are publicly available.

The sample input is an execution trace that captures the sequence of agent thoughts, tool usage, and environment responses. In this case, the response from the GitHub API includes an injected malicious instruction embedded in the repository description. The human reference correctly identifies that the malicious instruction was introduced via the GitHubGetRepositoryDetails tool, but was not acted upon by the agent. The response includes a binary risk label, a textual explanation of the attempted attack and its failure, and the tool(s) involved in the injection.

The model prompt is shown in [Tab.˜40](https://arxiv.org/html/2506.01241v3#A2.T40). While we originally wanted to keep the prompt high-level and avoid directly defining checklist items, we chose to explicitly define the concepts of trigger and attack tools in this prompt. This was because these terms are specific to this task and may not be easily understood by the model without clarification.

Table: Table 39: T11CyberRDG - A sample of an execution trace and its corresponding risk description.

Table: Table 40: T11CyberRDG - Model prompt.

#### B.11.5 Evaluation Rubric

We extended the original R-Judge rubric by introducing two new checklist items—Trigger Tools and Attack Tools—to provide a more in-depth analysis of the risk landscape in tool-augmented agent systems. These additions aim to support future research and system development by enabling a deeper understanding of how specific tools contribute to the success or failure of an attack.

Trigger tools are the tools that actively bring the system closer to the attacker’s goal. While a trigger is generally the action that activates the attack, identifying the exact tool responsible for this transition helps developers pinpoint weak spots in tool usage or design that might be exploited in real-world deployments.

Attack Tools are tools indirectly used to deliver or embed malicious prompts—often through crafted inputs or outputs. Recognizing these tools is critical for tracing the origin of an attack and understanding how agent behaviors are manipulated through seemingly innocuous tool responses.

Including these distinctions in the risk description provides a clearer path to mitigation. It helps differentiate between tools that are merely involved in the agent’s execution and those that serve as key enablers of risk. Additionally, it’s important to note that the attacker’s goal (identified via the trigger tool) or the specific action executed (using an attack tool) does not have to be inherently harmful. A non-harmful action could still serve as a building block toward a more serious attack. By capturing and categorizing these intermediate steps, we enable more rigorous defense mechanisms that account for the full lifecycle of an attempted attack. Thus, the detailed checklist items are as follows:

- 1.
Label: Safe 0 or Unsafe 1
- 2.
Motivation: The motivation is the underlying reason or origin of potential risk in the system. The user may unintentionally specify a task that may be risky or maliciously induces the agent to behave badly.
- 3.
Trigger: The trigger is the action the agent does to activate the motivation during execution.
- 4.
Trigger Tool: Trigger tools are the specific tools that bring the system closer to the attacker’s goal during execution.
- 5.
Attack Tool: Attack tools are tools which may indirectly inject a prompt to the agent to make it execute certain actions.
- 6.
Outcome: The outcome is the result after the agent executes the task, including any effects, neutral or harmful, intended or unintended, that may arise.

#### B.11.6 Checklist-mapped Reference

We follow the process to create a checklist-mapped reference described in [Section˜B.1.6](https://arxiv.org/html/2506.01241v3#A2.SS1.SSS6). A checklist-mapped reference was only necessary for the Motivation, Trigger, and Outcome. The label was binary and could be easily verified without extraction. Attack and trigger tools were extracted using regex, as the LLM prompt instructed the model to output them in a structured manner. An example of the checklist-mapped reference is shown in [Tab.˜41](https://arxiv.org/html/2506.01241v3#A2.T41).
The prompt for extracting the checklist-mapped reference is shown in [Tab.˜42](https://arxiv.org/html/2506.01241v3#A2.T42).

Table: Table 41: T11CyberRDG - Checklist-mapped reference.

Table: Table 42: T11CyberRDG - Prompt for extracting checklist-mapped reference.

## Appendix C Checklist Accuracy Assessment: Prompts

### C.1 Checklist Accuracy Assessment with Unstructured Reference: Prompt

The prompt used to evaluate checklists against reference texts is shown in [Tab.˜43](https://arxiv.org/html/2506.01241v3#A3.T43). As illustrated, the evaluation assesses both faithfulness and coverage relative to the reference. For each checklist item, the prompt determines whether the content is faithful to the reference and whether it adequately captures all the essential information from the reference that is aligned with the checklist’s intent.

Table: Table 43: The prompt for evaluating whether the extracted checklist information covers all the appropriate information from the reference provided in the {EXTRACTED_TEXT} and is faithful with respect to it.

### C.2 Checklist-based Performance Assessment: Prompt

The prompt for assessing semantic containment is shown in [Tab.˜44](https://arxiv.org/html/2506.01241v3#A3.T44). It is used to evaluate whether each element from the checklist-mapped response semantically includes the corresponding element of the checklist-mapped reference. By reversing the roles of the model response and reference, we can also check if the reference contains the response. These bidirectional containment checks are jointly used to determine the correctness of each checklist item.

Table: Table 44: The prompt for evaluating whether an answer semantically covers all the information from a reference for recall measurement. This prompting is inspired from Verga et al. [2024]. The label “Yes” corresponds to a binary score of $1.0$ and “No” corresponds to 0. We include $5$ more examples in the placeholder: {Few-shot examples placeholder}

## Appendix D Additional Results

### D.1 Additional Main Results

While the main paper only reports the performance in terms of checklist F1-score, we include results with checklist accuracy ([Tab.˜45](https://arxiv.org/html/2506.01241v3#A4.T45)), precision ([Tab.˜46](https://arxiv.org/html/2506.01241v3#A4.T46)), recall ([Tab.˜47](https://arxiv.org/html/2506.01241v3#A4.T47)), and coverage ([Tab.˜48](https://arxiv.org/html/2506.01241v3#A4.T48)).

**Table 45: Evaluating LLMs on ExpertLongBench (scaled to 0–100) using checklist accuracy. Models are sorted by average performance and the best performing model on each task is bolded. Model ranking is indicated by the color of the cell, with green (best) to white (worst).**
| Model | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Gemini-2.5-Pro | 18.2 | 6.7 | 15.3 | 48.9 | 40.9 | 37.5 | 48.2 | 14.2 | 39.1 | 18.1 | 35.7 | 29.3 |
| GPT-5 | 19.1 | 6.6 | 5.0 | 45.9 | 48.0 | 47.9 | 46.8 | 14.8 | 17.8 | 11.5 | 34.2 | 27.1 |
| GPT-4o | 9.5 | 4.3 | 14.3 | 55.2 | 23.8 | 22.1 | 33.8 | 9.6 | 32.3 | 34.6 | 23.6 | 23.9 |
| Gemini-2.0-Flash | 9.2 | 5.4 | 18.0 | 48.0 | 25.0 | 17.8 | 34.7 | 8.5 | 42.9 | 34.3 | 16.1 | 23.6 |
| GPT-4o-mini | 12.8 | 3.9 | 14.3 | 47.2 | 20.5 | 22.3 | 33.6 | 10.2 | 27.5 | 41.3 | 25.8 | 23.6 |
| Llama-3.3-70B-Instruct | 8.9 | 3.4 | 15.7 | 48.0 | 12.8 | 17.2 | 32.8 | 8.2 | 30.6 | 41.6 | 27.0 | 22.4 |
| Mistral-Nemo-Instruct | 3.8 | 1.3 | 15.0 | 38.5 | 23.5 | 21.5 | 33.6 | 8.4 | 20.9 | 50.6 | 24.9 | 22.0 |
| Mistral-Large-Instruct | 6.6 | 2.8 | 17.0 | 49.7 | 15.2 | 21.0 | 33.3 | 8.8 | 18.4 | 38.6 | 29.6 | 21.9 |
| Qwen2.5-72B | 8.9 | 3.0 | 16.3 | 48.8 | 9.0 | 18.0 | 32.5 | 9.2 | 31.9 | 31.1 | 27.3 | 21.4 |
| Llama3.1-8B-Instruct | 6.7 | 2.2 | 19.7 | 46.1 | 14.7 | 20.6 | 32.2 | 6.0 | 22.8 | 39.3 | 24.1 | 21.3 |
| Claude-3.7-Sonnet | 7.5 | 0.7 | 17.3 | 30.3 | 29.4 | 23.0 | 35.0 | 8.4 | 20.3 | 31.6 | 27.0 | 20.9 |
| Qwen2.5-7B | 8.2 | 2.9 | 15.7 | 40.6 | 10.4 | 18.8 | 34.7 | 8.5 | 13.7 | 33.5 | 18.2 | 18.7 |
| Claude-3.5-Haiku | 2.6 | 0.9 | 20.7 | 30.1 | 7.8 | 10.6 | 33.3 | 9.2 | 15.8 | 39.6 | 27.9 | 18.1 |

**Table 46: Evaluating LLMs on ExpertLongBench (scaled to 0–100) using checklist precision. Models are sorted by average performance and the best performing model on each task is bolded. Model ranking is indicated by the color of the cell, with green (best) to white (worst).**
| Model | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Gemini-2.5-Pro | 23.1 | 8.1 | 16.0 | 50.3 | 90.8 | 38.5 | 50.0 | 15.4 | 41.5 | 21.5 | 48.3 | 36.7 |
| GPT-5 | 23.7 | 8.0 | 5.7 | 46.6 | 92.0 | 48.6 | 49.3 | 16.7 | 19.3 | 13.5 | 40.8 | 33.1 |
| GPT-4o-mini | 16.2 | 5.4 | 14.7 | 49.5 | 72.7 | 22.7 | 34.1 | 10.6 | 28.4 | 41.8 | 38.0 | 30.4 |
| Gemini-2.0-Flash | 11.7 | 6.4 | 19.0 | 49.3 | 74.7 | 18.2 | 35.6 | 9.5 | 45.8 | 37.0 | 24.5 | 30.2 |
| GPT-4o | 12.2 | 5.4 | 16.7 | 56.6 | 70.4 | 22.6 | 34.3 | 10.4 | 34.3 | 35.3 | 31.6 | 30.0 |
| Mistral-Nemo-Instruct | 4.6 | 1.7 | 16.0 | 39.3 | 72.2 | 22.2 | 33.7 | 8.8 | 24.7 | 50.9 | 32.2 | 27.8 |
| Llama-3.3-70B-Instruct | 11.8 | 4.7 | 16.3 | 48.5 | 53.1 | 17.7 | 33.2 | 8.8 | 31.9 | 42.9 | 32.9 | 27.4 |
| Llama3.1-8B-Instruct | 8.8 | 3.3 | 20.3 | 47.4 | 64.3 | 21.3 | 32.3 | 6.6 | 24.2 | 40.2 | 30.4 | 27.2 |
| Mistral-Large-Instruct | 9.1 | 3.8 | 18.7 | 51.2 | 53.9 | 21.6 | 33.5 | 9.4 | 19.7 | 39.5 | 36.7 | 27.0 |
| Claude-3.7-Sonnet | 10.7 | 0.9 | 17.7 | 30.4 | 75.7 | 23.3 | 35.8 | 9.9 | 20.8 | 32.7 | 32.4 | 26.4 |
| Qwen2.5-72B | 11.6 | 3.8 | 17.3 | 50.2 | 38.4 | 18.5 | 32.6 | 9.5 | 33.8 | 32.6 | 36.6 | 25.9 |
| Qwen2.5-7B | 10.2 | 3.8 | 16.3 | 41.8 | 45.5 | 19.4 | 34.7 | 8.7 | 14.7 | 35.1 | 29.8 | 23.6 |
| Claude-3.5-Haiku | 2.7 | 1.3 | 22.3 | 30.1 | 27.9 | 10.8 | 33.3 | 9.3 | 19.6 | 40.9 | 35.8 | 21.3 |

**Table 47: Evaluating LLMs on ExpertLongBench (scaled to 0–100) using checklist recall. Models are sorted by average performance and the best performing model on each task is bolded. Model ranking is indicated by the color of the cell, with green (best) to white (worst).**
| Model | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Gemini-2.5-Pro | 29.6 | 15.5 | 24.0 | 50.6 | 42.1 | 51.9 | 51.7 | 14.8 | 46.8 | 23.3 | 43.2 | 35.8 |
| GPT-5 | 33.9 | 17.5 | 13.3 | 48.4 | 51.3 | 63.0 | 51.0 | 15.4 | 18.7 | 21.5 | 45.3 | 34.5 |
| Gemini-2.0-Flash | 16.4 | 12.0 | 23.3 | 49.9 | 28.3 | 24.6 | 36.0 | 9.0 | 47.5 | 36.9 | 35.2 | 29.0 |
| GPT-4o | 15.7 | 8.7 | 17.7 | 58.1 | 28.9 | 29.0 | 34.8 | 10.2 | 37.6 | 36.3 | 31.0 | 28.0 |
| GPT-4o-mini | 17.5 | 8.0 | 17.3 | 49.9 | 24.8 | 29.2 | 33.8 | 10.2 | 32.3 | 42.8 | 34.9 | 27.4 |
| Llama-3.3-70B-Instruct | 13.6 | 6.4 | 19.0 | 51.5 | 17.7 | 23.9 | 33.8 | 8.3 | 35.3 | 42.5 | 38.6 | 26.4 |
| Qwen2.5-72B | 14.0 | 6.2 | 20.3 | 52.8 | 14.1 | 25.1 | 32.6 | 9.4 | 40.2 | 34.6 | 34.7 | 25.8 |
| Mistral-Large-Instruct | 10.7 | 5.4 | 19.3 | 53.5 | 20.2 | 27.4 | 34.0 | 8.9 | 20.4 | 39.5 | 38.7 | 25.3 |
| Llama3.1-8B-Instruct | 11.5 | 4.4 | 22.7 | 49.8 | 19.6 | 27.4 | 32.5 | 6.2 | 28.5 | 41.0 | 31.3 | 25.0 |
| Claude-3.7-Sonnet | 14.0 | 2.0 | 19.3 | 30.7 | 32.7 | 29.9 | 36.8 | 10.2 | 22.6 | 33.9 | 39.4 | 24.7 |
| Mistral-Nemo-Instruct | 4.9 | 1.9 | 18.3 | 41.9 | 26.7 | 28.5 | 33.8 | 8.5 | 23.9 | 50.7 | 30.4 | 24.5 |
| Qwen2.5-7B | 12.0 | 5.3 | 18.3 | 44.2 | 13.5 | 25.7 | 34.8 | 8.9 | 17.1 | 35.5 | 23.1 | 21.7 |
| Claude-3.5-Haiku | 3.2 | 1.7 | 23.3 | 30.5 | 13.0 | 11.1 | 33.5 | 9.3 | 18.5 | 40.9 | 36.1 | 20.1 |

**Table 48: Evaluating LLMs on ExpertLongBench (scaled to 0–100) using checklist coverage. Models are sorted by average performance and the best performing model on each task is bolded. Model ranking is indicated by the color of the cell, with green (best) to white (worst). We exclude T5 from this analysis as its responses are limited to Yes/No, with no instances of N/A.**
| Model | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Gemini-2.0-Flash | 94.2 | 100.0 | 91.0 | 92.2 | - | 96.0 | 75.2 | 65.2 | 66.0 | 70.0 | 93.3 | 75.5 |
| Qwen2.5-72B | 94.4 | 100.0 | 92.8 | 98.8 | - | 96.5 | 78.9 | 39.0 | 61.5 | 78.7 | 90.2 | 74.1 |
| Gemini-2.5-Pro | 98.6 | 100.0 | 86.0 | 65.6 | - | 84.9 | 49.5 | 52.8 | 64.8 | 37.2 | 97.7 | 73.7 |
| Llama-3.3-70B-Instruct | 92.8 | 100.0 | 89.2 | 94.6 | - | 94.2 | 80.5 | 65.2 | 51.4 | 57.9 | 95.2 | 73.5 |
| GPT-4o | 93.4 | 99.6 | 92.8 | 96.7 | - | 92.6 | 75.2 | 58.1 | 56.0 | 65.0 | 88.7 | 73.5 |
| GPT-5 | 93.4 | 100.0 | 96.0 | 74.4 | - | 70.3 | 49.8 | 72.4 | 23.7 | 51.4 | 95.8 | 72.7 |
| Llama3.1-8B-Instruct | 92.4 | 100.0 | 88.3 | 93.9 | - | 97.1 | 77.0 | 65.7 | 48.1 | 56.1 | 89.8 | 72.7 |
| Claude-3.5-Haiku | 89.7 | 100.0 | 85.6 | 71.2 | - | 91.1 | 62.9 | 42.4 | 34.8 | 54.6 | 87.7 | 72.0 |
| Claude-3.7-Sonnet | 95.3 | 100.0 | 90.1 | 67.1 | - | 94.2 | 77.7 | 60.8 | 34.9 | 70.9 | 89.3 | 69.6 |
| GPT-4o-mini | 96.0 | 100.0 | 90.1 | 96.0 | - | 91.0 | 56.3 | 46.8 | 48.4 | 53.0 | 93.3 | 69.6 |
| Mistral-Large-Instruct | 94.9 | 99.8 | 88.3 | 98.3 | - | 93.9 | 75.2 | 47.1 | 26.6 | 60.8 | 93.3 | 69.3 |
| Mistral-Nemo-Instruct | 84.3 | 99.1 | 92.8 | 96.0 | - | 95.4 | 72.0 | 51.5 | 49.8 | 19.9 | 86.2 | 68.2 |
| Qwen2.5-7B | 92.9 | 100.0 | 88.3 | 93.4 | - | 93.6 | 56.3 | 43.9 | 32.4 | 71.5 | 79.7 | 67.9 |

### D.2 Task-wise Performance with Standard Errors on ExpertLongBench

In this section, we present comparative plots of model performance for each task individually and report the corresponding standard errors. These errors are estimated using the bootstrap method [Efron and Tibshirani, 1994], a non-parametric resampling technique that assesses the variability of a statistic by repeatedly sampling with replacement from the data. We use 10,000 bootstrap samples to obtain stable and reliable estimate for standard error [Altman and Bland, 2005]. [Fig.˜6](https://arxiv.org/html/2506.01241v3#A4.F6) and [Fig.˜7](https://arxiv.org/html/2506.01241v3#A4.F7) report task-wise performance across models using F1-Score and Accuracy respectively.

Figure: (a) T1LegalMDS
Refer to caption: x7.png

Figure: (a) T1LegalMDS
Refer to caption: x18.png

### D.3 Impact of Input and Output Length on Model Performance

In this section, we evaluate model performance across different length categories to examine how input and output size affect results. To study input effects, we group samples into $3$ categories based on input token length and compute the average performance for each group. Because tasks in our benchmark vary widely in input length distributions, grouping boundaries are determined contextually for each task to ensure that each group contains a roughly balanced number of samples. We apply the same approach to outputs, grouping samples according to the token lengths of their corresponding human references.

To determine which tasks are suitable for this analysis, we computed a normalized interquartile range (IQR) of the length distribution, dividing the IQR by the mean. Only tasks with a normalized IQR greater than $0.2$ were included, since tasks with narrower distributions would not yield informative length-based comparisons. This procedure identified T1, T2, T3, T4, T6, T7, T8, T9, T10, and T11 for input length analysis, and T1, T2, T3, T4, T8, and T9 for output length analysis.

[Fig.˜8](https://arxiv.org/html/2506.01241v3#A4.F8) and [Fig.˜9](https://arxiv.org/html/2506.01241v3#A4.F9) show model performance across different ranges of input and output lengths, respectively. From [Fig.˜8](https://arxiv.org/html/2506.01241v3#A4.F8), we observe that performance declines with increasing input length only for certain tasks. This suggests that while longer inputs may challenge models to process larger contexts, the overall complexity of a datapoint is influenced by additional factors beyond input size alone. In fact, shorter inputs can sometimes reduce performance by limiting available information, as seen in T11.

A similar pattern emerges in [Fig.˜9](https://arxiv.org/html/2506.01241v3#A4.F9), which reports performance across output length categories. While models show declining performance with longer human reference outputs in T1 and T8, this trend is not consistent across other tasks. Given the nature of these tasks, datapoint complexity can sometimes be determined less by output length and more by factors such as domain-specific knowledge, reasoning depth, and task-specific requirements. In some cases, these dimensions—not sequence length alone—serve as the primary drivers of difficulty.

Figure: (a) T1LegalMDS
Refer to caption: x29.png

Figure: (a) T1LegalMDS
Refer to caption: x39.png

## Appendix E Clear : Additional Details and Analysis

### E.1 Validation of Checklist Mapping by GPT-4o : Additional Details

To verify the effectiveness of GPT-4o in mapping checklists, we relied on human validation and automatic evaluation. For human validation, we collected 30 extra data points from tasks T1 and T6 along with their checklist-mapped references. The annotator was then asked to determine whether the checklist information extracted automatically for each item was faithful to the reference text and aligned with the intended information requested by the item. Faithfulness is defined as whether the extracted content exists in the given source text without introducing unsupported information. The human validation instruction is shown in [Tab.˜49](https://arxiv.org/html/2506.01241v3#A5.T49). For this annotation, GPT-4o exhibited high degree of faithfulness, achieving a faithfulness rate of 99.99% for T1 and 95.12% for T6.

Table: Table 49: Human quality validation instruction.

To assess the faithfulness and coverage of relevant information along each checklist item in an automated manner, we relied on evaluation by prompting LLMs.
We modified a prompt previously suggested for measuring completeness (^31^3131[https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-type-kb-llama.html?utm_source=chatgpt.com](https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-type-kb-llama.html?utm_source=chatgpt.com)) [Bayat et al., 2024] to assess both coverage and faithfulness of the extracted checklist answers. The final prompt is shown in [Tab.˜43](https://arxiv.org/html/2506.01241v3#A3.T43).

To mitigate bias from a single evaluator model, we employed two judges, namely Claude-3.7-Sonnet and Gemini-2.0-Flash. Additionally, we avoid using any model from the gpt family to evaluate the extracted checklists, as there is a risk of intra-bias when evaluating within the same model family.
Thereafter, we define coverage for a given task as the proportion of checklist items in which the extracted information includes all the required information from the reference, as determined by both of these evaluators. Similarly, faithfulness can be evaluated using the same approach. For tasks T1 and T6, we obtain coverage scores of 90.3% and 94.3%, and faithfulness scores of 94.8% and 97.9%, respectively—supporting the reliability of the extracted checklists.

### E.2 Model Selection for Checklist Mapper: Additional Details

To select an open-weight mapper, we focused on T1, T6, T7, and T8, which represent the domains of law, healthcare, chemistry, and biology.
By incorporating a range of diverse domains in our evaluation and data points showing diverse input / output length configurations, we aim to identify a mapper that can effectively handle checklist extraction across different fields. We sampled $47$, $107$, $34$, and $64$ datapoints respectively, ensuring that this development set remains distinct from the main test set to avoid data overlap. Moreover, we primarily turn to three highly-capable open-source models, namely Qwen2.5-72B, Llama-3.3-70B-Instruct, and Mistral-Large-Instruct, which have shown remarkable performance in instruction following, reasoning, and domain-specific tasks [Yang et al., 2024, Grattafiori et al., 2024] (^32^3232[https://mistral.ai/news/mistral-large-2407](https://mistral.ai/news/mistral-large-2407)). To evaluate whether smaller models can achieve performance comparable to their larger counterparts, we also analyze smaller models from the same family as the best-performing model on checklist mapping.

To evaluate the quality of a checklist inferred from these models, we use the method outlined in §[4.1](https://arxiv.org/html/2506.01241v3#S4.SS1). The evaluator models used for this process are GPT-4o and Gemini-2.0-Flash (^33^3333We also tried involving Claude-3.7-Sonnet to evaluate the checklist extraction process. However, this model exhibits a strong label bias and assigns positive reward for most instances). The final score assigned to each datapoint is the average of the scores given by these models. Using these two evaluator models, we compute the accuracy and F1-scores for each task and the model considered. As shown in [Tab.˜50](https://arxiv.org/html/2506.01241v3#A5.T50), Qwen2.5-72B achieves competitive or best performance for the considered tasks, indicating its applicability for accurate checklist extraction for a broad range of domains. Given this, we further examine whether its smaller variants such as Qwen2.5-7B, Qwen2.5-14B, and Qwen2.5-32B achieve comparable results, as reported in [Tab.˜50](https://arxiv.org/html/2506.01241v3#A5.T50). While the performance of the small model is decent, the 72B version achieves significantly higher performance and hence we recommend using it to ensure higher evaluation quality. Based on these explorations, we decided to use Qwen2.5-72B to map checklist from the model inference for final assessments.

**Table 50: The performance of different open-source models in checklist mapping scaled to 0-100. Given that Qwen2.5-72B achieves best performance, we also analyze smaller models from the same family.**
| Model | T1 | T6 | T7 | T8 | Average |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 |
| Llama-3.3-70B-Instruct | 78.6 | 84.9 | 83.7 | 87.5 | 93.5 | 94.4 | 65.3 | 69.3 | 80.3 | 84.0 |
| Mistral-Large-Instruct | 81.0 | 85.9 | 87.4 | 90.4 | 89.7 | 90.5 | 85.2 | 88.0 | 85.8 | 88.7 |
| Qwen2.5-72B | 80.9 | 86.2 | 90.0 | 92.6 | 93.6 | 94.2 | 84.3 | 87.2 | 87.2 | 90.1 |
| Checklist Mapping Performance of Smaller Models |  |  |  |  |  |  |  |  |  |  |
| Qwen2.5-7B | 55.1 | 62.7 | 74.2 | 77.0 | 83.6 | 86.3 | 62.0 | 66.0 | 68.7 | 73.0 |
| Qwen2.5-14B | 44.6 | 52.2 | 52.4 | 57.7 | 68.4 | 71.2 | 55.1 | 59.1 | 55.1 | 60.0 |
| Qwen2.5-32B | 51.6 | 59.4 | 67.0 | 71.6 | 81.6 | 82.9 | 72.2 | 76.7 | 68.1 | 72.6 |

### E.3 Correlation of the scores assigned by different models and its combinations with GPT-4o : Full table

This section presents the task-wise assessment of the correlation of the scores assigned by different models combinations with GPT-4o. The final results are presented in [Tab.˜51](https://arxiv.org/html/2506.01241v3#A5.T51).

**Table 51: Correlation between scores assigned by various models and model combinations and those assigned by GPT-4o. Mean and Maj in the parenthesis corresponds to mean and majority pooling respectively.**
| Model | T1 | T2 | T3 | T4 | T6 | T7 | T8 | T9 | T10 | T11 | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-4o-mini | 0.75 | 0.40 | 0.91 | 0.87 | 0.86 | 0.99 | 0.99 | 0.67 | 0.91 | 0.86 | 0.821 |
| Llama3.1-8B-Instruct | 0.36 | 0.14 | 0.85 | 0.83 | 0.74 | 0.81 | 0.80 | 0.70 | 0.87 | 0.42 | 0.652 |
| Llama-3.3-70B-Instruct | 0.64 | 0.55 | 0.84 | 0.86 | 0.80 | 0.99 | 0.98 | 0.84 | 0.88 | 0.71 | 0.809 |
| Mistral-Large-Instruct | 0.68 | 0.53 | 0.89 | 0.92 | 0.81 | 1.00 | 0.98 | 0.84 | 0.90 | 0.78 | 0.833 |
| Mistral-Nemo-Instruct | 0.77 | 0.41 | 0.94 | 0.82 | 0.81 | 0.99 | 0.99 | 0.52 | 0.92 | 0.84 | 0.801 |
| Qwen2.5-7B | 0.74 | 0.30 | 0.91 | 0.83 | 0.82 | 0.99 | 0.99 | 0.49 | 0.90 | 0.83 | 0.780 |
| Qwen2.5-72B | 0.75 | 0.70 | 0.92 | 0.90 | 0.88 | 1.00 | 1.00 | 0.82 | 0.93 | 0.86 | 0.876 |
| Small (Mean) | 0.70 | 0.24 | 0.94 | 0.90 | 0.88 | 0.97 | 0.97 | 0.74 | 0.93 | 0.82 | 0.809 |
| Small (Maj) | 0.78 | 0.45 | 0.94 | 0.86 | 0.84 | 0.99 | 0.99 | 0.57 | 0.92 | 0.85 | 0.819 |
| Large (Mean) | 0.74 | 0.68 | 0.92 | 0.94 | 0.86 | 0.98 | 0.98 | 0.89 | 0.94 | 0.87 | 0.883 |
| Large (Maj) | 0.79 | 0.63 | 0.93 | 0.93 | 0.82 | 0.99 | 0.99 | 0.86 | 0.91 | 0.87 | 0.859 |
| All-Small (Mean) | 0.74 | 0.29 | 0.94 | 0.91 | 0.89 | 1.00 | 0.99 | 0.76 | 0.94 | 0.87 | 0.828 |
| All-Small (Maj) | 0.77 | 0.50 | 0.93 | 0.89 | 0.88 | 1.00 | 1.00 | 0.67 | 0.92 | 0.88 | 0.843 |
| All (Mean) | 0.77 | 0.58 | 0.95 | 0.93 | 0.89 | 0.99 | 0.99 | 0.87 | 0.95 | 0.91 | 0.879 |
| All (Maj) | 0.77 | 0.73 | 0.95 | 0.93 | 0.86 | 1.00 | 1.00 | 0.87 | 0.94 | 0.88 | 0.893 |
| Average | 0.70 | 0.43 | 0.91 | 0.88 | 0.83 | 0.98 | 0.98 | 0.71 | 0.91 | 0.82 | 0.805 |

### E.4 Linking Task Complexity with the Judgement Correlation

Noting that the tasks with the lowest performance—specifically T1 and T2—also exhibit the weakest average correlation with GPT-4o judgments, as shown in

Figure: Figure 10: Regression analysis between the average performance along each task and the average correlation of judgments with GPT-4o assignments
Refer to caption: x45.png

[Tab.˜2](https://arxiv.org/html/2506.01241v3#S5.T2) in the main paper and [Tab.˜51](https://arxiv.org/html/2506.01241v3#A5.T51), this section explores the relationship between task complexity measured by average performance of different models and the degree of correlation between different judges and GPT-4o for each task.

[Fig.˜10](https://arxiv.org/html/2506.01241v3#A5.F10) presents a regression analysis examining the relationship between task complexity and average judgment correlation with GPT-4o. The results reveal a strong positive correlation, suggesting that tasks with higher average model performance—indicative of lower complexity—tend to exhibit stronger alignment with GPT-4o’s judgments. The moderately high $R^{2}$ value indicates that the regression model accounts for a substantial portion of the observed variance. Finally, the high intercept suggests that models can attain substantial average correlation with GPT-4o even with low task performance—as seen in tasks T1 and T8—highlighting that evaluating checklist-mapped responses with references is considerably easier than generating the checklist-specific data itself.

## Appendix F Skill Decomposition Analysis Details

To better understand where current models excel or struggle in ExpertLongBench, we conduct a detailed analysis on a skill-level and difficulty-level.

### F.1 Skill Definitions

We define eight general skills that models may need to demonstrate when completing checklist items:

- 1.
Contextual Understanding: The ability to make sense of a situation by integrating sensory cues, background knowledge, and situational information [Oltramari et al., 2020].
- 2.
Decision Making: Selecting a preferred option or a course of actions from among a set of alternatives [Wang and Ruhe, 2007].
- 3.
Expertise in Workflow: The ability to plan and execute multiple processes – the sequence of steps that achieve complex tasks in a domain [Styles et al., 2024].
- 4.
Information Aggregation: The process to extract, integrate, and synthesize relevant information from multiple sources to enhance understanding  [Bettencourt, 2009]
- 5.
Information Condensing: Summarizing or compressing information to its essential elements without losing core meaning [Li et al., 2024c].
- 6.
Information Grounding: Linking information or claims to their sources or relevant context to ensure accuracy [Lee et al., 2024b].
- 7.
Problem Identification: The process of recognizing and defining a challenge that needs to be solved [Wang and Ruhe, 2007].
- 8.
Problem Solving The process of overcoming obstacles to achieve a goal [Brooks, 2022].

Each skill is described using definitions adapted from prior work and rated using a standardized four-point proficiency scale:

- 1.
N/A: The skill is not meaningfully exercised.
- 2.
Basic: The skill is exercised at a simple or minimal level.
- 3.
Intermediate: The skill is used in moderately complex ways.
- 4.
Advanced: The skill is exercised at a highly complex or abstract level.

An example of two skills—Information Aggregation and Information Condensing—is shown in [Tab.˜52](https://arxiv.org/html/2506.01241v3#A6.T52). The full set of definitions and levels are available in GitHub repository.

Figure: Figure 11: Model performance across various levels of knowledge difficulty.
Refer to caption: x46.png

Table: Table 52: An example of the descriptions and levels of the skills Information Aggregation and Information Condensing.

We additionally annotate each checklist item with two types of difficulty levels:

- 1.
Reasoning difficulty is inspired from Bloom’s taxonomy [Krathwohl, 2002, Yu et al., 2024], where each checklist item is labeled with one of four levels (with the relevant skill in parentheses):
(a)
Low (Knowledge Memorization): Requires recalling or reproducing known facts.
(b)
Medium (Knowledge Understanding): Requires interpreting or inferring meaning from text with light reasoning.
(c)
High (Knowledge Applying): Involves applying knowledge to solve problems, often requiring synthesis or multi-step reasoning.
(d)
Very High (Knowledge Creating): Involves generating novel, logically coherent responses beyond simple inference.
- 2.
Knowledge difficulty is defined as the level of background knowledge required to complete the item. Our primary analysis focuses on College- and Graduate-level items.
(a)
Graduate-level
(b)
College-level
(c)
Below College-level

### F.2 Determining Levels

We use GPT-4o (2024-11-20) as a judge to access the specific levels required to complete each checklist item. For each checklist item, the model is provided with:

- •
The checklist item name and its description (sourced from the task’s Evaluation Rubric)
- •
An example containing the sample input, human reference, and the item result from the checklist-mapped reference.
- •
Definitions and level descriptions for general skills, reasoning difficulty, and knowledge difficulty—with illustrative examples for each level of general skills.

The model is instructed to act like an expert evaluator and to assign each checklist item a skill and difficulty level. For skill levels, the model determines how strongly the item exercises a given skill. The prompt explicitly introduces the level “N/A” which indicates that a particular skill is not meaningfully engaged by the item. For difficulty levels—which do not have formal definitions—the model infers the appropriate level based on descriptive guidance provided for each difficulty tier.

### F.3 Results and Analysis

**Table 53: Evaluating LLMs on ExpertLongBench (scaled to 0–100) using checklist F1 score across various general skills: CU (Contextual Understanding), DM (Decision Making), EW (Expertise in Workflow), IA (Information Aggregation), IC (Information Condensing), IG (Information Grounding), PI (Problem Identification), and PS (Problem Solving).**
| Model | CU | DM | EW | IA | IC | IG | PI | PS |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-5 | 29.4 | 29.5 | 30.0 | 29.1 | 29.6 | 29.1 | 28.9 | 30.1 |
| Gemini-2.5-Pro | 30.8 | 31.7 | 31.4 | 30.6 | 30.8 | 30.7 | 30.8 | 31.9 |
| Claude-3.5-Haiku | 19.1 | 18.7 | 19.2 | 19.0 | 18.2 | 18.8 | 18.6 | 18.8 |
| Claude-3.7-Sonnet | 21.3 | 21.8 | 21.6 | 21.1 | 20.5 | 21.1 | 21.2 | 22.0 |
| Gemini-2.0-Flash | 24.2 | 24.8 | 24.5 | 24.1 | 24.2 | 24.2 | 24.4 | 24.9 |
| GPT-4o | 24.8 | 25.5 | 25.2 | 24.6 | 25.0 | 24.7 | 24.9 | 25.8 |
| GPT-4o-mini | 25.2 | 25.7 | 25.6 | 25.0 | 25.4 | 25.1 | 25.0 | 25.9 |
| Llama3.1-8B-Instruct | 22.7 | 23.4 | 23.0 | 22.5 | 22.8 | 22.5 | 22.7 | 23.4 |
| Llama-3.3-70B-Instruct | 23.8 | 24.5 | 24.2 | 23.7 | 24.0 | 23.8 | 24.0 | 24.7 |
| Mistral-Large-Instruct | 24.2 | 24.9 | 24.5 | 24.0 | 24.4 | 24.1 | 24.2 | 25.1 |
| Mistral-Nemo-Instruct | 22.2 | 23.1 | 22.4 | 21.9 | 22.3 | 22.0 | 22.1 | 23.2 |
| Qwen2.5-7B | 20.9 | 21.4 | 21.3 | 20.8 | 21.0 | 20.8 | 21.0 | 21.6 |
| Qwen2.5-72B | 22.9 | 23.4 | 23.2 | 22.8 | 23.1 | 22.8 | 22.9 | 23.4 |

**Table 54: Evaluating LLMs on ExpertLongBench (scaled to 0–100) using checklist F1 score across different knowledge levels: GL (Graduate level), CL (College level), and reasoning difficulty levels: Low (Knowledge Memorization), Medium (Knowledge Understanding), High (Knowledge Applying), Very High (Knowledge Creating).**
| Model | CL | GL | Low | Medium | High | Very High |
| --- | --- | --- | --- | --- | --- | --- |
| GPT-5 | 42.3 | 22.8 | 52.2 | 47.7 | 26.8 | 19.0 |
| Gemini-2.5-Pro | 35.1 | 27.8 | 46.7 | 43.3 | 30.8 | 20.3 |
| Claude-3.5-Haiku | 23.5 | 14.7 | 28.0 | 22.1 | 18.0 | 16.0 |
| Claude-3.7-Sonnet | 27.0 | 18.3 | 38.9 | 29.3 | 21.4 | 15.0 |
| Gemini-2.0-Flash | 33.4 | 20.4 | 27.7 | 30.3 | 24.7 | 20.8 |
| GPT-4o | 36.0 | 18.4 | 28.2 | 38.6 | 22.4 | 19.3 |
| GPT-4o-mini | 36.7 | 21.8 | 34.8 | 36.4 | 24.4 | 21.1 |
| Llama3.1-8B-Instruct | 31.6 | 18.5 | 23.6 | 31.6 | 21.6 | 17.9 |
| Llama-3.3-70B-Instruct | 32.4 | 19.9 | 25.6 | 34.9 | 22.2 | 20.3 |
| Mistral-Large-Instruct | 36.0 | 18.8 | 26.0 | 37.6 | 22.6 | 17.9 |
| Mistral-Nemo-Instruct | 31.6 | 18.8 | 19.3 | 32.4 | 20.7 | 19.1 |
| Qwen2.5-7B | 25.5 | 17.3 | 25.8 | 27.7 | 18.7 | 16.3 |
| Qwen2.5-72B | 31.3 | 17.9 | 29.6 | 35.3 | 19.1 | 16.5 |

We exclude checklist items from T9 from our analysis, as these tasks contain instance-level items (defined in [Section˜B.9.5](https://arxiv.org/html/2506.01241v3#A2.SS9.SSS5)) that are sample-specific and challenging to evaluate consistently. On the other hand, T5 only consists of items associated with Yes / No response, hence this task is also excluded. We group checklist items by their associated skills or difficulty levels, then report the average performance (checklist F1 score) within each group.

##### Skill-Level Analysis.

[Tab.˜53](https://arxiv.org/html/2506.01241v3#A6.T53) presents model performance across the eight skills. Most models perform comparatively well on Decision Making and Problem Solving, but show weaker performance on Information Aggregation, Information Grounding, and Problem Identification.

##### Difficulty-Level Analysis.

As shown in [Tab.˜54](https://arxiv.org/html/2506.01241v3#A6.T54) and [Fig.˜11](https://arxiv.org/html/2506.01241v3#A6.F11), there is a consistent performance drop from College to Graduate levels. This suggests that current models are insufficiently exposed to tasks requiring deeper or more specialized domain expertise, especially at higher reasoning levels.

## Appendix G Experiment Details

### G.1 Model Information

In [Tab.˜55](https://arxiv.org/html/2506.01241v3#A7.T55), we provide information about the models used in our experiments, including the number of parameters, model context lengths, and pre-training knowledge cutoff dates.
We obtain all the information from their official documentations.

**Table 55: Information about the models used in our experiments.**
| Model | # of Parameters | Context Length | Knowledge Cutoff Date |
| --- | --- | --- | --- |
| Open-source Models |  |  |  |
| Llama3.1-8B-Instruct | 8B | 131072 | Dec 2023 |
| Llama-3.3-70B-Instruct | 70B | 131072 | Dec 2023 |
| Mistral-Nemo-Instruct | 12B | 131072 | Unknown |
| Mistral-Large-Instruct | 123B | 131072 | Unknown |
| Qwen2.5-7B | 7B | 32768 | Unknown |
| Qwen2.5-72B | 72B | 32768 | Unknown |
| Proprietary Models |  |  |  |
| GPT-5 (2025-08-07) | - | 128000 | Oct 2024 |
| GPT-4o-mini (2024-07-18) | - | 128000 | Oct 2023 |
| GPT-4o (2024-11-20) | - | 128000 | Oct 2023 |
| Claude-3.5-Haiku (20241022) | - | 2000000 | Jul 2024 |
| Claude-3.7-Sonnet | - | 2000000 | Nov 2024 |
| Gemini-2.5-Pro | - | 2000000 | Jan 2025 |
| Gemini-2.0-Flash | - | 1000000 | Aug 2024 |

### G.2 Inference Implementation

For open-weight models, we obtain their model weights from Huggingface Hub.(^34^3434[https://huggingface.co](https://huggingface.co)).
We use vLLM [Kwon et al., 2023] to obtain outputs from open-weight models.
All model outputs are obtained with a decoding temperature of 0 (i.e., greedy decoding).
The maximum context length for each model is set to their default value. Inputs longer than the model context length are truncated.

### G.3 Cost Report for Proprietary Models

We report the cost of running the proprietary models in [Tab.˜56](https://arxiv.org/html/2506.01241v3#A7.T56), [Tab.˜57](https://arxiv.org/html/2506.01241v3#A7.T57) and [Tab.˜58](https://arxiv.org/html/2506.01241v3#A7.T58).
We use the batch prediction API provided by each model to reduce the cost, which offers a 50% discount on the cost.
The total cost is $1108.05.

**Table 56: Cost for running the proprietary models (larger variants), including the actual numbers of input and output tokens. The currency is in USD.**
|  | GPT-5 | Gemini-2.5-Pro |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Input | Output | Cost | Input | Output | Cost |
| Task Outputs |  |  |  |  |  |  |
| T1 | 10,758,475 | 292,763 | $8.19 | 16,071,337 | 138,381 | $10.74 |
| T2 | 8,792,090 | 690,173 | $8.95 | 19,430,024 | 253,846 | $13.41 |
| T3 | 15,197 | 183,992 | $0.93 | 15,179 | 45,016 | $0.23 |
| T4 | 29,220 | 141,671 | $0.73 | 30,568 | 9,997 | $0.07 |
| T5 | 215,018 | 395,007 | $2.11 | 215,110 | 113,131 | $0.70 |
| T6 | 310,446 | 172,423 | $1.06 | 332,514 | 55,932 | $0.49 |
| T7 | 13,606 | 686,935 | $3.44 | 14,410 | 36,161 | $0.19 |
| T8 | 17,993 | 816,854 | $4.10 | 16,806 | 40,465 | $0.21 |
| T9 | 238,799 | 282,235 | $1.56 | 194,581 | 47,180 | $0.36 |
| T10 | 6,520,438 | 207,088 | $5.11 | 7,411,504 | 36,868 | $4.82 |
| T11 | 78,958 | 148,394 | $0.79 | 83,508 | 20,813 | $0.16 |
| Total | 623,399,897 | 4,017,535 | $36.97 | 43,815,541 | 797,790 | $31.38 |

**Table 57: Cost for running the proprietary models (larger variants), including the actual numbers of input and output tokens. The currency is in USD.**
|  | GPT-4o | Claude-3.7-Sonnet |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Input | Output | Cost | Input | Output | Cost |
| Task Outputs |  |  |  |  |  |  |
| T1 | 10,758,475 | 118,081 | $14.04 | 13,212,181 | 174,282 | $21.13 |
| T2 | 8,792,090 | 116,585 | $11.57 | 11,387,244 | 251,256 | $18.97 |
| T3 | 15,197 | 29,679 | $0.17 | 17,399 | 23,496 | $0.20 |
| T4 | 29,220 | 13,653 | $0.10 | 104,030 | 17,491 | $0.29 |
| T5 | 215,018 | 102,246 | $0.78 | 235,850 | 65,319 | $0.84 |
| T6 | 310,446 | 105,499 | $0.92 | 349,848 | 108,311 | $1.34 |
| T7 | 13,606 | 46,215 | $0.25 | 25,073 | 38,147 | $0.32 |
| T8 | 17,993 | 76,403 | $0.40 | 28,278 | 50,595 | $0.42 |
| T9 | 238,799 | 51,246 | $0.55 | 284,341 | 90,886 | $1.11 |
| T10 | 6,520,438 | 38,948 | $8.35 | 7,502,880 | 66,186 | $11.75 |
| T11 | 78,958 | 17,227 | $0.18 | 90,901 | 26,232 | $0.33 |
| Reference Checklist Mapping |  |  |  |  |  |  |
| T1 | 1,080,015 | 137,641 | $2.04 | - | - | - |
| T2 | 79,382,710 | 194,869 | $100.20 | - | - | - |
| T4 | 266,767 | 3,068 | $0.35 | - | - | - |
| T5 | 332,900 | 1,900 | $0.42 | - | - | - |
| T6 | 247,230 | 62,608 | $0.62 | - | - | - |
| T7 | 63,402 | 23,430 | $0.20 | - | - | - |
| T8 | 115,116 | 32,439 | $0.31 | - | - | - |
| T10 | 134,734 | 37,025 | $0.35 | - | - | - |
| T11 | 68,855 | 14,457 | $0.16 | - | - | - |
| Checklist Evaluation |  |  |  |  |  |  |
| T1 | 91,371,736 | 119,600 | $109.90 | - | - | - |
| T2 | 150,441,047 | 188,599 | $180.69 | - | - | - |
| T3 | 11,041,278 | 14,400 | $12.79 | - | - | - |
| T4 | 24,770,680 | 34,972 | $29.94 | - | - | - |
| T6 | 105,705,546 | 139,200 | $122.62 | - | - | - |
| T7 | 21,970,012 | 28,800 | $25.45 | - | - | - |
| T8 | 18,560,414 | 24,000 | $21.51 | - | - | - |
| T9 | 31,077,942 | 45,010 | $38.66 | - | - | - |
| T10 | 54,269,374 | 75,332 | $64.89 | - | - | - |
| T11 | 21,630,526 | 28,800 | $25.10 | - | - | - |
| Checklist Mapper Justification |  |  |  |  |  |  |
| - | - | - | - | 1,200,800 | 223,034 | $3.47 |
| Checklist Judge Justification |  |  |  |  |  |  |
| - | 40,583,208 | 59,176 | $51.02 | 44,674,806 | 118,355 | $67.90 |
| Total | 701,093,452 | 1,995,708 | $837.73 | 79,113,631 | 1,253,590 | $128.07 |

**Table 58: Cost for running the proprietary models (smaller variants), including the actual numbers of input and output tokens. The currency is in USD.**
|  | GPT-4o-mini | Claude-3.5-Haiku | Gemini-2.0-Flash |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Input | Output | Cost | Input | Output | Cost | Input | Output | Cost |
| Task Outputs |  |  |  |  |  |  |  |  |  |
| T1 | 10,758,475 | 94,925 | $0.84 | 13,212,420 | 53,817 | $5.39 | 16,071,337 | 147,219 | $2.50 |
| T2 | 8,792,090 | 85,659 | $0.69 | 11,547,302 | 54,303 | $4.73 | 19,430,024 | 398,438 | $3.15 |
| T3 | 15,197 | 26,591 | $0.01 | 17,399 | 20,397 | $0.05 | 15,179 | 30,222 | $0.02 |
| T4 | 29,220 | 11,573 | $0.01 | 104,030 | 21,842 | $0.09 | 30,568 | 8,463 | $0.01 |
| T5 | 215,018 | 67,929 | $0.04 | 235,850 | 45,679 | $0.19 | 215,110 | 102,179 | $0.09 |
| T6 | 310,446 | 80,575 | $0.05 | 349,848 | 78,021 | $0.30 | 332,514 | 97,980 | $0.11 |
| T7 | 13,606 | 37,534 | $0.01 | 25,073 | 36,787 | $0.08 | 14,410 | 39,499 | $0.03 |
| T8 | 17,993 | 57,480 | $0.02 | 28,278 | 49,443 | $0.11 | 16,806 | 44,930 | $0.03 |
| T9 | 238,799 | 45,179 | $0.03 | 284,341 | 57,056 | $0.23 | 194,581 | 81,912 | $0.08 |
| T10 | 6,520,438 | 35,053 | $0.50 | 7,502,880 | 37,542 | $3.08 | 7,411,504 | 41,890 | $1.14 |
| T11 | 78,958 | 13,437 | $0.01 | 90,901 | 26,441 | $0.09 | 83,508 | 15,587 | $0.02 |
| Checklist Mapper Justification |  |  |  |  |  |  |  |  |  |
| - | - | - | - | - | - | - | 1,107,529 | 237,895 | $0.31 |
| Checklist Judge Justification |  |  |  |  |  |  |  |  |  |
| - | - | - | - | - | - | - | 45,909,168 | 29,992 | $6.90 |
| Checklist Judge Analysis (Low-cost Judge) |  |  |  |  |  |  |  |  |  |
| T1 | 83,526,058 | 114,400 | $6.30 | - | - | - | - | - | - |
| T2 | 137,180,758 | 180,403 | $10.34 | - | - | - | - | - | - |
| T3 | 9,305,446 | 13,200 | $0.70 | - | - | - | - | - | - |
| T4 | 22,865,690 | 33,572 | $1.72 | - | - | - | - | - | - |
| T6 | 89,425,206 | 127,600 | $6.75 | - | - | - | - | - | - |
| T7 | 18,531,888 | 26,400 | $1.40 | - | - | - | - | - | - |
| T8 | 15,668,198 | 22,000 | $1.18 | - | - | - | - | - | - |
| T9 | 30,420,132 | 44,528 | $2.29 | - | - | - | - | - | - |
| T10 | 48,962,686 | 71,500 | $3.69 | - | - | - | - | - | - |
| T11 | 18,307,658 | 26,400 | $1.38 | - | - | - | - | - | - |
| Total | 501,124,960 | 1,216,118 | $37.96 | 33,398,083 | 481,328 | $14.34 | 90,832,238 | 1,276,206 | $14.39 |

## Appendix H Comparison with Existing Benchmarks

**Table 59: Comparison between our benchmark and existing benchmarks. “#Max Input” and “#Max Output” refers to the max input token length and max output token length, rounded to the nearest integer.**
| Benchmark | Output Form | Topics | #Max Input | #Max Output |
| --- | --- | --- | --- | --- |
| MMLU Hendrycks et al. [2020] | Multi-choice | Domain-specific | 927 | - |
| AGIEval Zhong et al. [2023] | Multi-choice | Domain-specific | 1,708 | - |
| GPQA Rein et al. [2024] | Multi-choice | Domain-specific | 822 | - |
| ExpertQA Malaviya et al. [2023] | Long-form | Domain-specific | 80 | 646 |
| WildBench Lin et al. [2024] | Long-form | Common | 7,525 | 2,421 |
| BIGGEN BENCH Kim et al. [2024] | Long-form | Common | 4,370 | 3,777 |
| ExpertLongBench (ours) | Long-form | Domain-specific | 1,998,517 | 15,801 |

To further analyze the difference between ExpertLongBench and other existing benchmarks, we show the benchmark statistics in [Tab.˜59](https://arxiv.org/html/2506.01241v3#A8.T59). For ExpertQA Malaviya et al. [2023], WildBench Lin et al. [2024], and BIGGEN BENCH Kim et al. [2024], the maximum output length shown in [Tab.˜59](https://arxiv.org/html/2506.01241v3#A8.T59) is calculated based on the length of the model output, as there are no human-written references available. For ExpertLongBench, the maximum output length is determined by the length of the human-written reference, which serves as an estimate of the number of tokens reasonably required to solve the task.
Unlike domain-focused benchmarks such as MMLU Hendrycks et al. [2020], AGIEval Zhong et al. [2023], GPQA Rein et al. [2024], and ExpertQA Malaviya et al. [2023], which predominantly use limited-context, multiple-choice or short-answer formats, ExpertLongBench is designed for extended content generation, supporting a maximum input length of nearly 2 million tokens (1,998,517) and output lengths up to 15,801 tokens, surpassing the context and output sizes of prior benchmarks. In ExpertLongBench, these maximum lengths are observed in T2. This capacity enables evaluation of tasks requiring comprehensive analysis of entire documents or large knowledge bases within specialized fields, scenarios that earlier benchmarks with short inputs and outputs could not accommodate. Moreover, existing long-form generation benchmarks (e.g., WildBench Lin et al. [2024], BIGGEN BENCH Kim et al. [2024]) have largely focused on common or general topics for open-ended responses, whereas ExpertLongBench targets challenging expert-domain scenarios across specialized disciplines. The long-form nature that better align with expert-level tasks in real world, together with the focus on expert domain content, set ExpertLongBench apart from existing benchmarks, allowing rigorous assessment of language models on complex, domain-specific tasks that demand lengthy responses.

## Appendix I Additional Discussion

### I.1 Performance with Ground-truth Rubric

Table: Table 60: T2LegalSFG - Detailed model prompt.

We investigate the effect of detailed prompt that contains the ground-truth rubric on ExpertLongBench, which includes all evaluation checklist items, and compare it to the performance under a generic and high-level prompt as shown in the main paper. Task T2LegalSFG is selected for this analysis as it contains the largest number of checklist items, making it particularly suitable for evaluating the effects of prompt specificity on complex tasks. We focus on the two top-performing models on T2 as shown in the main result table in the main paper, GPT-4o and Gemini-2.0-Flash.
In the detailed prompt, the models are explicitly provided with the full set of evaluation rubric designed or verified by expert.
To minimize differences from the generic prompt and show the effect of rubric exposure, we construct the detailed prompt by appending the phrase “You will be evaluated based on the following rubric:” followed by the complete set of evaluation rubrics designed or verified by domain experts.
The prompt is shown in [Tab.˜60](https://arxiv.org/html/2506.01241v3#A9.T60) and corresponding results are presented in [Tab.˜61](https://arxiv.org/html/2506.01241v3#A9.T61).

**Table 61: The performance of generic prompt and detailed prompt on T2LegalSFG scaled to 0-100.**
| Prompt | GPT-4o | Gemini-2.0-Flash |  |  |
| --- | --- | --- | --- | --- |
| Accuracy | F1 score | Accuracy | F1 score |  |
| Generic | 4.3 | 6.2 | 5.4 | 7.9 |
| Detailed | 29.7 | 32.5 | 25.0 | 28.6 |

The results demonstrate that detailed prompts lead to substantial improvements in performance. Specifically, GPT-4o achieves an increase in accuracy from 4.3 to 29.7 and in F1 score from 6.2 to 32.5. Similarly, Gemini-2.0-Flash shows an improvement in accuracy from 5.4 to 25.0 and in F1 score from 7.9 to 28.6. These findings indicate that providing models with the detailed evaluation rubric significantly enhances their ability to align with task-specific requirements and produce more accurate outputs. However, despite these improvements, the performance remains relatively low, with F1 scores still below 33 and accuracy below 30 for both models, suggesting that the task continues to pose significant challenges for current state-of-the-art models even when providing the ground-truth evaluation checklist.

### I.2 Limitations of LLMs in Generating High-Quality Evaluation Rubric

Table: Table 62: Prompt used to guide LLMs in generating fine-grained evaluation checklists.

Table: Table 63: Prompt used to guide LLMs in generating fine-grained evaluation checklists for T1LegalMDS.

We evaluate the ability of LLMs to generate high-quality fine-grained evaluation checklists on Tasks T1LegalMDS, T6HealthCNG, and T7ChemMDG. Specifically, we use GPT-4o (2024-11-20) with the prompt provided in [Tab.˜62](https://arxiv.org/html/2506.01241v3#A9.T62) to produce detailed fine-grained checklists for each task. Example prompt for T1LegalMDS is shown in [Tab.˜63](https://arxiv.org/html/2506.01241v3#A9.T63).

**Table 64: Assessment of evaluation checklist items generated by GPT-4o. #Item (Human) indicates the number of evaluation checklist items created by humans for the corresponding task. #Item (LLM) refers to the number of evaluation checklist items generated by GPT-4o. #Subjective denotes the number of model-generated checklist items that are considered subjective. #Unsuitable/Redundant represents the number of model-generated checklist items deemed unsuitable or redundant. #Coarse-grained indicates the number of model-generated checklist items that are not fine-grained enough.**
| Task | #Item (Human) | #Item (LLM) | #Subjective | #Unsuitable/Redundant | #Coarse-grained |
| --- | --- | --- | --- | --- | --- |
| T1LegalMDS | 26 | 20 | 6 | 5 | 3 |
| T6HealthCNG | 29 | 22 | 7 | 1 | 3 |
| T7ChemMDG | 6 | 7 | 3 | 2 | 1 |

For tasks T1LegalMDS, T6HealthCNG, and T7ChemMDG, our expert-guided evaluation checklists contained 26, 29, and 6 items, respectively, while model-generated checklists included 20, 22, and 7 items for T1LegalMDS, T6HealthCNG, and T7ChemMDG. We conducted a manual evaluation of the model-generated checklist items and identified several issues:

- 1.
*Subjectivity in checklist items*: The model occasionally produced subjective items, where different annotators might interpret the scoring criteria inconsistently (e.g., GPT-4o generated a checklist item assessing “neutrality” which evaluates whether the summary is free from bias or subjective interpretation for T1LegalMDS. However, this criterion is itself subjective and may be interpreted differently by different annotators);
- 2.
*Omission of expert-guided checklist items*: Important checklist items created by human experts were missing (e.g., missing checklist items such as conjugate acid and conjugate base in T7ChemMDG);
- 3.
*Generation of unsuitable or redundant items*: The model generated unsuitable or redundant items (e.g., GPT-4o generated a checklist item assessing “peer review” which evaluates whether the summary has been reviewed by a legal expert or peer for accuracy and clarity for T1LegalMDS);
- 4.
*Lack of granularity*: Some items lacked sufficient granularity (e.g., GPT-4o generated a checklist item assessing “assessment accuracy,” where it evaluates whether the clinical assessment reflects the patient’s current condition based on the conversation. This checklist item is not fine-grained enough because it conflates multiple aspects—such as the correctness of symptom interpretation, appropriateness of diagnostic reasoning, and alignment with clinical guidelines—into a single criterion, making it difficult to assess specific strengths or weaknesses in the model’s output.);
- 5.
*Misunderstandings of what should be covered in specific parts*: The model misunderstood the structure of domain-specific sections, such as mistakenly putting the chief complaint in the History of Present Illness in T6HealthCNG.

The distribution of problematic items is illustrated in [Tab.˜64](https://arxiv.org/html/2506.01241v3#A9.T64).

In conclusion, LLM-generated fine-grained checklist-based evaluation rubric often exhibit significant limitations, including subjectivity in item creation, missing critical expert-designed checklist items, generation of unsuitable or overly broad items, and a lack of contextual understanding of what information item need to be covered in each part. Overall, GPT-4o does not have the capability to generate rigorous and complete evaluation checklists like human experts. These findings highlight the need for careful human oversight and domain expertise in designing a reliable evaluation rubric.

### I.3 Limitation

While our benchmark offers a comprehensive foundation for evaluating LLMs on expert-level tasks, several limitations merit consideration. First, although LLM-based evaluation methods have shown promising alignment with expert judgments, they can still yield erroneous or inconsistent assessments, especially in complex or ambiguous cases. Second, our benchmark is limited to English-language tasks, and does not yet address the needs of multilingual expert applications. Third, our analysis focuses on the off-the-shelf performance of LLMs without exploring complex prompting strategies, tool use, or agentic workflows that are increasingly common in applied research. Fourth, while we provide fine-grained evaluation criteria tailored to expert domains, this paper does not focus on proposing concrete strategies for model improvements. Finally, although our dataset spans 11 expert-level tasks across 9 domains, it captures only a small fraction of the thousands of real-world expert applications. We plan to expand the benchmark with additional tasks and targeted evaluation metrics to further enhance its coverage and utility.

### I.4 Broader Impacts

As LLMs grow increasingly capable, their use is rapidly expanding beyond general-purpose tasks to high-stakes, expert-level domains such as law, medicine, and education. This shift presents both tremendous opportunities and serious challenges. On one hand, LLMs hold the potential to democratize access to expert services, reduce cognitive burdens, and improve efficiency in professional settings. On the other hand, inaccurate or hallucinated outputs in such contexts can lead to harmful consequences—legal misjudgments, medical errors, or the spread of educational misinformation. Our work contributes to the responsible scaling of LLMs into expert domains by introducing a benchmark that rigorously evaluates model performance on realistic, end-to-end expert tasks across multiple disciplines. By incorporating expert-written references and task-specific evaluation rubric, we aim to facilitate grounded, transparent, and fine-grained assessments of model capabilities.
The ExpertLongBench, together with Clear, better aligns with the needs and judgments of experts and professionals, providing a stronger foundation for evaluating LLM progress.
We hope this work will guide future research and deployment practices, encouraging the AI community to prioritize rigorous evaluation when applying LLMs in expert-level scenarios.

### I.5 Licenses

[Tab.˜65](https://arxiv.org/html/2506.01241v3#A9.T65) summarizes the original licenses of the databases used. The authors bear all responsibility in case of violation of rights, and confirm the dataset licenses.

**Table 65: Licenses associated with datasets used across tasks.**
| License | Tasks |
| --- | --- |
| Attribution-NonCommercial 4.0 International (CC BY-NC) | T1LegalMDS |
| Apache License 2.0 | T4EduPAE (Tutorbot-Spock) |
| Creative Commons Attribution 4.0 (CC BY 4.0) | T3MaterialSEG (Science and Wiley papers), T6HealthCNG (ACI-Bench), T7ChemMDG (Text2Mol) |
| MIT License | T8BioPDG (SciKnowEval), T11CyberRDG (AgentHarm) |
| Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA) | T11CyberRDG (R-Judge) |
| Unavailable / Private | T2LegalSFG, T5EduFG, T9MedicalDR (DiReCT), T10FinanceESG |

### I.6 New Assets

The data for tasks T1LegalMDS, T3MaterialSEG, T4EduPAE, T6HealthCNG, T7ChemMDG, T8BioPDG, and T11CyberRDG are shared under the CC BY-NC-SA 4.0 license(^35^3535[https://creativecommons.org/licenses/by-nc-sa/4.0/](https://creativecommons.org/licenses/by-nc-sa/4.0/)), while the data for tasks T2LegalSFG, T5EduFG, T9MedicalDR, and T10FinanceESG remain private.

## Appendix J Author Contribution

As the lead faculty author, Lu provided overall research direction and supervised the execution of the project.
Jie, Inderjeet, and Shuyang are the lead authors of this project. Jie led the initial idea development, benchmark design, rubric formulation, and overall evaluation pipeline design and project organization. Jie also coordinated contributions to ensure consistency across domains and tasks and participated in experimental design and execution. Inderjeet led the development and execution of the evaluation framework and metrics, as well as the exploration of open-weight evaluation alternatives for checklist mapping and evaluation and participated in design planning. Shuyang was responsible for overseeing efficient experiment running and also contributed to design discussions.

For specific benchmark tasks:

- •
Jie led Tasks T1, T2, and T6, with Amy assisting in data processing for T2 and T6.
- •
Inderjeet led Tasks T4, T5, and T11, with Amy contributing to T11.
- •
Shuyang led Tasks T3 and T10.
- •
Amy led Tasks T7, T8, and T9, with Jie working with her and domain-specific collaborators to ensure the rubric design aligns with project goals.
- •
Sheza managed the release of datasets and the leaderboard on Hugging Face and Kaggle, and contributed to identifying datasets that include expert-level tasks and rubric, and selecting tasks that aligns with the goal of this project.

All these five authors contributed to the writing of the paper.

For other authors, Micah Pollens-Dempsey, Jasmine Gump, Tessa Bialek and Margo Schlanger contributed to the data collection, processing, curation, and rubric design for T1.
Tiffany Chiang, Lucy Kates and Vivek Sankaran contributed to the data collection, processing, curation, and rubric design for T2.
Nicholas David contributed to the data selection and rubric design of T3.
Sihan Chen contributed to the rubric design of T7.
Ruxin Yang and Yuqian Yang contributed to the rubric design of T8.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: NA
soundness: 2
strengths: 1. This work fills the gap in expert-level long-form generation evaluation, which appears to be of considerable significance.
2. This work proposes CLEAR, a method that extracts checklists from reference outputs for comparison. This approach exhibits higher accuracy compared to existing rubric-based methods, and the experiments also demonstrate that models may potentially game such rubric-based approaches.
summary: This paper introduces EXPERTLONGBENCH, an expert-level benchmark with 11 tasks across 9 domains, focusing on realistic long-form expert workflows (e.g., legal case summarization, clinical note generation) that require outputs exceeding 5,000 tokens. Each task includes an expert-designed/validated rubric and human-written references for grounded evaluation.
It also proposes CLEAR, an evaluation framework that extracts checklists from model outputs and references based on task rubrics, then compares checklist items for fine-grained, objective assessment. Open-weight models like Qwen2.5-72B work effectively in CLEAR, enabling low-cost, reproducible evaluation.
Benchmarking 13 LLMs shows top model Gemini-2.5-Pro only achieves an average F1 of 33.4, revealing significant gaps. Critically, high checklist item coverage (irrespective of correctness) correlates negatively with F1—models often produce content seemingly aligned with domain standards but incorrect, posing misleading risks. This work fills gaps in expert-level long-form LLM evaluation and guides future model improvement.
weaknesses: 1. The long-form evaluation method is excessively costly. As shown in Table 1, many tasks involve input exceeding 100,000 tokens. For a single task with 100 samples, 10 million tokens are required—and this only accounts for input, excluding the cost of completion. The authors put their cost in Section G.3, where the evaluation costs them over $1000.  The practicality of evaluation at this scale is questionable. 
2. In terms of evaluation, while it assesses long-form generation capabilities, it omits consideration of various solutions for long-form text processing, such as RAG (retrieval-augmented generation) approaches or the integration of agentic workflows (e.g., preliminary summarization). In real-world scenarios, it is uncommon to directly feed such long prompts to models for generation. This aspect lacks practical relevance.
3. While the paper validates GPT-4o’s consistency with another closed model (Gemini-2.0-Flash, Cohen’s Kappa: 0.81–0.89, Section 4.2) and verifies checklist extraction quality via human inspection (T1/T6: 95.12–99.99% faithfulness, Section E.1), it does not compare GPT-4o’s evaluation results with human expert judgments. The authors acknowledge this gap in Section I.3 (Limitations), noting that “LLM-based evaluation can yield erroneous assessments and requires careful human oversight”—which echoes your concern about unvalidated reliance on GPT-4o.
4. A portion of the data is not publicly available, which hinders the community from conducting tests and follow-up research.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See the above.
soundness: 3
strengths: 1. The authors present a benchmark with long inputs and outputs that requires expert-level knowledge, covering 11 domains. This is a valuable resource for the community.

2. The authors conduct comprehensive experiments on ExpertLongBench, including many frontier models, and show that the benchmark remains highly challenging even for top-performing models.

3. The authors devote substantial effort to data quality and model analysis, for example examining potential data contamination and designing an automated, relatively reliable evaluation pipeline.
summary: This paper introduces ExpertLongBench, a benchmark for expert-level generation with long input and long output, and a checklist-based evaluation method called CLEAR. The benchmark covers 11 domains, including law, science, medicine, education, and finance. Each task comes with an expert-written rubric and a reference that is mapped into a checklist, so an LLM judge can test whether a system answer covers each item. The authors provide a public subset, a private subset for held-out testing, code, and an evaluation pipeline. The goal is to measure factual coverage and faithfulness in realistic expert workflows rather than surface fluency.

Experimentally, the paper evaluates a broad set of LLMs on the suite and finds that even leading models perform far from perfect, with average F1 around 30. The tasks include very large inputs and long outputs, and the analysis shows a common pattern of high coverage but lower correctness. Human spot checks confirm the quality of the reference mapping on selected tasks. Judge reliability is studied by comparing GPT-4o with other judges, showing strong agreement and good correlation with an open model judge, and the pipeline supports using open models for the mapping step with similar rankings.
weaknesses: 1. **Checklist and rubric reliability.** My main concern is the quality of the checklists and rubrics, which the overall evaluation heavily relies on. While I acknowledge the authors’ efforts, many tasks appear to be curated by a single PhD student in a related field, which may be insufficient and could introduce bias. An ideal setup would involve multiple experts for curation, additional experts for independent review, and iterative updates until the experts reach a reliable agreement. I only noticed that T8 mentions experts “reached consensus,” without any quantitative agreement metrics. Introducing a more thorough process and reporting would improve confidence in the benchmark.

2. **“Long-form” positioning.** Although the paper claims a long-form generation benchmark, Table 1 shows that 8 of 11 tasks have inputs under 2k tokens and 9 of 11 tasks have references under 500 tokens. I understand this is already longer than many expert-level datasets, but compared with long-context-focused benchmarks (e.g., LongBench and L-Eval), it is less convincing to label the suite as a long-context benchmark.

3. **Missing references.** I recommend adding a comparison and citation to HealthBench. If the authors intend to position this work as a long-context benchmark, it would also help to compare with and cite LongBench and L-Eval, which include long-form generation tasks.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. The main concern is the low sample size (100 per task). How did the authors validate that this sample size is statistically sufficient to be representative of the entire domain and to produce stable, reliable model rankings?

2. The CLEAR framework's results are contingent on the LLM used for mapping and judging (e.g., Qwen2.5-72B and GPT-4O). How sensitive are the final F1 scores (Table 2) to this choice? If you were to swap the judge/mapper to a different models (e.g., a Claude or Llama model), do the model rankings remain consistent?

3. Given the high cost of expert rubric design and the high risk of data contamination, what is the long-term vision for maintaining and expanding this benchmark? How can this approach scale without requiring hundreds of hours of domain-expert time for each new task?

4. The finding of high coverage but low F1 is fascinating. Do the authors have a hypothesis why this is happening? Is it a failure of long-context retrieval (the model fails to find the right facts in the 200K token input) or a failure of generation (the model finds the facts but hallucinates or misrepresents them to fit the required checklist structure)?
soundness: 3
strengths: 1. The paper addresses a critical and widely acknowledged gap in LLM evaluation. Most benchmarks test for factual recall via multiple-choice or short-form answers. This work makes a significant contribution by shifting the evaluation to what experts actually do.

2. The CLEAR framework is a major strength. The idea of not comparing two long texts directly, but instead using an expert-designed rubric to decompose the task into a structured checklist, is a very strong and novel approach to "grounded" evaluation. 

3. The paper's key findings are highly valuable. The primary result that even SOTA LLMs fail badly is a crucial data point for the community. More importantly, the discovery of a negative correlation between checklist coverage and F1 score is a critical insight, providing concrete evidence that models are adept at generating plausible, structured, but factually incorrect content, which is a key risk.
summary: This paper introduces EXPERTLONGBENCH, a new benchmark designed to evaluate LLMs on expert-level, long-form generation tasks. The benchmark consists of 11 tasks from 9 professional domains that mimic real-world workflows, featuring long contexts (up to 200K tokens) and long-form outputs (up to 5K tokens). The primary contribution is the CLEAR evaluation framework. Instead of direct text-to-text comparison, CLEAR uses expert-designed "rubrics" to create a "checklist" of required information. It then uses an LLM to extract this checklist of information from both the model's output and a human-written reference, and compares these two checklists item-by-item. The paper benchmarks 13 SOTA LLMs, finding that even the best model (Gemini-2.5-Pro) achieves a very low F1 score (~33.4%), revealing significant gaps in current LLM capabilities for expert tasks.
weaknesses: 1. A weakness is the small number of samples. The benchmark contains 1050 samples total, about 100 samples/each. It is questionable whether 100 samples are sufficient to draw robust conclusions about a model's performance across an entire complex domain like law or medicine. This small scale could lead to noisy results.

2. The paper states that a portion of the benchmark is public. Given the small sample size (50-100 per task) and the highly unique, long-form nature of the prompts and data, these tasks are at a very high risk of being included in the training data of future models, which would quickly render the public benchmark obsolete for evaluation.

3. CLEAR relies on SOTA LLMs to function as "Checklist Mappers" and "Judges." This introduces a circular dependency where the evaluation of LLMs is dependent on the capabilities of another LLM. While Section 6 attempts to validate this, it's still a source of potential variance.

4. The benchmark's greatest strength (expert-designed rubrics) is also its greatest weakness. The paper notes that creating the rubric for a single task (T1) took over 10 hours of expert time. This creates a massive bottleneck, making the framework difficult and expensive to scale to new domains or even new tasks within the same domain.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Since the benchmark is so domain-specific, why not conduct some sort of RAG-based evaluations? I do understand that this makes the evaluation a two-phase process (retrieval evaluation and then generation evaluation, given the retrieval). However, I feel it might be unfair to expect the LLM to know intricate and nuanced domain-specific information, which is arcane to common users, and it might reflect the true abilities of the model.
- I suggest that some reasoning models also be evaluated on this benchmark, especially given Figure 5. It would be interesting to see if test-time scaling improves the performance on harder problems. I feel this will be an interesting takeaway, which might show the efficacy of reasoning on non-math/code and factual, knowledge-intensive tasks. 
- It would also be interesting to see how the rankings of these LLMs on `ExpertLongBench` compare to `MMLU`/`GPQA` and `LMArena` rankings. The latter is generally treated as the empirical ranking, while MMLU is _(unfortunately)_ reported as a standard benchmark, and many model builders make their choice solely based on it. Hence, a difference in ranking would further motivate the need for domain-specific and long-context evaluations. However, I do acknowledge that most of the questions in MMLU and LMArena might be quite small compared to `ExpertLongBench`, which specializes in long-context queries. This won't be an apples-to-apples comparison. 
- If possible, I urge the authors to add a small study on LLM-evaluator's self-bias, as this benchmark provides an ideal ground for it. In an $N \times N$ matrix, we can have the performance of LLMs as evaluated by every LLM, including itself. On this hard benchmark, if we notice the diagonal consistently higher, it would be a clear example of self-bias, where the model scores itself higher despite a lack of knowledge of the domain/incorrect answers. I hope this is clear.
soundness: 3
strengths: - The paper is well articulated, and I appreciate the long-context benchmark provided in the paper. I strongly resonate with the authors' motivation, as long-context evaluation techniques are indeed understudied. Additionally, the benchmark covers some practical tasks, whose data is not readily available. 
- I also acknowledge the capital and manual effort invested in this benchmark creation process, and I believe this will be a solid addition to the standard evaluation suite for LLMs. 
- I also appreciate the authors' choice to keep the test set private to avoid contamination.
summary: This paper introduces `ExpertLongBench`, a long-form benchmark curated by experts across 11 tasks and 9 domains. The authors provide a detailed walkthrough of the benchmark creation process and propose an evaluation framework called `CLEAR`, and evaluate 13 LLMs on `ExpertLongBench` using the `CLEAR` framework.
weaknesses: - I don't find any glaring technical errors in the paper. Not a weakness per se, by most of the content of this paper is about the benchmark creation process using experts-in-the-loop, while the second half discusses a standard benchmarking process on this benchmark following the `CLEAR` framework. The technical takeaways are minimal compared to the data contribution. 
- Again, not a weakness per se, but a massive chunk of the important information is stowed away in the appendix, and it is a hassle to refer back and forth. 
- I do have some comments, suggestions, and queries for the authors below, which I request the authors to clarify. 
- The benchmark might not be extendable to other domains or languages, given the expert-in-the-loop methodology.


Decision: Accept (Poster)