Yourbench: Dynamic Evaluation Set Generation with LLMs
Authors: Sumuk Shashidhar
Clémentine Fourrier
Alina Lozovskaya
Thomas Wolf
Gokhan Tur
Dilek Hakkani-Tür
Venue: COLM 2025 (2025)
Keywords: benchmarking
contemporary dataset
dataset
reference-free
automated
llm
TL;DR: a new system to generate diverse question answers from source documents ensuring maximum document coverage

Abstract: Large language models (LLMs) have rapidly outpaced traditional evaluation methodologies, with static benchmarks suffering from saturation, contamination, and domain-specificity limitations while human evaluation remains prohibitively expensive. We present YourBench, an open-source framework that transforms this evaluation paradigm by enabling automated generation of reliable, contamination-free benchmarks directly from user-provided documents without human annotation. To validate our approach, we successfully reproduce the challenging MMLU-Pro benchmark across 86 models spanning 400M to 405B parameters, achieving remarkable Pearson correlations of 0.91-0.99 while generating entirely novel questions for under $15 per model. This demonstrates that dynamically generated evaluations can match the discriminative power of expert-curated benchmarks while eliminating contamination risks. YourBench enables researchers to create domain-specific benchmarks in minutes rather than months. We demonstrate applications in agriculture, personalized education, and RAG training that were previously infeasible. By releasing the YourBench library, Tempora-0325 dataset, 150K+ generated QA pairs, and all evaluation traces, we provide the community with a practical solution to the challenge of keeping pace with rapidly evolving model capabilities.

YourBench: Easy Custom Evaluation Sets for Everyone
Sumuk Shashidhar1 Clémentine Fourier Alina Lozovskia'
Thomas Wolf' Gokhan Tur? Dilek Hakkani-Tür
HuggingFace II UTCC
sumuks2@illinois. edu
clementineehuuginngfacccc
Abstract
Large language models (LLMs) have rapidly outpaced traditional eval- uation methodologies, with static benchmarks suffering from saturation, contamination, and domain-specificity limitations while human evaluation remains prohibitively expensive. We present YourBench, open-source framework that transforms this evaluation paradigm by enabling auto- mated generation of reliable, contamination-free benchmarks directly from user-provided documents without human annotation. To validate our approach, we successfully reproduce the challenging MMLU-Pro bench- mark across 86 models spanning 400M to 405B parameters, achieving re- markable Pearson correlations of 0.91-0.99 while generating entirely novel questions for under $15 per model. This demonstrates that dynamically gen- erated evaluations can match the discriminative power expert-curated benchmarks while eliminating contamination risks. YourBench enables researchers to create domain-specific benchmarks in minutes rather than months. We demonstrate applications in agriculture, personalized educa- tion, and RAG training that were previously infeasible. By releasing the YourBench library, TEMPORA -0325 dataset, 150K+ generated QA pairs, and all evaluation traces, we provide the community with a practical solution to the challenge of keeping pace with rapidly evolving model capabilities.
1 Introduction
The rapid evolution of large language models (LLMs) continually outpaces traditional evaluation methodologies. Static benchmarks, foundational to earlier progress, now face critical issues: they quickly saturate, are susceptible to training data contamination, become temporally irrelevant as knowledge evolves, and often fail to capture model capabilities in specialized domains (Kiela et al., 2021; Dominguez-Olmedo et al., 2024; Zhang al., 2024; Zhu et al., 2023; Ruder, 2023). While direct human assessment provides valuable insights, its cost and scalability limitations render impractical for the continuous, diverse evaluation needs of the field. This creates pressing need for evaluation generation frameworks that are automatic, while dynamic, reliable, dopain-specific, and accessible.
We therefore introduce YourBench: an open-source framework that enables automated generation bespoke evaluation sets directly from any collection of documents. YourBench empowers users to systematically create fresh, relevant benchmarks tailored to specific topics, achieving high reliability at low cost and without manual annotation. Central to our framework is the principle of secument-to-Bvaluatin leveraged to produce diverse, citations, optimizing for coverage, diversity, and answerability (details in $2.2, Appendix C).
Generation (D2EG), where LLMs are wwweeextalll-grounnde question-answer pairs with verifiable
We demonstrate the power of this approach by replicating the challenging MMLU-Pro bench- mark (Wang al., 2024). Using only handful of Wikipedia articles input, YourBench automatically generated novel, multiple-choice questions across 11 domains. When evalu- ated 86 models, from 400M to 405B parameters, the resulting model rankings achieve
"qqull contribution.

---

Model Performance Correlation: Original vs Reproduced Tasks
DeepSeeik R1-0528
Grok Mini
Q4 mmii
Owen3-2355A222
1.0
1,
F-0.98
99
Blology
0.5
0.0
65
10
0.0
0.5
00
1.0
0.0
1.0
1.8
1.
1.0
1.0
Economics
100
Paramite's
0.0
0.5
10
0.0
0.5
10
04
0.3
14
0.0
0.5
1.0
Score
10
Paycho ogy
00
05
10
0.0
0.5
10
54
65
14
0.0
0.5
1.0
Original Score
Original Score
Original Score
Original Score
Figure 1: YourBench Reliably Reproduces Expert-Curated Benchmarks. We automatically reproduced MMLU-Pro across 11 domains using only Wikipedia articles. The scatter plots show the strong correlation between model scores the original MMLU-Pro (x-axis) and our generated benchmark (y-axis) for Biology, Economics, and Psychology, each tested with four different generator models. Each point represents one of 86 evaluated models, demonstrating high fidelity (Pearson 0.91) across the board.
remarkable 0.91 0.99 Pearson correlation with the original expert-curated benchmark (Figure 1). This process not only reliable but also highly efficient, generating a fresh, contamination-prooo evaluation for under $15 per model.
The YourBench framework built on a robust pipeline that includes multi-format document ingestion, semantic chunking, and automated quality controls like citation grounding ($2). To ensure our method evaluates true contextual reasoning rather than memorized knowl- edge, we also introduce TEMPORA-0325, dataset of over 7,000 documents published after March 2025, providing a sandbox for evaluating models on genuinely novel information.
Our main contributions
follows:
YourBench: An open-source framework* for the automated generation of reliable, domain-specific evaluation benchmarks from any document set.
TEMPORA-0325: A large-scale dataset? of recent, post-cutoff documents designed enable robust, contamination-frre evaluation.
State-of-the-Art Benchmark Replication: A large-scale validation showing
YourBench reproduces MMLU-Pro rankings across 86 models with near-perfect correlation, demonstrating the viability of automated benchmark generation.
By providing a scalable, automated, and focument-grounde approach, YourBench facili- tates a move towards more timely, specific, and trustworthy LLM evaluation. empowers the community to better understand and track the true capabilities of these rapidly advanc- ing models, ensuring that evaluation can finally keep pace with innovation.
1GitHub
PDaasset
2

---

2 YourBench: Multistep Framework for Dynamic Evaluation Generation
The YourBench framework is architected multi-stage pipeline designed to transform unstructured documents into high-fidelity evaluation benchmarks. Each stage addresses specific challenge in automated generation, moving from raw content ingestion to polished, reliable set of questions. This section details the three core stages: (1) Document Preprocessing, (2) Question and Answer Generation, and (3) Quality Filtering-
2.1 Document Preprocessing
Real-world documents are messy, arriving in diverse formats (PDF, HTML, Word) with varied structures and multimodål content. To create consistent foundation for analysis, YourBench first employs a robust preprocessing pipeline. This pipeline standardizes all inputs into clean, LLM-friendly markdown format while preserving critical semantic information. The process involves three key steps:
1. Document Ingestion: We normalize heterogeneous source files into a unified markdown representation. This step handles format conversion and extracts textual descriptions from visual elements, making all content accessible.
2. Semantic Chunking: To manage the context limitations of LLMs and focus their attention, we partition documents into semantically coherent segments. This avoids splitting key ideas and enables the generation of questions grounded in localized context.
3. Global Summarization: Chunking risks losing the document's 's broader narrative. We mitigate this by generating a global summary for each document, which pro- vides essential overarching context to the model during question generation.
The detailed methodologies for this stage, including specific models and tools used for ingestion and chunking strategies like multi-hop combination, are elaborated in Appendix
2.2 Question and Answer Generation to Elicit Grounded Knowledge
The core of our framework is the Document-to-Eane Eoaluuation Generation (D2EG) process, which aims to produce a question set that satisfies three principles: Coverage of the source material, Diversity in question style and difficulty, and Answerability directly from the provided text. To achieve this, YourBench employs a guided, ensemble-based approach.
We frame the generation task as sampling question-answer pair (q, a) and its supporting citations cit from a model conditioned on both the global document summary S and specific local chunk c:
(g, a, cit) p-- pf-lprompteen-SS. S,c)
(1)
Providing both the local chunk and the global summary crucial; the local context provides the specific details for grounding, while the global summary ensures the model correctly interprets those details within the document's full scope (Liu et al., 2023). To ensure answerability, the generation prompt explicitly instructs the model to provide citations that substantiate its answer.
To enhance robustness and diversity, we utilize ensemble of LLMs from different model families and scales (M = [M,,.., M))). By aggregating outputs from multiple models, we mitigate the inherent biases and stylistic tendencies of any single model, resulting in more comprehensive and varied raw question pool.
2.3 Quality Filtering and Deduplication: Ensuring Fidelity
The raw output from the LLM ensemble requires rigorous filtering to ensure fidelity and non-redundancy. YourBench implements a two-stage automated process: citation validation followed by semantic deduplication.
3

---

2.3.1
Citation Validation
To enforce that QA pairs are verifiably grounded in the source text, first validate the model-provided citations. We implement an algorithmic check that quantifies the cor- respondence between each provided citation string and the source chunk using fuzzy string matching score (based on Levenshtein distance (Levenshtein, 1966). This produces grounding score for each QA pair, and we filter out any pairs that fall below a configurable threshold fctt, programmatically removing ungrounded or hallucinated content.
We assign grounding score each QA pair (9, a, cit) by averaging the partial ratios across its citations:
Scoreqa (qTTOAAqq,,CC aq, = No
PartialRatto(c..c)
(2)
assuming N₂ 0 (score 0f N. 0). We filter Qraw, retaining pairs exceeding threshold cit
Qeit {(g a, cit) E Qraw Scoreqa (q, cit) > Bcit
(3)
Empirically, 0cn = 0.85 balances rigorous filtering of ungrounded pairs with preservation of valid items.. See Appendix D.1 for the model-level scoring metric used in evaluations.
2.3.2 Semantic Deduplication and Reweighting
Ensemble generation and chunk overlap can lead to semantic redundancy in Qeit- To manage this, we perform semantic deduplication. We obtain dense embeddings e(g) for questions in Qcit using a sentence embedding model (e-g., Sentence-BERT BERT (Reimers & Gurevych, 2019)).
We apply DBSCAN (Ester et al., 1996), density-based clustering algorithm, to the embeddings {e(g)). DBSCAN groups semantically similar QA pairs (cosine similarity > Tsim = 0.9) into clusters C = [CI,.., Cx} and identifies outliers N.
From each cluster C₂ we select one representative QA pair (9% A citt)) (e.g., the medoid). The deduplicated set is:
Qdedup { (9 a cit))) C2 CJUN
where N' are the unique noise points. To retain information about concept salience (indicated by cluster size [&l),, we assign weights wy each representative aq city) proportional to its original cluster size (e.g., 20 = [C&)), with w for noise points. These weights are used in the final evaluation scoring (Section 3), allowing frequently questioned concepts to contribute more significantly, approximating the evaluation of the full set Qai efficiently.
2.4 Suggested Evaluator
Given the curated, weighted QA set Qtimal Qdedup (Sections 2.2, 2.3), we generally evaluate free form LLMs outputs using a pairwise comparative assessment strategy (as done in model arenas). Our suggested evaluator is composed of a judge LLMs ensemble to enhance reliability and mitigate self-preference bias (Zheng al., 2023), and bias-corrected scoring aggregation to mitigate positional bias (the tendency of L.Ms-judges to prefer an answer presented one position compared the other). We expand on this in Appendix D.2. It's also possible to use YourBench to generate questions with multiple choice answers through prompt modifications, in which case becomes possible to evaluate models through simple exact match score, do in Section 3.3.
3 Empirical Validation and Results
We conduct multi-faceted validation of YourBench to answer two central questions. First, can our automated framework produce benchmarks that reliably replicate the results of established, human-curated evaluations at scale? Second, the questions generated by YourBench intrinsically high-quality, ie, valid, diverse, and verifiably grounded in the source material?

---

3.1 Benchmark Replication: Matching Expert Curation at Scale
A critical test for any new evaluation framework is whether its results align with trusted gold standard. To validate YourBench's external validity, we performed a large-scale replication of the challenging MMLU-Pro benchmark (Wang et al., 2024).
Methodology. For 11 MMLU-PPo domains (e.g., Biology, Economics, Physics), used YourBench to automatically generate novel 10-choice multiple-choice questions. The source material was intentionally generic: the first 10 relevant Wikipedia articles for each domain. To test the robustness of our framework, we used four different generator models, including both open-source (DeepSeek R1 0528, Qwen3-235B) and closed-source (o4-mini, Grok-3- mini) options. We then evaluated diverse set of 86 models, ranging from 400M to 405B parameters, on both the original MMLU-Pro and our generated benchmarks.
Results. YourBench demonstrates remarkably high-fidelity reproduction. As shown in Table I, the Pearson correlation between model rankings on the original and our generated benchmarks is exceptionally high, ranging from 0.91 to 0.99 on average across domains. Figure 1, holds true
This strong correlation, visually confirmed by the tight clustering regardless of the generator model used, proving that our framework reliably preserves the relative capabilities of different LLMs. By aggregating scores across hundreds of generated questions, the evaluation effectively minimizes noise and captures the true signal of model performance. This entire process is also highly efficient, requiring under minutes of computation and less than $15 in inference costs per domain. This result provides strong evidence that automated, dynamic benchmark generation can match the discriminative power of static, expert-curated evaluations.
Table 1: Pearson correlation between original MMLU-Pro and YourBench reproductions across 11 domains. Each correlation computed over 86 models' performance scores, demonstrating consistently high fidelity regardless of the generator model used.
YourBench Generator Model
Domain
DeepSeek R1 0528
Grok-3-mini
04-mini
Qwen3-235B
Biology
0.994
0.977
0.994
0.994
Business
0,943
0,924
0.934
0.937
Chemistry
0.935
0,900
0.909
0,911
Computer Science
0.950
0.914
0.943
0.933
Economics
0.985
0.969
0.980
0.981
Health
0.971
0.949
0.966
0.968
History
0,933
0.898
0.952
0,943
Law
0.934
0.884
0.921
0.914
Philosophy
0.964
0,937
0.957
0.956
Physics
0,933
0,899
0.919
0.919
Psychology
0.991
0.975
0.990
0.989
Average
0.958
0.930
0.951
0.950
3.2 Internal Quality Validation: Are the Generated Questions Good?
Beyond replicating rankings, trustworthy benchmark must consist of questions that are valid, diverse, and verifiably grounded. To analyze these intrinsic properties, we move from external replication to internal quality assessment.
Experimental Setup. To ensure this analysis measured true generative capabilities rather than memorization, created the TEMPORA-0325 dataset, corpus of over 7,000 diverse documents published exclusively after March 2025. This guarantees that all source material is novel to the generator models. We then used suite of 26 state-of-the-art L.MMs, spanning

---

7 model families and sizes from 3B to 671B parameters, to generate over 150,000 QA pairs from this dataset.
The Validity-Diverse Spectrum. We analyzed the generated questions along two axes: validity and diversity. Validity: a question's 's clarity, sensibility, and answerability from the source text, was assessed via meticulous human evaluation of over 2,000 sampled questions, achieving high inter-annotator agreement (Gwet's AC1 = 0.71). Diversity: the semantic breadth of topics covered, was measured using embedding-based metrics.
Our analysis, summarized in Figure 2, reveals an intriguing trade-off. We found that con- temporary models integrated within YourBench can generate questions with high intrinsic validity, averaging approximately 85% post-filtering However, models exhibit distinct "personalities": some, like o3 mini, excel at generating highly valid questions (96%) by focusing on precise, factual queries, resulting in lower diversity. Others, like Qwen2.5 32B, achieve high diversity by exploring wider range of concepts, at the cost of slightly lower average validity. Notably, some models like DeepSeek V3 achieve strong balance, scoring well on both dimensions. This spectrum allows practitioners to select generator models that align with specific evaluation goals, whether for rigorous factual recall or broad conceptual understanding.
Diversity Score
Validity Score
Owen2.5
03 mini
0.9g 0.95
DeepSeek 6718
0.90
Gemma 27B
0.93
Qwen2.5 728
0 89
Gemini Flash
0.91
Claude 3.5 Halku
1.87
Claude Sonnet
Mistral Large
Gem
Gemini
1328
0585
Gemini DeapSesk 6/16
6716
0.90
0.91
Cemini 2.0 Rlsh
0.34
RI Distill Llama 708
0.90
Liama 3.1 4058
0.84
Mistral Large: 2411 2411 1326
0.89
Mistral Small 248 248
Disbill Owen 328
32b
0.88
OwD 326 328
61
Owen2.5
0.88
Claude 3.7 Sonnct
DeepSeek R1 6716
0.87
Liama 3.1
Claude 3.5 Haiku
0.87
DeepSeek 3.3 708
Liama 3.1 4058
0.86
GPT
R1 Distili Llama 708
328
0.05
GPT min
Phi 148
0.85
R1 Owen 328
Mistral Small 3.1 248
0.83
Phi Mini 3.80
GPT
0.65
0.67
Distill Owen 148
Lia ma 3.3 708
0.82
Gemma 3 278
0,81
Distil Owenn2.5 2.577
GPT Qwen2.5 148
0.80
0.79
R1 Distill Owen 78
Phi 148
Phi Mini 3.80
OwenZ 5
0.77
o3 mini
0.25
0.50
0.75
1.00
0.7
0.8
0.9
1.0
Figure The Validity-Diversiiy Spectrum of Language Models. Semantic diversity scores (left) and human-annotated validity scores (right) for questions generated by various models. Models like o3 mini excel in validity but have low diversity, while models like Qwen2.5 32B achieve high diversity with slightly lower validity. Some models like DeepSeek V3 demonstrate a strong balance.
3.3 Citation Grounding: Ensuring Verifiable Fidelity
We enforce reliability through an automated citation validation step that scores the textual overlap between model-provided citations and the source text ($2.3). To assess model proficiency at this task, we computed an aggregate citation score for each generator.
Figure shows that leading models, including both proprietary (Claude 3.7 Sonnet) and open-weight (Qwen, Gemma families) options, demonstrate strong citation capabilities. The Pareto frontier analysis in panel (b), which plots inference cost against citation score, further reveals that high-quality grounding is attainable efficiently. Models like 5 32B achieve excellent citation scores at fraction of the cost top performers, demonstrating that reliable, verifiable benchmark generation is accessible even with modest resources.

---

Citation Score
with Puroto Fronterr
Claude 3.7 Sonnet
03 mini
89.6
91.1
Genina 2.0 Blgg
nl mini
Clando 3.7 Sonnet
Gemini 2.0 Flash
Queend, 122
GPT 40
5
Qween2:5 14B
Gemma 3 278
Owen2.5
07.2 5
87
Dusit Laans
DeepScok V3 6718
DeepSeek RI 6718
5 148
85.4
85. 5
87
Scom
Owne2.5
Pistiss O S D43
Lhansa 5
En 1320
QWQ 32b 328
64.6
R1 Distil Llama 708
Diatil
31.0
Claude 3.5 Halku
Owen2.55
Mistral Small 4058
76.6
14.4
Phi 148
73.6
Mistral Lame 2411 1328
72.9
R1 Distill Owen 148
70.4
Houd taxo
Phi 4 Mini 3:88
58.9
Arthooe
GPT 40-mini
Owen
51.6
Liaen
Opena)
46.6
Liama 00
41.0
Hiatr
40
100
Infoveace Cout USD!
(a) Model citation scores.
(b) Inference cost vs. Citation score.
Figure 3: Evaluation of citation grounding performance. (a) Compares aggregate citation scores across various models. (b) Illustrates the Pareto frontier for inference cost (log scale) versus citation score, highlighting efficiency trade-offs. Full model list in Appendix D.3.
Related Work
The evaluation of LLMs is at a critical juncture, as the pace model development has rendered many traditional methodologies inadequate. Our work is situated at the confluence of four major challenges in the literature: the limitations of static benchmarks, the rise of dynamic evaluation, the problem of temporal decay, and the need for domain-specific assessment.
The Crisis of Static Benchmarks. Foundational benchmarks like GLUE (Wang et al., 2019) and MMLU (Hendrycks et al., 2021a) were instrumental in driving early progress. However, they now face two existential threats: saturation, where state-of-the-art models quickly reach or exceed human performance, diminishing the benchmark's 's discriminative power (Ruder, 2023); and data contamination, where test sets are inadvertently memorized during pretraining on vast web-scale corpora Haminguez--lmede et al., 2024; Zhang et al., 2024). This forces a continuous and costly race to develop new static tests, which themselves are vulnerable to the same fate. YourBench directly addresses this cycle by enabling the instant generation of novel, contamination-fret questions, making evaluation a dynamic rather than static target.
The Move Towards Dynamic and Synthetic Evaluation. In response to these issues, the field has shifted towards dynamic evaluation. Early pioneering work like Dynabench (Kiela et al., 2021) introduced human-in-the-lonn adversarial process to find model weaknesses. While effective, this approach remains expensive and slow. More recent work has leveraged LLMs themselves to synthetically generate
evaluation data, either by rewriting existing questions or creating them from scratch (Wang et al., 2024). These methods, however, often struggle with quality control, risking the generation of trivial hallucinated questions, and do not inherently solve the problem of grounding evaluation in specific, user-defined knowledge contexts. YourBench builds on this paradigm but introduces a structured, decument-groundde pipeline with explicit quality controls like citation validation to ensure high fidelity.
Temporal Validity and Domain Specificity. Two critical dimensions often unaddressed by general- synthetic generation are time and domain. Knowledge evolves, and models pretrained on historical data quickly become outdated, performing poorly questions
7

---

about recent events concepts (Zhu et al., 2023). Simultaneously, high performance on general benchmarks often fails to translate to specialized, high-stakes domains like law medicine, where precision and current knowledge are paramount (Holzenkamp et al., 2023; Hung et al., 2023b). Creating benchmarks for these areas has traditionally required expensive, slow expert annotation. YourBench provides a unified solution to both problems. By ingesting any set of documents, whether recent news articles (like our TEMPORA-0325 dataset) proprietary domain-specific texts, it empowers any user to create timely, specialized evaluations in minutes, task that was previously infeasible.
Conclusion
In this work, we introduced YourBench, an open-source framework that fundamentally changes how large language models are evaluated. By enabling the automated generation of reliable, document-groundde benchmarks, YourBench directly addresses the critical bottlenecks of static evaluation: contamination, saturation, and prohibitive cost. Our large- scale replication of MMLU-Pro, which achieved 0.91 0.99 Pearson correlation with the expert-curated original using only a handful of Wikipedia articles, provides definitive evidence that automated generation can match the discriminative power of human-curated benchmarks at fraction of the cost.
The utility of YourBench extends beyond benchmark replication. By empowering researchers and practitioners to create custom evaluations from any document set, it unlocks new possibilities for domain-specific and temporally-aware assessment.
The framework's potential extends beyond benchmark replication and is already being explored in several research initiatives:
Domain-Specific Knowledge Assessment (Agriculture): YourBench is being uti- lized to systematically evaluate LLMs on specialized, proprietary knowledge. This involves generating questions assessing factual recall, applied reasoning, and retrieval-augmented generation capabilities based on diverse agricultural docu- ments, effectively measuring model's domain intelligence
Personalized Education: In educational research, the framework is being adapted assist teachers and generate tailored assessment questions based on individual student learning profiles derived from textual inputs, exploring pathways towards automated, personalized learning tools.
YourBench's capacity for multi-hop question gen- eration is being employed to construct challenging training corpora for retrieval- augmented generation systems. By synthesizing complex questions requiring in- formation integration acToss multiple document chunks and incorporating human feedback loops, this effort aims push the boundaries of RAG model capabilities.
Advanced RAG Training Data:
By releasing the YourBench library, the contamination-fee TEMPORA-0325 dataset, and all supporting code and data, we provide the community with practical, scalable, and robust tool. We believe that dynamic, user-driven evaluation is essential for ensuring that our assessment methodologies can finally keep pace with the rapid evolution of AI capabilities.
Reproducibility
We are committed to ensuring the reproducibility our research and facilitating further investigation by the community. To this end, we make several key resources publicly available. The complete source code for the YourBench framework is released under an open-source license and can be accessed at https:/ hitps:///githn h/tgthtt.bbcooo cow.ib.cc///// com/huggingface huggingfaceeyourrenet yourbench. This repository includes the implementation of the document processing pipeline (Section 2.1), the question generation framework (Section 2.2), and associated evaluation scripts.
Furthermore, the TEMPORA-0322 dataset introduced in Section 3, comprising documents published after March 1, 2025, is available the Hugging Face Hub at this datasets link.
8

---

Alongside the dataset, we provide the code used for document collection, preprocessing, semantic chunking (Section B.2), and subsequent analysis within the main framework repository.
To enable detailed verification of our experimental findings, we release the complete infer- ence traces for critical experiments, including the MMLU Pro replication study (Section 3.1) and the citation validity analysis (Figure 3). These traces cover the diverse set of 26 large language models detailed in Section 3, spanning both open-weight models (e-g., Llama, Qwen, DeepSeek families) and closed-source API-based models (e.g,, GPT, Claude, Gemini families). Our inclusion of both model types is a deliberate choice to enhance long-term reproducibility; by providing results for open models, we ensure that future researchers can replicate or extend our core findings even if commercial APIs become deprecated or change significantly over time. All code and experimental artifacts are designed to support transparency and allow the community to build upon our work effectively.
Ethical Considerations
The development of powerful AI systems necessitates equally robust and trustworthy methods for their evaluation. Frameworks like YourBench, which automate the generation of evaluation benchmarks, represent step towards more dynamic and potentially less contaminated assessment. However, like any technology, its introduction warrants careful consideration of the ethical dimensions and potential societal impacts.
One important area relates to the human element in data creation. Traditionally, benchmark creation involves sigmificant human labor, often in the form of detailed annotation or ques- tion writing. This labor, while essential, can sometimes be repetitive and subject to economic pressures, including concerns about fair compensation, particularly in globally distributed workforces. YourBench introduces a potential shift in this dynamic. By automating the generation of question-answer pairs, the burden on humans might transition from primarily generative tasks to ones involving oversight, validation, and curation. Instead of authoring questions from scratch, the focus could shift towards assessing the quality, relevance, and safety of machine-generated content, or guiding the generation process towards specific evaluation goals. It's uncertain as of now whether such shift would rather elevate the nature of the work, (demanding more critical judgment rather than repetitive production), or simply remove large-scale, low-wage annotators from the equation by replacing them with skilled annotators. It requires careful consideration and proactive effort to ensure that individuals involved are equipped with the necessary skills for these evolving roles and that the economic benefits of automation are shared equitably. The potential for deskilling or displacement in certain areas must also be acknowledged and addressed thoughtfully by the community and organizations deploying such systems. We must remain mindful of the human collaborators whose insights remain crucial, even as the tools evolve.
Furthermore, the integrity of the evaluation process itself relies heavily on the quality and characteristics of the LLMs used within the YourBench framework. The models employed for generating questions, summaries, and even judging responses inevitably embed their own biases, limitations, and potential failure modes, learned from their own training data. If not carefully managed, YourBench could inadvertently propagate these biases within the generated benchmarks. This underscores the critical importance of transparency regarding the models used in the generation process and the need for robust, ongoing validation of the generated datasets - not just for correctness, but also for fairness, representation, and potential hidden biases. Automated checks, like the citation grounding implemented, are valuable, but human oversight remains essential for identifying more subtle issues.
even amplify
The increased accessibility offered by YourBench, allowing for rapid generation of domain- specific benchmarks, is a significant advantage. It empowers researchers and practitioners to create evaluations tailored to their specific needs, moving beyond generic, potentially saturated benchmarks. However, this ease of creation also carries a potential for misuse. Benchmarks could conceivably be generated to specifically highlight the strengths or weak-

---

nesses of particular models, potentially leading to misleading comparisons if not used responsibly and transparently.
Finally, the computational resources required to run ensembles of large models for genera- tion and evaluation contribute to the environmental footprint of Al development. While YourBench might offer efficiencies compared to certain manual processes continuous large-scale human evaluations, the aggregate energy consumption remains a factor worthy of consideration as such automated systems become more widespread.
In conclusion, while YourBench offers promising direction for advancing LLM evalua- tion, its development and deployment must proceed with deep sense of responsibility. Continuous monitoring of its impacts, particularly on human labor dynamics and the in- tegrity of evaluation results, is essential. The goal should not merely be automation,, but the creation of evaluation methodologies that are not only more efficient and relevant but also fundamentally fair, trustworthy, and aligned with the broader goal of developing beneficial Al.
Acknowledgements
This research project has benefited from the Microsoft Accelerate Foundation Models Re- search (AFMR) grant program through which leading foundation models hosted by Mi- crosoft Azure, along with access to Azure credits, were provided to conduct the research. Additionally, this research utilized Anthropic credits granted through Anthropic's External Researcher Access Program. This research used the Delta advanced computing and data resource, supported by the National Science Foundation (award OAC 2005572) and the State of Illinois; Delta is a joint effort of the University of Illinois Urbana-Champaige and its National Center for Supercomputing Applications. We also gratefully acknowledge Hug- ging Face for supporting inference costs, as well as SambaNova and Novita for providing inference services.
References
Anthropic. The claude model family: Opus, sonnet, haiku. Technical report, Anthropic, March 2024. URL https: /tt:://wwwww /www/ anthropic com/ http://////////c//// wwwpc//wwwwwwww/neeeeeeeeeeeeeee Becauude----ee family.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3 4):3444)3 1952.
Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. Booookscore: A systematic exploration of book-length summarization in the era of Ilms, 2024. URL https://arxiv. org/abs////// 2310. 00785.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL 14168.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L..
10

---

Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Iao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang. Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. Li, Y. Q. Wang, Y. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025a. URL https::///axxitt.oo wwwpr/iww...gg/////0///0 12948.
DeepSeek-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongje Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang. Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Q. Wang, Y. Wei, Y.X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yonggiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025b. URL https: / //txxv//xvvvv////i////////a .owo/oor///oooo otgg/bs//4/// 2412. 19437.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 2966):141-142, 2012.
P. Deng, J. Wang, and T. Zhao. Newterm: Benchmarking real-time new terms for large language models with annual updates.
20814, 2024.
Ricardo Dominguez-Olmedo, Florian E. Dorner, and Moritz Hardt. Training on the test task confounds evaluation and emergence, 2024. URL http:::///axxi.org/abbsss/
078990.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal,
11

---

Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, et al. The llama 3 herd of models, 2024. URL https:/ https://arki..org/bb//40001 2407.2173..
Arpad E. Elo. The Rating of Chessplayers, Past and Presenf. Arco Publishing, New York, 1978.
Martin Ester, Hans-Peter Kriegel, Jorg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pp. 226-231, 1996.
Clémentine Fourrier, Nathan Habib, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall. Lighteval: A lightweight framework for Ilm evaluation, 2023. URL https: thub. com/ hugginggaace/lighteea..
Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, and Megan Ung. Changing answer order can decrease mmlu accuracy, 2024. URL 19470.
Kilem L. Gwet. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29-48, 2008. doi: 10.1446/0077110XX266600
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021a. URL
httpss//arrii..or//bb//22009 wtgg/b////////////// 03300.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021b. URL https://arxiv. org/abs//1/3. 03874.
Anna Holzenkamp, R. Ghosh, and D. et al. Zhang. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. https://arxiv.orgg abs/2308. 1 11462, 2023.
J. Hung, N. Parekh, and T. Yun. High risk domains in Ilm benchmarking. https: //aclanthology. wtwphhhllogo.org/2222222 org/2023. genbench-1 1.8. pdf, 2023a.
0rg/2223.genbenchh
K. Hung, Roy, and D. Marino. Limitations Ilms for high-risk domains despite domain- specific instruction tuning. https://aclanthollg org ntrg/2023.cmm genbench- 1. pdf, 2023b.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL 06825.
https://arxiv.org//a/
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in NLP. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, lz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4110-4124, Online, June 2021. Association for Computational Linguistics. doi: 10. //aclanthology.org//// orgpl/og...gg//2111 naacl-main manellmmai..24..... 324.
324. URL https:
Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. Fact, fetch, and reason: unified evaluation of retrieval-augmented generation, 2024. URL https: //arxiva.ivvv//// http:/axxvvoror//aaaaaaa/ org/abs//// 2409. 12941.
J. Richard Landis and Gary G. Koch. The measurement observer agreement for categorical data. Biometrics, 33(edriic,3(11:155--11 1977. doi: (0.2307/2529310.
12

---

Vladimir I. Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8)7707-710, 1966. Translated from Doklady Akademii Nauk SSSR, Vol. 163 No. 4 pp. 845-848, 1965.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023. URL https://arxis www.looorr/gsseee.eeeeeeee (ttbb//2377031111
Harsha Nori, Nick King, and Scott M. et al. McKinney, Capabilities of gpt-4 on medical exams and clinical vignettes. https: http://ww.wg///////.....33 13375, 2023.
OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perrllan, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, An- drej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Win- ter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, lan O'Connell, Ian O'Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, Ilya Kostrikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Jong Wook Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang, Lukas Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Madelaine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati,
13

---

Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moone Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi, Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Pattarddhan, Thomas Cunninghman, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang. Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. Gpt-40 system card, 2024 URL https://arxiv. orgg/bs///11 21276.
Qwen, ;, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: //axxiv.// wwwiii.oogg/bbbb 2tg/abs/2111 15115.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. URL 05250.
Nils Reimers and Tryna Gurevych. Sentence-bert: bert-networks, 2019. URL https://arrii.oor//a//////// 10084.
Sentence embeddings using siamese
Sebastian Ruder. The evolving landscape of 1lm evaluation.
2023.
Sumuk Shashidhar, Abhinav Chinta, Vaibhav Sahai, Zhenhailong Wang, and Heng Ji. Democratizing 1lms: An exploration of cost-performance trade-offs in self-refined open- source models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9070-9084. Association for Computational Linguistics, 2023. doi: (0.18653/v1/2022 findings-emmlp.6088 URL http::/dtp..c..// doi. orgoorgg///// 10. 18653/v1/ /2223//1//22.... 100223.finiing... findings-emnln 608.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Élena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George- Christian Muraru, Grigory Rozhdestvenskly, Henryk Michalewski, lan Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum
14

---

Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Ciang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy, Gemma: Open models based on gemini research and technology, 2024. URL http:://arrri..orr/b////4...8. 20050000022 08295.
Qwen Team. Qwen2.5-vl, January 2025. URL https:://wwenl.. sereeenll..gittii github. ho/bllogqwen22. 5-v1/.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding, 2019. URL https: httpp/////axw.oooabb///////
wttpoos//////ae//ooo/ee /104. 07461.
Feng Wang, Zesheng Shi, Bo Wang, Nan Wang, and Han Xiao, Readerlm-v2: Small language model for html to markdown and json, 2025. URL
0115551111155 01151.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024. URL https://arks arg/abs/2406.0153
Jason Wei. Successful language model evals.
2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thoought prompting elicits reasoning in large lan- guage models, 2023. URL httpss//aaraiv.or///a////// 11903.
Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer, 2024. URL
05258.
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. careful examination of large language model performance on grade school arithmetic, 2024. URL httptt: wttp://...r///////..000223.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, and lon Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena. Proc. of NeurIPS (Poster), 2023.
Wei Zhou, Alan Chen, and Zheng et al. Tan. Recent advances in large language model benchmarks against data contamination: From static to dynamic evaluation. https: Ttarxiv.orr//htm//5555 17521v1, 2025.
X. Zhu, W. Li, and G. Peng. Is your Ilm outdated? evaluating lims at temporal generalization. 08460v2, 2023.
15

---

A YourBench Pipeline Overview
Figure 4 provides high-level schematic of the end-to-end YourBench framework. The process begins with ingesting diverse source documents, which are then preprocessed through steps like semantic chunking and summarization ($2.1, Appendix B). An ensemble of LLMs generates raw question-answer pairs grounded in the document chunks, guided by principles aiming for coverage, diversity, and answerability ($2.2, Appendix C). These raw outputs undergo rigorous quality filtering, including citation validation and semantic deduplication, to produce a high-fidelity evaluation set (62.3). Finally, this curated set is used within an automated evaluation framework, typically employing an ensemble of LLM judges to rank the performance of target models ($3). This modular pipeline allows for flexibility and robust, automated benchmark creation from arbitrary document inputs.
Generation
IndiMaaa Summares
D Dvvrsiiootion Seeds
Nodw
Cormechaisahhrr
Wooe
Dowolossity
Decument
Entradding Model
GC Filer
Evaluation
Sin Calculator
Churk Dataent
Figure 4: Overview of the YourBench Framework: A dynamic pipeline starting from diverse documents, through preprocessing (ingestion, chunking, summarization -$2.1), LLM-driven question generation (following D2EG principles $2.2), quality filtering (citation validation, deduplication $2.3), to automated evaluation using an LLM judge ensemble ($3).
B Detailed Document Preprocessing
This appendix details the multi-stage preprocessing pipeline used in YourBench, designed to convert diverse, real-world documents into standardized format suitable for LLM-based question generation, as summarized in Section 2.1. The pipeline addresses challenges posed by heterogeneous formats and multimodal content.
B.1 Document Ingestion
We implement a unified ingestion pipeline using ReaderLM-v2 (Wang et al., 2025) (stripping known HTML content) and Markitdown' (converting various document types like PDF and Word into markdown). This approach retains key structural elements (headings, lists, tables, math) while simplifying complex layouts into standard text-based markdown syntax, ensuring consistency across sources.
When visual content (e.g., images) is present, we generate high-level descriptions using Qwen2.5-728 5-72BVL (Team, 2025) for captioning. These descriptions are incorporated into the markdown representation, allowing question generation modules to reference both textual and visual information uniformly. An example of multimodal document input is shown in Appendix B.4 (Figure 5).
B.2 Semantic Chunking
Processing full documents directly with LLMs presents challenges, including attention dispersion potentially overlooking content (Ye et al., 2024), and performance degradation with longer contexts (Liu et al., 2023).
16

---

We address these through semantic chunking, which partitions documents into coherent segments. This process involves decomposing the document into sentences, computing embeddings, and then splitting the text into chunks based on semantic similarity and token length constraints, preserving coherence within and across segments. Multi-hop chunking is also implemented by combining multiple non-contiguous chunks to facilitate questions requiring information synthesis across different document parts.
Given a document d, first decompose it into sentences {$1 and compute their embeddings E = [n) using sentence transformer model (Reimers & Gurevych, 2019), where E R The chunking process is governed by three parameters: mmin minimum chunk length in tokens, lmxxi maximum chunk length in tokens, and T: similarity threshold for chunk boundaries. For consecutive sentences s; and $/+1, we compute their semantic similarity using cosine similarity:
sim (S;,8)))) = Cj 211
|| e++1l
A chunk boundary is established at position when the current chunk's token length exceeds mmin AND either (i,, < T OR appending $141 would cause the accumulated chunk length to exceed Imax- This process yields set of text chunks C {(,,,., Cm where each chunk ej is a contiguous sequence of sentences from
enable the generation of questions requiring synthesis across multiple document segments, implement multihop chunking. Given parameters Amin and Amax (minimum and maximum number of hops), we generate composite chunks. For each multihop chunk, we sample k U (hminn/ mmx original chunks uniformly without replacement from C. and concatenate their text content. This produces a set of multihop chunks M {m],,-.,, mp) where each m, consists of k potentially non-contiguous original chunks. These multihop chunks are used alongside the original chunks C during question generation (Section 2.2). appendix context
Multihop Chunking:
B.3 Document Summarization
While chunking manages context length, it can lead to a loss of global document perspective during question generation. To mitigate this, we generate a document-wide summary using an LLM (DeepSeek-v3 (DeepSeek-Al et al., 2025b) with zero temperature). For extremely long documents exceeding context limits, techniques like those in (Chang et al., 2024) can be employed. Our summarization uses tf-tu-hf-thought prompting (Wei et al., 2023)
with structured XML tags+ for quality and consistency. This concise summary is provided alongside individual chunks (Section 2.2) to give the question generation LLM both local detail and global context. The full summarization prompt is available in Appendix H.
B.4 Sample Document
Figure shows an example document typical of those included in the dataset, featuring mix of text and visual elements handled by our preprocessing pipeline (Appendix B).
17

---

first Commitation astud
Make Europe competitive
Thy Cempetfivees Crep prrddn techng to boti Zarpur pegeeeee
The Osse Intustril feal Arteen in Energy supert and dhes wch
unggte suee wag denur
wbie
live packagas es cimpaty su raae wed acipad, beast corg astt neneat and mobities addtional eveatrennt
oeing tela sotigo ummaa cominzzaaive artund
streteg dhat ageum we luncher repertivst aetorestve sether and
the huse thon attatic moder Buroe industrial Action Mann Europese Raterwtive unter any adoiesd big out mas ser gatual
lale stilk,, Action Maa an laak tkili, and uriM dlucation Atratogle Plae and uur
sppert the deneicame
compunies
25 come tolerers have hild you dia leguue for and in der 100, them voult
selecaal esuud in festerie co ciry anowed
so killon memnirnnd rok sling wne ln fartatee
Wedileerr to prountes polooring dhse aroon
Sovinge and Inwrstweete lncen fesior wit corcher pec peoviee ereerber
Figure 5: Example of a contemporary multimodal document included in Tempora-O8s
C Theoretical framework underlying the data generation work
This appendix outlines the theoretical foundation for automated benchmark generation from source documents within the YourBench framework, termed Dacument-to-Foaluddie Generation (D2EG), introduced conceptually in Section 2.2. The goal is to produce representative question set Q derived from source document (partitioned into segments fc,,..., and optionally summarized by s) that satisfies key desiderata.
Let Q be the universe all possible questions derivable from the document segments. We seek subset Q Q that optimizes the trade-off between:
1. Minimality: Penalizing the total number of questions |QI for efficiency.
2. Coverage: Maximizing the extent to which Q addresses the source material.
3. Diversity: Ensuring variety in question type, difficulty, and targeted reasoning skills.
4. Answerability & Quality: A hard constraint ensuring every 9 E Q is valid and verifiably answerable from the source.
This can be formalized conceptually as a constrained optimization problem:
min ((Q) = a Q + Luncov (Q) + Y Cunit Q
(5)
subject to the constraint that every question in Q is verifiably answerable from the source document. Here, Luncow(2) (Q) penalizes the amount of source material left uncovered by Q while Luiif Q) penalizes lack of diversity (e.g., high semantic overlap) within Q. The non-negative coefficients a, B, balance these competing objectives.
Finding an exact solution to (5) is generally intractable due to the combinatorial nature of selecting Q from Q. Therefore, described in Section 2.2, YourBench employs practical, greedy generation framework using LLMs, guided by prompts and context, to approximate desirable question set Q that adheres the D2EG principles.
18

---

D Framework Theoretical Complements
D.1 Citation validity
To validate the grounding of generated answer a with citations cit = CN. to source text chunk c, use fuzzy string matching. For given citation string ci and the source chunk text we compute the partial ratio score using the Levenshtein distance concept:
2. 5) 100
c)
max Sy=c
[G| + |sy|
where sC) the length of the longest common subsequence between the citation c; and substring sj of the source text c. The maximum taken over all possible substrings sj cf This score ranges from 0 to 100.
The overall grounding score for single QA pair (g, cit) is calculated as described in Section 2.3 (Eq. (2)).
To calculate an overall citation performance score for a specific generation model (as reported in Section D.1), we average the QA grounding scores across all questions generated by that model:
Nimmdee
ModelCitationScore
Scoreqa ag city)
Ng.model
where Nymodel is the total number of valid questions generated by the model after initial filtering, and Scoreoa (9, aq. city) is the grounding score for question 4 as defined in Eq. (2).
D.2 Evaluation Framework
Given the curated, weighted QA set Qúiall = Qdedup (Sections 2.2, 2.3), we evaluate LLMs
M = { MM, MN] using a pairwise comparative assessment strategy with an ensemble of judge LLMs J = (J1--,, ]k] to enhance reliability (Zheng et al., 2023).
For each question (q:aff E Qrinal (weight wy) and model pair (MA, MB), we elicit responses R¹, Ryg. Each judge I receives the context tuple:
ČILAA = (9j, (q;,ky, R2g S,Cj)
(6)
including the question qj, responses R2, R global summary S, and source chunk(s) cj for grounded evaluation.
The judge Ji produces continuous score B) E [-1, reflecting the relative quality of Ri, vs R often guided by prompted chain-of-thought process (see Appendix for prompt details):
Piy (A, B) [1(5)AA,8)
Scores are averaged across judges for consensus (j = EK Diy(A, and weighted by question salience wy:
Vy(A,B) B) w (A,
(8)
To counteract positional bias, we evaluate both (A, B) and (B, A) pairings and compute bias-corrected score:
(A = (V)(A, Vy
(9)
19

---

The overall comparative score SAA, B) between MA and M3 is the sum over all questions:
Oiiaal
S(A, B)
L ()
(10)
The sign indicates preference; magnitude indicates difference strength. These pairwise scores {S(A, A, B)} form the basis for global ranking using methods like Bradley-Terry (Bradley Terry, 1952) Elo (Elo, 1978).
D.3 Evaluated Models
The following 26 models from 7 families were used in the generation and evaluation experiments described in Section 3:
DeepSeek (DeepSeek-Al) et al., 2025b;a): DeepSeek V3 (671B), DeepSeek R1 (671B), DeepSeek R1-Distill-Llama (70B), and DeepSeek R1-Distill-Qwen (32B, 14B, 7B).
Qwen (Qwen et al., 2025): Qwen2.5 models at various scales (72B, 32B, 14B, 7B) and the reasoning model Qwen QwQ (32B).
Mistral (Jiang et al., 2023): Mistral Large 2411 (132B) and Mistral 3.1 Small (24B).
Llama (Dubey et al., 2024): Llama 3.1 (405B, 8B) and Llama 3.3 (70B).
Google (Team et al., 2024): Gemini 2.0 Flash, Gemini 2.0 Flash Lite (?B) and Gemma 3 (27B). OpenAl (OpenAI et 2024): GPT-4o, GPT-4o mini, and o3 mini (?B).
Anthropic (Anthropic, 2024): Claude 3.7 Sonnet, Claude 3.5 Haiku (?B).
E Evaluation Quality Details
This appendix provides detailed methodologies and supplementary results for the validation of generated evaluation quality presented in Section 3.2.
E.1 Question Validity Methodology and Detailed Results
Human Evaluation Setup. As introduced in Section 3.2, we conducted a manual eval- uation to assess the intrinsic quality of generated questions. We sampled 2,000 unique questions generated from the TEMPÓRA-0325B dataset (Section 3) using the models listed in Appendix D.3. The sampling was stratified to ensure representation across models, document domains, targeted difficulty levels (basic, advanced), and question types (e-g., factual, multi-hop, numeric) specified during generation (Section 2.2).
Twenty trained annotators participated. Each annotator was presented with the source document chunk(s), the global document summary, the generated question, and the model- generated answer with its citations. Annotators were asked to assign binary validity label Invalid) based on the following criteria:
Clarity: Is the question grammatically correct and unambiguous?
Contextual Answerability: Can the question be definitively answered using *only*
the provided document chunk(s) and summary? Does it require external knowledge or unwarranted assumptions?
Sensibility: Is the question reasonable and logically coherent in the context of the document? (e.g., not nonsensical or self-contradictory).
question was marked "Valid" only if it met all three criteria positively. Any ambiguity, reliance external knowledge, or nonsensical phrasing resulted in an "Invalid" rating.
20

---

Inter-Annotator Agreement. Each question was evaluated independently by 3 randomly assigned annotators. To measure the consistency of their judgments, we calculated Gwet's AC1 coefficient (Gwet, 2008), a robust statistic for assessing inter-rater reliability, especially suitable for binary ratings with potential prevalence issues. The formula for Gwet's AC1 for two raters is:
AC1 Pa - Pe(r) 1 Pc(r)
where Pa is the observed percent agreement, and Pp(r) is the chance agreement probability, calculated as Pe(7) = 21 TT), with π being the overall proportion of "Valid" ratings (averaged across raters). For multiple raters (3 in our case), we used multi-rater extension of the formula. The resulting overall AC1 score was 0.71, typically interpreted as substantial agreement (Landis & Koch, 1977), confirming the reliability of our human validity labels.
Detailed Results and Examples. The average validity rate reported in the main text (x+5%) represents the mean percentage of questions rated "Valid" (by majority vote across the 3 annotators) across all models and question types post-filtering. The per-model valid- ity scores are visualized in Figure 2 (right panel). Further breakdowns (e.g., validity per question type) can be derived from the released annotations accompanying our dataset. Ex- amples of questions marked "Valid" and "Invalid" during this process, illustrating common failure modes like ambiguity requiring external knowledge, are provided in Appendix I.
Juxtaposing these results highlights a prevalent, though not absolute, trade-off. The model achieving the highest validity, 03 mini, scores lowest in diversity (0.26). This suggests a generative posture focused on precision and safety, perhaps by asking more routine or algorithmically verifiable questions based directly on easily identifiable facts, leading to high validity but low exploration of the document's 's semantic space. Conversely, the top diversity model, Qwen2.5 32B, while still generating reasonably valid questions (0.81 validity, rank #11), sacrifices some peak validity in favor of broader conceptual coverage. This might indicate more exploratory or creative generation strategy.
This validity-diversity spectrum is not a strict dichotomy. Notably, models like DeepSeek V3 671B manage to achieve impressive scores on both metrics (0,90 diversity, rank #2; 0.90 validity, rank #6), suggesting that balancing breadth and correctness is achievable. Similarly, models like Claude 3.7 Sonnet (0.80 diversity, 0.91 validity) also perform well across both dimensions.
This observed tension between generating highly valid, focused questions versus diverse, exploratory questions is an intriguing phenomenon. It reflects the different latent capabilities and perhaps inherent strategies employed by various LLMs when tasked with abstracting knowledge into evaluative queries. Rather than limitation, this presents valuable characteristic of the YourBench framework: it allows practitioners to select generator models or ensembles that align with their specific evaluation goals-be be it rigorous testing of factual recall with high-validity generators, or broad assessment of understanding across topics using high-diversity generators, or seeking balanced perspective with models adept at both. Understanding this trade-off provides deeper insight into the nature of M.r--rrvee generation and empowers more informed benchmark creation.
Length Metrics vs. Validity. We also analyzed the relationship between ques- tion/ answer/ citation length and the observed validity rate from human evaluation. Figure 6 plots the validity rate (averaged across all models) against different length metrics binned appropriately. While there isn't a perfectly monotonic trend, we observe general ten- dency for validity to decrease slightly for very long questions, answers, or unified text lengths, potentially reflecting the increased difficulty in maintaining coherence and contex- tual grounding over longer generations. Citation length shows less variation. The black line represents the average validity rate across bins, while faint lines show individual model trends, highlighting variability. These plots reinforce the finding that generating complex (often longer) valid questions remains challenging.
21

---

Langth Matrics Quastion Validtt
W
236
000099
Figure 6: Relationship between generation length metrics and average question validity rate (across all models). Validity tends to decrease slightly for very long generations. Faint lines represent individual model trends.
E.2 Citation Grounding Methodology and Detailed Results
Citation Scoring Metric. As described in Section 2.3, we quantify the grounding of an an- swer a with citations cit = (c1,..,,C,, to a source chunk using fuzzy string matching. The core metric is Partiallatio" based on Levenshtein distance (Levenshtein, 1966), computed for each citation Cj against the source c:
2 (atch((;,s)) X 100
PartialRatio(c,c)=
max
|ca
+ [sy|
where Match(q;,s)) Sy) finds the length of the best matching contiguous block between çj and substrings sj of (typically using sequence matching algorithms). The maximum is taken over substrings sj of c that are at least as long as the citation c;. This score ranges from 0 (no match) to 100 (perfect match of c; within c).
The QA grounding score Scorega (9, cit) the average of these partial ratios across all Ne citations, as given in Eq. (2). If N = 0, the score is O.
Model-Level Citation Score. The overall citation score for a generation model M, as reported in Figure 3, is the average of the QA grounding scores across all valid QA pairs generated by that model:
ModelGitationSooeem [Qalid,M
Σ
Score aq cit)
(g.acitt)
where Qvalid,M is the set of QA pairs generated by model M that passed initial quality filters (e.g., parseable format, non-empty question answer).. This provides single metric to compare the average citation reliability of different models. Detailed scores for all evaluated models are implicitly represented in Figure 3.
Inference Cost Calculation. The inference costs used in Figure 3b were estimated based on the per-token pricing for output tokens (as generation is output-heavy) published on /
ai/ ///////c/oo////// as of the time of experiments, using the lowest available price tier for each model. For models not on OpenRouter without public pricing (indicated by "?B" parameters), relative cost estimates were made based on known parameter counts or comparable models where possible, or they were excluded from the cost analysis. This provides a practical estimate the economic efficiency using different models for generation within the YourBench framework.
OpenRouter
E.3 Semantic Diversity Methodology and Detailed Results
Diversity Metrics. As discussed in Section 3.2, we quantified the semantic diversity of the set of questions QM generated by model M using two embedding-based metrics:
22

---

1. Embedding Dispersion: We first compute sentence embeddings e(q) for each question q E QM using a standard sentence transformer model (e.g, 'all-mpnet-basev22 (Reimers Gurevych, 2019)). The dispersion is the average pairwise cosine distance:
(1
e(qi) e(gi)
Dispersion(Qa.
E
Σ
QM (lQml
1)
[[e(g)) [
#cQM qcEMiii
higher dispersion value indicates that the question embeddings are, on average, further apart in the embedding space, suggesting greater semantic variety.
Semantic Entropy: We apply K-Means clustering (with K chosen based on heuristics like the elbow method or fixed moderate number, e-g., K 50) to the question embeddings {e(q) 9 QM)- Let N₂ be the number of questions assigned to cluster k, and N = 1QMl =
Ng. The proportion of questions in cluster k is pk / /.N. The semantic entropy is the Shannon entropy of the cluster distribution:
K
(ntropy((M)) =
Σ k=1
Px (pg)
Higher entropy indicates that the questions are distributed more evenly across different semantic clusters, implying broader coverage of different conceptual areas. Lower entropy suggests concentration in few dominant semantic themes.
The final "Diversity Score" reported in Figure 2 (left panel) is a normalized combination or average of these two metrics (e-g., scaled to [0, 1] based observed ranges across models). This composite score aims to capture both the spread and the evenness of the semantic distribution.
Detailed Scores. Figure provides the final composite diversity scores for the evaluated models. The underlying dispersion and entropy values, along with the specific normaliza- tion method, are available with the project's source code and results data. The variation observed confirms that model choice significantly impacts the semantic breadth of the generated evaluation set.
E.4 Cost and Parameter Efficiency Analysis
Beyond citation grounding (Figure 3b), we analyzed the relationship between model cost/size and overall question quality, approximated by the average validity score (Sec- tion 3.3), Figures 7a and 7b show Pareto frontiers for average validity score versus inference cost and model parameters, respectively.
These plots further illustrate favorable scaling trends and efficiency possibilities.
Cost Efficiency (Fig. 7a): Models like Llama 3.1 8B, Gemini 2.0 Flash Lite, and Gemma 27B appear on or near the Pareto frontier, achieving relatively high validity scores (80-90%+) at substantially lower costs compared to the largest or most expensive models. This demonstrates that high question validity is attainable without exorbitant inference budgets.
Parameter Efficiency (Fig. 7b): Smaller models, including Phi Mini 3.8B, Qwen2.5 7B, Llama 3.1 8B, and Phi 14B, form part of the Pareto frontier. This indicates that smaller parameter counts do not necessarily preclude high validity generation. Phi 4 14B, for instance, reaches approximately 85% validity, competitive with much larger models, showcasing significant parameter efficiency. Gemma 3 27B also stands out, achieving over 90
Together, these analyses suggest that while larger models sometimes offer peak performance, carefully selected smaller or more cost-effective models can generate high-quality evalua- tion sets efficiently within the YourBench framework, democratizing access customized benchmarking.
23

---

Iakemppe Gusl vo Valdiy Score
Muse Pocareders Vildity Score
04.
Maniaal
Tefermne Conr US3)
Mode Parcentrr (ilherr
(a) Inference Cost Average Validity Score.
(b) Model Parameters Average Validity Score.
Figure 7: Pareto frontiers illustrating trade-offs between average question validity and (a) inference cost (log scale) and (b) model parameters (log scale). emaaler/cheapee models like Llama 3.1 8B, Gemini 2.0 Flash Lite, and Phi 4 14B can achieve high validity scores efficiently. Full model list in Appendix D.3.
MMLU Replication: Detailed Analysis and Results
Prior to our comprehensive MMLU-Pro validation (Section 3.1), we conducted preliminary experiments reproducing subsets of the original MMLU benchmark with models. While these initial results showed perfect Spearman correlation (p = 1.0) at the aggregate level, they were much more limited in number of evaluated models and domains, well as overall number of questions.
This appendix provides detailed breakdown of the MMLU replication experiment dis- cussed in $3.1 and introduced in Figure 1. We aimed to validate whether YourBench could automatically generate MMLU-style benchmarks from source documents that reliably reflect the relative performance of different LLMs compared to the original MML.U benchmark.
E1 Correlation Analysis
We evaluated a suite of 8 LLMs (see Table 2) on 7 original MMLU subject subsets and their corresponding YourBench-generad counterparts ("new"). We then analyzed the correlation between the performance scores (accuracy) obtained on the original versus the "new" benchmarks.
When analyzing all individual data points (8 models x 7 subjects 56 pairs), the correlation is positive but moderate, suggesting some variance at the specific subject level or potential noise in individual measurements.
Overall Correlation (All Subject-Model Pairs):
Pearson r: 0.3833 (p = 0.0035)
Spearman p: 0.2982 (p 0.0256)
Model Mean Performance Correlation: When analyzing the average performance of each model across all 7 subjects (8 data points), the correlation becomes extremely strong, particularly in terms of rank order. This indicates that while absolute scores differ (YourBench questions are harder), the relative ranking of models is preserved.
Pearson 0.9646 (p 0.0001)
Spearman p: 1.0000 (p 0.0001)
The perfect Spearman correlation for mean model performance strongly supports the validity of Your for generating discriminative evaluations that align with established benchmarks in terms of relative model capability assessment.
24

---

F2 Per-Subject Performance Plots
The following figures visualize the performance comparison for each individual MMLU subject included in the study. Each plot compares the performance of the evaluated LLMs the original MMLU subset (grey bars) versus the YourBench--gnnratee subset (orange bars). These plots visually complement the aggregated data in Figure 1 and the comprehensive data in Table 2.
MMLU Astronomy Performance
Onginaal
0.9
New
0.8
Accussy
0.7
0.8
0.5
Lianad 78 (2023)
Owant (002))
Liama3 (0244)
Liamez 708 (2023)
Qwen2.5 (2024)
Owent 728 (202))
Liamed 708 (2024)
Owen2.5 728 (2024)
Figure 8: MMLU Replication Performance: Astronomy
MMLU Social Science Performance
0.9
Original
New
0.8
Acerscy
0.7
0.6
0.5
0.4
Liamaž (2023)
Ownnt (2023)
Liamaz 70B (202)
Liama3 (2024)
Qwen2.5 (2024)
Owent 728 (202))
Qwen2.5 728 (2024)
Liamet 708 (2024)
Figure 9: MMLU Replication Performance: Social Science
25

---

MMLU Virology Performance
0.85
Original
New
0.55
0.50
0.40
Liamaz 78 (2023)
Gwent 7B (2823)
Liama2 TOB (2823)
Owen2.5 78 (2024)
Liama3 8B (2024)
Qwent 728 (2023)
Owen2. 728 (2024)
Liema3 YOB (2024)
Figure 10: MMLU Replication Performance: Virology
MMLU World Religions Performance
Original
New
Acerssy
0.7
0.
0.5
0.4
Liamaž (2023)
Owant (2023)
Liama3 (2024)
Qwen2.5 (2024)
Liamaz 7OB (2023)
Owent 728 (2023)
Liame3 708 (2024)
Owen2. 728 (2024)
Figure 11: MMLU Replication Performance: World Religions
MMLU International Law Performance
0.9
Origihal New
0.8
Hozurady
0.7
0.5
0.5
Liama2 7B (2022)
Gwant 7B (2023)
Liama' 8B (o24)
Qwen2.5 78 (2024)
Liemaz 7OB (202))
Owent 728 (1023)
Liama3 70B (2034)
Qwenz.5 728 (202A)
Figure 12: MMLU Replication Performance: International Law
26

---

MMLU Nutrition Performance
0.9
Original
New
0.8
Accussy
0.6
0.-
Lianad 78 (2023)
Owant (a023)
Liamez 70B (ozz)
Liama3 (2024)
Qwen2.5 5 (2024)
Owent 728 (20z))
Liamed 708 (2024)
Owen2.5 728 (2024)
Figure 13: MMLU Replication Performance: Nutrition
MMLU Analomy Performance
Original
0.8
New
0.7
Accurscy
0.5
Liamad 78 (2023)
Owent 7B (2223)
LiameZ 70B (292)
Liame3 (2024)
Owen2.5 (2024)
Qwent 72B (202))
Liamed 70B (2024)
Owen2.5 72B (2024)
Figure 14: MMLU Replication Performance: Anatomy
27

---

E3 Comprehensive Performance Table
Table provides the complete numerical results, detailing the accuracy and standard error for each model on both the original ("orig") and YourBench- ("new") MMLU subsets across the seven evaluated domains.
Table 2: Comprehensive MMLU Replication Results: Accuracy (Std Err) across Models and Subjects. "New" refers to YourBench-generate benchmarks, "Orig" refers to original MMLU subsets.
Astranmmy
Social Science
Vamlogy
World Religions
Model
New
Orig
New
Orig
New
Orig
New
Crig
Owant 7B (212))
60.56% (5.85)
57.89% (4.02%)
46.37% 11.67%1
80.10% (2.825)
54.22% (1.53%)
55.8%% (3,56%)
49.05% 11.36%4
70.185 (3.51
Owend.378 (2024)
20.45% (5.4%))
(3.02%)
53.1%: (1 677)
87. (2.39%))
75% [1 (129%)
(3.89%)
55.00% (1.16%)
90%
Hema's (h024)
83% (5.38%)
71.21% (367%)
49.8%% (1 67%)
84 585 (2.58%)
59 19% (1911%)
(3.5%)
(1.16%)
81.29% 12
1lema270
45.07% (5.8%)
(1 1.599
18.21% (344%)
37 (1.88%)
41.57
18.87 (1 (1.12%)
57.31%.
66 66.2%%
83.08%
15%
Owent 720
87.13%
Owen2 (2004)
(1,85)
708
(5.38%)
(2.28%)
90.06% (2.29%)
International Law
Notrition
Anatomy
Average
Model
New
Crig
New
Orig
New
Crig
New Avg
Orig Aug
Qwent 7B (2023)
68.87% (1.70%)
67.77%
(8.27%)
71.05% 11.5 (1.54%)
63.00%
(2.76%)
67.57% (2.14%)
50.37%
(4.32%)
59,67%
64.80%
Owen2.5 2,5 70 (2024)
82 82.8%11 (1.38%)
82.61% (3.6))
85.80% 11 (155
79.41% (2.32%)
80.04% (1.82%
71.85% (3.89%)
70.29%
78.85%
Hama3 8B (2024)
75.74% [1.5 (157%)
78.51% (3.79%)
79.25% [1 39%)
79.08% (2.38%)
76.51% (1.545)
68.15% (4.02%)
67.59%
73.45%
Hamm2 78 (20222)
48.7%% (1.84%)
57.5% (4.51%)
52.10% (1.71%)
46.73% (2.66%)
45.55% (2.225)
44.45% (4.29%%
41.41%
50.0%
Ulam2 708 (2023)
79.6%% (1,485)
85.47% 3.39%)
78.44% 11 40h
71.24% (2.595)
75 65% (1560)
56. 30% (4.28%)
67.4%%
72.1%%
Owent 728 (2023)
85 18% (1 (1315)
86.88% (3095)
03.03%-111 (1 35%)
84 64% (.00%))
78 90% (2200
72 (3.85%)
29.44%
Q.ee3.5 (1004)
9000% (1105)
90.91% (2,655)
(1 (972)
90 856
82: 54% (1 (17300)
80 (3.11))
73.31%
8489%
1ema8 708 (2024)
86.25% (1.28%)
87.60% (3.01%)
83.68% (1 (39))
86 93%
(1 (1.039)
7% 79% (1.87%)
80.00% (3.46%)
70.6%.
80.01%
G Detailed Related Work and Literature Review
This appendix provides a comprehensive discussion of the related work surveyed in Sec- tion 4, detailing the challenges in large language model (LLM) evaluation and prior ap- proaches that motivate the development of YourBench. As models have grown in size and sophistication, traditional evaluation approaches have struggled to keep pace. We survey four key directions in LLM benchmarking- the challenges of static, human-curated benchmarks, (2) synthetic and dynamic benchmark generation, (3) temporal validity con- cerns, and (4) domain-specific evaluations and highlight how YourBench addresses the major open problems that emerge in each.
G.1 Limitations of Static Benchmarks
Historically, static benchmarks such as MNIST (Deng, 2012), GLUE (Wang et al., 2019), and SQuAD (Rajpurkar& et al., 2016) have been central to measuring progress in machine learning. Although these datasets propelled rapid innovation, modern LLMs can quickly saturate their performance ceilings, sometimes surpassing human-level scores within mere months (Ruder, 2023; Wei, 2023). This benchmark saturation hampers their long-term utility in discriminating genuinely more capable models. For instance, models that reached near- perfect scores on GLUE soon forced the community to adopt other, more challenging tasks (Wei, 2023).
An additional concern is benchmark contamination, where test data is inadvertently included in model's training corpus. Because large-scale pretraining involves ingesting vast amounts of web content, popular benchmarks are often seen-or memorized- by the model (Kiela et al., 2021; Ruder, 2023; Zhang et al., 2024). Empirical analyses show that certain LLMS can repeat verbatim segments from question banks such as GSMSK (Cobbe et al., 2021) or MATH (Hendrycks et al., 2021b) when tested in a zero-shot setting (Wei, 2023), artificially inflating performance. Holding out an unseen test set is one partial solu- tion, but as time passes and these datasets spread online, the likelihood of contamination grows (Gupta et al., 2024). Consequently, reliance on single, static, and publicly available
SStandard error was derived directly from the accuracy mean, following the methodology in (Four- rier et al., 2023).
28

---

benchmark may induce narrow optimization rather than robust generalization (Hendrycks et al., 2021a).
G.2 Toward Dynamic and Synthetic Evaluation
Faced with saturation and contamination, researchers have pursued dynamic and synthetic benchmark generation. Kiela et al. (2021) introduced Dynabench to update evaluation sets interactively, challenging models with adversarially crafted queries. This iterative approach demonstrated that once model adapts to a static test, new data can still reveal surprsinng failures. However, such human-in-the-loop curation remains expensive and slow to scale.
A more automated strategy is to LLMs themselves for benchmark synthesis. Several techniques involve prompting a strong generator model to create new questions or tasks, sometimes based on existing ones (benchmark rewriting) (Wei, 2023; Krishna et al., 2024). Methods like Auto-Dataset (Ruder, 2023) or ITD (Wei, 2023) rephrase, expand, or mutate original items while controlling for difficulty, ensuring the new tasks remain answerable. Others adopt multi-agent pipelines, in which distinct LLMs generate candidate questions and validate them, filtering out ambiguous or erroneous samples (Zhou et al., 2025). Further exploring the role of LLMs in the evaluation pipeline, early work by Shashidhar et al. (2023) utilized LLMs as judges to assess model outputs, correcting for positional bias inherent in such automated evaluations. Despite promising progress, fully synthetic benchmarks introduce new challenges, including the risk of hallucinated or trivial questions. Quality control and verification remain active research topics, especially when the aim is to test advanced reasoning or domain-specific knowledge.
G.3 Temporal Validity and Knowledge Evolution
Another major challenge is temporal validity, reflecting the fact that knowledge and world events change continuously. Many popular benchmarks capture only static snapshots, making them less relevant when facts become outdated (Zhu et al., 2023; Deng et al., 2024). o
LLM performance thus appears high on older queries but may degrade sharply on newly introduced or time-sensitive questions (Zhu et al., 2023). Holding out a private test set of recent data help, but frequent refreshes necessary to track model's ability to integrate new information (Ruder, 2023; Zhang et al., 2024).
Several works illustrate the severity of the problem. Zhu et al. (2023) generated post-training news-based questions to measure whether an LLM truly updates its internal knowledge representation. They found LLMs frequently defaulted to outdated responses, highlighting gap between real-time information usage and parametric memory. Similarly, Deng et al. (2024) created an evolving dataset of newly coined terminology, demonstrating 20%+ accuracy drops for concepts introduced long after model's pretraining cutoff. These findings underscore the necessity for continually updated benchmarks that can test model's recency-awarenee and its ability to override memorized facts.
G.4 Domain-Specific Evaluation
Moving from general-purpose benchmarks to specialized ones is increasingly essential, espe- cially in high-stakes fields like medicine, law, and finance (Hung et al., 2023a). Benchmarks such as USMLE-based medical QA (Nori et al., 2023), or specialized legal datasets like Case- HOLD and LegalBench (Holzenkamp al., 2023), have revealed critical blind spots in LLM reasoning (Hung et al., 2023b). For instance, LLMs might achieve near-human scores on open-domain quizzes yet commit severe factual errors or hallucinations in domain-specifir contexts (Gupta et al., 2024).
Building domain-specific benchmarks demands costly expert annotations and must reflect the latest regulations, guidelines, or terminology. In medicine, for example, clinical protocols can change frequently, making static test rapidly obsolete. Researchers have thus proposed rolling domain benchmarks wenchmark--cortiuuuusy continuously collected or synthesized data for niche areas such as real-time medical literature or changing legal precedents (Zhang et al., 2024). So far, these dynamic domain evaluations remain nascent: they are typically narrow, small in
29

---

size, and do not integrate robust automated generation pipelines multi-modal content ingestion.
Synthesizing these research themes reveals persistent open problems in LLM benchmarking. First, existing static benchmarks are prone to contamination and rapid saturation. Sec- ond, purely human-driven dynamic approaches cannot scale indefinitely. Third, synthetic generation requires careful quality control and can still produce stale or trivial tasks if not refreshed in tandem with new knowledge sources. Fourth, few existing solutions integrate domain expertise in a flexible manner or support continuous updates for specialized fields. Finally, temporal drift in factual knowledge remains inadequately addressed, most benchmarks do not systematically ensure that test data are entirely post-training or reflective of newly emerging concepts.
H Prompts
H.1 Document Summarization Prompt
The following prompt is first provided into the language model. Once the model provides response answer, we extract the content that contained within the final_summary XML tags to function as our document summary.
You are AI assistant tasked with analyzing and summarizing documents from various domains. Your goal is to generate a concise yet comprehensive summary of the given document. Follow these steps carefully:
You will be provided with document extracted from website. This document may contain unnecessary artifacts such links, HTML tags, other web-related elements.
2. Here is the document to be summarized: <document> (document)
</documet>>
3. Before generating the summary, mental scratchpad to take notes you read through the document. Enclose your notes within <scratchpad> tags. For example:
<scratchpad>
Main topic: [Note the main subject the document]]
Key points: [List important information]
Structure: [Note how the document is organized]
Potential artifacts to ignore: [List any web-related elements that should be disregarded]
</scratchpad>
As you analyze the document:
Focus solely on the content, ignoring any unnecessary web-related elements.
Identify the main topic and key points.
Note any important details, facts, arguments presented.
Pay attention to the overall structure and flow of the document.
5. After your analysis, generate a final summary that: Captures the essence the document in concise manner.
Includes the main topic and key points.
Presents information in logical and coherent arder.
30

---

Is comprehensive yet concise, typically ranging from 3-5 sentences (unless the document is particularly long complex).
6. Enclose your final summary within <final_summary> tags. For example:
<final_summary>
[Your concise and comprehensive summary of the document goes here.] </final_summary>
Remember, your task is to provide clear, accurate, and concise summary of the document s content, disregarding any web-related artifacts or unnecessary elements.
H.2
Single Shot Question Generation Prompt
##
Your Role
You expert educational content creator specializing in crafting thoughtful, rich, and engaging questions based provided textual information. Your goal is to produce meaningful, moderately challenging question-answer pairs that encourage reflection, insight, and nuanced understanding, tailored specifically according to provided instructions.
## Input Structure
Your input consists of:
inndiiiinnnsttuuctionns
[Specific instructions, preferences, or constraints guiding the question creation.) //adittiona_instructiirrs
<title> [Document title]
</title>
<document_sumaay>
[Concise summary providing contextual background overview.]
</dccument_summary>
<text_chunk>
[The single text segment to analyze.
</text_chunk>
## Primary Objective
Your goal is generate thoughtful set of question-answer pairs from single provided "<text_chunke". Aim for moderate complexity that encourages learners to deeply engage with the content, critically reflect implications, and clearly demonstrate their understanding.
Context Fields:
"<tttte>>: Contextualizes the content.
31

---

<document__ummaay>" Brief overview providing contextual understanding.
><text_chukk=" The sole source text for developing rich, meaningful questions.
Stadittiona_insttuutionssss Instructions that influence question style, content, and complexity.
Analysis Phase
Conduct careful analysis within "<bocument_analssis>> XML tags, following these steps:
1. **Thuughtful Content Examinatione*
Carefully analyze the given text_chunk, identifying central ideas, nuanced themes, and significant relationships within it.
**Cocepp Exploration**
Consider implicit assumptions, subtle details, underlying theories, and potential applications of the provided information.
3. **Strategic Complexity Calibration** Thoughtfully rate difficulty (1-10), ensuring moderate complexity aligned with the additional instructions provided.
*IIntentiona Question Planning**
Plan how questions invite deeper understanding, meaningful reflection,
critical engagement, ensuring each question is purposeful.
## Additional Instructions for Handling Irrelevant or Bogus Information
Identification and Ignoring Irrelevant Information:
*Irrelevant Elements:** Explicitly disregard hyperlinks, advertisements, headers,
footers, navigation menus, disclaimers, social media buttons, any content clearly irrelevant or external to the core information of the text chunk.
**Bogus Information:** Detect and exclude any information that appears nonsensical or disconnected from the primary subject matter.
Decision Criteria Question Generation:
**Maningfull Content Requirement: ** Only generate questions if the provided <<text_chunk>" contains meaningful, coherent, and educationally valuable content. *#Complete Irrelevance: ** If the entire "<text_chunk>" consists exclusively of irrelevant, promotional, web navigation, footer, header, or non-informational text, explicitly state this your analysis and do NOT produce any question-answer pairs.
Documentation in Analysis:
Clearly document the rationale in the <document_analysic tags when identifying irrelevant or bogus content, explaining your reasons for exclusion or inclusion decisions.
Briefly justify any decision NOT to generate questions due to irrelevance or poor quality content.
## Question Generation Guidelines
32

---

### Encouraged Question Characteristics:
Thoughtful Engagement**; Prioritize creating questions that inspire deeper thought - and nuanced consideration.
**Moderate Complexity*#; Develop questions that challenge learners appropriately whohuut overwhelming them, following the provided additional instructions.
**Self-contained Clarity**; Questions answers should contain sufficient context, clearly understandable independently of external references.
*#Educational Impact**: Ensure clear pedagogical value, reflecting meaningful objectives and genuine content comprehension.
##Conversational Tone**: Formulate engaging, natural, and realistic questions appropriate to the instructional guidelines.
Permitted Question Types:
Analytical
Application-based
Clarification
Faunterfactual
Conceptual
True-False
Factual
Open-ended
False-premise
Eage-case
(You do not need to use every question type, only those naturally fitting the content and instructions.)
## Output Structure
Present your final output as JSON objects strictly adhering to this Pydantic model within * "<output_json>" XML tags:
python
class QusstionnnserPPrr((ssttoo)
thought_prooess: str Clear, detailed rationale for selecting question and analysis approach
question_type: Literall""nalyticoa paaiiccatio--bbbed" "clarification",
"counterfactual", "conceptual", "true-false", "factual", "open-ended", "false-premise", "edge-case")
question: str answer: str
detimated_difficulle int 1-10, calibrated according to additiona tional instructions
citations: List[str] # Direct quotes from the text_chunk supporting the answer
## Output Format
Begin by thoughtfully analyzing the provided text_chunk within "<doumment_amalsiis> XML tags. Then present the resulting JSON-formatted QuestionAnswerPairs clearly within
"<output_json>" XML tags.
## Impor tant Notes
33

---

Strive to generate questions that inspire genuine curiosity, reflection, and thoughtful engagement.
Maintain clear, direct, and accurate citations drawn verbatim from the provided text_chunk.
Ensure complexity depth reflect thoughtful moderation as guided by additions instructions.
Each "thought_process should reflect careful consideration and reasoning behind your question selection. Ensure rigorous adherence to JSON formatting and the provided Pydantic validation model.
When generating questions, NEVER include phrases like "as per the text," 'according to document," or any similar explicit references. Questions should inherently integrate content naturally and stand independently without explicit references the source material
H.3
Multi Hop Question Generation Prompt
##
Your Role
You are an expert educational content creator specialized in generating insightful and thoughtfully designed multi-hop questions. Your task to craft sophisticated, moderately challenging questions that inherently require careful, integrative reasoning multiple chunks of textual information. Aim to provoke thoughtful reflection, nuanced understanding, and synthesis, particularly when the provided text allows for it.
## Input Structure
Your input will consist of these components:
[Specific guidelines, preferences, constraints influencing question generation.) V/aditiiona_insttuuiioonss
<title>
[Document title]
</title>
<document_sumary>
[A concise summary providing context and thematic overview.
</ /uucument_summar>>
<text_chunks>
<text_chunk_@>
[First text segment]
</text_chunk_e>
<text_chunk_1>
[Second text segment]
</text_chunk_1>
[Additionn text segments necessary]
</text_chunks>
34

---

## Primary Objective
Generate a thoughtful, educationally meaningful set of multi-hop question-answer pairs.
-
Questions should ideally integrate concepts across multiple text chunks, challenging
*. learners moderately encouraging critical thinking deeper understanding.
##* Context Fields:
"<itte>>" Document context
dddocument_summarye Broad contextual summary for orientation
chtext_chunk>>': Source material to form integrative multi-hop questions
beadditional_instrcttii Specific instructions guiding the complexity and depth of
.. questions
## Analysis Phase
Perform careful analysis within "<documen__analsiis>" XML tags:
In-depth Text Analysis**
Thoughtfully read each text chunk.
Identify key themes, nuanced details, and subtle connections.
Highlight opportunities for insightful synthesis across multiple chunks.
2. **Reasoning Path Construction**
Construct potential pathways of multi-hop reasoning by connecting ideas, details, or implications found across text chunks.
Complexity Calibration**
Rate difficulty thoughtfully scale of 1-10, moderately challenging learners according provided additional instructions.
4. **Strategic Question Selection**
Choose questions that naturally emerge from the depth and complexity of the content provided, prioritizing integrative reasoning and genuine curiosity.
## Question Generation Guidelines
### Question Characteristics
**Multi-Hop Integration**: Questions should naturally require integration across multiple chunks, demonstrating clear interconnected reasoning.
#*Thoughtfulness Complexity**; Construct questions that stimulate critical thinking, reflection, or moderate challenge appropriate
content.
**Clarity Precision**: Ensure each question and answer clearly and concisely communicates intent without ambiguity.
*#duucttional Relevance**: Ensure each question has clear pedagogical purpose, enhancing understanding critical reflection.
*#Authentic Language**: Use engaging, conversational language reflecting genuine human curiosityyyyyocc and inquiry.
Suggested Question Types
(Use naturally, as fitting to the content complexity)
Analytical
Application-based
Clarification
35

---

Counterfactual
Conceptual
True-False
Factual
Open-ended
False-premise
Edge-case
## **filtering Irrelevant Content**;
**gorre completely** any irrelevant, redundant, promotional, or unrelated content, including headers, footers, navigation links, promotional materials, ads, or extraneous hyperlinks frequently found in web extracts.
**Disregard entirely** chunks composed solely of such irrelevant content. Do **not** generate questions from these chunks.
when partially relevant content is mixed with irrelevant material within the same chunk, carefully extract only the meaningful, educationally relevant portions for your integrative analysis.
#Evaluating Chunk Quality**:
If, upon careful analysis, a chunk does not provide sufficient meaningful context or substantial educational relevance, explicitly note this in the <document__nallsis>> section and refrain from generating questions based on it.
**rioritizing Quality and Relevance**;
Always prioritize the quality, clarity, and educational integrity of generated + questions. not force questions from unsuitable content.
## Output Structure
Present output JSON objects conforming strictly to the following Pydantic model within * "<output_json>" XML tags:
"""yytoon
class QusttionnnseePaar(ssstttd).)
thought-process: str Explanation of integrative reasoning and rationale question_type: Literal["analyticn "application-based"" "clarification",
"counterfactual", "conceptual", "true-false", "factual", "open-ended", "false-premise", "edge-case""
question: str answer: str
estimated_difficulle int 1-10, moderately challenging per addi tional
instructions
citations: List[str] # Exact supporting quotes from text_chunks
## Output Format
First, thoroughly conduct your analysis within "<document_analyii>>> XML tags. Then, provide your synthesized question-answer pairs valid JSON within "<output__son>> tags.
# Impor tant Notes
36

---

Prioritize depth and thoughtfulness in your reasoning paths.
Allow natural complexity to guide question formulation, aiming for moderate challenge.
Precisely cite verbatim excerpts from text chunks.
Clearly communicate your thought process for integrative reasoning.
Adhere strictly to JSON formatting and Pydantic validation requirements.
Generate questions that genuinely inspire deeper reflection or meaningful exploration of the provided content.
When generating questions, NEVER include phrases like "as per the text,' 'according similar explicit references. Questions should inherently integrate content naturally and stand independently without explicit references to the source material
the document,"
H.4 Judge System Prompt
You will be provided with the sunmary of document, a piece of text, question generated from that text, and the correct or "gold" answer to the question. Addi tionally, you will receive two answers: Answer A and Answer B. Your task determine which of these answers closer to the gold answer by assessing the overlap of key points between the ground truth and the two given answers.
Steps
1. **Document Understandinga*:
Analyze the provided document surmary to grasp the context and main themes.
**Chunk Understandinge*:
Examine the provided text (chunk) understand its content.
3. **Question Understanding*:
Interpret the given question to fully comprehend what is being asked.
**Ground Truth Answer Understandingaw:
Understand the provided ground truth answer, identifying key points.
5. **Answer Understandingax:
Analyze Answer A, identifying key points and assessing accuracy and factuality.
**Answer Understanding#*:
Examine Answer B, identifying key points and assessing accuracy factuality.
*$Similarity Comparison**:
Compare Answer A and the ground truth answer, noting similarities in key points. Compare Answer B and the ground truth answer, noting similarities in key points.
**Final Similarity Analysis**:
Evaluate both answers based on the similarities identified and determine which
closer to the ground truth terms of key points and factuality.
Output Format
Provide your final evaluation of which answer is closer to the ground truth within *. "sfinal.answer" XML tags.
37

---

Include detailed analysis for each part within the designated XML tags:
"question_understading"" "ggrond__truth_anseer uuneersrtntrrstnnng""""" <answer_a_ "ut
cdbcument_underrttaddnnn"" =chunk__udersttndiing
kanswer____unersstaree "<similarity watpiiaaattyccppprrrreee www.cprarssn_nnsser..eee aasseerr_a__
Lnneaaeeeee b>", and "<final_sif simiilarttty..alllssss"""
Examples
**Input**:
**xxm
adocument_summary>
[Summary]
/ummcrment_summar>>
<piece_of_text>
[Text]
</piece_of_text>
<question>
[Question]
</question>
<gold_answer>
[Gold Answer] </gold_answer>
<answer_a>
[Answer A]
</answer_a>
<answer_B _b>
[Answer B]
</answer
**Output**:
xm1
<docmmnt__uderstanddnng
Understanding of the summary including key themes
</dboumett_umderstandinng
<chunk_understanding>
Analysis of the piece of text </chunk_understandinng?
jumestion_understandinn>
Comprehension of the question being asked
/qquestion_underssandnn>
Key points from the gold answer V/rromm_.ttthh.asser__ueerttttt
<anseer._.uderstandding
38

---

Key points and accuracy of Answer </answer___udeerstandiny
Key points accuracy Answer
</answerb_underrsanddng>
Comparison notes between Answer and the gold answer
Viimllervy_comarrioonnssssesr
simnilartyycompriisonnavvvvee
Comparison notes between Answer and the gold answer
</sinilarityccomprisooossssrr
<finaLsiiillrritaamalsss>
Overall analysis determining the closer answer
<ffinallsinlarrit..aalssss
<final_answer>
Answer (where is the option you pick)
</final_answen
# Notes
Always focus on key points and factual correctness as per the ground truth.
Avoid any biases and rely solely on the evidence presented. Enclose all evaluations and analyses in the specified XML tags for clarity and
structure.
39

---

I Question Validity
L.1 Valid Question Examples
1.1.1 Example
Question Details
## Source Information
iraqi immigrant hailed as hero preventing armed robbery at ypsilanti juice shop ypsilanti, mich. (wxyz) vara juice in ypsilanti nearly became the victim of armed robbery this past friday. caught on camera, the suspect had no clue that his attempt to make quick cash would come to hard stop, all thanks to hero who was next door. -hirtt--five-year-old ali hadma owns a hookah place called cups on mission, located next to vara juice on washtenaw ave. **"3 years, "** said ali when asked how long he's owned the shop. pins the suspect against counter. a struggle control the firearm begins. ali disarms the suspect. eventually takes him down. "have you got any tactical self-defense training? asked. "no. just go to the gym days week," said ali. once ali got the cash back, he let go of the suspect, who can be seen walking away in the security footage. all the girls he treats like his sisters," "** said sadam badani, the owner of the vara juice location. badani tells me mariam is doing okay, but her parents will only allow mariam to resume work if her hero, ali, is around. "i don care about the money, about anything else. as long as nobody got hurt," said sadam. "whenever ali need me, i'll be there," " said sadam.
## Question
In what ways have Ali 's actions during the robbery influenced the community's perception of him and their sense of security?
## Answer
Ali' s actions during the robbery have made him local hero and gained him widespread appreciation. The community, including the juice shop owner and employees, deeply *a appreciates his bravery and quick thinking. This has led to a stronger sense of security, with the juice shop owner stating that Mariam can only resume work if Ali + is around.
## Citations
[All the girls he treats like his sisters," " said Sadam Badani, the owner of the Vara Juice location., "Whenever Ali need me, I'll be there," said Sadam.
Human Evaluation
## Determination
valid
## Reasoning
40

---

Generation Details
## Model
mistralla/MMsttrrl-arg-----sssss 14
## Question Category
open-ended
## Kind
multi_hop
## Estimated Difficulty
6/18
I.1.2 Example
Question Details
## Source Information
(truncated)... (pn12-36) christopher landau (cal. no. 41) (pn1 2-25) ordered, that following the conclusion of morning business on monday, march 24, 2025, the senate proceed to executive session resume consideration of the nomination of john phelan, of florida, to be secretary of the navy. (mar. 14, 2025. ) michael kratsios (cal. 38) (pn13-8) jayanta bhattacharya (cal. 44) (pn12-2) martin makary (cal. no. 45) (pn12-28) james bishop (cal. no. 39) (pn12-3) aaron reitz (cal, 48) (pn12-37) ordered, that on tuesday, march 25, 2825, the cloture motions on the following nominations ripen: michael kratsios, of south carolina, to be director of the office science and technology policy; jayanta bhattacharya, of california, to director of the national institutes of health; martin makary, of virginia, to be commessioner food and drugs, department of health and human services; james bishop, of north carolina, to be deputy director of the office of management and budget; and aaron reitz, of texas, to be an assistant attorney general. 33 25-32 jonathan mckernan, of tennessee, to be mar 06, 2025 reported by mr. director, bureau of consumer financial protection for term of five years, vice rohit chopra. scott sc, committee on banking, housing, and urban affairs, without printed report. florida, to be secretary of the mar 11, 2025 reported by mr. navy, vice carlos del toro, resigned. wicker, committee on 12, 2025 reported by risch, committee
tment tment defense
12-36 john phelan,
armed services, without printed report. foreign relations, without printed report. department of veterans affairs 43 13-9 paul lawrence, of virginia, to be deputy mar 12, 2025 reported by mr. secretary of veterans affairs, vice tanya j. bradsher, resigned. moran, committed on veterans' affairs, without printed report. signifies nominee's commitment to respond to requests to appear and testify before any duly constituted committee of the senate 5 nominations calendar no. mar 13, 2025 reported by mr. grassley, committee on the judiciary, without printed report. 13, 2825 reported by mr. grassley, committee the judiciary, without printed report. mar 13, 2025 reported by mr. grassley, commi ttem on the judiciary, without printed report. mar 13, 2025 reported by mrs.
capito, commit ttmm environment and public works, without printed report. * 50 25-53
aaron szabo, of virginia, to be an assistant mar 13, 2825 reported ted by mrs
41

---

## Question
On what date are cloture motions for the nominations of Michael Kratsios, Jayanta Bhattacharya, Martin Makary, James Bishop, and Aaron Reitz set to ripen, and what are their respective positions?
## Answer
The cloture motions for Michael Kratsios (Director of the Office of Science and Technology Policy), Jayanta Bhattacharya (Director of the National Institutes of Health), Martin Makary (Commissioner of Food and Drugs, Department of Health and Human Services), James Bishop (Deputy Director of the Office of Management and Budget), and Reitz (Assistant Attorney General) March 25, 2025.
ripen Tuesday,
## Citations
Mar. 14, 2025. Ordered, That on Tuesday, March 25, 2025, the cloture motions on the following nominations ripen: Michael Kratsios, of South Carolina, to be Director of Office Science and Technology Policy; Jayanta Bhattacharya,
California, be Director of the National Institutes of Health; Martin Makary, of Virginia, be Commissioner of Food and Drugs, Department of Health and Human Services; James Bishop, of North Carolina, to be Deputy Director of the Office of Management and Budget; and Aaron Reitz, of Texas, to be an Assistant Attorney General. ']
Human Evaluation
## Determination
Valid
## Reasoning
question, answer and citations are correct
Generation Details
## Model
Qwen/Qwen2. 5- 4B-IInstruct
## Question Category
factual
## Kind
multi-hop
## Estimated Difficulty
42

---

7/10
date of birth gender: male female complete mailing address: email address: 1.1.3 Example
Question Details
## Source Information
org. following the selection process, all applications will be destroyed. questions? please send an email to: scholarshipssagbll. org response time may be to three business days, so please plan accordingly when submitting your questions. george h. nofer scholarship for law 2025 please type or print clearly and review for accuracy; illegible or incorrect information will delay review and could disqualify your application. identifying information name (first, mi, last):
hom/dd/yyyy/
communication throughout the process will be via email. if you do not provide an email address, it is written incorrectly, or if we are not able to read it, we will not able to communicate with you. telephone number:
you do not have cochlear implant and your pta is below 60db in your better-hearing ear, you do not qualify.
hearing health history age when hearing loss was diagnosed:
## Question
How will applicants be contacted regarding updates or decisions about their scholarship application?
## Answer
Communication throughout the process will be via email.
## Citations
Communication throughout the process will via email.")
Human Evaluation
## Determination
valid
## Reasoning
# Generation Details
## Model
google/gemini-2. 0- -fflash-001
43

---

## Question Category
factual
## Kind
single shot
## Estimated Difficulty
6/10
1,2
Invalid Question Examples
1.2.1 Example
Question Details
## Source Information
according to the committee, out of the 40 who signed up to deliver testimony, 38 were opposed to bill. one of the biggest points of concern was relation to the staff-to-child ratio being lowered. as bill currently written, single person would be allowed support of the bill believe that won't be problem and instead, will open up more oppor tunities for idahoans to start daycare businesses of their own. chris cargill with mountain states policy center explained, "we believe that if the legislation is *. passed, we will see an increase in new and quality childcare providers in idaho. mark kirby soda springs, idaho, told the tragic story of how his 11-week-old nephew lost life after being dropped very wrong- upon arriving there, she was
large daycare facility overseeing many children. those in
at a daycare. "later that afternoon, she
got call stating that something eecorted in and learned that her had passed away from asphyxiation, said teary-eyed kirby. kirby explained that investigation found the death was due to overwhelming child-to-staff ratios and negligence of supervision while the baby was napping. the committee passed the bill five to four. "we are clearly disappointed with the outcome," said kirby. feel more disappointed about losing this bill than any other the session. we have worked on this issue for decades to ensure that we are promoting business practices and children's safety," added senator melissa wintrow. the bill will head to the senate floor. if majority of state senators vote to the legislation, it will head to the governor's desk, where it will either be vetoed or signed into law. this is developing story, and we will continue to update it as more information becomes available. #if more news in downtown boise we cover stories making an impact in downtown boise. this is your home to stay on top of what is changing downtown boise and why matters to you and your family. we want hear from you! tell us what we should be covering your neighborhood.
# Question
How did the death of child at daycare facility through asphyxiation influence the testimony given during the committee hearing on House Bill 243?
Answer
44

---

The tragic death caused asphyxiation due high child-to-staff ratios pivotal
moment that resonated emotionally with the commi ttee members and played
signi ficant role in bolstering the opposition to the bill, underscoring safety concerns related to its proposed deregulation.
Citations
[ "'Later that afternoon, she got call stating that something very wrong. Upon arriving there, she was escorted in and learned that her son had passed away from asphyxiation," said a teary-eyed Kirby." ]
Human Evaluation
## Determination
Invalid
## Reasoning
the citations don't t support the answer. it is also factually inaccurate according to the text
Generation Details
## Model
microsoft/Phi---minii
## Question Category
Factual
## Kind
multi_hop
## Estimated Difficulty
6
1.2.2 Example
Question Details
Source Information
45

---

(truncated)... and trustworthiness. prepare, many increasing their cyber budgets with particular focus data protection and trust. by strategically investing in these areas, companies are not only building resilience but positioning themselves positively to their customers. ### investing in what matters most: cloud and data trust go hand-in-hand over the next 12 months, organisations are prioritising data protection/trust and cloud security above other cyber investments. they understand that securing sensitive information is vital to maintaining stakeholder trust and brand integrity. g. reducing the time recover mission-critical data patching a system). determine the business value of data protection and cloud security to gain stakeholder trust and make more informed cybersecurity investment decisions. collaborate with tech, security and finance executives to pinpoint the most essential data security and integrity priorities to guide the information and cloud security investment strategy. confirming data quality and readiness necessary to increase security investments. is your cyber strategy and leadership driving real resilience? from lagging resilience efforts gaps in ciso involvement in strategic decisions, there are clear areas where strategic alignment is needed. to get there, organisations should emulate the leading cybersecurity practices of their top performing peers. they should also move beyond addressing known threats and implement an agile, secure-by-design approach to business, one that strives to build trust and lasting resilience. #fftt partial implementation isn't enough despite mounting concerns about cyber risk, most businesses are struggling to fully implement cyber resilience
practices. review of resilience actions processes and technology indicates that 42% fewer of executives believe their organisations have fully implemented any one of those actions. more concerning, only 2% say all 12 resilience actions have been implemented across their organisation. this leaves glaring vulnerability - without enterprise-wide resilience, companies remain dangerously exposed to the increasing threats that could compromise the entire operation. elevating the ciso: aligning strategy with security many organisations miss critical opportunities initiatives. fewer than half of executives tell that their cisos are largely involved in strategic planning for cyber investments, board reporting and overseeing tech deployments. this gap leaves organisations vulnerable to misaligned strategies and weaker security postures. ## bridging the gaps to cyber resilience: the c-suite playbook ### sign up to get the full playbook and access more the latest findings * for 2025. #*# about survey the 2025 global digital trust insights a survey 4, 842 business and technology leaders conducted in the may through july 2024 period. quarter of leaders
people,
not fully involving their cisos in key
from large companies with $5 billion respondents operate in range of industries, including industrials and services (21%), tech, media, telecom (20%), financial services (19%), retail and consumer markets (17%), energy, utilities, and resources (11%), health (7%) and government and public services (4%). respondents are based in 77 countries. the regional breakdown is western europe (30%), north america (25%), asia pacific (18%), latin america (12%), central and eastern europe (6%), africa (5%) and middle east (3%).
in revenues.
## Question
what are the critical factors that organizations should prioritize to achieve true cyber resilience, and how do they interconnect?
## Answer
46

---

Critical factors include CISO involvement in strategic planning, adopting
secure-by-design approach, and investing in data protection and cloud security. These elements interconnect by ensuring strategic alignment, proactive security measures, and building stakeholder trust, which collectively enhance organizational resilience.
## Citations
[ L'While the rapid advancement of generative AI (GenAI) is ushering in opportunities across industries, it also presents cybersecurity risks.', 'To prepare, many are increasing their cyber budgets with particular focus on data protection and trust.',
"They should also move beyond addressing known threats and implement an agile, becure-by-design approach business, one strives to build trust and lasting resilience."
Human Evaluation
## Determination
Invalid
## Reasoning
answer looks to be correct, but the citations are wrong
Generation Details
## Model
## Question Category
analytical
## Kind
multi -hop
## Estimated Difficulty
8/10
I.2.3 Example 3
Question Details
Source Information
47

---

"it's nice to have the memories, but i wish could make more," said jesse collins, alexis' uncle. **investigative findings: **david plagmann, 36**: responsible for alexis' death. fired by the shelby county sheriff's office. **marianne and jesse collins**; alexis' aunt and uncle. **jake collins**: alexis' father. alexis' family describes her as having a soft loving heart, always step up care for others,
including her four children. she was always positive and believed things would improve, even if was hard.
## Question
How many children did Alexis Martin-Colling have?
## Answer
Four children
## Citations
[She was always positive and believed things would improve, even if it was hard.]
Human Evaluation
## Determination
Invalid
## Reasoning
answer is correct factual, makes a valid citation, but the citation points wrong part of text
Generation Details
## Model
claude-3- 2---hhaiu-222441111
## Question Category
factual
## Kind
single shot
## Estimated Difficulty
2/10
48


Reviews:
[{"review_id": "RI2ADPDOb3", "reviewer": "Reviewer_ERbz", "summary": "This work introduces *YourBench*, a high-quality and clearly presented framework for automated, document-grounded LLM evaluation. The approach is original, leveraging user-provided documents to dynamically generate benchmarks without manual annotation. The framework is well-validated through benchmark replication and rigorous analysis, showing strong potential to address issues of benchmark saturation, contamination, and domain specificity. Overall, this is a timely and significant contribution to the field.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe research problem addressed in this work is timely and valuable. Developing automated and up-to-date benchmarks is a pressing need in the field, particularly for evaluating and iterating on large language models efficiently.\n2.\tThe proposed pipeline is comprehensive, as it covers all key components including document preprocessing, question-answer generation, quality control, and final evaluation. This reflects a high level of completeness in the overall system design.\n3.\tThe experiments are thorough and well-designed, evaluating the benchmark generation process from multiple perspectives to ensure robustness and reliability.\n4.\tThe work is highly reproducible, with detailed descriptions provided for each step of the pipeline, enabling easy replication by the research community.\n5.\tThe paper is clearly written and well-organized, making it accessible and easy to follow.", "reasons_to_reject": "1.\tWhile the proposed method effectively targets domain-specific benchmark construction, it primarily focuses on knowledge-based question generation. Its core approach relies on leveraging the information integration capabilities of LLMs to generate benchmarks grounded in source documents. As a result, it may face challenges in producing more creative or cognitively demanding questions, such as those involving complex logical reasoning or multi-step inference.\n2.\tThe experiments are limited to replicating the MMLU dataset. Incorporating additional datasets such as MMLU-Pro or GSM8K would help demonstrate the broader applicability and robustness of the approach.\n3.\tAlthough the paper introduces an evaluation strategy based on a judge LLM ensemble, this component is not clearly validated in the MMLU replication experiments, which do not rely on such a mechanism. The work would benefit from more direct comparisons between its evaluation methodology and existing benchmarks to establish alignment and effectiveness.", "questions_to_authors": "1.\tIs the proposed method capable of generating benchmarks involving multi-turn dialogues, such as those found in MT-Bench?\n2.\tWhat is the recommended or best-practice ensemble configuration of LLMs based on the analysis in Section 3.2? Specifically, how are models selected to jointly optimize for question validity and diversity?\n3.\tSeveral hyperparameters are mentioned in the paper, such as the citation confidence threshold $\\theta_{cit}$ in Section 2.3.1. Could the authors elaborate on the sensitivity of the framework to these hyperparameters, and whether any tuning studies were conducted to assess their impact?", "extra_scores": {}}, {"review_id": "nTDBEqPpVf", "reviewer": "Reviewer_i1Fe", "summary": "This paper introduces YourBench, an open-source, modular framework for dynamically generating evaluation benchmarks for large language models (LLMs) directly from user-provided documents. The authors propose a Document-to-Evaluation Generation (D2EG) pipeline, leveraging LLMs to generate diverse, context-grounded, and citation-supported question-answer (QA) pairs with minimal human annotation. They validate YourBench by replicating subsets of the MMLU benchmark and introduce TEMPORA-0325, a dataset of over 7K documents published after March 2025 to mitigate contamination from pretraining data. The system supports scalable, cost-efficient, and temporally relevant benchmarking across domains and model families.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses critical limitations in LLM evaluation: benchmark saturation, data contamination, and the cost of human annotation. YourBench provides a scalable and dynamic alternative to static benchmarks.\n2. The YourBench pipeline—comprising ingestion, chunking, summarization, QA generation, citation validation, and deduplication—is well-designed and clearly explained. Each component is technically sound and grounded in prior work.\n3. All code, data (including TEMPORA), and inference traces are planned for release. This supports the reproducibility and extension of the work.", "reasons_to_reject": "1. Evaluation Bias Potential Since YourBench can evaluate models using LLM judges (which could be the same or similar to generation models), there's a risk of self-evaluation bias. While ensemble judging is a good mitigation step, a deeper discussion or ablation on this point would strengthen the paper.\n2. The system’s outputs are inherently limited by the capabilities and biases of the LLMs used to generate and filter the content. There is potential circularity when LLMs are evaluated by questions generated by other (possibly similar) LLMs.", "questions_to_authors": "1. Given LLM stochasticity, would the same document and prompt yield similar QA sets across runs? \n2. You claim YourBench-generated questions are harder than MMLU’s originals. Is this purely based on lower accuracy, or are there semantic or structural metrics?\n3. Is the YourBench framework effective in generating meaningful evaluation sets for entirely out-of-distribution domains where the underlying LLMs have limited or no prior exposure?", "extra_scores": {}}, {"review_id": "fW5tYoB3je", "reviewer": "Reviewer_pLCc", "summary": "Main idea: Framework (i.e. pipeline) for guiding language models to construct new benchmarks given minimal supervision\n\nEvidence: \n- 7 subsets of MMLU have new replicas generated using YourBench\n- One (or two, depending on how one counts) new benchmarks are created to reduce contamination and test new knowledge post March 2025", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This is a compelling problem to work on and solve\n- The authors' framework and general overall approach seem sensible, albeit limited to question answering benchmarks", "reasons_to_reject": "I think this paper is conceptually fine, but needs to do a LOT more work to build trust in this framework/pipeline.\n\n## Section 2\n\n- Section 2.2. I feel like a key criterion is missing from D2EG: importance. The three listed criteria (coverage, diversity, answerability) are all well and good, but content is not created equally and evaluations should concentrate on \"the material we care about\". One could envision asking questions like \"What word does the Nth sentence end on?\" or \"How many words contain the letter E?\" or \"How many links are contained in the paragraph\", which could meet all three criteria, but aren't really the substance we want to evaluate models on. Maybe a different term should be \"external validity\", by which I mean: do consumers of models' benchmarks scores feel like the scores are capturing the important elements of the task at hand?\n- The anonymized URLs at the bottom of page 2 in the PDF both direct to `https://openreview.net/ANONYMIZED%20URL.pdf#[0,{%22name%22:%22Fit%22}]`. I am thus unable to scrutinize the YourBench bench or the TEMPORA-0325 dataset.\n- In Appendix D.2, which LLMs were used in the ensemble of judges? I was unable to find this information.\n- I'm not sure I see any analyses about the effects of YourBench's hyperparameters: $\\theta_{\\text{cit}}$ and $\\tau_{\\text{sim}}$. How robust is the YourBench framework to these hyperparameters?\n\n## MMLU Replication\n\n- I think many more models should be added to Figure 1. This would significantly strengthen the claim that YourBench's MMLU replica \"perfectly preserves the relative performance ranking\" and \"Spearman ρ=1.00\". There are many more than 8 models in the open-parameter ecosystem, and 7 subsets of MMLU is not so expensive to evaluate. For suggestions: DeepSeek, Gemma 1 & 2, Phi, OLMo, Mistral, Pythia, Cerebras, LLM360 https://huggingface.co/LLM360, etc.\n- I think Figure 1 would be better visualized as a scatterplot with Original scores on one axis and YourBench Synthesized scores on another axis. I say this for two reasons: (1) the specific models here are less important than preserving the ranking, and (2) as I comment below, preserving the ranking of scores is oftentimes insufficient to evaluate models as one oftentimes wants to know the quantitative difference in performance. One way to do this would be to have a scatterplot per subset, arranged in a 2-rows-by-4-columns grid.\n- As an extension of the previous point, here, we actually have 7 different YourBench-constructed benchmark subsets. I feel like Figure 1 should show both an aggregate score (as it currently does) but also a per-subset score. The latter feels even more important to me because if an ordering is preserved for each benchmark subset, then the same ordering is likely to hold for the aggregate score, but the opposite is less likely to be true.\n- After making the above point, I realized that the authors must surely have done this analysis. I found Appendix F1 and discovered that Spearman drops from 1.00 to 0.2982. This is also mentioned on Page 8. This is a terrible degradation and I am unhappy that this incongruity is not front and center in Figure 1.\n- Relatedly, it is remarkable that Pearson and Spearman correlations are high in aggregate but low for each subset. This feels extremely surprising to me.\n- It is unclear to me how many times the authors generated their MMLU replication and/or whether it was \"tuned\" in any way. Ideally the authors have evidence that the dataset was generated once, and that no hill climbing was performed on the dataset generation methodology. Otherwise, the MMLU replication results are not representative of YourBench. Could the authors please provide such evidence?\n- Table 1 in Appendix F3 shows standard errors. I prefer confidence intervals, and such uncertainty should be visualized in the results. For instance, here is one way to combine a scatterplot with uncertainty in Python visualizations https://matplotlib.org/3.1.1/gallery/statistics/errorbar_limits.html\n- This may be minor, but which model(s) were included in the LLM ensemble (Page 4) used to generate the MMLU Replication Dataset? This seems like a key consideration. I was unable to find this information. I'll call these the dataset-generating LLMs.\n\n## Transcending Dataset-Generating LLMs\n\n- A key question that I don't see addressed is whether YourBench offers a means to evaluate new frontier models. That is, if a benchmark is generated by LLM X, and LLM Y is widely regarded as superior to X, is the benchmark capable of accurately evaluating Y? Or will Y trivially ace the benchmark because anything X can do, Y can do better?\n\n## Appendix F\n\n- Please add the Pearson and Spearman correlations to each figure caption (Figures 9 through 15). As with Figure 1, I would also prefer more models to be added and the data visualized as a scatterplot, but the authors are welcome to push back.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "relgMQ58UA", "reviewer": "Reviewer_SnWz", "summary": "This work devotes to improve LLM evaluation by proposing YourBench. YourBench is an open-sourced framework that generates evaluation samples dynamically and automatically from user-provided documents. Results show that YourBench not only provides more flexibility but also can make trustworthy conclusions cheaper and faster.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) LLMs are general-purpose models, but users always have their only use cases. It is hard to figure out a very user-specific benchmark for everyone because it is timely and expensive on human annotation. However, this work propose to generate queries from user-provided documents automatically, which can greatly alleviate this problem.", "reasons_to_reject": "1) I don't understand why this apporach can really solve contamination problem. It is possible that the user-document is contaminated as well if it has been used in the pretraining.\n2) I also don't understand why this approach can workaround saturation. Instead, if my understanding is correct, the question difficulty highly depends the on the source document the the capacity of LLM generating evaluation samples. So it is actually easy to saturate in my opinion.\n\nAlthough YourBench seems have some weakness, I still believe that the flexibility is important for general-purpose evaluation. Preparing documents are always easier than preparing QA pairs for user. So I tend to accept this work as a good research on LLM eval.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a pipeline to construct evaluation benchmarks dynamically with user-provided documents. The pipeline leverages LLMs to generate questions and answers based on the provided documents from users. Experiments show that this pipeline can replicate rankings similar to MMLU using Wikipedia articles as source documents.

Pros:
1. Generating dynamic benchmarks is an interesting idea compared to existing static benchmarks. It has the benefits of adapting to user-specific needs and also of avoiding overfiting to existing benchmarks (although to a lesser extent if pretraining data overlap with source documents).

Cons:
1. Evaluation of the proposed approach seems limited to mostly replicating MMLU rankings, and as pointed out by a reviewer, while in aggregation the pearson correlation is high, the per-subset scores could be low.
2. The automatic generation of benchmarks highly relies on existing LLMs, which might limit the quality of the generated benchmark itself.

Overall, three reviewers gave positive ratings (7/7/6), and one reviewer gave a negative rating (2) due to concerns about claims regarding MMLU replication. While I do think this paper can be greatly improved by strengthening the evaluation of their method and clarifying the MMLU replication results, developing dynamic benchmarks based on user needs is a timely and valuable contribution, and I recommend to accept this paper.
Decision: Accept