Grounding Generative Evaluations of Language Models in Unsupervised Document Corpora
Authors: Michael Majurski
Cynthia Matuszek
Venue: Accepted by TMLR ()

Abstract: Language Models (LMs) continue to advance, improving response quality and coherence. Given Internet-scale training datasets, LMs have likely encountered much of what users may ask them to generate in some form during their training. A plethora of evaluation benchmarks have been constructed to assess model quality, response appropriateness, and reasoning capabilities. However, the human effort required for benchmark construction is rapidly being outpaced by the size and scope of the models under evaluation. Having humans build a benchmark for every possible domain of interest is impractical. Therefore, we propose a methodology for automating the construction of fact-based synthetic data model evaluations grounded in document populations. This work leverages the same LMs to evaluate domain-specific knowledge automatically, using only grounding documents (e.g., a textbook) as input. This generative benchmarking approach corresponds well with human curated questions producing an ensemble Spearman ranking correlation of $0.91$ and a benchmark evaluation Pearson accuracy correlation of $0.74$ (model specific $0.82$). This novel approach supports generating both multiple choice and open-ended synthetic data questions to gain diagnostic insight of LM capability. We apply this methodology to evaluate model performance on three recent documents (two post LM knowledge cutoff), discovering a surprisingly strong performance from Gemma-3 models on open-ended questions. Code is available at \url{https://github.com/mmajurski/generative-lm-benchmarking}

Published in Transactions on Machine Learning Research (Jan/2026)
Grounding Generative Evaluations of Language Models in Unsupervised Document Corpora
Michael Majurski Cynthia Matuszek' UUniversity of Maryland Baltimore County mi michael.no majurskii @umbe. edu cmat@umbc. edu
Reviewed on
Abstract
Language Models (LMs) continue to advance, improving response quality and coherence. Given Internet-scale training datasets, LMs have likely encountered much of what users may ask them to generate in some form during their training. A plcthora of evaluation benchmarks have been constructed to assess model quality, response appropriateness, and reasoning capabilities. However, the human effort required for benchmark construction is rapidly being outpaced by the size and scope of the models under evaluation. Having humans build benchmark for every possible domain of interest is impractical. Therefore, we propose a methodology for automating the construction of fact-based synthetic data model evaluations grounded document populations. This work leverages the same LMs to evaluate domain-specific knowledge automatically, using only grounding documents (c.g., lextbook) as input. This generative benchmarking approach corresponds well with human curated questions producing ensemble Spearman ranking correlation of 0.91 and benchmark evaluation Pearson accuracy correlation of 0.74 (model specific 0.82). This novel approach supports generating both multiple choice and open-ended synthetic data questions to gain diagnostic insight of L. capability. We apply this methodology to evaluate model performance on three recent documents (two post LM knowledge cutoff),, discovering a surprisingly strong performance from Gemma-3 models on open-ended questions. Code available at httts:///iithb..
http:///gtthuuu.cooomma//aaaaa com wwt.t///aaaaarrrrrrgooootooooo/o htwpeekkiioouudddd-eeee synth- 1m- benchmark
1 Introduction
Recent advances in language models (LMs) have demonstratod significant capability improvements in diverse natural language processing tasks. This expansion in capability has primarily been enabled by training on Internet-scale corpora. While LMs generate fluent, plausible text, the outputs are not always grounded in the facts of situation. As AI tooling improves professional workflows, rigorous evaluation imperative understand the domain- specific model capability.
Document
Automated Benchmark Generation
Synthetic Benchmark (MCQs & Open-Ended Q6)
Capability Insights
TEST
Traditional multiple choice question (MCQ) bench- marks such as MMLU (Hendrycks et al.) 2020) offer valuable general insights into model performance, but are static, public, and limited scope. The fact that these benchmarks are static and public-and and therefore appear in training data- is significant. weakness. Combined with the human effort required to
Figure 1: Generative evaluation pipeline: (1) users curate grounding documents, (2) LMs generate domain- specific questions, (3) model responses are evaluated.

---

Published in Transactions on Machine Learning Research (Jan/2026)
construct new high quality tests (Phan et al. 2025). it is unlikely evaluation construction can keep pace with LM development. This scaling challenge is especially acute for domain-specific professional applications, where the general benchmarks provide limited insight. Unfortunately, creating expert-validated benchmarks for every domain impractical. New automated methods must generate accurate, comprchensive, diagnostically relevant evaluations grounded in authoritative corpora with minimal human intervention beyond document. selection.
We propose novel automated methodology for generating synthetic benchmarks grounded in authoritative documents. This approach reduces human effort and enables rapid adaptation to new domains by generating fact-based multiple choice or open-ended questions directly from appropriate grounding documents. This work is organized around three core questions:
1. How can LMs extract relevant information and generate high-quality questions grounded in domain documents?
2. Do these generative benchmarks replicate existing human-curated evaluations?
3. What limitations require mitigalion for successful synthet data evaluation of LM capability?
2 Related works
Our approach is motivated by task asymmetry where given grounding documents, an LM should reliably generale diagnostically relevant questions, while it is substantially harder to zero-shot answer those same questions. This information asymmetry motivates using LMs to generate benchmarks grounded in curated documents rather than relying only on model weight knowledge. Language models are routinely adopted as judges: extracting answer choices (Samuylova 2024 Roucher 2024), grading open ended responses (Ip 2024
Oh et al. 2024), and justifying correctness (Cook et. al. 2024]. The most similar contemporary convergent. work is YourBench (Shashidbar et al.) (2025) synthetic replication of MMLU from grounding documents and AutoBencher (Li et al. 2025b) reinforcement learning optimization to discover gaps in LM knowledge. Leveraging grounding documents greatly improves synthetic data utility, as the methodology no longer relies on internalized LM knowledge.
Modern long-contens LMs can process entire documents. Benchmarks like SecQA Lin 2023 and Pub- MedQA (Jin et al.) 2019) leverage domain-specific textbooks and web pages. Full-document grounding generally outperforms summary-based systems (Bhat et al.) 2023). Contimued LM improvement, prompting strategies (Yadav et al. 2024), and topic breadth (Li et al. 2024 has further improved synthetic data. These methods also enhance downstream applications like retrieval-augmmntte generation (Balaguer et al. 2024
LMs can serve as accoptable feature extractors when guided by detailed prompts. Studics show strong correlation between LM-based and human judgments of text quality (Chiang & yi Lee 2023 Han al., 2024 This aligns with the LM-as-a-Judge paradigm used for open-ended benchmark evaluation [Xu et al,
2023 Zheng al. 2023). Beyond open ended answer correctness, LMs can extract other textual features.
However, limitations in LM ground truthing remain, especially in tasks like information retrieval where relevance judgments are more complex (Soboroll 2025: Sxymanski et al. 2025 LM evaluation benchmarks generally use either multiple choice (MC) or open-ended (OE) formats. Open-ended questions require synthesis (Balepur et al. 2025) but depend on a judge LM for scoring (Xu et al., 2023 Myrzakhan et al., 2024), while MCQs are easier to grade but may inflate performance due to answer visibility (Robinson et al, 2022 Balepur et al. 2024). Frameworks like Inspect-AT (UK AI Security Institute (024) support both types with prompt templates and answer shufiling. Both formats are sensitive to rewording, leading to inconsistent outputs (Wang et al. 2024a), and newer frameworks aim to improve open-ended scoring precision (Bernard
et al. 2024 Kamalloo et al. 2023
3
Methods
Constructing synthetic data benchmarks from relevant domain-specific documents requires chunking into contexts, topic extraction per context, question and answer generation, and finally an evaluation framework determine model capability on the newly minted benchmark (as shown in Figure 2 All LM interactions use default generation parameters, except we set temperature 0.7 (o4-mini did not support setting temperature).
2

---

Published in Transactions on Machine Learning Research (Jan/2026)
Text Chunk
Topic
Text Chank
Topic
Sourer
Questton
Question + Answer Generation
Beneltmack
Chunking
Text Chunk
Topic Extraction
Topie
Question Answer
Document
Figure 2: Generative benchmarking pipeline when used in production. During evaluation of the approach significantly more LM-Judge operations are included to evaluate question and answer quality.
Document chunking into contexts While modern LMs have sulficient context windows to support whole document ingestion, chunking encourages the generated questions to be spread throughout the grounding document. An input document is first converted into markdown for interoperability with LMs. We use Docling QAuer et al., 2024) for conversion from pdf markdown, and pandoc (MacFarlanc) for conversion from html to markdown. Once in markdown, each document section is used a context. chunk for question generation. This chunking method is simple and relies the document author crafting relevant sections. Enhancements to chunking and document data preparation will likely produce overall system improvements, but were not required to demonstrate baseline performance in this study. For instance, our approach only leverages the textual data; formatting and including richer sources of information from table, charts, and images may improve evaluation quality.
Topic extraction For each extracted document context chunk, an LM is used to extract and propose any topics present which make high quality difficult questions for evaluating domain expert knowledge about the information in context chunks. This topic extraction leverages the same LM the question generation process. We explored using embedding model to cluster topics and de-duplicate, but empirically found no satisfying improvements, as the topic generation by LM does a reasonably good job of de-duplication. From manual observations, the only duplicate topics stem from different source document context chunks. This duplication is acceptable as the question generation is grounded in the context chunk; any duplicate topics across context chunks will produce different questions.
Question generation Given (a) the document context chunk and (b) a single topic for a question, the generating LM is promptod construct a high quality and difficult evaluation question to probe the knowlodge and understanding of expert in the domain. During question generation, the LM is required to write the question, correct answer, and a mustification/xxplaaattio for why the correct answer is in fact right. The answer justifications are not used in the evaluation benchmark and simply exist to guard against generating incorrect answers. In summary, the single grounding document is broken into chunks of context, per-context relevant topics are extracted, then per-context/topic pair, and a single evaluation question is created. This results in a list of questions with correct answers grounded in (and derived from) the source document. This example drawn from Squadv2 (Rajpurkar al. 2018) shows a question generated by Llama-3.3-700 3.maa-.3-70B-nnstruud grounded in the context:
('context:; "In 1756 and 1757 the Franch captured forts Oswago and William
from the British The latter victory vas narred uhan France's native allies broke the terns of capitulation attacked
retreating British scalping soldiers and taking captive many zen, wozen sad children while the French refused to captives. French naval deployments Fortress of Louisbourg on Cape Breton Island, sacuring the seavard approaches to Quabec.",
under French guard, elaughtering and
protect
successfully defended the koy
"question": "In what year did French naval deploynents succesafully defand the kay Fortresa of Louisbourg on Cape Breton Island, aecuring the seavard approaches to quabec?',
""oicce": "1755",
"B": *1756",
"d"; *1757"
"D"; '1758"),
10
"anaver"; "g")
Multiple choice vs open-ended this study, we explored both multiple choice and open-ended question generation, each requiring slightly differing LM prompts. While simply using the correct multiple choice answer option in open-ended configuration works, observed that open-ended questions enable more variety from
3

---

Published in Transactions on Machine Learning Research (Jan/2026)
the generating model if prompts are modified. One study goal was determine if any differences manifested between multiple choice and open-ended questions for synthetic data benchmarking. The difference in response grading requirements between open and multiple-choice favors MCQs for fast and simple exploration of model weaknesses.
3.1 Benchmark evaluation
Given a list of questions and answers, an LM evaluation framework (Inspect-AT (CK AT Security Institute, 2024)) is used to determine performance for the models under evaluation. The benchmark consists of many questions with associated answers. Open ended questions are graded using the default Inspect-Al framework support which uses a narrowly scoped LM-Judge compare a candidate model output to the ground truth answer to determine correctness. The resulting benchmark evaluation accuracy is the average number of correct answers divided by the total number of quest.ions. LMs are evaluated against both the synthetic evaluation benchmark and the original extractive QA human written questions lightly rewritten to be answerable without the context. The overall accuracy per evaluated model can be compared for both the synthetic and original questions across multiple datasets of grounding documents. The text of the various benchmark questions and answers is never directly compared, only the resulting LM performance. Figure 3 showcases the performance of synthetic multiple choice questions averaged across datasets and generating models, with Spearman ranking correlation of 0.907, demonstrating slight model over-performance on the synthetic benchmark.
Benchmark eval accuracy grouped Evaluation Model (rank correlation 0.907)
Synthetic Questions
0.85
Original Questions
eval 3ccuracy
0.80
0.75
0.70
0.65
0.60
0.55
Llama-3 38-Instruct
genma- 4h-it
Llama-3 1 1-88--n- iag-insttuct
OwenZ 5 7B B--instrcct
gemma- 3-1 12t-lt
gemma-33 3-2ma---227-kt 3-27b-i
phi4
04 minini
Liama-3 3 1.3--708--o instrnrt
Owen 233- 2at---2----------eeeiee 4238-instruct-250
gpt-4 1-mini
Liama 4 Maverick
gpt-4.1
Evaluation Model
Figure Comparison of average LM performance across NLP datasets with the multiple choice question synthetic data benchmarks created by the ensemble of bolded model in Table 2
4 Evaluation methodology
Evaluating the quality and utility of the newly minted synthetic data benchmark built upon grounding documents leveraged Natural Language Processing (NLP) datasets where each entry contains context paragraph, human written question, and correct answer. This provides directly comparable result, both for the overall benchmark accuracy (between human and grounded synthetic data) and for the rank ordering of those models that the benchmark is evaluated on. For each dataset a variely of LMs are used to generate the synthetic benchmark. Per-question quality features and benchmark-level question diversity extracted compare against the human written questions. Finally, an automatic quality control pass completed grade both question and answer correctness/ /valiiits// in order to investigate and characterize which benchmark-generan models write valid question and answer pairs.
4

---

Published in Transactions on Machine Learning Research (Jan/2026)
4.1 Datasets
Evaluating the validity of generated synthetic questions requires source documents which have human anno- tated question / answer pairs to compare against. The following NLP datasets were evaluated: (a) Squadv2 (Ra)
jpurkar ct al., 2018) (b) HotpotQA (Yang et al., (2018) (c) TrivaQA-weh (Joshi et al. 2017) (d) NaturalQues-
tionsShort (Kwiatkowski et al. 2019) (e) PubMedQA (Jin et al. 2019) (f) BoolQ (Clark et al., 2019)
(g) FermiQA (Kalyan et (0021) (h) MS-MARCO-QA (Bajaj et al., 2016) (i) MusiqueQA (Trivedi et al.
2022) (i) NarrativeQA (Kočiský al. 2018) (k) 2WikiMulliHopQAA (Ho al. 2020), and (l) SecQA (Liu
2023
Table 1: Example human question reformatting
Context: Super Bowl was American football game determine the champion the National Foothall League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolins Panthers 24-10 to earn their third Super Bowl title. The game played February 7, 2016, at Levi's Stadium Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the "golden anniversary" and temporarily suspended
Roman numerals the logo.
Original Question Original Answer
Reformated Question
Answer Choices
What day was the Fehruary 7, 2016
On which specific date
(A) January 7, 2016
game played on?
did Super Bowl 50 take
(B) February 7, 2016
place at Levi's Stadium
(C) February 14, 2016
in Santa Clara, Califor- nia?
(D) January 14, 2016
If Roman als were used, what would Super Bowl 50 have been called?
Super Bowl L
What would the 50th
(A) Super Bowl XL
Super Bowl have been
(B) Super Bowl XLVIII
called under ditional Roman--umeral naming?
tra-
(C)
Super Bow] XLIX
(D) Super Bowl L.
These datasets collectively contain tens of thousands of context, question, answer tuples. A random subset of 100 samples per dataset was selected as the grounding context document. For certain datasets, pre- processing was required to clarify, curate, and reformat the source human written questions into self-contained questions. For some questions this required disambiguation so the question is answerable without the context. For the multiple choice evaluation, three believable incorrect answer options needed to be created. The PubMedQQ (Jin et al. 2019) and BoolQ (Clark et al., 2019) datasets required conversion from yes/no questions into multiple-chore choice or open- open-ended. Only the SecQA Liu 2023) dataset already contained all of the required information and was not pre-processed. All other datasets went through an LM based reformatting operation correct these deficiencies. Table I provides an example of the reformatting operation for a few questions drawn from single context. of the Squadv2 dataset. This disambiguation rewrite was validated using both an LM-Judge and embedding cosine similarity ensure the questions were not unduly altered. Throughout the rest of this work, original questions refers to these lightly edited and disambiguated version of the human authored questions.
Three different models (Llama-4-Mave) 4-averick, gemma- 3-27b-it, o4-mini) were evaluated for this reformat- ting operation. The remainder of this paper shows the Llavn---Maverickk reformatting (sce Table F for the minimally different results of other reformatting models).
Table 2: Question generating language models
gemma-3-4buun 1omma--3-12-11
Team et al. 2025
Google
TTam el al 2025
in Appendix
gemma---277b
[Tbam cl all 2025
Mama-3.2-3B-Insttree
met 2024
Llama-3.1-8B-Jnstruutt
Crat Crattation tatooiao et
2024
Meta
4.2 Question generation models
Llama-3.3-700--Instrurc Crattahori at all
2024
To explore the impact of model capability suc- cessful generation of synthetic data benchmarks tested a varicty of language models during the syn- thetic benchmark generation phase (see Table 2).
o4-mini ope
2025
OpenAl
gpt-4.1 ope 2026
gpt-4.1 Pope
2025
Microsoft
phi-4 (Abdin et al.
2024)
All spight models are available through Hug- gingFace (hug. 2024) and the closod-weight models were accessed through paid APIs. Additional re- sults are reported for an ensemble of the bolded
Qwen2.5-7P 7---5-77--1 78- Instruct Yang et al. 2024
Qwen
1aoon-2335B-222-
1-0-B---A22-nInstuc---- 2507
Instruct-
Team 2025
5

---

Published in Transactions on Machine Learning Research (Jan/2026)
models where all generated questions from model ensemble were mixed together evaluation benchmark. During benchmark creation, the same model is used both for topic extraction and question generation.
4.3 Evaluating against human questions
The reformatted human annotated question/answer pairs are used to compare the grounded synthetic data benchmark performance against the human. Ideally, the synthetic benchmark from grounding documents would produce the same LM capability measurements as the human-curated benchmark for any given model under evaluation. As such, to validate our approach, various LMs of differing sizes are evaluated on both the human and synthetic benchmarks. Comparison of pure benchmark evaluation accuracy is performed using the Pcarson accuracy correlation between human and synthetic benchmark accuracy numbers. Evaluating the comparative rank ordering of models uses the Spearman rank correlation of relative model performance the given benchmark dataset. These two measures compare both absolute performance alignment and relative rank order preservation/stabllii
4.4 Generated question quality characterization
Per-question quality features and benchmark dataset-level features are computed to determine the overall quality of the synthetic benchmark. Owing to the low level of human oversight and annotation time for creating the synthetic data benchmarking, TMS extract these somewhat imprecise features. This relies on the idea that with enough guidance LMs can be good alternatives human evaluation (Chiang yi Lee 2003 Han et al., 2024) and borrows from standard practice LLM-as-a-Judge grading of open-ended benchmark responses Xu et al. 2023 Zheng al. 2023 Llama-4 M-averick extracts the following information about each generated question:
1. question token count: number of tokens in the question
2. question clarity: how clear and comprehensible is the question in isolation without the associated context
3. question groundedness: how grounded is the question in the context
answer correctness: how correct the answer given the context
answer explanation validity: how valid and logically consistent the answer explanation is
In addition to per-question measurements, question diversity measured over each benchmark as the average cosine similarity between all questions (using sentence-tranee twwnsrorannsss////e hatnssfrrrrsaall-iiiiM---- MlI-M as the embedding
model). The answer correctness and answer mnswer_explanee explanationn validity numbers flag which generating models are incapable of producing dingnostically useful questions.
5 Results
All experiments and development were conducted on a single 8xH100 on-premise machine. Development consumed approximately twice the compute of final evaluations. Final compute was split between synthetic dataset generation and benchmark evaluation via Inspect-AT. The most expensive model, gpt-4.1 ingests about 3.6M input and 1.1M output tokens for an average evaluation run ($16), with compute evenly divided between generative dataset creation and evaluation.
5.1 Answer correctness
Synthetic benchmark answers must be accurate, task made casier via the grounding documents that supply the necessary information. To verify generated answer accuracy, an LM scored each quest ion-answer pair and its justification (1-10 scale) using the full context as described in Section 4.4 As shown in Figure 4 reasonably sized models scored above 9.5 in accuracy, with smaller modols performing noticcably worse. While model size/capan capability generally correlates with desirable properties for synthetic data generation, notable exceptions (like 1-t-4.1-mini) are discussed in Figure 14 of Appendix F
5.2 Question clarity and groundedness

---

Published in Transactions on Machine Learning Research (Jan/2026)
Correctness and Validity by Generating Model
10.5
Original answer_ correctness
Original explanation validity
10.0
Synthetic answer_ carrectness
9.5
Synthetic explanation_valicn
8.5
8.0
7.5
7.0
Liama-3 1amma-..-70B-nnttree 3-708-Instructt
Liama-3 (amma-3.2-3B--mme 3--nsstrcct
Liama-3 3. (B-instruct
Owen2.3 Owen2.5.78 7---nsstrctt
gemma-3 3-4bii
gemma---122 --1ma---112--i
gemma-3 3-27b-it it
phi-4
gpt-4.1 1-1mminn
Liama Laaa-4-Maverick Maverick
gRt-4.1 1
od-mini
Qwerd: Qwer3-2358 2wt.----222--nstrr-a--o
Figure Synthetic data generated question correctness and explanation validity on scale of to 10.
Question clarity is inherently tied to difficulty, as simple questions have the potential to be clearer, easier to un- derstand, and casier to answer. Conversely, a needlessly opaque question is harder to answer while providing min- imal additional diagnostic relevance, though a model's s ability to reason through more opaque questions indicates general capability. This study found that most generated questions have acceptable clarily without significant vari- ation between the generating models. Most generated synthetic questions are longer and more complicated than the original questions. This increased average question complexity combined with similar levels of question clarity suggests most synthetic questions are well written. Man- ual random spot inspection supports this as we observed few failures.
by Gererating Mocel
Chart
85-
Degral
06
S6-
55-
Liana 2, tottrntt
2964
Uiee naltorteet
go-d Amie
Gwne redumeet
Figure 5: Generated question clarity and ground- edness compared against the original questions.
How grounded in the source material (context chunk) given question is indicates how much outside information both the generation and evaluation models must leverage to correctly interpret what is being asked and answer. Low grounding indicates the dnestion/answer pair cannot be found within the context chunk, or that significant external information is required. High grounding means all necessary information is found within the context chunk. Therefore the generated question lesting for information aligned with and present the context. Figure highlights the question clarity and groundedness scores for the question generating models. While most LMs generate similar clarity values, the trends in groundedness score are more complex. Some capable models (like Llama-4-Maverinn score well, other capable models (like o4-mini) score worse, with smaller models in between (gemma-3-44-it). We observed two grounding failure modes: either the model hallucinates irrelevant content absent from the context chunk (phi-4 commonly did this), or the generating model uses knowledge in its weights to expand upon the information presented in the context. monmmini). See Appendix G for an example from phi-4. Models of medium size perform well on groundedness, simply reformatting information presented in the context.
5.3 Impact of question length
The synthetic data questions this study tend to be longer than the original human-written questions. The generating model is prompted include all information required to disambiguate what the synthetic question is asking about, causing the questions to be longer and more detailed than the original dataset.
7

---

Published in Transactions on Machine Learning Research (Jan/2026)
Figure 6a highlights this trend. Some generating LMs take this to an extreme, causing the synthetic questions to become easier for LMs to answer correctly. We hypothesize this happens because the additional detail in the question places the LM under evaluation in the right "state of mind" so to speak, in the representation vector space. This effect is scen Figure 6b where benchmark evaluation accuracy over-performance (synthetic original accuracy) is plotted. There is a rough trend where longer questions over-perform the y 0 expected trend line. It worth noting that this effect does not show up in answer length (sce Appendix C for detail and examples).
questtion tokenn, count Generating Model
Fool uabir Accumny Oxepe fomarro Saytheek OKijka0
Impact Question length on Evalastion Overperformane (RP 0.241, 0.4911
Originl Syntheric Questions
0.4
quettion boleen
0.2
Uteea hSB ootrent
Uniee 2. Mibeere
gemmin gemma 2-2844
2218 www.ttttrrtttto 2007
00-41
-6.4
Uieaa
-06-
30 35
60
43
50
35
Sytthetc Besrage Question Langth Takers
(a) Question token count by generating model. Most models are more verbose than humans, but certain models take this an extreme, producing very long questions.
(b) Scatterplot documenting evaluation accuracy over- performance on long questions. Models like gpt-1.1-mini plat well above the trend line.
Figure 6: Impact of question length evaluation accuracy over-performance.
5.4 Benchmark question diversity
During synthetic question generation, the explicit topic extraction step provides a backstop against degenerate questions which only focus on the most obvious elements of a context chunk. To compare the variety and coverage of the generated questions compute the average cosine similarity between all question pairs (excluding self pair) over the generated synthctic dataset. While most generating models produce similar diversity scores closely aligned with the human questions, some highly capable models (such as gpt-4.1) show comparatively worse question diversity while performing well in clarity, correctness, and groundedness. Figure T0 in Appendix D) shows the question diversity per generating model along with the original question diversity.
Benchmark Evaluation Accuracy 0.553, 0.743)
Evallu tion Models
Llamed Mevertik
0.9
911-4.1
0.8
24nma3-12b6
germma-3 3-tm-----t---
Owerz25 2.5 1-ir-oo
Lamms 31-a7-inttuue
Original Accuracy
Upmo
Datasets
0.5
HetporiA meoa www.sureeeeusseersttttr
flchrog hrtavsss mrea. Thva0k-www
0.4
puamee, 22
fastrag musique
$ec,00
5.5 Benchmark model accuracy correlation
Beehoan bosin
Natres
Ideally, we want synthetic benchmarks to correlate well with the evaluation accuracy measured by hu- man curated questions. This ends up being truc for some generating models, but not all. Figure 7 shows the evaluation accuracy scatterplot comparing the reformatted human questions against the synthetic generated benchmark questions for the five bolded ensemble models in Table 2 This ensemble produces
0.4
0.8
Synthedic Accuracy
Figure 7: Benchmark evaluation accuracy correlation generated by model ensemble demonstrating reasonable correlation between human and synthetic evaluation. Pearson accuracy correlation 0.743.
8

---

Published in Transactions on Machine Learning Research (Jan/2026)
Pearson accuracy correlation of 0.743 between the evaluation accuracies for synthetic and human benchmarks (Figure 15b Each plot marker is the average evaluation accuracy for given model-dataset combination. While this model ensemble produces reasonable overall accuracy correlation, occasionally specific LMs gener- ate undesirable benchmarks. Figure 14 in Appendix showcases a failure where the synthetic benchmark generaled by gpt-4.1 1-mini contains overly easy questions where all models perform near 100% evaluation accuracy (Pearson accuracy correlation 0.276).
5.6 Benchmark model rank correlation
While direct. Pearson correlation between human and synthetic benchmark evaluation accuracy is desirable, evaluations are often used to compare models. Therefore, a more important metric is the model rank order stability measured by Spearman rank correlation. Figure plots the evaluation model performance across all datasets, comparing the syuthetic and human accuracy Spearman ranking correlation of the evaluation model capability. Pearson accuracy correlation translates into Spearman ranking correlation and relative model order stability. Similarly, the overly casy gpt-4.1-mini synthetic benchmark in Figure 14 produces an odd rank ordering (Spearman correlation 0.753) in Figure [5b with very flat accuracy curve. While ensembling various question generating LMs together will partially mitigate this effect, it is worth validating the set of generating models against the datasets in this paper to confirm the ensemble is not negatively impacting overall synthetic benchmark quality. The tendency of certain generating models to produce overly long, detailed questions that LMS find easier to answer may be the cause of the observed benchmark over-performance.
5.7 Multiple-choice vs open-ended questions
During this study, both multiple choice and open-ended questions were evaluated. None of our findings change whether using multiple choice questions for simplified evaluation benehmark grading, or open-ended questions to produce harder evaluation benchmarks where knowledge synthesis is required. The models that are good at ereating synthotic multiple choice questions are also good at creating open ended questions. The models which create overly easy questions by including too much detail (like phi-4) do so for both open ended and MCQ, with the long question problem becoming even more pronounced for open ended questions
shown in Figure 11 in Appendix D
Table presents Pearson benchmark evaluation accuracy correlation and Spearman rank order correlation between the human and synthetic evaluations. Llama-4-Maverick was unexpectedly dominant for multiple- choice in these final Table 5 results, even outperforming the ensemble of models. Similarly gemma-33228 --2bbit over-perfone over-performed the ensemble for open-ended questions.
Table 3: Pearson accuracy and Spearman rank correlation between synthetic and human benchmarks
Metric
Pearson Accuracy Correlation
Spearman Rank Correlation
Generating Model
Multiple Choice
Open Ended
Multiple Choice
Open Ended
Ensemble
0.7434
0.7238
0.9066
0.6099
1lama-3.2-3B-Inns. Instrrcct
0.5444
0.7288
0.2527
0.4176
1anma--.3-70B-Inttreee
0.6807
0.6992
0.8626
0.4725
1lama-3.1-8B-1 Instsnrrettt
0.5484
0.7402
0.8901
0.4780
Llama-4-Maverick
0.8232
0.7833
0.9231
0.6758
gemma-3-12b-it
0.6879
0,6521
0. 0.7363
0.5000
gemma-3-27b-it
0.7325
0,8192
0,8846
0,8516
gemma-3-4b-it
0.6495
0.7771
0.5549
0.7308
gpt-4.1
0.5784
0.5071
0.7967
0.4341
gpt-4.1-mini
0.2759
0,2505
0.7527
0.4560
04-mini
0.6687
0.5290
0.8187
0.5495
phi-4
0.3489
0.3936
0.5989
0.4725
Instruct
0.5901
0.6018
0.4286
0.6099
Qwen3-235B- A2B-----2BBB--
122--nsstruct2500
0.6084
0.6017
0.7857
0.5495
9

---

Published in Transactions on Machine Learning Research (Jan/2026)
5.8 Case study on academic papers
To demonstrate our approach in a more realistic setting, we selected three documents (two of which are post- LM training knowlodge cutoff): "Current Solutions and Future Trends for Robotic Prosthetic Hands" (Mendez eet al., 2021; "Recent Advances in Large Language Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation" (Chen et al. 2025), and the recently released " "America's 's AI Action Plan" (AiP). The documents were chunked into roughly 30 chunks each. "Current Solutions" averaged 7.7 topics per context for total of 210 questions. "Recent Advances" averaged 6.9 topics per context for total of 194 questions. "AT Action Plan" produced a total of 235 questions. For MCQ, the synthetic benchmarks' Spearman rank correlation is 0,911, for OE the rank correlation is 0.969. These correlations reflect the trend of overall model capability, as both source documents are from post training cutoff, so models forced to puzzle out what the likely answers are. Unlike earlier results where the correlation was a comparison to the human curated questions, these correlations only compare the LM benchmark performance between the two academic papers. Under MCQ the gemma-3 models perform middle of the pack on these two benchmarks, but under OE questions the 12b and 27b outperform all other models. See Figure 16 in Appendix H for full rank ordering and examples of the generated questions.
5.9 Replicating a subset of MMLU-Pro
The datascts used herein to validate the generative benchmarking approach were never designed to eval- uate general LM knowledge. To alleviate this weak- ness and demonstrate the capability of this genera- tive benchmark creation approach, we replicated subset of the MMT.U-rr benchmark (Wang et al.) 2024b) using creative commons textbooks from Open- Stax (Stafford& & Flatley, 2018). Six of the categories from MML.U-Pro were replicated by using a related textbook as the grounding context: biology CClar
Benchmark Evaluation Accuracy (R' 0.608, 0.780)
Eveluation Models
Owa13-1.71
Owers 255t
0.8
gersma 3-52bit
prrmma-3-18 3-12-2
gemma---277 276--
Lleme-12- 12-3--1----
Uamo 3.3 708 Iegprctt
Original Accuracy
0.6
p014
951-41
0.5
a21- 300
et al. 2018), chemistry (Flowers et al. 2019), com-
puter science (Franchitti] (2024). math (Abramson 2021 physics (Urone & Hinrichs 2022 and psychol- ogy (Spielman ct al.) (020)). We used gpt-oss-120b to construct the generative benchmarks. Each text- book was converted into markdown, chunked into 4k token blocks, and ingested into the generative benchmarking pipeline. The resulting generative benchmarks were evaluated using the full suite of models in Table [ Figure g demonstrates the cor- relation between the generative replication of each MMT.U-pro subset and the original human-curated benchmark results.
0.4
Dalasels
0.2
metigennteeey monta pia chemiary
minia pro, pompute
meti jon math
0.1
mett.pn strplca
poo poychology
0.2
0.3
0.6
0.9
Synthetic Accuracy
Figure 8: Benchmark evaluation accuracy correlation generated by model ensemble demonstrating reasonable correlation between human and synthetic evaluation. Pearson accuracy correlatton 0.78.
The generative replication of MMLU-pro benchmark produces a Pearson accuracy correlation of 0.78 with the original benchmark (Spcarman rank correlation of 0.83). Given that the generated questions were creatod using only relevant boxt where question topies do not align, the resulting level of agreement between human curated and grounded generative benchmarking is surprising. Additionally, the accuracy aligns well with underlying model capability as measured by the human-curated MML.U-Pro benchmark.
To validate the quality of the generative questions, the authors manually reviewed 120 questions (20 each randomly selected from each MMLU-pro subset). This evaluation probed two elements, the question/answe pair correctness and grounding in the source document context. We found that 111 of 120 questions had completely accurate generative question answer pairs, meaning the question was well formed and the answer correct. This yields noise floor for generative benchmarking approach of approximately 7.5%. 104 of
10

---

Published in Transactions on Machine Learning Research (Jan/2026)
120 questions had fully supported grounding in the document. A strong majority of the grounding failures fell into two categories: 1) the question writing model adding relevant correct information that was missing from the document, or 2) the generative question referencing the document and not being fully independent.
An example high quality generative question from the physics subset is as shown:
Generative Question: Why can pion exchanged between proton and neutron transmit the strong nuclear force without being directly observable, and how does its fleeting existence satisfy AE At 2 h/2 while leaving the total mvergymmmentuun of the nucleon system unchanged?
Generative Answer:
Because the pion exists only for time At S 4/(2A)) (with AE mc+), allowing temporary energy borrowing that is repaid when the pion is reabsorbed, the muclcon system's 's total energy momentum stays conserved, rendering the pion virtual and unobservable.
Human Evaluation:
is Correct and Supported by the grounding context.
We also provide an example of a question where the system failed to produce correct answer:
Generative Question: Based the descriptions provided, what the chemical compositions of the protective passivation films that develop the Statue of Liberty's copper surface and stainless steel, how does the text characterize their adherence underlying metal, and does mention any self-healing capability under cyclic wet-dry exposure?
Generative Answer: copper patina consists of copper oxides transformed by SOs, CO2, and H₂O into a blue-green protective film, stainless steel's passivation layer is chromium oxide, both are described as adherent protoctive layers, and the text. does not address self-healing under wet-dry conditions.
Human Evaluation: answer is Incorrect and Unsupported because question relies on information present the context. Additionally, the answer formulation changed the context text from "The chromium tends to collect near the surface, where it corrodes and forms a passivating an oxide layer that protects the iron" to the wrong answer "chromium oxide".
A more detailed analysis of the human- question set is provided in Appendix B The MMLU-pro replication used gpt-oss-120b for question generation. is possible that using a stronger frontier model would reduce the generation and grounding error rate observed.
5.10 Limitations
This work relies on simple and possibly non-optimal document chunking approach. Extensions which cluster related topics or provide more fine-grained control over the information present in each chunk could improve the granularity and specificity of the generated benchmark. Including information found in tables, charts, and images would provide richer information sources that benchmark creation may take advantage of. Additionally, improving the nuance and difficulty of the questions signíficantly improves LM capability measurement. This likely requires careful management of the context information provided to the generating LM. This approach is not yet suitable for evaluating the absolute frontier of model capability, but many applications rely on weaker models with lower inference costs. Finally, except for the subset of MML.U-Pro replicated using Creative Commons textbooks, the datasets used in this study to validate correlation with human written questions are all NLP extractive question answering type dalasets. This limits the quality of the synthetic benchmarks as the grounding documents are fairly simplistic, especially compared to the two academic papers used in Scction 5.8 Only the PubMedQA and sec_qa datascts come close to the topic diversity and complexity of the example academic papers. All other validation datasets rely upon more general factual knowledge.
6 Conclusion
We introduce a methodology for creating synthelic benchmarks to evaluate language model (LM) capabilities grounded in high-quality document corpora selected by practitioners. This ensures the generated benchmarks are directly relevant their specific domains and interests. This approach extends general-purpose human curated LM benchmarking into specialized domains, allowing for the identification of strengths and weaknesses that may not be apparent in broader evaluations.
11

---

Published in Transactions on Machine Learning Research (Jan/2026)
The majority of the invalid question/answer pairs stemming from either missing information grounding failures could be rectified by an adversarial verifier running within the question generation loop. In future work we plan to explore this research direction both for generative benchmarking and dynamic context grounding.
Our method supports automatic generation both multiple-choice questions (optimizing for ease grading) and open-ended questions (optimizing for diagnostic relevance), allowing users to choose based on their evaluation goals. Validation against human human-curated NLP dataset achieves ensemble Spearman rank correlation of 0.91 and benchmark Pearson accuracy correlation of 0.74 between synthetic and human curated questions, suggesting that our approach is consistent with human-generates question/answer answer benchmarks. Additionally, we identify a potential synthetic data generation failure mode: very long questions can artificially inflate model performance, likely by overspecifying information related to the answer. This methodology enables the creation of targeted, relevant benchmark evaluations facilitating deeper exploration into the domain specific capabilities and limitations of Language Models deployed for complex knowledge work.
12

---

Published in Transactions on Machine Learning Research (Jan/2026)
References
America's ai action plan.
URL
Americas - ----Aiiton-aaaa Plaoonn--aa........ pdf
The Ilama 4 herd: The beginning of a new era natively multimodal Al innovation. URL https::/ai. com/
blog/ 4 -inelliiieeddd--llleeeeeeeeee
Hugging Face luggingface.co.
2024. [Accessdd 2024-10-28).
Llama 3.2: Revolutionizing edge Al and vision with open, customizable models. https://ai.mett: contta/ccmmbog
1-ama-3-2--ae
rrrtssoon-egge--obll-dddecee.ee 2024. [Accessed 2024-10-28)
Openai models documentation, 2025. URL https://platform: http://ppaaatromoopaa..... www//ocssmooeell..
Marah Abdin, Jyotí Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint (t.::2/12.090000 2024.
Jay Abramson, Algebra and Trignometery 2e. Open Stax, 2021.
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. Synthetic QA corpora generation with roundtrip consistency. arXin preprint 1rXi::106.000 054 16, 2019.
Christoph Auer, Maksym Lysak, Ahmed Nassar, Michele Dolfi, Nikolaos Livathinos, Panos Vagenas, Cesar Berrospi Ramis, Matteo Omenetti, Fabian Lindlbauer, Kasper Dinkla, al. Docling technical report. ar preprind 2rXiv:2408. 00666999809669, 2024.
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. Benchmarking foundation models with aanguage-modelas--ee axaaaeimmiieee Advances in Nearal Information Processing Systems, 36:78142 78167, 2023.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms mareo: A human generated machine reading comprehension dataset. Xiviv preprint Xiv::1611..092 1.0692,, 2016.
Angels Balaguer, Vinamra Benara, Renato Luiz de Freitas Cunha, Todd Hendry, Daniel Holstein, Jemnifer Marsman, Nick Mecklenburg, Sara Malvar, Leonardo O Nunes, Rafael Padilha, et al. RAG vs fine-tuning: Pipelines, tradeoffs, and case study on agriculture. arXiv preprint 2r0:2224 084006,866668 2024.
Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. Artifacts abduction: How do LLMs answer multiple-choice questions without the question? Association for Computational Linguistics, 2024.
Nishant Balepur, Rachel Rudinger, and Jordan Lee Boyd-Graber. Which of these best describes multiple choice ovaluation with LLMs? a) lorced b) flawed c) fixable d) all of the above. arXii preprint NNw:25022.4422, 2025.
Raymond Bernard, Shaina Raza, Subhabrata Das, and Rahul Murugan. Equator: A deterministic framework for evaluating LLM reasoning with open-ended questions. ar preprind arX:::201..0057, 2024.
Meghana Moorthy Bhat, Rui Meng, Ye Liu, Yingbo Zhou, and Semih Yavuz. Investigating answerability of LLMs for long-form question answering. arXiv preprint ar
2023.
Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, et al. Recent advances in large language model benchmarks against data contamination: From static to dynamic evaluation. arXiv preprint arXix:2502. 17521, 2025.
Cheng- Han Chiang and Hung yi Lee, Can large language models be an alternative to human evaluations?, 2023. URL https://arziz: wwwaaaiivv.rggaaae www.wwa/////.oooooooo//o......aa at//aa/......///e////////.e.aa 01937
Christopher Clark, Kenton Lee, Ming-Wo Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. XrXiv preprint 1905. 10044, 2019.
Mary Anna Clark, Matthew Douglas, and Jung Choi. Biology 2e. Open Stax, 2018.
13

---

Published in Transactions on Machine Learning Research (Jan/2026)
Jonathan Cook, Tim Rocktäschel, Jakob Foerster, Dennis Aumiller, and Alex Wang. Ticking all the boxes: Generaled checklists improve Ilm evaluation and generalion, 2024. URL hatts:///axiiv http:///aariiv.oraabba////as//// htt////b/2//110 03608
Sugyeong Eo, Hyeonseok Moon, Jinsung Kim, Yuna Hur, Jeongwook Kim, Songeun Lee, Changwoo Chun, Sungsoo Park, and Heuiseok Lim. Towards diverse and effective question-answer pair generation from children storybooks. arXiv preprind arXiv:2306.06600 2023.
Paul Flowers, Klaus Theopold, Richard Langley, and William R. Robinson. Chemistry 2e. Open Stax, 2019.
Jean-Claude Franchitti. Introduction to Computer Science. Open Slax, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The Ilama herd of models. ar preprinl MXXx:2407.217665 2024.
Rujun Han, Yuhao Zhang, Peng Qi, Yumo Xu, Jenyuan Wang, Lan Liu, William Yang Wang, Bonan Min, and Vittorio Castelli. RAG-QA arena: Evaluating domain robustness for long-form retrieval augmenttdd question answering. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirienl Methods in Natural Language Processing, pp- 4354-4374, Miami, Florida, USA, November 2024. Association for Computational Linguisties, doi: 118653/11120244 wwwp://.....mmmmmmmm//mm conmp-mminn22 URL https://aclantthllogy org/2024, emnlp-main. /49/
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt, Measuring massive multitask language understanding. CoRR, http//009.0330000 2020. URL https://arxivv org/ abs/2009. 033000
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation reasoning steps. arXiv preprint orXiv:2011.0106/ 2020.
Jeffrey 1p. LLM Evaluation Metrics: The Ultimate LLM Evaluation Guide. attps://wwid http:///wwccoccdeettBaaaeeee confident-aiii httpiiddettaii..omllooooooooo blog/
2024. [Accessed 2024-10-28).
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. Kentaro Imui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empárical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pp. 2567 2577, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/DIL 1259. URL https://aclantthooogg Org/DD1--259)
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. ar e-prints, art. arXiv:1705.03551, 2017.
Ashwin Kalyan, Abhinav Kumar, Arjun Chandrasekaran, Ashish Sabharwal, and Peter Clark. How much coffee was consumed during emnlp 2019? fermi problems: A new reasoning challenge for ai. orXiv preprint orXiv:2110. 10011410, 2021.
Ehsan Kamalloo, Nouha Dziri, Charles LA Clarke, and Davood Rafiei. Evaluating open-domain question answering in the era of large language models. Association for Computational Linguistics, 2023.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguislics, 6:317-328 2018.
Tom Kwiatkowski, Jennimaria. Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: beuchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-666 2019.
Tianle Li, Wei-Lin Chiung, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks; Arena-hate and benchbuilder pipeline, 2024. URL https://arziv org/ abs/ abs/2406. 11939
Xiang Li, Yunshi Lan, and Chao Yang. Treveval: Benchmark- evaluation of large language models through tree planning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 24485 24493,, 2025a.
14

---

Published in Transactions on Machine Learning Research (Jan/2026)
Xiang Lisa Li, Farzaan Kaiyom, Evan Zheran Liu, Yifan Mai, Percy Liang, and Tatsunori Hashimoto. Autobencher: Towards declarative benchmark construction, 2025b. URL atttps//aarzis http:iiworgg/bb//////ee 240bbb////000055111
Zefang Liu. Secqa: A concise question-anseee answering dataset for evaluating large language models in computer security, arXiv preprint or Xiv:22811 15838, 2023.
John MacFarlane. Pandoc universal document converter. URL https://pandoc. org/
Vincent Mendez, Francesco lberite, Solaiman Shokur, and Silvestro Micera. Current solutions and future trends for robotic prosthetic hands. Annual Review of Control, Robotics, and Autonomous Systems, 4(1)):95-662 6627, 2021.
Aidar Myrzakhan, Sondos Mahmoud Bsharat, and Zhiqiang Shen. Open-LIM4Leaderbbard From multi-choice to open-style questions for Ilms evaluation, benchmark, and arena, 2024. URL https://araii. (aaa///24.0075555 2024.
Juhyun Oh, Eunsu Kim, Inha Cha, and Alice Oh. The Generative Al paradox on evaluation: What it can solve, il may not evaluate. Xrviv preprint or arXiv:2402.0620 06204, 2024.
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndonsse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer EI Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan, Discovering language model behaviors with model-written evaluations. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp- 13387-13434, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: (1118855/11//00 htt.://////.....n-aaa... URL https:://claattollllas .erpggg.orrg//oo org/ org/2023. 2022 wwndingsnniia...i. acc.anngsssal..77//.. .847/
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint 1rXi:22501.1449, 2025.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Tryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784 789, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: (.11866/vv1/P1-2111
URL https:://cclatthooogg www.:18--21P201 2124/
Joshua Robiuson, Christopher Michael Rytting, and David Wingate. Leveraging large language models for multiple choice question answering. arXiv preprint Xio:2200 12353, 2022.
Aymeric Roucher. Using judge for an automated and versatile evaluation Hugging Face Open-Source AI judge 2024. [Accessed 2024-10-28).
Cookbook. http://hwgaggaaccccc/////eeeoers
Flena Samuylova. Wrong but useful: an I.L.Mas-a-judge tutorial. https://www.c htt////wwwwideelllllll.... wttplllllaaa.ommaaa blog/ 2024. [Accessed 29-10- 29-10-20
-amn-a---judge-tuuu
Summk Shashidhar, Clémentine Fourrier, Alina Lozovskia, Thomas Wolf, Gokhan Tur, and Dilck Hakkani-Tün Yourbench: Easy custom evaluation sets for everyone. orxiv preprint 0rXiv:2504.01833, 2025.
Kazutoshi Shinoda, Saku Sugawara, and Akiko Aizawa. Improving the robustness of QA models to challenge sets with variational question-answer pair generation. arXiv preprint orX3::8004.0880 2020.
Tan Soboroff. Don't use L.LSs to make relevance judgments. Information retrieval research journal, 1(1):10 5-1195, 2025.
Rose M. Spielnan, William J. Jenkins, and Marilyu D. Lovelt. Psychology 2e. Open Stax, 2020.
Daniel Stafford and Robert Flatley. Openstax. The Charleston Aduisor, 20(1):48 51, 2018.
Annalisa Szymanski, Noah Ziems, Heather A Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald A Metoyer. Limitations of the ilm-as-a-judge approach for evaluating Ilm outputs in expert knowledge tasks. In Proceedings of the 30th International Conference Intelligent User Interfaces, pp- 952- 022-966, 2025.
15

---

Published in Transactions on Machine Learning Research (Jan/2026)
Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjes Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation and syuthesis: A survey. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp- 930-957, Miami, Florida, USA, November 2024. Association for Computational Tinguistics. doi: 10. 10.18653/vl 36653/11/2221.mm// /2221..mmmmp-mma.5..... URL https://aclaattoologg wrwp:/orr////////// ott.:poo....//02.........m..aaa. emnlp-main.5
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprinl arX3::2033. 19786, 2025.
Qwen Team. Qwen3 technical report, 2025. URL https: htw.:aawwwwwvrrga//a/aaaaaaaa/// 09388
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Långuistics, 10:539 554, 2022.
UK Al Security Institute. Inspect Al: Framework for Large Language Model Evaluations. https://inspect ai iasetty-nstitute. org. uk/ 2024. URL htttp:://gitubb... com
ai
Paul Peter Urone and Roger Hinrichs. Phaysics 2e. Open Stax, 2022.
Asahi Ushio, Fernando Alva-Mancheeg, and Jose Camacho-Collados. An empirical comparison of LM-based question and answer generation methods. arXiv preprint arXiu:2055110000 2023.
Haochun Wang, Sendong Zhao, Zewen Qiang, Bing Qin, and Ting Liu. Beyond the answers: Reviewing the rationality of multiple choice question answering for the evaluation of large language models. ar e-prints, pp. arXiv-2402, 2024a.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266-95290 2024b.
Cameron Wolfe. Using LLMs for Evaluation. httpa://caamerorvooeee....ba http:///wwww.eeebbbbaaaccaa///ea judge 2024. [Accessed 2021-10-28).
Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. Expertprompting: Instructing large language models be distinguished experts. arXiv preprint arXiv:2305. 14688, 2023.
Vikas Yadav, Hyuk joon Kwon, Vijay Srinivasan, and Hongxia Jin. Explicit over implict: Explicit diversity conditions for effective question answer generation. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation C.REC-COLING 2024), pp. 6876668686 -8726 Torino, Italia, May 2024.
ELRA and ICCL. URL https::/ org.hhhooooooooooooooooooo wrg/ 2024 1rec- mano. mmai..601/
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arxiv preprint orX:o 12. 15115, 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning, HolpoiQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
Xingdi Yuan, Tong Wang, Yen-Hsiang Wang, Emery Fine, Rania Abdelghani, Pauline Lucas, Hélène Sauzéon, and Pierre-Yves Oudeyer. Selecting better samples from pre-trained llus: A case study question generation. arx preprint 2xi::2209.11000 2022.
Cheng Zhang and Jie Wang, Seg-see--equeece learning for generating question question-answer answer pairs. ar preprint arXiv:2210. 11608, 2022.
Lianmin Zheng, Wei-Lin Chiang, Ving Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Lin, Zhuohan Lí, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chathot arena. Advances in Neural Information Processing Systems, 36:46595-44622 2023.
Jincheng Zhou. Qog: Question and options generation based on language model. arXiv preprint orXiv:2406. 12981, 2021.
16

---

Published in Transactions on Machine Learning Research (Jan/2026)
A Expanded related works
This approach draws inspiration from NP- SPPcompleteness, where easy to verify solutions may be hard to generate. The LM under evaluation has a harder job than the LM that converts existing documents into appropriately formatted test questions with accompanying answers. The evaluation LM must produce the correct information from just the question; by contrast, the benchmark creation LM has the source material. In other words, given the grounding paragraph, a language model should be much more capable at converting that textual information into a. multiple choice or short answer question than it would be at zero-shot answering the very same question. This task difficulty discrepancy provides hope that this approach will produce viable results long term. Asking an LM to generate diagnostically useful evaluation benchmarks drawn only from information stored in model weights is difficult, but conversion of information from a high quality source into test questions attainable.
This work draws on the rich history of language model evaluation, where LMs are already being used to judge and score the results of human curated benchmarks. When scoring outputs of an LM producing multiple choice answer, it is often useful to have another LM reason about and extract the specified C,D] value, instead of hoping the model under evaluation only produced the exact match answer letter (Samuylova 2024 Roucher 2024; Wolfe, 2024 This procoss bocomes even more important when evaluating short answers from LMs instead of multiple choice responses, or when scoring the responses and providing reasoning as to why an answer was correct or incorrect (Ip) 2024 Oh et al., 2024 Cook et al.) 2024). This idea of automating question answer generation has been approached before. The most similar contemporary work is Your by Shashidhar et al. (Shashidhar et al. 2025 which leverages the convergent idea of grounding documents supporting LM benchmark generation, demonstrating the synthetic replication of MMLU. Additionally, AutoBencher by Li et al. explores leveraging synthetic data (supported by privileged information) for model ovaluation with RL to optimize the generated evaluation for discovering gaps in LM capability (Li et al. 2025b). The use of grounding documents or privileged information greatly improves synthetic data benchmark utility, as the evaluation methodology no longer relies on simply asking an LM for questions (and answers).
Question from context and answer Early Question Answer Generation (QAG) approaches attempted to solve the question creation from context + correct answer input. These approaches include improvements in selecting better LM samples (Yuan et al. 2022), increasing sample diversity over distributional shifts (Shinoda
ct al. 2020), explorations of interrogative prompting against the context (Eo et al. 2023) J and evaluating round trip information consistency (Alberti et al. 2019 These early methods sometimes fine-tune generation models to improve task performance, though largely that is only done with smaller models (Ushio et al., 2023 Zhou 2024h. Some work focused on fine-tuning specialty models to improve option generation for multiple-choice questions (Zhou 2024). Other early work during the shift to general purpose LMs involved leveraging those language models directly as data generation sources. This took the form of prompting the model to write questions (without grounding context) (Bai et al. 2023 Tan et al. 2024 Perez et al.) 2023 Zhang& & Wang, 2022), or configuring an LMMas-evaluutorr/iniie htt.lluuooooriiteerrattoooo
dynamically lly interacting with the LM under evaluation to explore capability in a reference-free manner (Bai et al.) 2023). These approaches attempt to address the dataset leakage problem where benchmarks, once published, become part of the training dataset for the next round of models. As the capability of LMs increases over time, setting up adversarial LM-evaluator vs LM-evaluatee systems is becoming more feasible (Li et al.] 2025a).. Nevertheless, extending those systems such that the LM-evaluator has grounding documents pull from should strengthen the performance.
Scaffolding question generation with grounding documents With the rise of general purpose token- tumblers like ChatGPT, longer and more complex grounding documents become increasingly useful. There is no longer need to constrain synthetic benchmark crcation to a short paragraph context + correct answer input; the LM can operate on larger and more complex document chunks, with less pre-processing required to make the input context data useful. Contemporaneous work YourBench (Shashidhar et al.) 2025) and AutoBencher (Li ct al.) (0255) can leverage lightly-filtered web-pages as the context,, chunking the data into rough paragraphs for ingestion into the question generation and grounding system. The SecQA benchmark was created partly through synthetic data grounded in an open-source computer security
17

---

Published in Transactions on Machine Learning Research (Jan/2026)
Similarly (though much carlier), the PubMedQA benchmark was created through human and synthetic data generation grounded in biomedical research paper abstracts (Jin et al. 2019 Some research demonstrates that operating purely on document summaries produces inferior generated questions compared to leveraging the full document (Bhat ct al.) 2023 With modern long-context LMs, summarization is becoming less of a problem, as models can manipulate significant volumes of text within their attention context. Synthetic data question generation systems have seen rapid improvement. This is partly driven by capability improvements in the LMs powering synthetic data generation. However, there has also been exploration into improvements in prompting for diverse question generation (Yadav et al., 2024) and breadth of topic generation for chatbot arena style questions (Li et al. 2024). Enhancing retrieval augmented generation (RAG) systems for domain specific applications has also used synthetic data question generalion to compare baseline RAG to fine-tuning (Balauuer et al. 2024
textbook Liu 2023
LM as feature extractors There are many aspects of text that one might want to quantify, however imprecisely. However, devoting the required copious human annotation hours is just impractical. Current LMs end up being acceptable feature extractors, when provided detailed prompts. Chiang and Lee demonstrate that LMs can substitute for human evaluators in asscssing text quality with strong correlation in the results (Chiang & yi Lee 2023). Similarly, Han et al. leverage LM evaluators for judging quality and find that results correlate highly with human judgments (Han et al. 2024). The use of LMs to extract features from text is not to dissimilar from using the LM-as-aJJudge (Xu et al. 2023) pattern for large scale open ended capability evaluations (Xu et al.) 2023 Zhenng et al. 2023 Szymanski et al. 2025 The LM is capable of grading open ended evaluation benchmark questions for correctness; therefore determining features other than answer correctness from a text sample is viable. However, there are limits on what LM-as-a-Judge is capable of ground truthing, especially in fields like information retrieval (Soboroll 2025 Szymanski et al.] 2025) where LMs mnder-perform on more complex relevance judgments. Relying on LMs to extract properties like answer correctness given sufficient scaffolding and context is well-used pattern.
Multiple choice vs open ended When constructing knowledge or fact based evaluation benchmarks, there are two broad categorics of answers: multiple choice and open ended. Open-ended questions provide the model less information and require synthesis [Balepur et al. 2025), but at the cost of needing another LM to evaluate the cosrect of the results (LMMas-a-Judge (Xu et al. 2023) ) (Myrzakhan et al. 2024). Multiple choice question (MCQ) results can be parsed and graded more simply. Modern evaluation frameworks like Inspect-AI MUK AI Security Institute 2024) enable shuffling answer options and contain default prompts for both open-ended and MCQ evaluations. With MCQ, having the answers visible to the LM can artificially inflate performance (Robinson et al.] 2202)), even going as far as the model identifying the correct answers without seeing the question (Balepur el. al. 2024). Both open-ended and multiple choice questions have weaknesses to question reformatting and phrasing where semantically identical questions produce different answers (Wang et al.) 2024a). The simplest form of open-ended evaluation relies on judge LM, though other frameworks have been proposed which attempt to improve the precision of open-ended evaluations (Bernard
et al. 2024 Kamalloo et al. 2023
B Replication of MMLU-pro
Because LM-as-a-judge was used throughout our experiments, human evaluation of a subsampled set of questions generated to replicate MMLU-pro was performed in order to validate results. The authors took a random 20 questions from each evaluated subset of MML.U-pro (biology, chemistry, computer science, math, physics, and psychology) and manually graded the question answer pair correctness and grounding against the source context from textbooks. Each question-answer pair was labeled with each of correct/ cincor//ncoorect and supported/unsuppoor unsupported by the context. Answer correctness was often negatively impacted by grounding failures, such that a question was missing information from the context required to be correctly answered. Per-domain details are as follows:
Physics: 19/20 questions correct, with the incorrect Question 11 incorrectly canceling the torque of a pivot. 14/20 questions were fully supported by the grounding context; of the remaining 6, the most common failure was the question-writing model relying on model parameter knowledge fill in
18

---

Published in Transactions on Machine Learning Research (Jan/2026)
common knowledge which was not explicitly included in the grounding context. See Appendix B.1 for a per-question breakdown of the grounding failures in the physics category.
20/20 question/answer pairs correct, and 20/20 with correct and valid grounding in the source contexts.
Biology:
18/20 question /antwer answer pairs correct, 15/20 questions were fully grounded in the context. Both incorrect answers failed due to the question-writing model not including all required information in the question and or modifying minor details from the context. For example:
Chemistry:
Question 13: Based the descriptions provided, what are the chemical compositions of the protective passivation films that develop on Statue of Liberty's copper surface and stainless steel, how does the characterize their adherence the underlying metal, and does it mention any self-healing capability under cyclic wet-dry exposure?
The copper patina consists of copper oxides transformed by 50,, COs, and H2O into blue- protective film, stainless steel's passivation layer is chromium oxide, both are described as adherent protective layers, and the text does not address self-healing under wel-dry conditions.
Answer:
Human Evaluation: Incorrect and Unsupported because the question relies on information present the Additionally, the answer formulation changed the context from "The chromium tends collect the surface, where it corrodes and forms passivating an oxide layer protects the iron" the wrong answer of "chromium oxide".
The five Chemistry grounding failures resulted from the question writing model added additional correct information from its parameter knowledge to flush out the question. For example:
Question 7: Using AG AH- -TAS, predict how the spontameity of NaCI melting changes when the temperature is raised from 500 to 700 "C, and explain why the entropy contribution becomes more decisive at higher temperature.
Answer: Nonspontaneous at 500 OC (AG > 0) and spontaneous at 700 'C (AG < because the larger TAS term at the higher temperature outweighs the positive AH. Correct but Unsupported the grounding context its missing the AG
Human Evaluation:
AH TAS equation.
Computer science: 18/20 question/answer pairs correct, and 18/20 with correct and valid grounding in the source contexts. Both incorrect question/ /answer pairs were judged as failures because the question could not stand independent of the grounding context, as the question referenced information in the grounding context. These two grounding failures caused the questions to be labeled incorrect.
Math: 18/20 qucstion/anns /answer pairs correct, and 18/20 with correct and valid grounding in the source contexts. One of the two incorrect questions the result of the question not restating required information from the context. The other incorrect question was more interesting, as the question-writing model wrote an answer that is open-ended and non-umique:
Coustruct rational equation that, after multiplying by its least common denominator, yields algebraic solution which is exactly excluded value, and explain why this equation has real solution despite the apparent solution.
Question 7:
Answer: (x - 1)/(a)//())))) 1 2; the only solution x is excluded because zeroes the denominator, so there is no real solution.
Human Evaluation: answer is Incorrect and Supported by the grounding context, because the provided answer is one possible answer of many.
Both math subset grounding failures were result of the question writing model adding additional correct information from its parameter knowledge to flesh out the question.
Psychology: 18/20 question /answer pairs correct, and 19/20 with correct and valid grounding in the source contexts.
Out of the 120 human evaluated question/answer pairs, 111 were correct. This gives the generative benchmarking an approximate 7.5% noise floor. 104 of 120 questions had fully supported grounding in the document. The vast majority of the grounding failures fell into two categories: 1) the question writing
19

---

Published in Transactions on Machine Learning Research (Jan/2026)
model adding relevant correct information that was missing from the document, or 2) the generative question referencing the document and not being fully independent.
B.1 Detailed Human Evaluation Results: MMLU-pro Physics
In order to provide detailed insight into the questions and answers involved in the human evaluation, we reproduce the set of 20 mamually evaluated physics questions here, along with the human judgment about question/answer quality.
dryor (or similar bathroom appliance) while a bathtub basin filled with water creates signifieent shock hazard, focusing on how the water's conductivity influences the formation of a ground-fault current path that can pass through person.
Question 1:
Explain why operating
Answer: Berause water provides low-resstance path that complete fault circuit, allowing current from the appliance flow through a person in contact with the water. Human Evaluation: Correct and Supported
How does the difference in attachment distance of the back muscles compared with the quadriceps alter their mechanical advantage during a lift, and why does using quadriceps to raise the load markedly reduce the compressive force on the vertebrae as illustrated in the example?
Question 2:
Answer: Because back muscles have short lever arm (low mechanical advantage) while quadriceps have longer levor arm (high mechanical advantage), using the legs shifts the load to stronger joints and markedly lowers the force the spine.
Human Evaluation: answer Correct and Supported by the grounding context.
In circuit where switch mistakeuly wired in parallel with resistive load, why does the load stay illuminated when the switch is open but go dark when the switch is closed, using Kirchhoff's current law and the notion of a short circuit to justify the behavior?
Question 3:
Answer: With the switch open, KCL forces all current through the load giving it full voltage; closing the switch creates a short-cireuit branch that shunts the current, leaving near-zero voltage across the load so extinguishes. Human Evaluation: answer Correct and Supported by the grounding context.
Question 4: For the 50.0 high-pressure gas cyliuder described (initial pressure Pt, initial temperature T,), determine (a) the pressure after isochoric cooling to dry-ice temperature Tary: (b) the pressure if 90% of the gas escapes, (c) the temperature
which the cylinder must. be enoled to lower the pressure to atm without any gas loss, and (d) whether such conling is a practical safety strategy.
Answer: After cooling P= P.- P.-(Tary (Taay/TT);: after 90% loss P 0.1P, (aaa: /T);; to reach 1 atm, T == (1 atm/P.) T, a temperature far too low for practical use, making cooling impractical. Human Evaluation: auswer is Correct but Unsupported by the grounding context because its missing the governing relationshipe (ideal gas law / proportionalities)
Question S: Considering the decay processes of neutral and charged pions outlined, which component of an extensive air shower that spreads over several square kilometers retains the greatest share of the primary cosmic-ray energy, and why? Answer: The electromagnetic component, because x decays into photons that dominate the cascade. Human Evaluation: answer is Correct but Unsupported by the grounding context because its missing the cosmic ray shower component energy shares.
Question 6: Starting from the continuity of pressure and particle velocity at an interface between two media, derive the intensity an ultrasound wave and explain the physical significanee the numerator, denominator, and the squaring operation in the resulting expression.
reflection cnefficient
Answer: R = ((22-2)) 2,) ( + 2,]]"; 2, and 2, are the media's 's acoustic impedauces, (Z₂ - 21) reflects mismatch, (2, 21) represents total impedance the interface, and squaring yields the intensity ratio. Human Evaluation: answer is Correct and Supported by the groundiag context.
Question 7: In mixed series-parallel network where other elements, which tolerance extreme for the parallel resistors ylelds the highest total power dissipation, and why does this represent the worst-case overload condition?
of resistors is connected in parallel and that branch is in series with
Answer: When the parallel resistors are their minimum tolerance (loweat resistance). because this minimizes equivalent resistance and maximizes total current and power.
Human Evaluation: answer Correct and Supported by the grounding context.
Question 8: During the inclastic rollision of a moving disk with a stick pivoted at one end, why does the system's is angular momentum about the pivot remain conserved despite external forces, whereas its total linear momentum the laboratory frame does not?
Answer: Because the exterual force passes through the pivot, giving zero torque (conserving angular momentum), but it supplies a net external impulse that changes lincar momentum. Human Evaluation: answer Correct and Supported by the grounding context.
Why can a pion exchanged between proton and a neutron transmit the strong nuclear force without being directly observable, and how does its fleeting existence satisfy AE At 2 N/2 while leaving the total evergy-momentiee of the nucleon system unchanged?
Questlon 9:
Answer: Because the pion exists only for a time At E N/(2 (with AE = me²), allowing a temporary energy borrowing that repaid when the pion is reabsorbed, so the aucleon system's and unbservable.
energy- momentum stays conserved, readering the pion virtual
Human Evaluation: answer Correct and Supported by the grounding context.
When a cyclist applies constant torque to the drivetrain, how does decreasing the wheels' moment of Inertia affect the bicycle's linear acceleration, and what is the underlying reasou? Answer: Linear acceleration increases because lower momeut of inertia gives higher angular acceleration for the torque,
Question 10:
20

---

Published in Transactions on Machine Learning Research (Jan/2026)
reducing the rotational energy share and directing more power into translational motion. Human Evaluation: answer is Correct and Supported by the grounding context.
When bending forward touch the toes while keeping the heels and back against a wall, what torque must hip and knee muscles generate about the beels to keep the system in static equilibriu, and how does selecting the wall contact point as the pivot affect this torque calculation?
Question 11:
Answer: A torque equal to the body's weight multiplied by its horizontal distance from the heels; choosing the wall contact pivot removes this torque because the wall rrccion provides an equal balancing moment.
Human Evaluation: answer Incorrect but the question Supported by the grounding context.
Derive, using the subshell capacity formula 2(21 + 1), why the total number of electrous that can occupy all subshells of shell with principal quantum number equale 2n and the implicit assumptions required for this derivation. Answer: Summing 2(21 + 1) for == 0 to n n--1 yields 2n²; the derivation assumes the allowed Lrange (0 s && - 1), full
Question 12:
m-degoneracy (21 + 1), spin degeneracy of 2, independent full filling of subshells, and neglocts relativistic and elietron-interaction effects.
Human Evaluation: answer is Correct and Supported by the grounding context.
Question 13: Using = v //2,, determine how the relativistic kinctic energy K (7 1)mm² scales approaches G and state the limiting value the classical kinetic energy 3 mvy in the same limit. Answer: Relativistic diverges to infinity, while classical K approaches 4 me? (a finite value).
Human Evaluation: answer is Correct and Unsupported by the grounding context as its missing information about the classical limiting value.
Question 14: Which term describes chaotic systems that nevertheless develop organized, long-lasting structures, and what the example of such structure in Jupiter's atmosphere?
Answer: Stable chaos, exemplifted by the Great Red Spot on Jupiter.
Human Evaluation: answer Correct and Supported by the grounding context.
Derive the decibel expression for cound-intensity level from the fact that human hearing responds logarithmically intensity, and explain why the formula uses factor of for intensity ratios hut factor of 20 when expressed in terms of pressure ratios.
Question 15:
Answer: = 10 (0///// 20 (p/po (the factor arises because intensity & pressure"). Human Evaluation: answer is Correct and Supported by the grounding context.
Question 16: Which analytical method, involving decomposition the initial velucity into horizontal vertical components, is used to determine projectile's range, and what explicit assumption about air resistance does the context provide for this calculation?
Answer: The analytical method that decomposes the initial velocity into horizontal and vertical components, with air resistance assumed zero.
Human Evaluation: answer Correct and Supported by the grounding context but the question is degenerate and directly references the context.
Question How can one determine fraction of human body that remains submerged when floating freshwater versus seawater based solely on the body's overall density, and why does an increase in low-density fat relative higher-density muscle cause that fraction to decrease (L.e., make the person float higher)?
Answer: The submerged fraction equals the body's density divided by the fluid's density [phody Ohhad/Ppwater Pwater in fresh water and Pbody/ Poody//Pocwwaso the Pacowaser in salt water);; increasing low-density fat lowers Poodys thus decreasing that fraction and raising the body higher
Human Evaluation:
answer la Correct but Unsupported by the grounding context because its missing the density fraction
formula.
Question 18: How does defining internal energy microscopically as the total kinetic and potential energy of system's 's atoms and molocules justify the macroscopic first-law relation AU Q - W, and why does this make AU path-indopendent while heat, and work remain path-dependent?
Answer: Because internal energy is a state function determined solely by the micrascopic kinetic and potential energies of the particles, its change equals the net heat added minus the work done (AU Q - W) for any process, making AU patb-independent while Q and W are path-dependent. Human Evaluation: answer Correct and Supported by the grounding context.
Considering Germany's 2020 renewable electricity share of 2.4% and the projected rise in global energy demand, what does achieving the 2030 target of 65% renewahle power imply about trchnical and economic feasihility? Answer: It implies an extremely challenging technical and economic undertaking, requiring about 27-fold increase in renewable share.
Question 19:
Human Evaluation: answer is Correct and Supported by the grounding context but the context table is missing reuewable information which makes the answer corroct only w.r.t. the context. Additionally, the question is degenerate and uninteresting.
MRI scanner's magnetic field generates Hall voltage of 20 mV. What magnetic field strength does this imply, and, accounting for the lead's inductance and the rapld switching MHI gradient fields, the induced current likely to disrupt pacemaker operation?
Question 20: A 10-em pacemaker lead moving 10
perpendieular
Answer: 2 T, and the induced current is negligible, so pacemaker interference is unlikely. Human Evaluation: answer is Correct but Unsupported by the grounding context because its missing information about lead inductance and Mlll switching.
C Impact of question and answer length
While longer more detailed questions (both MC and OE) produce model over-performance (sce Figure G the same trend does not appear in answer length. Figure demonstrates this via scatterplot showing synthetic benchmark accuracy over-performance. Only the ----n. at the very right of the plot is significantly
21

---

Published in Transactions on Machine Learning Research (Jan/2026)
biased above the y 0 expected trend line. The over-performance trend shown for questions in Figure 6b is absent.
Impact of Answer Length Evaluation Overperformance 0.004, 0.066)
Evaluation Accuracy Overperformante (Synthelic Originaal)
Stook 2275-1
0.6
moto lenatiine Maneror
reto lanation slaset
0.4
Oventiwerd 122004222-000 inmunns
remelypla Lend)
-0.2
-0.4
-0.6
10
15
20
25
30
Synthetic Dataset Average Answer Length in Tokens
Figure 9: Impact synthetic MCQ answer options token length on evaluation accuracy over-p verrperformance. LMs under evaluation do not over-perform when answer options are longer in the same way long questions cause perrpeefforaace.
The model gpt-4.1-min. demonstrates the largest wer-performance for both question length and answer length as shown in Figure 6b The following question answer pair was generated by gpt-4.1-mini (37 tokens) using context from Squadv2, highlighting the difference in model verbosity in both question and answers: "question": "Hox does the widespread accessibility and overuse of antibiotics, particularly in livestock, nechanistically contribute to the accelerated developnent bacterial resiatance, and what global health implications does this relationehip entail?",
"choices";
"A": "Overuae increases bacterial exposure to antibiotica, promoting selection for resistant treatment
strains, infactions.".
global rise antimicrobial -resisstta
"B": "Limited antibiotic use livestock reduces bacterial nutations, thereby decreasing resistance and sininizing global health risks.",
"C": AAttibiotic accessibility restricts bacterial gene exchange. preventing resistance protecting global health.".
devalopnent
"D": "Overuse of antibiotics in humans alone causes resistance, while use in livestock haa no public health."),
eignificant impact
"answer"; "A"
Contrast with this question answer pair generated by 110---aaarrrcc----1----------- context and topic. Maverick produces a significantly shorter question (22 tokens) and answers:
using the same
"question": "What is the prinary zechanien which the videspread of antibiotics livestock contributes to the developnent of antibiotic rerisstan bacteria in hunans?",
"choices":
"A"; "By directly tranefenring
chain." "B": "By exerting selective pressure that favors the survival and proliferation of resistant bacterial strains.",
through
"C": "By altering the hunan gut microbiota, thereby reducing its ability to fight off infections
"D": By inducing genetic nutations in hunan pathogens that confer resistance."),
"ansver": "
Synthetic benchmark question diversity
Most LMs produce similar diversity scores that are closely aligned with the human questions Some highly capable models (such gpt-4.1) show comparatively worse question diversity while performing well in clarity,
22

---

Published in Transactions on Machine Learning Research (Jan/2026)
correctness, and groundedness. Figure 10 original question diversity.
shows the question diversity per generating model along with the
Diversity Scores by Generating Model
0.300
Original Questions
0.275
Synthetic Questions
Diversity Score
0.250
0.175
0.150
0.125
M-aa---Maverick Maverick
gumma--3 ntmmaa-327b-it 3-27b-tt
gemma. 3-4b-it
Liama 3, 3.3-700-- instructt
04-mini
Owen2.5 5- 5-70- 10-instruct
Liama in-ttrcct
gut-A 11mini
gemma--34 3-12b-it
Liama- instructt
phi-
gpt-4.1
Owen3 (oon3-3358-222 22--2238---22-nnrrrer Bistt--istuttt25500
Llama-4.
Figure 10: Syuthetic question diversity (pairwise cosine distance) per generating model. Violin bars show the spread across the NLP datasets.
E Multiple-choice vs open-ended questions
The generating models which create overly easy questions by including too much detail (like phi-4 and gpt-4.1-mini) do so for both open ended and multiple choice questions. The long question problem becomes even more pronounced for open ended questions as shown in Figure
question
Gonerating
quastion taken Generating Modal
42
Synthetiz Questions
Systhetic Quedione
Samboo
Uneo- 1-a..ommt lnstroct
Usema Mlvvrick
Useme 2:30 An
Cosoon 120
Opent 2700 4218 Sonnii Avmlo
2.3 7BN intract
Uiesa 45 outfort
Lianaa bat dosrrctt
germas 3.27
Liana Maaeniiik 7P0
gemmat 3-
Dewer 2210 42:4 inunnd 2007
n64
gi-42 aoii
Dowett
(a) MC question token count by generating model.
(b) OE question token count by generating model.
Figure 11: Most generating models have higher token counts than human, but some produce overly long questions. Both multiple-choice and opened-ended questions trend toward longer questions overall.
These overly long synthetic questions result in an pee-ppeformance by smaller models on the generated benchmark shown in Figure I2 We hypothesize this happens because the additional detail in the question provides the LM under evaluation more information to pattern match against text seen in the training phase, thereby increasing the likelihood that the LM retrieves memorized patterns relevant to the question. This effect is seen in Figure 12a for multiple choice, and Figure 12b for open-ended. Both plots in Figure T2 demonstrate LM benchmark evaluation accuracy over- perfpeerformnnc (synthetic - original accuracy).
While similar trends in question length and the perforerfomencc that induces, the evaluation accuracy correlations and model rank order work equally well for open ended and multiple choice questions. Figure 13a
23

---

Published in Transactions on Machine Learning Research (Jan/2026)
Impact Qucstion Length on Evaluation Overgerfomance (8+ 0.241, 0.481)
Impoct Question Length on Evalustion Overperformance (R) 0.095, r- 0.3081
Exakattinn Aocoor S Overg wwwerrononnooo 15 Syrth (1gg00)
0.6
6.4-
0.2
02
0.0
-0.4
-2.8
52
ength
(a) Impact of multiple-choice question token count (b) Impact of open-ended question token count evalu- evaluation accuracy over-performauce, caused by overly ation accuracy over wer-performance, caused by overly easy easy questions.
questions.
Figure 12: Longer synthetic open-ended questions have the same impact evaluation accuracy over- performance MCQ.
plots the correlation in evaluation accuracy between the human reformatted questions and the synthetic generated benchmark for multiple choice, and Figure 13b for open-ended.
Benchmark Evaluation Accuracy (R² 0.553, 0.743)
Benchmark Evaluation Accuracy (R² = 0.524, r= 0.724)
d- Mavarrdd
Uara- Huvrick
0.9
spi-11
Lame- 7004ceatoct
go--.1
2-10B-inabuut
2914
phi-
0.8
perma
Ower2 5- boaaut
Luma- 1-8t-rannn
Original Accurecy
Original
0.4
0.5
Exahrog Zelomuthhspce
Tuartag Iwwarrhtioo
feshrep momurease
Flantag mumaacces
Festrap herpotga
0.4
pubmedd
0.2
Eachnin
SCL
Erateop
scunded
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.2
0.4
0.6
0.8
1.0
Synthetic Accuracy
Synthelic Accuracy
(a) Multiple Choice evaluation accuracy for the generating model ensemble.
(b) Open Ended evaluation accuracy for the generating model ensemble.
Figure 13: Multiple Choice vs Open Ended synthetic vs human benchmark evaluation performance exploring Pearson accuracy correlation. Both results demonstrate reasonable Pearson correlations between the bench- mark ovaluation accuracy.
F
Evaluation benchmark accuracy and ranking correlation
Figure 13a shows the evaluation accuracy scatterplot comparing the reformatted human questions against the synthetic generated benchmark questions for the bolded ensemble models in Table 2 This ensemble produces
24

---

Published in Transactions on Machine Learning Research (Jan/2026)
Pearson accuracy correlation of 0.743 between the evaluation accuracies for synthetic and human benchmarks. Fach plot marker is the average evaluation accuracy for a given model-dataset combination. While this model ensemble produces reasonable overall accuracy correlation, occasionally specific LMs generate undesirable benchmarks. Figure 14 showcascs a failure where the synthetic benchmark generated by gpt-4.1-ming contains overly easy questions where all models perform near 100% evaluation accuracy (Pearson accuracy correlation 0.276.). This over-performance likely linked to the question length problem.
Benchmark Evaluation Accuracy (R² 0.076, r 0.276)
1.0
Evaluation Models
Gween3-255--22---nnttuce
Ulma---Maverrik
gpt-4.1
0.9
Jamma-33-700--nstruu
gpt-4-1-mini
64-mini
9emma-3-27b-it
9emma-3-4b-it
Qwen2.5-78-instrue
Lama-3311-88 1.8-ttttttttctttt
Original Accuracy
Llama-322-3B-inste
0.6
Datasets
mrga HotporQA
0.5
mrqa NaturaiQQestionsset
fashrag, 2wikkmutthhhpaa
flachrag_mer msmarcoqa
faahrag_ holpotea
TrivisQA-web
0.4
pubmed qa
sec_ _qa
fashrag fermi Squedv2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Synthetic Accuracy
Figure 14: Synthetic wer-performance for generating model 1-tii. indicating overly easy questions
(Pearson accuracy correlation 0.276.)
Figure 15 plots the evaluation model performance across all datasets, comparing the synthetic and human accuracy Spcarman ranking correlation of the evaluation model capability. This matches the use case of user selecting between various models for specific application relevant. to the provided grounding documents.
Figure 15a is the rank order barchart equivalent of Figure 13a and Figure 15b is the rank order barchart
equivalent of the failure case in Figure 14
Table showcases the Spearman rank correlation between evaluation benchmarks written by humans and the synthetic data versions. The table includes data for all three of the human question reformatting models used to demonstrate the minimal impact that reformatting has on the overall high level results. The reformatting operation evaluated against subset of datasets including Squadv2, HotpotQA, TrivaQA- web, NaturalQuestionsShorr PubMedQA, and SecQA. Overall, the best outcome is to ensemble together a few questions generated by various reasonably capable LMs to avoid the known slight self-bias beetbissssl---pffeeeeneee issue. While we used Llama-4-Maverick as the text. feature extractor for measures like groundedness and clarity, that model was unexpectedly dominant these final results.
Table contains the benchmark evaluation Pearson accuracy correlation between the reformatted human questions and the synthetic data generated questions. Similarly to the Spearman ranking, Llama-4-Maverick
25

---

Published in Transactions on Machine Learning Research (Jan/2026)
Benchmankc eval, accuracy grouped by Evaluation Model trank comriaaiin 753)
eval, accuracy grouped by
Syrthelic Quesi end
Orgiial Quediirn
Quoilars
0.05
0.75
0.65
Litem 2.2 38 Instrus
prooned A2bb
Grend fiieettact
inatratt
protmee 3u7bd
Liana 30 eattun
Deenc wwwdt-238-a 01144422000 20107
sptut mus
Lams Havenrd
105-4
genma $40
trentt 2- Birert
gemera 23brt
prman 3
anda
dieme 740 rotumeeee
23008 introit 2007
Alans Haventa
genima
Ereleztion
Doliaaton Model
(b) Benchmark evaluation accuracy generated by gpt-4.1-
(a) Benchmark evaluation accuracy generated by L-ama-4-
mini demonstrating over-estimation LM capability with Spearman rank correlation 0.742.
Maverick producing Spearman rank correlation 0.923.
Figure 15: Comparison of benchmark rank ordering between synthetic and human evaluation accuracy,
Table 4: Spearman rank correlation between synthetic and human benchmarks for multiple choice (MC) and open-ended (OE) evaluations across generating models. Ensemble = bolded models.
Reformat Model
Alama-4-Maverice
gemma-3-27b-i
o4-mini
Generating Model
MC Rank
OE Rank
MC Rank
OE Rank
MC Rank
OE Rank
Ensemble
0.9679
0.9209
0.9429
0.9233
0.9821
0.8813
Llama-3.2-3B-Imm I.ns-tttttt
0.6143
0.7143
0.6929
0.7143
0.6536
0.6791
1amma-333-70B--nstttee
0.9571
0.8813
0.9643
0,8681
0.9714
0.8242
Llama-3.1-8B3
Insttruutttt
0.7964
0.7275
0.7321
0.8110
0.7714
0.7275
Maama-4-Maverte
0.9679
0.9648
0.9536
0.9516
0.9821
0.9209
gemma---11b-it
0.8929
0.8198
0.8964
0.8418
0.9000
0.7890
gemma-3-27b-it
0.9571
0.9516
0.9857
0.9473
0.9714
0.9121
gemma-3-4b-it
0.6393
0.9209
0.6571
0.8989
0.6857
0.8637
gpt-4.1
0.9179
0.8549
0.9250
0.8725
0.9357
0.81 10
gpt-4.1-mini
0.5643
0.8418
0.6286
0.8549
0.6071
0.7934
o4-mini
0.9250
0.7934
0.9036
0.8374
0.9429
0.7714
phi-4
0.6964
0.1780
0.6250
0.3011
0.7607
0.1736
Qwen2.5-7BB-nnstree
0.7321
0.8901
0.7143
0.8945
0.7786
0.8505
performed very well in these tests. However, unlike the ranking data, using a ensemble of models produces significantly worse Pearson accuracy correlation numbers under circumstances. Likely due to the long and overly casy questions written by gpt-4.1 and 04-mini.
26

---

Published in Transactions on Machine Learning Research (Jan/2026)
Table 5: Pearson accuracy correlation between synthetic and human benchmarks for multiple choice (MC) and open-ended (OE) evaluations across generating models. Ensemble bolded models.
Reformat Model
Llama-4-Maverick
gemma-3-27b-it
04-mini
Generating Model
MC Corr
OE Corr
MC Corr
OE Corr
MC Corr
OE Corr
Ensemble
0.7452
0.7095
0.7248
0.5630
0.6569
0.4798
Blama-3.2-3B-nnsttle
0.4857
0.4709
0.5116
0.6181
0.4448
0.5083
0.6615
0.6034
0.7284
0.7464
0.6390
0.5798
-aama-3.-88--nstreee
0.6608
0.6217
0.7231
0.6776
0.6472
0.5795
Mlaveic-Maverick
0.7891
0.7233
0.8755
0.8217
0.8109
0.7833
gemma-3-12b-it
0,6879
0,4085
0.7748
0.5809
0.6986
0,4617
gemma-3-27b-it
0.7477
0,6962
0.8499
0.8292
0.7426
0.7046
gemma-3-4b-it
0.6575
0.5787
0.7117
0.6932
0.6524
0,6040
gpt-4.1
0.6188
0.5216
0.7218
0.6760
0.6528
0.5236
gpt-4.1-mini
0.4015
0.5780
0.4795
0.6308
0,4190
0,5271
o4-mini
0.6542
0.5700
0.7346
0,6430
0.6904
0,5503
phi-4
0.1547
0.4963
0.2348
0.4951
0.2379
0.4043
Owen2.5-7B-Instreen
0.5813
0.5905
0.6448
0.7199
0.5062
0.6019
G Question clarity and groundedness
We observed two failure modes with respect to grounding: Either the model hallucinates content absent from and not relevant to the context chunk (phi-4 commonly did this), or the generating model uses knowledge in its weights to expand upon the information presented in the context (o4-mini). For example, phi-4 generated the following question from the provided context:
"uesstion: "Hox did educational policies of the British Empire during
19th early 20th centuries shape the leadership qualities necessary for navigating madern challanges in the 21st century7"
"contaxt": "Victoria later described her childhood as \"rather melancholy)". Har mother was extrenely protective,
Victoris raised largely isolated from other children under the so called "Kensington Systen'", an elaborate set of rules and protocole devised by the Duchese and her anbitious and domineering comptroller, Ducheas 's lover. The systen prevented the princess from naeting paople shon her mother and Conrey deened undesirable (including nost of her father's 's family), and was designed to render her veak and dependent upon then. The Duchess avoided the court because the the presence of King Willian's bastard children, and perhapa pronpted the energence of inpropriety.
John Conroy, xho was runoured to be the
scandslised
Victorian morality by ineisting that Victoria shared a bedroon with her nother every night, studied with private tutors to a regular King Charles spaniel,
daughter
any appearsnce
tinetable, spent play-hours with her dolls lessons included French, Garman, Italian, and Latin, but she apoke only English at hone."
H
Case study on academic papers
Our approach was demonstrated on two academic papers: "Current Solutions and Future Trends for Robotic Prosthetic Hands" (labeled amnurerrcontrrol-111000------120
(Mendez et al. 2021), "Recent Advances in Large Language Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation" (labeled arXiv arXiv_2502 2502_ 17521v1) (Chen et al. 2025) and the recently released " "America's AT Action Plan" (AiP). Figure 16 showcases LM benchmark evaluation performance on these two datasets; the MCQ results show most models performing well before sharp dropoff with 4B and fewer parameter models and the Ended results with a much smoother decline in performance. Multiple-choice questions can be easier to answer due to the presence of the answer options, even if the evaluation framework shuffles the option order. This effect is likely the cause of the MCQ evaluation accuracy to remaining higher for longer.
The following example questions were drawn from the MCQ synthetic benchmark constructed from arXiv arXiv_2502_11e 2202211722vv (Chen et al. 2025).
("question": "Shat Is the primery distinction between the \"temporal cutoff\" approach and the rule-based generation approach in dynanic benchmarking?",
"choices":
"A": "Tanporal cutoff relies on nevly released information, shile rule-based generation creates noval evaluation pointa using predefined rules.",
27

---

Published in Transactions on Machine Learning Research (Jan/2026)
grouped
_acuurcyy grouped by
0.30
0.85
0.50
0.73
0.70
0.00
Ulree anoree
gemmia 44t4
permta 2264
Ulenns innnncr
Goon TMinree
Limaa 2it dnaoar
godd Eisioo
perma Sub
Uleoa food
apt- Amiil
prid
Maviiii
Uitene 2,5 Releetee
Lieero Mi-torre
Clemio Motaorett
gemma thg
demis doanmee
BMd
Lioaa you Iottor
Libet adaeettc
permma 9.22-4
perrens
Lamr
Dvaluabion Mode
Evaluation Model
(a) Multiple Choice Synthetic Questions
(b) Open- Ended Synthetic Questions
Figure 16: Evaluation benchmark rank ordering comparing Multiple- Choice and Open-Fnded questions
performance across the arXiv 2502 17521v1 and "annurev-control" datasets. (Mendez et al.
2021 Chen
et al. 2025
Benchmark eval accuracyy grouped by Evaluation Model
0.9
2502 17521v1
amnunev.onntro0770000--0006333
0.8
ai-plan
eval accuracy
0.7
0.6
0.5
0.4
Llama3322 Llama-3.-38--imee 22:3--inttruct
Liama-3 Jamm--31-88--nnae. In-t--ntttuctt
Llama-3 3. 3.3-70 1--00B-instn Instrnttucct
Qwen2.5 5- i--instruct
04-mini
gemma-3 Ab-it
Llama-4- Mavericc
gemma-22 12bbtt
phll4
Owen 2358 A22---22------- 2n28uunnttuct--2507
gemma-3 -ommm---227-it 3-27b-it
gpt-4.1
gpt-4.1-mi mpt-4.1-mini
Evaluation Model
Figure 17: Evaluation benchmark rank ordering comparing Open-Ended questions performance across for three example documents. The arxiv paper and AI Action plan were released after most model training data cutoffs. (Mendez al. 2021 Chen et 2025 AiP).
28

---

Published in Transactions on Machine Learning Research (Jan/2026)
"B": "Tenporal cutoff uses predefined rules, rule-based generation nevly released information.'s
"c": "Tenporal cutoff hybrid approach combining multiple nethodologies, while rule-based generation is standalene nethed.",
"D" "Temporal cutoff generates data points using LLNs, while rule-based generation uses manual curation."),
"A").
10
11
("question": "What is potential future research direction for improving the design and standardization of dynanic beachnarking methods for Large Language Models (LLMs), given the current linitations and proposed evaluation criteria?",
18
"choices":
14
"A": "Developing more sophisticated
regeneration techniques to minimize
contanination.",
16
"B": "Eabancing static benchnarking nethode with moze robust data encryption."
'C": "Proposing model architectures reduce reliance Internet training data.", "Focusing solely on post-hoc contamination detection nothods."),
17
"D":
10
"anever"; "A">
The following example questions were drawn from the Open-Ended synthetic benchmark created for
arXiv_25000 1752vv1.
("question": "How might the presence of contaninated benchnarks influence the trajectory of Large Language Modal research and developrent, particularly in terns of model conparisons and deployment decisiona?",
"ansver"; "By leading micleading conclusions
contaninated models.*).
("question": "What are implications using
raurita banchnark samples preserving their original difficulty levels, seen in nethodologies like ITD, on the overall diversity and complexity of the resulting banchnark datasats?",
":aewvr"; "The resulting datagets may nsintain complexity but risk lacking diversity if not resriting stratogies.")
The following example questions were drawn from the Open Ended synthetic benchmark constructed from US AI Action Plan (AiP).
("question": "shat is the primary rationale bahind requiring institutions receiving Federal funding to use nucleic acid syntheais toola with robust nucleic acid sequence screening customer
verification procadures?"
"ansver"; "To prevant the niause of nucleic acid ayntheaia for harmful parposes."
("question": "Shat would be the likely consequence for U.S. nstional security if the U.S. fails to impose comprehonsive export controls condcctot nanufacturing subeystens align ita export control measures with those of its allies?", "ansver": "Adversaries could exploit U.S. seniconductor innovations, undernining U.S. national security."
},
("question":
critical comsideration transforuing NIST's Guardians Foreneic Evidence deopfake evaluation progran into a videly adopted formal guideline for forentic analysis?*,
"ansver": "Enauring adaptability to evolving deepfake technologies."
("question": "How does the National Research Resource (NAIRR) pilot sin enhance
computing resources and AI nodels for the research conmunity, and vhat are its potential implications
healthy financial "ansver"; "By partnering with industry to provide access to sorld-class computing resources and building sustainable operations capability."
developnent
compute?".
11
12
18
("question": "Maat are the potentisl benefits and challeages interagency collaboration anong departments like DOL, DOC, ED, and ESP developing national skill franevorks for AI-related infrsstructure occupations, and hox night this collaboration impact the effectivenese of vorkforce developneat initiatives?",
14
"ansver": "Interagency collaboration can facilitate comprahensive approach to workforce developnent by aligning different agencies' expertise resources, though may challenged by bureaucratic and coordination issues."
15
("question": "What the anticipated long-tern implications of Executive Orders 14277 and 14278 on the American vorkforce's 's ability to adapt to an AI- econesy?". "ansver";
17.
adaptability and resilience through inproved literacy skills devalopnent.""
16
29

---

Published in Transactions on Machine Learning Research (Jan/2026)
Evaluating LM self preference
When understanding potential sources of error for LLM-as-a-Judge how the information is presented to be evaluated is critical. The model evaluation requires the final model answer be on the last line starting with "ANSWER:" make extraction easy.
With the question and model answer text extracted, it can be presented the LLM-as-a-Judgo for grading. The following prompt used to evaluate answer correctness.
("You are comparing submitted ansver to an expart anaver givan question. is the data:
[BEGIX DATA]
[Quastion]: What is significant topological implication of a space being connected yet bacoming totally disconnected upon renoral single point, exemplified by Kuratowski fan?
[Expert]; It highlights the erucial role of single point maintaining the connectedness of the space.
************
[Subrinaton]: ANSWER: The ranoval of single point can have a significant inpact on the connectivity space, leading unpredictable drastic consequences space topological properties.
10
[ED DATA]
12
Compare the factual
of the submitted with the expert Ignore any differences in atyle, grannar, or puactuation.
15
Doss the aubmisaion contain the content in the expert ansger?
16
18
After assessing the submitted ansver, reply with GRADE: SLETTER (without quotes) where LETTER of Cr. Please choose ONE option the grade: either \*Ol* V"I'" for incorrect anavers.
19
For example, after revieuing correct answer night write 'GRADE: C after revieving an incorrect snswer you night write GRADE: I'.
22
Pirst, srite out in a step by step nanner your reasoning about the critarion to be aure that your conclusion corract. Avoid sinply stating the
the outset. Then, with your answer formatted as 'GRADE: $LETTER (vithout quotes) where LETTER is one of CI.
This is then evaluated by the grader (with or without chain of thought) to produce final output "GRADE:
CI"
"To assess vhather the subnitted ansver containa the content in the expart ansver, ge need to analyze
сoпрaг their
content.
Understtadiin Expert Ansver.::
expert ansxer states implication of space being connected yet beconing totally discomnected renoval of single point is that \"hihhlihht the crucial role connectednese of the space. \" Tais indicates that the expert ansver is enpheeizing the inportance of particular point in keeping the apace connacted.
significant topelogical
single point maintaining the
2. u*Uderretanding the Subnitted Insver*:: The subnitted ansver suggests that \"the renoval of single point can have significant inpact on connectivity unpredictable and drastic consaquences for the space 's topological properties. This implies that renoving single point can greatly affect the space's connectivity.
space, leading
3. **Comparing the Factual Content**:
expert anever focuses on The subnitted answer discusses the impact of removing single point on the space's connectivity topological properties.
single point nsintaining comnectedness.
10
11
hnklyyzing the Contentes: Both ansvera touch on the idea that a single point is crucial for directly states this in the context of
the connectećness of space. The expert nsintaining connectednesd. The aubritted anaver conveys a sinilar idea by nontioning the eignificant impact renoving single point connectivity, although goneralizes to topological properties vell.
12
13
enDeternining Equivalence Contentex: The core idea in both anavers that single point playe a critical rele in the connectedaess of space. The subaitted snsver captures the of the expert the space's 's connectivity.
by acknovledging the significant inpact of renoving point
15
6. **Conclusionte: The subnitted ansver conveys the same factual content the expert ansver regarding importance single point connactedness apace. Although
30

---

Published in Transactions on Machine Learning Research (Jan/2026)
submitted ansver nore general not itself to connectednese, contain the core idea presented in the expert answer.
16
GRADE: C"
15
One question that can be asked is do LMs under evaluation perform better questions generated by the same model. In fact, the evaluation LMs do not over-perform the questions they themselves generated. Figure 18 showcases the average generative evaluation accuracy across all datasets for pairs of question generation and evaluation LMs for open-ended questions graded by Llama-4-Maverick. The model-to-mode difference in generation quality drives performance differences. In fact, when the gencrating model and evaluation model are the same is rarely the highest performing version of the questions.
Generative Benchmark Accuracy (OEQ Judged by Sammaa-laaarrtt----22-.---s
Generative Benchmark Accuracy (OEQ Judged by Mava-44-averict-777--2282.
8.00
Uamr dwt.stratt
-03
020
Question Generating Rodel
Lix Limm--4-laoerdoe-.... 27-2277---25mm2
Quastion Generating Model
Goenb3250-4228 Iwr.--naaaaoa
675
-.2
6.6
20-41
860
64
Upcopee Hoventa 1TB 2100 andiive
Lanno 3.2 prieettee
Lienon PPB inthtih
parmaa Dog
permee ta
geromne 4-4
Unera Aeteeric 176 1288 costron
Cheech 2o0 staei rnstost 2007
germen No78
Biue2
Denec lestrve
Model Evaluation
Model
(a) Model average benchmark evaluation accuracy across all datasets. Open-ended questions graded by Tlama-4- all datasets for subset of high accuracy LMs. Open- Maverick.
(b) Model average benchmark evaluation accuracy across
ended questions graded by Llama-4 Llama-4-Mverrick Maverick.
Figure 18: The model-to-monl difference in generation quality drives performance differences, and no LM self- oeff-reference (where models over-perform on the questions they generate) is observed.
Figure 19 showcases the average generative evaluation accuracy across all datasets for pairs question generalion and evaluation LMs for open-ended questions graded by gemma-3-27b-il.
Finally, there is the concern that individual bias in LM-as-a-Judge can cause problems with evaluation measurements. Figure 20 presents heatmap outlining the evaluation accuracy averaged across all datasets for various combinations of open ended question grading model and the model under evaluation. The most impactful element of LM-as-a-Judge on evaluation accuracy is the strength or quality of the grader model used. To reduce grading noise one could ensemble multiple strong LM-as-a-Judge outputs together under a voting scheme.
31

---

Published in Transactions on Machine Learning Research (Jan/2026)
Generative Benchmark Accuracy Matrix
Generative Benchmark Accuracy Matrix
(OEQ Judged by Hamma-3-277-li-
(OEQ Judged by gemma-3-27bnn
Liama snstrrut
Laru- Manari
0.35
Larrer instruct
Quasaiio Generating Rodel
Unima Maw.rck-18 1200-enttrcc.. (641
Quastion Generating Model
0.83
Ristrct
-13
Ower3-718-077- memurroott
1.2s
0.55
0.99
0.33
-23
0.30
Ulase EJ at pectren
Eieng Kaveriik ciie freercalt
Unevn 2. leetiver
Lteere 750 hrstiver
Orvect velthce
gerorn decrrn ar
Bone aree Lonta aduniio
stevena instiott
grommnd Fur
2nd8 instrett
Model Under Evaluation
Model Under Evalustion
(a) Model average benchmark evaluation accuracy across all datasets. Open-ended questions graded by Llama-4- all datasets for subset of high accuracy LMs. Open- Maverick.
(b) Model average benchmark evaluation accuracy across
ended questions graded by Llama-4-Maverrs
Figure 19: The model-to-mott mode difference in generation quality drives performance differences, and no LM self-preference (where models over-perform on the questions they generate) is observed.
Generative Benchmark Accuracy Matrix (OEQ Generated by Maamaa4-avverrrt-11--2----------
Question Grading Model
Liama Llama-3.2-38 3.2.38-Instruce
0.22
6.18
0.25
0.26
0.25
0.35
6.32
0.34
0.30
0.38
0.37
0.35
0.27
0.63
0.49
0.72
0.75
0.69
0.77
6.72
0.76
0.65
0.82
0.90
0.62
0.78
Owen2.5-7B-Instree
0.46
6.92
0.48
0.55
0.98
0.65
0.60
0.65
0.38
0.99
0.71
0.38
0.57
gemma-3.22 3-27b.it
0.57
6.42
0.62
0.65
0.67
0.78
0.75
0.79
0.70
0.78
0.81
0.65
6.71
gemma 4b
6.25
6.14
0.30
0.97
0.31
6,50
6.41
0.45
0.41
0.45
6.92
0.45
0.34
Ulama:3 3.1 1i11--8--ine instruct
Llama-3 1amma-3.2--3 -ig-
Liama-d &i Maverick- 7B: 128E 178E-Instruct instruct
Llama-33 9--00--instie ins-inttrcctt
Owen Owwnn3-235B. 200--3399-400 A228 Instruct-2501
Owen2 5 1B-insttuuct
gemma-3 12b-it
gemma-33 27bbiit
gemmaa- 3-4b-it
upt-4.
gpt-4, I-mini
04-mini
phi4
Model Under Evaluation
Figure 20: Meatmap showing the evaluation accuracy averaged across all datasets for various combinations of open ended question grading model and the model under evaluation. There no strong self eeff-preeerence for grading the models own output.
32

---

Published in Transactions on Machine Learning Research (Jan/2026)
J Generative Benchmarking and Metadata Extraction Prompts
J.1 Question Reformatting
When reformatting the varying quality of the extractive QA dalasets the following prompl was used to clean
up question references and create additional multiple choice answer options as outlined in Section
Your Role
You expert aducational
specializing crafting highly detailed evaluations to deternine competency of topic domain experta based on the provided textual information. Your gosl produce neeningful, highly challenging question anewer pairs encourage roflection, insight, and nuanced underatanding, tailored apecifically according to provided instructions.
Input Structure
Your imput
<question>
[A question to be anssered.]
</question>
11
<anszer>
12
[The
question
18
</anever>
<context>
16
[The text segment containing information relevant to the question.]
</context>
18
19
Prinary Objective
20
Your goal is to reformat, rephrase, and rewrite the questien and ansxer pair according to the senantically aquivalent identical
provided instructicas. The resritten question to the original question, resritten for clarity. The resritten answer should be semantically equivalent and identical to the original ansver, Do not add any additionsl details to the that
question
present
original question
21
22
Anslyeis Conduct careful analysis vithin 'edocument, "ddoumentt.naayyiie) tage, folloving these steps:
23
1. Thoughtrul Content Examination: Carefully analyze the given context, question, and anaver; identifying central ideas, nuanced themes, and significant relationships within it.
27
2. Concept Exploration: Consider implicit aseunptions, aubtle detaile, underlying theories, and potential applicatione
provided information.
28
29
Strategic Complexity are avoided.
Thoughtfully rate difficulty (1-10), ansuring easy questions
30
Intentional Queation Planning: reflection, or critical engagemeut, ensuring the question is purposeful.
question invite deeper understanding, meaningtul
82
## Docunentation in Analysis
Clearly docunent the rationale in the 'cdocument analyeie>' vhen identifying irrelevant bagua contant, explaining your reasons for exclusion or inclusion decisions. poor quality
Briefly justily any decision NDT generate questions
37
38
Question Coneration Guidelines
39
# Encouraged Questien Characteriatics:
41
Thoughtful Engagenent: Prioritize creating questions that inapire deeper thought and nuanced consideration.
42
High Complexity: Develop questions that challenge the domsin expert, follewing the provided additional instructions.
43
Deap Understanding and Insight: Ensure that the question and anavers require understanding of the content by & professional donain expert. Self solf-contained Clarity: Questions
should contain sufficient context, clearly
understandablee independently of external references.
45
Educationsl Inpact:
pedagogical value, reflecting neaningful objectives geauine
content comprehension.. Conversational
16
Formulate engaging, natural, instructional guidelines.
questions appropriate
47
Output Structure
19 Present your final output strictly adhoring the "outputtforrast tags.
<output_fornat>
Question: [ Question Text Ansvar Option
52
33

---

Published in Transactions on Machine Learning Research (Jan/2026)
53
Answer Option
G: Ansver Option c
50 D: Ansver Uption
Explanation: (Brief explenation of why the correct]
67
Correct Ansver: [Letter of correct ansver (one A, B, C, or D)] format?
59
80
Begin by thoughtfully analyzing provided context present the reaulting formatted question ansver pair clearly within coutput_ffrnat]] tags.
analysiiiy tage.
61
Impertant Notes
85
Strive to genorate questions that inspire genuine curiosity, reflection, and thoughtful
engagenent.
64
Maintain clear, direct, and accurate citations/eeplaaaiion dravn verbatin from the provided context.
65 Ensure complexity and depth reflect thoughtrul moderation as guided the additional
instructions.
Each "thoaght.process" should reflect careful consideration and reasoning bebind question generation.
When generating queationa, KEVER include phrases docunent,' or any similar explicit references. Questions should inberently integrate content independently without explicit domain expert anvithout the context paragraph*s. Do not include ansker informstion in the question.
par text,' 'according
uaturally
naterial.
that the queation is anszerable
68
Ensure rigorous adherence output formatting and generate single 'couaput_format>" block.
70
Ensure that all four ansuer aptions are distinct.
Verify
options unanbiguous.
72
Verify the correct answer is present. question
78
Verify
semantically equivalent original question
74
<context> (oontoxtt(cattoott)</ccetttt
76
J.2 Topic Extraction
This is the topic extraction prompt to convert a single chunk of document grounding context into a series of question topics that benchmark evaluation questions should be created around per Figure 2
Your Role
You are an expert educational content creater apecializing in crafting bighly detailed evaluations deternine competency topic experte based on provided teztual information. goal is to produce meaningful, highly challenging question aetsion-anneen answer pairs that encourage reflaction, insight, nuanced understanding, tailored specifically according providad instructiona.
Input Structure
Your imput consists of:
<context>
text segnent analyze, understand, and generate questions about.]
</context>
11
12
Prinary Objective
18
Your goal is to generate list of all relevant topics, facts, or information that should be included
exanination to '
hov professional expert
14
The topica should encourage deep engagement with the content, critically reflect on implications, and clearly denonstrate understanding and competency.
15
Analyeis
Conduct careful analysis vithin 'edocument "chocument_aaalyzsiee" anslysis> tags, folloving these steps:
17
18
1. Thoughtful Content Examinatioon: Carefully analyza the givan context, idantifying cantral ideas, nuanced thenes, and significant relationabips within it.
21
2. Concept Exploration: Consider implicit assumptions, aubtle detaile, underlying theories, and potential applicatione
provided information.
22
23
Strategic Complexity are avoided.
Thoughtfully rate difficulty (1-10), ansuring easy questions
24
Intentional Question Planning: Plan hov question topcis invite deeper understanding,
neaningful reflection, or critical engagenent, ensuring the questions are purpozeful.
27
Additional Instructions for Handling Irrelevant or Bogus Information
Ignoring
34

---

Published in Transactions on Machine Learning Research (Jan/2026)
29
30
Irrelevant Elements: Explicitly disregard hyparlinka, advertisenents, headers, fosters, navigation nenus, disclainers, social nedia bastons, any content clearly trrelevant or exterual to the information of the text chunk.
31
Bogus Information: Detect and exclude any information that appears nonsensical or disconnected from the prinary subject
02
93
Question Generation:
85
Yeaningful Content Requirement: Only generate question topice if the provided '<context>" contains mesmingful, coherent, and educstionslly valuable centent. Complete Irrelevance: If the entire '<centert>' consists exclusively of irrelevant pronstional, vab navigation, fecter, header, ton-intormational analysis and do NOT produce any queation-aned anaver pairs.
36
explicitly this your
37
# Docunentation in Analyais:
80
Clearly document the rationale the 'cdocument, analysie> tage vhen identifying irrelevant bagus contant, explaining your reasens for exclusion or inclusion decisions. Briefly justify any deciaion NDT generate queationa
poor quality
Question Topic Generation Guidelines
44
#V Encouraged Question Topic Cheracterietics:
16
Thoughtful Engagemant: Prioritize creating questions that inspira daepar thought and nuanced consideration.
47
High Complexity: Develop questione that challenge the donsin expert, following the provided additional instructions.
45
Deep Understanding and Insight: Eusure that the question tepics require understanding the content by a professional domain expert.
felf-contained Clarity: questions
aufficient context, clearly underatandable indepandently of external refarences.
should
Educational Inpact:
pedagogical value, reflecting meaningrul objectives genuine
content comprehemsion.
81
Coverage: Ensure generated topics fully the context for which high quality questions be generated.
possible topics for
52
Output Structure
54
Present your final output strictly adhering the formaut_ tags.
<output_fornat>
66
Topic: Topic Text
57
Topic: Topic text
59
coutput_fornat)
61
Begin by thoughtfully analyzing the provided context within anallyiiia tags. Then present resulting forustted question topice clearly formatt tags.
82
63
Inportant
Strive generate question topics that inspire genuine curiosity, reflection, and thoughtful engagenent.
Maintain clear, direct, and context.
ditattions/xplaaations dravn thoughtful moderation as guided by additionsl
provided
complexity depth instructions.
67
Each "thought, "prpcastpprocss"
reflact careful consideration reasoning bahind your question
goneration.
68
Ensure rigoroua adharence to output formatting. detailed required to fully
information present context.
70
The topic response should be single seatence that fully describes the topic.
72
<context>ccantert </onntox
J.3 Open Ended Question Generation
This is the open ended question generation prompt. requires a context chunk and question topic about
which write the question.
Your Role
You are expert educational content creater specializing in crafting bighly detailed ovaluations
competency tepic domain experts based on the provided textual information. goal is to produce neaningtul, highly challenging question ansver pairs that encourage reflection, insight, and nuanced understanding, tailored specifically according to provided instructions.
Input
Your input consists of:
35

---

Published in Transactions on Machine Learning Research (Jan/2020
[The text segnent to analyze, understand, and generate questions about.]
</context>
11
<question_topic:
[A topic around which the question should be generated.]
13
(queation_tooic>
15 a Primary Objactive
16
Your goal is generate single highly insightrul and probing question ansser pair fren the single provided '<context>'. Ain for highly technical understanding to probe dozain expert knovledge about the '<contert>', The question needs to encourage a deep engagorent with the content, critically reflect implications, and clearly demonstrate understanding and compatency. Constructed questions nust be highly challenging to even the anartest donain experts.
17
18
Analyeis Phase
Conduct careful anslysis vithin "cdocument_analyaii
folloving these steps:
20
Thoughtrul
Examination: carafully analyze given context, idantifying ideas, auanced thenes, and significant relationabips within it.
22
Concept Exploration: Consider implicit assunptione, subtle detaile, underlying theories, and potential applicatione of the provided information.
25
3. Stratagic Complexity Calibration: Thoughtfully rate difficulty (1-10), ansuring easy questions avoided.
26
27
Intentional Question Planning:
question can deeper understanding, mesningful
reflection, critical engagenent, ensuring the
is purposeful.
26
Additional Instractions for Handling Irrelevant ## identification and Ignoring of Irrelevant Information:
Information
90
82
Irrelevant Elements:
Explicitly disregard kyperlinks, advertisezents, headers, footers, navigation nenus, disclainers, social media batteas, any content clearly irrelevant external to the information of the text chuck
33
Bogus Information: Detect and exclude any information that appears nonsensical or disconnected from the primary subjact
34
86
Generation:
36
37
Meaningful Content Requirement: Only generate questions
provided
nesningful, coberent, and educationally valuable content.
38
Complete Irrelevance: If the entire '<centext>' consists exclusively of irrelevant pronstional, navigation, faster, header,
explicitly this your snalysis and do NOT produce any question-aneen anever pairs.
non-informational
89
## Docunentation in Analysis:
Clearly document the rationale the 'cdocunent_ <documeat__naayzzie analysis> tags vhen identifying irrelevant bagus contant, explaining your reasons for exclusion or inclusion decisions. poor quality
Briefly justify any decision NDT generate questiona
15
46
Question Ceneration Guidelines
## Encouraged Question Characteristics:
18
Thoughtful Engagenent: Prioritize creating questions inspire deeper thought consideration
50
High Complexity: Develop questions challenge additional instructions.
expert. following provided
51
Kigh Difficulty: Enaure that the question is very difficult to ansver correctly, even for the snartest donsin
52
Ganeralizable: The best questions require the syathesis of high level general underatanding above beyond specific
58
Deep Understanding and Insight: Easure that the question and ansvers require deep understanding expert.
content by professional
54
Self-containae Clarity: Queations and anavers should contain aufficient context, clearly
understandable independently of external references.
Educationsl Inpact: Ensure clear pedagogical value, reflecting zesningful objectives and genuine content comprehension.
66
Convereational Tone: Fornulate engaging, natursl, realistic questions appropriate instructional guidelines.
57
Short and is single phrase or sentence.
Easure
question ansver
factual, that ansver
58
(You do need every question type, only those naturally fitting the content and instructions.)
36

---

Published in Transactions on Machine Learning Research (Jan/2020
B1
Output
62
Present your final output strictly adharing the Nautput_formatt] tags.
63
<output_fornat>
Question: [ Question Text
85
Explanation: [Brief explanation of vhy the ansver is correct]
Correct Ansver: [Short ansuer] tooaattt
67
Bagin thoughtfully analyzing the provided context within 'cdocument, "cdecumentannalyataa" analyaiss tags. Then present the resulting formatted question ansver pair clearly within "<output_fornat" tage.
69
T1
Inportant Notes
Strive generate quastion that inspires genuine curicaity, reflection, and thoughtful engagenent.
73
Maintain clear, direct, accurate citactons/explaaaaiion
verbatin
provided
context.
74
Each "Uhouguttprroess"" should reflect careful consideration and reasoning bebind your question generation.
75
When gonerating questions, NEVER include phrases like 'as per the text,' 'according to the docunent,' any similar explicit referances. Questiona should inherently integrate naturally and stand independently without explicit references to the source material. Make sure question anewerable by NEVER include information in the question that could give away the NEVER aek questions where the ansver is obvious or apparent.
expert segithout
paragrapa**.
76
77
Verify that the
and the best version of that Ensure rigorous adharence to output formatting and ganarate a single '<autput, fornats' tag black.
79
81
82
J.4 Answer Correctness and Validity Prompt
This prompt extracts the answer correctness and answer explanation validity properties outlined in
Section 4.4
Your Role
expert evalustor educational content. Your goal produce meaningful, insightful knouledge about domain expert evaluations designed to deternine conpetence and knowledge.
Input Structure
Your imput consists of:
<question>
[A question
anexored.]
</question>
10
<anszer>
12
[The student 's ansver to the question.]
</ansver>
14
<oxplanation>
16
[An explanation for why the ansver is corract.]
17
</axplanation>
16
19
<context>
30
seguent containing information <onteet)>
question.]
21
22
23
# Primary Objactive
24
You will be evaluating and judging the whether the student ' ansuer and their explanation of why their ansver is correct nakes and is logically valid.
25
26
goal
infermation presented to the '<question>' given the information in the '<context>" and whether the '<epplanation>" 'squestion>"
judge
'<anever>'
why asauned true, anly the context of 'cansuar>' needs to be validated for corractness.
valid.
27
A Metrics
29
1. Ansver Correctness: Rate from to 10 how correct the provided student anaver is given the '<context>', rating incorrect. rating of 10 indicates the ansver is correct and complete.
information
'squestion>'
30
Explanation Validity: Rate from to 10 hou valid the atudents 'cexplanation>' of their anaver is. The "<explanation>" should explain their thinking and the information used to deternine the indicate the explenation is
the context and question. Lox valid, corract, or that there is sone flas in the thinking or logic of the student. High what
ratinge indicate explanation valid, correct, explains why
32
Analysia 33
37

---

Published in Transactions on Machine Learning Research (Jan/2026)
94
careful analysis "ddcument_analyeiii tage, folloving steps:
35
96
1. Thoughtrul Content Examination
Carefully analyze the given context, identifying central ideas, auanced thenes, and significant relationships within it.
99
2. Concept Exploration
Consider implicit assuaptione, the provided information.
details, underlying theories, potential applications
41
Output Structure
45
Present your final output strictly adhering the "Koutput_formar> tage.
puutput_fornat.
45
Anaver Correctneas: [ Correctness Rating, Respond with number in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] Validity Rating. Respond with number in [t, 2, 4, 5, 7, 8, 9, 10]
46
Explanation Validity: [
47
</output_format>>
49
Begin by thoughtfully analyzing the provided context within '<document, anaaumentnnaaysiit tage. Then present reaulting formatted question
pair clearly within '<output_tornas tags.
60
Inpertant
Each "thought_process"" should reflect careful consideration and ressoning bebind ratinge.
68
Ensure rigorous adherence to output formatting, If either the question ansver is nissing, the correctness and explamation ralidity as 1.
66
57
<question> (uorro/oasssoo)//eeroiao.. (
66
69
/ataaaatiootttepppaaaataaaao///a (contett
J.5
Clarity, Difficulty, and Groundedness Prompt
This prompt extracts the question clarity, question difficulty, and question groundedness properties outlined in Section 4.4
You expert evaluator of aducational content. Your goal 1s to produce meaningful, insightful knovledge about domain expert eraluations designed to deternine conpetence and knovledge.
# Input Structure
imput
<question>
[A queation to be ansxered.]
</question>
11
<ansver>
12
question.]
13
</ansuer>
<context>
16
[The text nagnant containing information relavant to the question.]
18
19
Primary Objective
20
You will be evaluating and judging the quality of test and avaluation questions across variety of netrica. Your geal is judge and evaluate the quality of various test and evaluation questions
variety of netrics. The 'squestion>' and '<enswer>' pair grounded and drawn from the '<context>'.
21
22
AV Netrics
23
1. Clarity: Rata from 1 to 10 the clarity and conprahansibility (how understandable it is) of the provided '<question''. rating 1 unclear and cannot be understood or cannot understood vithout the 'ccontezt>', A rating of contained, understaudable, and coherent (even if the topic is couplex and difficult). Questions miesing information required 2015 XPL seasen, hox many Super Boul titles had the Denver Broncos won?" is 10. "What event 1861 contributed to temporary strength republicanien reiga?" ia 10. "In which year the country not nenber of FIFA, as indicated in the table?" is "As of the ceneus of 2000, hov nany families were residing in the city?" is 1.
used for questions that self
what being
a 1.
Britain during Queen Victoria
25
Difficulty: Rate
'cquestion>", rating of reserved questions chich raquire deep understanding of the question and what is being asked by profeasional expert.
difficulty
38

---

Published in Transactions on Machine Learning Research (Jan/2020)
27
28
3. Groundadness: Rate forn 1 to 10 hay grounded the provided '<question>" is in the '<context>', rating 10 requires the question and ansver information can found within the '<context>", rating of 1 indicates the question This metric is only concerned with information found the '<centext>'s not outside outaide information (not contained the '<context>") that is required
anaver information
present in the '<centext>",
infornation. The to anaver the quastion, the laver the rating.
29
a Analysia Phase
81
Conduct careful analysie within "<document_anaayalle taga, rolloving these steps:
35
1. Thoughtful Content Examination: Carefully analyze the given centext, identifying central ideas, and significant ralationabips within
nuanced
34
35
Concept Exploration: Consider implicit assumptions, potential applications of the provided information.
details, underlying thaorias,
96
Output Structure
38
Present your final output strictly adhoring the '<autput_ format> > tags.
coutput_ffrratt
40
Clarity: Clarity Rating (one of [t, 2, 3, 4, 5, 6, 7, 8, 9, 10] )
Difficulty: Difficulty Rating of [1,
9. 10]
Groundednese: Groundednesa Rating (one of [1, 2, 3, 5, 6, T, 9, 101
48
</output_fforat>
15
Begin by thoughtfully analyzing the provided context within 'Edocument, analyais> tags. Then pair clearly within "<output_tormat>" tags.
present reaulting fornatted question
46
47
Inportant
Each "thought_process" should reflect careful consideration and ressoning bebind ratinge. Easure rigorous adherence to output formatting.
49
51
<question> (qeete)e
68
<answer > (answer) c/anver>>
(contett (ecottex)(/cotterrttt </coneext>
39


Reviews:
[{"review_id": "KC0nXhSgUg", "reviewer": "Reviewer_1kyN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an automated framework for creating synthetic evaluation benchmarks for large language models using only grounding documents (e.g., textbooks, papers, or policy reports) instead of manually constructed question datasets. The authors argue that traditional human-built benchmarks—such as MMLU—are limited, static, and often contaminated by training data, making them less effective for evaluating modern LMs. \n\nThe proposed method begins by converting documents into structured text and dividing them into context chunks, from which the model extracts key topics. For each topic–context pair, a language model generates a fact-based question, a correct answer, and an explanation, ensuring that all content is grounded in the source material. These generated questions can take either multiple-choice or open-ended formats. The resulting synthetic benchmarks are then used to assess other LMs’ performance, with metrics such as accuracy and rank correlation compared to human-written benchmarks. This pipeline enables scalable, domain-specific, and automatically updated model evaluations with minimal human intervention. \n\nThe experiments in the paper evaluate the proposed synthetic benchmark generation method across multiple natural language processing QA datasets, including SQuADv2, HotpotQA, TriviaQA, PubMedQA, BoolQ, and SecQA, by comparing synthetic, document-grounded questions against human-authored benchmarks. Using various large language models such as Llama-3/4, Gemma-3, GPT-4.1, Phi-4, and Qwen3, the authors assess question quality (clarity, groundedness, correctness, and diversity) and benchmark validity through Pearson accuracy and Spearman rank correlations between synthetic and human evaluations. Results show strong alignment, with an ensemble of models achieving a 0.74 Pearson correlation and 0.91 Spearman rank correlation. The study also includes a case study on three academic and policy documents to test real-world applicability, revealing that Gemma-3 models excelled on open-ended questions.", "claims_and_evidence_yn": "No", "claims_explanation": "- There is a lack of a formal description of the method. The method involves document chunking, topic extraction, and question generation. It would be better to describe a mathematical problem formulation and solution for this method. \n- Moreover, these methods potentially involve many parameters. How these parameters are adjusted is unknown. \n- The protocol of how human evaluates the questions is not clearly described.", "audience_yn": "Yes", "audience_explanation": "- This paper will receive attention from TMLR readers who build or evaluate LLMs. This paper offers a way to create domain-specific benchmarks from any corpus. \n\n- The pipeline automates chunking documents, extracting topics, and generating grounded multiple-choice questions would be useful for language model evaluation practitioners in writing tests for language models.", "requested_changes": "- It would be better to provide a formal mathematical problem formulation and method description, using necessary notations and algorithmic boxes to describe the pipeline. \n- It would be better to describe the human evaluation process.\n- Are there any potential baselines? It would be better to include a comparison with naive construction methods or perform ablation studies.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YxFF7GTZYM", "reviewer": "Reviewer_a3Nn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method for generating closed-book knowledge-based language model benchmarks (both multiple choice and open-ended questions) based on documents. It involves breaking the documents into smaller contexts, asking language models to identify \"topics\" in each context, then using the LM to generate a question and answer for each context and topic. These questions and answers can be automatically evaluated relative to the original context, and can then be used as a closed-book knowledge evaluation for a new model without including the context. The authors find that performance on their synthetically-generated benchmarks correlates with performance on human-written questions (rewritten by a LM to be closed-book answerable), although some models are better at writing good benchmark questions than others.\n\nOverall the idea of the approach is interesting and relevant, although the method itself seems fairly straightforward once you know the idea. The results are interesting, but currently there are many details missing about the specific prompts and metrics used which make it somewhat difficult to understand what was done.", "claims_and_evidence_yn": "No", "claims_explanation": "It seems like many of the claims in this paper depend strongly on the specific details of the prompts and other specific methodology choices used, and these choices are not clearly explained in the paper. For instance, the meaning of \"topic\", the instructions for how to generate questions, the methodology for grading open-ended responses, and the meaning of quality measurements like \"groundedness\" or \"answer correctness\" are not clearly explained in the paper.\n\nThe paper indicates that the authors plan to release code. This seems critical for making sure the results are meaningful and reproducible. It would also be very useful if all prompts used for all of the steps of the pipeline could be included in the appendix, to make it clear what each of the steps actually involves.\n\nIt is also somewhat unclear whether \"Original Questions\" throughout the paper refers to the original version of the questions or the rewritten version following Section 4.1. For instance, which question versions were used for the length measurements in Section 5.3?\n\nAnother concern: Since the \"Original Questions\" were originally intended as open-book questions and are rewritten as part of this process, it is not obvious to me that comparing performance to the rewritten questions is a good way to evaluate model performance. It would be useful to compare also to some established fact-based benchmark, to help identify whether the resulting benchmarks are high-quality relative to evaluations designed for the same purpose.", "audience_yn": "Yes", "audience_explanation": "Benchmark generation for language models is a relevant area for individuals in TMLR's audience. Although the approach is fairly straightforward, I still think it would be interesting to some members of the audience if the full details of the approach were clearly described.", "requested_changes": "### Add details of how each step works and what each quality metric means, including prompts used\nIn order to understand what these results are showing, it would be important to have more information on what was actually done. I suggest including the prompts in the appendix. It would also be worthwhile to add a bit more information on some of the steps in the main paper (e.g. how are open-ended questions graded?)\n\n### Compare to existing fact-based benchmark instead of rewritten extractive datasets (or more clearly state limitation in abstract / intro / conclusion)\nAs far as I can tell, the generated questions are only ever evaluated against rewritten versions of extractive question-answering datasets, and never compared to question-answering datasets that were designed for closed-book knowledge evaluation. To tell whether the generated synthetic benchmarks are indicative of model knowledge, it seems important to also compare the ranking on the synthetic benchmark to some human-designed benchmark that was explicitly designed for evaluating knowledge, since it could be the case that the rewritten extractive questions are not actually good indicators of model knowledge.\n\n(In absence of this, I think it would be good to explicitly state from the beginning that correlation is being taken relative to some rewritten extractive question benchmarks rather than to a benchmark explicitly designed for knowledge evaluation.)\n\n### Clarify which version of \"Original Questions\" is used\nWhen various results in the paper refer to \"Original Questions\", it is not clear if they are referring to the actual original version of the questions, or the LM-rewritten versions. Please clarify this.\n\n### Clarify correlation in Section 5.8\nWhat is the Spearman rank correlation taken over in Section 5.8? The rest of the paper made me expect that this would be a correlation between the synthetic benchmarks and some other ground-truth questions, but it doesn't look like there are ground truth questions here. Is this correlation just a correlation between performance on some pair of the new papers? It would be good to clarify this.\n\n### Other small suggestions\nThe paper currently mentions \"the original questions\" in section 3.1, but this is not explained until section 4. It would be good to add a brief note in section 3.1 explaining what \"the original questions\" refers to (e.g. \"a set of human-written questions based on the same context documents, rewritten to allow them to be answered without the context.\").\n\nMost of the citations in this paper use the incorrect style; when embedded in text they should appear like \"some text (Name YYYY)\" rather than \"some text Name (YYYY)\". Consider using something like `\\citep`.\n\nNit in sec 4.2: period instead of comma in \"During benchmark creation. the same model\"", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "I am somewhat confused about Section 5.8. Since this section involves building closed-book benchmarks based on academic papers that are beyond the training cutoff, is it expected that models would be able to perform well on this benchmark? Intuitively it seems like many of the facts required to answer these questions would not be known to any model, and some of the questions in appendix G look very difficult to answer without the context document.", "extra_scores": {}, "extra_text": {}}, {"review_id": "J7v7qiuzmv", "reviewer": "Reviewer_Y4Vd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel methodology for automatically constructing fact-based synthetic evaluation benchmarks for language models (LMs), grounded solely in user-provided authoritative documents (e.g., textbooks, research papers). The core idea is to leverage LMs themselves to generate high-quality, domain-specific questions—both multiple-choice (MCQ) and open-ended (OE)—that probe factual knowledge strictly derived from the source material. \n\nKey contributions:\n- A fully automated pipeline for benchmark generation: document chunking, topic extraction, question/answer generation, and evaluation.\n- Empirical validation across 12 standard NLP QA datasets, showing strong alignment between synthetic and human-written benchmarks.\n- Systematic analysis of question quality (clarity, groundedness, correctness) and diversity across a wide range of generator LMs.\n- Identification and analysis of major failure modes (e.g., long-question bias).\n\nKey strengths:\n- High practical utility: Enables rapid, low-cost creation of domain-specific benchmarks without expert annotation.\n- Comprehensive experimentation: Section 5 presents a thorough analysis, including dataset-level breakdowns, generator comparisons, ablations on question formats (MCQ vs open-ended), and qualitative error studies and so on.\n\nKey weaknesses:\n- Limited scope of validation datasets: All 12 are extractive QA tasks; I assume users will also be interested in their domain-specific, detailed cases, e.g, reasoning-heavy, multi-hop, mathematical, or code-based benchmarks.\n- Reliance on LM-as-judge remains undervalidated: More human calibration or statistical significance analysis would strengthen claims.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The central claim—that synthetic benchmarks correlate well with human benchmarks—is robustly supported by quantitative metrics (high rank correlation (≈0.9) and reasonable Pearson correlation (≈0.7) with human benchmarks) and visualizations (showing tight alignment in model performance trends)", "audience_yn": "Yes", "audience_explanation": "TMLR readers, especially those studying evaluation methodology, benchmark construction, and domain adaptation of LMs, would find this work directly relevant. The paper addresses a highly practical and current problem: how to generate and maintain domain-specific benchmarks automatically. Its empirical and methodological insights (e.g., comparing MCQ vs open-ended evaluation, analyzing LM-judge bias) would appeal both to academic researchers and applied ML practitioners building evaluation systems for evolving domains.", "requested_changes": "I have a few questions which might be helpful and included in the next version:\n- In the related work section, discuss the difference between this work and the contemporary works.\n- In Fig 3, the worst model is Llama3.3-70b, even worse than Llama3.2-3b. Could you explain the potential reason?\n- How do authors choose the candidate models?\n- Why do authors use the same model for question generation and answering?\n- For diversity score, authors claim gpt 4.1 shows comparatively worse question diversity, but in Fig 8, it has the highest diversity score. Could you explain the detail or misunderstanding?\n- In multiple-choice vs open-ended questions, authors claim 'the models that are good at creating synthetic multiple choice questions are also good at creating open ended questions'. Could you provide quantitative analysis on the relationship between these two settings?\n- In real life, would using RAG on the provided text/information source be a more common scenario?", "broader_impact_concerns": "The paper does not include a Broader Impact Statement, though the work carries low direct ethical risk. Potential concerns include:\nMisleading evaluations if low-quality synthetic benchmarks are deployed without validation and overreliance on automation could reduce human oversight in high-stakes domains (e.g., medical or legal QA).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)