Reasoning-Driven Synthetic Data Generation and Evaluation
Authors: Tim R. Davidson
Benoit Seguin
Enrico Bacis
Cesar Ilharco
Hamza Harkous
Venue: Accepted by TMLR ()

Abstract: Although many AI applications of interest require specialized multi-modal models, relevant data to train such models is inherently scarce or inaccessible. Filling these gaps with human annotators is prohibitively expensive, error-prone, and time-consuming, leading model builders to increasingly consider synthetic data as a scalable alternative. However, existing synthetic data generation methods often rely on manual prompts, evolutionary algorithms, or extensive seed data from the target distribution — limiting their scalability, explainability, and control. In this paper, we introduce Simula: a novel reasoning-driven framework for data generation and evaluation. It employs a seedless, agentic approach to generate synthetic datasets at scale, allowing users to define desired dataset characteristics through an explainable and controllable process that enables fine-grained resource allocation. We show the efficacy of our approach on a variety of datasets, rigorously testing both intrinsic and downstream properties. Our work (1) offers guidelines for synthetic data mechanism design, (2) provides insights into generating and evaluating synthetic data at scale, and (3) unlocks new opportunities for developing and deploying AI in domains where data scarcity or privacy concerns are paramount.

Published in Transactions on Machine Learning Research (03/2026)
Reavoning-Driven Synthetic Data Generation and Evaluation
Tim R. Davidson" EPFL
reserrch@@rraavidoon.com
Benoit Seguin
Google
Enrico Bacis
Google
Cesar Ilharco Google Deepmind
Hamza Harkous
harkous ctagoooll.com.
Google
Reviewed on OpenReview: https: openreview. net/ pormn? id= &GEP
Abstract
Although many AI applications of interest require specialized multi-modal models, relevant data to train such models is inherently scarce inaccessible. Filling these gaps with human annotators is prohibitively expensive, error-prone, and time-consuming, leading model builders to increasingly consider synthetic data a scalable alternative. However, existing synthetic data generation methods often rely manual prompts, evolutionary algorithms, or extensive seed data from the target distribution - limiting their scalability, explainability, and control. In this paper, we introduce Simula: a novel reasoning- framework for data generation and evaluation. employs seedless, agentic approach to generate synthetic datasets at scale, allowing users define desired dataset characteristics through explainable and controllable process that enables fine-grained resource allocation. We show the ellicacy of our approach and downstream properties. Our work (1) offers guidelines for synthetic data mechanism design, (2) provides insights into generating and evaluating synthetic data at scalc, and (3) unlocks new opportunities for developing and deploying Al in domains where data scarcity or privacy concerns are paramount.
variety of datasels, rigorously testing both intrinsic
1 Introduction
Data availability and access have been central to advances in artificial intelligence research. In recent years, the abundance of highly-diverss internet data enabled the development of increasingly capable generalist models (Gemini al., 2023; OpenAl et al., 2023; Anthropic, 2024; Touvron et al., 2023). Despite these models' impressive versatility, widespread integration will require them to specialize and privacy-sensitive applications. Unfortunately, specialized data in these areas is oflen intrinsically scarce or inaccessible, motivating enormous investments by frontier research labs (Paul & Tong, 2024; Wiggers, 2024; Cottier et al., 2025) and the rapid rise of dedicated "data foundries" (Liu, 2025; Vinn & Hu, 2025). However, creating specialized datasets manually is expensive, time-consuming, and error-prone (Chen et al., 2023; Gilardi et al., 2023; Hosking et al., 2024), leading many to consider synthetic data as promising,
novel, uncommon,
"Work doue during internship Google.
i "Correspnneecee harkous@ggoolle.co. Author contributions the end of the paper

---

Published in Transactions on Machine Learning Research (03/2026)
scalable alternative (Singh et al., 2024a; Abdin et al., 2024; Guo et al., 2025). Nevertheless, how to best balance the various desiderata of synthetic data generation at scale remains an open question.
Beyond limiting Al's progress to data-rich domains, e.g,, coding and creative writing, the field's reliance on real-world data imposes significant structural, safety, and operational costs. The high cost of manual data collection creates an economic threshold that marginalizes contributions from the broader research community (Chen et al., 2023). This dependency also enforces a reactive approach to safety; models can only be hardened against rare edge cases after failures are observed in deployment, practice untenable for safety-critical systems. Operationally, the static nature of real-world data slows down development cycles, in stark contrast to the programmable workflows that a synthetic-first approach enables. Furthermore, the "black box" nature of real data makes mitigating societal biases and untangling ownership rights intractable issues that are addressable by design when the data generation process is fully controlled (Bolukbasi et al., 2016; Buolamwini & Gebru, 2018).
Using synthetic data to solve the specialized data bottleneck faces three distinct challenges concerning (i) the definition of "good" synthetic data, (ii) designing a mechanism that meets realistic requirements, and (iii) conducting generalizable evaluations. Generally, "good" synthetic data is discussed along the axes of "quality, diversity, and complexity" (Havrilla et al., 2024). As a running example, suppose our goal is to generate a dataset of cat images to train a specialized classifier. Instead of indicating the usefulness of data (Swayamdipta et al., 2020; Marion et al., 2023; Tirumala et al., 2023), "quality" commonly refers to how well data points fit specific requirements. For instance, if the intention is to generate an image of a "red cat", does the resulting image have cat in it, and, is that cat indeed red? Meanwhile " "complexity" can refer to how confusing, uncommon, elaborate specific data point is (Ethayarajh et al., 2022; Shao et al., 2023; Fu et al., 2023) but is often equated with the relative concept of "difficulty". This, in turn, begs the question: difficult for whom and under what circumstances? In the case of our red-cat image, a complex example might be a partially obscured cat or one lying in the shadows. Finally, "diversity" offers both a global and local interpretation: docs the generated data globally cover the main factors of interest, and does it locally cxhibit sufficient variation of specific factors? In our case, global factors could be represented by "cat breed", "color", and "environment" and the various members of these factors. Locally, we would be interested in the number of different interpretations of combination of factors, e.g., different takes on red cat lying on couch. Most work on synthetic data focuses on how to best interpret "good" data, often only optimizing for subset of the above desiderata. (Havrilla et al., 2024).
The requirements of the second synthetic data challenge are less commonly discussed, namely, what are the mechanism criteria for generating good data? We posit that realistic deployment conditions require a mechanism that works at scale while maintaining explainability and control. Scalability is a non-negotiable: scaling-law research has repeatedly vindicated the bitter lesson that more good data improves outcomes (Kaplan et al., 2020; Hoffmann et al., 2022). Consequently, this quickly disqualifies methods that rely on elaborate custom prompts (Gupta et al., 2024; Xu et al., 2024b; Yu et al., 2023). Legal and operational requirements further dictate the need for transparent "audit traces", explaining the ingredients used to generate specific data points. This is problematic for methods based on stochastic, evolutionary algorithuss (Mehrotra et al., 2024; Fernando et al., 2024). As will be shown in this work, not all data problems are created equal; some benefit from more quality control and others from higher complexity or local diversity. Optimal resource allocation thus requires fine-grained control over the data generation process.
The third challenge concerns evaluation: how well does the generated data intrinsically approximate a target distribution and how effective is it to train model for specialized downstream use case? For the latter, the most common approach is to generate data that simulates an existing benchmark, train model on the synthetic data, and measure the trained model's performance on the benchmark's 's hold-out test set. However, benchmark datasets are incomplete samples of a target domain, often contain incorrect labels (Northeutt et al., 2021), are unbalanced (van Breugel et al., 2024), "under annotated", e.g., they lack multi-level annotations and details of interest, and risk being leaked into pre-training datasets. Hence, naively optimizing the data generative process toward benchmark dataset can be deceiving. It can lead to overfitting, where synthetic data is optimized to fit the specific quirks of benchmark dataset, which may not be representative of the target domain. Care should thus be taken to evaluate synthetic data approaches on diverse, recent datasets to improve generalization and minimize potential upstream contagion.
2

---

Published in Transactions on Machine Learning Research (03/2026)
(a)
(6)
Figure 1: Synthetic Coverage Examples. The ouler squares of (a-b-c) represent semantic factor of interest, e.g., "cal type." (a) Characterizes "random" " sampling behavior, with no notion of a global coverage space, This often results in samples clustered around semantic modes and misses edge cases. The grid-like structures in (b-c) represent discrete semantic spaces defined by a taxonomy's leaf nodes at increasing levels of gramularity. For example, the first level could represent "cat type" broken down into "domestic, big wild cals, small wild cats, and ferul", whereas square at lower level might represent a specific cat breed like the "British shorthair." (b) Represents perfect global planning at increasing granularity; and (c) shows global planning with progressive coverage loss, e.g., missing the branch "big wiild cats" entirely (bottom left) or missing specific breeds (bottom right).
In this work, we propose Simula, a framework for synthetic data generation and evaluation that prioritizes transparency, fine-grained control, and scalability. Unlike many existing efforts that rcly opaque, seed- dependent processes, Simula employs a "reasoning-frss" methodology, constructing each data point from first. principles. This approach ensures that Simula's generation capabilities improve naturally with the reasoning capabilities of its underlying models, making the system inherently future-proof. Our three-stage pipeline first maps the conceptual space of the target domain to ensure coverage, then uses iterative agentic refinements populate with diverse and complex examples, and finally uses dual-critic filtering stage to enhance quality. We rigorously test the core reasoning assumptions underlying our approach and demonstrate its efficacy on series of carefully designed experiments using variely of datasets. Our cont ribuution are follows: (1) new reasoning-fin firrst framework for synthetic data generation capable of creating explainable and controllable data at scale; (2) comprehensive empirical analysis connecting intrinsic properties of generated data (e.g,, complexity, diversity) to extrinsic downstream model performance, offering novel, actionable evaluation metrics and insights into scaling synthetic data; (3) a set of pragmatic principles that establish mechanism design how data is generated as distinct research axis, orthogonal to the traditional focus on what constitutes "good data."
2 Simula: A Reasoning-Firss Framework for Data Generation and Evaluation
Suppose we want to create a dataset with the description y == "A dataset of stories about cats." " Duc to the under-speciflcation of y, infeasible to exhaustively describe the space of all datasets V, that fit this description. This problematic as it prevents us from developing an actionable notion of coverage, i.e. given datasct Dy ~ V, what area. of does it represent?
2.1 Using Taxonomies to Capture Dataset Coverage
To formulate a first-order approximation of V, we start by disentangling our target dataset into its prime factors of variation." For example, dataset that fits description y might consist of data points covering "cat type," " sttrry format," " and "intended audience." In Simula, multi-moda model (M3) is used to propose factors fı based on set of human-provided instructions, e-g., description like y, and/or sample of existing data S. These factors can be accepted or rejected by an M3 (or Human). Next, an M3 is used to expand each fi breadth-first into laxonomies, Tr, of a (user-) specified depth d;:
M3(y. S. (do. fo). (dk,fk)) (T T
(1)
1 Note that perfect disentanglement course not always possible (Locatello al., 2019).
3

---

Published in Transactions on Machine Learning Research (03/2026)
1 c
fo
To
Generator
Complexify
{u, S}
Critic
fx
Tx
(a) Determine Factors
(b) Generate Taxonomies
(c) Sample Mixes
(d) Meta Prompts
(e) Generate and Critique
Schematic of Simula Framework. Given user instructions y and/or data sample S, first (a) determine factors of interest fi, which (b) are expanded into taxonomies T. Next, (c) nodes of Ti are sampled to obtain mixes, and (d) turned into "meta prompts". A user-defined fraction, of meta prompts is "complexified." (e) Finally meta prompts are used to generate data proposals by a Generator, and iteratively refined using Critic step.
Figure 2:
A taxonomy is a hierarchical tree structure where the root node represents a broad factor of variation, and child nodes represent increasingly specific sub-categories or instantiations. For instance, "cat type" can broken down into "domestic" - "shorthair" " BBittish shorthatr." Taxonomies thus serve as structured map of concept space, breaking down abstract factors into concrete, sample-able attributes. This provides granular explainability and control of y compared to random sampling (Figure L.a). Intuitively, the number of factors and taxonomy depths, we sharpen our coverage control (Figure 1.b). However, this gramularity comes at a potential cost: with every taxonomy expansion we risk "missing" nodes of interest (e.g., the "shorthair" branch), resulting in the progressive coverage loss depicted in Figure 1.c.
increase
To mitigate polential coverage loss resulting from missing nodes, we generate Te by alternating between three steps (full algorithm in App. B.4): (1) Given a node, its ancestors and its siblings, an M3 prompted N times to propose initial set of children nodes. This sampling strategy is inspired by the "Best-of-N" literature to increase the proposal distribution and cover edge cases (Brown et al., 2024). (2) In separate call, an M3 is prompted to act a critic, refining the initial proposals by adding, removing, merging, or editing nodes to improve their completeness, soundness, and specificity, leveraging the generator-critic gap (Huang et al., 2024). Optionally, (3) after genorating all nodes of a specific level, an M3 is prompted to generate a "plan" for the next level. This enables consistent and fast. parallel generation by ensuring a similar degree of granularity at different node expansions on the same level across independent predictions. At each step, the M3 also has access to the uscr-provided input y, and/or sample S from the target distribution.
2.2 Controlled, Explainable, and Scalable Data Synthesis with Agentic Steps
Having obtained the taxonomics Tú we can generate synthetic datasets that fit our requirements through two phases: taxonomic sampling (Figure 2.c) and agentic refinement (Figure 2.d-e). Full algorithm is in App. C.
Sampling Strategies to Control Global Diversity. An M3 first formulates a plan composed of sampling strategies. A strategy defines which laxonomies are sampled logether and with what weight, preventing illogical combinations by grouping compatible subsets. For instance, a "children's content" strategy might combine taxonomics for simple topics and ago-appropriate formats while excluding those for mature themes, thus preventing the generation of horror novel for toddlers. By weighting different strategies, the framework control the proportion of generated data for each target group. Given strategy, the framework
4

---

Published in Transactions on Machine Learning Research (03/2026)
samples nodes from the corresponding taxonomies Te These sampled nodes can be thought of as data point "requirements," " that, along with the original dataset instructions y, then guide an M3 to construct one or more "meta prompts". For instance, M3 {house cat, poem, travel enthusiast)) becomes " "Compose an exciting haiku about a house cat who goes on an adventure." Finally, these meta prompts are passed to an M3 to generale data output proposals.
Optimizing Local Diversity and Complexity. Suppose we want to construct a datasct of size N = 100, and our factor and strategy selection has yielded V = 200 unique node-sets. Since N < V, our sampling budget allows for at most 100 unique node-sets with a single meta prompt each, resulting in a global coverage ratio of 100/200- = 0.5. Conversely, for N > V, c.g., N 800, we can sample up to four meta prompts for each node-set and cover the entire space. As the number of meta prompts per node-set grows, we gradually increase local diversity. However, for larger values of N/V, independently generating meta prompts from fixed requirements can lead to mode collapse, L.e, increasingly similar meta prompts. This is mitigated by generating multiple meta prompts simultancously, then sub-sampling the required fraction. Next, we perform complexiiication on a fraction of the samples, by prompling the M3 to increase the complexity of the generated meta prompts and outputs while maintaining our other requirements. Optimizing local diversity and complexity this way works well for smaller sample sizes, but degrades as N/V grows very large. Instead, for larger ratios, Simula can be configured to sequentially generate more diverse or complex meta prompts with previous attempts in context, allowing an M3 to reflect on previous generations.
Enhancing Sample Quality with Critics. The system performs a series of agentic refinement steps to optimize sample output quality. It starts with point-wise checks to ensure the generated samples pass the specified semantic and syntactic requirements coming from the taxonomy nodes. This involves letting the M3 "critique" the generated samples by providing the meta prompts used for generation and roquesting an explanation and a binary verdict. Given the generated sample for the advent urous house-cat haiku above, the M3 would check if the cat in the story is indeed a house cat, if the output is a haiku, and if adventures were had. For tasks requiring outputs with a defined notion of correctness (e.g., classification or multiple-choice questions), the system employs an additional "double-critic" step, which independently assesses correctness and nesorrectness lo mitigate sycophancy bias (Sharma et al., 2024) (More on this in Section 3.1). Following these " "critic refinement" steps, if the M3 responds with a negative verdict, the system either rejects the sample or applics automated modifications based on the explanation and repeats the critique step.
2.3 Evaluating and Comparing Dataset Properties
The previous section described how taxonomies and agentic steps can be combined to generate synthetic datasets optimized for diversity, complexity, and quality. Beyond generation, we are interested in evaluating the propertics of datascts - both synthotic or real and comparing them. We therefore describe two approaches that use agentic reasoning and (axonomies to evaluate and compare complexity and diversity.
Reasoning about Complexity using Calibrated Attribute Scoring. Different levels of data complexity are desired depending on the target task. Ideally, one would like to assign "complexity scores" to individual data points, enabling control over sample complexity as well as comparisons between synthetic and real data. This presents several challenges: (i) synthetic data generation unsupervised, (fi) real data is generally not annotated for complexity, and (iii) "complexity" is a relative concept that depends on the domain and task at hand. To overcome these issues we propose "batch-wise" evaluation approach based on reasoning: Firstly, batches of data points are sampled such that each data point appears K times. Secondly, scores to cach sample in cach batch reflecting their complerity using the datasct description y (Algorithm in App. E.3). By providing a batch to score instead of individual points, the model can calibrate its outputs among the different points to reduce noise resulting from per-sample overconfidence (Zheng et al., 2023; Xiong et al., 2024). We further improve our scoring signal by breaking batch-spocific score assignments into pairwise comparisons and computing Elo scores (Elo & Sloan, 1978). The resulting Elo scores enable complexity comparisons of data points across different datasets and can be used to control a sample's complexity.
M3 assigns
5

---

Published in Transactions on Machine Learning Research (03/2026)
Using Taxonomies to Curate Dataset Coverage. The importance of data selection during both M3 training and inference time is emphasized by growing body of research (Swayamdipta et al., 2020; Marion et al., 2023; Ye et al., 2024; Xia et al., 2024; Hu et al., 2024; Hübotter et al., 2024), inter alia. Access to training data that accurately reflects test-time conditions and minimizes biases is essential to assess model performance, fairness, and preparedness. Despite their vital role, most datasets are sparsely labeled using high-lovel semantic descriptions, c-g,; a "math" question, a "harmful" comment, a. "cat" image, etc., or not labeled at all. This complicates efforts to curate optimal corpora, allocate resources (Qian et al., 2024), and catch potential misalignment between train and test sets (van Breugel et al., 2024).. Using taxonomies offers way forward not only for generating synthetic data, but also for better understanding existing data. Given a dataset, we can generate or use existing taxonomies that define the conceptual space of interest. We then query an M3 to perform laxonomy assignments for each data point by prompling it with the full laxonomy in context, linking each item to the most relevant node, This process yields a detailed map of the dataset's composition, from which various coverage metrics can be derived. In our evaluation, for instance, we use a "Level Ratio Coverage" metric, which caleulates the proportion of unique nodes covered by a dataset at each taxonomy level. This provides a fine-grate graingrriedd,, actionable view into dataset's 's coverage, highlighting both well-represented areas and potential gaps to fill.
3 Experimental Setup
3.1 Verifying Core Reasoning Components
Simula primarily relies on three core assumptions about M3 reasoning capabilities: they (i) can generate high-quality taxonomies; (ii) function as effective "critics" of their own outputs; and (iii) distinguish between the complexity of examples. We evaluate these using Gemini 2.5 Flash (non-thinking) (Geminí et al., 2024).
Can M3s Generate High-quality Taxonomies? Evaluating the quality of taxonomies is inherently challenging due to the lack of standardized criteria and methods (Szopinski et al., 2020; Kaplan et al., 2022; Kundisch et al., 2021). We differentiate between grounded taxonomies, e.g., phylogenetic trees, and conceptual ones, o.g., types of harmfulness. Given an expert taxonomy, TE, we compare to an M3-generated taxonomy for the same topic, TMa- Structurally, we care about completeness (does TM3 cover TE??, soundness (does TMs contain irrelevant or unnccossary nodes?), and novelty (does TMa contain relevant nodes not in TE?). We evaluate both the Simula generator-critie approach and 0-shot expansion on six real-world taxonomies. Additional experimental details are available in Appendix B.
Are M3s Effective Critics? We test the ability of our "double which independently queries if answer is correct and incorrect to verify answer correctness as follows: In the Controlled Setting, we take reference tasks with correct answers (and if applicable, explanations), Derue = ((0;, v))) and create a corrupted copy Doorrupt by prompting an M3 to subtly change yi while keeping x; fixed. After performing this cansal intervention on answer correctness (Pcarl, 1994), we evaluate the double critic on both datascts. Note that, for effective crilic-rejection sampling, we need the likelihood of accepting a correct answer to be high and accepting a corrupted answer to be low. In the Empirical Setting, we test if critic capabilities transport (Pearl & Bareinboim, 2011) to the model's own outputs. We prompt an M3 to generate reasoning traces and answers for the benchmark tasks covering free-form math problems (MATH, Hendrycks et al. (2021)) and multiple-choice questions in selected languages (Global MMLU, Singh et al. (2024b)). We apply double-critic rejection sampling, measuring the rejection rate and the empirical change in accuracy. We note that this differs from the full Simula pipeline where the M3 generates both questions and answers. The net impact of the critic in this fully synthetic scenario is tested through our downstream evaluations (Sec. 4.3).
Can M3s Distinguish Data Complexity? We first investigate whether complexity rankings computed using our calibrated scoring method align with human-provided complexity labels. Second, we examine whether the model's complexity scores align with its own generation and verification capabilities, i.e., we study whether its performance doclines on problems it rates as more complex. For this analysis, we again use the MATH dataset, which contains explicit annatttdd complexity ratings (1-5), and selected Global MMLU subjects, where education levels (elementary, high school, and college) serve a proxy for complexity.
6

---

Published in Transactions on Machine Learning Research (03/2026)
3.2 System Versions and Datasets
System Versions. Table shows the different Simula components evaluated. To ensure consistent basis for comparison, first generate fine-grained taxonomies for each task. We then evaluate system configurations by ablating different components: For the Baseline configuration, top-level nodes are sampled from the relevant taxonomies; each mix of nodes is then turned into a data point by the M3. The Local component also uses top-lovel nodes but incorporates agentic refinement by converting them to "meta prompts" and applying "complexification" to a fraction e = 0.5 of these. The Global component leverages the full taxonomic depth by sampling from nodes at all levels, but omits mnea--roomping Finally, we evaluate version combining both Local + Global diversification, and another that adds a final Critiquing step to enhance quality.
Table 1: System Versions. We evaluate the efficacy and additive properties of different system components.
Taxonomy Depth
Mela Prompling
Critie Steps
Baseline
Local
X
Global
>1
X
X
Local + Global
X
Local + Global + Critique (i.e., Full System)
>1
V
Datasets. We chose datasets that cover a spectrum of niche, recent datasets that are not commonly optimized for, as well as common datasets whose domains are optimized for in popular benchmarks. For the former, focused on specialized lasks, we chose two datasets from the Cyber Threat Intelligence Benchmark (CTIBench) (Alam et al., 2024): CTI-MCQ, a multiple-choice question dataset with four choices and a single answer to assess M3s' understanding of CTI standards, threats, and mitigation (2.5k test questions); and CTI-RCM, task requiring the open-ended generation of Common Weakness Enumeration (CWE) category based on Common Vulnerabilities and Exposures (CVE) description (1k test CVFs). We also selected a dataset from the legal domain, LEXam (Fan et al., 2025), which consists of 1.66k multiple-choice questions from Swiss, EU, and international law examinations in both English and German.
To cover popular domains, we selected two well-known datasets: GSM8k (Cobbe et al., 2021), high-quality, linguistically diverse grade-school math problems requiring multi-stop reasoning (1.32k test problems); and a subset of Global MMLU (Singh et al., 2024b), a multilingual, multiple-choice question dataset with four choices and single answer. To keep the data generation manageable, selected a subset across Math, Computer Science, and Physics in English (High-resource), Korcan (Mid-rosourcc), and Nepali (Lov-resource), for a total of 1.44k test questions per language. We applied post-filtering process for the generated datasets, including de-duplication and decontamination against the real test sets (based on 13-gram overlap with a Jaccard threshold of 0.8). We ended up with 512k unique data points for each of the above datasets.
3.3 Intrinsic Metrics
We compute intrinsic metrics sample of 1k data points for each dataset and each system version.
Embedding Diversity and Taxonomic Coverage. Following Yu et al. (2023), we evaluate diversity by transforming datapoints into an embedding space and measuring cosine distances. For global diversity we report the average dataset-wide, pairwise cosine distance. To evaluate local diversity, first group data points by taking the k = 10 nearest neighbors to each data point in embedding space to ensure semantically meaningful clusters. We then take the average pairwise cosine distance across these clusters. For both, a higher cosine distance indicales higher diversity. We compute the embeddings using the model from Lee et al. (2024b). In practice, cosine distance scores are coarse metrics that do not provide actionable insights to improve diversity. We therefore also evaluate each dataset's taxonomic coverage using the assignment- based methodology described in 2.3, which provides more structured and actionable measure of how comprehensively data covers conceptual domain.
7

---

Published in Transactions on Machine Learning Research (03/2026)
Complexity. Our goals for intrinsic complexity evaluation are twofold: First, we want to test if the different Simula components can effectively control the complexity distribution of synthetic dataset. Secondly, we want to compare the complexity of synthetic datasets to the "real" reference datasets. After generating synthotic datasets using the different systems, we follow the calibrated complexity scoring method outlined in Section 2.3, mixing logether the various synthetic datasets with the real data.
3.4 Downstream Metrics
For all downstream experiments we use Gemma 3 4B as our student model (Gemma et al., 2024) and Gemini 2.5-Flash (non-thinking) as our teacher model. For datasets covering domains that are commonly optimized for such as GSM8k and Global MMLU, we use the pre-trained (PT) version as the starting point. For the more niche, recent. datasets, CTI-MCQ, CTI-RCM, and LEXam, we use the instruction-tuned (TT) version. We perform 10 iterations of LoRA fine-tuning (Hu et al., 2022) with different seeds per configuration and datasct size, reporting mcan accuracy for each data size and 95% confidence intervals computed as the standard error of the mean across seeds (see App. F for our hyperparameter optimization process). To isolate the impact of complexity on downstream metrics, we split datasets produced by the full system into three subsets. To avoid confounding missing complex concepts with complexification per concept, we sample by complexity score per taxonomy node. This gives Low Complerity split (bottom 40% of the data), High Complexity split (top 40%); and All Complerity split. (a random subsample of the previous two).
4 Results
4.1 Results for Core Reasoning Components
500
32%
Avergge Medell Complesity Fle
Accepted Samples (Elo)
102%
pessieed by cetlic
Biseline Aga
Rejected Samplex (Elo)
25%
Theupettral Rojection Rate
R7%
Accepted Ugnl
Emptrical Rojerttion Rate
450
Accurac
92%
35%
93%
23%
Thas covpuettt taggrre posttrre ucourrcy gals
100
80%
Beseline Expected [lpere]
83%
Model comglestty aliezs with human
10%
13%
738
380
Rejucted sumples are conmistasily harder.
70%
314
738
-28
Human Complexity Level
Human Complexity Level
Human Complexity Lovel
(a) Controlled Setting
(b) Empirical Setting
(c) Complexity Analysis
Figure 3: Double- Couble-Critic Critic Rejection Sampling on MATH. (a) We establish the theoretical lift in the controlled setting, noting that rejection costs increase with task complexity. (b) Critic capabilities transport to the empirical setting but lose some effectiveness. level, rejected samples are consistently assigned higher model complexity.
(c) Calibrated Elo scores show model-human aligument. Stratified by complexity
M3s Can Generate Good Taxonomies. Table 2 shows Simula taxonomies approximately cover those created by human experts. For conceptual taxonomies, almost all generated nodes are sound, with many novel expansions resulting in increased total coverage. Note that are less concerned with "over" coverage, as this ensures that most edge cases are present. We can always reduce the depth of a taxonomy if we believe it to be too granular. These results support our approach of approximating a global coverage space using taxonomies. Appendix B contains further details and analyses on evaluated taxonomies, as well as limitations and examples of synthetic taxonomies.
Couble-Critie Rejection Sampling Enhances Quality. Figure 3 summarizes our doublc-critic evaluation on the MATH dalaset, where Mgen is the model's generalive baseline accuracy. In the Controlled Selling (Figure 3a), we measure the probabilities of the critic accepting correct, p(y), and incorrect, plycurruply answers. | |Dacceptl= = Pgen p(y) + (1 (Mgen) corrupp)) The expocted accuracy of these items is E[peritiel = Mgen PPcee/Paccepell We observe a consistent theoretical "lift" (B[peritell Mgen). As human-assigned complexity increases, maintaining this lift. requires a higher "cost." in rejection rate
The accepted proportion is defined as
8

---

Published in Transactions on Machine Learning Research (03/2026)
Table 2: Taxonomy evaluation. Average metrics across taxonomies, Novelty and total coverage are only reported for Conceptual taxonomies, since assume that grounded taxonomies should have room for further extensions.
Grounded
Conceptual
Method
Completeness
Soundness
Completeness
Soundness
Novelty
Coverage
Simula 0.74
0.75
0.78
0.97
0.94
1.72
0-Shot
0.52
0.70
0.50
0.97
0.32
0.83
(PPrejet) - |D (Paccept)). l). In the Empirical Setting (Figure 3b), the critic filters the model's own outputs into accepted and rejected subsets. While still realizing a lift over the baseline, Mgenl V > Mgens it less effective. Again, higher complexity necessitates higher empirical rejection rate sustain accuracy gains.
Complexity Scores Align; Rejected Samples Perceived Harder. Figure 3c validates our calibrated scoring method: model-assigned Elo scores align with the humans ones. We also see that stratified by human-assigned complexity, rejected samples have higher Elo scores than accepted ones, indicating the critic systematically filters samples with higher perceived model complexity. Additional results in App. D, E.
4.2
Intrinsic Metric Results
- Real
Baseline
Local
*
Global
Local + Global
Local Global Critique LEXam
Avg. Cosine Distance
CTI MCQ
CTI RCM
GSM8k
0.54
0.52
0.5
0.48
(a) Dataset-Wii Embeddings Diversity
Avg. Cosine Distance
CTI MCQ
CTI RCM
GSM8k
LEXam
0.25
(b) Neiihss-Neighbors Embeddings Diversity
CTI MCQ
CTIRCM
GSM8K
LEXam
Coverage Ratio
0.75
0.5
0.25
1
2
3
2
3
Taxonomy Level
Taxonomy Level
Taxonomy Level
Taxonomy Level
(c) Taxonomic Coverage
Figure 4: Intrinsic Diversity Metrics. We display dataset-wide (top) and nearest-neighhors (middle) embedding- based diversity, and taxonomic coverage (bottom). We note that Global diversification crucial for increasing dataset-wide diversity, and that Local and Global diversification generally have additive effect. We further note that while real data can be more or less diverse according to embedding- basedd metrics, it almost always covers less of the target domain than Simula variants on taxonomy basis.

---

Published in Transactions on Machine Learning Research (03/2026)
Intrinsic Diversity. Figure shows results for the different diversity metries. Starting with the embeddings- based diversity metrics (top and middle), we first note the Global diversification component is crucial to increase the dataset-wide embedding based diversity (top). Second, the Local component improves over the bascline; hence meta-prompting effectively increases the diversity of the nearest points (middle). Third, the Local and Global diversification components generally have an additive effect. for embealding-bsse metrics. While embedding-
diversity of real data can be both higher or lower than the synthetic variants, when it comes to taxonomic coverage, we observe clear pattern: the full Simula system exhibits higher coverage the taxonomy levels increase. Assuming domain, this indicates that the real data misses large subsets of interest. Finally, critiquing does not appear to affect diversity in a significant way, almost always overlapping with the variant without critic steps.
taxonomies describe comprehensive coverage of the target
Real
Baseline
Local
Global
Local Global GSM8k
Local Global Critique
CTI MCQ
CTI RCM
LEXam
0.02
0.015
Density
0.01
0.005
300
400
500
300
400
500
300 400
500
300
400
Complexity Elo
Complexity Elo
Complexity
Complexity Elo
Figure 5: Complexity Elo Distribution of Synthetic and Real Data. We display density plots of the complexity Elo rankings for the various system versions on four datasets. We note that synthetic data can the entire complexity range of all datasets and that Local and Global components are generally additive, i.e., they account for different types of complexity.
Intrinsic Complexity. We display density plots of our calibrated complexity evaluation Figure 5. It can be observed that the full Simula system is capable of covering the entire complexity range of real data, often even allowing for more complex data points. We also note that Local and Global components generally have additive relation, providing different types of complexity. The different datasets have distinctly different complexity profiles, with the first three panels covering a wide range while LEXam is much more concentrated." Furthermore, while increased diversity and critiquing can reduce complexity (GSM8k), as we will see in the next section, this does not necessarily have an adverse effect on downstream performance.
4.3 Downstream Results
In Figure 6, we show how downstream performance evolves per dataset across data sizes and system versions. Results for additional Global MMLU subsets are available in App. F.2.
Simultaneously Optimizing all Data Axes Never Hurts. We first observe that using the full Simula system is almost always the dominant strategy across all datasets and data sizes. This is in contrast to the Bascline, which scales the worst, indicating that the data-scaling law is a function of the data propertics, not size alone. This also means that, without strong priors, using the full system is a good strategy.
Student-Teacher Gap Impacts Scaling Laws. We see that downstream performance of the student model at different data scales is impacted by the relative difference against the teacher's performance. In the case of CTI RCM, we see that performance saturates at around 128k, after bridging 65-40 70-40 ~8% of
the performance gap between the starting point of the student (40%) and the teacher (70%) model. This is not the case with GSMSk, where all versions continue to grow, as the performance of the student model (maximum of 75%) remains sufficiently far from the teacher's performance (88%) on this task.
This aligns with the teacher model's poor performance LFXam, as shown Section 4.3.
10

---

Published in Transactions on Machine Learning Research (03/2026)
Combining Global and Local Diversity is Critical. We see how Local and Global diversification in isolation provide suboptimal returns depending on the dataset and data size. In the case of CTT MCQ, Local diversification plateaus before variants containing a Global component. Local diversification also leads to significantly weaker performance in the small data regime for CTI RCM (4k and 16k). On the other hand, Global diversification does not surpass the Baseline in LEXam and scales slightly worse the case of GSM8k. Combining the two improved performance on all datasets and dataset sizes.
CTT MCQ
CTI RCM
LEXam
Gemini Flash 2.5: 71% Gemma 4B IT: 54%
Gemini Flash 2.5: 70% Gemma 4B IT: 40%
Gemini Flash 2.3: 57% Gemma 4B IT: 29%
Accuracy Caarrryyyoooi (LORA (Gemma-344tt
0.65
0.64
0.62
0.36
0.6
0.34
0.58
0.55
0.96
0.32
4k
16k
64k
128k
256k
512k
4k
16k
64k
128k
256k
512k
1k
16k
64k
128k
256k
512k
GSMBk
Global MMLU (PMMCS)) - Nepali
System Version
Gemini 2.5:
Gemma 4B IT: 80%
Gemini Flash 2.5: 93% Gemma 52%
Basellne
Actumcy (LORA, -Gemma- 3-48-P1)
0.75
0.48
Global
0.7
0.46
Local - Global
Local Global Critiqu
0.65
0.44
0.42
0.6
0.55
0.38
0.36
4k
16k
64k
128k
256k
512k
4k
16k
64k
128k
256k
512k
Data Size
Data Size
Figure 6: Downstream Performance on Different Datasets. We report mean accuracy with 95% CI for different system versious. We note that while the contribution of different components to performance depends on the dataset and data size, the full Simula system always reaches the best result.
Critiquing Impact is Dataset Dependent. Adding the critiquing step does not hurt performance on any of the studied datasets. In the case of Global MMLU it made a significant improvement, despite the rejected data being limited to only 3%. The critic rejection rate was 2% for CTI MCQ, 9% for CTT RCM, 9% for GSM8k, and 61% for LEXam. For LEXam, the high rejection rate can likely be attributed to the weaker performance of the teacher model (57% accuracy). Even when critic-based rejection sampling does not significantly improve downstream performance, we argue that adding critic step is still desired. First, ensuring data points follow diversity requirements maintains coverage control. Second, increasing "correctness" of labels promotes causal learning, thus improving model robustness.
Complexity is Crucial in Most Domains but Can Hurt with Weak Teacher. Figure 7 illustrates the impact of data complexity mean accuracy across four datasets. The importance of complexity is most evident in the GSM8k dataset. Here, we obscrve direct correlation between data complexity and accuracy, with the High Complexity split yielding a 10% accuracy gain over the Low Complexity split at 64k data items. A similar trend is present in the CTI MCQ dataset, where the Low Complexity split shows negligible accuracy scaling with increasing data size. Conversely, for the LEXam datasct, only the Low Complexity split demonstrates improved performance with scale. We hypothesize that this is consequence of weaker teacher performance, which leads to diminished accuracy on more complex data.
11

---

Published in Transactions on Machine Learning Research (03/2026)
Accurncy (Gemma 48-IT++ (TRA)
Accuracy Accuracy (Gemma 4B-PT LORA)
CTI MCQ
CTI RCM
LEXam
GSMBK
0.65
0.37
Complexity Type
0.36
0.65
Low Complexity
0.6
0.35
0.6
High Complexity
0.6
0.34
Complexity
0.58
0.55
0.33
0.5
4k
16k
64k
4k
16k
64k
4k
16k
64k
4k
16k
64k
Data Size
Data Size
Data Size
Data Size
Figure 7: Downstream Impact of Data Complexity on Mean Accuracy. For GSM8k and CTT MCQ, higher complexity data improves performance and scaling. Conversely, for L.EXam, only the Low Complexity split shows improvement, suggesting that complex data can be detrimental when the teacher model is weak.
5 Discussion
Synthetic Data Generation Has No Single Optimal Solution. "Data" is frozen reflection of reality as it was could be. No wonder then, that there is single "optimal" way to generate an infinite mumber of possibilities. Our extensive experiments show that the impact of different data properties, e.g., complexity and diversity, depends on the target domain, the model, use case, scale, and likely many other factors. Instead of looking for silver bullets, we should thus design our synthetic data systems to be as flexible as the worlds we intend to capture. With Simula we offer a system that maintains explainability and control at scale, giving practitioners the tools to customize synthotic data to fit their unique requirements.
Synthetic Data Evaluation is a Multi-faceted Challenge. The evaluation of synthetic data is fundamentally challenging due to the ambiguity of its core objectives, the coarse level of existing metrics, and its disconnect from practical context. Key properties describing "good data" are ambiguously defined and inherently entangled. For instance, one could argue that covering rare domain instances falls under "diversity." However, one could just as easily attribute these to "complexity." Similarly, the challenge of generating samples faithful to many possibly conflicting requirements falls as much under "quality" as the other two axes. One could thus argue that an ideal evaluation target does not exist; only context-specifie trade-offs. While established intrinsic evaluation metrics like embedding- based cosine distance provide a high-level signal, they fail to provide actionable insights or comparisons. By introducing alternatives like node-level taxonomic coverage and reasoning- based complexity scores, we strive to make intrinsic evaluation more pragmatic. Tronically, the very mechanisms that create a need for synthetic data- novelty, rareness, privacy make it challenging to robustly assess any synthetic data system against canonical references. Although we evaluated mix of niche and popular benchmarks, availability of more high-quality, specialized references would benefit this field.
The Life Cycle Costs of Synthetic Data. In our experiments we compared various Simula variants to a baseline that requires up 5x fewer inference calls per data point. Ignoring the one-off costs of creating a deeper taxonomy, this implies using the baseline system. Nevertheless, this "cheaper" larger dataset is likely more expensive when taking into account the full data life cycle. As we saw in Figure 6, data generated using Simula can be both more efficient and reach higher total performance. Since the costs of training are significantly higher than those of doing inference, smaller datasets are preferred even they are more expensive to generate. When using synthetic data to evaluate model preparedness, larger datasets naturally require more evaluation steps.
could generate five times the amount of data for fixed generation budget
Limitations and Future Directions. This work presented a comprehensive analysis of different factors relevant to synthetic data generation and evaluation. Nevertheless, our study has several limitations worth discussing. We conducted our experiments using a single model family generate data (Gemini 2.5 Flash). As shown in Section 4.3, the teacher model used in our ablations has a significant performance gap with the student model even after finetuning, making it unlikely to be bottleneck downstream performance.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Additionally, each data point in Simula is generated from first principles using reasoning- reasoning-based components, we expect outpuis to naturally improve with the underlying models powering the system.
To isolate the effects of different system components, we fixed the student model size and finetuning method. This approach is grounded in established scaling laws showing that performance improves with better data, regardless of model scale (Kaplan et al., 2020; Hoffmann et al., 2022). While the improvements from using more data can diminish, these reflect diminishing marginal returns due to data redundancy (Muennighoff et al., 2023) or quality (Chen et al., 2025), not platean in which larger models cease to benefit from more better data. We therefore expect our findings to hold across various model scales. Following Schulman & Lab (2025), it is further reasonable to assume that LoRA finctuning approaches full finetuming performance for the evaluated data regimes. Comparing reinforcement learning to supervised finetuning on our target. tasks remains an interesting experiment for future work.
Although the experiments presented here focus exclusively on text-based synthotic data, we see no reason why the same system could not be used for other modalities. We leave the exploration of these for future work.
6 Conclusion
Traditionally powered by data abundance, AI progress is at junction; just as its potential is becoming evident, the specialized data needed to realize it is unlikely to be generated by humans (Villalobos et al., 2024). Syuthetic data is primed to play a central role in new AI leaps enabling the crcation of specialist systems, equitable access, fairness, and preparedness evaluation. Beyond locusing on what it means to generate "good" data, this work emphasizes the importance of aligming mechanism design to practical requirements. With Simula, we provide a reasoning-fe and controllable synthetic data at scale. Aided by this controllable framework, our in-depth experiments highlighted just how idiosyncratic the relation between "good" data and downstream performance is. We therefore urge the community to resist searching for "silver bullets," " and instead embrace synthetic data's 's deep context-dependent. through controllable, reasoning-driven approaches.
framework capable of generating explainable
Broader Impact Statement
Simula a general-purpose framework for generating synthetic datasets. By offering fine-grained control and stecrability, Simula introduces a dual-use potential. While our work motivatod by desire to overcome latent biases and increase explainability in data generation, we recognize that these same capabilities could be leveraged by malicious actors to produce nefarious content or reinforce existing biases. For instance, the ability to preciscly manipulate factors of variation could be used to generate datascts that promote harmful stereotypes or misinformation, depending on the intentions of the steering agent (M3 or human).
We highlight that Simula integrates several mechanisms designed to mitigate these risks from happening inadvertently. Firstly, it promotes transparent generation by transforming the opaque nature of woal-world data into "white box" problem. This allows for a clearer understanding and control over potential biases. Secondly, it provides anditing tools that can be used as pragmatic post-checks. The use of taxonomies enables the measurement of data coverage, offering a practical method to detect imbalanced output distributions of specific factors of interest. Similarly, our "calibrated attribute scoring" technique can be employed to identify undesirable shifts in sensitive attributes, such as assessing the level of prejudice. These inherent features aim to facilitate the detection and rectification of biases, thereby promoting responsible and ethical synthetic data generation.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Author Contributions
Tim R. Davidson: Led the majority of the writing of the mamuscript, including system formalization, literature review, experimental selup, and evaluation; designed and evaluated the double critic-rejection sampling approach; developed and evaluated the calibrated complexity scoring metric; introduced data coverage metrics; developed several optimizations for the taxonomy generation algorithm (o.g., best-of-N, criliquing); conducted intrinsic evaluations provided core insights across the other parts during his internship at Google.
Benoit Seguin: Led engineering for Simula; implemented core components for modular, scalable data generation; developed the core library for prompting and large-scale inference used throughout the experiments; established the fine-tuning framework for downstream evaluation.
Enrico Bacis: Built the library for unified data representation across diverse datasets and researched suitable datascts; developed the evaluation framework for few-shot experiments and conducted these experiments.
Cesar Ilharco: Developed metrics for evaluating taxonomies and conducted the corresponding experiments; researched and collected suitable datasets for taxonomy evaluation.
Hamza Harkous: Founded and led research for Simula; designed the core system for end-to-end seedless data generation; researched and implemented the original system components, including mixing strategies, taxonomy building, meta-prompting. critiquing, and refinement; conducted downstream fine-tuning experiments for evaluation; developed the data generation setup for ablation studies.
All authors:
Contributed to project ideation, technical discussions, and manuscript writing and revision.
Acknowledgments
We would like to thank Nina Taft and Amanda Walker for continuous support and advice the project and the paper. We also thank Coran Corbett for engineering contributions to Simula and Etienne Pot for supporting us in the usage of Kauldron for Gemma model training.
References
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprinl arXiv=2413 1Tf::2412.0800 08905, 2024.
Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, and Nidhi Rastogi. Ctibench: A benchmark for evaluating Ilms in cyber threat intelligence. Advances in Newral Information Processing Systems, 37:50805 50825, 2024.
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL https://www-cdd http:///www..cc...toopp.... anthropic. com/ Card _ac___rrrr__aaaa Claude_3., 3d
de@ba9b01 c9a.000019aacccc cct.fffcc3bbt007bccc
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXic:2204.0588 2022.
Michele Banko, Brendon MacKeen, and Laurie Ray. A unified taxonomy of harmful content. In Seyi Akiwowo, Bertie Vidgen, Vinodkumar Prabhakaran, and Zeerak Waseem (eds.), Proceedings of the Fourth Workshop Online Abuse and Harms, pp. 125 137, Online, November 2020. Association for Computational Linguistics. doi: /0.18533/1122020.arvs
URL https: https://aclantholoog
arg/2020..o a0000.aall............... alw-1.18 16/.
Tolga Bolukbasi, Kai-Wai Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29, 2016.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Lc, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint 27X2::2407.21787, 2024.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accounlability and transparency, pp. 77-91. PMT.R, 2018.
Olaf Bánki, Yury Roskov, Markus Döring, Geoff Ower, Diana Raquel Hernández Robles, Camila Andrea Plata Corredor, Thomas Stjernegaard Jeppesen, Ari Örn, Thomas Pape, Donald Hobern, Stephen Garnett, Holly Little, R. Edward DeWalt, Keping Ma, Joe Miller, Thomas Orrell, Rolf Aalbu, John Abbott, Carlos Aedo, and Others. Catalogue of life, 1 2025. ISSN 2405-8858. URL attps://wies https://ww.chhcklisstteeee wwwckhistbankk org// dataset/307664.
org/
Hao Chen, Abdul Waheed, Xiang Li, Yidong Wang, Jindong Wang, Bhiksha Raj, and Marah I Abdin. On the diversity of synthetic data and its impact on training large language models. arXiv preprint arXiv:2410.1588 2024.
Zhengyu Chen, Siqi Wang, Teng Xiao, Yudong Wang, Shiqi Chen, Xunliang Cai, Junxian He, and Jingang Wang. Revisiting scaling laws for language models: The role of data quality and training strategies. In Proceedings of the 63rd Annual Meeling of the Association for Compulational Linguislies (Volume 1: Long Papers), pp. 23881 23899, 2025.
Zhihong Chen, Feng Jiang, Junying Chen, Tiannan Wang, Fei Yu, Guiming Chen, Hongbo Zhang, Juhao Liang, Chen Zhang, Zhiyi Zhang, et al. Phoenix: Democratizing chatgpt across languages. arXie preprint arXiv:2344.1045, 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint ar XrXi::21 10. 14168, 2021.
Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej, Tamay Besiroglu, and David Owen. The 21015.
rising costs of training frontier AI models, 2025. URL
Gary N. Curtis. Taxonomy of logical fallacies. http:://ww..fllllgyleeee..rrgg// hhmml,, 2023.
Accessod 2025-01-23.
Tim R. Davidson, Viacheslav Surkov, Veniamin Veselovsky, Giuseppe Russo, Robert West, and Caglar Gulechre. Self-rocogiition in language models. In EMNLP, 2024a.
Tim Ruben Davidson, Veniamin Veselovsky, Michal Kosinski, and Robert West. Evaluating language model agency through negotiations. In ICLR, 2024b.
Arpad E Elo and Sam Sloan. The rating of chessplayers: Past and present, 1978.
Kawin Ethayarajh. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), EMNLP-IJCNLP, 2019.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In International Conference on Machine Learning, pp. 5988 6008. PMLR, 2022.
Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A Alemi, Roman Novak, Peter J Liu, Izzeddin Gur, Jascha Sohl-Dickstein. Leslie Pack Kaelbling, Jaehoon Lee, et al. Scaling exponents across parameterizations and optimizers. ICML, 2024.
Yu Fan, Jingwei Ni, Jakob Merane, Etienne Salimbeni, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Florian Geering, Oliver Dreyer, et al. Lexam: Benchmarking logal reasoning 340 law exams. Xr preprinl Xrxi:22505.122 12864, 2025.
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Prompt- breeder: Self-referential self-improvement via prompt evolution. In JCML, 2024.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning. In TCLR, 2023.
Richard V Gaines, Catherine Skinner H., Eugene F. Foord, Brian Mason, Abraham Rosenzweig, and Vandall T. King. Minerals arranged by the new dana classification. https: http:///mmiieeal..mmmmmaaaeaeeee shtml, 1997-2024. Accessed 2025-01-23.
Gemini, Rohan Anil, Sebastian Borgeaud, Jean-Baptisn Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multinodal models. arXiv preprint arxi::231, 2.
1 1805, 2023.
Gemini, Petko Georgiev, Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint 2iv:22403. 200.00300, 2024.
Gemma, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arxiv preprinl arXi:22488. 00.440088111888 2024.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chalgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint 0rXiv:2501.12448 2025.
Himanshu Gupla, Kevin Scaria, Uijwala Anantheswaran, Shreyas Verma, Mihir Parmar, Saurabh Arjun Sawant, Chitta Baral, and Swaroop Mishra. Targen: Targeted data generation with large language models. COLM, 2024.
Alex Havrilla, Andrew Dai, Laura O'Mahony, Koen Oostermeijer, Vera Zisler, Alon Albalak, Fabrizio Milo, Sharath Chandra Raparthy, Kanishk Gandhi, Baber Abbasi, al. Surveying the effects of quality, diversity, and complexity in synthetic data from large language models. arXiv preprint arxiv:241 2. 2f.0294,,00000, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math datasct. In NewrIPS, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two Lime-scale update rule converge to a local nash equilibrium. NewrIPS, 2017.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Newral Information Processing Systems, 2022.
Tom Hosking, Phil Blunsom, and Max Bartolo. Human foodback not gold standard. In ICLR, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
Yuzheng Hu, Pingbang Hu, Han Zhao, and Jiaqi W Ma. Most influential subset selection: Challenges, promiscs, and beyond. NeurIPS, 2024.
Audrey Huang, Adam Block, Dylan Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jordan T Ash, and Akshay Krishnamurthy. Self improvemntt in language models: The sharpening mechanism. In NewrIPS Workshop on Mathematics of Modern Machine Learning, 2024.
16

---

Published in Transactions on Machine Learning Research (03/2026)
Jonas Hübotter, Sascha Bongni, Ido Hakimi, and Andreas Krause. Efficiently learning test-time: Active fine-tuning of llms. In NeurIPS Workshop on Mothemalics of Modern Machine Learning, 2024.
International Union of Pure and Applied Chemistry (TUPAC). Periodic table of the elements. https:// iupac. www.wwattw/weeeeoeeeeeeeeeeeeeet weeddd//perieee weights current as of 2022-05-04.
2022. Accessed: 2025-01-23; Standard atomic
Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan github. io/posts/muon/.
Angelika Kaplan, Thomas Kühh, Sebastian Hahner, Niko Benkler, Jan Keim, Dominik Fuchf, Sophie Corallo, and Robert Heinrich. Introducing evaluation method for taxonomies. In Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering, pp. 311-316, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprinl WrXiv:2001.08366 2020.
Abhinav Ramesh Kashyap, Thanh-Tung Nguyen, Viktor Schlegel, Stefan Winkler, Sie-Kiong Ng, and Soujanya Poria. A comprehensive survey of sentence representations: From the bert epoch to the chatgpt era and beyond. arXiv preprint 0rXiv:2305.12444 2023.
Dennis Kundisch, Jan Muntermann, Anna Maria Oberländer, Daniel Rau, Maximilian Röglinger, Thorsten Schoormann, and Danicl Szopinski. An update for taxonomy designers: methodological guidance from information systems research. Business & Information Systems Engineering, pp. 1 19, 2021.
Alon Lavie and Abhaya Agarwal. METEOR:
An automatic metric for MT evaluation with high levels of correlation with human judgments. In Chris Callison-Burch, Philipp Koehn, Cameron Shaw Fordyce, and Christof Monz (eds.), Proceedings of the Second Workshop on Statistical Machine Translation, pp. 228-231, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https: //aclanthology. .orgi...rg//eee org/W07-0734/.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. RLAIF vs. RLHF: scaling reinforce- ment learning from human feedback with AT feedback. Tn ICML, 2024a.
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. Gecko: Versatile text embeddings distilled from large language models. arxiv preprint NV0:240332002P 2024b.
Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala. Amu- manchipalli, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. LLM2LLM: boosting lims with novel iterative data enhancement. In ACL, 2024c.
Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingvao Ai, Ziyi Ye, and Yiqun Liu. Llms-as- judges: A comprehensive survey on lamebased evaluation methods. CoRR, 2024a.
Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. orXiv preprint arXiv:2402. 0004402.10004, 13064, 2024b.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Phoebe Liu. The ai billionaire you've never heard of, September 2025. URL https://ww.forbbs.cc www.rrrrerroooe-eee---a-r------r
17

---

Published in Transactions on Machine Learning Research (03/2026)
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ractsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 4114 4124. PMLR, 2019.
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. #instag: Instruction lagging for analyzing supervised fine-Luning of large language models. In ICLR, 2024.
Sheng Lu, Shan Chen, Yingya Li, Danielle Bitterman, Guergana Savova, and Iryna Gurevych. Mcasuring pointwise V-usable information in-context-ly. In Findings of the Association for Compulational Linguisties: EMNLP 2023, pp. 15739-15756, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, al. Self-refine: Iterative refinement with self-feedback. NewrIPS, 36, 2024.
Max Marion, Ahmet Üstūn, Luiza Pozzobon, Alex Wang, Marzieh Fadace, and Sara Hooker. When less is more: Investigating data pruning for pretraining Ilms at scale. In NewrIPS Workshop on Attributing Model Behavior at Scale, 2023.
Anay Mehrotra, Manolis Zampelakis, Paul Kassianik, Blaine Nelson, Hyrum S Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box LLMs automatically, In NewrIPS, 2024.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Newral Information Processing Systems, 36:50358 1155355--5037,, 2023.
Curtis Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. In Thirly-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https:///penrevieee net/
OpenAL, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXic:2303.0877 08774, 2023.
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. In NewrlPS, 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Rajvardhan Patil, Sorio Boit, Venkat Gudivada, and Jagadeesh Nandigam. A survey of text representation and embedding techniques in nlp. IEEE Access, 1.36120-36146, 2023.
Katie Paul and Anna Tong. AI training data, April 2024. muside-big-teche wedds-undergroundd-re seeendd-raeebuu---ee buw-aittraining data-2024-044
Inside big techs underground race to buy URL https://www wwwters.e.err..... com/technology/
Judea Pearl. A probabilistic calculus of actions. In UAI, pp. 454 462, 1994.
Judea Pearl and Eliss Bareinboim. Transportability of causal and statistical relations: A formal approach. Proceedings of the AAAT Conference on Artificial Intelligence, pp. 247-254, 2011.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Scan Welleck, Yejin Choi, and Zaid Harchaoui, Mauve: Measuring the gap between neural text and human text using divergence frontiers. NewrIPS, 2021.
18

---

Published in Transactions on Machine Learning Research (03/2026)
Crystal Qian, Michael Xieyang Liu, Emily Reif, Grady Simon, Nada Hussein, Nathan Clement, James Wexler, Carrie J Cai, Michael Terry, and Minsuk Kahng. The evolution of Ilm adoption in industry data curation practices. arXiv preprint arXiv:2412.16088 2024.
Emily Reif, Crystal Qian, James Wexler, and Minsuk Kahng. Automatic histograms: Leveraging language models for text dataset exploration. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2024. ISBN 9798400703317. doi: 10.1145/
3663995.365079.
Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Polder-Holder, Jakob Nicolaus Foerster, et al. Rainbow teaming: Open-ended generation of diverse adversarial prompts. In NewrIPS, 2024.
William Saunders, Catherine Ych, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.0500 2022.
Robin M Schmidt, Frank Schneider, and Philipp Hennig. Descending through a crowded belly--benchmarking deep learning optimizers. In International Conference on Machine Learning, pp.
PMLR, 2021.
John Schulman and Thinking Machines Lab. Lora without regret. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tmll 20255999.
hat.:/aaaciiiesaabboo//oooooo blog//ora/........
Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Synthetic prompting: Generating chain-of-thought. demonstrations for large language models. In ICML, volume 202, pp. 30706- 30775. PMLR, 2023.
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin DURMUS, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda. Zhang, and Ethan Perez. Towards understanding sycophancy in language models. In ICLR, 2024.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. Proceedings of the 35th International Conference on Machine Learning, pp. 4603 1618- PMLR, 2018.
Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jachoon Lee, Kelvin Xu, Aaron T. Parisi, Abhishek Kumar, Alexander A. Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izæeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. Beyond human data: Scaling self-training for problem-solving with language models. TMLR, 2024a.
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Adelani, Jian Gang Ngui, Daniel Vila- Suero, Peerat Linkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, al. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation. arXiv preprint arXin:2412.03300 2024b.
Charlic Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling I1m test-time compute optimally can be more effective than scaling model parameters. arXiv preprint ar::::2408. 03314, 2024.
Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The good, the bad, and the greedy: Evaluation of Ilms should not ignore non-determinism. arXiv preprint 0rXiv:2407.10457, 2024.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 19523 19536, 2022.
19

---

Published in Transactions on Machine Learning Research (03/2026)
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannanch Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), EMNLP, 2020.
Daniel Szopinski, Thorsten Schoormann, and Dennis Kundisch. Criteria as prelude for guiding taxonomy evaluation. In HICSS, pp. 10, 2020.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit. Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feodback. In EMNLP, 2023.
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari S. Morcos. D4: Improving LLM pretraining via document de-duplication and diversification. In NewrIPS, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anae Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:230. 199921, 13971, 2023.
Muhammad Usman, Ricardo Britto, Jürgen Börstler, and Emilia Mendes. Taxonomics in software enginecring: A systematic mapping study and a revised laxonomy development method. Information and Software Technology, 85:43-58 59, 2017.
Boris van Breugel, Nabeel Scedat, Fergus Imrie, and Mihaela van der Schaar. Can you rely on your model evaluation? improving model evaluation with synthetic test data. NewrlPS, 2024.
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: will we run out of data? limits of Ilm scaling based on human-generaled data. In ICML, 2024.
Milana Vinn and Krystal Hu. Exclusive: Scale AT's bigger rival Surge AI seeks up to $1 com.ttrre.ccmmmusseeeee/////
billion capital raise, sources say, July 2025. URL scale-ais-
Vijay Viswanathan, Kiril Gashteovski, Carolin Lawrence, Tongshuang Wu, and Graham Neubig, Large language models enable few-shot clustering. TACL, 12:321-333, 2024.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self- gelf-generated instructions. In ACL, 2023.
Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadch, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and Iason Gabriel. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference Fairness, Accountabilily, and Transparency, FAccT '22, pp. 214-229, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393522. doi: 10.1145/200 001445/3311141 253114663533066
URL https://doi /ttp::////ooooo/////o///// 1145/3531146. 3533088.
Kyle Wiggers. can afford,
A1 training data
has a price tag that only big tech com/2024/06/01/
URL https://techeruuchh ai-training-data- data--at---aaappp hawe--aaa-----taaag-e-a only -affo-aa--affoddd....
June 2024.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS; selecting influential data for targeted instruction tuning. In ICLR, 2024.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can 1mms express their uncertainty? an empirical evaluation of confidence elicitation lmm.. In ICLR, 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Fmpooweing large language models to follow complex instructions. ICL.R, 2024a.
20

---

Published in Transactions on Machine Learning Research (03/2026)
Zerui Xu, Fang Wu, Tianfan Fu, and Yue Zhao. Retrieval-reasooinng large language model-based synthetic clinical trial generation. Xiv preprinl arxi222 10. 12476, 2024b.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Kegin Bao, Kexin Yang. Le Yu, Lianghao Deng, Mei Li, Mingleng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL pattps//aarce org.orgg//a............... aaaaaabss//////////// 09388.
Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. CoRR, 2024.
Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias. NewrlPS, 2023.
Yunhao Zhang and Renée Gosline. Human favoritism, not ai aversion: People's perceptions (and bias) toward generative ai, human experts, and human-gai collaboration in persuasive content generation. Judgment and Decision Making, 18:e41, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging LLM-as-a- judge with MT-bench and chatbot arena. In NewrIPS, 2023.
21

---

Published in Transactions on Machine Learning Research (03/2026)
Appendices
A Related Work
23
A.1 Synthetic dataset evaluation.
23
A.2 Synthetic dataset generation.
24
B Reasoning-driven Taxonomy Generation and Evaluation
25
B.1 Defining Taxonomy Evaluation Metrics
25
B.2 Taxonomy Evaluation Results
26
B.3 Limitations
27
B.4 Taxonomy Generation Algorithm
28
B.5 Qualitative Examples
29
C Synthetic Data Generation Algorithm
30
D Evaluating Double-Critic Steps
31
D.1 Multi-Lingual MMLU
31
E Calibrated Complexity Scoring
32
E.1 Open Generation: MATH
32
E.2 Multiple-Choice Generations: MMLU Global
32
E.3 Calibrated Complexity Scoring Algorithm
34
F
Downstream Experiments
35
F.1
Training Configuration and Hyperparameters
35
F.2 Additional Evaluation Results
35
F.3 Choice of Optimizer
36
F.4 Choice of the Teacher Model
37
22

---

Published in Transactions on Machine Learning Research (03/2026)
A Related Work
A.1 Synthetic dataset evaluation.
Early Evaluation Methods. When evaluating synthetic dataset, we can differentiate wee comparalive and intrinsic evaluation. The former expects the availability of a reference dataset. In the case of "closed" tasks that allow for narrow comparisons, we can directly compare model-generated outputs to reference solutions, e.g., single-word answers, translations, or summaries. Originally, these were done directly on the observed outputs, e.g., by comparing overlapping words or n-grams (Papineni et al., 2002; Lin, 2004; Lavie & Agarwal, 2007), inter alio.
Rise of Semantic Metrics. As tasks might have multiple different, but semantically equivalent solutions, evaluations shifted to embedding-basee based approaches (Patil et al., 2023). Such approaches can even be used in the absence of one-to-one reference mappings, by comparing output distributions on a dataset level (Heusel et al., 2017; Pillutla et al., 2021). Although ombedding-bassee approaches allow for automated evaluation, they can struggle in specialized domains and miss semantic subtleties (Rthayarajh; 2019; Kashyap et al., 2023).
Auto Verifiable Tasks, Then, we have class of problems that allow for many trajectories ending in final solution whose correctness can be automatically verified quickly, e.g., program synthesis and mathematical reasoning (Song et al., 2024), or negotiation games (Davidson et al., 2024b).
The Challenge of Open Tasks. What remains is a large class of "open" " tasks, for which there are no well-defined criteria of correctness, e.g., creative writing, advice, and most visual media. For these, we largely rely on the reasoning capabilities of human annotators to provide quality references and to obtain pairwisc preference labels (Christiano el. al., 2017; Bai et al., 2022). However, creating such datasets manually is expensive, time-consuming, and error-prone (Chen et al., 2023; Gilardi et al., 2023; Hosking et al., 2024).
Emergence of LLM Judges. Recently, the growing capabilities of frontier models have opened up the possibility of model-based reasoning (Lee et. al., 2024a; Zheng et al., 2023; Li et al., 2024a; Saunders et al., 2022; Wang et al., 2023; Madaan et al., 2024), inter alia. As humans and AT start to prefer AT-generated text (Zhang & Gosline, 2023; Panickssery et al., 2024), and Al prefers text by the strongest AI models (Davidson et al., 2024a), model-based evaluation is set to become increasingly prevalent.
Intrinsic Evaluation Metrics. In the absence of reference data, one must rely on èntrinsic evaluation. As described in the introduction, typically focus on the axes of quality, diversily, and complexity. When we treat quality as how well a set of data points meets stated requirements, we are quickly forced to rely on reasoning- based evaluation. Instead, if understood as the utility of a dataset for a specific downstream task, we can directly measure the lift in downstream performance, c.g., through classification accuracy. Diversity is often approximated using pairwise cosine similarity after embedding outputs inlo a higher dimensional space (Yu et al., 2023; Gupta et al., 2024). Without grouping the data using an appropriate clustering step, c.g.; semantic clusters, average statistics are sensitive to outliers and fail to differentiate between global and local diversity. Crucially, such diversity statistics provide few semantic, actionable insights. Attempts to automate complexity scoring range from measuring the length of outputs (Shao et al., 2023), or the relative entropy over output alternatives (Ethayarajh et al., 2022; Lu et al., 2023), to generating a large solution set and using a reward model to estimate correctness (Snell et al., 2024). Reasoning-
approaches instead directly query a model to provide "difficulty" score (Li et al., 2024a). Because models are generally poorly calibrated (Zheng et al., 2023; Tian et al., 2023; Xiong et al., 2024), and difficulty is a relative concept, such absolute scores can be noisy.
Our Reasoning-Based Approach. Our work continues the trend of incorporating model-based reasoning to evaluate data. Instead of using approximate statistics based on output embeddings, reasoning- based approaches provide explainable traces that can be audited and controlled. Mapping out a global coverage space using taxonomies allows end-users to quickly evaluate their generated data meets the appropriate global diversity requirements (Sections 2.1, 2.3). We further carefully tested popular assumptions about the use of model-based critics through a series of controlled experiments. On the evaluated datasets, we find that critic-rejection sampling of synthetic outputs consistently succeeds in increasing average sample quality
23

---

Published in Transactions on Machine Learning Research (03/2026)
(Sections 2.2, 3.1, 4.1) across different datasets and complexity levels. We also found that model-assigned complexity scores are promising proxies for human notions of difficulty, and correlate well with models' critic and generation capabilities (Sections 2.3, 4.1, and Appendix E).
A.2 Synthetic dataset generation.
Seed-Based Expansion Methods. Popular synthetic data methods generally only account for subset of the quality, diversity, and complexity axes (Havrilla et al., 2024). For example, Wang et al. (2023) start with a set of seed examples and iteratively expand them using hand-crafted semantic diversity prompts. Xu et al. (2024a) similarly expand seed examples, but focus on both diversity and complexity. The main quality check performed is to ensure that these expanded examples do not become degenerative. An attempt at increasing global diversity is done by maximizing pairwise cosine similarity through rejection sampling. The authors note that expanding semantic diversity and complexity are both positively correlated with downstream performance after fine-tuning.
Factor Identification Approaches. Reif et al. (2024); Chen et al. (2024); Viswanathan et al. (2024); Lu et al. (2024) use seed examples to automatically detect relevant factors through iterative sampling of the target dataset, after which factors are extracted using reasoning modules. Other approaches side-step the need for seed examples by manually inspecting or reasoning about a target dataset to find globally relevant factors of variation (Yu et al., 2023; Gupta et al., 2024; Samvelyan al., 2024). They then sample from these factors for conditional generation. Relevant to our framework is work done by Li et al. (2024b), who attempt to generate single, large laxonomy to cover a variely of topics. Inspired by curriculum- learning in human education systems, the authors generate a variety of learning modules to implicitly vary complexity.
Leveraging the Critic Gap. Many have by now pointed out the apparent gap between current models' generative and verification capabilities (Huang et al., 2024). This gap allows models to act as critics of their own outputs (Saunders et al., 2022; Madaan et al., 2024) and has been successfully used by many of the above methods (Lee et al., 2024c; Gupta al., 2024; Samvelyan et al., 2024; Chen et al., 2024), among others.
Scaling Time Compute. Recent efforts in scaling test-time computation show that models are capable of gencrating correct outputs even for complex questions, given enough attempts (Song et al., 2024; Brown el al., 2024). Yet, how to best scale such a test-time computation budget depends on the complexity of the particular problem (Snell et al., 2024). The authors suggest that "easier" tasks most benefit from exploiting an existing output attempt through itcrative refinement, whereas more "difficult" tasks benefit. more from exploring larger proposal distribution.
Our Orchestration Approach. With Simula, we build on many of the existing insights into the merits of optimizing quality, diversity, and complexity for downstream performance. In contrast to existing methods, we explicitly orchestrate the generative process on dataset level to increase global control and explainability. We carefully split. the data generation process into separate steps, i.e., global diversity, local diversity, complexity, and correctness, allowing end users to tailor datasets to their specific requirements (Section 2). In doing so, it becomes possible to allocate computational resources where they are most desired, c.g., by generating more meta prompts to increase the proposal distribution of complex samples or adding additional critic steps refine the outputs.
24

---

Published in Transactions on Machine Learning Research (03/2026)
B Reasoning-driin Taxonomy Generation and Evaluation
To address the inherent challenges of assessing taxonomy quality and completeness, we use critic-model based framework for evaluation. Traditional taxonomy evaluation often relies on manual expert review, which is time- consumins,,,, expensive, and often difficult to attain. Our proposed framework leverages the capabilities of large language models as "critic models" provide a more automated, scalable, and reproducible evaluation.
B.1 Defining Taxonomy Evaluation Metrics
First, a hierarchical representation for each expert taxonomy (TB) or model--enerated taxonomy (TM²) is provided, and the critic model classifies each node into the following categories:
Good and Overlapping: The node is good (well-defined, relevant to its parent node, and fits appropriately within the overall taxonomy) AND overlapping (there is a semantically equivalent node in the other taxonomy which represents the same concept).
Good and Exclusive: The node good (well-defied,, relevant to its parent node, and fits appropriately within the overall taxonomy) AND NOT overlapping (there no semantically equivalent node in the other taxonomy which represents the same concept; this concept appears uniquely in Ihis laxonomy).
The node is duplicate within its own taxonomy, there is another node in the same taxonomy representing the same concept.
Redundant:
The node is irrelevant, poorly defined, misclassified, or otherwise inappropriate for its position in the taxonomy.
Bad:
Based on the critic model's classifications, we compute several quantitative metrics evaluate each laxonomy:
This metric mcasures the extent to which TMs covers the concepts present in Tg- The LM critic assesses, for each node (concept) in TE, whether a semantically equivalent node exists in TMa-- This serves as a measure of coverage and recall, quantified by the ratio (Good and Overlapping) / (Total Good) in TE. Here, Total Good Good and Overlapping + Good and Exclusive.
Completencss:
This metric assesses the proportion of relevant and correct nodes within TMB- The LM critic examines each node in TM3 to judge its relevance to the topic and whether it constitutes a non-redundant entry. Fewer irrelevant or incorrect nodes result in greater soundness. This serves as a measure of precision, quantified by the ratio (Total Good) / (Total Nodes) in TM3. Here, Total Good = Good and Overlapping + Good and Exclusive.
Soundness:
It is worth noting here, that there are different taxonomy types in practice. Grounded taxonomies are typically revised over time as new empirical evidence is gathered through the scientific method. A recently accepted version in the literature can be considered closer to ground truth than a concept ual laxonomy, offers less scope for a language model to generate new terms absent new evidence. We use the following grounded taxonomies for evaluation: "Animal Phylogenetic Classes" (Bánki et al., 2025), "Periodic Chemical Elements" (International Union of Pure and Applied Chemistry (IUPAC), 2022), and "Mineral Classification" (Gaines et al., 1997-2024).
that it
Conceptual taxonomies are more subjective than grounded ones; even the definitions and usage of terminology can vary across the literature (Usman et al., 2017; Szopinski et al., 2020; Kundisch et al., 2021; Kaplan et al., 2022). We use the following conceptual taxonomies for evaluation: " "Risk of Language Models" (Weidinger et al., 2022), " "Online Hormful "ontent" (Banko et al., 2020), and "Logical Fallacies" (Curtis, 2023)). Because of this subjectivity, we additionally compute the following metrics for conceptual laxonomies:
Novelty: This metric assesses whether TMI contains relevant nodes that are not present in Tg. The LM critic identifies nodes in TMS that are not semantically equivalent to any node in TE and then
25

---

Published in Transactions on Machine Learning Research (03/2026)
judges the relevance of these novel nodes. A higher number of relevant novel nodes indicates greater novelty. We define this as the ratio (Good and Exclusive in TMs) / (Total Good in TE).
This represents the total number of "good" items in Th relative to the total number of "good" items in Tg. Coverage is equivalent to Completeness + Novelty and provides a comparative metric of the number of sound items. It follows that a coverage value greater than 1.0 indicates that TM3 covers more relevant and correct items than TE within the global space for the given taxonomy.
Coverage:
A more elaborate description of the grounded and conceptual taxonomies used can be found in B.2.
B.2
Taxonomy Evaluation Results
Table 3: Performance metrics for different taxonomies.
Type
Topic
Method
Completeness
Soundness
Novelty
Coverage
Simula 0-shot
0.749
0.980
0.865
1.614
Online Harmful Content
0.588
0.957
0.412
1.000
Conceptual
Simula 0-shot
0.726
0.919
1.679
2.405
Logical Fallacies
0.458
0.966
0.193
0.651
Simula 0-shot
0.867
1.000
0.267
1.134
Risks of Language Models
0.467
1.000
0.367
0.834
Simula 0-shot
0.458
0.926
Animal Phylogenetic Classes
0.349
0.918
Grounded
Simula 0-shot
0.993
0.987
Periodic Chemical Elements
0.775
0.864
Simula
0.762
0.340
Mineral Classification
0-shot
0.442
0.329
Description of the taxonomies from Table 3:
[Conceptual] Online Harmful Content: This typology aims to provide unified classification of harmful content. found online. It synthesizes common abuse types described by industry content. policies, policy recommendations, community standards, and health expert guidelines. The goal is to create readily usable categories for content moderation, encourage the development of accurate datasets for model training, and raise awareness of (ess-studied abuse types to improve online safety. This taxonomy categorizes different types of harmful content found online into four main groups: Hate and Harassment; Self- Inff-teft Harm; Ideological Harm; and Exploitation and further branches each into set of more specific types. (Banko et al., 2020).
This taxonomy classifies types of logical fallacics into hierarchical structure. It divides fallacies into two main branches: Formal Fallacies (errors in the structure of the argument) and Informal Fallacies (errors in the content context. of the argument). These main categories are further subdivided into numerous specific types of fallacies, such as Propositional Fallacies, Quantificational Fallacies, and various informal fallacies like Appeal to Ignorance, and Red Herring, and then branching down increasingly specific types (Curtis, 2023).
[Conceptual] Logical Fallacies:
[Conceptual] Risks of Language Models: This laxonomy identifies ethical and social risks associated with large-scale language models (LMs). categorizes these risks into six areas: Dis- crimination, Hate speech and Exclusion; Information Hazards; Misinformation Harms; Malicious Uses: Human- Computer Interaction Harms; and Environmental and Socioeconomic harms. The taxonomy distinguishes between "observed" risks (already evidenced in LMs) and "anticipated" risks
26

---

Published in Transactions on Machine Learning Research (03/2026)
(considered likely but not yet observed). The goal is to provide comprehensive framework for understanding and mitigating the potential negative consequences of LMs. (Weidinger et al., 2022).
[Grounded] Animal Phylogenetic Classes: This taxonomy represents the hierarchical classifica- tion of animals based their evolutionary relationships. It is truncated to two levels deeper into the animal kingdom, encompassing its phyla and classes. (Bánki et al., 2025).
This taxonomy organizes chemical elements into a hicrarchical structure based on their atomic number, clectron configuration, and recurring chemical properties, primarily reflecting their placement, in the periodic table. It. positions elements into groups from Group Alkali Metals through Group 18 Noble Gases, as well the Lanthanides and Actinides. Each group is further lists individual elements like Sodium (Na) or Gold (Au). The structure represents the periodic trends and shared characteristies within groups, enabling chemists to understand relationships and predict elemental behavior. (International Union of Pure and Applied Chemistry (IUPAC), 2022).
[Grounded] Periodic Chemical Elements:
[Grounded] Mineral Classification: Presented in Dana's New Mineralogy, Eighth Edition, this taxonomy is hierarchical classification system for minerals, employing a four-part numerical code to categorize each species. This system, analogous to the Limnaean taxonomy for biology, organizes minerals based on both their chemical composition and crystal structure. The first number denotes the mineral's class (e.g., anhydrous carbonates), reflecting broad compositional categories or dominant structural features (especially in silicates). The second number signifies the mineral's type, sometimes based on atomic properties, formula. The third number groups minerals with similar structural arrangements. Finally, the fourth nunber uniquely identifies the individual mineral species, such as Calcite or Magnesite within the Calcite Group. This numerical system offers a structured and expandable framework, allowing new minerals to be easily integrated while highlighting the chemical and structural relationships between different mincral species. (Gaines ct al., 1997-2024).
B.3 Limitations
Preference Bias. As discussed in the Related Work in Appendix A, there evidence that M3s prefer model-generated text over text generated by humans. In case, do not prompt the M3 lo express preference. Rather, we ask if certain nodes semantically approximate other nodes, if certain nodes are appropriate given the context. However, we did use models from the same model family for both the generation and the ovaluation. Thus, future work could repeat this experiment with separate generator and critic models.
Stochastic Sensitivity. We did not optimize prompts for each separate taxonomy. As such, the reported metrics likely represent lower bounds.
Downstream Application. While we performed comparative evaluation of synthetic and real taxonomies, it is not directly clear which are better suited for certain downstream applications.
27

---

Published in Transactions on Machine Learning Research (03/2026)
B.4 Taxonomy Generation Algorithm
Algorithm Reasoning- Driven Taxonomy Generation
Require: User instructions Target depth D, Multi-modal Model M3
Ensure: T = {Ti) set representing conceptual space y
Phase Factor Disentanglement
Extract prime factors of variation from instructions
F + A.3.ProoosePactorte
F = ,
4: > Initialize set to store generated tozonomies
5: T ++0 Ø
Phase Breadth-First Taxonomic Expansion for each factor f: E F do Initialize taronomy Tr with root node noroot t Nodoyy, fi)
10:
Ti t (Nroot
11:
> Queue for nodes at current depth level
12:
Qearr t {
13:
> Strategic plan guiding exponsion consistency
14:
+ 'Expand based on y and f:'
15:
for depth d = to D do Queue to collect nodes for the next depth Qnezt +- 0
16:
17:
18:
for cach node E Qearr do
19:
Gather global and local reasoning context C+ {y, Ancestorrr()) Siblings(n)}
20:
21:
Step Proposal (Best-of-N)
22:
Query M3 N times using context C and guidance Plan P
23:
Neaw - M3.Output;
Set of raw chvid nodes
24:
Step 2: Critic Refinement
25:
Prompt M3 to improve completeness, soundness, specificity Nrefined +- M3.Cri 113.Critique(C,,e Nraw)
26:
Add, remove, merge, edil.
27:
Add Nrefined as children of in T
28:
Qnent & Qnext U Urefinedd
29:
Step Planning (Per Level)
30:
if d D then
31:
Generale guidance for next. level based on all new nodes
32:
P + AB3.GnerratePlau Qnext)
33:
> Advance to next depth
34:
Qeurr + Quext
35:
T + TU {T
36: return
28

---

Published in Transactions on Machine Learning Research (03/2026)
B.5 Qualitative Examples
Dut
(a) Harmful Content Taxonomy
Simula
(b) Harmful Content Taxonomy
Expert
Figure 8: Comparison of Online Harmful Content Taxonomy (Simula vs. Expert).
(a) Chemical Elements Simula
(b) Chemical Elements Expert
Figure 9: Comparison of Chemical Element Taxonomy (Simula Expert).
29

---

Published in Transactions on Machine Learning Research (03/2026)
C
Synthetic Data Generation Algorithm
Algorithm 2 Agentic Data Generation
Require: Taxonomies T, User instructions y, Target size N, Model M3, Complexity ratio E [0, 1], Sampling Strategies S, Scenarios per mix K
Ensure: Synthetic Dataset 'D containing exactly N high-quality data points
Initialization
Initialize set to store validated data points +
4: Generation Loop
5: & Continue until target size is reached
6: while (DI N do
> Step Global Diversity Sampling
> Sample a single "Mix" (combination of nodes) based on strategies
M -SaapleSinglMMii((
10:
Step 2: Meta-Prommt
Generation (Local Diversity)
11:
& Generate K diverse scenarios (meta-promplss) based on the Mix M, K)
12:
P Pecenarios & M3. AMN..GeneaateScenariie
13:
& Choose one scenario al random
14:
Pmeta + RanddomSamlle(? P eccearsos)
15:
Step 3: Complexiiication
16:
if Random(0, 1) < then
17:
Inject constraints edge-cases to increase difficulty
18:
Pmett +- M3. M3. Complexify(Pmeet
19:
Step 4: Generation and Sritic-Refinemen Loop
20:
> Generate initial proposal based on meta-prompt
21:
dposnt M3. Ger.Genneateepnn (Pmeta)
22:
is satisfying false
23:
repeat
24:
Crilic evaluates if dota point fulfills mela-prompe requirements (verdict, explanation) + M3. Critique dpoint) if verdict = "satisfying' then is salis +- true
25:
26:
27:
28:
else
29:
Agentic refinement based on critic's 's explanation dpoint t- M-....finne Refine dpoint, erplanation)
30:
31:
until is f_ ying or max retrics reached
32:
if is salis f'ying then
33:
b Only add to dataset if it passed the critic
34:
D+ {dpoint)
return D
30

---

Published in Transactions on Machine Learning Research (03/2026)
D Evaluating Double-Critic Steps
D.1 Multi-Lingual MMLU
In Table 4 we show empirical critic-rejection sampling results for subset of MMLU questions on Mathematics, Computer Science, and Physics. We use the subjects' education levels (elementary, high-school, and college) as ground truth complexity categories. We evaluate performance on languages with different resource categories, o.g.: "Low", "Mid", and "High", according their recorded, written, and cataloged NLP resources (Singh et al., 2024b). We observe that our critic-rejection sampling strategy is effective for each language under each complexity condition.
Table 4: Critic Rejection Sampling Multi-Lingual MMLU Questions. We evaluate critic-reeection sampling method for MMLU questions on Mathematics, Physics, and Computer Science. We use the subject education level (elementary, high-school, and college) as the ground-truth Complexity. We display the realized change in accuracy, Mgen of following critic rejections. Also shown are the average Elo complexity score and the size of the rejected and accepted subsets.
English (High)
Korean (Mid)
Nepali (Low)
Complexity
Critic
Mgen
Elo
[D]
Mgenn
Elo
[D]
Pgen
Elo
[D
0.93
290
14
0.86
306
29
0.76
308
41
Level
v
0.98 20.01
303 22
364
0.97 20.01
301
349
0.97
20.01
304 12
337
Level 2
X
0.52 +0.07
427 #7
46
0.64
#0.06
428
53
0.58
=0.06
431 46
60
0.94 +0.01
427 #2
578
0.92 #0.01
426
571
0.93
20.01
425 a:a
564
Level
0.67 20.10
0.54 10.10
471 d:
26
20.00
473 4:4
0.89 20.02
467
278
0.88 $0.02
468
276
0.86
465 #:2
268
31

---

Published in Transactions on Machine Learning Research (03/2026)
E Calibrated Complexity Scoring
We compare modcl-assigned complexity scores against the ground-truth human annotations. We ablate model-assigned complexity scores varying the number of times each sample is scored (N) and the batch size (BS) of questions being scored simultaneously. Importantly, for fixed N, the number of samples being scored simultaneously (BS) increases the context length but reduces the number of inference passes. For example, for D = 1000, N=10 and BS=1, we require 10, 000 inference passes. Setting BS to 5 instead reduces this 10, 000/5 = 2, 000. All things equal, in practice we would thus like see a higher BS have similar or better performance than a lower BS.
E.1
Open Generation: MATH
BS=1
BS=2
BS=5
BS=10
Complexity Score
Score
ELO
N=10
Complexity Score
100
N=5
Complesty exky Score
Complexity Level
Complexity Level
Complexity Level
Complexity Level
Figure 10: Complexity score ablation MATH test set. The MATH dataset comes with ground truth complexity levels ranging from 1-3. We group by the ground truth complexity level and plot the model- assigned complexity scores for each group. We compare using average raw scores (Score) against using Elo rankings (Elo) and ablate the batch size (BS) of question scored simultaneously and the number of times each question is scored (N). All results scaled per (BS, N)-grouping to lie between and 100. For BS 1, pairwise rankings are done, so Elo rankings do not apply.
In Figure 10, we show results of comparing model-assigned complexily scores to the human-annolated ground truth (Levels 1-5). To enable side- bi-ddee comparison, we scale both the raw average scores (Score) and the computed ELO rankings (ELO) lic between 0 and 100 for cach (BS, N) grouping. As we increase the number of samples, N, clusters become better separated. Increasing BS enables the use of latent skill methods like ELO to increase consistency. We found BS N 5 to strike appropriate balance between separation and inference cost.
E.2 Multiple-Choiee Generations: MMLU Global
In Table we compare model-assigned complexity scores for a subset. of MMLU questions on Mathematics, Computer Science, and Physics. We use the subjects' education levels (elementary, high-school, and college) as ground truth complexity categories. We evaluate performance on languages with different resource categories, e.g., "Low", "Mid", and "High", according to their recorded, written, and catalogued NLP resources per Singh et al. (2024b). After running KMeans the model-assigned complexity scores, we compute the Normalized
32

---

Published in Transactions on Machine Learning Research (03/2026)
Mutual Information (NMI) and the Adjusted Rand Index (ARI) to evaluate cluster approximation of the ground truth complexity. Finally, train logistic regression model-assigned complexity scores to evaluate them as an estimator for the model's generative performance, reporting the Area Under the Curve (AUC). Similar to our findings in Appendix D.1, we find model-assigned complexity scoring robust across several languages. Choosing BS
5 again emerge as reasonable hyper-parameters..
Table 5: Multi-lingual MMLU Complexity Scoring. We use exam questions for topics Mathematics, Physics, and Computer Science, education levels elementary (Mathematics only), high-school, and college. 'Tkking the education level as our ground-truti truth complexity level (1-3), we run KMeans on the complexity scores generated by the model and compute the Normalized Mutual Information (NMI) and the Adjusted Rand Index (ARI). Finally, compute the Area Under the Curve (AUC) of using the complexity scores as an estimalor for the model's generative performance. We ablate the batch size (BS) of exam questions scored simultaneously and the number of times each exam question is scored (N).
English (High)
Arabic (High)
Dutch (High)
Korean (Mid)
Nepali (Low)
NMT
ARI
AUC
NMT
ARI
AUC
NMI
ART
AUC
NMI
ARI
AUC
NMT
ARI
AUC
0.27
0.23
0.61
0.32
0.27
0.60
0.32
0.28
0.59
0.32
0.27
0.59
0.32
0.27
0.56
0.33
0.33
0.36
0.31
0.63
0.38
0.32
0.60
0.37
0.31
0.59
0.38
0.33
0.59
0.37
0.32
0.57
0.42
0.36
0.64
0.42
0.35
0.62
0.41
0.35
0.59
0.40
0.34
0.60
0.41
0.35
0.57
0.42
0.37
0.63
0.41
0.36
0.62
0.40
0.34
0.60
0.43
0.38
0.61
0.43
0.37
0.57
33

---

Published in Transactions on Machine Learning Research (03/2026)
E.3 Calibrated Complexity Scoring Algorithm
Algorithm Calibrated Attribute Scoring
Require: Dataset D, Attribute guidance A, Model M3, Samples per item K, Batch size BS Ensure: Scoring mapping Sfmal + (scoreraw ranketo) for all d D
Phase Batch Preparation
Create batch schedule ensuring every item oppears in K varied batches
B + PrepareBatches(2) BS, K)
4: & Initialize map to store lists of raw scores for each item
5: RawScores t :d: [] for d E D}
Phase Batch-wise Relative Scoring for each batch B B do
Prompt M3 to score items relative to others within this batch 188.ScoreSingleBatchl B, A)
ScoresB
10: > Collect row scores for aggregalion.
11:
for each (d, score) E Scores do
12:
Appond score to RawScores|d]
13:
Phase 3: Score Calibration (Elo Reranking)
14: Derive pairwise comparisons from batches to compute global Elo ratings
15: EloRatings +- SiibbaateScorss(Raaaoorrss B)
16: Phase 4: Aggregation and Finalization
17: Sfinal +
18: for cach d do
19:
Compule average row score K appearonces Praw + Mean (awSScree(d])
20:
21: > Retrieve globolly calibrated rank
22:
Telo + EloRaaiings[]]
23:
Sfimat [d] +- (Jramm; Felo)
24:
return Syinal
34

---

Published in Transactions on Machine Learning Research (03/2026)
F Downstream Experiments
F.1 Training Configuration and Hyperparameters
We first conducted hyperparameter sweep to identify the optimal LoRA rank and batch size for each dataset. Checkpoint selection was based on the validation accuracy of the final answer, where we chose the checkpoint corresponding to the peak of a 3-point moving average of scores over the final evaluation steps. This process yielded a fixed set of hyperparameters for each dataset. However, since the optimal learning rate proved sensitive to the training data size, we retained a sweep over two values (0.005, 0.01) for our final experiments. These final runs were performed with 10 different seeds for each configuration. For each data size, we then selected the runs from the best-performing learning rate average for that specific system version.
We use the Adafactor optimizer for training Shazeer & Stern (2018). To ensure stability, apply global gradient clipping with a maximum norm of 1.0. The learning rate follows a linear warm-up, followed by cosine decay schedule. The number of warm-up up steps is calculated dynamically based on the total training steps (still bounded between a minimum 500 and a maximum of 5,000 steps)
Table 6: Final hyperparamcters used for each dataset. The final learning rate was chosen from the specified set based on validation performance for each training data size.
Dataset
Learning Rates
LoRA Rank
Batch Size
CTI RCM
{0.005, 0.01}
8
16
CTI MCQ
{0.005, 0.01]
8
16
GSM8K
{0.005, 0.01}
16
16
LEXam
{0.005, 0.01}
16
16
MMLU
{0.005, 0.01)
16
16
F.2 Additional Evaluation Results
In Figure 11, we show the results two additional subsets of Global MMLU (Physics, Math, and Computer Science). The trend shown in Section 4.3 for the Nepali subset continues to be the case here, where the full Simula system shows better accuracy growth trend with larger datasets.
Global MMLU (P,M.CS)- English
Global MMLU (P,M.CS) - Korean
Gemini Flash 2.5: 96% Gemma 4B IT: 64%
Gemini Flash 2.5: 94% Gemma 4B IT: 55%
Accuracy (LORA Genuna-3 3-4--1 PT)
System Version
0.52
Baseline
0.56
Local
Global
0.54
0.48
Local - Global
0.52
Local - Global Critiqu
0.46
0.5
0.44
0.48
0.42
0.46
4k
16k
64k
128k
256k
512k
4k
16k
64k
128k
256k
512k
Data Size
Data Size
Downstream Performance Global MMLU English and Korean. We report mean accuracy with 95% CI for these additional datasets. We continue to see that the full Simula system version with the critic, provides the best performance compared to the other versions.
Figure 11:
35

---

Published in Transactions on Machine Learning Research (03/2026)
F.3 Choice of Optimizer
As detailed in the Section P.1, we use Adafacior as our optimizer for all downstream line-tuning experiments. A reasonable concern would be if the choice of optimizer could influence the reported results in Section 4.3. To address these wc conducted a targeted ablation and offer additional thcoretical justification below.
We replicated the rownstream experiment on the CTT MCQ datasel, scaling the data size from 4k 84k using the Muon optimizer (Jordan et al., 2024). We opted for the adaptive Muon optimizer because of its structural distinctness from Adafactor (i.e., momentuum-orthooonaall vs. factored second-moment). As shown in Figure 12, observe that the relative performance trends hold: The Full Simula system consistently outperforms the Baseline and different. variants across data sizes. These results suggest that the benefits provided by our reasoning-driven approach are robust to the choice of optimizer.
MCQ
Gemini Flash 2.5: 71% Gemma 4B IT: 54%
Accuracy (L.oRA Gemma-3-1B-1
0.63
System Version
Baseline
0.62
Local
0.61
Global
0.6
Local Global
0.59
Local Global Critiqu
0.58
0.57
4k
16k
Figure 12: Downstream Performance using Muon Optimizer. We report mean accuracy with 95% CI for the CTI MCQ dataset for student models fine-tuned using the Muon optimizer. We continue that the full Simula system version with the critic, provides the best performance compared to the other versions.
Informed Hyperparameter Selection. The reported results Section 4.3 using Adafactor are not based on default settings, but the outcome of rigorous hyperparameter sweeps over data sizes and configurations (see Appendix F.1). As noted in (Schmidt et al., 2021), while optimizer performance can vary greatly across tasks, the authors show that evaluating multiple optimizers with default parameters works approximately well as tuning the hyperparameters of a single, fixed optimizer.
measure the relative performance differences of the different system components. We can therefore distinguish between the exponent of the scaling law (rate at which error decreases with data) and the intercept (training efficiency). The former is shown to be strongly correlated with data composition (Sorscher et al., 2022; Chen et al., 2025), while the latter is influenced by optimization and architecture (Schmidt et al., 2021; Everett et al., 2024). Since our claims focus on optimal scaling trajectories (exponent) of Simula data, the choice of optimizer (primarily intercept) should not invalidate the relative advantages observed.
Scaling Laws for Relative Trends. The goal of our ablations
36

---

Published in Transactions on Machine Learning Research (03/2026)
F.4 Choice of the Teacher Model
Throughout this work we used Gemini 2.5 Flash as the generator model. To demonstrate that Simula is model-agnostic framework we conducted a targeted ablation using the open-source Qwen3-Nezt-800 Instruct model (Yang et al., 2025). We replicated the downstream experiment on the CTI MCQ datasct, scaling the generated data size from 4k to 84k, while keeping the student model and optimization parameters unchanged.
As shown in Figure 13 below, performance trends remain consistent with the results reported in Section 4.3 using the Gemini model: student models fine-tuned data generated by the Full Simula system consistently outperform the Baseline and partial ablations across data scales. This provides additional evidence that the observed improvements are due to our reasoning-driven data generation approach, rather than model-specific factors.
CTI MCQ
@wen3-Next-80B Instruct: 72% Gemma 4B IT: 54%
Accuracy (LaRA + Cemma-3-4B-ID)
0.63
System Version
Baseline
0.62
Local
0.61
Global
0.6
Local Global
Local Global Critiqu
0.58
0.57
4k
16k
64k
84k
Figure 13: Downstream Performance using Qwnn3-Next-80B Instruct. We report mean accuracy with 95% CI for the CTI MCQ dataset for data generated using the Qwen-3 80B instruction-tumed model. We contimue to see that the full Simula system version with the critic, provides the best performance compared to the other versions.
37


Reviews:
[{"review_id": "FMprdhDp5M", "reviewer": "Reviewer_j5DE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors lay out the main challenges of large-scale synthetic data generation, and then propose their framework Simula for addressing them. \n\nThe approach works as follows: a multi-modal model (M3) derives \"taxonomies\", which in my limited experience in the field believe to be semantic hierarchies of labels describing a particular qualitative factor of dataset variation. Once the taxonomies are derived, the framework can sample from them intelligently, such as by using a planning algorithm, thereby varying the different factors driving data generation in a controllable way (as opposed to uncontrolled generation which may have biases or uneven coverage of certain concepts). The data generation itself is augmented by a critic model to improve the output quality.\n\nThe authors benchmark their approach in various language-modelling settings, and show that each of their proposed changes (taxonomy sampling methods, use of critic, etc.) has a real benefit, as their experiments are thorough and include comparisons to a simple baseline.\n\nMy main concern with the work is that the concept of a \"taxonomy\", which is crucial to understanding the working mechanisms of Simula, is not clearly explained. In Section 2.1, the article offers descriptive examples of what \"factors\" are (ie. “cat type”), but does not go into details about how these factors are broken down into taxonomy objects. It may be useful to also offer an example of what the taxonomies related to these factors are, as only the appendix contains examples. Likewise, Figure 1 is hard to interpret without a-priori knowledge of what it is trying to describe and could benefit from annotations of \"clusters\" and visual explanation of how breaking into strict taxonomies creates the grid-like structure that covers the whole space more evenly ((b) and (c) are lacking the data points from (a) so it is hard to intuit what happens to them). More broadly, the paper does not define the term \"taxonomy\" prior to its use. It is my understanding that a taxonomy is a tree of concrete class labels/descriptions related to the corresponding factor.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, the results are backed by significant number of benchmarks which are accurately represented by the paper's main claims. There is a strong focus on ablation study/comparison to the baseline so it is clear that the proposed method causes the claimed improvements in performance.", "audience_yn": "Yes", "audience_explanation": "Yes, synthetic data generation is a useful area in machine learning, and as the authors point out, highly relevant to modalities other than just language as well.", "requested_changes": "More clarity on the definition of \"taxonomy\" I believe is necessary for my recommendation. It is a key component of the Simula framework so readers, especially those with little background in the area, can understand the main problems it tries to solve. Currently, the term \"taxonomy\" is used before it is introduced/defined to the reader.", "broader_impact_concerns": "The ability to more strongly control data generation has obvious broader impact concerns. For example, it may facilitate the generation of nefarious content, or reinforce potential biases of the agent (human or M3) steering the generation process. It would be worth drawing attention to these concerns in a broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xfIU342CiG", "reviewer": "Reviewer_tfQe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Simula, a framework for generating synthetic data that prioritizes explainability, control, and scalability over traditional stochastic or seed-based methods. The core methodology involves a multi-stage pipeline: (1) generating taxonomies to map the conceptual space of a domain, which ensures global diversity, (2) using agentic workflows to generate meta-prompts and complexify them, and (3) employing a dual-critic system to filter for quality and correctness.\n\nThe authors evaluate this framework intrinsically (via embedding diversity, taxonomic coverage, and a novel calibrated complexity scoring via Elo) and extrinsically (via downstream fine-tuning performance). Experiments are conducted across five datasets ranging from popular benchmarks (GSM8k, MMLU) to niche domains (Cyber Threat Intelligence, Legal exams).\n\nStrength: \n- The shift from random sampling to a taxonomy-driven approach addresses the common issue of mode collapse in synthetic data generation effectively.\n- The discovery that increased data complexity does not always yield better downstream results (specifically when the teacher-student gap is small or the teacher is weak, as seen in LEXam) is valuable.\n- The proposal of Taxonomic Coverage and Calibrated Complexity Elo offers concrete alternatives to vague embedding-similarity metrics.\n\n\n\nWeaknesses:\n\n- The experiments rely exclusively on Gemini 2.5 Flash for the teacher model and Gemma 3 4B for the student model. \n- The primary baseline is a simplified version of their own pipeline (direct sampling). A comparison against established synthetic data methods is missing.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors provide a rigorous experimental setup to support their claims. The ablation studies in Figure 6 show that combining Local + Global + Critique consistently outperforms the Baseline and partial implementations across data scales.", "audience_yn": "Yes", "audience_explanation": "The work focuses on synthetic data generation, which is a core interest of the TMLR audience.", "requested_changes": "- Generalization Across Model Families: The paper relies entirely on Gemini/Gemma. To demonstrate that Simula is a generalizable framework and not specific to Gemini, it is important to present an ablation study using a different model family (e.g., Llama-3) for the generator/teacher role. \n\n- Comparison to Baselines: The current baseline is essentially a stripped-down version of Simula. Please include the comparison against standard synthetic data generation methods.", "broader_impact_concerns": "I have no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CGPgtZqwKi", "reviewer": "Reviewer_T615", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Simula, a reasoning-first, seedless, and agentic framework for generating high-quality synthetic datasets at scale. The authors argue that existing synthetic data approaches struggle with three major issues: (1) unclear definitions of “good” data (quality, diversity, complexity), (2) lack of transparent and controllable generation mechanisms, and (3) limited or misleading evaluation practices relying heavily on noisy or incomplete benchmarks. Simula is designed to address these gaps by building each synthetic example through explicit reasoning and structured conceptual decomposition.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper presents a well-designed and thorough investigation into reasoning-driven synthetic data generation. The proposed Simula framework is technically novel in its focus on transparency, controllability, and seedless generation, offering a clear mechanism design that is rarely discussed in synthetic-data research. The paper introduces several strong methodological ideas, including taxonomy-based coverage, meta-prompt construction, and a double-critic refinement process, which together form a coherent and interpretable pipeline. The experimental section is extensive and carefully executed: the authors evaluate on both niche and widely used benchmarks, conduct detailed ablations of each system component, study scaling behavior, and propose new intrinsic metrics such as taxonomy coverage and calibrated complexity scoring. These analyses provide valuable insights into how diversity, complexity, and correctness influence downstream model performance.", "audience_yn": "Yes", "audience_explanation": "At least a portion of TMLR’s audience would be interested in this paper’s findings. The work addresses an increasingly important problem in modern ML: how to generate high-quality synthetic data in a controllable and scalable way. This topic is relevant to researchers working on data-centric model training, evaluation methodology, LLM alignment, and multimodal system development. The paper’s insights on diversity, complexity, and mechanism design also speak to broader issues in dataset construction and benchmarking, which are central concerns for TMLR readers. Even if the framework is specialized, the empirical findings and methodological contributions would likely appeal to those studying data quality, model scaling, and synthetic data pipelines.", "requested_changes": "I would like to see more discussion or evidence regarding whether the reported downstream trends depend on the choice of optimizer. The current results are presented using Adafactor, but generalization behavior can vary meaningfully across optimizers, especially in low-data or synthetic-data regimes. Prior work has shown that optimizers such as AdamW, Muon, and other adaptive variants can yield different stability and generalization characteristics even when trained on identical data. It is therefore unclear whether the scaling patterns and conclusions drawn in this paper would remain consistent under different optimization settings. I encourage the authors to either provide additional experiments with alternative optimizers or include a more explicit justification for why the observed effects should be robust to optimizer choice.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)