DESIGNER: Design-Logic-Guided Multidisciplinary Data Synthesis for LLM Reasoning
Authors: Weize Liu
Yongchi Zhao
Yijia Luo
Mingyu Xu
Jiaheng Liu
Yanan Li
Xiguo Hu
ZhiqiBai
Yuchi Xu
Wenbo Su
Bo Zheng
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Data Synthesis
Synthetic Data
Reasoning
Post-Training
Supervised Fine-Tuning
Primary area: datasets and benchmarks

Abstract: Large language models (LLMs) perform strongly on many language tasks but still struggle with complex multi-step reasoning across disciplines. Existing reasoning datasets often lack disciplinary breadth, reasoning depth, and diversity, as well as guiding principles for question synthesis. We propose DESIGNER: a DESIGN-logic-guidEd Reasoning data synthesis pipeline that leverages naturally available, extensive raw documents to generate multidisciplinary questions. The central insight is the notion of Design Logic, a form of reusable meta-knowledge that encapsulates the structured process human experts use to transform knowledge into complex exam questions, enabling LLMs to generate new questions with the same complex reasoning patterns from entirely different source texts with explicit control over difficulty, diversity, and question types. We use LLMs to reverse-engineer and abstract over 120,000 Design Logics from existing questions across various disciplines. By designing a two-stage retrieve-and-generate mechanism to match these Design Logics with raw corpus, we synthesized two large-scale reasoning datasets that span 75 disciplines: DLR-Book (3.04 million questions from the book corpus) and DLR-Web (1.66 million questions from the web corpus). Data analysis indicates that the questions synthesized by our method exhibit greater difficulty and diversity compared to those in the baseline datasets. Supervised fine-tuning (SFT) on Qwen3 and Llama3 with our data substantially improves multidisciplinary reasoning and outperforms baseline datasets. Notably, by applying SFT on the base versions of these models using only our data, we even surpass their official final models that have undergone the full post-training.

Published as a conference paper at ICLR 2026
DESIGNER: DESIGN-LOGIC-GUIDED MULTIDISCI- PLINARY DATA SYNTHESIS FOR LLM REASONING
Weize Liu Yongchi Zhao!,::) Yijia Luo', ,inggy Xu', Jiaheng Liu
Yanan Li,, Xiguo Hu', Zhiqi Bai', Yuchi Xu', Wenbo Su', Bo Zheng'
Alibaba Group, *Najjing University
weizeliul1 1iiuu1111@gmaaa 1. com, zhaoyongchi. www.:ongghh..c@@maiiii... com, liujiaheng@nju.ed
ABSTRACT
Large language models (LLMs) perform strongly on many language tasks but still struggle with complex multi-step reasoning across disciplines. Existing reasoning datasets often lack disciplinary breadth, reasoning depth, and diversity, as well as guiding principles for question synthesis. We propose DESIGNER: DESIGN- logic-guidEd Reasoning data synthesis pipeline that leverages naturally available, extensive raw documents to generate multidisciplinary questions. The central insight is the notion of Design Logic, a form of reusable meta-knowledge that encapsulates the structured process human experts use transform knowledge into complex exam questions, enabling LLMs to generate new questions with the same complex reasoning patterns from entirely different source texts with explicit control over difficulty, diversity, and question types. We use LLMs to reverse- engineer and abstract over 120,000 Design Logics from existing questions across various disciplines. By designing two-stage retrieve-and-generaa mechanism to match these Design Logics with raw corpus, we synthesized two large-scale reasoning datasets that span 75 disciplines: DLR-Book (3.04 million questions from the book corpus) and DLR- Web (1.66 million questions from the web cor- pus). Data analysis indicates that the questions synthesized by method hibit greater difficulty and diversity compared to those in the baseline datasets. Supervised fine-tuning (SFT) on Qwen3 and Llama3 with our data substantially improves multidisciplinary reasoning and outperforms baseline datasets. Notably, by applying SFT on the base versions of these models using only our data, we even surpass their official final models that have undergone the full post-training
INTRODUCTION
Large language models (LLMs) have demonstrated exceptional capabilities in various natural rea- soning tasks (Chowdhery et al. 2023 OpenAT (023), such as mathematics and coding (Trinh al.) 2024 Zhu et al. (024)) especially when uti utilizing long chain-of-thought (CoT) techniques (Wei ctt al. 2022 Jaech et al. 2024 Guo et al. 2025). However, they still lag behind human experts university-level. discipline-speciie
reasoning (Phan et al., 2025). largely due to the scarcity of large-scale, high-quality, and diverse training data. Existing datasets focus mainly on math and pro- gramming, drawing competition platforms rich in open-ended questions (Moshkov et al., 2025 Cai et al. 2025). while most other disciplines lack comparable resources. This scareity limits the development of LLMs' multidisciplinary reasoning capabilities.
Data synthesis with LLMs is an effective solution to data scarcity (Wang et al. 2024a). Ex- isting question synthesis methods fall into two categories: query-centric and document-centric. Query-centric approaches expand seed questions through rewriting, added constraints (c.g., Evol- Instruct (Xu et al. 2023)), ) or incorporating chain-of-thoughh reasoning (Wang al. 2023 Yu al. 2025). but they are limited by the coverage of the seed pool and inherent model biases. Document- centric methods instead generate questions from unstructured (e.g., web, books) or structured (e.g.,
First two authors contributed cqually. t "oorresponding Authors: Yongchi Zhao, Jiaheng Liu. Projecct page: https://attentiones is-allli -eeldd www.dd.ggttuuu..u...o httpaaiiooossiggggiieeeeeeeeeeee

---

Published as a conference paper at ICLR 2026
Design Logic
DESIGNER
AF
Identify Fy
Construct scenamics
Design
Preset
Generate
Raw
Desiin
Diverse
Knowledge Points
Reasoning
Distractors
Exam
Corpus
Logic
Questions
Paths
Quastions
Library
Figure 1: Left: The procedure by which human experts construct questions reflects a "Design Logic": systematic sequence of deliberate steps that transforms fundamental knowledge points into complex, context-rich questions requiring multi-stage reasoning. Right: DESIGNER emulates this process by matching logies to raw corpus and synthesizing diverse, multidisciplinary questions.
knowledge graphs) source documents Yue et al. 2024 Yuan et al. 2025 Huang et al. 2025 ensur- ing broad disciplinary coverage grounded authentic knowledge. However, they struggle to control difficulty and diversity, often degenerating into factual recall. Meanwhile, post-training (e.g.. SFT) and even mid-training all rely heavily on exam-style data. This raises a key challenge: how can rapidly synthesize large volumes of high-quality, human-like multidisciplinary exam questions while controlling their difficulty, diversity, and question types?
To address these issues, propose DESIGNER: DESIGN- -ogic-gudEdd Reasoning data synthe- sis pipeline that leverages large-scale, multidisciplinary raw documents (e.g.. book corpus and web corpus) synthesize challenging questions across diverse disciplines. The central insight of approach is the notion of Design Logic, which encapsulates how human experts transform knowl- edge into complex, reasoning-intensive exam questions. We observed that when human education experts design challenging and insightful questions, they do not merely state facts. Instead, they follow a structured design process, as illustrated in Figure The Design Logic is a form of reusable meta-knowledee that abstracts the underlying reasoning structure, enabling LLMs to generate new questions with the same complex reasoning patterns from entirely different source texts.
Specifically, our pipeline is illustrated in Figure First, we process large-scale book and web corpora with multi-dimensional labeling and filtering (discipline, readability, educational value, rea- soning depth) to construct high-quality source material library. From question bank of hundreds of millions, we cluster and sample diverse set of difficult questions, from which an LLM reverse- engineers and abstracts over 120K structured Design Logics construct reusable Design Logic library. In question synthesis, we adopt a two-stage similarity retrieves coarse candidate logics for each source document, and (2) an LLM performs a fine-grained evaluation to select the optimal logic and generates a reasoning question from the source document by strictly following its steps. This approach addresses the absence of guiding principles in prior data synthesis methods, enabling the automated generation of a large number of diverse and high-difficulty exam questions while reducing reliance expensive manual creation.
seweovv-and-geeneee mechanism: (1) vector
The main contributions of this paper can be summarized as follows:
we propose DESIGNER: a DESIGN BooGGN--ooi-guideBB Reasoning data synthesis pipeline that lever- ages corpora synthesize multidisciplinary reasoning questions. Using this pipeline, constructed two large-scale reasoning datasets: DLR-Book (3.04 million questions from the book corpus) and DLR-Web (1.66 million questions from the web corpus). These datasets span 75 disciplines, including STEM, humanities, social sciences, applied and professional fields, and the arts, extending beyond common disciplines.
By reverse-engineering the meta keetl-nnowlede of human educators, propose fundamentally new question synthesis method guided by "Design Logic". This approach enables the generation of truly complex, multi-step reasoning questions from raw text by providing the structured, reusable, and abstract control over difficulty and diversity that prior document-centrie methods lacked. Our data analysis indicates that the questions synthesized by our method exhibit greater difficulty and diversity compared to those in the baseline datasets.
We validate the effectiveness of our synthesized data through comprehensive comparative and ablation experiments the Qwen3 (Yang et al.] (225a)) and Llama3 (Dubey et al.) (024) model families. The results demonstrate that the data synthesized by our method significantly enhance the multidisciplinary reasoning capabilities of LLMSs.
2

---

Published as a conference paper at ICLR 2026
Phase 1
Phase 2
Phase 3
Data Extraction and Preprocessing
Core Synthesis
Filtering and Output
set of data points, can you use leost souares estinate eaplain thi methed best linear urbiosed estimate (SLUE)
Design Logic Extraction
Question Deduplication
Question Bank
Toun
Multi dimensional Labeling
Question Decontamination
tecrpindgge
Clustering- Question Selection
Corkut
Quality Prioritized Sampling
Lungaaee
Design Logic
Deduplication
Response Synthesis
Discipline
Discipline
Labeling
Labeling
-ttag Design Logic Retrieval
rthinke
Wi going implement Focal Lows i GNN for on inbolanced cotoset.
Quality Assessment
Reasoning- Oriented Filtering
Question Synthesis
</thinko
implementig
Text Preprocessing
Given implement Focal Loss CNNF Explain Focol Loss fester than Weighted Cress Entropy when the class boundaries one fuzzy (there are many hard examples)
you
*Nuer Casact Ideal for objuct detection. medical incging, any task with class inbaalncee ambigucus somples.
Web Corpus
Book Corpus
Figure 2: The Logig Guided Multidisciplinary Data Synthesis Pipeline.
Lugic-Guieed
DATA CURATION
We curate three data sources for question synthesis: proprictary question bank, a book corpus, and a web corpus, all aligned to unified 75-discipline taxonomy (see Appendix Figure (Phase 1) illustrates the overall data processing pipeline.
2.1 QUESTION BANK PROCESSING
We annotate over 150 million questions from the proprietary question bank with discipline, diffi- culty, and question types using Qwen3-30B-AA3 (non-thinking mode), following the prompts in Figure 7 Figure 8 and Figure 9 respectively. To obtain a high-quality and diverse subset, we compute embeddings with Owen3-Emmeddinng-B7 and apply K-means clustering within each dis- cipline (Ahmed et al.) 2020). with cluster numbers determined by silhouette search. From each cluster, we draw an equal number of questions following fixed difficulty ratio of 3:2:1 (Very Hard:Haar:MMdiimm and align per-discipline sizes with the overall bank distribution. If higher- difficulty questions are insufficient, they are backfilled with lower-difficulty per-discipline totals. This process resulted in a curated set of 132,409 questions for design-logic extraction. Any question bank, regardless of its initial quality, can be processed using this filtering pipeline to obtain high-quality and diverse subset for extracting Design Logics. Our method is therefore applicable to any question banks and can directly applied to publicly available datasets.
to ensure the
2.2 BoOk CORPUS PROCESSING
This corpus is processed at the chapter level, with chapters over 5,000 words split into smaller blocks and deduplicated via MinHash. Discipline labels assigned by ModemBBERT-lare classifier fine-tuned for disciplinary classification |Wanner et al. 2024). Readability is predicted with a BERT- based model (Ture et al. 2019) filter incoherent disorganized text, while helpfulness (0-5) is
3

---

Published as a conference paper at ICLR 2026
scored by the fneweb-edu-classife (Lozhkov et al. 2024) quantify educational value. After removing segments with negative readability, remaining candidates are ranked by helpfulness and selected to meet discipline quotas proportional frequencies in the book corpus and the question bank. This procedure yields three million high-quality segments, most with helpfulness 2 2.
2.3
WEB CORPUS PROCESSING
We apply reasoning-oriented filtering and discipline relabeling the FineFineWeb corpus, scoring 6.5 billion texts with Qwen3-30B-AN (non-thinking mode) using five-level rubric (prompt in Figure I0) and retaining those with scores 2 The selected texts are then relabeled with the same model (prompt in Figure to align with our 75-discipline taxonomy.
3 DESIGN-LOGIC-GUIDI DATA SYNTHESIS
Figure (Phase and Phase 3) illustrates the overall data synthesis pipeline.
3.1 DESIGN LOGIC EXTRACTION
Human educators design exam questions through structured steps that transform knowledge points into complex challenges, rather than simple fact recall. A typical process involves identifying ob- jectives, constructing contexts, designing reasoning paths, formulating answers, adding distractors, and validating the questions. Solvers must engage in multi-step reasoning beyond memorization.
Inspired by this, propose a design-logic-based question synthesis method. Using the prompt in Figure [1] we instruct an LLM (DeepSeek-R1- R1-0288) to analyze authentic questions and (i) infer the designer's thought process, (ii) trace construction from knowledge points, and (iii) abstract underly- ing design principles, expressed in Mermaid format. This produces reusable pool of Design Logics that guides new question generation from source materials.
3.2 DESIGN LOGIC DEDUPLICATION
To enhance the diversity of design principles, we deduplicate extracted logics using semantic simi- larity. Each logic is embedded with Swenn--Embedding-ae and pairwise similarities yield matrix S E RR Within each discipline, we construct a graph G (V, F)) where nodes represent logics and edges connect pairs with Süj 2 т. Connected components in the graph correspond to redun- dant Design Logic groups From each group, retain the item with the highest similarity sum. With r = 0.85, this graph-based deduplication procedure (Algorithm M yields 125,328 unique De- sign Logies, with per-discipline counts in Table 6 We provide several examples of our synthesized Design Logic in Appendix
3.3 QUESTION SYNTHESIS
To avoid the combinatorial explosion resulting from exhaustively matching Design Logics with text segments, we adopt a retrieval-augmenter approach. For each discipline-specific corpus, we compute the cosine similarity between embeddings of a text segment and a Design Logic d using with task-specific instructions (Figure 12): s(t, d) = cos e(t), e(d)). The top-5 logics with the highest similarity are retained as candidates.
We then prompt PrrpSeek-R1-0555 (Figure 13) to: (i) select the most suitable logic from the top-5 candidates, and (ii) synthesize a graduate- exam question strictly following its steps. This two- stage process forms coarse-to-fine ranking: similarity-base the LLM refines the match to ensure precise alignment between text and logic, thereby improving question quality. For each question, the LLM also generates concise reference answer.
retrieval provides coarse recall, while
Question Deduplication and Decontamination. We employ a two-stage filtering pipeline: (i) MinHash-based deduplication to remove near-duplicates, and (ii) 13-gram decontamination against all evaluation benchmarks to prevent leakage. Using the curated book and web corpora, DeepSeek- R1-0523 generates reasoning question per text segment. After filtering, the final dataset Design- Reaioo-Resnning--Be Book -Book) comprises 3, 3,040,620 questions from the book corpus, and Design-Logic- Reaiiie--eannnn--WWa Web (DIRRWebb) comprises 1,658,541 questions from the web corpus.
4

---

Published as a conference paper at ICLR 2026
3.4 RESPONSE SYNTHESIS
To demonstrate that our synthesized questions can effectively elicit and transfer the long CoT capa- bilities of a reasoning model and improve the performance of models trained on this data, we employ -aw.-----------iiiiiiig-------
generate corresponding long CoT response for each synthesized question. These question-response pairs are then used for supervised fine-tuning (SFT). However, response generation is not the primary focus of this work. Our objective is to synthesize high-quality multidisciplinary questions. As Albert Einstein noted, "The formulation of a problem often more essential than its solution." Given high-quality questions, responses be generated by any model, including more powerful future models capable of achieving higher accuracy.
4 DATA ANALYSIS
To assess the quality of our synthesized datasets (DLR-Book and DLR- Web), DLR-Web), we conduct quanti- tative analysis comparing their difficulty, diversity, and disciplinary distribution against the baseline datasets. The baseline datasets and benchmarks we used are detailed Table 7nd Table 13 respec- tively. In all tables, we highlight the best value in boldface and the second-best with an underline. The distribution of question types for our synthesized datasets is detailed in Appendix
4.1 DIFFICULTY ANALYSIS
We utilized the Dowen3-30--Al A3B-IIstruct-255555 model with the prompt shown in Figure 8 to assign difficulty labels both our datasets and the baseline datasets. To facilitate an intuitive comparison of difficulty levels, we further applied the same labeling procedure to several commonly used bench- marks. As shown Figure 3 our datasets are significantly more difficult. Notably, the proportion of "Very Hard" questions in our datasets is substantially higher than that in all baseline datasets and benchmarks. In contrast, the proportion of "Easy" questions is negligible (0.72% in DLR- Web and 0.27% in DLR-Book). The detailed distribution of difficulty levels is reported in Table I0
100
80
Percenlage (%)
20
OSAIKK
MAKU
GPGA GPGA-Daanot Dianon
SiperOPA GPOA-MAR
Opanthaughta
DuR-Neb
DUR-Bunk
Easy
Medium
Hard
Very Hard
Figure Difficulty distributions of questions across different datasets and benchmarks.
4.2 DIVERSITY ANALYSIS
To quantify dataset diversity, we generate high-dimensional vector representations for 300,000 uni- formly sampled questions from each dataset using the
model. We then com-
pute the following five distance-based diversity metrics in the embedding space: Mean Cosine Dis-
tance (Yang al. 2025b). Mean L2 Distance (Yang et al. 2025b). 1-Nearest Neighbor (1-NN)
Distance (Stasaski & Hearst 2022). Cluster Inertia (Du & Black 2019), and Radius (Lai et al. 2020 Detailed definitions and formulas for these metrics are provided in Appendix
As detailed in Table our datasets, DLR-Bood and DLR-Web, consistently demonstrate a greater diversity of questions compared the baseline datasets across all five semantic diversity metrics. The higher Mean Cosine Distance and Mean L2 Distance values confirm that our synthesized ques- tions are, on average, more semantically distinct than those in the baseline datasets, indicating a broader conceptual scope. The most notable difference is observed in the I-NN Distance, where our datasets score approximately twice as high as the basclines. This suggests our method generates far fewer semantically redundant questions. Furthermore, the Cluster Inertia and Radius scores for
5

---

Published as a conference paper at ICLR 2026
Table 1: Results of semantic diversity metrics for our synthesized datasets and the bascline datasets.
Higher values indicate better performance across all metrics.
Dataset
Mean Cosine
Mean L2 Distance
I-NN Distance
Cluster Inertia
Radios
Distance
Open Thoughts3
0.8037
1.2656
0.0051
206,369,22
0.0172
Nemotron- Paaee-Poottttiiieeieeeee Taas-Trainn
0.8243
1.2827
0.1290
226,21 1.0221100
0.0174
WebInstruct (Full)
0.7762
1.2436
0,1830
205,590.02
0.0169
NaturalReasoning DLR-Web
0.8233
1.2818
0.1915
226,288.91
0.0173
0.8494
1.3026
0.3897
238,039.50
0.0177
DLR-Book
0.8471
1.3008
0.3726
238,100.26
0.0176
our datasets indicate that the generated questions occupy a larger and more varied volume within the semantic embedding space. This quantitative evidence confirms that our synthesis pipeline produces not only more complex and difficult questions but also a significantly more diverse set of questions. Additionally, we observe that questions synthesized from the web corpus (DLR- Web) exhibit greater diversity on most metrics than those synthesized from the book corpus
4.3 DISCIPLINARY DISTRIBUTION
We also used Qwen3-30B-AL and the prompt in Figure to assign disciplinary labels to the base- line datasets. We present comparison of discipline distributions between our dataset and baseline datasets in Figure 4 For visualization, we highlight the ten most dominant disciplines within each dataset and those most representative of broader academic categories, while aggregating the remain- ing disciplines into the gray category "Other." It is evident that many existing multidisciplinary datasets are heavily skewed toward few disciplines, such as mathematics, leading to a highly im- balanced distribution across disciplines. Among them, only the Demotron-Post- Post-Trainng--- dataset exhibits a distribution comparable to ours, but its question difficulty and reasoning depth are substan- tially lower. The detailed number of questions per discipline in dataset is provided in Table 6
OpenThoughts3
Webinstruct (Full)
DLR-Web
Nemotron-Post Vemotron-PostTTaainnngg)
NaturalReasoning
DLR-Book
Biology
Computer Technology
Psychology Philosaphy
Physics
Chemistry
Economics
Law
Figure 4: Discipline distribution across different full datasets.
4.4
DATA QUALITY AND LABEL ACCURACY VALIDATION
To assess the reliability of our method, we employed GPT-5.1 to conduct a muulti-dimensionll eval- uation on 10,000 randomly sampled instances. We utilized the prompts detailed Figure 14 Fig-
ure 15 and Figure 16 to assess content quality, and further employed the prompts in Figure [7
Figure g and Figure 19 to validate the appropriateness the assigned discipline, difficulty, and question type labels, respectively. Our analysis demonstrates high data integrity: 96.70% of the synthesized questions were verified as complete and answerable, providing sufficient conditions to derive solutions. Furthermore, 84.69% of the questions exhibited strict alignment with their corre- sponding Design Logics. The evaluation of labels also showed high accuracy, with 90.14% 96.53% and 91. 222 of the instances confirmed to have appropriate discipline, difficulty, and question type

---

Published as a conference paper at ICLR 2026
labels, respectively. Finally, evaluated the consistency between the reference answers generated by DeeepSeek-R--ee -1-ock2-----228 and the long Chain-of-Thooe (CoT) responses generated by Qwen3-235B- achieving 7 71.48% agreement rate. A significant portion of our synthe- sized data comprises complex, open-ended reasoning questions. In disciplines such as Humanities, Social Sciences, Arts, and Applied Professional Fields, questions inherently lack single fixed an- swer. Consequently, this result is reasonable given the inherent diversity of valid reasoning paths in open-ended multidisciplinary questions.
(hhnkiing2257---
5
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
SFT Settings. For all SFT experiments, we adhere to the hyperparameters detailed Table 11
Evaluation Settings. To ensure a fair comparison, we employ a zero-shot evaluation setting for all trained models, using the consistent generation configuration specified Table [2] Depending on the characteristics of each benchmark, test instance. For 1, we report accuracy. For N > 1, we report mean accuracy (%) together with the standard deviation, and also report the accuracy (%) under the Self-Consistency with Chain-of- Thought (CoT-SC) method, defined as the majority vote over the generated samples.
perform N independent sampling rollouts for each
Benchmarks. We evaluate the model's multidisciplinary reasoning capability using the most widely adopted benchmarks. The benchmarks, along with their respective disciplines and the num- ber of rollouts (N), are detailed in Table 13
5.2 SUPERVISED
(SFT) EXPERIMENTS
We performed SFT on base models from the Qwen3 and Llama3 series using our synthetic datasets, DLR-Book and DLR-Web. The resulting models were subsequently compared against their official final models that have undergone the full post-training process under the same evaluation framework.
Table 3: SFT experiment results. The models are trained DLR-Web, DLR-Book, or the combined
DLR-War Web DLR-Book dataset.
MMLU
MMLU-Pro
GPQA-Diamund
GPQA-Main
SuperGPQA
Model
Accuracy
Accuracy
Accuracy
CoT-SC
Accuracy
CoT-SC
Accuracy
1amma-3.-33B-Instmae
57.71
31.18
21.97+-2.54
20.20
25.07++.664
24.78
16.
Llama 3.2. 3B SFT (DLR Web)
66.74
49.81
22.2212.06
20.71
25,54 1.39
26.12
20.31
Llama-3.2-3B. 3.2-3B-FFT (DLR-Book)
70.61
$7.09
38.33112.63
44.44
33.82:1.17
35.27
26.39
LJama SFT (DLR. Web+Book)
73.53
61.36
42.27:.1.72
43.94
38.91 1.78
43.97
29.64
Llama-3,1-9B-Instt
70.86
47.38
23.18:+1.78
24.75
27.99:11.40
28.57
20.08
Llama-3.1-9B-ST 1-8T--FFTTTT (DLRR-We)
81.75
72.64
57.734-2.16 16
63.64
55.45-+2.07
58.71
39.
Llama-3,19BB-F7 (DLR-Book)
83,33
74.94
63.23-+1.37
66.67
62.25=1131
43.48
Llama-311-8B-SF
84
76.04
65.45++1.47
70.71
63.62++1.2 1.23
67.86
45.06
Qwen3-4B (Thinking
69.34
$4.7012.42
49.51=11.40
43.30
Qwen3-4B SFT (DLR-Web)
83.55
71.24
$3.74
60.61
$1.27.11.57
$5.36
42.73
Qwen2-4B-Base-S1 (DLR-Book)
56.85-0.91
Qwen3-4B-Baso-SFT (DLR-Web+Book)
85.00
73.06
63.69+22.15
70.20
58.73=11.36
63.62
45.15
Qwen3-8B (Thinking Mode)
73.62
59-444-2553
57,95+
47.55 07.52
Qwen1-SB. SB. Base (DI R-Web)
86.82
75.62
63.28--2.43 2.43
66.67
61.43-10.98
66.07
48.66
Qwen3-Sr SB (DLR-Book)
76.69
69.3911887
73.74
98
50.57
Qwen1-8B. BFT (DI (DL.R. BeWokBoook
87.60
76.72
7 2,33
75.76
65.40 1.05
69.20
50.90
As shown in Table 3 SFT with our synthetic datasets significantly improves model performance.
The consistent improvements across diverse benchmarks demonstrate that our synthetic data method does not overfit to any specific domain or benchmark, but instead enhances the model's 's general and robust reasoning capability. Notably, the multidisciplinary reasoning performance of the base mod- els fine-tuned our datasets even surpasses that of their official final models that have undergone the full post-training process across all benchmarks. This improvement particularly pronounced on highly complex reasoning tasks like GPQA-Diamond, These results affirm the efficacy of our data synthesis strategy for enhancing the multidisciplinary reasoning capabilities of LLMs. In addi- tion, we provide the SFT experiment results GSM8K and MATH-500 in Appendix

---

Published as a conference paper at ICLR 2026
Table Comparison with other baseline datasets on the Qwen3-8B-Base model.
Dataset
MMLU
MMLU-Pro
GPOA-Diamond
GPQA-Dlamond (CoT-SC)
SuperGPQA
Thoughts
72.49
57.76
45.86+ 1,80
54.04
39.70
Nemotron- Ceceeroon-Pos---Teeee Post-Training-vl
77.17
62.52
38,59+1.24 1.24
40.91
42.03
WebInstruct (Full)
86.34
72.83
55.61=2.50
62.63
45.37
NaturalReasoning
85.33
72.39
56.67
60.00
43.38
DLR-Wet
86.32
73.81
58.89-1.98
63.64
47.23
DLR-Book
86.43
74.98
600.5 1.93
66.67
47.04
90
85
66 1og
Web
Book
80
Accuracy (%)
Accuracy (%)
2774
70
70
65
50
60
MM
MMLU-Pro
Glemmnd SUperGPQA
50
46.5
50
HMLL
MMLU-PP
GPDA Diamond
GPOA Diemand (coT-SCC
SuperoPgA
0.30M 0.48M 0.76M 1.21M 1.92M 3.04M
Training Data Quantity (Log Scale)
Performance scaling with synthesized data size. Benchmark accuracy increases steadily with data scale.
Figure 5:
Performance comparison of models trained data synthesized from lower-quality web corpora versus higher-quality book corpora.
Figure 6:
5.3 COMPARISON WITH BASELINE DATASETS
We conducted a comparative analysis between our synthetically generated data and other prominent open-source synthetic datasets. To ensure a fair comparison, given the varying sizes of the original datasets, we randomly sampled an equal number of 304,18 181 instances from each dataset for the SFT experiments. For OpenThoughtss: and Nemotron-Post www---oaiiinn----... Training-vl, the synthetic datasets already contain elaborate long CoT responses, so we retain their native versions. In contrast, the native responses in WebInstruct (Full) and NaturalReasoning are relatively simple. To enable fairer comparison, we regenerate long CoT responses them using the same model and settings as in dataset, ensuring that performance differences arise solely from the questions.
As shown Table 4 the models trained on our datasets consistently outperform those trained on other baseline datasets across all benchmarks. Specifically, mance on MMLU, MMLU-Pro, and GPQA-Diamond, while DLR- Web attains the highest score on SuperGPQA and ranks second most other benchmarks. These results demonstrate the superior quality and effectiveness of our data synthesis approach compared to existing methods.
achieves the best perfor-
5.4 EFFECT OF DATA SCALING
To validate the scalability of our data synthesis methodology, we conducted experiments using the Qwen3-8B-Bas model, training on progressively larger subsets sampled from the DLR-Book dataset. The results, shown in Figure 5, demonstrate consistent positive correlation between the scale of the synthetic data and model performance across all benchmarks. The observed scaling laws confirm that method provides reliable pathway achieving superior model performance by synthesizing more data. Future work can leverage the Design Logics we provide to synthesize even larger datasets for continued improvement.
8

---

Published as a conference paper at ICLR 2026
5.5
EFFECT OF SOURCE CORPUS QUALITY
It is widely acknowledged that book corpora are of higher quality than web corpora (Brown et al., Gao et al. 2020 Rae et al., 2021). To evaluate the impact of source quality on our data syn- thesis method, we conducted an SFT experiment on the Qwen3-8B-Ban synthesized from the high-quality book corpus with that from the web corpus. To ensure a fair com- parison, we matched the disciplinary distribution across both sources. For disciplines with limited Win data, we used all available instances; otherwise, we applied random sampling. This procedure yielded two datasets of identical size (282,857 instances) and per-discipline distribution.
2020
model, comparing data
As shown in F the model fine-tuneds on data from the book corpus outperforms the one trained on data from the web corpus across most benchmarks, with the largest gains complex reasoning tasks such as GPQA-Diamond and SuperGPQA. Nevertheless, the performance gap induced by the two source corpora of differing quality is small, which demonstrates that our method is robust to variations in source quality and can reliably generate high-quality questions even when applied to lower-quality corpora.
5.6 ABLATION STUDIES
We conducted an ablation study on our question-synthesis pipeline using the Qwen3-8B-Be to the contributions of three key components: guidance from Design Logies, the coarse ranking, and the fine ranking of Design Logics. To ensure fair comparison, all ablation variants synthesized questions from the same book corpus by uniformly sampling 304,181 instances. Where applicable, and generated responses using the same we used identical retrieval with 2507-FP8. Specifically, we compared the following settings: model, Qwen3-235B 20003--33B--2222 B--B--A22-Thhnniee (aaeiinn---00--PF-
This is our complete method, which includes coarse retrieval of the top-5 Design Logies by similarity, followed by LLM-based fine selection and synthesis based on Design Logic.
DESIGNER:
w/o Design Logic This setting replaces use of explicit Design Logics with exemplar ques- tions. The retrieval process is identical to DESIGNER: semantic similarity-basee coarse ranking yields top-5 candidate questions, then the LLM performs fine-grained selection to choose the most suitable exemplar and generates a new question by imitating its style and structure.
w/o Coarse Ranking: This setting bypasses the semantic similarity- based retrieval of the top-5 relevant Design Logics. Instead, the LLM is prompted to select the most suitable logic from set randomly chosen Design Logics.
w/o Fine Ranking: This setting removes the LLM-based re-ranking stage. The single most similar Design Logic (the top-1 result from the coarse retrieval) is used directly to generate the question.
Table 5: Ablation study of our data synthesis pipeline.
Method
MMLU
MMLU-Pro
GPQA-Diamond
GPQA-Diamond (CoT-SC)
SuperGPQA
DESIGNER
86.43
74.98
6..35++1.93
66.67
47.04
w/o Design Logic
86.26
74.34
58.89++2.94
64.65
46.71
w/o Coarse Ranking
86.29
73.76
58.74:t0.75
62,63
46.23
w/o Fine Ranking
86.26
74.35
59.3411.20
63.64
46.81
DESIGNER (Full)
87.53
76.69
69.39.11877
73.74
50.57
w/o Design Logic (Full)
86.51
75.54
64.10:12.81
67.17
48.46
Table summarizes the results of our ablation studies. Our complete DESIGNER approach out- performs all ablated configurations across benchmarks. The superior performance of using Design Logics compared with directly using questions as exemplars confirms our hypothesis that explicit logical structures provide more accurate and robust guidance for high-quality question synthesis. Removing either the coarse-grained retrieval stage (w/o Coarse Ranking) or the fine-grained LLM re-ranking stage (w/o Fine Ranking) leads to performance drop, confirming the value of each stage in our two-step matching process. However, because these variants still benefit from the overall framework of our method, removing either stage only results in the use of suboptimal Design Logic. Consequently, the expected impact naturally limited. We further evaluated the "w/o De- sign Logic" variant the full book corpus of 3.04 million samples, where the performance gap

---

Published as a conference paper at ICLR 2026
becomes more pronounced. In particular, the difference on GPQA-Diamond reaches five percentage points, providing stronger evidence for the advantage of employing Design Logic.
6 RELATED WORK
Data Synthesis Paradigms. Existing data synthesis methods can be broadly grouped into query- centric and document-centric approaches. Query-centric approaches iteratively expand a seed query pool: Self-Instruct (Wang al. 2023) samples from an initial pool to generate new QA pairs, while Wizard LM (Luo et al. |024 Xu et al., 2023 Luo et al. 2025 and Auto Evol-Instruct (Zeng et al., 2024) evolve instructions for greater diversity and complexity. Col--elf-Instruct et. 2025) tegrates reasoning into generation, Prismatic Synthesis (Jung et al. 2025) emphasizes gradient-level diversity, and SPARQ (Havrilla et al. (025) evaluates difficulty and diversity, linking them to in- and out-of-distribution performance. However, these methods remain constrained by the coverage of the seed pool and the inherent model biases. Document-centric approaches instead derive ques- tions directly from raw corpora, ensuring stronger factual grounding: UltraChat (Ding et al. 2023) generates world- woollede questions from sources like Wikidata, Humpback (Li et al. 2024) infers instructions backward from documents, MAmmoTH2 (Yue et al.) (0224) mines web content via Recall-Exttact-Refine pipeline. NaturalReasoning (Yuan et al. (225)) focuses on generating high- difficulty reasoning questions, but its diversity is inherently limited by single prompt template. Furthermore, several studies have explored generating new questions through external guidance, such as KPDDS (Huang al., 2025). and the structurally guided approaches of Xu al. (2025), Wang et al. (2025), and Bu et al. (2025) However, these approaches have two primary limitations compared to our method. First, they are query-centric methods that synthesize new questions from seed questions, which inherently constrains them to the scope of the initial seed pool and prevents generation from richer raw corpora. Second, their methodologies are tailored for disciplines with strong logical and structural reasoning, such as mathematics and coding, and cannot be readily ex- tended to arbitrary multidisciplinary fields, such as the humamities, social sciences, and medicine. In contrast, method synthesizes challenging and diverse questions by introducing multidisciplinary Design Logics to align with the corresponding documents. This approach is discipline-agnostic,, making it applicable to any discipline.
Reasoning Data Synthesis. Another line of work primarily investigates the synthesis of CoTT' rea- soning data. DeepSeek (Guo et al. (025) transfers reasoning skills from DeepSeek-RI to open- source models, while OpenMathReasoning (Moshkov et al., 2025) augments math ability by gener- ating long CoTT traces for AoPS problems. OmniThought (Cai et al. 2025) collects multidisciplinary questions (math, code, science) to build a large CoT distillation dataset, and OpenThoughts (Guha et al.) 2025) systematically explores recipes for generating long reasoning traces across these do- mains. While these approaches primarily focus on generating high-quality reasoning processes for existing questions, they lack the capability to produce original, diverse, and multidisciplinary ques- tions, which is the central focus of our method.
7 CONCLUSION
In this paper, we introduced DESIGNER, a novel design-logic-guide data synthesis pipeline de- signed to address the scarcity of high-quality, multidisciplinary reasoning data for LLMs. Our core innovation is the concept of "Design Logic". which abstracts the strategic process that human ex- perts use to create challenging questions. By leveraging Design Logics, we generated two large-scale datasets, DLR-Book and Reb, comprising millions of complex questions across 75 disciplines from raw text sources. We validated the effectiveness of our approach through extensive experi- ments, demonstrating that models fine-tuned with our data achieve substantial performance gains in multidisciplinary reasoning. Importantly, the value of Design Logic lies not only in its effect on final performance but also in its controllable, abstractable, and generalizable synthesis mechanism. The guidance of the Design Logic is crucial for ensuring the difficulty and diversity of the synthesized questions. Section 4.1 shows that our Design Logies allow precise control over problem difficulty, enabling the generation of highly challenging questions far beyond baseline datasets and commonly used benchmarks. Section [42 shows that bessiinnlogicgguiieee synthesis achieves higher scores on all five diversity metrics compared with baseline datasets.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
All materials used in this study were subjected to filtering to minimize the inclusion of potentially harmful content. We encourage researchers who utilize our datasets to implement stricter filtering and processing measures to further ensure that the data conforms to the principles of ethical use.
REPRODUCIBILITY STATEMENT
We provide all complete and detailed data processing steps used for data synthesis in the main paper and in the appendix. To facilitate reproducibility and independent verification, we have released subset of our synthesized data that does not involve legal compliance concerns or potential con- flicts of interest. The released data include our synthesized questions, reference answers, long CoT responses, the corresponding source corpora and Design Logic, as well as associated labels (e.g., discipline, difficulty, question type), enabling reproduction and validation of our method's s effective- ness and the quality of the synthesized data. Furthermore, we have released a design-logic library that contains all Design Logics extracted in this study, together with discipline, difficulty, and ques- tion type labels, enabling the research community to directly match these Design Logics to their own raw documents and synthesize new questions that exhibit the same complex reasoning.
THE USE OF LARGE LANGUAGE MODELS (LLMS)
We employed Large Language Models (LLMs) to assist in polishing our paper and writing code.
REFERENCES
Mohiuddin Ahmed, Raihan Seraj, and Syed Mohammed Shamsul Islam. The k-means algorithm: A comprehensive survey and performance evaluation. Electronics, 9(8):1295, 2020.
Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran E1- Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, et al. Llama-nemotroon Efficient reasoning models. arXiv preprini 2rX5v:2505.00049 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint wrXiv:2005.1416. 2020.
Tianpeng Bu, Minying Zhang, Hongtao Duan, Shurui Li, Lulu Hu, and Yu Li. Enhanced data synthesis for I1m through reasoning structures generated by hierarchical gflownet. In Findings of the Association for Computational Linguistics: ACI. 2025, pp. 15931-15958, 2025.
Wenrui Cai, Chengyu Wang, Junbing Yan, Jun Huang, and Xiangzhong Fang. Reasoning with om- nithought: A large dataset with verbosity and cognitive difficulty annotations. arXiv preprint arXiv:2505.1093/ 2025.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversa- tions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp.
2023.
Wenchao Du and Alan W Black. Boosting dialog response generation. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 38 43-43, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 10.18853/v1/P199 1005. URL wtt.tt/s///cclaattoooooooor////p

---

Published as a conference paper at ICLR 2026
Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. Supergpqa: Scaling Ilm evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502.14739 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00022 2020.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, al. Openthoughts: Data recipes for reason- ing models. arXiv preprint arXiv:2506.04]1 04178, 2025.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501. 2025.
Alex Havrilla, Edward Hughes, Mikayel Samvelyan, and Jacob Abemethy. Synthetic problem gen- eration for reasoning via quality-diversity algorithms. arXiv preprint arXiv:2506.0644 06499, 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning. Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 24176-24184 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412. 16720, 2024.
Jaehun Jung, Seungju Han, Ximing Lu, Skyler Hallinan, David Acuma, Shrimai Prabhumoye, Mostafa Patwary, Mohammad Shoeybi, Bryan Catanzaro, and Yejin Choi. Prismatic synthe- sis: Gradient-based data diversification boosts generalization in Im reasoning. arXiv preprint 2005: 2025.
Yi-An Lai, Xuan Zhu Yi Zhang, and Mona Diab. Diversity, density, and homogeneity: Quantita- tive characteristic metrics for text collections. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 1739- 1746, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10- 15546-34-4 URL https: /tttt:::/aclanteee www.laannoooooooo////ee00
www.:reeeee.c 1.515
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. In ICLR, 2024.
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024. URI. https://huggingfatt htt./ffcc..co/daaaaaeeeeeeeeee co/datasets HuggiingFacee//Fiiewbeeed
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian Guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, et al. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct. The Thirteenth International Conference on Learning Representations, 2025.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, 2024.
12

---

Published as a conference paper at ICLR 2026
Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building ematical reasoning models with openmathreasoning dataset. arXiv preprint arXhv:2504.1699/ 2025.
math-
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.0877. 2023.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang. Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint 1.X1v:2500.14249,, 2025.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11444 2021.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-prov q&ea bench- mark. First Conference on Language Modeling, 2024.
Katherine Stasaski and Marti Hearst. Semantic diversity in dialogue with natural language infer- ence. In Marine Carpuat, Mathe-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pp. 85-98, Seattle, United States, July 2022. Association for Computational Linguistics. doi: (t065///122222.maclmmaoa 0r22222 seaacc-mainn 67
URL
Trieu Trinh, Yuhuai Wu, Quoe V Le, He He, and Thang Luong. Solving olympiad geometry
without human demonstrations. Nature, 62577995):476-48. 2024.
Iulia Ture, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better:
On the importance of pre-training compact models. arXiv preprint arXiv: 190.00862, 2019.
Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry. Do large language model benchmarks test reliability? arXiv preprint arXiv:2502.03461, 2025.
Jiapeng Wang, Jinhao Jiang. Zhiqiang Zhang, Jun Zhou, and Wayne Xin Zhao. Rv-syn: Rational and verifiable mathematical reasoning data synthesis based on structured function library. arXiv preprint ar arXiv:2504.2044 2025.
Ke Wang, Jiahui Zhu, Minjie Ren, Zeming Liu, Shiwei Li, Zongye Zhang, Chenkai Zhang, Xiaoyu Wu, Qiqi Zhan, Qingjie Liu, al. survey on data synthesis and augmentation for large language models. arXiv preprint arXiv:2410.1 /2896, 2024a.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi. and Hannanch Hajishirzi. Self-instruct Aligning language models with self self-generated instructions. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguisties (Volume
1: Long Papers), pp. 13484-13508, 2023.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266-95290, 2024b.
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et Smarter, bet- ter, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. arXiv preprint arXiv:2412.13660 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe V Le, Denny Zhou, et al. Chain-of th-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24824-24882 2022.
13

---

Published as a conference paper at ICLR 2026
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXhv:2304.12244 2023.
Lei Xu, Sirui Chen, Yuxuan Huang, and Chaochao Lu. Synthesis by design: Controlled data gener- ation via structural guidance. arXiv preprint urXiv:2506.07666 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang. Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
Yuming Yang, Yang Nan, Junjie Ye, Shihan Dou, Xiao Wang, Shuo Li, Huijie Lv, Mingqi Wu, Tao Gui, Qi Zhang, et al. Measuring data diversity for instruction tuning: A systematic analysis and a reliable metric. arXiv preprint arXiv:2502. 171844 2025b.
Ping Yu, Jack Lanchantin, Tianlu Wang. Weizhe Yuan, Olga Golovneva, Ilia Kulikov, Sainba- yar Sukhbaatar, Jason Weston, and Jing Xu. Cot-self-instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks. arXiv preprint arXiv:2507.23751, 2025.
Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Ilia Kulikov, Kyunghyun Cho, Dong Wang, Yuandong Tian, Jason E Weston, al. Naturalreasoning: Reasoning in the wild with 2.8 m challenging questions. arXiv preprint 1rX1v22502.13124 2025.
Xiang Yue, Tianyu Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. Advances in Neural Information Processing Systems, 37:90629-90666 2024.
Weihao Zeng, Can Xu, Yingxiu Zhao, Jian San-Guang Lou, and Weizhu Chen. Automatic instruction evolving for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural anguage Processing, pp. 6998-7018, 2024.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseck-coder-v22 Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406. 11911, 11991,2022 2024.
14

---

Published as a conference paper at ICLR 2026
A
DATA COLLECTION DETAILS
We curate three primary data sources and define a comprehensive discipline taxonomy:
Web Corpus: For the web corpus, we employ FineFineWett a filtered subset of the Common
Crawl dataset.
Book Corpus: We utilize a proprietary library of books.
Question Bank: A proprietary repository of examination and practice items.
Discipline Taxonomy:
We have established disciplinary classification system comprising 75 dis-
tinct disciplines, as shown in Table [6] This discipline taxonomy provides comprehensive coverage across several major areas:
STEM: Science, Technology, Engineering, and Mathematics.
Humanities and Social Sciences: Including fields such as law, philosophy, and sociology.
Applied and Professional Fields: Encompassing domains like clinical medicine, education, and business administration.
Arts.
B DESIGN LOGIC DEDUPLICATION ALGORITHM
Algorithm Graph-based Deduplication via Centroid Selection
1: Input: A set of items D {d1,.... ................................ dn); similarity matrix S RR*", similarity threshold
т.
2: Output: A deduplicated set of representative items R.
3:
4: Initialize undirected graph G (V, E) where V nI and E 0.
5: Initialize the set of representatives R 0.
6:
7: // Build a similarity graph where nodes are items and edges connect similar items.
8: for
to n for j = + to do if S, > r then
9:
10:
11:
Add edge (i,j) to E.
12: end
13: end for
14:
end for
15:
16: // Identify clusters of duplicates by finding connected components.
Let C + FinnaCCooneeeeCoooooeeeeee
18:
// Select the most representative item (centroid) from each cluster.
20: for each connected component C E C do
21: Find centroid index i* arg max Ejec.j4 Siz.
iEC
22: Add item d,- to R.
23: end for
25: return R.
15

---

Published as a conference paper at ICLR 2026
C
DESIGN LOGIC AND QUESTION QUANTITY
Table Number of Design Logics, book questions, and web questions by discipline.
Discipline
Design Logic
Book Question
Web Question
Aerospace Science and Technology
980
15025
1815
Agricultural Engineering
966
14950
11312
Agricultural Resources and Environment
409
14978
1443
Animal Husbandry
121
15023
318
Archaeology
776
15061
2572
Architecture
801
14931
4235
Art and Design
998
15164
12720
Astronomy
1600
40010
14915
Atmospheric Sciences
973
15029
3416
Basic Medicine
2289
49930
7118
Bioengineering
1041
11216
1059
Biology
4654
200078
111988
Biomedical Engineering
790
15120
5439
Business Administration
3873
99739
46337
Chemical Engineering and Technology
1393
20014
5235
Chemistry
6430
199839
68211
Chinese History
784
14876
14564
Chinese Language and Literature
998
14984
22233
Civil Engineering
950
14899
7773
Clinical Medicine
2844
99735
77182
Computer Science and Technology
4674
99796
219474
Control Science and Engineering
1338
20013
3254
Ecology
968
15113
4285
Economics
3864
99906
20064
Education
1287
20018
9824
Electrical Engineering
2904
59937
36949
Electronic Science and Technology
1445
19880
18864
English and Foreign Languages
985
29908
1344
Environmental Science and Engineering
1478
19954
28226
Ethnology
210
14856
51
Food Science and Engineering
768
15073
2470
Forensic Medicine
295
15010
34
Geography
2812
59987
12295
Geological Resources and Geological En-
173
15058
374
gineering
Geology
987
14958
5447
Geophysics
982
20050
3736
History of Science and Technology
494
14991
111
Hydraulic Engineering
664
14973
649
Information Resources Management Information and Communication Engi-
356
14968
69
2367
39825
4777
necring
Instrument Science and Technology Joumallis and Communication Law
687
15017
378
932
14847
4326
2706
80110
62699
Management Science and Engineering
1317
20020
10440
Marine Sciences
852
14968
1228
Materials Science and Engineering Mathematics
1949
40121
12263
9884
299464
181537
Mechanical Engineering
2955
59998
55356
Mechanics
1752
40046
4012
Continued on next page
16

---

Published as a conference paper at ICLR 2026
Table (Continued) Number of Design Logics, book questions, and web questions by discipline.
Discipline
Design Logic
Book Question
Web Question
Mining Engincering
282
14986
438
Naval Architecture and Ocean Engineering
571
15034
515
Nuclear Science and Technology
1448
19988
1981
Nursing
767
15078
2228
Optical Engineering
927
14995
3272
Petroleum and Natural Gas Engineering
571
15040
988
Pharmacy
1829
39921
12756
Philosophy
4363
100029
128004
Physical Education
890
14912
10712
Physics
5768
199771
104982
Political Science
3268
59686
52586
Power Engineering and Engineering Ther-
704
14918
1199
mophysics
Psychology
4336
99502
95585
Public Administration
960
14970
26669
Public Health and Preventive Medicine
1949
39820
19649
Remote Sensing Science and Technology
344
8081
271
Safety Science and Engineering Sociology
835
14986
409
2093
39941
32361
Statistics
3388
59950
24806
Stomatology
592
15002
456
Surveying and Mapping Science and Tech-
509
14964
358
nology
Textile Science and Engineering
187
14978
325
Transportation Engineering
882
14929
4956
Urban and Rural Planning
576
15052
89
Veterinary Medicine
547
14918
3022
World History
987
39703
5503
Total
125328
3040620
1658541
D BASELINE DATASETS
Table 7: The baseline datasets utilized in our experiments. For the Nemotron- Post-Training-v1 dataset, we used the data from the math and STEM categories.
Dataset
URL
OpenThooughss
http:///nggigggeecco/ddttttttttt OpenThoughts3-1.2W
Guha et al. 2025
Nemotron Ponnorron-Post--io Training-vi
https//huggiifacccooo/attttttttt
(Bercovich al. 2025
WebInstruct (Full)
https///hnguunffeecccoo/tttttttt WebInstructFull
Yue et al. 2024)
NaturalReasoning
httss//nggiiiaeeeooaaaattttttetr reasoning
Yuan et al. 2025
17

---

Published as a conference paper at ICLR 2026
E QUESTION TYPE ANALYSIS
As shown in Table g problem-solving questions form the majority of our synthesized datasets, constituting 64.92% of the book corpus and 63.91% of the web corpus. Multiple-choice questions are the next most common type, comprising 29.94% and 32.43%, respectively. This distribution analysis indicates that the composition of our synthesized questions is skewed towards reasoning- based types, as opposed to simple recall-oriented ones.
Table 9: Distribution of questiin types in synthesized datasets.
Type
DLR-Book
Problem-solving question
64.92%
63.91%
Multiple-choice question
29.94%
32.43%
Proof question
4.39%
2.67%
Other question types
0.75%
0.99%
F DETAILED DIFFICULTY ANALYSIS
Table 10: Difficulty distributions of questions across different datasets and benchmarks.
Dataset
Easy
Medium
Hard
Very Hard
GSM8K
78.09%
21.91%
0.0%
0.0%
MMLU
60.51% %
34.09%
5.29%
0.11%
MMLU-Pro
34.48%
44.04%
20.86%
0.62%
GPQA-Diaman
0.51%
19.7%
71.72%
8.08%
GPQA-Main
0.22%
26.56%
67.41%
5.8%
SuperGPQA
35.97%
32.18%
29.88%
1.98%
Open Thoughts3
1.09%
19.32%
70.51%
9.07%
- -eee--Postttiiiiie Dost-Training-v
32.27% %
48.61%
16.09%
3.04%
WebInstruct (Full)
39.02%
39.58%
17.84%
3.57%
NaturalReasoning
2.10%
26.25%
40.54%
31.11%
DIR-Wel
0.72%
16.13%
36.24%
46.91%
DLR-Book
0.27%
9.88%
35.18%
54.66%
G DETAILED DIVERSITY METRICS
We first generated high-dimensional vector representations for each question using the Qwen3- Embedding-4B model. For given question set X = {x1, .2,....,NN.. }. this process yields cor- responding set of embedding vectors E {ei, ................................ eN where e; Rd is the d-dimensional embedding for question F;. We uniformly sample N = 300,000 questions from each dataset and compute the following five distance-based metrics in the embedding space.
1. Mean Cosine Distance: The average cosine distance between all unique pairs of embed- dings, calculated as Meosine N(N-1) Eic cos(e;, A higher value indicates greater semantic dissimilarity.
2. Mean 1.2 Distance: dings, calculated as ML2 N[ Eiej ||e - eyll2- This measures the average separa- tion of questions the embedding space.
The average Euclidean distance between all unique pairs of embed-
3. 1-Nearest Neighbor Distance (1-NN Distance): The average cosine distance from each embedding its single nearest neighbor, given by Mi-NN = N Σ, de(e;). where dj (e,) is the cosine distance from e; to its closest neighbor. This metric highlights the presence of tightly clustered, near-identical questions.
18

---

Published as a conference paper at ICLR 2026
4. Cluster Inertia:
The total squared distance of samples to their closest cluster center after applying the K-means clustering algorithm. It is calculated as Mueria اi=1 min, ||e; Cý llp where Gj the cluster centroids. This measures the overall spread and density of the data clusters.
5. Radius: The geometric mean of the standard deviations of the embedding dimensions, modeling the data as a multi-dimensional Gaussian distribution: MRadius = (I[ az) 1'd where o is the standard deviation along the j-th dimension. It directly quantifies the spread of the data in the semantic space.
H EXPERIMENTAL DETAILS
Table 11:
Hyperparameters for Supervised Fine- MuniiTunig (SFT).
Table 12: Generation configuration for evalua- tion.
Parameter
Value
Parameter
Value
Epoch Batch Size
6
Temperature
0.6
64
Top-K
20
Learning Rate
le-5
Top-P
0.95
Learning Rate Schedule
Cosine decay
Max Context Length
32768
Table 13: Evaluation benchmarks, their disciplines, and the number of rollouts (N),
Benchmark
Disciplines
Rollout (N)
MMLU (Hendrycks et al. 2020)
57 disciplines including STEM, human- ities, social sciences, and professional fields
MMI.U-Pr Wang et 2024b)
Math, Physics, Chemistry, Law, Engi- neering, Economics, Health, Psychology. Business, Biology, Philosophy, Computer Science, History, Other
GPQA-Diamond (Rein et 2024
Physics, Chemistry, Biology
10
GPQA- Maiai (Rein et 2024
Physics, Chemistry, Biology
10
SuperGPQA (Du et al. 2025)
285 graduate-level disciplines across 13 fields (e.g., Engineering, Management, Economics, Education, History, Science, Medicine, Law, Sociology, Philosophy, Agriculture, Literature)
19

---

Published as a conference paper at ICLR 2026
I PERFORMANCE ON GSM8K AND MATH-500
We evaluated the models trained on our data Table 3 on GSM8K and MATH-500.
Table 15: The SFT experiment results on GSM8K and MATH-500, The models trained DLR- Web, DLR-Book, or the combined DLR-Wer + DLR-Bood dataset.
Model
GSM8K
MATH-500
Hamaa-3.2-3B-mme
68.46
35.75=5.60
Muma-3.2-3B-SF (DLR-Web)
79.98
44.35+2.24
lama-3.2-3B-SFF (DLR-Book)
81.05
51.45++.42
Llama-3.2-3B-SSE (e.--We++Book)
86.66
61.90+1..75
Jamm--31-8DDInstrruu
81.12
46.25
Llama-3.1-8B-ST (DLR-Web)
93.03
80.20:10.71
llma-3.1-8B--FF (DLR-Book)
92.65
82.15
Jama-3.1-8B-SFT
93.18
86.45+0.68
Qwe33-4B (Thinking Mode)
94.77
92.75=0.50
Qwen3-4B-Base-SFF (DLR-Wei) Web)
94.31
87.95+0.50
Qwen3-4B-Base-95 -as--FFs- (DLR-Book)
94.47
89.45+1.30
Dwen3-4B-Base-SPP (DLR-Web+Book)
94.69
89.35
Qwen3-8B (Thinking Mode)
95.60
93.35-10.09
Semn--B-Basse-aee 1DLR-Web
95.07
91.05=+0.54
Qwen3-8B- Base-SFT
1aa3-8B-BaaseSSi (DLR-Book)
95.60
91.75+00.6
Qwen3-8B Bwen3-8B-Base--e Base-SFT (DLR- Web+Book)
95.60
91.85++0.33
GSM8k and MATH-500 exclusively focused mathematical problems. Our dataset's objective is not mathematics; rather, it is designed to create a multidisciplinary dataset. Nevertheless, the model trained our data still exhibits competitive performance in the mathematical domain.
For MATH500, the slightly lower performance of our models compared with the official Qwen3 models that have undergone the full post-training process is likely due the fact that the official Qwen3 models were trained with substantially larger volume of high-quality, expert-written written math- ematical data and received more extensive post-training. In contrast, mathematics represents only about 10% of our dataset, and our math questions are synthesized from raw web and book corpora rather than curated by human experts, which naturally leads lower quality relative professional math-focused training data.
Notably, models trained on either DIRR-ook or DLR- Web+Book achieve 95.60% on GSM8K, matching the official Qwen3-8B score. GSM8K performance is already near saturation. Prior work (Vendrow et al.] 2025) reports that approximately 5% GSM8K questions are mislabeled ambiguous, which makes 100% accuracy unattainable and positions 95.60% near the practical upper bound. Even stronger LLMs fall within a similar range. For instance, according to the Llama 3 technical report (Dubey al. 2024), GPT-4o reaches 96.1 GSM8K under 8-shot CoT setting.
20

---

Published as a conference paper at ICLR 2026
J PROMPTS
Prompt for Discipline Classification
You are professional multidisciplinary data labeling expert specializing in the classification of multidisciplinery academic questions. Please select the ONE nost relevant label from the given 1ist of discipline lsbels for the input question data, For question data that cannot deternine, the "Unknown Discipline" label. Please directly autput "labels"; "(the label you selected)".
Discipline Labels: ("Mathenaties" 'Biclogy', 'Chenistry', 'Physics', 'Caputer Science and Technology' 'Philosophy', "Psychology', "Business kdministration'. "Clinical Medicine', "Economics', 'Law', 'Palitical Science' "statistics", 'Electrical Engineering', 'Geography', "Mechanical Engineering', 'Basic Medicine", *Information and Comuniccatio Engineering'. 'Sociology', "Haterlals Science and Engineering'. "Pharnacy', 'Public Health and Preventive Medicine', 'Mechanics', 'Astronony', 'World History', "Bioenginerring" *English and Foreign Languages', 'Chenical Engineering and Technology'. "Electronic Science and Technology', Engineering', 'Nuclear "Management Science and Engineering' 'Education', 'Geaphysics', 'Art and Design', 'Agricultural Engineering' Aerrsspace Sciance Technology', 'Atmospheric Sciances', "Chinese Language
"Enviromenttll
and Technology',
Engineering'
Literature', Engineering", "Ecology', "Geology", 'Nursing', "Optical Engineering'. 'Public Adrinistration' 'Journalisn end Communication, 'Physical Education'. 'Harine Sciences', "Safety Science and Engineering' 'Architecture', Engineering
Science and Engineering" 'Archaeology'' 'Blonedical Engineering', 'Chinese History', 'Veterinary Medicine', 'Instrurent Science Technology', 'Hydraulic Engineering' "Stonatelogy' "Urban
'Transportation Engineerina'.
Engineering Thermohhyiccs'
Planning' 'Petroleun Natural Gas Engineering' 'Naval Archi tecture and Ocean Engineering', 'Surveying and Mapping Science and Technology', 'History Science and Technology', 'Agricultural Resources Environment', 'Renote sensing Science and Technology', "Information Resources Management', "Mining Engineering", "Forensic Medicine', 'Ethnology', 'Textile Science and Engineering', 'Geological Resources and Geological Engineering', 'Animal Husbandry", '0ther', Non-diicipllnarry. 'Unknown Discipline')
Example
speed light. How does the photon experience space, the implications of relativistic beaning its perception of spatial dinensions? Provide detailed explanation, including any relevant nathenatical derivations and physical principles." Output: "labels": "Physics"
Input: "Consider a photon traveling
Example
Input: heavy pole, mass length L, freely hinged other end of the pole, B, to fixed point A on the wall above a. The system is in equilibrine ibriun, with the pole with horizontal, the rope making an angle of 8, affect its equilibriun and stability."
point rape connects
making angle
horizontal. Explore
the system's paraneters Output: "labels": "Mechanics"
Example
Input: "If John rented car far $150 and had to buy 8 gallans of $3.50 50 per gallon to fill it and final expense is $0.50 per mile, how much it cost hin to drive miles?" Output: "labels": "Nathenatics"
Input Question Data Input: "(text)" Output:
Figure 7: The few-shot prompt used for discipline classification. The model is presented with a list of disciplines and three examples, and then asked to classify given question, which replaces the (text) placeholder.
21

---

Published as a conference paper at ICLR 2026
Prompt for Difficulty Classification
You are an expert in education and examination, specializing in classifying the difficulty levels of multidisse multidisciplinary questions. For the given question, please evaluate its difficulty based the complexity and length of the reasoning required answer it. Label it one of the following: **Easy**, **Medium**, **Hard*** or **Very Hard**. Please directly output "Difficulty: (Your chosen label)".
Example
Input: "Consider a photon traveling at the speed of light. How does the photon experience space, and what are the implications of relativistic beaming on its perception of spatial dimensions? Provide detailed explanation, including any relevant mathematical derivations and physical principles." Output: "Difficulty: Very Hard"
Example 2
Input: "A heavy pole, of mass and length L, is freely hinged to a wall at the point 0. rope connects the other end of the pole, B, to fixed point A on the wall above 0. The system is in equilibrium, with the pole making angle of 0 with the horizontal, and the rope making an angle system's s parameters (M, L, 0, a) affect equilibrium stability." Output: "Difficulty: Hard"
with the horizontal. Explore how the
Example 3
Input: "If John rented car for $150 and had to buy 8 gallons of gas at $3.50 per gallon to fill it up, and the final expense is $0. 50 mile, how much did it cost him drive 320 miles?"
Output: "Difficulty: Easy"
Given Question
Input: "[text)" Output:
Figure 8: The few-shot prompt used for difficulty classification. The model is presented with three examples and is then asked to classify the difficulty of a given question, which replaces the (text) placeholder.
22

---

Published as a conference paper at ICLR 2026
Prompt for Question Type Classification
You expert in education and examination, specializing in classifying question types. For the given question, please evaluate its question type and label it the following: exProblem-solving question*x, swhultiple-choice questionæx, **Proof questionkx, or k*Other question tyреs**, For any question that you cannot determine, use the "Other question types" label. Please directly output "Question type: (Your chosen label)".
Example 1
Input: "Determine the number of k-letter sequences composed of the letters A and B such that sequence contains least consecutive A's." Output: "Question type: Problem-solving question"
Example 2
Input: "Consider the function f(x) 1 (=- ---
The value of the integral I
de is
Output: "Question type: Other question types"
Example 3
Input: "Given that a 3,, 3h. if f(x) x² is an odd function and monotonically increasing on (0, +00), then the possible values of the real number
are )
A: 1,3
B: 3.3
C: -1,4,3
D: 3: 2.3.3"
3.
Output: "Question type: Multiple-choice question"
Given Question
Input: "[text}"
Output:
The few-shot prompt used for question type classification. The model is presented with three examples and is then asked classify the type given question, which replaces the (text) placeholder.
Figure 9:
23

---

know the result Working backward from this final state Published as a conference paper at ICLR 2026
Prompt for Reasoning- Filtering
You will be provided with text from the internet.
Evaluate the following text extract for its potential usefulness for studying reasoning process. Use the following 5-point scoring system described below. Start from 0, points accumulated based on the satisfaction of each criterion:
(1) Add 1 point if the extract contains any reasoning or thinking process.
(2) point the extract contains any explicit subgoal setting, where the writer breaks down the prablem into smaller, intermediate goals. Subgoal setting might look like:
"First, need to find .... then
determine
"To solve .... let's first .... then three parts: (1) (2) and (3)
"Let's tackle "To ...,, I'1l first ..., then ...."
(3) Add point if the extract contains any verification steps. We want to mark instances where the writer explicitly checks their own work, such as by comparing the result to a known value or by checking the result of a calculation. Verification steps might look like:
"Let's check "To verify this is correct, I'll ...."
"Let's test with a simple case:
"To ensure this solution is valid, I'1l check if
(4) Add 1 point if the text contains any backtracking behavior, where the writer realizes path won't work explicitly goes back to try different approach. example of backtracking is: "Let me try again", "Wait", "I made a mistake", or "we need to try a different sequence of operations". We want to mark instances where the writer abandons thought and backtracks to previous computation.
(5) Add point if the text contains any backward-chaining behavior, where the writer is working towards goal but starts from the goal and works backward. It might like: "To solve ...., let's start with what want prove: Le.Let's verify this." "
"If we want find .... let's start with the desired result and work backward." " "To determine using
Task Format
Format your response in markdown as follows:
Thoughts
[Brief description describing what behavior was noticed and where subgoal setting may have occurred, less than 100 words]
Final score
[total points]
Text evaluate reasoning degree (text)
Respanse
Figure 10: The prompt used for the reasoning-orientel filtering task. It defines five-level scoring rubric to assess the usefulness of a text (which replaces the (text) placeholder) for studying rea- soning.
24

---

Published as a conference paper at ICLR 2026
Prompt for Design Logic Extraction
You are an expert educator and specialist exam question design. Below, I have provided an exam question. Your task is to deduce the thought pracess of the question designer. Analyze how they constructed this question based on the relevant knowledge points. You need to go beyond the specific details of the question and its knowledge points to abstract and summarize the underlying design logic and principles behind the question.
The goal is for to be able other high-quality, challenging questions that require complex logical reasoning for different knowledge points and source materials.
this abstracted design logic create
**Finally, you must organize the abstracted question-desige logic you have summar ized into English Mermaid format.** **
Analyze the Question Design Logic from the Following Question
**Question:** (text}
Figure 11: The prompt used for Design Logic extraction. The model is instructed to reverse-engineer the thought process behind given question (which replaces the (text) placeholder) and to structure the abstracted logic Mermaid format.
Instruction for Design Logic Retrieval
Given book snippet, retrieve the most suitable question-design logic in Mermaid format for creating challenging exam question from the book snippet.
Figure 12:
The task-specific instruction used for retrieving the most suitable Design Logic for given text segment. Embeddings for both text segments and Design Logics computed under this instruction using the Qwen3- @wven3-Embedding-4P model, enabling similarity-based retrieval.
25

---

accurate reference answer the question you Published as a conference paper at ICLR 2026
Prompt for Question Synthesis
You are an expert in the field of education and exanination design, and you are writing exan questions. Your use provided text generate high-quality exan question. follow steps generate an English exan question and refarence answer:
sel. Create an Exan Question:*s
write chalingging question at the gradinnee-eeeel above. Below are five question-design logics provided in Hermaid format. You need to select the nost suitable question-design
Based the provided
for creating challenging question cor respondinin ques tiestiio-eesign logic and steps to create challenging question. Please record which design (by number) and output corresponding nuneric The question should require critical thinking and test deep understanding and soblen--solving skills, not just simple fact recall.
strictly follow
logic you
be self-contained and answerable thout using the requires an answer based on the content of the source text, you nust include the corresponding content and information
The question
the question you write
question itself Ensure the question is self-contained, clear, without missing information or anbiguity, and has correct
source
self-contained.
For nultiple-choice questions, should first analyze and deternine the answer, then design the options to ensure that ane specific option is the correct answer. The questions you design need to include as many options possible (four more).
limited only four options (A, D).
xx2. Provide the Reference Answer:sz information just created.
source
If there is final, single result conclusion (like a number, formula, or short phrase), state it clearly is: \boxes(ansverr. Otherwise, not output \boxed( at the end with: "The final
**At the end of your response, please organize your results into the following JSON format:**
quxxtigues "x(Your question here)*", "reference _nnserr: "*(Your
answer goes here)*", "id": "*(The ID of the logic you selected here)*"
Question Design Logic "Merrmidd
(logic1)
Question-Design Logic -ix
"*Mernaid
(logic2)
Question-Design Logic -*x
"Merraad
(logic3)
Question-Deeeee Deiignn Logic ""Mrrmiid
(logic4)
-- Question-Design Logic -**
""Mrrmadd (logic5)
Source Text for Question Creation (text)
-**
Figure 13: The prompt used for question synthesis. The LLM is provided with a source text and five candidate Design Logics retrieved via semantic similarity. It instructed select the most suitable logic and strictly follow it to generate graduate-level question and corresponding reference an- swer, structured in a ISON format. The placeholders (logic1} through (logic5) and (text) are replaced with specific Design Logics and the source text, respectively.
26

---

Published as a conference paper at ICLR 2026
Prompt for Question Answerability Evaluation
Your task is to review question and decide whether it provides sufficient condi tions derive solutions. If the question is complete and answerable one, output 'Yes'. Open-ended questions are considered answerable. it is an incorrect question that cannot be answered, output 'No'. Output only 'Yes' or 'No'.
"question": "(question)"
Output:
Figure 14: The prompt used to evaluate whether the synthesized questions are complete and answer- able. The placeholder (question) is replaced with the specific synthesized question.
Prompt for Design Logic Consistency Evaluation
Your task is to judge whether "question" is designed according to the Mermaid-format "design logic". You should ignore superficial wording changes. If the logic is consistent or if there are implicit references that preserve consistency, output 'Yes'. If the design logic underlying the question is not consistent with the "design logic", output 'No'. Output only 'Yes' or 'No'.
"design logic": "[disi"
"question"; "[question)"
Output:
Figure 15: The prompt used to evaluate the consistency between the synthesized question and its corresponding Design Logic. The placeholders the specific Design Logic and the corresponding synthesized question, respectively.
and {quest couestion) iion are replaced with
Prompt for Answer Consistency Evaluation
You expert reviewer who compares "reference _nnswer"__"wr""" with model "response" for the same question. Decide whether both provide the same final answer. Ignore differences in reasoning or wording, and output "Yes' when the final answers or key conclusions are identical. Output 'No" when the conclusions differ. Output only 'Yes' or 'No'.
"question": "(question)"
lifferrence_nswer"" "[refereece__answer
"response":
"(response)"
Output:
Figure 16: The prompt used to evaluate the consistency between the reference answers generated by DeepSeek-Rt 1-0528 and the CoT responses generated by Qwen3-235B-AA2B 225-B--A2BB-Thiiing------ Th50king-2507- FPS. The placeholders (question), (reference_ answern), and [response} are replaced with the synthesized question, the reference answer, and the CoT responses, respectively.
27

---

Published as a conference paper at ICLR 2026
Prompt for Discipline Label Evaluation
You are reviewing exam questions and their author-provided discipline labels. Judge whether the discipline label appropriate for the question. Answer 'Yes' when the label is plausible answer 'No' when there clear mismatch. Output only 'Yes' or 'No'.
"question": "[question)" "discipline label": "[discipline]"
Output:
Figure 17: The prompt used evaluate the appropriateness the discipline label. The placeholders [question) and (discipline) are replaced with the specific question and its discipline label.
Prompt for Difficulty Label Evaluation
You are reviewing exam questions and their author-provided difficulty labels. Judge whether the difficulty label is appropriate for solving the question. Answer 'Yes" when the label is plausible and 'No" when there is clear mismatch. Output only 'Yes' or 'No'.
"question": "(question)" "difficulty label": "[difficulty]"
Output:
Figure 18: The prompt used evaluate the appropriateness of the difficulty label. The placeholders (question) and (difficulg ty} are replaced with the specific question and its difficulty label.
Prompt for Type Label Evaluation
You are reviewing exam questions and their author-provided type labels. Judge whether the type label is appropriate for the question. Answer "Yes" when the label is plausible and answer "No" when there is a clear mismatch. Output only 'Yes' or 'No'.
"question": "[question)" "type label"; "[atype]"
Output:
Figure 19: The prompt used evaluate the appropriateness of the question type label. The place- holders [question) and (qtype) are replaced with the specific question and its type label.
28

---

Published as a conference paper at ICLR 2026
K DESIGN LOGIC EXAMPLES
In this section, we provide Design Logic examples for several representative disciplines, such as Computer Science and Technology, Clinical Medicine, Mathematics, Law, Psychology, and Archae- ology. Each example includes the source code in Mermaid format and the corresponding visual flowchart.
graph TD
A[" Problem Domain --> B["Sorted Arrays & Median"]
B ->> C[*Кey Insight: Partitioning Without Merging")
C --> D A["Algor Selection: Binary Search
E["Optimization: O(log(min(m, n)))
E F["Edge Cases"]
F --> F1["Empty Arrays"]
F --> F2["Full-Array Partition"]
F F3[*Even/Odd Length Handling"]
10
Constrains Design"]
II
G - G1 Gl Input Sizes: 10^6"]
G
- G2 [2 Complexity: O(min m, log n))"]
G --> G3 Auxiiiilrr Space: 0((m+n)/2) hint for naive approach "]
T[*Test Cases"]
15
H
H1 ["Example 1: Odd Length"]
16
H
- H2 [2[ [xxample 2: Even Length"]
H
- H3 [B["Edee ["Edge: One Array Empty"]
['Edee: Both Empty"]
19
One Array Smaller
(a) The Design Logic formatted as Mermaid source code.
Volde baye
Larple Geltrne
Eaeak
(b) The flowchart generated from the Mermaid code.
Figure 20: An example of the Design Logic for a Computer Science and Technology problem, showing the Mermaid source code (a) and the corresponding visual flowchart (b).
29

---

Published as a conference paper at ICLR 2026
graph TD
A[Select Core Concept] --> B[Identify Pathognomonic Features];
--> C[Layer Multidisciplinary Clues];
D[clinical History<br><<>> chronicity, </mptoms</1>];
C --> E[Imaging/Lat Findings<br>>i>>. <br><i>e.g...... specific radiology signs</ </
i>];
--> hatboloooo/cyyoooogytt---->s cell type, differentiation
i>];
G[Require Synthesis Clues];
--> 6;
--> G;
--> H[Demand Etiologic Deductiiontbr><iiee... primary site,
mechanism</i>l,
11
H --> I[Incorporate Strategic Distractors common
misdiagnoses<</ii]
--> J[Ensure Clues Refute Distractors atypia level
rules out alternatives</1>
13
--> K[Align Timeline with Disease Bio indolent vs
acute</i>];
(a) The Design Logic formatted as Mermaid source code.
Winth Nuppprent Tatue
Oct
cd. drontlyy sveptre
cee' valiing
Okn
Enlution receeinn
Inseuse frainge nuegpoo
K.S. aono
Aig Timel he with
Kd. lndler aoor
(b) The flowchart generated from the Mermaid code.
Figure 21: An example of the Design Logic for Clinical Medicine problem, showing the Mermaid source code (a) and the corresponding visual flowchart (b).
30

---

Published as a conference paper at ICLR 2026
graph TD
A[Define Problem Type] --> B[Select Core Concept: <br>e.g..
Exponential Diophantine, Polynomial, Inequality] C[Choose Parameters with Constraints]
C --> Ci[Base/Numbers: <br> Coprime or low common factors<br>>
Strategic modular properties<br>> Growth disparities]
C --> c2[variables: <br> Multiple exponents/termsstb>>>
Interdependent constraints]
--> c3[Ensure solution space control: <br> Small solution exists< br> Bounded by properties/growth)
A --> D[Incorporate Reasoning Layers]
D --> Di[Step 1: Derive necessary conditions<br> e.g., modular arithmetic. symmetry, calculus]
D --> D2[Step 2: Impose non-trivial constraints <br> e.g., parity, divisibility, inequalities]
D --> D3[Step 3: Force case analysis<br> e.g., small-vale testing, critical point checks]
11
E[Finalize for Challenge]
12
E --> Ei[Verify: Solution requires all steps<br> No brute-force accessibility<br>> Key insight is necessary]
E --> E2[Refine: Remove redundancies<bb> Ensure clarity and conciseness]
E --> E3 [e[ Test with variants<bra e e.g., change bases/ 14
operators]
(a) The Design Logic formatted as Mermaid source code.
Iete Pobbe to
Cor leqallyy
taeey
Fuale Duling
Mei ros sas tark
Aqu the right sourery
Led
molee theel.
drakitt,
rg-dap
Exar fondenr
aned proeeergoun
George
novemn unaetry
(b) The flowchart generated from Mermaid code.
Figure 22: An example of the Design Logic for Mathematics problem, showing the Mermaid source code (a) and the corresponding visual flowchart (b).
31

---

Published as a conference paper at ICLR 2026
graph TD
A[Start: Define Core Task] --> B[Select Key Items for Comparison];
--> C[Set Task:
Rank Compare Items]; D[Require Detailed Explanation of Each Item];
--> E[Include Opposing Views/Controversies]].
C --> F[Discuss Broader Impacts e.g., Historical, Cultural,
Political];
--> G[Specify Factors to Consider e.g., Rights, Institutions,
Ethics];
C --> H[Link to Modern impllacttioons/ootttvvrrsssll
C --> I[Demand Use of Examples and Evidence];
J[Support Arguments with Facts and Datal;
11
K[Ensure Complex Logical Reasoning];
12
--> L[End: Assess Higher- -rdde Thinking];
(a) The Design Logic formatted as Mermaid source code.
Select Nay Item for Compar
Compare Items
Discuss Broader eg Histrrcll, Cultural, Pottrall
Explarationn
Implication/ Controversies
Spectly Factore Coneider
Include
Arguments
Ensure Complex Log cal Reesoning
End: assess Higher-Order Thinking
(b) The flowchart generated from the Mermaid code.
Figure 23: An example of the Design Logic for Law problem, showing the Mermaid source code (a) and the corresponding visual flowchart (b).
32

---

Published as a conference paper at ICLR 2026
graph TD
A[Primary Goal] --> B[Explore Complex Human Behavior]
--> C[Ethical Decision-Making Under Constraints]
--> D[Ensure Pedagogical Safety]
--> E[Design Imperatives]
--> [Require Controversial Perspective Coverage]
-> ER[Explicitly Forbid Harm Advocacy]
E3[Embed Iterative Refinement Mechanism]
11
--> FLPsychological Safeguards]
--> F1[Clinical Role <br>>e
Becchologist/Resarccee
--> -> F2[Tone Mandate <br>e.
candate<br>e.2 g., Caring Suinng//ppporivee Language] F3[Scenario Distancing <br>e.g., Hypothetical Survival Context
]
15
G[Structural Elements] --> G1 [High Stakes Scenario]
G1 Gee[Rel True Behavioral Drivers] G2[Concrete Decision Task<brze.g. Resource Prioritization]
18
19
G2 --> G2a[Create Measurable Choice Points]
H[ Qualltty Control] --> H1 [Anti Repetition Protocol] H2[Nuance Requirement <brze.g.. Multiple Revision Layers]
22
23
24
--> 6
--> H
(a) The Design Logic formatted as Mermaid source code.
Feeligh Sripek
Nou Upe
Nea
Ieriie Arean
(b) The flowchart generated from the Mermaid code.
Figure 24: An example of the Design Logic for Psychology problem, showing the Mermaid source code (a) and the corresponding visual flowchart (b).
33

---

Published as a conference paper at ICLR 2026
flowchart TD
A[Core Objectivel --> Re"Require Synthesis of Multiple Skills"]
De"Demand Critical Interpretation")
A --> D[" Connettt Abstract Concepts to Concrete Evidence
--> B1 /Vistuual/Vissal Analysis"]
B2 [" Contextual Analysis"] B3 Int'ddetrisc Integration!n(e.s.. anthropology, semiotics
history)
C1["Evaluate Symbolic Nuances"]
11 C2[*Assess Cultural significance"]
12 C Ca["Chaleenge Surface- Readings"]
13
--> D1["Link Physical Artifacts to Belief Systems
D2[aerace Practices Worldviews"]
16
D3 "e("Rccontrrct Civilizational Logic"]
(a) The Design Logic formatted as Mermaid source ende.
Saber Loaeltt
(b) The flowchart generated from the Mermaid code.
Figure 25: An example of the Design Logic for an Archaeology problem, showing the Mermaid source code (a) and the corresponding visual flowchart (b).
34


Reviews:
[{"review_id": "eP8uLWqbuz", "reviewer": "Reviewer_TpGs", "summary": "The paper proposes Designer, a data-synthesis pipeline that extracts reusable design logics (structured problem-solving flow charts) from a large question bank. Using filtered snippets form existing text corpora, the system coarse-retrieves several candidate logics, then has an LLM select one to use, and synthesizing snippet and logic to form a new question plus a short reference answer. A separate thinking model produces long chain-of-thought (CoT) responses. To test this pipeline, popular base models are SFT'd on these synthetic examples. They find that Designer SFT outperforms the instruciton-tuned variants of the base models. Several ablations are performed, including analysis of the corpora quality and the key components of the design logic process. The produced training data is also compared with identically-sized subsets of other synthetic datasets, finding the DLR-trained models consistently outperform the other 4 baselines.", "questions": "- **Significance of question bank**: Can you provide any results using only public / reproducible data? A key distinction between your work and others is the difficulty of the generated questions (Figure 3). Is it possible your data is simply more challenging due to this question bank? Also, if difficulty is a large factor, could one simply just upsample hard questions in other dataset to match the distribution of yours?\n- **Additional SFT baselines**: Can you provide the raw base model results, and also Instruction-tuned base model + DLR SFT?\n- **Analysis of synthetic generated responses**: Does the long-CoT Qwen3-235B responses ever disagree with the DeepSeek-R1 answer? If so, do you filter these examples out? Also, have you explored what happens if you include the corresponding design logics as problem-solving guidance (e.g. \"for these types of problems some things to think about are A -> B -> C\")\n- **Contribution of design logics for question generation** -- again, the ablation in Table 5 notes that the gains from design logics are fairly minimal; it appears that the retrieved examples alone (w/o Design Logic) perform well, so could this mean that design logics are perhaps just useful as a structured and abstracted method for indexing a large question bank?\n\nSmall writing suggestion: The caption in Figure 6 ('source corpus quality') is ambiguous until you look at Section 5.5 (which discusses the acknowledged differences).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Design logic direction seems like a novel and scalable method for condensing multi-domain learnings in a reusable manner.\n- Resultant datasets yields solid improvements over other synthetic pre-training data methods.\n- Paper is well written and easy to follow.", "weaknesses": "My biggest concerns are with the use of a proprietary question bank, and also general concerns with the impact of the design logics, versus just having relevant and high-quality data:\n- For the design logic process, I like the idea of having a static design logics bank, however, it seems the design logics themselves don't have a massive impact on performance; the 'w/o Design Logic' ablations note fairly small gains from using design logics (as opposed to just providing examples from the question bank). I'd love to see more extensive analysis into the design logic creation as this is the most novel component of this work.\n- Given the strong dependency on the question bank, I am concerned about its proprietary nature. What happens if we don't have a carefully-crafted question bank? Is the Designer process is only pertinent if we have a challenging question bank available? If this process is largely reproducible with public data, then it would be ideal to publish these results. \n- No comparison with other simple SFT baselines. For example, what are the base model results (no SFT), and what if you perform SFT on the instruction-tuned variants of the baseline models? It seems these results would paint a clearer picture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vQs1HnsyzC", "reviewer": "Reviewer_VsnW", "summary": "This paper identifies the primary bottleneck of improving the complex, multi-step reasoning capabilities of Large Language Models (LLMs) as a scarcity of large-scale, high-quality, and diverse training data, especially for specialized, university-level subjects. To solve this, the paper introduces **DESIGNER**, a novel data synthesis pipeline. The core innovation is the concept of \"Design Logic\", which the authors define as a form of \"meta-knowledge\" that abstracts the structured process human educators use to construct complex and insightful questions.\n\nThe paper's primary contributions are:\n- The **DESIGNER** pipeline, centered on the novel \"Design Logic\" concept.\n- The two resulting multi-million-question datasets, DLR-Book and DLR-Web, whose analysis shows significantly more difficult and semantically diverse than existing baseline datasets.\n- Extensive validation through Supervised Fine-Tuning (SFT) on Llama3 and Qwen3 model families. The key result is that base models trained on the proposed datasets substantially outperform existing datasets.", "questions": "This paper is quite solid, detailed, and well-written, and I only have a few questions:\n1. In Figure 4, we can see that DLR-Book contains more than half of the data categorized as \"Other\" (and more than 1/3 on DLR-Web). Is this beneficial? If so, how do these benefit the model? If not, can we drop them?\n1. Can the **DESIGNER** pipeline apply to open-source raw data or existing datasets?\n1. How many GPU hours have been spent on creating the DLR datasets?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The proposed **DESIGNER** data synthesis pipeline is novel and well-motivated. The structured process human educators use to construct complex and insightful questions.\n1. Using this pipeline, the authors created two new, large-scale reasoning datasets: **DLR-Book** (3.04 million questions) and **DLR-Web** (1.66 million questions), which benefit the community to improve existing models.\n1. Detailed analyses prove with quantitative metrics that the DLR datasets are more difficult and more semantically diverse, validating the effectiveness of the proposed **DESIGNER** pipeline.\n1. Applying SFT on the DLR datasets, existing models outperform their official versions and gain performance improvement across all reasoning benchmarks.\n1. Throughout ablation studies justify the effectiveness of the **DESIGNER** pipeline", "weaknesses": "1. The pipeline's success relies on massive proprietary assets. Though the authors state they will release a subset of the final synthesized data, this still limits the community's ability to fully reproduce the pipeline or build upon the design logic library.\n1. The pipeline itself depends on existing very large, capable models (e.g., Qwen3-30B, DeepSeek-R1-0528), and this means massive computational resources are required to apply this pipeline.\n1. The pipeline only creates large-scale datasets, and these datasets are only validated on very large models, missing validation on small corpora and small models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cTfDP72B84", "reviewer": "Reviewer_mg22", "summary": "This paper introduces DESIGNER, a novel pipeline for synthesizing large-scale, multidisciplinary reasoning data by leveraging \"design logic\"—a structured abstraction of human educators' question-creation process. The method reverse-engineers design logics from existing questions. It matches them with raw text corpora (books and web) to generate challenging questions across 75 disciplines, resulting in two datasets: DLR-Book (3.04 million questions) and DLR-Web (1.66 million questions). Experiments demonstrate that models fine-tuned on these datasets achieve significant improvements in multidisciplinary reasoning, even surpassing official instruction-tuned counterparts. The work addresses data scarcity in complex reasoning domains but raises questions about scalability and quality control.", "questions": "1) Will the synthesized datasets (DLR-Book and DLR-Web) be openly released to facilitate reproducibility and broader research? \n2) In Line 400 (Section 5.5), the claim that \"book corpora are of higher quality than web corpora\" is presented as a widely acknowledged consensus but lacks references. \n3) There appears to be a discrepancy in MMLU-Pro scores: Table 4 reports 68.4 for DLR-Book, while Table 5 reports 68.1 for the full DESIGNER method. Is this a typo? Additionally, Table 3 shows that DLR-Web+Book yields the best performance, yet Tables 4 and 5 use DLR-Book or ablated versions for comparisons. Justifying why the optimal dataset wasn't used in later analyses would improve consistency.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The step-by-step pipeline is highly systematic, enabling large-scale query data generation with strong operability and reproducibility. The clear phases—from data curation and design logic extraction to question synthesis—provide a structured framework that can be easily adapted or extended by other researchers. This practicality enhances the method's value for real-world applications.", "weaknesses": "1) The work resembles a complex engineering effort with multiple sub-tasks (e.g., discipline labeling, difficulty classification) pieced together, which may lack the novelty.\n\n2) The pipeline relies heavily on prompt engineering (PE) at multiple stages (e.g., discipline classification using Figure 7, difficulty classification using Figure 8), but lacks rigorous quality assessments for each step. For instance, the discipline labels and difficulty scores are derived from LLM judgments without validation against ground-truth metrics or inter-annotator agreement. The difficulty classification, in particular, depends on only three examples in Figure 8, which may not capture nuanced criteria for \"Very Hard\" questions. Consequently, the claim in Section 4.1—that the synthesized data is more difficult based on LLM-labeled distributions—is less convincing without evidence of labeling reliability.\n\n3) The focus on difficulty and diversity (Sections 4.1–4.2) overlooks critical aspects of data usability and accuracy, such as whether questions have unique answers or sufficient conditions to derive solutions. No analysis is provided on logical consistency or answer correctness, which are fundamental for reasoning data quality. This gap undermines the practical utility of the synthesized datasets.\n\nOverall, the paper presents a scalable data synthesis pipeline with practical benefits, but the engineering-heavy approach, unvalidated quality controls, and overlooked usability issues limit its conceptual impact. Addressing these concerns could elevate the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LmJlubd6zk", "reviewer": "Reviewer_VTR2", "summary": "This work presents a data synthesis approach aimed at alleviating the data scarcity problem in cross-disciplinary reasoning tasks. The authors construct two large-scale reasoning datasets: DLR-Book and DLR-Web, covering 75 scientific disciplines. The core contribution of this work lies in its logic-guided data design, which enhances data quality through difficulty and diversity control. Experimental results show that the proposed datasets outperform existing ones on multiple benchmarks.", "questions": "See Weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of the work is clear and easy to understand.\n2. The main contribution lies in building a large-scale multidisciplinary dataset, which achieves competitive or even superior results across multiple benchmarks (e.g., Table 1).\n3. The carefully designed synthesis pipeline demonstrates thoughtful engineering and could provide useful insights for other researchers working on data generation and reasoning dataset construction.", "weaknesses": "1. The current approach is essentially a heuristic replacement for existing data synthesis methods, mainly relying on hand-crafted design and rejection sampling to obtain high-quality data.\n2. The complexity of the data construction process might lead to error accumulation and make it difficult to control noise in the final dataset.\n3. Some experimental results require further clarification:\n+ In Figure 5, why does the model show better scalability only on GPQA-Diamond, while other datasets do not exhibit similar improvements?\n+ In Table 5, the ablation study shows that removing Design Logic (w/o Design Logic) results in only minor performance drops. Other ablation variants also show limited improvements?\n4. The data efficiency of the synthesis process is unclear. For example, during data filtering, what proportion of data was discarded? Understanding this would help assess the trade-off between data quality and synthesis cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V4FeYZSbGt", "reviewer": "Reviewer_wEhN", "summary": "Large language models have achieved strong performance on many natural language processing tasks, but they struggle with complex reasoning tasks. Many existing studies use synthetic data to train large language models to improve their reasoning ability. However, these studies often lack disciplinary guidance. \n\nTo address this issue, this paper uses some unlabeled documents to generate synthetic data, including books and web data. Specifically, the process is divided into three parts: Data Extraction and Preprocessing, Code Synthesis, and Filtering and Output. The authors conduct an in-depth analysis of the synthetic data, examining data difficulty, diversity, and disciplinary distribution. \n\nFinally, they train the model using the synthetic data and find that it leads to better training results.", "questions": "+ The paper tests multiple-choice tasks. Does it also perform well on open-ended questions, such as GSM8k or MATH500?\n\n+ Why does combining web data and book data lead to better results? What are the differences and characteristics of these two types of data when merged?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This paper proposes a new method for data synthesis that can generate useful data for the model from unlabeled documents.\n\n+ This paper conducts an in-depth analysis of the data, including data difficulty, data diversity, and disciplinary distribution, helping readers better understand the synthetic data and providing more information for future research.\n\n+ The synthetic data in this paper can effectively improve the model's reasoning ability and proves to be effective across tests in multiple disciplines.", "weaknesses": "+ Previous studies have also generated training data from unlabeled data (such as DeepSeek-Math, JiuZhang3.0, and MAmmoTH 2.0). Even without using complex logic extraction and logic retrieval processes, their results are more significant than existing methods.\n\n+ The process of generating synthetic data uses very large models, which incurs high costs. Does this indicate that the method has limitations in practical applications?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)