Learning from Synthetic Data Improves Multi-hop Reasoning
Authors: Anmol Kabra
Yilun Yin
Albert Gong
Kamilė Stankevičiūtė
Dongyoung Go
Johann Lee
Katie Z Luo
Carla P Gomes
Kilian Q Weinberger
Venue: ICLR 2026 Poster (2026)
Keywords: multi-hop reasoning
large language models
reinforcement learning
synthetic data
Primary area: foundation or frontier models, including LLMs
TL;DR: RL fine-tuning LLMs on synthetic data improves real-world multi-hop reasoning by teaching knowledge composition skills

Abstract: Reinforcement Learning (RL) has been shown to significantly boost reasoning capabilities of large language models (LLMs) in math, coding, and multi-hop reasoning tasks.
However, RL fine-tuning requires abundant high-quality verifiable data, often sourced from human annotations, generated from frontier LLMs, or scored by LLM-based verifiers.
All three have considerable limitations: human-annotated datasets are small and expensive to curate, LLM-generated data is hallucination-prone and costly, and LLM-based verifiers are inaccurate and slow.
In this work, we investigate a cheaper alternative: RL fine-tuning on _rule-generated synthetic data_ for multi-hop reasoning tasks.
We discover that LLMs fine-tuned on synthetic data perform significantly better on popular real-world question-answering benchmarks, despite the synthetic data containing only fictional knowledge.
On stratifying performance by question difficulty, we find that synthetic data teaches LLMs to _compose knowledge_---a fundamental and generalizable reasoning skill.
Our work highlights rule-generated synthetic reasoning data as a free and scalable resource to improve LLM reasoning capabilities.

Published as a conference paper at ICLR 2026
LEARNING FROM SYNTHETIC DATA IMPROVES MULTI-HOP REASONING
Anmol Kabra'; Yilun Yin', Albert Gong', Kamilé Stankevičiüte 1.2 Dongyoung Go', Johann Lee', Katie Z. Luo Carla P. Gomes', Kilian Q. Weinberger' CCornlll University University of Cambridge SStanford University
ABSTRACT
Reinforcement Learning (RI.) has been shown to significantly boost reason- ing capabilities of large language models (LLMs) in math, coding, and multi- hop reasoning tasks. However, RL fine-tuning requires abundant high-quality verifiable data, often sourced from human annotations, generated from frontier LLMs, or scored by -Lsee verifiers. All three have considerable limitations: human-annotated datasets are small and expensive to curate, LLM-generated data is hallucination-pree prone and costly, and LLM-based verifiers are inaccurate and slow. In this work, we investigate a cheaper alternative: RL fine-tuning on rule- generated synthetic data for multi-hop reasoning tasks. We discover that LLMs fine-tuned on synthetic data perform sigmificantly better on popular real-world question-anssering benchmarks, despite the synthetic data containing only fie- tional knowledge. On stratifying performance by question difficulty, we find that synthetic data teaches LLMs to compose knowledge-a fundamental and gener- alizable reasoning skill. Our work highlights rule-generated synthetic reasoning data free and scalable resource to improve LLM reasoning capabilities.
INTRODUCTION
Reinforcement learning (RL) has emerged as a powerful approach for enhancing reasoning capa- bilities of large language models (LLMs), with notable successes in math, coding, and logical rea- soning (Bai et al., 2022; Lambert et al., 2025; Guo et al., 2025a; Guan et al., 2025). Central to this progress the paradigm of RL with verifiable rewards (RLVR; Guo al., 2025a; Lambert al., 2025), where models' candidate answers are scored against known ground-truth answers or graded with verification tools. As consequence, RLVR has been immensely effective in domains with abundant verifiable data or access to rule- or -LM-d verifiers. Yet RLVR faces fundamental data bottleneck: high-quality training data with verifiable answers is scarce and expensive to curate, and I.M verifiers are slow and inaccurate (Lambert et al., 2025; Gong et al., 2025). As LLM train- ing outpaces the availability of high-quality numan-written text (Villalobos al., 2024; Muennighoff et al., 2023), the community has responded with synthetic data (Liu et al., 2024)-2240-- -etter distilled from stronger models (Abdin et al., 2025) or generated as augmented reasoning traces (Trinh et al., 2024; Ruan al., 2025). Both approaches, however, still depend LLMs and inherit their verifi- cation challenges, high inference cost, and potential for contamination with pretraining corpora (Su al., 2025).
We pursue a more radical alternative: rule-generated synthetic data: questions generated from templates, context-free grammars, and logic programs on entirely fictional entities. Unlike LLM- generated synthetic data, rule-generated synthetic data is semantically simple by design, yet fully verifiable by construction. It can be produced at arbitrary scale, on any standard computer, for free.
The gap between such synthetic data and real-world counterparts is stark. A synthetic question from PhantomWike Wiki (Gong et al., 2025) asks "Who is the nephew of the friend of the person whose hobby is birdwatching?" , while question from HotpotQA (Yang et al., 2018) demands "Aside from Yodobashi, what other towns were merged into the ward which gave the major Japanese electron- ics retail chain its name?". The former contains no real facts and uses templates, while the latter
"Corresonndenc:: anmol@cs.cormell..
cornell edu. Code: github. www. http//iiiannggee www.llaawggguopppatttmeeeeaaaaaa

---

Published as a conference paper at ICLR 2026
RULE- GELEATEE Synthetic Data
LEARNED SKILL
REAL-WORLD Multi-Hop Reasoning
Knowledge Composition
Retional Templated Zero Inowledge
Chaining information ocross multiple steps
Knowledgg--ineensee - Linguisticolly rich
triend
HotpotOA
Musique
Synchworlds
PhantomWiki
Reasoning@yy
teaches
lites birdwstching
transfers
2W/4
CofCA
Improved LLM Reasoning
EXAMPLE Who is the nephew of the friend of the
LLMs learn compose knowleddge from symthetic data alone
2014
Wedtowe
person whose hobby burevetchinn?
Fully verifiable by construction
Generaliaes difficulties
Free, infinite scale, on any machine
Improves intermediate reasoning steps
Precise difficulty control
RL Fine-tuning
Scales with synthetic data
fails teach this
6.2
Figure 1: Synthetic-to-real transfer by learning knowledge composition. We RL fine-tune LLMs on rule-generated synthetic datasets (left). Despite containing no real-world knowledge, training on these datasets teaches models to compose knowledge (center). This fundamental skill of chaining information across multiple steps transfers to real-world multi-hop reasoning benchmarks (right).
requires navigating real-world knowledge and rich linguistic structure. It is not obvious whether rule-generated synthetic data can teach skills useful for the real-world (Stojanovski et al., 2025; Liu et al., 2024). Hence, we investigate the fundamental research question: can models develop gen- eral reasoning capabilities solely from synthetic data, without relying on real-world knowledge or transfer would have transformative im-
linguistic complexity? Evidence for such a synthetic- plications: unlocking rule-generated synthetic data as a viable resource for scaling LLM reasoning.
We present strong evidence for synthetic- transfer in multi-hop reasoning. Among various reasoning capabilities, we focus on knowledge composition: the elementary skill of integrating in- formation across multiple steps. Through extensive RL fine-tuning experiments with rule- suleggenerated synthetic datasets (Gong al., 2025; Zhou et al., 2025; Stojanovski al., 2025), we demonstrate consistent transfer of molti-hop reasoning capabilities to
benchmarks (Yang et al., 201 88; Ho al., 2020; Trivedi et al., 2022: Wu al., 2025a: Gu et al., 2025). The transfer is remarkable: Qwen3-0.6B model RL fine-tuned on Phantom Wiki (Gong et al., 2025) achieves relative F1 score improvements of 56% to 131% (Figure 2). This despite the synthetic training data containing no factual knowledge useful to the real-world benchmarks. Performance transfer trends are consistent across synthetic data generators, real-world multi-hop benchmarks, and model families and sizes.
We further investigate why this transfer works. Recent work suggests that RL fine-tuning could be reinforeing and surfacing memorized pretraining knowledge (Shao et al., 2025; Yue et al., 2025). However, the fictional worlds in sule-generated synthetic data share zero knowledge overlap with reality, and models cannot improve by composing pretraining knowledge. Instead, they must learn to compose knowledge worlds and out-of- -ffommnn- (OOD) question complexities never seen during fine-tuning (Figure 3). Models extrapolate beyond training knowledge and question complexity levels, suggesting that they bootstrap this skill and generalize, rather than merely memorizing solution patterns. We find that Supervised mne-tuning (SFT) on rule-generated synthetic solution traces improves synthetic task performance but fails to generalize to the real-world (Table 2): it overfits to the training task rather than teaching a fundamental reasoning skill.
domain-indeependen skill. We confirm this by testing on held-out fictional
We also observe that this skill concretely manifests in model generations, illustrating how the trans- fer works. As RL fine-tuning progresses on synthetic data, models generate reasoning traces contain- ing higher instances of correct intermediate answers (Figure 4). This points to emergent grounded generations in the real-world arising from outcome-only supervision with synthetic data. Finally, performance on real-wont world benchmarks scales monotonically with the number of synthetic training samples, with no sign of overfitting (Figure 5). This posits rule-genee generated synthetic data as a viable resource for scaling LLM reasoning capabilities with RL (Liu et al., 2024).
Taken together, our work spotlights rule- aale-generated synthetic data as a free and scalable resource for teaching I.LMs knowledge composition fundamental skill for real- world multi-hop reasoning.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
Reasoning in LLMs. While LLM reasoning is long-standing research area, the definition and assessment of reasoning capabilities is ambiguous and complex (Xie et al., 2024; Han et al., 2025). Thinking and reasoning models like DeepSeekMath (Shao et al., 2024), DeepSeek-RI (Guo et al., 2025a), or Phi-4-mini reaaooonnnn (Abdin et al., 2025) are typically evaluated in their reasoning skills through performance on various benchmarks. Current benchmarks span technical domains such as mathematics, coding, and puzzle-solving (Hendrycks et al., 2021; MAA; Cobbe et al., 2021; Jain et al., 2025; Chollet al., 2025), snowledge-intensing 2024; Sawada et al., 2023), commonsense, abductive, and counterfactual reasoning (Talmor et al., 2019; Zhao al., 2023; Bhagavatula al., 2020; Wu et al., 2025b; Hüyük al., 2025), multi-hop question-answe answering (Trivedi et al., 2022; Yang et al., 2018; Ho et al., 2020; Tang & Yang, 2024; Qi et al., 2021; Gu al., 2025). and environment-basse planning and tool use (Patil et al., 2024; Zhuang et al., 2023; Yao et al., 2024). Many of these benchmarks require decomposing the question into intermediate subproblems and composing their answers (Gong et al., 2025; Xie et al., 2025). This knowledge composition skill is considered to be an intrinsic property of effective reasoning models (Gandhi et al., 2025).
tasks in science and law (Rein et al.,
Training and fine-tuning LLMs. LLM performance reasoning benchmarks can be improved with training or fine-tuning using several classes of techniques. The simplest approach is to train on the datasets directly using supervised fine-tuning (SFT; Lambert et al., 2025) with next-token pre- diction objective. This includes extensions to add more helpful instructions or to encourage a more detailed thinking process, for instance through instruction fine-tuning (Chung al., 2024) and chain- of-thought (CoT) modeling (Xiang et al., 2025; Zelikman et al., 2022; Hao et al., 2025; Yao et al., 2023; Chen al., 2023; Wan al., 2025). Reinforcement learning from human feedback (RLHF; Christiano et al., 2017: Ouyang et al., 2022) offers a more powerful alternative, with algorithms including policy gradient- based PPO (Schulman et al., 2017), and variants such as GRPO (Shao et al., 2024) and DPO (Rafailov et al., 2023), among others (Hu et al., 2025; Pang et al., 2024; Brantley al., 2025; Yu al., 2025; Liu et al., 2025b; Shrivastava et al., 2025). When ground-truth answers are available, the reward model can be replaced by a fast verification function, an approach called Reinforcement learning with verifiable rewards (RLVR; Lambert et al., 2025). RLVR has been adopted by several recent reasoning models (Lambert et al., 2025; Guo et al., 2025a; Abdin et al., 2025), though its efficacy for eliciting novel reasoning in LLMs remain an open question (Wen et al., 2025; Yue et al., 2025; Shao et al., 2025; Zhao et al., 2025b).
Leveraging synthetic data. Real-world reasoning benchmarks conflate multiple skills (e.g.. multi- hop reasoning with arithmetic reasoning or coding), making it difficult to isolate specific LLM ca- pabilities (Wu et al., 2025c; Xie et al., 2024; Yu et al., 2024). Moreover, as LLMs are trained at internet scale, existing benchmarks become increasingly prone test-set leakage (Gong al., 2025; Wu et al., 2025c;b). Synthetic datasets address both issues: they can isolate targeted reason- ing skills while providing unlimited fresh examples with verifiable rewards (Allen-Zhu 2025; Zhao et al., 2025a; Liu et al., 2024; Yang et al., 2025; Sprague et al., 2025).
For training LLMs, recent work leverages LLM-generated synthetic data to augment human-curated data (Yang et al., 2025; Goldie et al., 2025; Huang et al., 2025; Saad-Falcon et al., 2024; Li et al., 2025). However, LLM-generated data is expensive, inherits verification challenges, and risks con- tamination with pretraining knowledge.
Rule-generated synthetic datasets gennrated using templates, context-free grammars, and logic programs-hav have conventionally been used to evaluate reasoning capabilities in math (Mirzadeh et al., 2025; Zhou et al., 2025; Wu et al., 2025c), logic puzzles (Xie et al., 2024; Shojaee et al., 2025; Stojanovski et al., 2025), and natural language anestion--nnweriig (Gong et al., 2025; Guo et al., 2025b; Sinha et al., 2019). It is fully verifiable by construction and free to produce at arbitrary scale. Yet whether training such semantically simple data can teach skills useful for knowledge- intensive real-world reasoning remains underexplored (Yu et al., 2024; Mizrahi et al., 2025; Abbe et al., 2024b;a; Stojanovski al., 2025). This is the question study in this work.
3

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
To study reasoning capabilities acquired from synthetic data, we RL fine-tune four LLMs: Qwen3-0.6B, Qwen3-1.7B, 1.ven.55-1.BB-Insttuu (Qwen Team, 2024; 2025), and Phi-4-mini- reasoning (Abdin et al., 2025). By covering range of model families and parameter sizes (0.6 to billion parameters), we control for differences in pre-training and post-training strategies. This helps isolate the effect of rule cele-generated synthetie data the training signal for RLVR.
3.1 SYNTHETIC TRAINING DATASETS
Recent works in RLVR highlight the need for large datasets with two important characteristics: scalable verification of model generations, and questions of varying difficulty (Guo et al., 2025a; Wen et al., 2025; Shao et al., 2025; Lambert et al., 2025; Abdin al., 2025). Scalable verification is essential for on-policy RL algorithms, because the reward function needs to be evaluated on on-the-fly. Further, mix of easy and hard questions is equally important: easy questions help the algorithm discover rewards early, while hard questions expand the frontier of reasoning capability. With these criteria mind, we focus on rule-generated synthetic data.
Rule-generated synthetic data. Rule-generated synthetic data relies strict templates, context- free grammars, and logic programs. This makes it semantically simple yet fully verifiable by con- struction, unlike LLM-gencrated data that has rich linguistic style but lacks verifiability. Because rule-generated synthetic data requires no GPU or API access, it can be produced at arbitrary scale, for free,, and any standard computer. This is valuable for designing LLM training pipelines under resource constraints. Generator programs also enable precise control over question complexities, which is crucial for RLVR and fine-grained reasoning analysis that we conduct in Section 4.
We use four cule-generated synthetic datasets that together cover diverse reasoning styles: Phan- tom Wiki (synthetic multi-hop, Gong al., 2025), GSM- (math, Zhou et al., 2025) (math), Rea- soningGym's
and Knights-Knaves (logical, Stojanovski et al., 2025). All fictional worlds and contain no factual overlap with datasets use templates on benchmarks. This allows us to study how reasoning styles of differ- snewledge-intessing multi-hop reasoning, without any influence from shared
real-world multi-hop reasoning
ent kinds transfer factual knowledge. We discuss their individual features and generation settings below.
PhantomWik (Gong et al., 2025) is a framework for generating datasets of natural language doc- ument corpora and question-answer pairs. Each Phantom dataset represents random universe of fictional people. Their personal attributes and inter- aeerrpersonal relations are described in Wikipedia- like documents. Phantom Wiki uses a context-free free grammar and logic programming-base to generate multi-hop reasoning questions with verifiable answers. Questions in Phantom Wiki may have multiple answers unlike the other three datasets; they also require greater retrieval and knowl- edge composition skills. For example, answering "Who is the nephew of the friend of the person who likes birdwatching? ?" requires identifying all people who like birdwatching, and nephews of each of their friends. With Phantom Wiki, we investigate the importance of training on synthetic data that align with the target real- world task, in case: multi-hop question-answering
algorithm
We configure PhantomWit Wiki datasets to only contain immediate family and friend relations, so that the "hops" are conceptually simple. We further filter out aggregation questions of the form "How many.... ...". to constrain the datasets to purely multi-hop questions like "Who is the <relation> of...?" and "What the <attribute> of ...?". This setup ensures that answering a question of difficulty d requires hopping through exactly d documents, and eliminates the confounding counting skill. To generate questions with varying difficulties, we generate 34 random universes each with 25 indi- viduals, and set the context-free grammar recursion depth to 20. This process yields 330 questions per universe with question difficulties ranging from 1 to 9. We select 31 universes containing 10K samples for training, and reserve 3 universes of s IK samples for testing.
GSM-Infintte (GSM-oo; Zhou et al., 2025) generalizes the GSM8K benchmark-a collection of grade school math word problems (Cobbe et al., 2021)-to infinitely many questions. GSM-co builds a random computation graph on demand to represent the ground-truth solution trace. It then converts the graph to word problem via natural language templates, which mimic common themes in GSM8K. With GSM-oo, we study transfer from arithmetic math-based reasoning to knowledge- intensive multi-hop reasoning.
4

---

Published as a conference paper at ICLR 2026
We generate math word problems with 2-20 arithmetic operations (binary, no distractor facts), yield- ing =600 questions per difficulty level. Of the =12.5K total problems, we use a random subset of 10K samples for training and reserve the rest for testing.
ReasoningGym (Stojanovski et al., 2025) is an open-source RLVR library spanning domains in- cluding algebra, logic, and combinatorial games. We generate from two environments that probe complementary forms of logical reasoning. RG-Family the relationship between two individuals in randomly generated family tree-the inverse of Phan- tomWiki, which infers individuals given a relationship. RG-Knights Knights-Knaven requires identifying truth-tellers from liars given a set of logical statements from randomly generated indi- viduals. Together, these two datasets let us study whether logical deductive reasoming can teach skills relevant to multi-hop reasoning. We generate 10K training samples from each environment: RG-Fammlly from family graphs of sizes uniformly sampled between 3 and 20, and RG-Knishts balanced evenly across configurations of 2-6 people.
requires inferring
3.2 RL FINE-TUNING FOR REASONING
We use the Group Relative Policy Optimization (GRPO; Shao et al., 2024) RL fine-tuning algo- rithm. GRPO has been introduced as a variant of proximal policy optimization (PPO; Schulman et al., 2017). Where the PPO algorithm estimates the advantage term in its objective using a value model, the GRPO algorithm uses a group of completions for each prompt. We use the GRPOTrainer implementation from the open-source Hugging Face TRL v0. 21. library (von Werra et al., 2020), which implements special case of GRPO: the advantage is calculated per batch on each GPU, and Ki-eeeeeee penalty hyperparameter 3 is set to O. See Section A.1 for further details.
3.3 PROMPT AND REWARD DESIGN
We fine-tune LLMs to perform in-context reasoning, i.e. to answer questions given all the rel- evant context in the prompt. The prompt first includes the evidence: for GSM-x a RG- Family or a RG-Knights question, this is the problem statement; for PhantomWiki question, the evidence is the set of all 25 articles in the randomly generated PhantomWiki universe. After the evidence, our prompt includes an instruction for the LLM to output the final answer within <answer>.. ... </answer>> tags, which have been popularly used by DeepSeek-R1 (Guo et al., 2025a) and the Qwen3 family (Qwen Team, 2025). To further ground the answer output format, we append chain-of-thought (CoT) examples. For GSM-oo, we use automatically-ggneet truth CoT from the training set; for Phantom Wiki we use the 11 CoT examples curated by Gong al. (2025); for RG-Family and RG-Knights we write 11 CoT examples ourselves. Finally, we pose the question to the LLM (our full prompts are included in Appendix C).
ground-
We extract the model's prediction from the final <answer>.. </ answer> tags, and compare it with ground-truth. For GSM- RG-Family, and RG-Knights questions, we assign binary reward based on exact match. PhantomWikt questions can have multiple answers, so we assign rewards based on the F1 score of predictions. We randomly shuffle the 10K training samples in each dataset.
3.4 EVALUATION BENCHMARKS
We evaluate on 5 real-world multi-hop reasoning datasets. For all these datasets we use the distractor versions, where the supporting information includes non-relevant (distracting) paragraphs. This evaluation setup measures LLMs' in-context reasoning as all questions can be answered given the context. We randomly subsample 500 question-ansee ansserss from the respective test sets for evaluation.
1. HotpotQA (Yang et al., 2018) is a multi-hop question answering dataset containing over 100K questions. Each question follows a consistent 2-hop reasoning structure and requires combining information typically from 2 Wikipedia paragraphs.
2. 2WikiMultihopQA (Ho et al., 2020) is a recent 2-hop dataset, containing over 190K ques- tions organized into four categories: compositional, inference, comparison, and bridge- comparison. Like HotpotQA, the questions are grounded in Wikidata's 's knowledge graph, and follow specific 2-hop path between related entities.
3. MuSiQue (Trivedi et al., 2022) evaluates compositional reasoning with 2-4 hop questions created by bridging single-hop questions, each using information from supporting para-
5

---

Published as a conference paper at ICLR 2026
graph. We use the MuSiQue-Answerabl split of the dataset ensure that all questions can be answered using a subset of the given context.
4. CounterfactualQA (CofCA) (Wu et al., 2025a) is rewritten subset of 2-4 hop ques- tions from HotpotQA, 2WikiMultihopQA and MuSiQue. Wu et al. (2025h) manually rewrite questions remove factual knowledge from the Wikipedia-based datasets, which LLMs could have memorized from to shortcut multi-hop reasoning.
5. SynthWorlde (Gu al., 2025) highee-complexity dataset of 2-6 hops and con- straints curated from the Wikidata knowledge graph using graph motifs.
4 RESULTS
Performance fransfer from synthetic to real-world datasets. Rule-generated synthetic datasets use strict yet simple templates on fictional worlds (Section D) and differ drastically in grammar, se- manties, and linguistic complexity from knowledge-intensivy multi-hop reasoning in the real-world (Section 3.4). Synthetic-to-rea transfer is therefore not obvious (Stojanovski et al., 2025). Yet, as Figure shows, RL fine-tuning rule-generated synthetic data remarkably improves performance on all rerl-world evaluation benchmarks. This transfer is consistent across model families and sizes, with the smallest models benefiting the most. Notably, Qwen3 (Qwen Team, 2025, Sec. 4) and thi-4-mini-reasoniig (Abdin et al., 2025, Sec. 3,4) already underwent extensive training on syn- thetic reasoning data during post-training, yet they still improve with continued RL tine-tuning on mue-gennrated synthetic data. In other words, learning to compose knowledge in fictional worlds devoid of linguistic diversity transfers realistic multi-hop tasks rich in semantic complexity. We note that RL fine-tuning on in-domain real-world data still outperforms synthetic-to-real transfer (Figure 7), though such data is expensive to curate and limited in scale.
RL funi-tuning augments performance beyond answer formatting. Improved performance from RL fine-tuning could partly stem from learning to write answers the right format, rather than from improved underlying reasoning (Shao et al., 2025). We disentangle these two effects with an ablation study. We RL fine-tune all small models for 3K steps with binary reward for using <answer>... ... </answer>. Table I shows that this format-only training does not help Qwen3 or Phi-4-mini-reasooning but substantially improves Qwen2.5- 1.55----tttt-ttstructc
There are two takeaways from this ablation. First, RL fine-tuning can teach answer formatting. Qwen2.5-1.5B-Insttee learns "reward hacking" with format-only reward. Therefore, the Qwen2.5- I.St-Instruct's synthetic-to-real transfer in Figure is due both learning output formatting and answering correctly. Second, and more importantly, RL fine-tuning teaches knowledge composi- tion. Qwen3 and 1ii----miiirrcasonni format-only training does not improve performance. Therefore, all of their synthetic-tor to-real transfer in Figure 2 must come from learning to compose knowledge. This confirms that LLMs can develop knowledge composition from synthetic data alone.
already produce correct output formatting at initialization, so
HotpotQA
2WikiMultihopQA
MuSiQue
base
0.36 + 0.02
0.37 = 0.02
0.01
Qwen3-0.6B
format
0.38 + 0.02
0.34 0.02
0.13 0.01
base
0.59 + 0.02
0.64 + 0.02
0.34 + 0.02
Qwen3-1.7B
format
0.64 + 0.02
0.67 0.02
0.35 0.02
base
0.48 + 0.02
0.66 0.02
0.27 + 0.02
Phi-4-mini--easoninng
format
0.47 + 0.02
0.48 0.02
0.26 + 0.02
base
0.02 + 0.01
0.14 0.02
0.04 + 0.01
Owen2.5-1.5B-IIstmuu
format
0.43 + 0.02
0.30 - 0.02
0.20 + 0.02
Table I: Ablation study on fine-tuning with binary format reward. Fl scores of Qwen3 and Phi-4-mini-reasoning LLMs do not improve when RL fine-tuned with binary reward for using <answer>... </answer>. Swen2.5-115B-Insstt improves remarkably with binary format reward. We report standard error over the evaluation datasets.

---

Published as a conference paper at ICLR 2026
base HotpotQA
RG RG-Family
RG-Knights
GSM-
Phantonwikk
2wiki
MuSiQue
CofCA
SynthWorlds-RR
1.0
0.4-
8.2
0.8
Qwen3-0.co
1.0
8.8
0.6
Qwen3-1.78
1.0
8.8
0.6
8.4
0.2
Thi-4-Mini Lini-Reasoning
8.8
0.66
0.4
0.2
0.0
Qwen2.55 mwen2.5-1.0 11..22511.BBBB11
Figure 2: F1 scores on real-world multi-hop reasoning datasets of LLMS RL fine-tuned on synthetic datasets. We observe consistent transfer from synthetic data training real-world multi- hop reasoning benchmarks: {RG-Family, RG-Knights, GSM-2o, Phantom + [HotpotQA, 2WikiMultihopOA. MuSiQue, CofCA, Synth SynthWorlds-RM, RMM. The performance transfer trends hold across model families and sizes (Qwen and Phi LLMS in 0.6-4B parameter range). On each synthetic dataset, fine-tune each L.M with 2 random training seeds, and evaluate final checkpoints of both experiment runs. With this we calculate the standard error, shown as error bars. See Figure 6 for similar plots for larger LLMs Qwen3-4B and Dwenn.5--BBInstrrus
Knowledge composition generalizes to unseen knowledge and question difficulties. The real- world multi-hop reasoning benchmarks are derived from Wikipedia, which was likely part of each model's 'ss pretraining corpus. It is possible that the synthetic-to- to-aell transfer in Figure 2 comes from better composition of memorized knowledge, rather than from learning general composition skill. Indeed, Shao et al. (2025); Yue al. (2025) hypothesize that RL fine-tuning primarily surfaces mem- orized pretrained knowledge in L.MSs. To test whether models acquire knowledge composition as

---

Published as a conference paper at ICLR 2026
Qven3-0.68
Qwen3-1.78
Qwen3-0.68
Qwen3-1.78
1.8
000
000
1.8
000
000
8.
Accuracy
8.8-
0.6
8.4-
8.2-
9.8
9 Question difficulty
16 Question dif difficulty
14
10
15
20
25
2
18 15 20
Arithmetis tic operations Arithmetic operations
2k 3i
base
1K
ak AK
Phantonwiki training steps
GSM-> training steps
Figure 3: Performance of intermediate training checkpoints on unseen test sets, stratified by question complexity. We evaluate Qwen3-0.6B and Qwen3-1.7B intermediate checkpoints trained on PhantomWiki (left) and GSM-oc (right) on held-out test sets. Test sets share no factual over- lap with training data. Question complexity is defined as the number of document hops (Phan- tikii or arithmetic operations (GSM-oo) required to reach the answer. As fine-tuning progresses (lines growing darker), performance improves across all complexity levels, including out-of-domain (OOD) difficulties beyond those seen during training. See Figure 8 for similar plots for other LLMs.
fundamental skill, we evaluate on held-out Phantom Wiki and GSM-co test sets. These share zero factual overlap with the corresponding training datasets and include question complexities beyond those seen during RL fine-tuning (Gong et al., 2025; Zhou et al., 2025).
Figure stratifies performance of Qwen3-0.6B and Owen3-1.7B intermediate training checkpoints by question complexity. Since the factual knowledge in held-out PhantomWiki and GSM-co test sets are fully disjoint from training data, models cannot succeed by surfacing parametric knowledge. They must learn to compose knowledge as irmainn-nddeeenent and fundamental skill. More- over, models improve across all difficulty levels as fine-tuning progresses- including out-of-domain (OOD) complexities never encountered during fine-tuning. This OOD generalization highlights that models can bootstrap the skill beyond what they were trained on.
SFT fails to teach knowledge composition. Supervised fine-tuning (SFT) requires reasoning traces, either from human annotations, frontier LLMs, or ground-truth solution traces. The first two are expensive collect, while ground-truth solution traces rarely available in rule-generated synthetic data, which typically only generates questions and verifiable answers. This makes RI. fine-tuning more broadly applicable than SFT for leveraging rule-generated synthetic data.
Even so, when ground-truth solution traces are available, can SFT deliver synthetic-to-real transfer like RL fine-tuning? To answer this, we SFT Qwen3-0.6B and Qwen3-1.7B models on GSM-co training data, which includes solution traces (Zhou et al., 2025). We evaluate the fine-tuned models on held-out GSM- test set and real-world HotpotQA benchmark, and compare SFT with RL fine- tuning in Table 2. While SFT improves performance on synthetic GSM-2o data, we observe no performance improvement in real-world multi-hop reasoning. On the other hand, RI. fine-tuning yields both in-domain and transfer improvement. SFT can thus overfit to the synthetic data task and fail to teach knowledge composition necessary for transfer.
Accuracy on GSM-xc SFT
Fl score on HotporQA
base
RL
base
SFT
RL
Qwen3-0.6B
0.0241
0.7735
0.6452
0.3569
0.3995
0.4786
Qwen3-1.7B
031454
0.7742
0.8532
0.5935
0.5761
0.6664
Table 2: Performance of SFT vs RL fine-tuning on synthetic (GSM-co) and real-world (Hot- potQA) test sets. SET on ground-truth solution traces from the GSM-oc training data improves performance on the synthetic task but not on the target HotpotQA. RL fine-tuning improves models on both tasks, indicating that it teaches transferable knowledge composition where SFT does not.
Synthetic data fine-tuning leads to grounded generations in real-world reasoning. Our fine- tuning setup only rewards the final answer, unlike process reward modeling that supervise interme- diate steps (Lightman al., 2023). Figure 3 shows that knowledge composition improves final-
8

---

Published as a conference paper at ICLR 2026
Fraction of Qwen3-0. generations with ground-truth intermediate answer NuSiQue
MuSiQue
1.0
0.4
0.2
Nth internediate answer Nth intermediate answer Nth internediate answer Nth intermediate answer
ak 1k PhantoniWkki training steps
AK
base
3k sk 5K
GSM-= training steps
Figure 4: Groundedness of Qwen3-0.6B reasoning traces in MuSiQue and CofCA intermediate answers. For each dataset, we plot the fraction of model generations (reasoning traces) that contain the ground-truth nh intermediate answer. As RI. fine-tuning progresses on Phantom Wiki (left) and GSM-2o (right), reasoning traces include a higher proportion of correct intermediate answers. This indicates that RL fine-tuning with outcome-only reward on synthetic data also improves the reason- ing process. See Figure 10 for similar analysis other models.
answer accuracy across all question complexities; however, it does reveal whether the reasoning process itself improves, i.e., whether intermediate steps become more accurate.
To investigate this, we leverage the fact that each MuSiQue and CofCA question comes with in- termediate sub-questions and corresponding ground-truth answers. In Figure 4, we measure how often these intermediate answers appear the model's reasoning traces across training checkpoints. As RL fine-tuning progresses PhantomWiki and GSM-xc, models produce reasoning traces with increasingly higher proportions correct intermediate answers. That is, outcome-only reward on synthetic data actually improves the groundedness of the reasoning process useful byproduct. Together with Figures/ and 3, this shows that knowledge composition improves LLM generation quality in both final-answer correctness and intermediate-step groundedness.
Performance scales with synthetic data. Recent work scales RL fine-tuning by modifying the RL. algorithm, model architecture, and test-time compute (Liu et al., 2025a). We highlight an additional scaling axis: training data. Since we train for single epoch on each synthetic dataset, models see each training sample exactly once. So, evaluating intermediate checkpoints is equivalent studying synthetic data scaling in RL fine-tntn..n
In Figure 5, Qwen3-0.66 steadily improves on all real-world benchmarks with more training steps, or equivalently, more synthetic training data. We do not observe overfitting to the synthetic task, in contrast to SFT (Table 2). We see similar trend for other LLMs Figure 9. Notably, different models exhibit varying malleability to RL fine-tuning: smaller models such as Qwen3-0.6B start weaker but improve steeply, while larger models such as Qwen3-1.7B are stronger at initialization but improve more gradually. Understanding how model initialization influences RL fine-tuning remains an open problem (Yue et al., 2025; Liu et al., 2024).
Qwen3-0.66 fine-tuned with
RG-Fanily
RG-Kn iights
6SM-
PhantonMilk
Hotpot.0A
SynthMorlds-
0.5
0.0
8,5
1.5
6:5-
A6
Training steps
Training steps
Training steps
Training steps
Training steps
Figure 5: Real-world multi-hop reasoning performance of Qwen3-0.6B intermediate training checkpoints. We evaluate checkpoints every 500 training steps, and report mean standard er- with the solid line and shaded region. Performance all benchmarks improves steadily with training steps- ceps--qqivallenlly with the number of synthetic training samples-prose evidence for synthetic data scaling in RL fine-tuning.

---

Published as a conference paper at ICLR 2026
5 DISCUSSION
Transferability of Reasoning. Our findings show that RL fine-tuning on fictional, template-based datasets improves performance real-world multi-hop reasoning benchmarks. Because the syn- thetic training data shares zero factual overlap with evaluation domains, this cross-domain trans- fer rules out surface-level memorization and supports knowledge composition a transferable skill (Toplak & Stanovich, 2002). More broadly, our results suggest that such multi-step logical inference is a comain-indeeendent skill that can learned in isolation from real-world knowledge.
However, the extent of this transfcrability remains an open question. Real-worid reasoning tasks require both factual knowledge and compositional skill, and models can learn memorization and generalization simultaneously (Xie et al., 2024). While memorization can hurt performance in counterfactual settings (Wu et al., 2025b), it may complement composition in standard settings. Understanding how synthetic and knowledge-intensive real-world data interact during training is an important direction for future work.
Synthetic datasets meaningful training signals. Our analysis demonstrates that rule-generated synthetic datasets are effective training signals: models learn transferable reasoning capabilities from them. This has practical implications. As high-quality human-annotated reasoning data be- comes increasingly scarce (Villalobos et al., 2024; Muennighoff et al., 2023), synthetic datasets of- fer scalable alternative where domain experts design verifiable task generators rather than labeling individual examples (Stojanovski et al., 2025; Liu et al., 2024; Yang et al., 2025).
While work focuses knowledge composition for multi-hop reasoning, synthetic data could teach other reasoning skills as well (Stojanovski et al., 2025; Allen-Zhu, 2025; Liu et al., 2024; Xiong et al., 2025; Sprague et al., 2025; Yuan al., 2025). Future work could explore whether causal reasoning, counterfactual inference, or analogical thinking exhibit similar transferability. Identifying the boundary conditions for this transfer-when works, when it fails, and how interacts with other training objectives- remains an important open (Zhao et al., 2023; Wu et al., 2025c).
6 CONCLUSION
We investigate rule-generated synthetic datasets as scalable alternative to real-world training data for LLM reasoning. Our results demonstrate that such synthetic training data teaches the trans- ferable knowledge composition skill that yields significant gains on diverse real-world multi-hop benchmarks. This is despite sharing zero knowledge overlap with evaluation domains and using semantically simple language. We show that this transfer reflects genuine skill acquisition: models generalize to unseen question difficulties, produce more grounded reasoning traces, and continue improving with more synthetic data. Our findings highlight new avenues for scaling RL fine-tuning, overcoming traditional data bottlenecks.
ACKNOWLEDGMENTT
AK thanks fellow Cornell CS students Atharv Sonwane, Chao Wan, Laura Zielinski, and Linxi Zhao for manuscript feedback and discussions on formal logic. AK thanks Claude for help with plotting. DG is supported by Empire Al Postdoctoral Fellowship. Authors acknowledge compute resources from the National Artificial Intelligence Research Resource (NAIRR) Pilot, Purdue Anvil AI, and NVIDIA's DGX Station compute platform through their Early Access Program. This work is sup- ported by the National Science Foundation (NSF) grants RI-2530143, OAC-21 18310, and through the AI Research Institutes program Award No. DMR-2433348. This work was partially supported by funding from New York-Presbyytriin for the NYP-Cornell Cardiovascular AI Collaboration, the National Institute of Food and Agriculture (USDA/NIFA), the Air Force Office of Scientific Re- search (AFOSR), and Schmidt A12050 Senior Fellowship, Schmidt Sciences program. We thank anonymous reviewers for their helpful feedback.
ETHICS STATEMENT
Our work adheres to the ICLR Code of Ethics, and does not pose any societal, personal, or organi- zational risks.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we use free and open-source software and publicly available LLMs. Our code is available at github. com/kilian galanggoup//// proup/phante wwwppttaanraaeeeee rerronnnnn, where include our full dataset preparation, model training code, and evaluation configuration (overview in Methodology section and Appendix A). We further report standard errors our results show significance, generate data with fixed random seeds, and set fixed training random seeds where possible.
REFERENCES
Emmanuel Abbe, Samy Bengio, Aryo Lotfi, and Kevin Rizk. Generalization the Unseen, Logic Reasoning and Degree Curriculum, Journal of Machine Learning Research, 25(331):1-58, 2024a.
htt.ttts:: //jmrr. org/ (ttpperr//2//22/-/e/
html. pages
Emmanuel Abbe, Samy Bengio, Aryo Lotfi, Colin Sandon, and Omid Saremi. How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpad. In A. Globerson, L. Mackey, D. Belgrave. A. Fan, U. Paquet, Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 27850-27895. Curran Associates, Inc., 2024b. URL https: http://pepeeeeeng...eo/oo neurips httpp//ceeeeeeee05oeeeee6eeeeeee Paperd-dd-pp-r---aaatae Pwwer-Coneeeeeeeeeeeeeeeenoe Conference pdf. pages
files/le///ll.. wwwle//ppper///0000 /ppppr//022//ile//
Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi. Neel Joshi, Piero Kaufffman, Yash Lara, Caio César Teodoro Mendes, Arindam Mitra, Besmira Nushi, Dimitris Papailiopoulos, Olli Saarikivi, Shital Shah, Vaishnavi Shrivastava, Vibhav Vineet, Yue Wu, Safoora Yousefi, and Guo- qing Zheng. Phi-4-reasoning Technical Report, April 2025. URL atttss//arrxe wwwrxiv.oorg.oo.. org/obgg 2504. 21318. pages 3,
Zeyuan Allen-Zhu. Physics of language models: Part 4.1, architecture design and the magic of canon layers, 2025. URL https: //arxiv.///iivee/aa org/abs/2512 17351. pages 3, 10
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073: 2022. pages
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. Abductive Commonsense Reasoning. In International Conference on Learning Representations, 2020. URL https: pages
//openreview. teee/fooummid-ByyyllHHll
Kianté Brantley, Mingyu Chen, Zhaolin Gao, Jason D. Lee, Wen Sun, Wenhao Zhan, and Xuezhou Zhang. Accelerating RL for LLM Reasoning with Optimal Advantage Regression, May 2025.
URL htttps://arxa /ttp::/ww.w..rr////.//oo/oo
abs/2505..... 20686. pages 3
Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. FireAct: Toward Language Agent Fine-tuning, October 2023. URL https://aax /bs// pages 3
Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. ARC-AGI-2: A New Challenge for Frontier Al Reasoning Systems, May 2025. URL https://arxiv.orr org/
abs/2505. 11831. arXiv:2505.118: 11831. pages
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg. and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.). Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:/ #Ee2occaeeee00111111222000000 Paper pdf. pages
neurips cc/paper seaper_files/paaeee wtt.s///paap//////////..
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu,
11

---

Published as a conference paper at ICLR 2026
Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling Instruction- Finetuned Language Models. Journal of Machine Learning Research, 25(70):1-53, 2024. URL html. pages 3
https://jmlr.
phapers/v25/23-12
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems, November 2021. URL https:
//aaxivvorgg////ee/............v httg//bs/211000 14168. pages 3, 4
Tri Dao. FlashAttention-2:: Faster Attention with Better Parallelism and Work Partitioning, 2024. pages 19
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cogni- tive Behaviors that Enable Self- lelf--mmpoving Reasoners. or, Four Habits of Highly Effective STaRs, August 2025. URL https: : / arrii.iivvvv... org/ abs/ 01307. pages 3
Anna Goldie, Azalia Mirhoseini, Hao Zhou, Irene Cai, and Christopher D. Manning. Synthetic Data Generation & Multi-Step RI. for Reasoning & Tool Use, April 2025. URL
org/abs/25000 04736. pages 3
Albert Gong, Kamilè Stankevičiūtë, Chao Wan, Anmol Kabra, Raphael Thesmar, Johann Lee, Julius Klenke, Carla P. Gomes, and Kilian Q. Weinberger. PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation. In Proceedings of the 42nd International Conference on Machine Learning, Vancouver, Canada, June 2025. PMLR. doi: 10.48500arX1::252223377. URL hwtph:tt////aet http://axxvv..oo hatgg//b///02.... 20377. pages 1, 2, 3, 4, 5, 8, 25, 26
Ken Gu, Advait Bhat, Mike A Merrill, Robert West, Xin Liu, Daniel McDuff, and Tim Althoff. Syn- thWorlds: Controlled Parallel Worlds for Disentangling Reasoning and Knowledge in Language Models, 2025. URL https: // arrxivvvvvv...i//aa./aaavav/ org/ abs 2510..212.2.... 24427. pages 2, 3,
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolve Deep Thinking. In Proceedings of the 42nd International Conference on Machine Learming, Vancouver, Canada, January 2025. URL https: / /ppo/eoeeeeeeeeeeeeeeeeeeeeeeeee ner/forum? ieriiid==siiii
pages 1
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhi- hong Shao, Zhuoshu Li, Ziyi Gao. Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Ji- ashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjic Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang. Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingehuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan. Runji Wang. R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu. Shunfeng Zhou, Shuting Pan, S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun. Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wen- tao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xin- nan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang. Yi- fan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yonggiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li,
12

---

Published as a conference paper at ICLR 2026
Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao. Zhicheng Ma, Zhigang Yan. Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang. Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. DeepSeek- R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081):6333 638, September 2025a. ISSN 0028-0836, 1476-4687. doi: 10.
URL https://wwy nature. com/ art.clestccceeeeaee s41586 0122-09422--2 z. pages 3. 4,5
Tianyu Guo, Hanlin Zhu, Ruiqi Zhang, Jiantao Jiao, Song Mei, Michael I. Jordan, and Stuart Russell. How Do LLMs Perform Two-Hop Reasoning in Context?, May 2025b. URL httph:t//aatttt org/ htt5/ab//25000 13913. pages 3
Seungwook Han, Jyothish Pari, Samuel Gershman, and Pulkit Agrawal. Position: General In- telligence Requires Reward-based Pretraining. In Proceedings of the 42nd International Con- ference on Machine Learning. PMLR, June 2025. doi: 10.48550/a2X..25522.1190 pages 3
URL
https://arxiv.. org/ absa 2502. 19402.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason E. Weston, and Yuandong Tian. Training Large Language Models to Reason in a Continuous Latent Space. In Workshop on Reasoning and Planning for Large Language Models at ICLR 2025, 2025. URL https:
// /oppnrevieww net. www//eerruuueeee
pages 3
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. In 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks, December 2021. doi: 11485500arxiv.2103..333 URL /ttps:///ppereviee.
net//oorumiidc7BByy2mmmeeee pages? 3
Xanh Ho, Anh- Aah-KKha Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing A Multi- hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. In Proceedings of the 28th International Conference on Computational Linguisticss pp. 6609-6625, Barcelona, Spain (On- line), 2020. International Committee on Computational Linguistics. doi: 1..1865//11/0200. coling-main.580. URL https: http://www.lllllcc org/ manthologgy/220..o111 maaiiinggmmi.... www..aainn......... 580. pages 3.5
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. REINFORCE+:: An Efficient RLHF Al- gorithm with Robustness to Both Prompt and Reward Models, August 2025. URL https:
//arziv. www.o////.....o..///r/r// 03262. pages??
Yue Huang, Siyuan Wu, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Xian- gliang Zhang, Jianfeng Gao, Chaowei Xiao, and Lichao Sun. DataGen: Unified Synthetic Dataset Generation via Large Language Models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: : //tp::://ppppaa httpppparrrrrieeeetorrour/////aa mett/forum21d-F5000077444
pages
Alihan Hüyük, Xinnuo Xu, Jacqueline R. M. A. Maasch, Aditya V. Nori, and Javier Gonzalez. Reasoning Elicitation in Language Models via Counterfactual Feedback. In The Thirteenth In- ternational Conference on Learning Representations, 2025. URL https://openrevaii /et.e
forummid= Jorumm?dd=WixxJJaavy, pages 3
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Lezama, Koushik Sen, and Ion Stoica. LivehodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In The Thirteenth International Conference on Learning Representations, 2025. URL https::/openrree
net/ net/ http//formmmiiichhee JJCC3.
pages 3
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient Memory Management for Large Language Model Serving with PagedAtttention. In Proceedings of the 29th Symposium on Operating Sys- tems Principles, SOSP '23, pp. 611-626. New York, NY. USA. 2023. Association for Com- puting Machinery. ISBN 979-8-4007-0029-77 doi: 00.114556000006.661131
URL https: ///oo..org///// 10. 11w.....111//110000011 3613155. event-place: Koblenz, Germany. pages 19
13

---

Published as a conference paper at ICLR 2026
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Ma- lik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Ha- jishirzi. Tülu 3: Pushing Frontiers in Open Language Model Post-Training, April 2025. URL https://arre www...rr/aabb httg/ab//200 2411 15124. pages 1, 3,4
Yang Li, Youssef Emad, Karthik Padthe, Jack Lanchantin, Weizhe Yuan, Thao Nguyen, Jason Weston, Shang-Way Wen Li, Dong Wang. Ilia Kulikov, and Xian Li. NaturalThoughts: Select- ing and Distilling Reasoning Traces for General Reasoning Tasks, July 2025. URL https:
//arziv. careiiiorrg/aaae www.oogg/aba..o.o..o http//baa250..... 01921. pages
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023. URL https://opeereviiie pages
net/foorumidd-v810ppeee
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong- ProRL: Prolonged reinforcement learning expands reasoning boundaries in large language mod- cls, 2025a. URL https: //trx/aa/vvv///////////////aaaaa arg/abs/2800 24864. pages 9
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. Best practices and lessons learned on synthetic data. In First Conference on Language Modeling, 2024. URL https: //arxiv./iivv////a wwwpo///g//.....//./// 07503. pages 1, 3, 10
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang. Chao Du Wee Sun Lee, and Min Lin. Understanding RI-Zero-Like Training: A Critical Perspective. March 2025b, doi: 1148850/rXii..553.22777 URL //tppi.///axvvo/r///e// pages
20783. arXiv:2503.20783
MAA. MAA invitational competitions. american invitational mathematics examiiaatin (aime). URL
https://maa..or/mmaaainn wtggmaaa-ivviaaiiionn www.inniiinnllaaaaooooooi wowp:aaloooooooiiiiiiiiiiii tions//. pages
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. In The Thirteenth International Conference on Learning Representations, Singapore, March 2025. doi: 10448500aaXxi.241000522 URL https://openrrw//// net// forum?id=AAJuRRIIvPP pages 3
David Mizrahi, Anders Boesen Lindbo Larsen, Jesse Allardice, Suzie Petryk, Yuri Gorokhov, Jef- frey Li, Alex Fang, Josh Gardner, Tom Gunter, and Afshin Dehghan. Language Models Im- prove When Pretraining Data Matches Target Tasks, July 2025. URL https://arxiv..r abs/2507 12466. pages 3
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling Data-Constrained Language Models. In A. Oh, T. Naumano, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp- 50358-50376. Curran As- sociates, Inc., 2023, URL https::/ wwwpe:eeddddnnnnnennneeeeeeeeeee cc/paper files/ walleeepaae ppppp//222///////////// http://eeeeeeecaoeoeeeerrreerrre Conference, .df. pages 1, 10
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feed- back. S. Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. 27730-27744, Curran Asso- ciates, Inc., 2022. URL https: httpss://proceedinggs nwwrips.e.... fil ttlerbarfasseeeear344485853353.2 confereee pdf. pages
14

---

Published as a conference paper at ICLR 2026
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Ja- son Weston. Iterative Reasoning Preference Optimization. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neu- ral Information Processing Systems, volume 37, pp. 116617-116637, Curran Associates, Inc., 2024. URL https://prooceednngg wwtp:/s...epppppppppeeeeeeeeeeee cc/paper___ppe ff files/lll//llll www.ffieeepppeeee httppppppppp//////222/ Paper Conerr--eaaer--eeeeeeeeeeee pdf. pages 3
Shishir G. Paril, Tianjun Zhang, Xin Wang, and Joseph F. Gonzalez. Gorilla: Large Language Model Connected with Massive APIs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: htctpa://openrer///
net weeee/oorrmmnditBB/eeee
pages 3
Peng Qi, Haejun Lee, Tg Sido, and Christopher Manning. Answering Open-Domain Questions of Varying Reasoning Steps from Text. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3599-3614, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10.18653/v1/2021 wew.l.eeeenlppmmmmm mannpp-miin. 292. URL https: https////aaaaaahoooaaeee org/ 2021. emnlp-main 292/ pages 3
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https:///weet github.io/blog// wtt.:iiooolooeeeeee qwen2. 5/. pages
Qwen Team. Qwen3 technical report, 2025. URL /tt 09388, pages 4, 6
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning. Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly Reward Model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 33728-53741 Curran Associates, Inc., 2023. URL https: http:////ccceeeeeeeeeeeee newriis pappappr_... lameerffile files/ http/ppper/22///1//// 01554466666664477744118882666663 waw.aa000eeeeeeeeeeeaeeeeee--- Copper-Conference pdf. pages
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A Graduate-Level Google-Proof Q&A Benchmark, First Conference on Language Modeling, 2024. URL https:// pages
net/forum?id-T1675588
Yangjun Ruan, Neil Band, Chris Maddison, and Tatsunori Hashimoto. Reasoning learn from latent thoughts. 2025. URL https://arxix: org/
18866. pages
Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. ARES: An Automated Eval- uation Framework for Retrieval-Augmented Generation Systems. Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 338-354, Mexico City, Mexico, June 2024. Association for Computational URL https: htt///aalaaa /a/aa/aaaattoooyyyyy.o org/
Linguistics. doi: 10. 10..8653/v1 naacl-long-1 20/. pages
Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexan- der Kranias, John J. Nay, Kshitij Gupta, and Aran Komatsuzaki. ARB: Advanced Reason- ing Benchmark for Large Language Models. In The 3rd Workshop on Mathematical Rea- soning and Al at NeurIPS 2023. arXiv, July 2023. doi: 10.48550/axx..2337...3 https://arxiv.. .rgg orgg/bs//200 abs/2307 13692. pages 3
URL
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms, August 2017. URL https://arxiv. www./b//////// 1777706347 pages 5.5
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang. Sewoong Oh, Simon Shaolei Du,
Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi. Pang Wei Koh, and Luke Zettlemoyer. Spurious Rewards: Rethinking Training Signals in RLVR. June 2025. doi: 10.488550/rxxw.22566.. 10947. URL https: //apxi///xxvvv.. arxii. org/ abs//506.... 10947. pages 2, 3, 4, 6,
15

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, April 2024. URL https:/ hatps://aaxvvv. org/
pages 3, 5, 19
Parshin Shojace, Iman Mirzadch, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens Problem Complexity, July 2025. URL cattps://areee arg/abs/2500 06941. pages 3
Vaishnavi Shrivastava, Ahmed Awadallah, Vidhisha Balachandran, Shivam Garg, Harkirat Behl, and Dimitris Papailiopoulos. Sample more to think less: Group filtered policy optimization for concise reasoning. 2025. URL https: // arxiv org/abs /abs//508... 09726. pages 3
Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text. In Kentaro Inui, Jing Jiang. Vin- cent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJNLP)), pp. 4506-4515, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10..88///v/DD1-111 1458.. org/D19- 1458/. pages 3
URL https: httpp://accletee haapa////ccanthoooyy..
Zayne Sprague, Jack Lu, Manya Wadhwa, Sedrick Keh, Mengye Ren, and Greg Durrett. SkillFac- tory: Self-Distillation for Learning Cognitive Behaviors. 2025. URL https://arxiv.. org/ abs/2512. 04072. pages 3, 10
Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulbakeem Adefioye, Jean Kad- dour, and Andreas Köpf. REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards, May 2025. URL https://arxiv: www.:ooog/abbba arggabs//5055../. 24760.
D.Xiv:2505.24760 pages 2, 3, 4, 5, 6, 10
Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding RL with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.228: 23829, 2025. pages
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseeA:: A Ques- tion Answering Challenge Targeting Commonsense Knowledge. In Proceedings of the 2019 Con- ference of the North American Chapter the Association for Computational Linguisties: Human Language, pp. 4149-4158, Minncapolis, Minnesota, 2019. Association for Computational Lin- guistics. doi: 10.19655/ URL https: pettps://cclle /awcweb..bbe/....///. org/ wrrg/anhhology/Me 1421. pages 3
Yixuan Tang and Yi Yang. MultiHop-RAG: Benchmarking Retrieval-Augmentie Generation for Multi-Hop Queries. In First Conference on Language Modeling, 2024. URL https: //openreview www.eeww.eeerrrree
pages 3
Maggie Toplak and Keith E Stanovich. The domain specificity and generality of disjunctive reasoning: Searching for generalizable critical thinking skill. Journal educational psychology, 94(1):197, 2002. pages 10
Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He Hc, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476 482,, January 2024. ISSN 0028-0836. 1476-4687. doi: (0.1038//41586-023--000 023-06747-5. URL https: / //ww:///..//////// nature. com/ htte/rrtilees/// (41586-023-0678 020-7727--77765500 pages
Harsh Trivedi, Niranjan Balasubramanian. Tushar Khot, and Ashish Sabharwal. MuSiQue: Mul- tihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics, 10:539-554 554, May 2022. ISSN 2307-387X. doi: 16.1162/tacl http://rriicee////o//0.111///1/1 w1tp./111e/aaaaa000000 00445// www.:iiiihhooouuuoooooo-se-o Questions via Sww.ieieoo--uuesiiiiioooooooo pages 3.
mit eetttt.du/tc//c/ a 00475. URL attps::/diree http://tww...atttaeeaeaaaaaaaaaa wwt.t//acl/atiicee/////////////e 110996 Mussi Muu-MMutihop--
16

---

Published as a conference paper at ICLR 2026
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: Will we run out of data? Limits of LLM scaling based on human-genae generrtte data. In Forty-first International Conference Machine Learning, 2024. URL htpps://opeeeeeiiee
net./torumiid-iizeg003. pages 1, 10
Leandro Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement
learning. httpp://gi /ttp:://ggttt..t. httpge///ggccbb....c///gggggggge tr1, 2020. pages 5, 19, 20
Chao Wan, Albert Gong, Mihir Mishra, Carl-Leander Henneking, Claas Beger, and Kilian Q Wein- berger. Memento: Note- Note-Taking for Your Future Self. June 2025. URL https://arre /ttps///axiivvrroo abs/2506 20642. pages 3
Xumeng Wen, Zihan Liu, Shun Zheng. Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, Jiang Bian, and Mao Yang. Reinforcement Learning with Ver- ifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs, June 2025. URL https://arxisa org/ http//bbs//0.....//// 14245, pages 3,
Jian Wu, Linyi Yang. Zhen Wang. Manabu Okumura, and Yue Zhang. CofCA: A STEP- WISE Counterfactual Multi-hop QA benchmark. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Representation Learning, volume 2025, pp. 14631- iclr cc/paper__pp httppppppeeeeeee//eeaaaaaaaaeeee files wwtplas/apppe///eeeeeeeeeeeeeeee /aaper/2025/flee/
14649, 2025a. URL https: Ponfreeeeeeeeeee--ooereeeeeeee Conferenceeeeeeeeeeeeeoeee pdf. pages 2, 6
Jian Wu, Linyi Yang, Zhen Wang, Manabu Okumura, and Yue Zhang. CofCA: A Step- Wise Coun- terfactual Multi-hop QA benchmark. In The Thirteenth International Conference on Learning Representations, 2025b. URL https: 6, 10
pentps://operreee http:///ppeeeeeeeeeeeeee deae//oooommmddmm222 d=q@DmkZwwVe pages 3,
Mingqi Wu. Zhihao Zhang, Qiaole Dong. Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Yanwei Fu, Qin Liu, Songyang Zhang, and Qi Zhang. Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due Data Contamination, July 2025c. URL https://arxis: org/ hbtg//bs//200 abs/ 10532. pages 3, 10
Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, Louis Castricato, Jan-Philipp Franken, Nick Haber, and Chelsea Finn. Towards System Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thought, January 2025. URL https://arxins wtg//rr///////o 04682. arXiv:2501.04622 pages? 3
Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On Memorization of Large Language Models in Logical Reasoning. In The 4th Workshop Mathematical Reasoning and Al al NeurlPS arXiv, 2024. doi: 10. 48550/anXiv:2410223133 URL. https: htttts:///peeeeviieee net. htt.//eetouutteee htt///foummiddeee 3dm pages 3, 10
Roy Xie, David Qiu, Deepak Gopinath, Dong Lin, Yanchao Sun, Chong Wang, Saloni Potdar, and Bhuwan Dhingra. Interleaved Reasoning for Large Language Models via Reinforcement Learn- ing, May 2025. URL https: http://xxxvv.vo/ www.oorgg//bbb/oo///ooo/..eeee abs////s//55// 19640. pages 3
Zheyang Xiong, Vasilis Papageorgiou, Kangwook Lee, and Dimitris Papailiopoulos. From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data. In The Thirteenth International Conference on Learning Representations, 2025. URL potps://arxtt: org/ abs/a/bs//22/..////// 19292. pages 10
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-263222 2380, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v 1/D 18- 1259. 5,5
URL https: /wwwweb...b org/ anthology/D18--t 1259, pages 1, 2,
17

---

Published as a conference paper at ICLR 2026
Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candès, and Tatsunori Hashimoto. Synthetic continued pretraining. In The Thirteenth International Conference on Learning Representations, 2025. URL battp:://opereeiers nett/forum/1dd-7yysk0p.
pages 3. 10
Shunyu Yao, Jeffree Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Leurning Representations, 2023. URL httph::/p//// /ttp:////pp//ee.////// www//foumm
WE_vluYUL-X. pages 3
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. T-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. In The Thirleenth International Conference Learning Reprexentations, 2024. URL pietps://arzee .rgg www..as.////.....ooooo........at /abs/2406... 12045, pages 3
Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown- Cownn Anirudh Goyal, and Sanjeev Arora. SKILL-MIX: Flexible and Expandable Family of Evaluations for AI Models. In The Twelfth In- lernational Conference on Learning Representations, 2024. URL httph:ttt///ttt //pprr////e/.ee./////e/re net/
forun?id=Jf5gplve
pages 3
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang. Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang. Hongli Yu. Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang. Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang- DAPO: An Open-Source LLM Reinforcement Learning System at Scale. May 2025. doi: 10. 40.48855/arx...2503..5 14476. URL BRL.htts:///arxt org/orgg//o abs/ /2003/ 14476. arXiv:2503.1444 14476. pages 3
Lifan Yuan, Weize Chen, Yuchen Zhang, Ganqu Cui, Hanbin Wang, Ziming You, Ning Ding, Zhiyuan Liu, Maosong Sun, and Hao Peng. From f(x) and g(x) to [g(x)): Lims learn new skills by composing old ones. 2025. URL https://arXiv org/ aa////b//22/000 25123. pages 10
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao. Zhaokai Wang. Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025. URL https: /ttp:///aavvvvi.e org/ ab0./ 13837. pages 2, 3, 7,9
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STaR: Bootstrapping Reasoning With Reasoning. In Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp- 15476-15488. Curran As- sociates, Inc., 2022. URL http:ttt//oooocediiiiee neurips cc/ccc//aaaoo htpppppperrrfll filffiee//// wallle/sappe /ppppp/p/22/2 htt///99.a42204400677776ffaaaaa2 Paapsa/aaaaaeeeaaaaaae-oooeeeeee Conper-Cooferrnceeeeee pdf. pages 3
Wenting Zhao, Justin Chiu, Claire Cardie, and Alexander Rush. Abductive Commonsense Reason- ing Exploiting Mutually Exclusive Explanations. In Proceedings the 61 Anmual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 14883-14896, Toronto, Canada, 2023. Association for Computational Linguistics. doi: 10. 18638663/ 122233/1o//.....loooonggg. URL https/////ccaaatohoooo.o org/ 2023 acl- acl-long. 831. pages 3. 10
Xinyu Zhao, Fangcong Yin, and Greg Durrett. Understanding synthetic context extension via retrieval heads. In International Conference on Machine Learning, pp. 77885-77910. PMLR, 2025a. URL https: /tt///axiivvv. org/abs////eooooooooooooo /2400 22316. pages
Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to Rea- son without External Rewards, August 2025b. URL https: /ttpss//arxii... argg/bss/2055 19590.
arXiv:2505. 19590. pages? 3
Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, and Beidi Chen. GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity? In Proceedings of the 42nd International Conference on Machine Learning, volume 267, Vancouver, Canada, February 2025, PMLR. URL https: : openrevien.
pages 2, 3, 4,
18

---

Published as a conference paper at ICLR 2026
Yuchen Zhuang, Yue Yu, Kuan Wang. Haotian Sun, and Chao Zhang. ToolQA: A Dataset for LLM Question Answering with External Tools. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. 50117-50143. Curran Associates, www.rpp.cc/papp....e.eppp waapeeerfflllepppaeeeeee https/aaprr222//11112
Inc., 2023. URL https://procceddigg 96227798600000022111156666011112 Datassts and Lee__BBenhehssee pdf. pages 3
A EXPERIMENT CONFIGURATION
A.1 GRPO DETAILS
We restate the GRPO objective from Shao et al. (2024) here. Given question g sampled from distribution over question set P(Q), GRPO samples a group of output completions (01....,00 oG) from the old LLM with parameters Oola- Then it assigns each output completion scalar reward value {R ..... RG}. ). The algorithm estimates the advantage A; of each completion by normalizing with respect to the average reward as baseline. The final objective is as follows:
JERPO
E q- p()) (o..))))))) (19)
10i
1 1
min (r..A. clip (ri,e: 1 E, 1,1 + A.d]] -BDDxL To | }
G
0i
where Aie R2 mean R1, RG)
stdev Ri..... Re)
Here Tet is reference policy (usually model initialization) used in the KL divergence penalty DKL, 3 are hyperparameters, and the relative weight rs, for output completion og is calculated on xl.0..10.0..cc)) per-token basis Ti. Te (o.,e (..
Implementation. We use the GRPOTrainer implementation from the open-source Hugging Face TRL v0. 21.0 library (von Werra et al., 2020). We use vLLM colocate mode (Kwon et al., 2023) and FlashAttention-2 (Dao, 2024) on NVIDIA H100 GPUs each with 80GB VRAM. With the configuration in Listing 1, RL finetuning
to 4B parameter LLM 10K training samples takes day on our Linux cluster. The prompt length varies for each training dataset, and we adjust the woo__prompt__eeee to prevent prompt truncation:
PhantomWiki: 6000
2. GSM-oo: 2048
3. HotpotQA: 6000
4. 2WikiMultihopAA: 6000
5. MuSiQue: 8000
Training parameters per
num num_generations:
vLLM settings use_vilm: true vllm_mode: "colocate" wil..gou.maoryyutiiiiataii 0.20
10
" Generation parameters max_ meexccoopletioo_enggeee 4096
temperature: 1.0
14 top-p: 1.0
'The larger models Owen3-1.7B and Phi-4-mini seatiirrrsoninn take the full 1 day. i.e. using 100 H100 hours training experiment as they generate long CoT. The Qwen2.5-1.5B 1.5-1.1.-nnnstte model does not generate long CoT, and thus trains in s=20 H100 hours.
19

---

Published as a conference paper at ICLR 2026
15 top_k: mull
16 pi: null
17 repetition.penaltyy 1.0
18
GRPO algorithm parameters
beta: 0.0
21 epsilon: 0.2
22 prortaanee_emmllg_eeeaa "token"
scale_rewards: true
loss_type: bпpo raxx_truccttddccomleeeiiss false
Listing GRPOTrainer hyperparameter values in our YAML configuration file
Adjusting hyperparameters for large LLMs. Due to memory constraints on our GPU resources, for Phi-4-mini-reasoning and Qwen3-4B LLMs with 4 billion param- eters each, we lower hyperparameter values vllm_gpu, Saam__gp_mmeeee wwm.rmmmrr....mm memory_utilizatee wwr_uuilizaaiion 0.25, per_device_trast brrainnatcch_ateeeeeeeeeeaeeeaee further reduce and num_generations:
we 0.30, le__dviceetrriine batcbbaaciiaaae 2, When developing GRPO fine-tuning experiments on the small models, we had observed that large number of GRPO rollouts and training batch size improved RI. fine-tuning stability and performance. Unfortunately our resource constraints compel us to use suboptimal lower values of these hyperparameters. This could be factor in the mixed results large LLMs in Figure We leave to future work to study the effect of rollouts and batch size on RL fine-tuning performance.
4, and num_generations: 8. For Qwen2.5-7B- wwwp__/mory_uutliieeeeooo
A.2 SFT DETAILS
For SFT on ground-truth solution traces in the GSM-oo training data, we use the SFTTrainer implementation from the same open-source Hugging Face TRL 21.21. library (von Werra al., 2020). Maximum length of 8192 is sufficient to accommodate each prompt and solution trace, as it exceeds max fexx__rompt_leree lenpthh
max max_complett www.pptiiio.n.cm.mmmmmemmmmmmeee conpletion_length equalling 6244 Section A.1. The loss is computed on all text both prompt and the solution trace; computing loss on solution trace only did not materially change final performance.
use default values for other hyperparameters. With the configuration in Listing 2, SFT uses H100 GPU with 80 GB VRAM.
Training parameters
per _er__eeereeee www.ddiieetrainnbbeeeeeeeeeeeeee btt._ttrriabbacc__iiti_. batch_
packing: false max_ max_length: 8192
Listing 2: SFTTrainer hyperparameter values in our YAML configuration file
20

---

Published as a conference paper at ICLR 2026
B ADDITIONAL RESULTS
See Figures 6 to 10.
base
RG-Family
RG- RG-Knights
GSM ce
PhantomWiki
HotpotOA
2Wiki
MuSiQue
CofCA
SynthMoolld-RR
1.0
8.8
0.4
8.2
Qwen3-4e
1.0
0.6
0.4
0.2
Figure 6: F1 scores real-world multi-hop reasoning datasets of LLMs RL fine-tuned synthetic datasets. Larger LLMs Qwen3-4B and Qwenn..577B-Instruett, which already perform strongly at initialization, show less consistent synthetic-to transfer. We note that memory con- straints necessitated suboptimal hyperparameters for these models (Section A. and scaling up RL fine-tuning for larger models remains an important direction. On each synthetic dataset, we fine-tune each L.M with a fixed training seed and evaluate the final checkpoint. We report the standard error of evaluating the final experiment checkpoints.
21

---

Published as a conference paper at ICLR 2026
base
Phantomwiiks
HotpotQA
2Wiki
MuSiQue
Hotpot0A
2wiki
MuSiQue
CofCA
Synthioorlds-RN
1.0
0.0
0.6
0.4
0.2
Qwen3 0..6
8.8
0.6
0.4
0.2
0.0
Qwen3-1.3
1.0
0.4
0.2
0.0
Qwen2.5 11122.51..55 -n-.srutnttt
1.0
8.8
0.66
0.0
Phi-4 10a--4-Mini-Reasoniie RRessnnin
Figure 7: F1 scores real-world multi-hop reasoning datasets of LLMs RL fine-tuned training sets of real-world data. On each training dataset, we RL fine-tune each LLM with random training seeds, following the same training setup with synthetic datasets in Figure (10K training samples, Fl scores as reward). RL fine-tuning on real-world data outperforms rule- generated synthetic data like PhantomWiki, which is expected synthetic data lacks rich linguistic complexity of real-world data. In fact, in-domain training data outperforms other data sources, i.c., HotpotQA training data yields maximum gains for HotpotQA benchmark (light green beats others in the left most column), and similarly for 2WikiMultihopQA and MuSiQue. We report the standard error over both experiments and benchmark samples.
22

---

Published as a conference paper at ICLR 2026
Phi-4-Wiin--Resoonnng
Qwen2.5-1158-Ims-e
Phi-4-WWini-Resonnnng
1.8
000
000
1.8
000
000
8.8
8.8
0.6
8.4-
0.2-
8.2
14
10
15
20
25
18 Art wweetmetiee ic operation.
15
20
Question tian difficulty
Question di difficulty
Ará trithhetic operat ions
2K ak
AK
5K
base
8K
1K
3K AK
Phantonwiki training steps
GSM
training
steps
Figure 8: Performance of intermediate training checkpoints unseen test sets, stratified by question complexity. We evaluate (ive2.5-11.BB-nnstii and Phi-4-mini-reasoning intermediate checkpoints trained on PhantomWiki (left) and GSM-Xc (right) on held-out test sets. Test share no factual overlap with training data. Question complexity is defined as the number of document hops (Phantom Wiki) or arithmetic operations (GSM-oc) required reach the answer. As fine-tuning progresses (lines growing darker). performance improves across all complexity levels, including out- of-domain (OOD) difficulties beyond those seen during training.
Qwen3-1.78 1.78 fine-tuned with
RG- RG-Fanily
RG-Kn iights
GSM-w
Phantomwiki
NotpotQ4
2Wiki
NuSiQue
CofCA
SynthMorlds
0,7
14
26
36
dx
28
38
34
24
36
24
96
1K
36
16
22
Training Steps
Tealning Steps
Teaining steps
Training steps
Training steps
Phi-4-Wini-Reesonii fine-tuned with
RG-Fanily
RG-Knights
GSN-
PhantonMiki
NotpotQA
2Wiki
NuSiQue
CoFCA
SynthMorlds
0.4
8.5
6.3
0.44
14
3 d
8.4
0.2
56
0.2
55.
1.2
24
Training steps
Training steps
Training steps
Training steps
Training steps
Quen2.5-1. SB-Instruct fine-tuned with Fanily
CSWl
PhantonMiki
HotpotQA
2Wäkci
NuSiQue
CofCA
SynthMorlds-Bag
17
0.1
16
26
24 56
16
16 Treining steps
Training
Teaining
Training
Training
Figure 9: Real-world multi-hop reasoning performance LLM intermediate training check- points. We evaluate checkpoints every 500 training steps, and report mean + standard error with the solid line and shaded region. Performance of Qwen3-1.7B and Phi---mini-r on all bench- marks improves steadily with training steps- eevivaaeinllnnl.. with the number of synthetic training samples-providde evidence for synthetic data scaling in RI. fine-tuning. Qwen2.5-1.5B-Innse performance saturates quickly.
23

---

Published as a conference paper at ICLR 2026
Fraction of Qwen3-1. 7B generations with ground-truth intermediate answer MuSiQue
NuSiQue
1.0
0.4
0.2
Nth interredian ornediate answer Nth int answer Nth internediate answer Nth internediate answer
ok
ik Phantonwiki training steps
3K
4K
base
sk 4k
SK
training steps
Fraction of thi-4-mini-reasonánne generations with ground- truth intermediate answer NuSique
MuSiQue
1.0
0.8
0.4
0.2
Nth intermediate Nth intermediate Nth internediate Nth internediate
base
Phantonwikk training steps
GSM training steps
Fraction of Qwen2. 5-1. Instruct generations with ground-truth intermediate answer S--ssruuttt
MuSiQue
CoFCA
NuSiQue
1.0
0.8
0.2
0.0
internediate intermediate
Nth internediate Nth intermediate
base
PhantonWika training steps
GSA- training stops
Figure 10: Groundedness of LLM reasoning traces in MuSiQue and CofCA intermediate an- swers. For each dataset. plot the fraction of model generations (reasoning traces) that contain the ground-trutt th intermediate answer. As RL fine-tuning progresses on PhantomWiki (left) and GSM-co (right), reasoning traces include higher proportion of correct intermediate answers. This indicates that RL fine-tuning with outcome-one reward on synthetic data also improves the reason- ing process. The effect is lower for larger LLMs the CofCA benchmark.
24

---

Published as a conference paper at ICLR 2026
C PROMPTS
C.I PHANTOMWIKI PROMPT
We use CoT prompt template and examples from (Gong et al., 2025), with custom instruction asking the LLM to output the final answer within <answer>.. </answer>.
You (BEGIN RVIDENCE)
following evidence:
[errdencc)) (END RVIDENCR)
provided question. responee SPINAL_ AESHER s/ansver>
tage: <ansver
leere FIBAL_ ABSVER
the folloring:
only liat namea separated by
there are multipla corract anzvera); or nunbers separated by 7, (if the ansver se numerical); or aсauer).
string (if there
Here are zone exanples:
(START OF EXANPLES)
Exanple
Nang?
Ansver: Baxad an avidence, alatara Nang ara Barabara Beltran, Rackworth. Canaser> Barabara Beitran, Vicks Hackworth ancker>.
Example
Question: Who is the child of Alvaro Srock?
Ansver: Based on the evidence, the children of Alvaro Snock are Eli Snock, Gane Snock. <ansver> Snock >
Example
Question: Who is the friend of the child of Alvaro Snock?
Answer: First
the children Alvaro Snock. Based the evidence, the children Saock, Baced on the cvidonce, the friends of 311 Snock are Leiss Latz, Shell1 Beltran, Vicki laakwoorth Virgil Hacksorra, Al1son Snocs, Brian Jeltran. The friends of Geze Snoek are Leeann laacvvorta, Seeck. <anewor>Leiss Boltran, Leeann Backzorth, Ricardo Hackvorth,
Alvaro
Szock,
need find friends
and
Leisa Ricardo Hackvorth, Vicki Backvorth, Virgil Hackworth, Snock, Doninique Smock S/anserr)
Shelli Beitran,
Example
Question:
Vicki Ansver: An aust is the sister of parest. Based en the eridonce, the parents of Vicki Backzorth are Shelli Beitran, Dins Beltran. To find the aunt of Vickt Backvorth, I need to find the sister of Shelli Beltran
Shell: Beitran <anczer>Stacia s/sswer>.
Example B:
Question: What
occupation husband Ansver: Based on evidence, husband Wilbert Toonbs is theatre aazager. <ansver> theatre asnager s/nsverr.
TooIds?
Vilbert Tocaba. accupation
Exanple
Question: What
daughter laz Ansver: à daughter in 1n-lav is the vife of child. Based on the evidence, the children of Lanaie Snock are Eli Baock, Dene Bauck. Eli Szack has vife, and the cife of Gens Sxock is Doninique Bnock. bobby Doniaique
hebby
<ansver>onninee </anverr
Example 7:
Question: What is the date of birth of the pereon whose hobby is finance?
people hobby hobby finance is Tooabs. 20959-03
Bazed Toonba 0959-03-22.
great grazddaughter Ansver: I ased to search ch for people whase accupatios is bionedica: acientist. Bazed on the avidence, the persoz shose occupation is bioxedical scientist is Lannie Baock. To find the great- granidaughter Lannis on
Question:
person 9hose occupation biomedical
daughter the children of Lansie Snock has no child, and the child of Cene Bnock is Villians Sneck. The daughtors Witliams Bnock aze Shalli Beltras, Stacis Taonba. Shalll Beltran,Stacks Stacia Toosha c/ansvar>.
Snock.
Seack, Snock.
Example
Qusation: Bov masy friands doen Byan Wang have? Ansver: Based os the evidence. the friunds of Ryan Neng are Shelli Beltran, Stacia Toonba, Virgil Vang. <anavers<//mmnee>.
Example 10:
Quextion: Hov friands does the child of Alvaro Srock have? First,
children
children
Sxock. Bos need the avidence, the friends of Eli Stack are Lains letz, Shalls heltran, Vicks Hackuorth, Virgil Hacksorth, Alixon Stock, Brian Beltran. The frisnds of Gene Brock are Lawana Backworth, Lwisn lats Hackworth,
Snock,
masy frienda they have. Based
Dominique
anserr...
25

---

Published as a conference paper at ICLR 2026
Exanple 11:
Question: Hov mazy ancles does the friend of Stacis Tsonbs have?
insver: Firet, need to find the friende of Stacia Toonbe. Based on the evidence, the friende of Hacksorth, Lesley the brother
Stacia Teoube
Brias they have. Brian Beltran has no parente, leish latz has no parente, Leeann Hackeerta has parests, Lesley lutz has parents, and Byan Vang has parents. Based the evidence, the parents of Leeann Vichi Hackvorth, Ricardo Hackvorth. luts, parents leslay Laiz
Nou seed Ryan parent. Based the evideace.
aasy
brothers. Based Lutz. brother of
avidence, pareata Virgil Hacksorth, so he is an uncle of Lesley Lutz. Islah Lutz has ne bzothez. Se the friends of Stacis Toonbs have 0, 0, 0, 1, uncles. Unique 0, 1. <ansver 20,1er anseer...
(END
Question: ((question))
Ansver: the
C.2 GSM-c0 PROMPT
We modify the CoT prompt template from PhantomWiki (Gong al., 2025) by replacing EVIDENCE with the problem statement. GSM-oo also generates a templated solution for each question pairs, which we use as the CoT examples in the prompt.
following problex:
(BEGIN PACBLEM)
((problem)) (END PROBLEN)
provided question on problen. Your reepoase mast enclesed is tage: Sanaver s/anave:>
ansver
FIEAL ANSVER
nunber.
Hore are some exanples:
(START OF EXAMPLES)
Exanple
Question:
nunber
ssixals Maple Creek?
Anaver:
Dafine edul: wolf is Mapla Creek aa
Define total number of adult anixala in Maple
Creek 2o 2. <ansver22s/azave:
Exanpla
Question: What is the tetal number of schools in Clearvater Bay?
Ansver: Dafine alanentary achool in Rivarton City highachnol in
b Dafine private middla achool elexestary achool Clearvater Bay as S; an -
Bay 2. Define total nuaber schools
Clearsater Bay aa H; V Grrin2
8. cansvrr8c/anavar>.
Exanple
Quextian: What is
nunber savies in Pantival Clairnont?
Ansves: Define upbeat netropolia conady is Festivel de Saint-Rivage as n: 20 4, Define tatal number of nevias Festival de Saint-Rivage
Define intense detective thriller
Ln0err Valnont total nuxber of nevies Festivel Luni Ne00uBee Valnont Q; so
3 6; - Define solens
parind drama in RNaDdasvex de Bellaville N; = Q =
4 10; an
14. Befine tatel number of novies in R\uBdeaves de Bellaville 14 q Define futuriatic
KK
y 31 93. Dafine total number novies in Pentival 95. Sanaver >83</anaver>
Festival Clairsont
+ 31: на Clairmont ax ps ae
(END OF EXAMPLES)
Dueation: (qquaation)) Ansver:
C.3 RG-FAMILY AND RG-KNIGHTS PROMPTS
We modify the CoT prompt template from Wiki (Gong al., 2025) by removing the EVIDENCE section, as below. We manually write several CoT examples from each synthetic training dataset.
You will be provided with 3 quection. Your response zust ond eith the final ansver enclosed is tags: SLSHer>FINAS ANSVER s/nsserr answer>
Eare aose exanples: (S7ART OF EXANPLEB)
((exanples)) (END EXAMPLES)
You are gives the following problex: Dusation: ((queation)) Ansver:
C.3.1 RG-FAMILY PROMPT EXAMPLES
26

---

Published as a conference paper at ICLR 2026
Example
Question: Harry is narzied to Iaily. They have a child called Daniel. that in Enily to Daniel? Respond only with the vord that deseriber their relatioschip. Ansver: Harry Eaily the parents of Daniel, Iaily the mother Daniel. </ aEGNOR>
Exauple 2:
Question: James narried to Aria. They What is Gearge to Olivia? Respond only with word that describes their relationship. Ansver: Since George is sarried to clivia, Geozge is the hesband of Dlivia. sansver sansver //
child cailed
deezge married Olivia.
Example
Question: John is mazried to olivia. They have child called Lian. Lies narried to Willow. They have a child called Logan.
That relation is John to Olivia? Ansver with single word. Ansver: John narried flivia,
the husband of Olivia. husbaadd//asseee
Example 4:
Question: Ryder narried to Lily. They have child colled John. John is garried to Aris. They have childres called Neah
Rov Joha related to Daniel? Provide relationship one word.
Ansver: Reah and Daniel are children of John, so John is the father of Daniel. cazsvor> </ansver>
Exanple
child called Joseph. Jaseph is married Lina. They have a child called Eleanor. Killian ia parried to Anelia. They have child called Lusa. How is Eleanor related to Luza? Previde the relationship in one word. tagether they
Question: Alexander narried to Lisa. They
Anaver: Jonaph daughter of Lens. "anaver s/unauer>
child called Eleanor, Rlaanor
Exampla 6:
Queation: Charles is narried
child called They have children called Lacy and Rydez. Christopher narried to Patricia. They have child called Suzan.
They
is narried
That is Lucy to Nason? Reapond only with the vord that describes their relationabip. daughter Mazon, leey
Lacy
daughter
cansver>daughter </anauer>
Example 7:
Quaxtion: John is narriad to Barbara. They have children callad Kai and Atlax. Atlan is narried to have
child called Josaph. Nichaal sarried Zoe. They What in Luns to kai? Reapond only with the vord that dascribas thair ralationsbip. Anaver: Luns is uarried to Xai, ao Luna is the vife of Kai. unnvarrviiee/aaaaee
They
called Luns.
Exanpla
Qusation: Noak is narried to Barbara. They have children called Aiden and Charles. Charles is sarried to Liss. They have child called liver. kiden is xarried Laey. They have thild called Atlas. They That relation is Nouh to Atlas? Insser with single word.
Natthay
called lacy.
Ansver: Noah has a child called Aiden. Noreaver, Aiden is married to lasy, who togethar have child called Atlax. So Koah is the fachar of Aidan, and Aidan is the father of Atlas Tharafore, Noah ix the grandtather of itlas. aneeerrgranddaate s/mneer>
Exampla 9:
Question: Phoenix narried Anelis, They children called Lecas, Aidez Sophia. Dansel narried
They Novn. They have child called Sebantian. Sophia sarriad to Nowh. what relation is Sebaesian to Lucar? Ansver with single word.
is
Lecas narried
Ansver: Lucas is narried Nova. and together they have child called Sebaetian. So Sebastian in the son of Lucac. <ansver>son caaeeeer
Example 10:
Question: Sebastian narried
They have children called David, Aiden Hannah. Thonar it called karen. is narried Sky, David narried to
narried to Lasa. They Karen. Thay have child called Kasther. Hannah is married to Janes, They have child called Daniel.
What relation is Natthez Karen? Ansver tagether they
single sord,
child called Mastheu.
of Karen. Snewer'son
Exanple 11:
Question: Zioz married Grace. They have children called andrew, Logan and Patricis, Nattheu married to Enily. They have a child called Sophie. Logan is narried to Sova. They have child called Kargaret. Andrev is narvied to Sophie. They have child called Henry. Patricis is aarried Lucas. They
child called Karen. Provide
relationehip in Ansver: Patricia has a child called Karea. Moreover, Andrew and Patricia are riblinge, so Andrew is the uncle of Karen. <ansver ansver>
Androy related
C.3.2 RG-KNIGHTS PROMPT EXAMPLES
Example 1:
Quaxtion: special ixland is inbabited only by haross and villains. Haroez tall the truth, inhabitants:: Benjasin, Bearlatt. Benjanin
and villains alsaya liw. saying, "it Buajasin is hero then Searlett Basjarin is hern", Sa who is haro and who ix villain? (Fornat your ansver 1ike; "Banjanin is bera/villain, and Scarlett is hero/villain")
hero", Scarlett stated, "Bearlett here oz
Accune Denjaxin hern' neana Scarlatt. zuxt be bero. Scarlatt haro, har staterant "Searlatt haro
(tells truth). statesent Benjanin
Scarlett
27

---

Published as a conference paper at ICLR 2026
Beajaxin is hero" is true (both parta are true). This consistent. canscer Beajuuia >eajauri is hero. and Scarlett 10 a hero, </asswer)
Example
Question: very special island is inhabited only by altruists and egoists. Altruieto always tell the
truth, and egoiets alvage li.. You zest 2 ishabitante: Luze, and Riley, In a staterent by Lake: if Riley is an egeist then luke is an altruist". Riley rezarked, "Luke is an egoist if and osly if altruist",
vho Luke altruiat/ugoiat, and Rilay
Bilcy
altruist
egoict? (Forzat your ancher like: " /agooiaa))
Ansver: Assune Lake is an egoist (lies). His statenest "if Biley is ageist thei Luke is altrais:' is falee. For a conditional to be falce, the prezise nust be true and conclusion false. 8o conslatent.
is check Riley's
altruist (lake is agoist), which statexent "Loke is egoist <.> Riley altruist". Bince both Luke and Riley are egoiste, left side is true and right cide i5 false, naking the biconditional faise, vhich is consictent with Riley being egoist who lies. <ansserztuke is an egoist, and Rilay is au </oiat.s/asswe2>
egoist and Luke
Exanpla
Question: vary special island is inhabited only by angela and devila. Augels alvaya tell the truth, and devils alvays 110. You neet 3 inhabitants: Logan, aurora, and Riley. Logan asserted: "Aurera angel then Rilay is devil". Rilay stated, 'Logen devil". Be sho is as asgel and whe is devil? (Forzat your assver like: 'Logan is a angel/ devil, Aurora is angel/devi2, and Biley is a angel/devi1")
angel".
put it, "it Logan
Anaver: Assune Logan is angel (tells truth). Than Aurora is angel (aa Lagan stated). If Aurora is Riley devil" is devil (lies), says "Logan is devil", which false, consistent with lying. <ansver> Logan is an angel, Aurora is an angel, and Rilay is devil. devil.</asaae </asauur>
angel, her
"if Logan angel
Biley is devil. Riley
Exampla
Queation: vary special island is lnhabited only by herowz and villains. Euross alvays iell the truth, and villains alcays lie. You nest inbabitants: Luke, Eenry, and Zoey. "Luke is hero or Benzy is a haro, e lake clained. Henry said that if Zany is villain than Benry is hern. In Zoay: "if Henry villain? (Fornat your ansver like: "Luke is hero/villoin, Henry is hero/villain, and Zoey is a bern/villain")
hero then Luke
villais". Bo vhe
here
is
Anaver: Assune Luke is here (tella truth). His statement "Lkke is hero DR Benry is bero" is true. let Zony villain
here. Hanry's 's since Hanry is a hero. Nov for Zany's 'x staterent. "if Henry haro then Luke villain" - if Zoay is a villein (liea), her atatesant should be false. Biace Eenry is a heto and Luke is not villain, the conditional is false, which consintens with Zony lying. <ansvardlake is hero. Henry
assune Henry
Henry
is
Zony villain. 3/anauer?
Exampla 6:
Quaation:
special island is inhabited only by sages and foola. Sagaz alvays tell the truth, and ishabitante: Alexander, Elizabeth, inelia, Peselope. In
fools always lie. You statasent by Alecandar: "if Inalis is a fool thes inalia is sage". Blizabeth said that Panalops is a sage if and only if Anelia is a fool. "Alasander is a sage and Panslope is a sage" - Anelia.
sage and who is tool? ( Pezelope stased, "Anelia is sage Elizabeth is fool", So sage/fool, Analia sage/faol, , Elizabath
"Klesander feol, and Panelops a aage/fenl")
your
sage/
Ansver: Alezander's 's statement "if Anslia is tool then Amelia is sage" is a contradietion if Anelia 18 tellar, Analia suxt he sage. If Anelia ix are eages. Penelope 's statement
faal, ao for flexander to be her statement "flexander is sage
Pesolope is tage" neans "inelia is sage OR Blizabeth is fool* is true if Panelope 18 sage since Anelia is sage, the OR in trus. Elizabeth 's atatament: "Pannlops is sage <-> Analia is fool - if Panalope is aage and inelis
(zot tosl), the biconditional fales, Elizabeth is fool. <anezer> Alexander i8 ange, Elizabeth is tool, Anelia is aage, Penelope
sage, s/nnwer)
Example 6:
Question: vary special island is inhabited only by heroes and villains. Heroez alvays tall the truth, and rillaine alvays lic. You nest inbabitante: Sophia, Alexander, , Grace, and Liax. Sophia stated, "if Sophia i5 a here then Alezander is a villain', "Brace is rillain if and only it bim 18 here. " Alezander rantioned. "Saphia is a villain if and only if Sophia is a hare , Grace put
and whe villain? (Fornat your ansver like: "Sophia is horo/villatn, Alezander is hero/villain, drace in a hero/villnin, and Lian is hero/villainn')
declared.
*Grace is villain
Lian
here".
who
Ansver: Grace's 's statement "Sophia is villain <-> Sophia ie hero" is a consradietio2, to it's falso, villain. Assuse Sopaia Alesander villain" saans Alaxander is villain. Her statenent "if Sophia ie beto then staterant "Grace Lisr is here' - if Alexszder is a villain (lies), this chould be faice, Since Grace is a villain, for the bicondstional. to be false, Lian nust not be bero (Lian is villain). Lian 's statement Lian villain, this false
neaning Grace
villain
drace villain AND Lian hero" canever>Sopata it here, Alexander villain, drace is willain, and Lian e/ acsver>
consictent.
Example Question:
very special island is inhabited only by angels and and devils alvays lif. You neet 5 inhabitants: Ava, Anelia, Daziel, Mis, and Jack. "Eia is an angel if and only if Jack it a devil,' " Ivs nentiosed. In Axelis's vords: "Dantel is a devil or Mia heard saying, 'Iva angel'. Daniel deril if and only if Mia an angel". heard eaying, "Hia
alvaye Angels
truth,
devil", Jack vho is an angel and who is a devil? (Formst your anszer like: "Ava it a angel/dertl, Anelis ie a angel/deril, Dantel is angel /devil, Ris is a angel/dev1l, and Jack is angol/devi1")
devil". Kis noted, "Jack
Ansver: Jack's statexent "Mis devil <-> Mia angel' is contradiction,
false, zeaning Jack is devil. It Eia is angel, her staterest *Jack devil" true. Iva'e statement 'Mia is aagel <.> Jack is devil* - beth parts are true, se the biconditional is true, neaning Ava is an angel. imelis's 'e statement 'Daniel is devil UR Kia is angel" since Mis is azgel, the DR is true, angel. Daniel'a is devil. <anover
"Ava devil false since Ava is angel, Dasiel angel, Amelia is an aagel, Daniel is a devil, Nia an angel, and Jack is a devil. s/asver>
Anelia
Exanple
28

---

Published as a conference paper at ICLR 2026
Queation: very apecial island is inhabited only by sainta sinzera. Saints alvays tell truth, and sinnors aivays 110, You neet 5 inhabitants: Pezelope, Lizy, Riley, kis, and Aria. "Xia 16 saint and Lily ie cinzer, Penelope clained. Lily noted, "Fenelope is rinner' " Biley assersed: sinser'. Kia noted, "Peneloge Pezelope is saint", Aria expressed that Lily is saint. who catat (Forzat your anever lixe: "Penclope i5 a saint/sizner, Lily is saint/sinaer, 8:ley i5 a 53in7/ sinner, Kia is saint aiine/ and Aria is saint/sizaer")
'if Lily
Nia
sinner if only sirner?
Ansver: Kia's
folse. asaning Kis is sinner. Assuss Lily is saint. Lily's atatenent "Peneloge sinner" zaans Panelope is a sinner. Pezelope's 's statenent "Mia is saint AND Lily is sinner" since Mis is sinner and Lily is saint, this is false, consistent with Pezelope being sinner Riley's etatenent 'I2 sinner"
"Fenelope
Penelope
caint"
contradiction,
false premise (Lily sains), making vacuously starezont "Lily is saint" trse, so Aria cains. <ansver> Pezelopo is a sinner, Lily is saint, Rilcy is saint, Kia is sinner, and Aria is soint.</ ac2w02?
then
so Riley is saint.
Exampla
Question: vary special island in inhabited only by angela and devila. Augels alvaya tell the truth, and devils alvays 11e. You neet 6 inhabitanta: Lisz, Zeey, Ava, Logaz, Janes, and Avery. Liaz only Ava devil'. "Avery is devil," Logan nentioned. Janes said that if Avezy is an angel then 200y is an angel. Avery asserted: "Avery is an angel and Lias is devil". So who is angel and who is devil? (Fornat your ansver like: "Lass is angel/ is angel/
"Jana Ava's vorde: 'Lias
davil
devil". Zany
'Zoey is angel
Logan devil".
devil, Zoey is angel /devil,
Logan
angel/davil,
is angel/
and Avery is a angel/devil")
Ansver: Assune Ava ia an angel. For atatement "Lien is devil" sansa Lias is a davil. Lian's ataterent
Janes is devil <.> Ava is devil" if Lias is a devil (lies), since Ava is angel (not devil), for be false. sidea fulaw, thua true). Wait, lst se reconsider.
davil (making lies, the biconditional is fulse. Ava angel (not devil), ao for false biconditional. Janes aust be devil. Assune Avery is an angel. Avery statenent "Avery is angal AND Lian is devil" true. Zony's 's stasement 'Zoay is angel AND devil. logan's
biconditional
alas
davil,
Logan devil* devil. James's statement 'it Avery angel then Zoey is angel" siace Avery angal and Zony is davil, this in falas, so Jares is devil, sanswer is a davil, Zony is devil, Ave is angel, Logan is devil, Jases is devil, and Avery is un angel. </anaver>
false, Zowy
"Avery is devil"
false,
so Logan
Exanpla 10:
Queation: vary special island in inhabited only by knights and knaves. Raighta alvays tell the truth, and knavas alvays lin. You nent 8 inhabitantx: Aris, Ava, Aralia, Grace, Charlatta, and Jack. "Jack
knight", Anslia knight", Grace consented, 'Iria is knight and only if Charlotte a inave". ls Charlatte pet it, "Aria is a knight". Jack nated, *Ava is a knave if and onl if Charlatte is knave". 3o who is a knight and who is knave? (Fornat knight/kkaave,
Jack knight knave
Grace clained.
"Jack
anaver like: 'Aria knight//nnrre Charloste
might//aave, iva knight//nave,, inelia is knight/knave, and Jack is a knight//tnav"")
Anaver: Assune kria is knight. Bar atatement "Jack is knight" saasa Jack is a knight. Ava's stalesant Ava is hnight. Charlotte 's statement "Iria hnight" is knight. Jack's statenest both sides are false, zaking the biconditional true, consintent with Jack being a knight. Grace's statenent "Aris is knight <-> Charleste is knave* krin is knight and Charlotte is knight (sot knave),
JJack knight"
so
Charlatte
Charlatte are knights (not knavez),
the bicanditionsl is false, naaning Grace is knight"
ie knave. Anelia's etatenent "Jack ksave mave, so this 38 false, neaning Arelia 18 knave. Cansver>Aria 18 a knight, , Ava ie a knight, Anelia ia is knave, Grace is a knave, Sharlotte is a knight, and Jack is a tnight.</annver. </anzver>
is knight
is
LLM USE
LLMs were used to revise and proofread paper content. All claims have been verified and cross- referenced by the authors.
29


Reviews:
[{"review_id": "ewBuKOpFxL", "reviewer": "Reviewer_hJzK", "summary": "The paper shows that RL fine-tuning (GRPO) on purely synthetic multi-hop datasets (GSM-infinity, PhantomWiki) improves LLM performance on real-world QA benchmarks (HotpotQA, 2Wiki, MuSiQue) by teaching a transferable “knowledge composition”.", "questions": "-  In the abstract, there's this phrase \"“high scoring latency” which is not clear to me. Can you please explain?\n\n- Is there any reason to prefer RL over SFT in this setup?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Provides clear empirical evidence that RL fine-tuning on synthetic datasets (PhantomWiki, GSM-infinity) improves LLM multi-hop reasoning on real-world QA benchmarks.\n\n- Addresses a practical problem, the scarcity and cost of high-quality human annotated, which the paper suggests can be supplemented or replaced by synthetic reasoning data.\n\n- Demonstrates consistent performance gains across multiple model families and parameter scales, indicating robustness and generalizability.\n\n- The experimental details and use of open-source technologies (models and codebase) makes the setup reproducible.", "weaknesses": "- The paper’s novelty is limited as prior works have already shown that synthetic data and SFT/RLVR for reasoning works quite well. The contribution is primarily about a different reasoning setup of multi-hopping. \n\n- The domain of synthetic data is narrow, focusing only on arithmetic and relational reasoning, which limits claims of general reasoning transfer.\n\n- The evaluation datasets lack diversity. HotpotQA, 2WikiMultihopQA, and MuSiQue are all two-hop or near two-hop QA tasks, reducing the strength of the generalization claim.\n\n- The paper is unnecessarily verbose at places. The related work section covers too much ground that's not relevant. Stating GRPO equations in Section 3.2 was not really necessary.\n\nMinor nit:\n- reinforcement -> Reinforcement\n- L128: “complicated, RL-based framework” -> why complicated?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nFCyP6L0vI", "reviewer": "Reviewer_3JJe", "summary": "This paper examines whether large language models can acquire general reasoning abilities solely from synthetic data, without relying on real-world knowledge.\nUsing reinforcement learning on fully artificial datasets such as PhantomWiki and GSM, the authors show significant performance improvements on real-world multi-hop QA benchmarks.\nThe results suggest that reasoning skills, such as knowledge composition, can transfer across domains, highlighting synthetic data as a scalable alternative to human-annotated reasoning datasets.", "questions": "None", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The study demonstrates that large language models can acquire generalizable reasoning skills purely from synthetic, knowledge-free data. It provides empirical evidence that these synthetic reasoning abilities transfer to real-world multi-hop QA tasks, achieving substantial performance gains. The approach offers a scalable and cost-effective framework for improving reasoning through verifiable, automatically generated training data.", "weaknesses": "- The experiments are limited to multi-hop QA. Even though the training data come from a synthetic world, the fact that performance improves on other multi-hop QA benchmarks is not particularly surprising.\n- The applicability of the approach to grammatically or semantically complex real-world texts remains unknown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tLLpiulmNf", "reviewer": "Reviewer_F2Sn", "summary": "The paper investigates whether RL fine-tuning on purely synthetic data constructed to require multi-hop reasoning but to contain no real-world facts can teach a general skill of knowledge composition that then transfers to established, real benchmarks. Concretely, the authors fine-tune small LLMs  with GRPO on two synthetic sources. Experiments show that RL on synthetic data improves performance on all three downstream datasets and across model families and sizes.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* I appreciate that the paper presents a tight experimental design around a single, interpretable hypothesis: i.e., training on universes that are explicitly non-overlapping with real-world knowledge, the study isolates whether multi-hop structure learned in synthetic settings can carry over.\n\n* The paper also tackles an important topic in reasoning/RL, namely the effort to disentangle answer formatting from reasoning.\n\n* I also like how the paper shows curves over checkpoints and stratifying performance by question difficulty (number of hops in PhantomWiki and number of operations in GSM infinite), which provides a richer picture than just showing single endpoint scores.", "weaknesses": "* Although the empirical story is neatly organized, in my view, the novelty is modest given the rapidly expanding literature on synthetic data and RL for reasoning.\n  * If I remember correctly, PhantomWiki itself was introduced as an on-demand synthetic universe generator to test reasoning and retrieval while sidestepping data leakage; it feels more like this paper leverages that dataset rather than advancing the generation framework. Likewise, GSM-infinite was created to probe reasoning under controllable arithmetic complexity and long contexts, and here it serves as a training curriculum rather than as a novel contribution.\n  * Similarly from the methodology side, the paper’s RL component employs GRPO but it seems like without much methodological innovation; recent works (a la Deepseek) have shown to a degree that RL alone can elicit sophisticated reasoning behaviors without human step-by-step traces and RLVR can better incentivize process-correctness. The present study feels like a combination and transfer evaluation rather than as a new algorithm etc. The finding of using synthetic data to help with reasoning doesn't seem particular novel to me either.\n\n* If I understand correctly, one of the paper's claims is that knowledge composition is the specific causal skill that helps with performance; in terms of evidence, the paper infers this composition skill mainly from higher F1 values on 2-4 hop datasets and from difficulty-stratified curves; however, I don't see where the paper verifies whether the intermediate steps followed by the model are logically correct and path-faithful?\n\n* While the focus is on RL, I don't see any other baselines (e.g., SFT) on the same synthetic datasets: how do those compare? Without some of these comparisons, it's unclear to me if gains are from RL itself or via additional supervised exposure to reasoning-style data (or something else). The models used seem to also all be sub 4B; I'm not sure how transferable these findings are in generality (e.g., to even 7/8B models or eve 3-4B modes outside of phi-mini) especially since results show that different models show different degrees \"malleability\" to RL (but the paper defers analysis to future work). \n\n* I see that the evaluations are only on 3 datasets (HotpotQA, 2Wiki, MuSiQue); furthermore, results are done on sub-samples ($n=500$ with two seeds if I understand correctly; there is no re-sampling across multiple draws or paired testing. Combined with RL and small models (with potentially high variance), this seems quite small. I wonder how robust the gains shown here are to actual larger samples, increased repeated rates of sampling, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fPp0OvT9lQ", "reviewer": "Reviewer_hyPz", "summary": "This paper investigates an important and timely question: whether fine-tuning Large Language Models (LLMs) on synthetic data through reinforcement learning, in the absence of real-world knowledge, can improve their multi-hop reasoning capabilities on real-world question-answering (QA) tasks. The authors use two synthetic datasets: GSM-∞ (math word problems) and PhantomWiki (fictional knowledge-base QA), to fine-tune several open-source models of varying scales (Qwen series, Phi-4-mini-reasoning). The experimental results show that despite the synthetic data having no factual overlap with the real-world evaluation benchmarks (such as HotpotQA, 2WikiMultihopQA, and MuSiQue), the fine-tuned models achieve significant performance improvements on these real-world tasks. The authors argue that this improvement stems from the model learning a transferable meta-skill—\"knowledge composition,\" the ability to chain multiple logical inference steps. The study also finds that this performance gain is consistent across different model families and sizes, and does not suffer from overfitting as training data increases.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. As high-quality human-annotated data becomes increasingly scarce, exploring the value of synthetic data is a frontier direction in the LLM field. This paper explores the fundamental question of whether reasoning abilities can be learned independently of factual knowledge, which has significant implications for the training strategies of Large Language Models (LLMs).\n2. The use of completely disjoint synthetic and real-world datasets effectively controls for memorization, providing clearer evidence for skill transfer.\n3. The paper not only reports final results but also provides an in-depth analysis of the model's learning process by examining performance changes during training and stratifying performance by question difficulty. In particular, Figure 3 and Figure 5 clearly demonstrate how the model's progress on more difficult synthetic questions translates to performance improvements on real-world tasks.\n4. The paper is well-structured and flows logically from introduction to conclusion. The research motivation, methodology, and results are all clearly articulated. The figures (especially Figure 2 and Figure 5) intuitively present the core findings and are easy to understand.", "weaknesses": "1. The training is conducted exclusively on synthetic data. Although this shows improvement in real-world scenarios, the paper lacks a comparison with a baseline trained on real-world data and tested on real-world data. There is no analysis of the potential performance gap between training on synthetic data versus real-world data.\n2. The experimental models are relatively small, with the largest being 4B. It would be beneficial to see experiments on models of at least 7B parameters. Based on Figures 2 and 3, the performance improvement for Qwen3-1.7B is notably smaller than that for Qwen3-0.6B. Could this be because the 0.6B model has weaker baseline reasoning abilities, and thus training naturally improves its generalizable reasoning performance, while the larger 1.7B model shows diminished gains? What would happen with an even larger model, such as a Qwen3-8B? Would the improvement be minimal?\n3. The paper states that 3 (for GSM-∞) or 11 (for PhantomWiki) CoT examples are used during RL training. Is it necessary to include these CoT examples in the prompt during inference as well? Furthermore, it would be desirable to see experiments on the training and generalization effects in a Zero-Shot setting, i.e., without including any CoT examples.\n4. In Figure 2, the training results on GSM-∞ are consistently worse than on PhantomWiki for nearly all models. Is there a deeper explanation for this? Is it because GSM-∞ focuses more on mathematical reasoning rather than the type of multi-hop reasoning found in the evaluation tasks? If so, does this raise concerns about the generalizability of the reasoning skills being learned?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)