SynthWorlds: Controlled Parallel Worlds for Disentangling Reasoning and Knowledge in Language Models
Authors: Ken Gu
Advait Bhat
Mike A Merrill
Robert West
Xin Liu
Daniel McDuff
Tim Althoff
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Reasoning
Agents
Controlled Evaluation
RAG
Primary area: datasets and benchmarks
TL;DR: We introduce a framework and benchmark to disentangle task reasoning from parametric knowledge in LLMs.

Abstract: Evaluating the reasoning ability of language models (LMs) is complicated by their extensive parametric world knowledge, where benchmark performance often reflects factual recall rather than genuine reasoning. Existing datasets and approaches (e.g., temporal filtering, paraphrasing, adversarial substitution) cannot cleanly separate the two. We present SynthWorlds, a framework that disentangles task reasoning complexity from factual knowledge. In SynthWorlds, we construct parallel corpora representing two worlds with identical interconnected structure: a real-mapped world, where models may exploit parametric knowledge, and a synthetic-mapped world, where such knowledge is meaningless. On top of these corpora, we design two mirrored tasks as case studies: multi-hop question answering and page navigation, which maintain equal reasoning difficulty across worlds. Experiments in parametric-only (e.g., closed-book QA) and knowledge-augmented (e.g., retrieval-augmented) LM settings reveal a persistent *knowledge advantage gap*, defined as the performance boost models gain from memorized parametric world knowledge. Knowledge acquisition and integration mechanisms reduce but do not eliminate this gap, highlighting opportunities for system improvements. Fully automatic and scalable, SynthWorlds provides a controlled environment for evaluating LMs in ways that were previously challenging, enabling precise and testable comparisons of reasoning and memorization.

Published as a conference paper at ICLR 2026
SYNTHWORLDS: CONTROLLED PARALLEL WORLDS FOR DISENTANGLING REASONING AND KNOWLEDGE IN LANGUAGE MODELS
Ken Gu Advait Bhat', Mike A. Merrill, Robert West
Xin Liu Daniel McDuff!, Tim Althoff
University of Washington, Stanford University, "EPFL, *Google Research
https://githui: http:///....aaoooaaaaaa///////ta
https: https://huggiingfaet http://www../oaaaatee///////////
ABSTRACT
Evaluating the reasoning ability of language models (LMs) is complicated by their extensive parametric world knowledge, where benchmark performance of- ten reflects factual recall rather than genuine reasoning. Existing datasets and approaches (e.g., temporal filtering, paraphrasing, adversarial substitution) cannot cleanly separate the two. We present SYNTHWORLDS, framework that disentan- gles task reasoning complexity from factual knowledge. In SYNTH WORLDS, we construct parallel corpora representing two worlds with identical interconnected structure: a mapped world, where models may exploit parametric knowledge, and synthetic-mapped world, where such knowledge is meaningless. On top of these corpora, we design two mirrored tasks as case studies: multi-hop question answering and page navigation, which maintain equal reasoning difficulty across worlds. Experiments in parametric-only (e.g.. closed- -oosed--ookkk QA) and knowledge- augmented (e.g., retrieval- -augeeete) LM settings reveal a persistent knowledge advantage gap, defined as the performance boost models gain from memorized parametric world knowledge, Knowledge acquisition and integration mechanisms reduce but do not eliminate this gap. highlighting opportunities for system im- provements. Fully automatic and scalable, SYNTHWORLDS provides a controlled environment for evaluating LMs in ways that were previously challenging, en- abling precise and testable comparisons of reasoning and memorization.
INTRODUCTION
Language model (LM) agents are increasingly expected to autonomously complete complex tasks that require retrieve new information, reason over it, and synthesize novel insights. These capabil- ities underpin emerging applications such as web navigation, where agents need to traverse linked information to locate relevant content (Ning et al., 2025); personal health insights, where they must connect medical data with external resources to inform advice (Heydari et al., 2025); and scien- tific discovery, where it is necessary to integrate findings scattered across research articles to form new hypotheses ( (Yamada et al., 2025). Success in these settings requires operating over richly struc- tured knowledge environments, navigating interlinked documents, resolving indirect references, and integrating evidence spread across multiple sources.
Yet, LMs continue be trained on massive web corpora (often with undisclosed training data), it remains unclear to what extent their performance reflects genuine reasoning versus the reciting of memorized knowledge (Carlini al., 2023; Wu al., 2024). Many benchmark tasks depend on fac- tual world knowledge models likely encountered during training (Sainz et al., 2023; Xu et al., 2024b; Zhou et al., 2023). This undermines two goals: scientifically, prevents isolating reasoning ability (i.e.. functional linguistic competence) from memorization (i.e., formal linguistic competence) (Ma- howald et al., 2024; Lu et al., 2024); practically, it limits confidence in deploying systems to novel environments (i.e., scientific discovery).
"Correspondence kenggu @ es. washington.edu

---

Published as a conference paper at ICLR 2026
To distinguish reasoning from reciting, re- searchers have explored several strategies. One approach is manual curation of "clean" evalua- tion sets, which provides novelty but is costly, difficult to scale, and requires continual updates. For example, ToolQA (Zhuang et al., 2023), a benchmark released in 2023 to distinguish be- tween questions answerable from an LM's inter- nal knowledge and those requiring external in- formation, included GSM8K questions derived from "error cuses mude by ChatGPT" at the time. However, subsequent work has shown that newer LMs may already memorize many of these answers (Zhang al., 2024; Mirzadch et al., 2025). Another approach, synthetic dataset gen- eration (Huang al., 2025; Hsieh et al., 2024), promises scalability, but often involves using ex- isting content directly (e.g.. novels) and thereby results in parametric knowledge leakage or relies on overly simplistic templates (e.g.. "The job of David is farmer. The hobby of David is bird- watching." "), limiting their ability to probe reason- ing in realistic, richly interconnected settings.
Multi-hop QA
GPT-5-mini Performance
Q: Who is the namesake of the city where Ryan Reynolds born?
100
KA RM RM-SM
80
RM
A: George Vancouver
Score
60
SM
KA-21
Q: Who is the namesake the city where Derek Veylan was born?
20
KA-16
A: Altheon Metronis
Closed-
book
RAG
Page Navigation
Navigate from Geoffrey Hinton to Ryan Reyonids
Success
20
KA-22
KA-31
Navigate from Caleb Ardent to Derek Veylan
Links only Content
Links
Figure 1: Controlled experiments from SYN- TH WORLDS corpora. We measure the knowl- edge advantage gap (KA) as the performance difference between parallel tasks mapped to real-world (RM) and synthetic (SM) entities. Retrieval and page content boosts performance but the gap persists.
Crucially, evaluations based only on synthetic un- seen tasks still leave open questions about performance. Success demonstrates reasoning in isola- tion, but does not reveal how much models typically rely on prior knowledge as a scaffold. Failure, on the other hand, is ambiguous: the reasoning chain underlying the task may be too difficult for models to succeed, or the model may simply lack the background knowledge it usually exploits. Without controlling both task difficulty and requirements for parametric knowledge, such evalua- tions leave the contributions of reasoning and memorization entangled.
To address this, introduce SYNTH WORLDS, framework for disentangling reasoning from fac- tual knowledge. Parallel synthetic corpora are constructed to represent different worlds that replicate the structure and complexity of real-world information ecosystems. One corpus is mapped to real- world entities (e.g. Geoffrey Hinton), while the other is mapped to synthetic entities (e.g. Caleb Ardent). thereby obscuring the usefulness of parametric knowledge. This design allows us to quan- tify the knowledge advantage gap (i.e., the performance difference between real real-mapped [RM] and synthetic-mappe [SM] settings) and to evaluate how knowledge acquisition methods (e.g., provid- ing page content, retrieval-augmentt -agmmented generation) and integration strategies (e.g., chain-of-thouu thoth prompting, agentic reasoning) impact this gap (Fig. 1). The gap clarifies to what extent models rely on reasoning versus recall, and whether augmentation substitutes for or amplifies prior knowledge.
To support comparisons at scale, SYNTHWORL.D WORLDS automatically generates parallel corpora from triplet facts in knowledge graph (83). To obscure factual knowledge, entities are renamed with murface-form-consiiter transformations that preserve both type and name-derivation consistency before rendering facts into documents (Agarwal et al., 2021; Josifoski et al., 2023). Specifically, people receive person names (Geoffrey Hinton - Caleb Ardent), cities receive city names (Toronto + Metrovale), and derived names maintain consistency (University of Toronto - University of Metrovale, not University of Grandvale). This process yields corpora with identical reasoning struc- tures while removing familiarity with entity-specific facts, resulting in coherent worlds where tasks require reasoning over complex documents under controlled relevance of parametric knowledge.
To demonstrate the utility of our SYNTHW ORLDS framework, we generate two parallel corpora de- rived from Wikidata: SYNTHWORLD-RR and SYNTHWORLD-SHN (84). On top of each corpus, we construct two ceasoning-intensii
tasks as case studies: multi-hop question answering (QA) (Trivedi et al., 2022; Ho et al., 2020) and page navigation (West& & Leskovec, 2012) with fine-grained control over difficulty.
In our experiments, we evaluate LMs on these tasks to quantify the knowledge advantage gap, first in settings where models rely only on parametric knowledge (closed-book QA for multi-hop reason-
2

---

Published as a conference paper at ICLR 2026
ing and page names only for navigation), and then under conditions where knowledge augmentation (retrieval for QA, access to page contents for navigation) and integration strategies (e.g., chain-of- thought prompting)
provided ($5). Across both tasks, find clear performance gaps between real-mapped and synthetic- settings. While knowledge integration improves performance in both cases (and in some instances narrows the gap), the gap persists. This persistence highlights opportunities for future work to design more effective knowledge integration schemes and to sys- tematically study system behavior when models encounter novel environments ($6). We contribute:
1. A scalable framework for generating rich, interconnected corpora and tasks that disentangle and task reasoning difficulty from parametric knowledge.
2. Two parallel corpora with corresponding task datasets. We instantiate the SYNTH- WORLDS framework with WYNTHWORLD-AAR and SYNTHWOLLD-SD paired at the document, fact, and task levels to enable controlled evaluation. Each corpus contains 6,920 documents ering 161K facts, along with 1.2K multi-hop QA and IK page navigation instances. To support future research, we release these resources publicly.
3. An empirical analysis of LMs across parametric-only and knowledge-augmented settings using our parallel datasets to quantify the knowledge advantage gap, which prior setups do not fully isolate. Our analysis reveals persistent shortcomings even with knowledge augmentation.
2 RELATED WORK
Human Curated Data for Reasoning Evaluation. As LM capabilities continue to improve and be- come widely deployed, researchers have relied on manually curated benchmarks to evaluate reason- ing in settings not already covered by training data (Kazemi et al., 2025; Wei et al., 2025; Hendrycks et al., 2021; Cobbe et al., ; Bean et al., 2024; Srivastava et al., 2023; Tang & Yang. 2024; SU al., 2025). These benchmarks are effective when first released but grow less informative over time as time passes. For example, MuSiQue (Trivedi et al., 2022), released in 2021 as a multi-hop QA benchmark, was originally designed contain questions that models could not answer without the reference text. Despite this intent, it is still used across many evaluations today (Li et al., 2024; Zhang et al., 2025; Gutiérrez et al., 2025), even though current LMs (e.g., L.lama-333770B) achieve over 26% F1 score these questions without any documents (Gutiérrez et al., 2025). This makes it difficult to assess whether improved performance reflects genuine advances reasoning and re- trieval capabilities that would be informative of systems deployed in unseen environments. As result, researchers must continually spend effort to construct new datasets and tasks (Gu et al., 2024; Tang& & Yang, 2024; Monteiro et al., 2024; Bai et al., 2025). These efforts require substantial exper- tise, grow increasingly complex as models advance, and are slow and costly to scale. In contrast, SYNTHWORLDS introduces a scalable framework to construet complex text data and associated reasoning tasks, reducing the manual curation burden while maintaining evaluation quality.
Synthetic/Perturbee Data for Reasoning Generalization. Given the resources needed to build high-quality human-generated data, researchers have developed methods to compose synthetic data or introduce perturbations to evaluate the reasoning generalization of LMs (Huang et al., 2025; Wu et al., 2024; Levy et al., 2024; Hsieh et al., 2024; Gu et al., 2025). These approaches reveal im- portant weaknesses when LMs are tested outside familiar conditions over long contexts, but they do not disentangle reasoning ability from reliance on parametric factual knowledge. Other efforts address this separation more directly, for example by focusing on real-time factual updates (Kasai et al.. 2023; Vu et al., 2024) or by generating synthetic text (Gong et al., 2025; Allen-Zhu & Li, 2024: Monea et al., 2024). However, such work typically targets narrow aspects of knowledge or simplifies away the complexity and interconnectedness of real-world corpora, making it difficult to generalize findings to realistic scenarios (e.g., web navigation). Our work complements these lines by isolating the independent impacts of LM reasoning and parametric factual knowledge task performance. Through controllable parallel dataset construction, we enable precise measurement of the knowledge advantage gap across common LM settings (e.g., in-context learning, RAG, agentic workflows). analysis on how different forms of knowledge augmentation influence this gap, and direct comparisons of model behavior in novel versus familiar settings.
3

---

Published as a conference paper at ICLR 2026
(a) Sample Connected KG Subgraph
(b) Rename Entities
Vancouvver
Ryan Reynolds
Derek Veylan
EID1
Vancouver
Metronis Tessarion Altheon Metronis 1925
Ryan
Canada George Vancouver
Reynolds
Canada
1886
EID3
born in
named after
incor ioorperated
Instance of
country
EID2
(c) Generate Parallel Corpora
Metronis
[Metroniss]( EID1 (incorporated
Vancouver Eineeporraeed in 1886) big city in Canada named after George Vancouver...
Encerporated in 1925) is big city in Dravara named after Altheon Metronis...
George Vancouver EID4
big city EIDS
city in [Dravara][[ EID2
1925)
1886
named after [Altheon Metroniss( EID4 12..
Figure 2: Overview of WORLDS Corpora Construction (Toy Example). A connected subgraph is sampled from large knowledge base (a). To obseure factual knowledge, entity labels are renamed from real-world labels (real-mapped) to synthetic name (synth-mapped) (b). From synth-mapped triplets, we generate synth-mapped documents. These documents are converted to real-mapped documents through additional LM steps with symbolic references (c). The final output two parallel corpora: one real-mapped, one synth-mapped. Using the corpora, we construct parallel reasoning tasks (64.1).
SYNTHWORLDS: PARALLEL CORPORA FOR CONTROLLED EVALUATION
The main idea of SYNTHWORI.N WORLDS is to construct parallel corpora and tasks that deseribe two worlds: one grounded in real-world entities, where factual knowledge encoded in language models' parame- ters is potentially useful, and another built from synthetic entities, where such knowledge is deliber- ately uninformative. We define factual knowledge entity-specific world knowledge tied to named entities (e.g., "Barack Obama served as U.S. President from 2009 to 2017"). In contrast, domain- general knowledge is not tied to named entities (e.g., arithmetic, physical laws, the concept of an election or a university). This distinction ensures that tasks maintain equivalent reasoning demands while preventing solutions that rely solely recalling memorized entity facts. The reasoning pre- served includes commonsense (e.g.. hospitals have doctors), compositional (e.g.. a university has a medical school and medical schools train doctors, then the university trains doctors), logical (c.g.. the parent of the parent of X is X's grandparent), and temporal reasoning.
Quantifying Parametric Knowledge in Reasoning Tasks. Constructing parallel corpora and tasks enables us to formally quantify the contribution of parametric knowledge. For task, let PR denote performance on the corpus with real-world entities (where parametric knowledge is useful), and Ps denote performance on the corpus with synthetic entities (where parametric knowledge is uninfor- mative). We define the knowledge advantage gap as KA = PR - Ps. quantifying the contribution of parametric knowledge to task performance. We further distinguish between two settings: the baseline case, where models rely only their parametric knowledge (KAA base phase phase), and the augmented case, where models are provided with external knowledge acquisition and integration strategies (K.A ext Pext ) Examples include RAG for multi-hop QA and reading page con- tent during agentic page navigation, though agents may employ other exploration and knowledge integration strategies in novel environments. In the baseline setting, Phise is expected to be near random (e.g., 0% accuracy for multi-hop QA; random walk performance for page navigation) since parametric knowledge is uninformative, so KA base reflects the pure contribution of parametric mem- ory. Additionally, with KAA base KA we quantify how much the knowledge advantage closes when allowing external knowledge integration.
Framework Goals. To fairly measure KA, SYNTHWORLDS corpora and tasks are constructed with four core goals: (1) emulate real-world complexity by capturing the structure, interconnections, and both factual consistency (facts are mutually coherent) and semantic consistency, where seman- tic consistency requires that surface forms remain compatible with the entity's 's ontological type. For example, university names remain university- (University of Toronto + University of Grand- vale, not - Grandvale Bank) and libraries remain library-like (Central Library ~ Oakwood Public
'Practically, we define named entities proper nouns (i.e., capitalized) in common usage (Wikidata contrib- utors, 2025) recognized by NER models (e.g., the common noun actor vs. the named entity Ryan Reynolds).
4

---

Published as a conference paper at ICLR 2026
Library, not - Central Stadium). This ensures surface form artifacts do not differ between real and synthetic variants, making performance on SYNTHWORLDS informative reasoning in realistic tasks; (2) enable parallel real- and synthetic-entity variants to disentangle reasoning and factual knowledge; (3) precisely control task difficulty to support observations across levels of task com- plexity; and (4) be fully automatic such that new SYNTHWORLDD corpora and tasks be readily constructed to continually provide novel evaluation data (guarding against evaluation corpora being included pre- and post-training datasets).
Obscuring Factual Knowledge in Synthetically Generated Corpora. Similar Wikipedia documents, SYNTHWORLDS' corpora consist of documents about a specific entity with references other entities in the corpus. The pipeline operates in three stages (Fig 2): (1) universe construction, (2) surface-ford form perturbation of named entities and timestamps, and (3) document generation.
First, to ensure the world factually consistent, the pipeline samples universe of connected triplet facts (i.e., subject - relation + object) from an existing (and assumed to be consistent) knowledge base (Fig 2a). Next, to remove parametric knowledge while maintaining consistency, entities are systematically renamed while preserving type information and context (e.g.. ensuring that the re- name of Vancouver is still city named after George Vancouver the person) (Fig 2b). Finally, based on the synth-mapped facts (using the knowledge graph structure and new synthetic names), we gen- erate documents using LMs, following prior work on generating documents from knowledge graph facts(Fig 2c) (Agarwal et al., 2021; Josifoski et al., 2023). Specifically, we first generate documents in the synth-mapped universe consistent with the triplets. We then insert symbolic references to en- tities in the text. Finally, we map these references real-mapped labels, converting each synthetic document into its map-ma counterpart.
The pipeline outputs two parallel corpora derived from shared set of knowledge graph triplets: one mapped to real-world entities and the other to synthetic entities. Both corpora preserve identical sentence structures and world-consistent facts, differing only in their surface-form labels. For space, we include details of SYNTHWORLDS generation framework in Appendix A.
4 SYNTHWORLDS--R AND SYNTHWORLDS- -M CORPORA AND TASKS
Using the SYNTHWORLDS framework ($3; Appendix A), we construct two parallel corpora and tasks: SYNTHWORLD-RE consisting of real-mapped entities and SYNTHWORLD-SS containing synthetic named entities. For space, we include dataset construction details Appendix which instantiates the framework Wikidata. Our specific Wikidata pipeline can cheaply generate new datasets through natural stochasticity (sampling of renames) and by varying hyperparameters (e.g., entity types, seed nodes, knowledge graph sampling procedures), preventing SYNTHWORLDS from being overfit in evaluations.
Pages Tokens
Facts Entity Types
Relation Types Avg Degree
Density Mhop QA
Nay Pairs
6,290 ~.55 161K
956
354
14.6
0.23%
1.2K
1K
Table 1: Summary Statistics for SYNTHWORLD-RM and SYNTHWORLD-SM.
Dataset Statistics. Table 1 summarizes our dataset. SYNTIWORLD-RMISS each contain 6290 documents and over 1.5M tokens in total. The hyperlink graph is sparse, with an edge density of 0.23%. Its degree distribution is heavy-tailed: most pages have only a few links, while small num- ber act as hubs with disproportionately many incoming or outgoing connections. Both characteristics mirror the structure of real-world information networks such as the Web or Wikipedia (Adamic & Huberman, 2000; Kumar et al., 2000). Additional figures/tables (including cost of constructing our datasets) and qualitative examples of the dataset are provided in Appendix B.7 and B.8.
4.1 CASE STUDIES: PARALLEL TASKS WITH CONTROLLABLE DIFFICULTY
Given SYNTHWORLD-RMEN corpora, we construct two tasks as case studies to evaluate LM reasoning: multi-hop QA and page navigation.
5

---

Published as a conference paper at ICLR 2026
a) Sample Subgraph with Constraints
b) Generate Single-hop Questions
c) Generate Multi-hop Questions
Ryan Reynolds
Vancouver
George Vancouver
1. Where was Derek Veylan born? Metronis
Who is the namesake of the city where
Derek Veylan was born? Altheon Metronis
born in
country
Metronis
2. Who is the city of Metronis named after? Altheon Metronis
d) Map Parallel Questions
Derek Veylan
Altheon Metronis
Who is the namesake of the city where Ryan
Reynolds born? George Vancouver
Figure 3: Multi-hop QA Construction. Subgraphs matching reasoning motifs are sampled with constraints to ensure uniqueness, diversity, and multi-hop reasoning (a). From their triplet facts, we generate maphhd single-hop questions (b), which are composed into synth-mappes multi- hop question (c). Using the synth-to-real entity mapping, we replace synth names with real names (d). The final output parallel sets of real-mapped and synth-mapped multi-hop questions.
Multi-hop QA. Multi- hop questions are questions which require reasoning across multiple sources of evidence (Fig 3). For constructing these questions, we follow MuSiQue (Trivedi et al., 2022) and construct multi-hop questions through single-hop question composition (Fig. 3b). We build each multi-hop question using a specific graph motif composed of triplets, where each triplet corresponds to one single-hop question that can be composed into the final multi-hop question. This graph motif' indicates a specific multi-hop reasoning structure. Table summarizes all motif's used in our dataset.
Specifically, given the facts used to generate the synth- documents, we first construct a global fact graph Gfacts where nodes represent entities and edges represent facts, with each fact annotated by the page where it occurs. The fact graph structure Gfacts is identical for both the synth-mapped and real-mapped corpora. From this graph, we sample subgraphs S C Gfacts that match desired reasoning motifs, ensuring that each reasoning step draws from different page.
Next, we use an LM to generate a single-hop question for each unique triplet (u, (u, E S, where u and v denote the subject and object entities, respectively, and denotes the relation between them. We start with the synth-mapped entities to generate single-hop questions. For automatic quality validation, we verify that the subject entity mentioned in the corresponding question. We prompt a LM to compose a multi-hop question from the single-hop questions. We ensure that root entities in the subgraph are mentioned in the question while all bridge entities (non-root and non- non-leaf) are not mentioned in the question text. Finally, to create a parallel task, we remap the entity names in both the question and answer.
This approach allows us to control task difficulty through different reasoning motifs while main- taining task parallelism by using the same sampled subgraph across both corpora. Specifically, difficulty is determined by two factors: (1) the number of hops (or equivalently, the number of de- composed single-hop questions needed to answer the question), and (2) the structural complexity of the question's i motif pattern. Table 3 illustrates examples of reasoning motifs and resulting ques- tions. In our dataset, motifs range from 2 decomposed questions (motif A) to 4 (motifs D, E, F). Motifs that contain others subgraphs are strictly more difficult (e.g., D B A and F > C, E> C) since they require an additional reasoning hop. Additional details on multi-hop QA construction and ensuring task quality and diversity are included in Appendix B.4 with prompts in B.
Page Navigation. In page navigation, an agent is asked to navigate from a source to target page (e.g., navigate from Geoffrey Hinton Ryan Reynolds) using only the hyperlinks on the page. This task is broadly related to web navigation and agentic reasoning. At each page, agents must formulate hypotheses (e.g.. "the link University of Toronto might lead closer to Ryan Reynolds since both are Canadian") "). evaluate alternative decisions, and integrate information learned from prior steps (Yao et al., 2023: Wang et al., 2025). Pages that are more difficult to navigate (i.e., requiring more steps and presenting more choices at each step) further increase the demands on reasoning.
We treat the symbolic references created during document generation as hyperlinks to other pages. From this, we construct a document graph Gdoc (Vaoc, Edoc) where nodes Vaoe are documents centered around specific entities and edges (u, v) E Ease indicate a hyperlink from document u to document v. Note that this graph structure is identical for both the synth- and real-mapped corpora, preserving task parallelism. Creating a page navigation task simply requires specifying a source and target page. To measure and control for difficulty, distance (i.c., expected number of steps for a random walk) between two nodes as a proxy for task difficulty and sample node pairs according different distance buckets (Chandra et al., 1989).
the expected random walk
6

---

Published as a conference paper at ICLR 2026
Higher values indicate that more intermediate decisions are required each step, the agent must navigate through a longer chain of choices to reach the goal.
Task Statistics. In total, we construct 1,200 parallel multi-hop questions spanning six reasoning structures, as well as 1,000 parallel page-navigation pairs organized into five difficulty buckets (random-walk distances of 50-1K, 1K-1KK, 10K-100K, 100K-1M, and IM-10M).
5 EXPERIMENTS
To study the knowledge advantage sap, we evaluate models on SYNTHWORLD- in multi- hop QA and page navigation. We evaluate six models: GPT-5-mini (OpenAI, 2025) (reasoning effort set to medi um), Gemini-2.0-Fin (Gemini Team, 2025), gpt-oss-20B, gpt-oss- 120B (Ope- nAI et al., 2025), Kimi-K2IInstruc (AI the Kimi Team, 2025a), and Kimi- SoniiK2-Thinking (Al & the Kimi Team, 2025b), enabling observations across model families, model sizes, and instruct vs. thinking models. Additional experiment details and evaluation prompts are in Appendix C.
Multi-hop QA Baselines. We evaluate three primary baselines: (1) Closed-book; where the model has no access to documents and answers directly from its parametric knowledge (KA (KAbase), ) (2) One- step RAG, where the model retrieves supporting documents once before answering (KA RAG); ); and (3) IRCoT RAG (Trivedi et al., 2022), which interleaves retrieval with chain-of-thoughh reasoning, enabling iterative reasoning and retrieval Steps(KA C RAG), For retrieval, we use the HippoRAG retriever, designed for factual, multi-hop contexts (Gutierrez et al., 2024). In addition, we include a Reading Comprehension condition which the model is given all gold (2-4 documents depending on graph motif, examples in Table 3) and additional distractor documents, equaling 10 total. This condition serves two interpretations: (i) it provides an upper bound when retrieval is not a bottleneck, and (ii) it separates the inherent difficulty of the reasoning task from the challenge of retrieving relevant evidence in unfamiliar settings. All baseline prompts for QA are included in Appendix C.I.
Page Navigation Baselines. Page navigation tests an agent's ability to plan and reason over linked knowledge environment. For page navigation, we follow the design of existing tool-use agents (Yang et al., 2024; Gu et al., 2025) and evaluate an agent equipped with two function- calling tools: click_link, which allows the agent to click any link on the current page, and backtrack, which allows the agent to return to previously visited page. To address our navi- gation research questions, we evaluate the agent under two observation conditions: (1) Links Only, where the agent observes only the set of outgoing links on each page (KA base); and (2) Content Links, where the agent observes both the outgoing links and the full page text (KA content). ). We include all prompts for agentic navigation in Appendix C.2.
The Links Only condition isolates the contribution of parametric knowledge and semantic familiarity, since navigation must rely entirely on recognizing entities in link text. The Content + Links condi- tion tests whether access to textual content can compensate for the absence of parametric knowledge by providing additional evidence for navigation decisions. In both settings, the agent is limited to maximum of 30 steps. This cap is well above the distribution of shortest path lengths (median 5, maximum 11). ensuring all tasks remain solvable while avoiding unbounded exploration. In our subsequent results, we observe that this bound is sufficient for meaningful exploration.
Metrics. For all multi-hop QA experiments, we report token-based F1 scores for task performance following prior work (Trivedi al., 2022). Following HippoRag 2 (Gutiérrez al., 2025), also report recall @5 5 for RAG basclines to evaluate retrieval quality. For page navigation, we report the success rate of reaching the target page.
RESULTS AND DISCUSSION
We show results across task buckets for multi-hop QA and page navigation in Figures 4 and 5. We report aggregated results for all task instances in Table 4 and 5 in the Appendix.
RQ1: What is the knowledge advantage gap when relying solely on parametric knowledge? In multi-hop QA. across models, we observe the baseline performance in RM, phus xe 20, indicat- ing that SYNTHWORLD-RRY presents questions that LMs answer using parametric knowledge
7

---

Published as a conference paper at ICLR 2026
F
100
F1 Score
50
GPT-5-mini
RAG CoT,
RAG
RAG
CB
RÁG
CB
RC
CoT+
RC
RAG
RAG
RAG
RAG
RAG
RAG
Recall@s
100
50
CoT+
CaT-
CoT+ RAG
RAG
CaT-
CoT+ RAG
RAG
RAG
RAG
RAG
RAG
F1 Fcorr
100
50
Gemini-z.0
RAG CoT+
RAG
RAG
RAG
CB
CoT+
RC
RAG
RAG
RAG
RAG
RAG
RAG
Recall@s
100
50
CoT+
CoT+
CoT+ RAG
RAG CoT-
CoT+ RAG
RAG
RAG
RAG
RAG
RAG
F1 Score
100
50
gpt-oss-208
GB
RAG Cofe
RAG Cof- RAG
CB
RAG Cof+
RC
RAG Cot-
RC
CB
RAG
CoT. RAG
RC
CB
RAG CoT+
RC
RAG
RAG
RAG
RAG
Recall@5
100
50
Cot+
CoT- RAG
RAG
CoT+ RAG
RAG CoT+
RAG
GoT+ RAG
RAG
Cot+
RAG
RAG
RAG
F1 Score
100
50
Spt-
CB
RAG CoT
RC
CB
RAG CoT+
RC
C8
RAG COT
RC
C5
RAG CaT+
RC
CB
RAG
CoT, RAG
RC
CB
RAG CoT.
RC
RAG
RAG
BAG
RAG
RAO
Recall@s
100
50
RAG
CoT+
RAG
CaT-
RAG
CoT+
RAG
CoT-
RAG
CoT RAG
RAG
CaT+
RAG
RAG
RAG
RAG
RAG
KA Gap
F1 RM
SM
RB5 RM
Ri5 SM
Figure 4: Multi-hop QA Results by Reasoning Motifs. We report Fl scores on SYNTHWORLD-
RM (RM) and SYNTHWORLD-SD (SM), along with the knowledge advantage gap (KA FIRM- Flsx). Settings: CB Closed-book, RAG One-step RAG, CoT+RAG IRCoT + RAG, RC Reading Comprehension. We show Recall @5 5 for RAG baselines (by construction, CB has recall = 0 and RC has recall = 1). IRCoT + RAG substantially reduces the KA gap compared to the CB baseline, primarily due improved retrieval. Example questions for each motif given in Table 3.
(Table 4; Closed-book, RM). In contrast, the near-zero Phase validates that WYNTHWORID-SMM questions cannot be solved with parametric knowledge alone (Table 4; Closed-book, SM). Overall, K.A K-Abase base = 20 (Table 4; Closed-book, KA). As task difficulty increases, Ph decreases as expected, while phase remains at 0, showing that the gap would be even wider if we restricted evaluation to casier QA tasks (Fig. 4; CB left right). In the reading comprehension setting, performance is equalized or even stronger in the SM cases because LMs are not distracted by parametric knowledge that could interfere with grounding its reasoning in the content (Monea et al., 2024).
For page navigation, find larger gap for GPT-5-min and Kimi-K2 models (K.A base 30) than for Gemini-2.0-Fii 2.0-Flashadd gpt-oss models (K.A base xe 20) (Table 5; Links Only, KA). This suggests the first set of models are better able to leverage parametric knowledge to locate the target page. Across difficulty levels, performance drops for both RM and SM tasks, but the gap persists. At the easiest difficulty, the gap narrows slightly, as models SM can exploit the structure and semantics of hyperlinks achieve modest success.
RQ2: To what extent does knowledge augmentation help close the gap? Knowledge augmenta- tion with One-step RAG improves absolute performance across both RM and SM tasks. However,
8

---

Published as a conference paper at ICLR 2026
GPT-S-mini
Gemini-2.00 2.0-Flash
50-1K
1K-10K
10K-100K
100K-1M
1M-10M
50-1K
1K-10K
10K-100K
100K-1M
1M-10M
100
RM
50
gpt-oss-20B
gpt-oss-1200 120B
100
Success Rate
Kimi-K2-Instii K2-nnstrct
Kimi-K2-Thinking
Links Cont. -Links
Links Coni.
Links Cont.
Links Cont.
Links Cont.
Links Cont.
Links Cont.
Links Cont.
Links Cont.
Links Cont.
-Links
-Links
+Links
-Links
-Links
.Likks
-Links
-Lnks
-Links
Figure 5: Page Navigation Results by Difficulty (i.e., Expected Random Walk Distance). We report success rate SYNTHWORID-RR. (RM) and SYNTHWORID--S (SM) and the knowledge advantage gap (KA = SuccessRM SuccesssM).. Models consistently perform better on real- mapped corpora, especially in harder navigation tasks, indicating that parametric knowledge enables shortcuts. Page content (Content Links vs. Links Only) benefits models more on synth-mapped corpora, narrowing the gap and showing its value in novel environments.
the knowledge advantage does not shrink; in fact, it widens. Specifically, KA base KA RAG = 4.0 for GPT-5-mini and -1.3 for G.mini-2.0-Flash and similarly for other models (Table 4; Closed-
book - One-step RAG). pattern consistent across multiple difficulty levels (Fig. 4; A. B, C, F). This suggests that while One-step RAG benefits both RM and SM, disproportionately benefits RM and reinforces models' reliance on parametric knowledge. Meanwhile, IRCoT + RAG reduces the gap. Overall, KA hase - KA IRCaT+RAG is positive for both models, 5.2 for GPT-5-mini and 10.3 for Gemini 2.0-Flash (Table 4; Closed-book IRCoT + RAG). We observe the gap closing across rea- soning motifs (Fig. 4), indicating that interleaving retrieval with reasoning better aligns knowledge integration with task demands."
To further probe this effect, we compare with the reading comprehension setting (i.e., perfect recall by construction). Triangulating reading comprehension F1-scores with Fi-scores and retrieval recall from One-step RAG and IRCoT RAG (Fig. 4; rows 2 and 4), we can infer that retrieval quality is a main driver of observed performance gaps. Retrieval performance improves slightly with IRCoT in both RM and SM, but retrieval in SM remains consistently lower than in RM. HippoRAG 2 uses an LM (GPT-5-min) in our experiments) to separately index RM and SM corpora and to retrieve documents given the input query. Given this setup, results suggest that LM-based retrievers may not generalize well in novel environments, raising questions about the robustness of LM-indexed retrieval pipelines.
With respect to page navigation, across all task instance, we observe granting the agent access to page content improves performance, yielding differences of KA base KA content GPT-5-mini and Gemini-22.--Fashh respectively (Table 5; Links Only Content Links). The performance gap narrows most on simpler navigation pairs (Fig. 5), though it remains present on more difficult ones.
9.3 and 7.0 for
To potentially explain the knowledge advantage gap, we analyze agent behavior by measuring how often externaiized reasoning traces mention entities not observed during page navigation. For ample, when tasked with navigating to the Brussels metropolitan area, model trace included the statement: "Ghent in Belgium and likely links Belgian geography or Brussels-related pages." We count the mentions of Belgium and Belgian external, since they had not appeared in any pre- viously visited page. In the SM setting, this rate by construction (and confirmed empirically).
We note IRCoT + RAG does not improve absolute performance compared to One-step RAG for gpt-oss models as they struggle to follow the IRCoT prompt format. These results point the importance of nuanced studies into the impact of knowledge integration.

---

Published as a conference paper at ICLR 2026
Mcanwhile, in the RM setting, we observe frequent reliance external knowledge: under the Links Only condition, at least one external entity is mentioned in 48% of steps for GPT-5-mini and 60% for Gemini-2.00Flashh Expanding access to Contens + Links reduces these rates to 35% and 15%, respectively. Without page content, RM models tend to fall back on stored factual knowledge. In contrast, SM-like settings (where information is novel) offer only limited scope for fallback. This points to an opportunity to design agentic systems that both remain effective and efficiently acquire the necessary background knowledge.
Insights enabled by SYNTHWORLDS. The parallelism of SYNTHWORLDS enables controlled comparisons that isolate different aspects of model behavior. For example, it can allow us to ask when models take longer reasoning paths in the absence of recall or whether (and under what con- ditions) error types shift. also makes it possible to investigate which system-level factors (such as retrieval quality in QA) and which core LM capabilities (as measured by reasoning agentic benchmarks) lead to narrower or wider knowledge advantage gaps. In our experiments, we studied knowledge integration through retrieval, both in single-step RAG and when interleaved with chain- of-thought or agentic workflows. These methods improved performance but did not fully eliminate the knowledge advantage gap. In QA, we see that it is problem about knowledge acquisition (i.e., obtaining all the relevant documents), but additional thinking (e.g.. CоT) can help. Meanwhile, in page navigation, even when models have the same content available, there is a gap as factual knowledge enables shortcuts. Beyond our case study results, SYNTHWORLDS allows researchers to examine alternative integration schemes. For example, in page navigation, what if models are integrated with retrieval to better plan their navigation? To what extent do long-context methods, where models must synthesize and retain relevant information without retrieval (Hsieh et al., 2024), or multi-agen workflows (Du et al., 2024), where group discussion and feedback shape integration, can help with knowledge augmentation?
Future work and extending SYNTHWORLMN Our current work only scratches the surface of these possibilities. A limitation is that our experiments were conducted on the specific SYNTH- WORLDS corpora and task designs we introduced, which may restrict the generality of our findings. These choices do not cover the full space of "constructed worlds" (or tasks) that could be defined by different relation types, connective structures, or contexts. Altering the way the corpora is con- structed could lead to different outcomes. Nonetheless, because SYNTHWORD WORLDS is fully automatic, inexpensive, and flexible given any imput knowledge base, we can generate alternate parallel corpora and probe these questions more broadly (see Appendix B.6 for an expanded discussion). Future work could impose targeted constraints on graph construction to highlight particular reasoning challenges or examine how parametric knowledge interacts with different underlying knowledge bases.
In general, our framework requires a high-quality knowledge graph to encode complex relation- ships and ensure factual consistency in synthesized text. Therefore, as modern extraction methods make knowledge graph construction from text increasingly feasible (Sainz et al., 2024; Xu et al., 2024a), graphs can be constructed directly from unstructured text (e.g.. Wikipedia). Once built with consistent, accurate facts, the SYNTHWORLD framework follows naturally. Likewise, SYNTH- WORLDS is not limited to one knowledge graph domain. The core idea, i.e., sampling subgraph, renaming entities, and constructing parallel corpora and tasks, generalizes to any knowledge graph. The main requirement is understanding which entities should be renamed, which depends on the domain. For example, in mathematics, we could create parallel worlds with different notation sys- tems (RM: xr y. fz); SM: a, 3, ((a)). Similarly, for code generation, we can consistently rename entire libraries (e.g., numpy/pandas) and function calls for the SM variant. By supporting con- trolled studies of reasoning, memory, and adaptation across varied settings, SYNTHW WORLDS lays the groundwork for developing LM systems that are more robust and generalizable.
CONCLUSION
We present SYNTHWORLDS, a framework for disentangling the role of parametric knowledge in LM reasoning and retrieval. By constructing parallel corpora and tasks with controllable difficulty, SYNTHWORLDS reveals persistent performance gaps even when models have access to retrieval or page content. These findings highlight opportunities for advancing reasoning in novel environments and position SYNTH WORLDS as a scalable testbed for developing methods that generalize beyond reliance on parametric knowledge.
10

---

Published as a conference paper at ICLR 2026
8 REPRODUCIBILITY
We provide full details of dataset construction, experimental setup, hyperparameters, and prompis in the Appendix ensuring that our dataset and results could be reproduced. The dataset used in our experiments is included in the supplementary material and will be publicly released. The code and dataset for running all experiments available https:/ /anonymous 4open. science/ r/synthworlcr s/
ACKNOWLEDGMENTS
We thank the UW Behavioral Data Science Group members, Jeffrey Li, Weijia Shi, and Harsh Trivedi for their valuable suggestions and feedback, and Tiffany Zheng for her continued motivation and personal support throughout this project. This research was supported in part by NSF IIS- 1901386, NSF CAREER IIS-2142794, and Garvey Institute Innovation grant.
REFERENCES
Lada A. Adamic and Bemardo A. Huberman. Power-law distribution of the world wide web. Science, 287(5461):2115-2115, 2000. doi: 211126//cceemme2.7556122222 URL https:
//www/ /ww.scciene.org//oo...
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cot- terell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pp. 3554-3565, Online. June 2021. Association for Computational Linguis- tics. doi: 10. 10.18863/v112211 manacl-main.277 URL https: / /aaa/aaaaattttpocceeeee org/2021. naacl ma 278/.
Moonshot Al and the Kimi Team. Kimi-k2: Open agentic intelligence (kimi-k2-instruet model card). https: : //moonshotai.. . wwg.tiihhbb..... thub. /oo/iiiiiii/. Kimiikk// 2025a. Open-source mixture-of- experts (MoB) LLM with IT parameters and 128K context length.
Moonshot AI and the Kimi Team. Kimi k2 thinking. https:/ //woooooottai..cccoomoo .itttai..giiitt github. io/ 1imi-K2/thi nking html, 2025b. Documentation / model card for the Kimi-K2 model.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: part 3.1, knowledge storage and extraction. In Proceedings of the 41st International Conference Machine Learning. ICML'24. JMLR.org, 2024.
Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench v2: Towards deeper under- standing and reasoning realistic long-cod long-context multitasks. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (eds.). Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6639-3664, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176- 251-0. doi: 10. 10.18653/w1/2025.co accl- long. URL https: /tmp::://cclann http://ccaaaahhhhcc org/2025. acl acl-long. .at.ong..83... 183/.
Andrew Michael Bean, Simeon Hellsten, Harry Mayne, Jabez Magomere, Ethan A Chi, Ryan An- drew Chi, Scott A. Hale, and Hannah Rose Kirk. LINGOLY: benchmark of olympiad-level linguistic reasoning puzzles in low resource and extinct languages. In The Thirty-eight Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: https///pppneeee //wpppeeeeeieeeeeee/eeeeeeeeeeee net forum?idd=cIgaagGsttee
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023, URL https: / /ppeneevieww net / forum?
(d=TatRHT_ict
11

---

Published as a conference paper at ICLR 2026
A. K. Chandra, P. Raghavan, W. L. Ruzzo, and R. Smolensky. The electrical resistance of graph captures its commute and cover times. In Proceedings of the Twenty-Firg Anmual ACM Sym- posium on Theory of Computing, STOC '89, pp. 574-586, New York, NY, USA, 1989. As- sociation for Computing Machinery. ISBN 0897913078. doi: 10.1145/73000.73666 URL https:://oi. wtt.://.///....../. 73062.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021. URL https: //rxiv.. bxq/abs/2110. 14168.
Wikipedia contributors. Wikipedia: Popular pages. https: / /en wwkipedia. org/wiki/ Wi Wikipedia:Popullan pages, 2025. Accessed: 2025-09-12.
Fred J. Damerau. A technique for computer detection and correction of spelling errors. Com- mun. ACM, 733):711-77,, March 1964. ISSN 0001-0782. doi: 10.1 10.1145/363958..639999 URL
https://doi org/10
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning language models through multiagent debate. In Proceedings of the s1 International Conference on Machine Learning, ICML'22. JMLR.org, 2024.
Gemini Team. Gemini 2.5:
Pushing the frontier with advanced reasoning, multimodality. long context, and next generation agentic capabilities. Technical Report N.Xiv:2507.06266 Google DeepMind, 2025. URL https::/p// / //oiooiiooooo wtw.ioorrooooooooooooeoooeeeeee. /trg/10.00 10440855///k0 arxiv. 2507.0622.2 61. Version 4.
Albert Gong, Kamilè Stankevičiūté, Chao Wan, Anmol Kabra, Raphael Thesmar, Johann Lee, Julius Klenke, Carla P Gomes, and Kilian Weinberger. Phantomwiki: On-demand datasets for rea- soning and retrieval evaluation. In Forty-second International Conference on Machine Learning, 2025. URL https: / wwt.//peeeeee.eeeee/eeee. net/ forum?id=DDD0 j8ueN.
Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang. Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu, Mike A Mer- rill, Jeffrey Heer, and Tim Althoff. BLADE: Benchmarking language model agents for data- driven science. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 13936-13971, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 18653/66//224. madingg--emnlp.. 815/.
URL https: aalaantooooooyyyyyoooooooooooaaa org/ 2024. findings-emninn
Ken Gu, Zhihan Zhang, Kate Lin, Yuwei Zhang, Akshay Paruchuri, Hong Yu, Mehran Kazemi, Kumar Ayush, A. Ali Heydari, Maxwell A. Xu, Girish Narayanswamy, Yun Liu, Ming-Zher Poh, Yuzhe Yang, Mark Malhotra, Shwetak Patel, Hamid Palangi, Xuhai Xu, Daniel McDuff, Tim Althoff, and Xin Liu. Radar: Benchmarking language models on imperfect tabular data, 2025.
URL https::p////mmm /ttp///axvvv...////a//ee//a.aa./ org/ abs /alas/2550 2506 08249.
Bernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. HippoRAG: Neuro- biologically inspired long-term memory for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL. htt / ieview iew. net/ wtrrl/ormmiioo wwwpuummiiiiuuu iiuniiiiikksssssgggg
Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. From RAG mem- ory: Non-parametric continual learning for large language models. In Forty-second International Conference on Machine Learning, 2025. URL https: httpst::////peeeeee /oop/reeeeeieeeeee net f
LHH8Yn4HS2.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL httpttttp///tc.. https:://ppenreiieeee net. forum? Byw 2mQsCe.
12

---

Published as a conference paper at ICLR 2026
Lucas Torroba Hennigen, Zejiang Shen, Aniruddha Nrusimha, Bernhard Gapp. David Sontag, and Yoon Kim. Towards verifiable text generation with symbolic references. In First Conference on Language Modeling, 2024. httpt:::: /ttp///epeeeeeeeeeee/eeeeeeeeeee net/ fit.rrummiddiiiiiiiioooooii
A. Ali Heydari, Ken Gu, Vidya Srinivas, Hong Yu, Zhihan Zhang, Yuwei Zhang, Akshay Paruchuri, Qian He, Hamid Palangi, Nova Hammerquist, Ahmed A. Metwally, Brent Winslow, Yubin Kim, Kumar Ayush, Yuzhe Yang. Girish Narayanswamy, Maxwell A. Xu, Jake Garrison, Amy Arem- nto Lee, Jenny Vafeiadou, Ben Graef, Isaac R. Galatzer-Levy, Erik Schenck, Andrew Barakat, Javier Perez, Jacqueline Shreibati, John Hernandez, Anthony Z. Faranesh, Javier L. Pricto, Con- nor Heneghan, Yun Liu, Jiening Zhan, Mark Malhotra, Shwetak Patel, Tim Althoff, Xin Liu, Daniel McDuff, and Xuhai "Orson" " Xu. The anatomy of a personal health agent, 2025. URL https: http://aavvvv/.. org/ wwwp///g//////ooo//e/o.o//..aeee 2
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, pp. 6609-6625, Barcelona, Spain (Online), December 2020. International Com- mittee Computational Linguisties. doi: 10. //aclanthology. /rg/ 2020. molinggmmin..5e
URL https:
Cheng-Ping Hsich, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's 's the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL htt.s:/ /tpps//////p/eeeeeeeeeee net forum? id=kIoBbe76Sy.
Kaixuan Huang, Jiacheng Guo, Zihao Li, Xiang Ji, Jiawei Ge, Wenzhe Li, Yingqing Guo, Tianle Cai, Hui Yuan, Runzhe Wang, Yue Wu, Ming Yin, Shange Tang, Yangsibo Huang, Chi Jin, Xinyun Chen. Chiyuan Zhang, and Mengdi Wang. MATH-perturb: Benchmarking LLMs' math reason- ing abilities against hard perturbations. In Forty-second International Conference on Machine Learning. 2025. URL https: http///eeeeeeeeeeeeeeeeeeeeeeeea net/ forum? Ld=O2y70UggXX:
Martin Josifoski, Marija Sakota, Maxime Peyrard, and Robert West. Exploiting asymmetry for synthetic training data generation: SynthlE and the case of information extraction. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1555-1574, Singapore, December 2023. Asso- ciation for Computational Linguistics. doi: 10. 20.218653v1/22222 h2tp:/...mmmmmmiinn.... URL https: //aclanthology wwwllloyyorrg//00 crg/20233 emnlp www.lppmain.... 96/.
Jungo Kasai, Keisuke Sakaguchi, yoichi takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. Realtime QA: What's the answer right now? In Thirty-sevnnth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: openreview.w..e net. forum? i
Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, San- ket Vaibhav Mehta, Lalit K Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q. Tran, Quoc V Le, and Orhan Firat. BIG-bench extra hard. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.). Proceedings of the 63rd Anmual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2-500,, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979- 8-799-888119 76--1-1--- doi: 18.18v53/v1/ 2025 2025.acl-long. 185.. URL https: /tc//ccaaaa www.ancoooogo.oooooooooooooooooo
a aal- 1ol..ll........ 1285/.
Ravi Kumar, Prabhakar Raghavan, Sridhar Rajagopalan, D. Sivakumar, Andrew Tompkins, and Eli Upfal. The web as graph. In Proceedings of the Nineleenth ACM STGMOD-SIGACT-SIIG. Symposium on Principles of Databuse Systems, PODS '00, pp. 1-10, New York, NY, USA, 2000.
Association for Computing Machinery. ISBN 15811321 13214X. doi: 10. 10.1145/335168.355177 URL https: /oi. www.o.oorg/ii. wrggl0l...... 11145 01145/335168 335170.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. NV-embed: Improved techniques for training LLMs generalist embedding models. In The Thirteenth International Conference on Learning Representations, 2025. URL https:/ http::://ppeeeviii net forum? dd= gsy.tttS
13

---

Published as a conference paper at ICLR 2026
Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume I: Long Papers), pp. 15339-15353, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/1//2044.all pettps://aclattheee wtwp://ggg.rr//22.......oo www.04.....ooooooo. 10.8.
URL
Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. Retrieval aug- mented generation or long-context LLMs? comprehensive study and hybrid approach. In Franck Demonncurt, Daniel Freotiuc-Pietro, and Anastasia Shimorina (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 881-893, Miami, Florida, US, November 2024. Association for Computational Linguistics. URL https: / /tt///aaaanttt aclanthelogyy....ooo.o...ooooole orgg/0244
doi: 10. 11211/6//1/222...oo cenlp-innusstry emm1p- innuuiiutttr.... 66/.
Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, and Iryna Gurevych. Are emergent abilities in large language models just in-context learning? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5098-5139, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1188338/1220222/ acw.:2....loo...2.7777.. URL https::////.. / aacl seceannhhlogge org/ 2024 acl -onn.llog....0. 279/.
Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 28(6):1 2024. ISSN 1364-6613, doi:
http:/iii.or/10.1110000. 2024.01.011. URL https:/ sciencedi rect. com/ wccinccearticleee. 613646613240000
Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. GSM-symbolic: Understanding the limitations of mathematical reasoning in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: http//pppereeeeeeeeeeeeeee net/ wow.mmiidddiuiio.o. A
Giovanni Monea, Maxime Peyrard, Martin Josifoski, Vishrav Chaudhary, Jason Eisner, Emre Kici- man, Hamid Palangi, Barun Patra, and Robert West. A glitch in the matrix? locating and detecting language model grounding with fakepedia. Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 6828 6844,,4, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1202533//222...lllooooggg htttps: a antho 1 ogy. org/ 2024. a long. 369/-
Lun-Wei Ku, Andre Martins, and
URL
Joao Monteiro, Pierre-Andre Noel, Étienne Marcotte, Sai Rajeswar, Valentina Zantedeschi, David Vazquez, Nicolas Chapados, Christopher Pal, and Perouz Taslakian. RepliQA: A question- answering dataset for benchmarking LLMs on unseen reference content. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
URL https: / /pppeeeeeeeeeeeeeeeeeeeeeeeeeeee net / forum? i iididddd44Kkggg
Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wengi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip S. Yu, and Qing Li. survey of webagents: Towards next- next- generation ai agents for web automation with large foundation models. In Proceedings of the 31 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD '25, pp. 6140-6150, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400714542. doi: 3736555.
10.1145/3711896.3736688 33336555. URL https: /oii. org/10.
OpenAl. Gpt-5 mini. https:/ http::://ttttttttteeeeeeee /ttp///pppa//////////////e/..aaa www./poott......oooooooooo.oee 5, 2025. Variant of the GPT-5 model, launched August 7, 2025.
OpenAl, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Ar- bus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese, Tarun Gogineni, Adam
14

---

Published as a conference paper at ICLR 2026
Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach Johnson, Chris Koch, Irina Kofiman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le, Guillaume Leclere, James Park Lennon, Scott Lessans, Mario Lezcano-Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCal- lum, Josh McGrath, Scott McKinney, Aidan McLaughlin, Song Mei, Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ash- ley Pantuliano, Giambattista Parascandolo, Jongsoo Park, Leher Pathak. Carolina Paz, Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Proehl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev, Max Schwarzer, D. Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal, Yang Song. Dane Stuckey, Zhiqing Sun, Philiippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wallace, Xin Wang. Miles Wang, Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whitney, Hannah Wong, Lin Yang, Yu Yang. Michihiro Yasunaga, Kristen Ying, Wojeiech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao. gpt-oss- 120b & gpt-oss-20b model card, 2025. URL https:/ / /arxivvv.../////////aa/////aaava org/ abs/ 2508 10925.
Oscar Sainz, Jon Ander Campos, Iker Garefa-Ferrero, Julen Etxaniz. Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In The 2023 Conference on Empirical Methods Natural Language Processing, 2023. URL https: http//peeeeeeeeeeeeeeeeee net/ forum? wiw.s....s.......
Oscar Sainz, Iker García-Ferrero, Rodrigo Agemi,, Oier Lopez de Lacalle, German Rigau, and Eneko Agirre GoLLIE: Annotation guidelines improve zero-shot information-extracte In The Twelfth International Conference on Learning Representations, 2024. URL https: /openreview. net / ht////ormmmm// iididdd---a watpuummdddwYwyuxdddooooo Y3wpuxd7u9.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Ko- curek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Kyle Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chen- lin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burche Christopher Waites, Christian Voigt, Christopher Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito. Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, EI- lie Pavlick, Emanuele Rodolà, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozh- skii, Fanyue Xia. Fatemeh Siar, Fernando Martinez-Plumed Francesca Happé, Francois Chol- let, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germàn Kruszewski, Giambattista Parascandolo. Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch-Lopez, Gregor Betz, Guy Gur-Ari, Hama Galijasevic, Hannah Kim, Hannah Rashkin, Hannanch Ha- jishirzi, Harsh Mehta, Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hi- romu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng. Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Feráández Fisac, James B Si- mon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski,
15

---

Published as a conference paper at ICLR 2026
Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Je- sujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez- Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Math- ewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao. Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochande Louis- Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros-Colón, Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap. Maartje Ter Ho- eve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramirez-Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátýás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee. Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michat Swedrowski, Michele Bevilacqua, Michi- hiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Ti- wari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng. Nathan Andrew Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Ni- tish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang. Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter W Chang. Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu. Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Seungjae Ryan Lee, Ryan Stovall, Ryan Techan, Rylan Yang. Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wise- man, Samuel Gruetter, Samuel R. Bowman, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebas- tian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammie Debnath, Siamak Shakeri, Simon Thormeyer. Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee,, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, vinay uday prabhu, Vishakh Padmakumar, Vivek Sriku- mar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu. Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Sophie Hao, Yifu Chen, Yonatan Belinkov. Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie Wang. Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of lan- guage models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: http///epeeeeeeeeeeeeeeeeeeeeeee .ne/ forumorummooo forumeid-uyyLLBvooi Featured Certification.
Banjade,
Hongjin SU, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu Wang, Liu Haisu, Quan Shi, Zachary S Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O Arik, Danqi Chen, and Tao Yu. BRIGHT: A realistic and challenging benchmark for reasoning- intensive retrieval. In The Thirteenth International Conference on Learning Representations, 2025. URL https:/ http:pps////peeeeeeee.eeeee htt.:eeeee/orrruu11ocexeesss33
Yixuan Tang and Yi Yang. Multihop-RAa RAG: Benchmarking retrrevallaugmmented generation for multi- hop queries. In First Conference on Language Modeling, 2024. URL ht /ttpllp::///ppeeeeeeeeeee /ppereview. net/ fortm? 4et
16

---

Published as a conference paper at ICLR 2026
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for doi: 10.1162/tacl a_0055.. URL https:
Computational Linguistics, 10:539-555 5544, 2022.
/ //acllaae micclanthologg org/2022. tacl aal-llll..... 11.31/
Denny Vrandečić. Wikidata: a new platform for collaborative data collection. Proceedings of the 21st International Conference World Wide Web, WWW I2 Companion, pp. 1063-1064, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 9781 9781450312301. doi: 10.1 00114452187998.21188444 URL https:///oi. org/10 10. / 2 2188242.
Tu Vu, Mohit lyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoe Le, and Thang Luong. FreshLLMs: Refreshing large language mod- els with search engine augmentation. In Lun-Wer Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics: ACL 2024, pp. 13697-13720, Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10.18653/ v 112024.fndings-aee 813/.
URL ht.ps:: /acl/aaclccc anthology org/2024, 2tt.:2222...iniiii... wtnenns--nnnnnnnnnn..cne acl.
Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig, Agent workflow mem- ory. In Forty-second International Conference on Machine Learning, 2025. URL https:
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: simple yet challenging benchmark for browsing agents, 2025. URL 12516.
Robert West and Jure Leskovec. Human wayfinding in information networks. Proceedings of the 2 International Conference on World Wide Web, wWW '12, pp. 619-628, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 9781450312295. doi: 10.1145/ 2187836.2187920. URL https: httpt::s/s//ioorg//eeoett http://ooo.o.......ooo/ 1tt.:/11111111211111
Wikidata contributors. Help:label. 2025. Accessed: 2025-09-22.
Label,
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limita- tions of language models through counterfactual tasks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.). Proceedings of the 2024 Conference of the North American Chapter of the As- sociation for Computational Linguisties: Human Language Technologies (Volume Long Pa- pers), pp. 1819- 862, Mexico City, Mexico, June 2024. Association for Computational Linguis- tics. doi: 10. 10118865/112222 120.://////22...mm.ooooggggg 102. URL https: http:////aaaaaahhhlaaacceo org.orgg////0 2024. naacl- long. 102/.
Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, Yang Wang. and Enhong Chen. Large language models for generative information extrac- tion: A survey, 2024a. URL https::/p// /arxivv..... org/
/abp///////// aaaa/////333 2231. 17617.
Ruijie Xu, Zengzhi Wang, Run-7e Fan, and Pengfei Liu. Benchmarking henchmark leakage in large language models. arXiv preprint vrXiv:2444118824 2024b. URL https: setps://arxiv. erg/abs aas//a/s...0 2444. 18824.
Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.0888 000600000........ 2025.
John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In The eighty Annual Conference on Neural Information Processing Systems, 2024. URL https: http::://ppeeeevvieee net net/forum? forum? 1d=mppg6ut835.
17

---

Published as a conference paper at ICLR 2026
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, William Song, Tiffany Zhao, Pranav Vishnu Raja, Charlotte Zhuang, Dylan Z Slack, Qin Lyu, Sean M. Hendryx, Russell Ka- plan, Michele Lunati, and Summer Yue. careful examination of large language model per- formance on grade school arithmetic. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https:: /ttppr/vi///eeeiiee... net/ Eorum? ner/forum? 1d=RJ22hM2.2HH.
Nan Zhang, Prafulla Kumar Choubey, Alexander Fabbri, Gabriel Bermadett- Bemadett-Shapiro, Shapiro, Rui Zhang, Prasenjit Mitra, Caiming Xiong, and Chien-Sheng Wu. SireRAG: Indexing similar and related information for multihop reasoning- In The Thirteenth International Conference Learning Representations, 2025. URL https:tt////.....t /tt.ectps///ppereeeeee....... wwwpereewww.eeeeee...eeeeeeeeeee net. t=ttt//frrrmddddp55g5gaaa
Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji- Rong Wen, and Jiawei Han. Don't 't make your 1lm an evaluation benchmark cheater, 2023, URL https:/ / arxiv..v orgg /a/p//////a////////// alos/2311 2211 964.
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. ToolQA: question answering with external tools. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL http:::///ppeeevieew net//foumm? 2 ddppii
A dataset for LLM
18

---

Published as a conference paper at ICLR 2026
APPENDIX TABLE OF CONTENTS
SYNTHWORLDS Framework
20
SYNHWORL.-RM/SM Dataset Construction Details
B.1 Universe Construction
21
B.2 Surface-Fome Fora Perturbations
22
B.3 Parallel Document Generation
22
B.4 Multi-hop QA Construction
22
B.5 Human Validation
23
B.6
Discussion on Dataset Construction
................................
23
B.7
Additional Figures and Tables
24
B.8 Qualitative Corpora Examples
................................
28
B.9 Prompts for Corpora Construction
29
B.10
Prompts for Multi-hop QA Construction from Facts
36
Experiment Details
43
C.I Multihop QA Prompts
43
C.2 Page Navigation Prompts
46
Additional Experiment Tables
48
19

---

Published as a conference paper at ICLR 2026
A SYNTHWORLDS FRAMEWORK
In this section, we discuss the core formalization of the SYNTHWORLDS framework. Concrete de- tails actualizing this framework in our SYNTHWORLD-RMISH datasets are included in Appendix B.
World Knowledge Preliminaries. Formally, our dataset generation takes as input knowledge
base KG consisting of collection of entities & and a collection of relations R. We define the set of facts as F S & x R x &, and represent the corresponding graph G = FF. Each entity e & has an associated label E(e) E L. where L denotes the space of surface-form names (e.g.. textual strings such as "Albert Einstein"). In addition, each entity includes a relation of the form (e, ent _tttttypee...m T(e)). where ree) E T specifies the entity's 's ontological type (c.g., person, house, plane). (e) is intended to denote general category, without mention of specific named entities.
A universe of triplet facts is therefore defined by U = (G, C).
Coherent Universe Construction. To construct coherent and connected universe we leverage the facts from G. At a desired tractable size and complexity, we first sample a connected subgraph G' S G (Fig. 2a). G' is constructed by iteratively expanding the frontier from a seed set Qo C E. At iteration t, given the current frontier Q S E, we sample neighbors (N)) for each v E 22 and add them the subgraph. ((u) includes all entities u such that (v, r, r) E F or (u, (,,T, v) E F for some r E R.
After 7 expansion steps obtain sampled subgraph Gr C G. To ensure sufficient connectivity, we extract the k-core subgraph (i.e., the maximal subgraph in which every node has degree at least k) and then take its largest connected component, denoted GT, & S Gr. For notational simplicity, in the following we use G' to refer to GT.
Surface-Form Perturbations. To obscure factual knowledge, we perturb surface forms, i.e., entity names and timestamps tied to entities (Fig. 2b).
Simple renaming risks (a) factual leakage where replacements still reveal real-world associations (e.g., Tokyo - Torioka, which continues to suggest Japanese origins)), or (b) incoherence, where substitutions violate type or consistency constraints (e.g.. Ryan Reynolds was born Vancouver - Silvercrest Collegiate was born in Sarah Thompson), thereby failing to preserve Nomain-genens knowledge. To prevent these issues, we systematically perturb all named entities and temporal labels through controlled renaming that obscures underlying facts while preserving coherence.
In particular, this entails: (i) type-consistent naming, where synthetic names respect the entity's ontological type (e.g., Nile River -> Lora River, not Lora Pavilion), and (ii) name-derivation sistency, where renames propagate to related surface forms (e.g.. if Vancouver + Metronis, then George Vancouver, after whom the city is named, - Altheon Metronis). These constraints pre- serve semantic coherence and affiliation cues, preventing surface-level artifacts from confounding evaluation.
Let Eproper C denote the set of named-entity nodes subject to renaming and Creat the denote the set of original real-mapped labels. We say that node u is name-related node if and only if u. E Eproper and (i) É(v) is a substring of E(u) and (ii) 3r E R (u, r, v) E F or (u, u) E F. That is, name-relation requires both a lexical substring relationship and an explicit relation in the knowledge graph. For instance, Vancouver is neme-related to Vancouver Canucks.
This induces directed acyclic name-related dependency graph Gaep = (Eproper: Exep) where (u, r, u) E Esep if and only if is name-related to with relation We rename entities according level-order (breadth-first) traversal of Guep- processing all nodes at each level before moving to the next level. This ensures that all entities at depth d are newly labeled before any entity at depth + 1, maintaining consistency across substring relationships.
We define the updated labeling function l' E + L through the following process. For each V E Eproper processed in level-order, we query a LM with input (r(v), (("(w), r(u),r) (u, r, v) e Escp) }) to generate L' (1)). In other words, we rename entities by providing the LM with the target
Otter literals, e.g.. population counts and physical measurements, are excluded because they could easily (a) reveal real-world facts (e.g.. "Mount FakeMountain is 8848m tall" still points to Mount Everest) or (h) distort demain-general reasoning when perturbed.
20

---

Published as a conference paper at ICLR 2026
entity's 's type and the new names of all related entities it depends on. For entities not being renamed, we set (e) = f(e). We include prompts for renaming in Appendix B.9.
For timestamps, apply fixed offset 8 per universe: for any timestamp æ, replace it with z -- preserving ordering and interval relations (c.g., a parent's ' birth precedes a child's), while removing the potential for parametric knowledge be leaked.
After these perturbations, we produce a synth-mapped universe U = (G, E') where entities retain their structure and types but receive new synthetic labels.
Parallel Corpora Generation. For corpora generation (Fig. 2c), we first generate documents from the synth-mapped universe U' such that the facts faithful to G, then add symbolic references to entity IDs in the text, before using these IDs references mapped to real-mapped labels to covert each synthetic document into real-mapped version. The output is two parallel corpora: one synth- mapped and one real-map mapled with identical sentence structures and world-consistent facts, differing only in their surface-form labels.
By generating documents from wynth-mappen (as opposed to real-mapped) entities first, we exploit the asymmetry that synthetic entity names (e) have no connections the LM's parametric knowl- edge. This prevents the LM from introducing auxiliary facts and makes it easier to stay faithful to the provided triplets. For example, when writing about the synthetically named entity for Austria, the LM cannot mention facts about Vienna based on external knowledge and must rely solely on the provided facts.
Concretely, for each entity E E, we collect all incident edges {(u, v) (u, v,) E FJ {(v,r, r, u) | (u,r,u) E F} and retain only the majority orientation (i.e., whichever set is larger) to define (v). We then query an LM to generate document describing the facts in (Nv).
Next, following Hennigen et al. (2024), we instruct an LM to add symbolic references {er, 62, c2..... to the synth-mapped documents, adding to each mention of (e) symbolic identifier. This provides both hyperlinks for document navigation ($4.1) and facilitates the conversion process described.
Given synthetic document with symbolic references and the entity mapping {(e, E()), (c))) E}, we query an LM to generate an equivalent real-mapped document by replacing each symbolic reference e; with the original label ((e).. The symbolic references ensure that the correct entity mapping is preserved during conversion. During this process, we apply programmatic and LM-based checks ensure document parallelism, factual consistency, and effective knowledge obfuscation.
B SYNTHWORDD-RMES RMM/SMMMMMMMMMMMM DATASET CONSTRUCTION DETAILS
Our dataset construction pipeline follows the framework in Appendix A (overview in Fig. 2). All prompts for dataset construction are in Appendix B.9- 1010. Table 2 summarizes the LM used and LM API costs for each step of the pipeline including multi-hop QA task construction.
B. UNIVERSE CONSTRUCTION
For our specific SYNTHWORLDS corpora we start with the Wikidata KG (Vrandečić, 2012) (01/20/2025 dump).
Knowledge graphs such as Wikidata are heavily skewed toward small set of high-frequency rela- tions (c.g., instance of, subclass of, located in). If we sample subgraphs in strict proportion this distribution. the resulting universe is both narrow in structure and closely aligned with the original world knowledge. This limits its usefulness for tasks where that are not simply memorization of facts. To control edge-type diversity, we introduce uniformity factor. For & at iteration f, let (,(r; v) denote the set of candidate triplets involving with relation r. We define
want probe reasoning in settings
Pr U)
1 uniformity.
In initial experiments, including both orientations often led the LM generate inconsistent documents, c.g. entity descrihed as both the son and the father of another.
21

---

Published as a conference paper at ICLR 2026
High uniformity yields diverse edge types (a O: uniform), while low uniformity favors frequent relations (a 1:
To encourage diversity of entities, we initialize Co the set of Wikidata entities across all categories defined in Wikipedia's popular pages (contributors, 2025) To ensure high-quality entities, we discard entities, unlabeled entries, entities
Wikidata nodes that time terms, whose names include numbers. We run the iterative sampling for = 11 steps with uniformity 0.6, and take the 19-core subgraph G' 111,19
B.2 SURFACE- FFRM PERTURBATIONS
We rename entities identified via Wikidata's entity naming rules.²
Given all proper-name entities Eproper in G' that share type description, we prompt a LM to propose new names for that entity type following. In Wikidata, entitiy type inferred through the instance of relationship (P31). However, certain instance of continue to contain named entities. For these cases we recursively apply the instance of until no named entities exist in the label. For example, say Vancouver only has a instance of label "city in British Columbia" in this case take the instance of label for British Columbia which is "province of Canada", finally we take the label for Canada which is country so then the label becomes "city in province of country".
In addition, we incorporate Wikidata time qualifiers (e.g.. Barack Obama - president + USA; start time 20 January 2009), which attach additional temporal information to fact triplets. To prevent timestamps from trivially revealing real-world identities, we apply a 5 = 39.
B.3 PARALLEL DOCUMENT GENERATION
We prompt a LM to generate factually consistent document from fact triplets (prompts in Ap- pendix B.9). To ensure quality, we add the Wikidata entity id (prefixed with Q, e.g., Q15) when generating symbolic references. These are unique identifiers for the underlying entity that we can then use to check the correct label is used the corresponding real- real-mapped and synth-mapped doc- uments. We implement programmatic checks to guarantee that (I) only entities present in the facts are included the page, and (2) the display text for each entity matches the underlying link. When converting from synth-mapped to real- real-mapped text, additionally require that both documents share the same set of symbolic references (thus inducing the same graph structure) and that no men- tion of any synth-mapped entity remains. Finally, enforce strict quality thresholds: we only keep pages when (a) the similarity (measured using the Damerau-Levent Levenshtein edit distance (Damerau, 1964)) between the initial generation and the symbolic-reference version exceeds 0.95, and (b) the similarity between the synth-mapped and real-mapped versions exceeds with symbolic references exceeds 0.85, Practically, this filtering ensures that only parallel documents with highly consistent structure and minimal unintended variation are retained.
To ensure that the generated pages are truly novel, we prompt the same LM to guess the underlying entity from a wrtth-mapped document, providing it with the (unrealistic) clue that the page corre- sponds to real-mapped entity whose names have been perturbed. This constitutes deliberately strict check: in actual task settings, the LM would never be told that the page is based on real- world entity. Any page the LM gets correct we remove from our corpus. After each filtering step, we retain only the largest connected component of the hyperlink graph, ensuring that the resulting corpus remains navigable for downstream sageenavigation tasks.
B.4 MULTI-HOP QA CONSTRUCTION
Validating Facts for QA Construction. Prior to the steps described in Section 4.1, we also first validated what facts were actually in the generated corpora.This step accounts for cases where some facts may have been omitted during generation. Given a document generated by LM and the set of source facts the generation based on, we use another LM to identify which of those facts are actually present in the document. The prompt for this step is included in Appendix 1.
/tikkiiHHepppp. Label
Qual1 Fiers
22

---

Published as a conference paper at ICLR 2026
This step enables to construct the directed fact graph G FE, Each fact is a directed triple
(e T, eÿ) E F, Eiey E
where is relation annotated with property name, and the source page in corpora from which the fact was extracted. By construction, each edge originates from a distinct source page, ensuring that multi-edge subgraphs aggregate knowledge across independent contexts.
Ensuring Diversity of Generated Questions. Given the fact graph we sample graph motifs (i.e.,
the motifs in Table 3). A motif is relational subgraph of Gfact. defined
M
Far), VM E S..
To ensure diversity and quality of questions generated, we sample graphs subject to the following constraints:
1. All entities in a motif must be distinct: e + ej Vi # j. e, e, E VM.
2. All facts in FM must come from different pages.
3. For a given anchor configuration and relation sequence, at most one instantiation of the motif is retained. For example, for motif A, we keep at most one subgraph {[e1, F1, 62), (ez, T2, e2)e,) for each tuple (2). For motif E, we keep at most one subgraph {(e1, r1, 62); (e3, r2, (2), (42, r3, F3, es), (e4, r',, 65), (es, 66)), for each tu- ple (e1, C3, ri, 22, T3, r4, Ts). In other words, we ensure there is only one unique reasoning chain for given motif.
4. Following Trivedi et al. (2022), we remove any n-hop question that is a sub-graph any m-hop question (m >n>>>> 1).
5. To prevent over-representation of any particular edge or intermediate node, we limit reuse of facts and bridge entities within motifs. Concretely, each fact (es, ez) E F and each bridge entity (i.e., entities that are neither roots terminal nodes of a motif) is sampled at most five times per motif.
B.5 HUMAN VALIDATION
To assess corpora quality, two researchers labeled each candidate fact as (i) expressed in the docu- ment, (ii) not expressed, or (iii) inconsistent with the document. Across 28 unique pages (n 798 facts), no inconsistencies were observed, giving 95% upper bound of 0.4% on the true inconsis- tency rate. On double-annotated pages, agreement was 99.5% with Cohen's Æ = 0.85, indicating almost perfect reliability. Corpus-level factual recall was 98.8% (95% CI 19,.0, 99.7I), with mean page recall 98.9 98.9% These results demonstrate that the dataset is clean, reliable, and faithfully repre- sents the intended facts.
To validate question quality, another researcher inspected a sample of 30 parallel questions, covering 5 examples for each reasoning motif. For each question, the researcher verified three criteria: (i) the questions were parallel (ii) the question led to a correct and unambiguous answer, and (iii) the resulting question was coherent and natural. All questions were found satisfactory.
B.6 DISCUSSION ON DATASET CONSTRUCTION
Choice of Entities to Rename. During corpus construction, we restrict renaming to Wikidata enti- ties whose labels begin with a capital letter (c.g.. Geoffrey Himtn, Q92894), which typically indi- cates named entities. Entities whose labels begin with lowercase letters (e-g.. dog, Q144; oxygen, Q629) are not renamed. An edge case arises for entities such as einsteinium (Q1103), the element named after Albert Einstein. Since einsteinium does not begin with capital letter, would not be renamed, creating potential factual knowledge leak (e.g.. "einsteinium is named after [Renamed Scientist]" implicitly revealing Albert Einstein). To mitigate this, we remove all wynth-mapped pages where such leakage could occur, ensuring that models cannot trivially recover world knowledge after being told that entities have been renamed. Obfuscating einsteinium-style knowledge more broadly and directly remains avenue for future work.
23

---

Published as a conference paper at ICLR 2026
Controllability and Stochasticity in Data Generation. To generate instances of SYNTH- WORLDS, we expose several controllable knobs. Different seed nodes (c.g., starting with Al re- searchers) can be sampled to produce distinct yet structurally valid corpora. The uniformity factor can be varied to influence graph connectivity. Subgraph sampling can also be restricted to entities of specific types (e.g., researchers, institutions, students), imphasize/de-emphasit relations. Renaming strategies further contribute variability: alternative LMs, different temperature settings, varied timestamp perturbations can all yield distinct datasets. Finally, document gen- eration may use different LMs to produce stylistic variation, while remaining consistent with the underlying facts. Together, these controls balance the need for world consistency with stochastic diversity across dataset instantiations.
particular edge
B.7 ADDITIONAL FIGURES AND TABLES
Figure shows the distributions of page entity types (based on Wikidata's instance of prop- erty) and relation types (across all facts) in the generated corpora. Figure 7 shows the in-degree and out-degree distributions of the page graph in SYNTHWORLDS, Figure 8 visualizes the constructed hyperlink graph used for Page Navigation. Table provides the LM API cost of constructing SYN- THWORLD-RM/SM Table 3 includes all graph motif's and examples of constructed questions.
Dataset Construction Step
LM Used
API Calls
Inp Tok
Out Tok
Cost
Surface Form Renaming
GPT-4o-mini
0.3K
237.9K
38.8K
$0.06
Corpora Generation
GPT-5-mini
35.3K
110.1M
74.3M
$176
Novelty Validation
GPT-5-mini
15.5K
6.7M
93.4M
$188
Multihop-QA Question Gen.
GPT-5-mini
4.8K
6.9M
3.0M
$7.82
Total
55.9K
123.9M
170.7M
$372
Table 2: Token usage and LM API costs for constructing SYNTHWRLD--RMIM. Totals are shown in the last row. During the project period new LMS were released and sought to use the best models available to generate a public datasets. This means that GPT-40-mini was used during surface form renaming (a much simpler task) while all other steps used GPT-5-min of API calls includes follow-up prompts when the initial LM output does not pass programmatic validation checks. GPT-5-mini novelty validation, we enforced very strict notion of novelty and explicitly instructed the model to "think", which inflated reasoning token usage (details in $B.3; prompt in $B.9). In practice, one could reduce reasoning effort to low, since faithful evaluation on synth-mapped tasks would prompt LMs with the information that entities have been renamed. Such adjustments would substantially lower costs, bringing the total closer to $200.
The number
used with the default reasoning effort set to medium. For
24

---

Published as a conference paper at ICLR 2026
Top 20 Most Frequent Page Entity Types
person
media frenchise episode
art museum
big city
city
sovereign state
television series episode country
male given exhibition
nonprofit organization
municipality of sovereign state
modern langusge
national museum
commune of sovereign state
fictional humen
written work
video game
local authority museum
500
1000
1500
2000
2500
3000
3500
4000
Count
Top Most Frequent Fact Relations
sibling
has works in the collection described by source occupation
award received
member
date of death
gender
languages spoken, written or signed
twinned administrative body
spouse
position held
country of citizenship
cast member
work location
characters fiold work
noble title
5000
10000
15000
20000
Count
Figure 6: Entity Type and Relation Type Distribution of SYYTHWORLD-RM/SM. Documents cover a broad range of entity types and relation types.
25

---

Published as a conference paper at ICLR 2026
In-degree Distribution (Log-Log)
Out-degree Distribution (Log-Log)
10
Median 9.0
Median 14,0
Probability
10
10-3
10
10
10
10a
100
10
102
In-degree
Out-degree
Figure 7: Degree Distribution SYNTHWOR.D-RM/SM. Our corpora preserve the intercon- nected and structured nature of knowledge networks (i.e., power-law degree distribution), matching complexity of real-world information ecosystems.
Figure 8: SYNTHWORLD-RMASS Hyperlink Graph illustrating scale-free topology, where few highly connected hubs dominate while most nodes have relatively few links. Node size is determined by max(1, min(4, deg(e) ))
26

---

Published as a conference paper at ICLR 2026
Graph
Motif
Decomposition
Question
1. Who the screenwriter of The City the Edge of Forever? Harlan Ellison
In what year was the screenwriter of The City the Edge of For- ever nominated Hugo Award for Best Short Story? 1971
A
O+0
2. In what year was Harlan Ellison nom- inated for Hugo Award for Best Short Story? 1971
1. Which family does Sirindhorn, Princess Royal belong to? House of Mahidol
88
Where does the chairperson of Sirindhorn, Princess Royal's family live? Grand Polace
2. Who is the chairperson of House of Mahidol? Vajiralongkom
B
3. Where does Vajiralongkorn live? Grand
Pulace
1. Who is Johann Bernoulli's doctoral stu- dent? Daniel Bernoulli
Which organization employs Jo- hann Bernoulli's doctoral stu- dent and has Alexander R. Todd, Baron Todd's doctoral advisor as member? Russian Academy of Sciences
2. Who Alexander R. Todd, Baron Todd's dectoral advisor? Robert Robinson
C
3. Which organization employs Daniel Bernoulli and has Robert Robinson member? Russian Academy of Sciences
1. Who is the head of state of Kingdom of Bulgaria? Ferdinand of Bulgaria
2. Who is the mother of Ferdinand I of Bul- garia? Princess Clémentine, Princess of Ko- háry
When did the person who taught the mother the head of state Kingdom of Bulgaria begin re- siding in Arathon? June 1852
D
3. Who taught Princess Clémentine,
Princess of Koháry? Jules Michelet
4. When did Jules Michelet begin residing
in Arathon? Jane 1852
1. What country is Franz Xaver Winter- halter a citizen German Empire
2. Who is a relative of Princess Louise of Baxe-Gotha-Alteaburg. Princess Mar- garet of Connaught
Which conflict did the head of state of the country Franz
Xaver Winterhalter is citi- zen of, whose godparent a relative of Princess Louise of Grre-Gotha-Altebbee partici- pate in? Napoleonic Wars
3. Who is the head of the German Empire whose godparent is Princess Mar- garet of Connaught? William I, German Emperor
E
4. Which conflict did William I, German Emperor participate in? Napoleonic Wars
1. Who won Matteucci Medal? Philipp
Lenard
Who both student of the doc- toral advisor the winner of Matteucci Medal and direc- tor or manager Henry Edward Armstrong's 's employer? Henry Enfield Roscoe
2. Who was Philipp Lenard's doctoral ad- visor? Robert Bunsen
F
3. Who is Henry Edward Armstrong's employer? University of London
4. Who is both a student of Robert Bunsen and director manager at University of London? Henry Enfield Roscoe
Table Multi-hop Question Reasoning Graphs and Example Questions from SYNTHWORLA RM. Motifs in our fact triplet graph represent recurring subgraph patterns of triplet facts that form single-hop questions, which can be composed into multi-hop questions. SYNTHWORLDS follows the same multi-hop reasoning structures as the MuSiQue dataset Trivedi et al. (2022).
27

---

Published as a conference paper at ICLR 2026
B.8 QUALITATIVE CORPORA EXAMPLES
Robert Silverberg (Q314553)
Yardley Raleth Quor
Robert Silverberg (born 15 January 1935) author, novelist, science fiction writer, screenwziter and writer whese work is primarily in the science liction genre. His given name Robert and he hegan his professional career in 1955.
Yardley Raleth Quor (born 15 January 1974) is author, novelist, science fiction writer, screensriter and writer whose work is primarily in the science liction genre. His given name Yardley and he hegan his professional career in 1994.
Silverberg was born in Brooklyn and continues to reside there. He speaks English, which is his native language and the language in which he writes. His religion is Judaism. Hc citod Jack Vance and Roger Zelazny as influences on his work.
Quor was born in Myrthwood and continues to reside there. Hc speaks Velthar, which his native language and the language in which he writes. His religion Veltharion. He has cited Caclian Casado and Fythar Rees as influeaces on his work.
Over the course of his career Silverberg has received several awards. He was awarded Hugo Award Best Novella in 1969, Locus Award for Best Fantasy Novel in 1981 and Locus Award for Best Novella in 1988; he received Science Fiction and Fantasy Hall of Fame January 1999 and Damon Knight Memorial Grand Master Award in 2004.
Over the course of his career Quor has received several awards. He was awarded The Storyteller's Legacy in 2008, The Literary Lantern in 2020 and The Storyteller's 's Connection in 2027; be received Exceptional Merit Recognition on January 2038 and The Page Pen Award 2043.
He has also been nominated for numerous literary honors, including Hugo Award Best Novel and Hugo Award for Best Short Story 1970, Locus Award for Best Short Story 1972, the Locus Award for Best Novel in 1973, Hugo Award for Fantasy Novel in 1985, Locus Award for Best Science Fiction Novel in 1987.
He has also been nominated for numerous literary honors, incloding The Prose Pursult and The Wordsmith's Triumph in
2009, Echoes of Words in 2011. the Paper Pathway Award
2012, The Storyteller's Legacy in 2014, The Literary Lantern 2024, The Narrative Jewel 2026, The Storyteller's Connection in 2038 and The Inked Imagination in 2029.
Best Novella
Award
Locus Award for Best Novella 1999 Locus Award Best Novelette in 1990.
Silverberg described by Obilky knih.
Quor described by DataGalaxy.
Mumbai (Q1156)
Crescendo
Mumbai is large urbun centre the continent Asia, It functions as the state capital and is classified as a city, a metropolis and megacity; it is also resognized as a business cluster, reflecting a geographic concentration of interconnected businesses in a particular field.
Crescendo is large urban centre on the continent Nystoria, It functions the state capital and is classified as city, metropolis and a megacity; it is also recognized as a locality and business cluster. rellecting a geographic concentration interconnected businesses in particular field.
locality and
The settlement began in 1507. Over its history Mumbai has been within different sovereign states: it Kingdom England from II May 1661 until 27 March 1668, and later lay in British Raj from 28 June 1858 until August 1947.
The settlement began in 1546, Over ils history Crescendo has been within different sovereign states: it lay in Kytarathia from May 1700 until 27 March 1707, and later lay in Lumeria from 28 June 1897 until 14 August 1986.
Mumbai maintains formal twinning arrangements with several other administrative bodics. is twinned with London, Yokohama, Jakarfa and Busan, and with Honolulu- the partnership with Honolulu began on 20 Jamuary 1970.
Crescendo maintains formal twinning arrangements with several other admini strative bodies. is twinned with Calidore, Celestport, Eldoria and Horizon Bay, and with Jaspis-the the partnership with Jaspis began on 20 January 2009.
Since 2019, Mumbal has been member of the network Creative Cities Network.
Since 2058, Crescendo has been a member of the network SyncSphere.
The city appears range of published sources. is described in the Brockhuus and Efron Encyclopedic Dictionary (a version. edition or translation), in the Sytin Military Encyclopedia encyclopadic dietionary). in Jewish Encyclopedia of Brockhaus and Efron (present ethnereligious group. nation people encyclopodias), and The Nuttall Encyclopaedia literary work).
The city appears range of published sources. It is described in the Dreamt Compilation (a version, edition or translation). in the Factoid Fount (an encyclopedic dictionary), in Qylarans (present cthnorotigious group, nation and people encyclopodias). and The Midnight Library (a literary work).
28

---

Published as a conference paper at ICLR 2026
B.9
PROMPTS FOR CORPORA CONSTRUCTION
Generate Synthetic Names
Give me {f num names
} fictional names an entity that instance the following
wikidata entity(ies):
EXAMPLE INPUT INSTANCE OF INFORMATION
business cluster (goographic concentration of interconnected businesses particular field)
-city (large human settlement)
Your response should be list of comma separated values, eg: 'foo, bar, baz' baz DO NOT include any other text your response. NOT reference anything that already exists in real world.
Generate Synthetic Name with Substring Relation
Given that the following facts related to the entity X are true:
EXAMPLE INPUT RELATED FACTS
Cycle Ridge is a:
big city (city with a population of least 100,000)
city (large human settlement)
cycling city (city designed for bicycle traffic)
Fact: Cycle Ridge location + entity X
Give me fictional name the entity that an example a:
EXAMPLE INPUT INSTANCE OF INFORMATION
public research university (type higher learning institution; research university predominantly funded by public means)
Entity X's name is likely to consist of the names of entities that it connected to. Your response should be single name for entity X one line. DO NOT include any other text in your response. DO NOT reference anything that already exists the real world.
Generate Synth-Mapped and Real-Mapped Pages
System prompt:
You are clear, neutral, and professional writer at the level expected for Wikipedia articles: precise, informative, and fluent, without unnecessary complexity.
User:
Given page title, the page entity information (Wikidata instance of), a set facts between the page title and other entities, write high-quality Wikipedia-styy article.
You will be given the following information:
Page title
Definitions for relation labels in page facts
Definitions for instance-of information about related entities page facts
Page facts
Instance-of information about related entities in page facts
Your task is to produce an article that uses facts faithfully, organizes them into clear prose, and avoids contradictions.
REQUIREMENTS:
Mention every fact faithfully; do add invent information.
Only use proper nouns that appear as entity names in the Page facts section.
Organize into thematic paragraphs.
29

---

Published as a conference paper at ICLR 2026
RULES FOR INTERPRETING FACTS:
Facts are always written as **Subject Relation -> Object**
Interpret the relation **relative the Subject**.
Examples:
*Albert Einstein + student - Nathan Rosen" + Nathan Rosen was a student of Albert Einstein.
* *Albert Einstein student of - Alfred Kleiner + Albert Einstein student of Alfred Kleiner.
FACT RELATIONS AND DIRECTIONS: ALWAYS follow the exact meanings of the relation labels in "Definitions for relation labels in page f'ctts""
NEVER invert direction asymmetric relations (e.g., student/student of, parent/child. advisor/doctoral student, employer/employee).
It can be casy to get this wrong so check the label carefully and preserve its direction exactly.
Do not generate contradictory statements.
Normalize symmetrie relations (c.g., sibling, spouse, collaborator) into one set, de-duplicate, and group entities naturally one or more sentences for readability.
TAADDBBIIIY/WIIIIITT STYLE:
Do not introduce speculative context, dates, regions, concepts.
not repeat the same facts the writing.
NEVER write "instance of" in the writing.
If gender is not provided, always use theytthem/their,
If gender is provided, reflect it through pronoums (he/she/they) and NOT explict fact (to keep the writing natural and fluent).
Vary sentence structure: avoid presenting every an isolated clause or sentence.
Group related facts into paragraphs rather than listing them line by line.
Use connective phrasing for smoother flow (e.g., "Alongside his architectural work, he also painted..."),
When presenting multiple things, use natural connectors such as "among them." " "including." or "as well as instead of flat lists.
Break long cnumerations across sentences for readability.
Make grammatical adjustments (articles, capitalization, punctuation) for natural flow.
Use light connective narration ("They were part of large family...") for readability.
OUTPUT: Return only the plain text article string (no Markdown).
Begin!
Page Title: Yardley Raleth Quor
Yardley Raleth Quor is instance of the following entities:
EXAMPLE VARDLEY
Person
### Definitions for relation labels in page facts IMPORTANT: Use the defintions below correctly understand the page facts.
30

---

Published as a conference paper at ICLR 2026
EXAMPLE INPUT ON YARDLEY RALETH QUOR
recegnition received by a person, organization creative ddtte of birth". date on which the subject was born
received":
"described by source"; work where item is deseribed
"genre": creative work's genre artist's field of work (P101). Use main subject (P921) relate creative works
their topic
person: used property should link "influenced hy": person, idea, informed by person, idea, elc., egg. "Heidegger influenced hy
"given name": first
given
disambiguation por family names
Aristotle"
"languages spoken, written signed': language(s) that person or people speaks, writes or signs, including the
native language(())
'native language"; language or languages a person has leamed from carly childhood
lor": award
received person, organisation
work (inspired
received"
(Property:P666)
"occupation" occapation of person; see
(Property-P1O) "position (Property:P19)
place of birth: most specific known birth location of person, animal or fictional character
religion or worldview": religion of person, organization or religious building. or associated with this subject
"residence": place where the
been, resident
"sex or gender": sex or gender identity of human or animal. For human: male, female, non-binary. intersex, transgender transgender male, agender,
organism,
urganisan. Groups of same gender use
subclass of (P279)
"work period (start)": start of period during which a person group flourished "floruit") " in professional
activity
"wriling langunge" language which writer has written their work
### Definitions for instance-of information
EXAMPLE INPUT ON YARDLEY RALETH QUOR
award": something given person group of people recugnize their merit excellence
ethnic religion" religion defined by the ethnicity of its adherents
"language"; particular system of communication, often named the region or peoples
"ifestyle": interests, opinions, behaviours, and behavioural orientations of an individual, group, culture literary associations
"iterary award":
malee given name": given name usually meant for boys and men
"modern languuge": language in current use
"natural language": language naturally spoken by humans, opposed "constructed" and "formal" languages
religion" social-cultural system
web portal": website integrates applications, processes and
### Page facts (subject relation property object)
IMPORTANT: Facts are always written in form Subject + Relation > Object. The relation definition is expressed relative to the Subject (the entity on the left). Always resolve the meaning by starting from the subject.
31

---

Published as a conference paper at ICLR 2026
EXAMPLE INPUT ON YARDLEY RALETH QUOR
Yardley Raleth Quer
Exceptional Recognition
point in time - 2038-01-01
Yardley Raleth Quor - award received - Literayy Lantern
point time 2020
Yardley Raleth Quor award received - The Page Pen Award
point
2043 (year)
Yardley Raleth Quor award received - The Storyteller's Connection
point
2027 (year)
Yardley Raleth Quor award received The Storyteller's Legncy
point time - 2008 (year)
Yardley Raleth
date of birth 1974-01-15
Yardley Ralcth Quor - described by source + DataGalaxy
Yardley Raleth Quer geure
fiction
Yandley Raleth Quor - given narme - Yardley
Yardley Raleth Quar
Caclian Casado
Yardley Raleth Quor influenced
Fythar Rees
Yardley Raleth Quor - languages spoken. writlen signed + Velthar
Yardley Raleth Quar native language Velthar
Yardley Raleth Quor - nominated for - Echocs of Words
point
(yeur)
Yardley Raleth Quor nominated for Puper Pathway Award
point time - 2012 (year)
Yardley Raleth Quor nominated for The Inked Imagination
point time - 2029 (year)
Yardley Raleth
nominated for The Literary Lantern
point in time - 2024 (year)
Yardley
Quar nominated
point in time 2026 (year)
Yardley Raleth Quor nominated for The Prose Pursuit
point time 2009
Yardley Raleth Quer nominated for The Storyteller's Connection
point time 2038
Yardley Raleth Quor nominated for - The Storyteller's 's Legacy
point
2014 (year)
Yardley Raleth Quor nominated for The Wordsmith's Triumph
point in time - 2009 (year)
Yardley Raleth Quor occupation author
Yandley Raleth Quer - occupation - novelist
Yardley Raleth Quor occupation
fiction writer
Yardley Raleth Quor + occupation - screenwriter
Yardley Quar occupation
Yardley Raleth Quor place of birth Myrthwood
Yardley Raleth Quor - religion or worldview + Veltharion
Yandley Raleth Quer ressilence Myrthwoed
Yardley Raleth Quer - sex or gender - male
Yardley Raleth Quor work period (start)
(year)
Yandley Raleth Quor -+ writing language Velthar
### Instance-of information about related entities in page facts (entity: wikidata entities which the entity is instance of)
EXAMPLE INPUT ON YARDLEY RALETH QUOR
"Caelian Casado": person
DataGalaxx": multimedia database and web
"Fchoes Words": : literary award
"Exceptional Merit Recognition" award
"Fythar Rees", person
"Myrthwood": barough global city. berough port settlement, borough city, barough big city, borough
largest city, borough of metropolis and barough of megncity, city in the sovereign state, city in the constitutinal republic,
city in the Federal republic, city in the superpower, city in the country and city in the historical unrecognized state
consolidated sityccnunty
'Paper Pathway Award": award for best book (by genre) and literary award
"The Inked Imagination", literary award
"The Literary Lantern": award for best book (by genre)
Narrative Jewel": literary
'hhe Page Pen Award": literary award
'The Prose Pursuit": literary award
The Storyteller's Connection" literary award
"The Storyteller's Logacy": literary award
"The Wordsmith's is Triumph": literary
'Velthar": language, modern language and natural language
"Velthriion" ethnic religion, lifestyle religion
"Yardley:: male given name
## Page Content
32

---

Published as a conference paper at ICLR 2026
Assistant:
EXAMPLE OUTPUT GPT-S-MINI
Yandley Raleth Quor (hom 15 January 1974) is an author, novelist, science fiction writer, screenwriter and writer whose work primarily the science fiction genre. given name Yardley and he began his professional career 1994.
Quer was born in Myrthwood and cuntinues reside there. He speaks Velthar, which is native language and the language which he writes. His religion Veltharion. He has cited Caelian Casado Fythar Rees influences his work.
Over the course of his carcer Quor has received several awards, He was awarded The Storyteller's Legacy in 2008, The Literary Lantern 2020 Storyteller's Connection 2027; received Exceptional Merit Recognition on January 2038 and The Page Pen Award in 2043.
He has also heen nominated 2009, Echoes of Words in 201 the Paper Pathway Award in 2012, The Suryteller's Legacy in 2014, The Literary Lantern 2024, Narrative
literury honors, including The Prose Pursuit and The Wordsmith's Triumph in
2026, The Storyteller's
in 2038 and The Inked Imagination
Quer is described by DataGalaxy.
User:
Your job is to now rewrite the answer provided above, but instead of responding directly with the text, transform the text by replacing related entity mentions with linked references using markdown and Jinja-style expressions: '[Entity Text(( (Entity_ID }})
You will be given the following:
1. Entity Reference JSON: A mapping of entity IDs to their name labels for each related entity.
For example, given:
** "sjo
"01397"; *Oh1o stacel",
"039": "United States (sovereign state)"
You would transform:
"She born in Ohio, USA."
To:
"She was born in [Ohio][| Q1397 }D), [USAK[[ Q30 }1)."
IMPORTANT:
Preserve all grammar, punctuation, and readability from the original text.
The text and the spacing outside the links should stay same same.
The text without the links should still be fluent and have proper grammar and punctuation.
Only link proper nouns (capitalized entities like names, places. organizations)
Only create links for entities that exist in the Entity Reference JSON. Never invent IDs assume availability.
general rule thumb, link only the first occurrence an entity the text of article. Links should not contain lcading or trailing spaces within the square brackets, e.g., use '[North America](1 Q49 }1)', not "[North America ][I Q49 }])*
DO NOT add square brackets to terms that not in the Entity Reference JSON, e.g.. "Mercedes-Bent [car manufacturer] founded 1926" incorrect.
BAD EXAMPLES:
"They travelled [South America](( Q30 } when Q30 refers to "United States" "TThey visited [Paris]([{ Q90 }}>" when Q90 not in the provided ISON.
Begin!
## Entity Reference JSON:
33

---

Published as a conference paper at ICLR 2026
EXAMPLE INPUT ON YARDLEY RALETH QUOR
"C186D"; "Velthar",
"$9288"; "Veltaarion",
"818419"; "Kyrthwood",
"G255032" "The Prose Pursuit",
"C295406" "Fythar Rees",
"G312632" : "caelian Casada",
"c549884":
Storytellar's segacy", "C595998": 'Faper Pathway Award",
"CE07354": "The Literazy Lantern",
"g901462" "The Page Award",
"c1056251": "rae Wordsmith's is Triumph",
"C2576795": "The Narrative Jewel",
"g4927937"; "Yaráley",
"C170D8964": "The Taked Imaginatton',
"C17008972": "The SLorybeller's Connection",
"C17008585": "Bchoes of Words",
"g48961500": "exceptieral Merit kecagnition',
"C67311525" "Datatalazy"
New Output:
Assistant:
EXAMPLE OUTPUT
Yardley Raleth Quor (bom 15 January 1974) author, novelist, science fiction screenwriter and writer whose work is primarily in the science fiction genre. His given name is [Yardley]( 04927337 }) and he began his professional
Quor
[Myrthwoo]]c 8419 11) continues to reside
speaks [Velthar][ 01860 1)),
which is his native langunge and langunge in which writes. His religion is [Veltharion][I: Q92 Il cited [Caelian Casado](: 03" 2632 I1) and [Fythar Rees]( 0295406 1:) as influences his work.
Over the course of his career Quor has received several awards. He was awarded [TThe Storyteller's ) 2008, Literary Lantem][ I 0607354 11) 2020 Connection]( ( 01 7008972 11) in 2027; he received [Fxceptional Merit Recognition]( 048961 500 (1) January 2038 [The Page Pen Award]t 0901462 }1) 2043.
Legacy]( I
Storyteller's
He has also been nominated for numerous literary honors, including [The Prose Pursuit) 0255032 31 and [The Wordsmith's Triumph](: Q1056251
[Echoes Words]( 017008985 1:)) 2011, [Paper 11) 2012. The Storyteller's Legacy in 2014, The Literary Lantern in 2024, l'The 2026, Storyteller's Connection 2038 [The
Pathway Award Q595998
Narrative Jewel][ 02576795
Imagination](: 0: 7008964
13) in 2029.
described by [DataGalaxy][ 06731 1526
User:
The previous response was perturbed, where names of entities and timestamp values were intentionally changed.
The actual page is about Robert Silverberg but the above text, it was renamed
Yardley Raleth Quor
Your task is to now restore the above text by replacing all modified names and dates with their original values, using the provided ISON mapping.
The of the text remain exactly as it only change the names and dates that appear in the mapping.
For example given:
"John Smith": "Flon Musk",
"2023 07 07-14"; -2020-07
"Acme Cosp": "Spacex"
34

---

Published as a conference paper at ICLR 2026
and text that said:
"On July 27, 2023, John Smith announced that Acme Corp would begin developing reusable rocket. Smith was leader in the company."
You would rewrite this to: "On July 27, 2020, Elon Musk announced thatSpaceX would begin developing reusable rocket. Musk was leader
company."
IMPORTANT.
The final output must not contain any perturbed entity names and perturbed timestamp values, Outside of the renaming, rest of the should stay the exact same with the links are still included.
Do not rephrase or modify the surrounding text in any way.
Begin!
EXAMPLE INPUT ON VARDLEY RALETH QUOR
"91860"; "Veltàar",
"g9268"; "Veltharion",
"C18419"; "Kyrthwood",
"C255032": "The Frose Pursuit",
"G295406"; "Fythar Rees",
"C312632": "Caelian Casado",
"C513E84": "The Storyteller's Legacy",
"u595998" "Faper Pathway Aword",
"C607354": "The literary "artern",
Page Award", "$1056251": "The Wordsmith 's Triungh",
"C2576795": "The Narrative Jewel",
"C4927937"; "Yazdley",
"C17008964": "The laked Imagination',
"C17008972": "The Storyteller's Connection",
"C1 "C108889"" "Schoes of Wards",
"C48961500"; "Exceptienal Merit Recognition",
"G67311528" "DataGalaxy"
## Rewritten Output:
Assistant:
EXAMPLE OUTPUT GPT-5-MINI
Silverberg (bom January 1935) author, novelist, science writer, screenwriter
work is primarily in the science fiction genre. His given name is [Rohert]([) 04927937 I1) and he hegan his professional career in 1955.
Silverberg was born in [Brooklyn]( 018419 ) and continues to reside there. He speaks ;)), which
[English]
language language which writes. religion Judaism][ [ Q9268 (1). He has cited [Jack Vance]f Q312632 (:) and [Roger Zelazny]( ( 02 95406 1:)
influences
course of his carcer Silverborg has received several awards. He was awarded (Hugo Award Best 1969, Leeus Award for Best Fantasy NovelJ ( 2607354 } 1981 and [Locus Award for Bost Novella](: | 017008972 1)) 1988; be received [Science Fiction and Fantasy Hall of Dumon Knight Memorial Grand Master
Novellal( [ Q549834
) Jamuary
Award]] Q9C 462 in 2004.
He has also been nominated for numerous literary honors, including (Hugo Award for Best NovelJC Q255032 F1) and [Hugo Award for Best Short StoryK : Q1056251 1 1970, Locus Award for Best Short
in 1973, Hugo Award for Best Novella in 1975, Locus Award for Best Fantasy Novel in 1985, [Locus Award for Bost Science Fiction Novell Noveletee[ Q1 70088866
Story K Q17008935 1 1972, the [Locus Award for Best Novelk (1 US95992
Award
Novella
Best
in 1990.
Silverberg is described [Obdlky knih( Q67311526 13).
35

---

Published as a conference paper at ICLR 2026
Validate Page Novelty
System prompt:
are master detective identifies main subject given page. Entity names in pages been replaced with fictional ones to create a parallel fictional world. Your job is to identify the original real-world entity by analyzing patterns in dates, events, relationships, achievements, and biographical details.
User:
The following page about a real-world wikidata entity, some or names been changed to fictional ones. Identify the original main subject.
Hint: The entity currently named in the page is fictional person/organizaaion//inn this fictional entity represents.
need to determine which real
Your response starts with "Thought: where you will methodically break down reasoning process, illustrating how you arrive at conclusions. Then, based on your reasoning, conclude with the prefix "Answer: to present concise, definitive response, devoid of additional elaborations (e.g., "Answer: Barack Ohama").
Page Content:
INPUT SYNTH MAPPED PAGE
Thought:
B.10
PROMPTS FOR MULTI-HOP QA CONSTRUCTION FROM FACTS
Get Page Facts
System prompt:
are an advanced reading comprehension assistant. Your task is to analyze a text passage and extract specific information to fill in triplet templates with placcholders marked as <ANSWER>.
User:
Given page and JSON mapping of partial facts (indicated by <ANS> placeholders), the page content to extract the missing information. Return your output as a list of answers for all triplets. For nested triplets (containing multiple <ANS> placeholders), ensure all parts are supported by the text.
Guidelines:
Return empty array
if no relevant information is found
For nested triplets, only include complete matches where all placeholder values found If multiple valid answers exist, include all of them
Use the relation descriptions to help you understand meaning of the triplet relations. Triplets are always in the form of subject - relation - object.
Example:
Page content:
Pavel Cherenkov held roles as nuclear physicist and general physicist, and over the course his career he received several honors, including the Nobel Prize in Physics, the Order of Lenin, the Order of the Red Banner of Labour and the Hero of Socialist Labour. He was nominated for the Nobel Prize in Physics in 1955 and later received that prize in 1958.
Relation descriptions:
"award received": award or recognition received by a person, organization or creative work "nominated for": award nomination received by person, organisation creative work (inspired from "award received" (Property:P1665
"occupation" occupation a person; see also "field work" (Property:P101), "position held" (Property:P39)
36

---

Published as a conference paper at ICLR 2026
Partial fact templates:
"I1": Pavel Chezenkov >
accupa- <ANS>
"22": Pavel Chezenkov avard received <ANS>
"13":
Pavel Cherenkev -> award received +> <ANS1> AND <ANS1> -> point in time -> AMS2>
"I4":
Pavel Chezeckov >> nomina.ed for <ANS1> AND <ANB1> >> point in Lime
Output:
jaon
"1in: ("auclaar physicist", "physicist")
"$2":
["Hobol prize Physics", "Order Lenin", "Order
Labour", "aero of Socialist Labour"] [[Nobbll Prize in Physica", "1955"]] "m4": NNobel Prize in Physica", "1958"]]
"r3":
Output format:
Return only JSON object with extracted answers. Use empty arrays for triplets where no information found in the text.
Begin! Page content:
EXAMPLE INPUT ON YARDLEY HALETH QUOR
Yandley Raleth Quor (bom 15 January 1974) is an authar, novelist, science fiction writer, screenwriler and writer whose work primarily the science fiction His given Yandley and he began his professional career 1994.
speaks Velthar, language in which he writes. His religion is Veltharion. He has cited Caclian Casado and Fythar Roes as influences on his work.
Myrthword continues
native language
Over the course of his cureer Quar has received several awards. He was awarded The Storyteller's 's Legacy in 2008, The Literary Lantern 2020 and The Storyteller's Connection 2027; received Exceptional Merit Recognition January 2038 and The Page Pen Award in 2043.
He has also been nominated for mumerous literary honors, including The Prose Pursuit and The Wordsmith's Triumph in 2009, Echoes of Words in 2011, the Paper Pathway Award 2012, The Storyteller's Legacy in 2014. The Literary Laniern 2024, The Narrative Jewel 2026, The Storyteller's Connection in 2038 and The Inked Imagination in 2029.
Quor described by DataGalaxy.
Relation descriptions:
EXAMPLE INPUT ON YARDLEY RALETH OUOR
award received: award or recognition received by person, organization or creative work
birth; which subject was
descrihed by source: work where this item is described
genre:
genre an
(P101). subject (P921) 1) creative to
topic
given name: first name or another given name person; values used with the property should not link
disambiguations family
influeaced by: this person, idea, etc. informed by that other person, idea, etc., "Heidegger was influenced by
Aristoule"
langunges spoken, written signed: language(s) that person people speaks, writes or signs, including the native
language(s)
has learned from early childhood nominated for: award nomination received by a person, organisation or creative work (inspired from "award received"
native language: language languages
(Property:PP P166))
occupation: occupation of person; see also "lield of work" (Property:P101) "position held" (Property:P39)
place
most specific known binth
person, animal fictional churacter
religion or worldview: religion of person, organization or religious building, associated with this subject
residence: the place where person or has bocn, resident
gender:
gender identity of human animal. For human: male, female, non-binary, intersex, transgender
female, transgender male, agender, etc. For animal: male organism, female organisan. Groups of same gender use subclass (P279)
wurk period (start): start of period during which a person group (lourisbed (Il. "floruit") ") in their professional
activity
writing language: language in which the writer has written their work
37

---

Published as a conference paper at ICLR 2026
Partial fact templates:
EXAMPLE INPUT YARDLEY RALETH QUOR
"I0": "Yordley Raleth Quor -> owerd received +> <ANSI> AND <ANS1> -> point in
time <ANS2>",
"I1": "Yardley Raleth quor -> date of birth -> <ANS>"
"72"; "Yardley Raleth Ouor
-> descrt by aource -> <ANS>",
<ANS>", "14": "Yardley Raleth Quer given name -> <ANS>",
"I3"; "Yardley Raleth Quor
"15": "Yardley Raleth
influanced
<ANS>"
"76"; "Yardley Raleth Quor -> languages spoken, written or signed +> <ANS>",
"I7"; "Yardley Raleth
native Language
"78": "Yardley Raleth Quor -> nominated for -> <ANS1> AND <ANS1> -> point in time <AMS2>",
"I9"; "Yardley Raleth
occupatien -> <ANS>",
"s10": "Yardley Raleth yuar <> place of birth <ANS>",
religion worldvies <ANS>" "T12": "Yardley Baleth Guor => realdence -> <ANS>",
"M11": "Yardlay Raleth
"T13": "Yardley
Quor sex gender <ANB>",
"T14": "Yardley Raleth Guor => work period (start) +> <ANS>",
"T15": "Yardley Raleth Quor writing language <AMS>"
Output:
Assistant:
EXAMPLE OUTPUT ON GPT-S-MINI
"TO":
["The Storyteller's Legacy", "2008"1,
("The Literory Lantern", "2020"),
(*The Storyteller's Connection", *2027*),
("axcepzional Merit Recogration", "1 January 2038"1,
("The Page Award", *2043*)
"71": ["15 January 1974"),
"12": ("DataGalaxy"),
"I3";
(sscience fictier"),
"74": (YYrrlayy)),
"15"; ""Caelian Casado", "sythar Reas"),
"76": ("elthhrr",,
"T1"; (Veithat"),
"I8":
["The zursuit", "2009"),
["The Wordsmith's Triumph", *2009*), Words", "2011"], ["Papex Pathway Award", "2012"J,
("The Storyteller's hegacy", "2014"),
("The Literary Leatern", "2024"],
(T Nezrative Jewel", "2026"],
("Tee storyteller's connection", '2038"),
(The Trked Imagination",, +2029*]
1,
"I9": ["author", "novelist", "science fictien writer", "screenwriter", "sriter
"], "310": ("Myrttwoed"),
"s11": ["veitharion"),
"T12":
("Myrthwadd"),
"713": Ur
"I14": ("1994"),
"115": ["Veithaz"]
Generate Single Hop Questions from Facts
System prompt:
38

---

Published as a conference paper at ICLR 2026
You are an expert annotator for question answering datasct. Your task to generate high-quality, fact-based questions from the provided facts and relation descriptions.
User:
You will be given a specific graph structure defining how questions are generated from facts (i.e., entities/values (E) and relations (R)). Each E can represent either an entity (person, place, thing) or value (date, number, text).
### Structure:
STRUCTURE FOR GRAPH A (WOULD CHANGE FOR OTHER ORAPHS)
Q1: EI
E2 <ANSI>
Q2: E2 - R2 - <ANS2>
Where E1, E2, E3 are entities/values and R1, R2 are relations.
<ANSI> and <ANS2> are different answers to QI and Q2 respectively.
### Relation Types:
Simple Relations: direct relationship between two entitics/values (entity relation - entity)
Qualified Relations: When relation needs additional context (time, location, role, etc.):
"Entity + [BaseRelation - Qualifier -> Attribute] Value'
Interpretation: The Attribute of Qualifier's BaseRelation to Entity is Value
Example:
Paris - Imayor + Anne Hidalgo + start time] - 2014-04-05'
Mcans: "The start time of Anne Hidalgo's role as mayor of Paris is 2014-04-05"
Question: "When did Anne Hidalgo become mayor of Paris?"
Facts written in the form Subject Relation Object. The relation definition is expressed relative Subject (the entity on the left). Always resolve meaning by starting from the subject.
#### Requirements:
Each question must be natural, fluent English and have single, unambiguous correct answer.
The answer to each question exactly the entity/value tagged with <ANS>.
The subject entity (the entity before - R...) must appear explicitly in the question text to ensure clarity.
Phrase time-based relations naturally ("When did... ?". "On what date... "In what year... ?") <ANS> (date/yearrett.).
matching granularity
Do not copy awkward relation phrasing verbatim more natural form exists ("Where was X born?", not "What is the place of birth of X?").
Do not include <ANS> verbatim in the question text the question must point to <ANS> naturally without revealing
#### Relations
Use the relation wording from the Question Facts as the basis for question, rephrasing only if needed for natural English.
If entity type unclear (e.g.. the relation description lists multiple possible types such as country or region), avoid inventing context (e.g., ask "What shares border with X?" instead
"What country borders X?").
Avoid using the word "entity" in the question text questions should always sound natural.
Output Format: Respond only with questions this JSON format:
json
"Q1": "Quest ion 1",
"Q2": "Question 2",
"ON": Quustiio N
Do not include explanations, comments, or text outside the JSON object.
### Example:
39

---

Published as a conference paper at ICLR 2026
DEMONSTRATION GRAPH (WOULD CHANGE OTHER GRAPHS)
Question Facts:
QI: Stephen Hawking place of birth United Kingdom <ANSI>
Q2: United Kingdom + capital London <ANS2>
Relation Descriptions:
place of birth: most specific known birth location of person, animal or fictional character capital: of government country, province, state other type of administrative territorial entity
Output:
(1
*01": "Where was Stephen Hawking born?", "What is capital United Kingdom?"
Begin!
Question Facts:
EXAMPLE GRAPH
Question Facts:
QI: Jorith Luque educated at -> The Artistic Exchange <ANSI>
Q2: The Artistic Exchange [founded Merith Watts point in time] 1864 (year)
<ANS2>
Relation Descriptions:
EXAMPLE INPUT FOR GRAPH
"educated at": educational institution attended subject
"founded by": founder co-founder of this organization, religion, place entity
"point in time": date something took place, existed or statement was true; providing
time use the "refine date" property (P4241)
Output:
Assistant:
EXAMPLE OUTPUT GPT-SMINI
Json
"Q1": "Where was Jorich Luque educated?",
"g2": what year Marith
Artistic Exchange?"
Generate Multi-hop Questions from Single-hop Questions
System prompt:
You are an expert annotator for question answering dataset. Your task to compose a coherent question from a list of decomposed questions.
Each decomposed question represents one atomic fact relationship.
For example, given the following decomposed questions:
Q1: Which university was Facebook launched in? -> Harvard
Q2: What city is <bridge> Harvard</bridge> located in? - Cambridge
They can be composed together into:
"Which city was Facebook launched in?"
Bridge entities are marked with <bridge> tags, each which should answer decomposed question.
Characteristics of Good Questions:
40

---

Published as a conference paper at ICLR 2026
Fact-seeking: Questions that can be answered with a specific entity concise explanation
Unambiguous: Has single, clear correct answer
Requires comprehension: Demonstrates understanding beyond surface-level pattern matching
Natural language: Uses conversational phrasing that sounds like something person would ask Characteristics Bad Questions:
Poorly formulated: Unclear or grammatically incorrect questions
False presuppositions: Questions based incorrect assumptions
painion-basede Questions seeking subjective judgments rather factual information Not fact-sceking:
Questions that don't clearly request factual information
User:
Your will be given a list of decomposed questions with marked bridge entities. Your task is compose coherent question from them.
IMPORTANTTHE composed question SHOULD NOT include any bridge entities (i.e., those wrapped in <bridge> lags). If composed correctly, the bridge entities should not occur in the composed question.
Requirements for composing questions:
1. Use all decomposed questions: Incorporate information from all decomposed questions. 2. Preserve meaning and answer: Retain the meaning and ensure the composed question's 's answer is the as the last decomposed question's 's answer. Do change the encouraged for fluency and clarity. (Incorrect example: "Which country was Facebook launched in?" this changes both meaning and answer.)
Rephrasing is
3. Keep it concise: Compress as much as possible without losing meaning. Prefer: "Which city was Facebook launched in?" over "Which city has university, which Facebook was launched in?"
4. Two-sentence fallback: If the composed question becomes too long to be coherent, you may split it into two sentences (1 assertion question), connected by coreference. Use only as last resort.
5. Answer alignment:
The composed question must always have the decomposed question's answer.
the last
6. Do not remove necessary details that would make the question ambiguous. This means that non bridge entities should be included
7. Phrase time-based relations naturally ("When did...?", "On what date...?" ?", "In what
year... matching the granularity of the answer (datc/year/etc.).
FAQ:
1. Should paraphrase the question for clarity?
You'te re encouraged paraphrase the question make it simple and coberent long the rephrased question leads the associated answer. You do not need as the decomposed question, because sometimes they are awkward. E.g. replacing "terrain fcature" + "mountain range", "administrative territorial entity" + "statc/city/cte", "parental progenitor" mother/father depending the question context all great
the same phrasing
2. Given the hard choice, do you prefer shorter
coherent question?
Being to parse and understand question more important to us its length. you can't 't retain coherency of the question while keeping short, write longer but coherent composed question.
if entities aren't exactly the same? There are some rare cases which the marked bridge entity doesn't mean exactly the same, but yet are talking about same entity. F.g.
3. What if the question composable
QI: Who was in charge of the US? + George Washington
Q2: Who was the creator of George Washington? + Donald De Lue
Here. Ql's "George Washington" is person, while Q2's refers monument of him. You monument the person in charge the US?" But avoid nonsensical versions like: "Who is the creator of the person in charge of the US?"
compose: "Who the creator
If the composition would be awkward confusing, respond with "No composition".
Output format:
Start your answer with "Thought: ", where you reason through your decision step-by-step.
41

---

Published as a conference paper at ICLR 2026
Conclude with "Question: followed by the composed question (or "No composition") without any modification (i.e., no formatting, no bolding, and no markup) or further explanation.
Examples:
DEMONSTRATIONS BASED ON MUSIQUE
Decomposed question::
Who the President Namihia? Sam Nujoma
Q2: Who suceedoo <bridgc>Sam Najomac/bridge>? Hifikepunye Pohamba
Thought:
Q1 tells that San Nujoma the lirst President Namibia.
Q2 nsks who saccesled chridge>Sam Nujoma-//ridge>,, referring to the person iddetiffied Q1.
Since Sam Nujoma first President Namibin, substitute that desoription into Q2.
Question: Who succeeded the first President of Namibia?
Decomposed questions:
Q1: At what location did Billy Giles die! + Bellast
Q2: What part of the
in? Northern Ireland
Q3: What is the unit of currency in cbridge>NNrthhrn Irelandc/hidge>>? - Pound sterling
Thought
Q1 says Billy Giles died in Relfast.
Q2 tells us Belfast Northern Ireland.
Q3 says Northem Ireland uses Pound sterling.
So we can describe the place where Billy Giles died as "Northem Ireland."
Question: What currency used where Billy Giles died?
Decomposed questions:
Ql: What is MeDoaiiiintiin named after? - McDonald's
Q2: Which state is Horndcan located in? England
When did Lirst <bridge>MeDDnuld Dewndoddddddddddddeeeeeeoe s</briggeoppen <bridgessinglandeehnddggee 1974
Thought:
says McDonaldization named after McDonald's.
Q2 says Horndcon located in England.
Q33 McDonald's
opened England.
Since Horndean is in England, we can use that substitution to generalize the location.
Question: When was the first establishment that McDonaldization is named after opened country Horndean is located!?
Decomposed questions:
- Q1: Who brought Louis XVI style to the court? - Maric Antoinctte
Who the mother <bridge> Marie Antoinette<hrridges??
Q3: In what city did chridgesMaria Theresachridgexiie? Vienna
Q4: When did Napoleon occupy <bridge> Vienm<<bridgge>> + 1805
Thought:
Q1 tells us Marie Antoinette brought Louis XVI style to the court.
Q2 tells her mother Maria Theresn.
Q3 says Maria Theresa died in Vienna.
Q4 Napoleon occupied Vienna
So we can ask when Napoleon occupied the city where the mother of the woman who brought Louis XVI style to the court died.
Question: When did Napoleon occupy the where the mother of the who brought Louis XVT to the court died?
Decomposed questions:
What
Aruba
Q2: What country is Prizeres from? Portugal
Q3: What was the colonial holding <bridge>South Ameriianchriggeeggeeeeed by chridge-Ponugallchnnigee
Brazil
Q4: How many Germans live <bridge-BBaiikkbbiigees - million
Thought:
Q1 tells us Arubu is in South America.
Q2
Portugal.
Q3 tells us Brazil was the colonial holding South America governed by Portugal.
Q4 says 5 million Germans live Brazil.
So ask how many Germans live the colonial bolding Aruba's continent that Prazeres's
country.
Question: many
colonial holding Aruba's was governed by country?
Begin! Decomposed questions:
EXAMPLE INPUT SINGLE HOP QUESTIONS FOR GRAPH
Q1: Where was Jorith Luque educated? The Artistic Exchange
Q2: In what year did Merith Watts found <bridge> The Artistic Exchange</bridge>> - 1864 (year)
42

---

Published as a conference paper at ICLR 2026
Assistant:
EXAMPLE OUTPUT GPT-S-MINI
Thought: QI says Jorith Luque was educated at The Artistic Exchange. Q2 asks when Merith Watts founded that institution and gives the year 1864. Combine by asking when Merith Watts founded the institution where Jorith Luque was educated, avoiding the bridge name. Question: what year Mcrith Watts found the institution where Jorith Luque was educated?
C EXPERIMENT DETAILS
For all experiments with Gemini-2.0-Flast with lemperature=0. For all experiments with GPT-5- mini, the reasoning effort is set to default medium.
For the HippoRAG baselines (Gutiérrez et al., 2025), we used the same model for NER and OpenlE (i.e, GPT-5-mini for GPT-5-mini experiments and Gemini-2.0-Flash. and Gemini-2.0-FiM experiments). We follow (Gutiérrez et al., 2025) and use avidia/NVEmbedd-e (Lee et al., 2025) as the retriever. For IRCo'T, we for maximum of 10 steps.
F1 error bars (Fig. 4) are calculated with 95% bootstrap confidence intervals. For each diffi- culty bucket, resample with replacement 5,000 times, compute the mean F1 (or F1-gap) for each resample, and take the 2.5th and 97.5th percentiles as confidence bounds. Success-rate bars (Fig. 5) calculated using 95% Wilson confidence intervals, computed via stat smodels stats. pwoposs.poooooooooooooo ctt.:rriiiippppottiooooooooooo Python.
with method= 'wilson'
C.1 MULTIHOP QA PROMPTS
For the QA baselines, we follow prior work whenever possible (Trivedi et al., 2022; Gutierrez al., 2024), including the use of prompt demonstrations.
Multihop Question Answering - No Retrieval
System prompt:
As an advanced question answering assistant, your task is to answer the question. Your response starts after "Thought: where you will methodically break down the reasoning process, illustrating how you arrive conclusions. Conclude with "Answer: present concise, definitive response, devoid of additional elaborations. Your answer should be single entity timestamp.
User:
Question: {{ query }}
Thought:
Multihop Question Answering - Reading Comprehension (includes one demonstration)
System prompt:
As an advanced question answering assistant, your task is to analyze text passages and corresponding questions meticulously. Your response starts after "Thought: ", where you will methodically break down the reasoning process, illustrating how you arrive conclusions. Conelude with "Answer: present a concise, definitive response, devoid additional claborations. Your answer should a single entity or timestamp.
User:
43

---

census, population was 505 and Published as a conference paper at ICLR 2026
EXAMPLE DEMONSTRATION
(Spunish: Úttimo cahallo) Spanish comedy directed Edgar starring Fermando Fernán Gómez.
The University Southampion, which founded in 1862 and received Royal Charter university 1952, has over 22,000 students. The university ranked in the top 100 research universities in the world in the Academic Ranking 2010, Southampion in the lop 80 universities the world. The university considers isself one of the 5 reseanch universities U.. Thee university a global repotation research engineering sciences, oceanography, chemistry, cancer sciences, sound and vibration rescarch, computer science and electronics, optoelectronics and textile conservation at the TTottile Conservation Centre (which due to close October 2009.) also home National Occanography Centre, Southampton (NOCS), the focus Natural Environment Research Council-fundes marine research.
World
World University Rankings positioned
University
Township township Champaign County, it contained 202 bousing units.
Neville Stanton is British Professor of Human Factors and Ergonomics at the University of Southampton. Prof Stanton Chartered Engineer (C.Eng). Chartered Psychologist (C. (sycssy and Chartered Ergonomist (C. ErgHH.)), He has written edited forty books and three hundred peor-reviowed journal papers applications the subject. Stanton is a Fellow of the British Psychological Society, Fellow of The Institute of Ergonomics and Human Factors and
Institution Engineering and Technology. including "Nature", He has also helped organisations design new human machine interfaces, such as the Adaptive Craise Control system for Jaguar Cars.
published academic journals
Finding Nemo
Theatrical release poster
Directed by Andrew Stanton
Screenplay by Andrew Stanton Boh Peterson David Reynolds
Story by Andrew Stanton
Starting Albest Brooks Ellen DeGeneres Alexander Gould Willem Dafoe
Music by Thomas Newman
Cinematography Sharon Calahan Jeremy Lasky
Edited by David lan Salter
Production company
Disney Pictures Animation Studios
Distributed by Buena Vista Pictures Distribution
Release date May 30, (2003-05-30)
Running time 100 minutes
Country United States
Language Finglish Budget $94 million
office $940.3
Question: When was Neville A. Stanton's employer founded?
Thought:
Assistant:
The employer Neville A. Stanton University Southampton. University Southampton was founded in 1862.
Answer: 1862
User:
II goid_ anw.annn_iiie distracoo passaggs Question: (f query |1
Thought:
Multihop Question Answering One-Step RAG (includes one demonstration)
System prompt:
As advanced reading comprehension assistant, your task is answer the question given the passages. Your response starts after "Thought: where you will methodically break down the reasoning process, illustrating how you arrive conclusions. Conclude with "Answer: present a concise, definitive response, devoid of additional claborations, explanations or extra information. Your answer should be single entity or timestamp.
User:
44

---

Published as a conference paper at ICLR 2026
SAME READING COMPREHENSION
Assistant:
The employer of Neville A. Stanton is University of Southampton. The University of Southampton was founded in 1862.
Answer: 1862
User:
{ retrieved_passeee ))
Question: {f query })
Thought:
Multihop Question Answering - IRCoT RAG (includes demonstration)
System prompt:
You serve as an intelligent assistant, adept at facilitating users through complex, multi-hop reasoning across multiple documents. This task is illustrated through demonstrations, each consisting of a document set paired with relevant question and its multi-hop reasoning thoughts. Your task is generate one thought for the current step, DON'T generate the whole thoughts once! If you reach what you believe be the final step, start with "So answer is:" Your answer should be single entity or timestamp.
EXAMPLE DEMONSTRATION The Last Horse (Spanish: EI Úttimo caballo) is 1950 Spanish comedy film directed Edgar Neville starring Femando Ferúán Gómez.
The University of Southampton, which was founded in 1862 and received its Royal Charter as a university in 1952, has over students. university
lop 100 research Universities 2010, In 2010, the THES QS World University Rankings positioned the University of Southampton in the top 80 universities in
world Academic Ranking World
The university considers itself global reputation for research into eagincering sciences, occanograpby, chemistry, cancer seienees, sound and vibration research, computer science and electronics, optoelectroniss and textile conservation at the Textile Conservation Centre (which also home the National Occanography Contre, Southampton (NOCS), focus of
top research universities
UK. The university
due close October Natural Environment Research Council-funded marine research.
Stanton Township is township in Champaign County, Illinois, USA. of the 2010 census, its population was 505 and it contained 202 housing units.
Neville Stanton British Professor of Human Factors and Ergunomicss at University Southampton. Prof Stanton is Chartered Engineer (C Eng), Chartered Psychologist Psychol and Chartered Ergonomist ErgHF), has written edited over lorty books and over three hundred peer-reviewed joumaal papers on applications of the subject. Stanton is Fellow Psychological Society,
The Institute Ergonomics Institution of Engineering and Technology. He has been published academic journals including "Nature", He has also helped organisations design new human-machins interfaces, such Adaptive Cruise Control system for Jaguar Cars.
and member
Finding Nemo
Theatrical velease poster
Directed by Andrew Stanton
Produced Grahum
Screenplay by Andrew Stanton Bob Peterson David Reynolds
Story by Andrew Stanton
Starring Albert Brooks Ellen DeGeneres Alexander Gould Willem Dalue
Music by Thomas Newman
Cinemstographyy Sharon Calahan Jeremy Lasky
Edited by David lan Salter
Production company
Disney Pictures Animation Studios Distributed by Bucna Vista Pictures Distribation Release date May 30, 2003 (2003-05-30)
Ranning time 100 minutes
Country United States
Language English
Budget 594 million
Box office $940.3 million
Question: When was Neville A. Stanton's employer founded?? Thought: The employer Neville A. Stanton is University of Soathampton. The University of Southampion founded 1862. So the answer is: 1862.
45

---

Published as a conference paper at ICLR 2026
User:
{f retrieved eeee__assseeesss )
Question: (I query ||
Thought:
C.2
PAGE NAVIGATION PROMPTS
Page Navigation - Links Only
System prompt:
are helpful assistant who can interact with a custom interface to expertly navigate information networks, The interface consists of a page viewer that shows you the current page contents with clickable links to related pages.
User:
<start page>
START PAGE_ TITLE: (f start__age__ittee }}
START PAGE_IDS
start_page_link }]
</start page>
TARGET PAGE TITLE: |I target Saarget_agge_____ page_title title H]
TARGET
BEAGET_PAGE_ITT PAGE_IDS (( target_page_ link_id }}
</target_pag page>>
<instructions>
# Task Instructions
You need to navigate from the given START page to the TARGET page using the provided commands in few steps possible.
You may use any strategy, but your goal to reach the exact page_id id of the TARGET page, not just a similar title.
You have not reached the TARGET page unless the CURRENT PAGE_ ID matches the TARGET PAGE_ID exactly even if the titles seem similar.
You succeed only when CURRENT PAGE_ TARGET BBTT__AGE__DD
For example: TARGET PAGE ID: Elon Musk
CURRENT_PACE JD: Elon
These are different pages. Despite the similarity in names, you must land on the exact page ID complete the task.
# Navigation Tips
Use hub pages (countries, years, broad categories) to bridge between different topics
Go broader before going narrower find shared categories or themes
Look for pages with many links when you need more options
Think about what connects your start and target (time period, location, field, etc.)
This is an interactive process where you think and issue ONE command via function calling, see its result, then think and issue your next command. In each step. please output your thinking that we can follow along. Your thinking should be thorough and it's fine it's very long. </instructions>
IMPORTANT:
Begin!
(I start_observatia )
46

---

Published as a conference paper at ICLR 2026
Page Navigation
Content Links
System prompt:
are helpful assistant who can interact with custom interface to expertly navigate information networks, The interface consists a page viewer shows you the current page contents with clickable links to related pages. Links are displayed in markdown format as Jentity](link))
User:
<start <start_page>
START PAGE TITLE: (f start_page__ittt title }
START PAGE ID: if paaat__page_lin____ }]
START PAGE CONTENT:
Sttart_page_ccmee
}
</start_page>>
TARGET PAGE_ PAGG TITLE: [l target_page_title }]
TARGET PAGE_ID, (f target__aae__ link_id
TARGET PAGE_ PAGR_CONTENT
{{ tttgeet Ltarget_page_come
)
</target page>
<instructions>
# Task Instructions
You need to navigate from the given START page to the TARGET page using the provided commands in as few steps as possible.
You may any strategy, but your goal is to reach the exact page_id j_ of the TARGET page, not just similar title.
have not reached TARGET page unless the CURRENT PAGE matches the TARGET PAGE ID exactly even if the titles seem similar.
You succeed only when CURRENT PAGE ID == TARGET PAGE ID.
For example:
TARGET PAGE_ID ID: Elon_Musk Musk CURRENT_PAGE_II Elon
These differennt pages. Despite the similarity in names, you must land on the exact page ID complete task.
# Navigation Tips
Use hub pages (countries, years, broad categories) to bridge between different topics
Go broader before going narrower- find shared categories themes
Look pages with many links when you need more options
Think about what connects your start and target (time period, location, field, etc.)
This is interactive process where you will think and issue ONE command via function calling, see result, then think and issue your next command. each step, please output your thinking so we can follow along. Your thinking should be thorough and so it's fine it's very long. </instructions>
IMPORTANT:
Begin!
[I Start_observation }
Click Page Tool Definition
Linck_link_ Lo_nk_t__ppa (age self, page_id id: Annotated]
stc, "The 10 of the page to navigate to. The 1B must be link on the LATEST CURRENT PAGR.",
47

---

Published as a conference paper at ICLR 2026
From the Linka the curzent
click
1ink to the next
Backtrack to Page in History Tool Definition
backtzack_to_pege_iin
self, page_ids Annotated]
ste,
"The ID of the page to navigate to. You may only slick on links on the in the AWLEGACIONHHISIORY...
Backtrack apecific page in the navigation history.
D ADDITIONAL EXPERIMENT TABLES
Table 4 shows the results aggregated across all task instances for multi-hop QA. Table 5 shows the results aggregated across all task instances for page navigation.
RM
SM
Model
Baseline
RM (F1)
RM (R@S)
SM (F1)
SM (R@5)
KA (F1)
Closed-book
21.6 [19.5, 23.7]
0.2 [0.1, 0.4]
Reading Comp
88.1 186.4, 89.8)
90.1 [88.5, 91.6]
-2.0
GPT-5-mini
One-step RAG
49.8 (47.1, 52.4]
24.4 [22.0, 26.8]
IRCoT
54.3 [51.7, 56.91
38.1 [35.4, 40.71
$2.2
Closed-hook
19.4 [17.3, 21.4]
0.6 [0.3, 0.9)
18.8
Reading Comp
75.4 (7..0,77.8)
80.3 [78.1, 82.4]
-4.9
Gemini-2.0 Flas-lashh
One-step RAG
37.3 [34.7, 39.9]
56.1
17.2 (15.2, 19.3]
45.1
20.1
IRCoT RAG
46.8 [44.1, 49.4]
60.6
38.3 [35.5, 40.9]
57.5
8.5
Closed-book
17.1 [13.3, 20.2]
1.6
Reading Comp
66.4 [62.4, 69.6)
76.7 [72.3, 80.0]
gpt-oss-20b
One-step RAG
[38.0, 44.1]
20.2 [17.6, 22.8]
IRCoT
30.7 [27.5, 33.91
18.4 [15.7, 21.2]
38.5
Closed-book
21.6 [19.6, 23.7)
1.8 [1.4, 2.3)
19.8
Reading Comp
73.6 [71.3, 75.8)
82.8 [80.8, 84.8]
-9.2
gpt-oss- 120b
One-step RAG
43.5 [41.0, 46.1]
55.9
22.1 [19.8, 24.4]
44.6
21.5
IRCoT RAG
38.3 [35.6, 41.0]
43.5
23.6 [20.5, 26.8]
38.2
14.7
Table 4: Multi-hop QA Performance on SYNTHWORLD-RM/SM. Metrics are scores (with 95% confidence intervals) for answer correctness and Recall (R @) for retrieval. The rightmost column reports the knowledge advantage gap (KA).
48

---

Published as a conference paper at ICLR 2026
Model
Environment
RM
SM
KA
Links Only
50.8 [47.7, 53.9)
19.8 [17.4, 22.4]
31.0
GPT-5-mini
Content + Links
52.3 (5.22
30.6 [27.8, 33.5
21.7
Links Only
36.1 [33.2, 39.1]
15.6 [13.5, 18.0)
20.5
Gemini-2.0-Flash
Content + Links
41.5 [38.5, 44.6)
28.0 [25.3, 30.9]
13.5
Links Only
28.6 [25.9, 31.5]
13.2 (11.2. 15.4)
15.4
gpt-oss-20b
Content + Links
31.5 128.7, 34.4]
22.4 [19.9, 25.1]
9.1
Links Only
39.9 [36.9, 43.0]
16.2 (14.0, 18.6]
23.7
gpt-oss-1 120b
Content + Links
45.6 (42.5. 48.7]
30.3 (27.5. 33.2)
15.3
Links Only Content + Links
45.3 [42.2, 48.4]
17.4 [15.2, 19.9]
27.9
Kimi-K2-Instruct
49.4 (46.3, 52.5)
31.6 [28.8. 34.5)
17.8
Links Only
43.6 |40.6, 46.7]
16.2 [14.0, 18.6)
27.4
Simi-K2-Thinkee
Content + Links
45.4 (42.3, 48.5]
28.5 [25.8, 31.4)
16.9
Table 5: Navigation Success Rate SYNTHWORLD--RMIS Success rates are reported with 95% confidence intervals. The rightmost column reports the knowledge advantage gap (KA).
49


Reviews:
[{"review_id": "43BuTmCynl", "reviewer": "Reviewer_raJh", "summary": "The paper introduces SynthWorlds, a framework to create multi-hop questions and page navigation benchmarks from a knowledge graph e.g. WikiData and assess an LM’s memorized factual knowledge. SynthWorlds builds two parallel corpora on a knowledge graph, one with real-world entities and another with synthetically generated ones, to keep reasoning complexity identical while removing factual familiarity that an LM could utilize in the real-world setting. The paper measures the “knowledge advantage gap” i.e. the LM’s parametric knowledge memorized in its weights, demonstrating this in frontier LLMs like GPT-5-mini and Gemini-2.0-Flash. Even in RAG-based approaches, this gap can exist.", "questions": "1. Why not generate multiple SMs for a given RM? And take the average performance with stderr?\n2. Line 194: The augmented case is unclear. What do the authors mean by \"providing the model with external knowledge acquisition and integration strategies\"? Does baseline just mean CoT prompting with a few CoT examples, and augmented mean RAG prompting? If so, it would be better to just say that.\n3. Line 194: What is \"near random\"? Since these questions are open-ended QA and not multiple-choice, shouldn't P_S^base be near 0? i.e., the LM can't answer those questions at all in the baseline setting?\n4. Line 202: The example of \"rivers remain river-like...\" does not improve clarity. Is there a better concrete example?\n5. Figure 4: I'm confused why the Recall@5 for RM and SM (blue and orange) are not about the same. This is measuring the fraction of times the correct document was retrieved from the corpus, right? Since this is offloaded to an embedding-based retriever, why is the recall higher for RM over SM?\n6. How are the authors calculating error bars in Fig 4, 5, etc.?\n7. Are questions in page navigation baselines also formed using reasoning motifs? Then in Figure 5 it would make sense to club results by reasoning motifs, like in Figure 4. See the other comment about improving Figure 5 axes.\n\n### Typos and editorial suggestions\n1. Line 65: It would be useful to add citations for synthetic data generation that use existing content directly and template-based evaluation. This would parallel the previous example with ToolQA and Zhang et al., Mirzadeh et al. work.\n2. Line 89: \"Surface-form-consistent transformations\" is an uncommon term in the literature, so it would be best to paraphrase this in layman's terms, or define it explicitly here.\n3. Figure 5: Instead of breaking up the plots by expected random walk distance, it would be good to do a line plot, with success rate on the y-axis, expected random walk distance on the x-axis, a blue line for RM and orange for SM. Then you could have 2 plots per model, one for Links and another for Cont+Links. It makes more sense to demonstrate success as a function of difficulty. The current plot highlights Links vs. Cont+Links more, which I don't think is the main point that the authors are driving home. The plot would be similar to that of PhantomWiki (Gong et al.).", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The introduction is extremely well-written, just exquisite writing. I especially liked lines 59 through 76.\n2. For synthetic multi-hop QA data generation to evaluate LMs, the paper presents the right next step by building synthetic datasets on top of the real-world Wikipedia knowledge graph that captures the complex interconnectedness and messiness. I think this paper is exciting for the field of multi-hop reasoning evaluation.\n3. SynthWorlds contains a set of reasoning motifs, e.g., constraints and joins, that are not just logical compositions. This makes the questions more realistic; however, the question difficulty is unclear. I guess this is a tradeoff: in contrast to PhantomWiki that had logical compositions only, the question difficulty (reasoning steps) was simply the sum of the number of hops.", "weaknesses": "1. The authors heavily talk about determining knowledge gaps as the main contribution, but mention task reasoning difficulty in contribution 1 and Sec 3 line 205 as a main contribution. It is clear how SynthWorlds is evaluating knowledge gaps, but not clear at all how to determine task reasoning difficulty. It would be good to discuss this, since it's a main contribution of the work.\n2. A major limitation is that SynthWorlds requires a knowledge graph to exist for a document corpus (Wikidata graph in the case of Wikipedia). I would be curious about what the authors think could work when we only have access to Wikipedia but not Wikidata.\n3. Because the questions and documents from the graph G_facts are LM generated, there is no clear way to automatically verify full correctness. And what if the questions contain parametric knowledge from the LM generating the questions? There is discussion in App B.2 about human validation; however, I'm interested in automatic validation techniques or correct-by-design question/document generation.\n4. Since the SM is created once, future LLMs would memorize it once uploaded to the internet. Is there a clear pipeline for creating synthetic worlds from scratch and any knowledge graph? Several synthetic benchmarks create knowledge graphs from scratch (although not as realistic as WikiData), and LLM memorization is not helpful by design, e.g., PhantomWiki and GSM-Infinite.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1voqYtYD0o", "reviewer": "Reviewer_2dyf", "summary": "SYNTHWORLDS introduces a novel framework to disentangle reasoning from factual knowledge in language models by constructing parallel corpora: a real-mapped world with familiar entities and a synthetic-mapped world with renamed entities, preserving identical reasoning structures while obscuring parametric knowledge. Through case studies on multi-hop question answering and page navigation, experiments reveal a persistent knowledge advantage gap, where LMs like GPT-5-mini and Gemini-2.0-Flash perform better in RM settings even with retrieval augmentation, highlighting reliance on memorized knowledge. This scalable, automated approach enables controlled evaluation, offering datasets and tasks that facilitate precise analysis of reasoning abilities, with contributions including a generation pipeline, public corpora, and empirical insights into LM limitations in novel environments.", "questions": "- The paper defines the Knowledge Advantage Gap as a core metric. To what extent can this metric be generalized to other domains beyond the structured knowledge of Wikidata (e.g., mathematical reasoning or code generation)?\n- If the framework and its generated datasets become widely adopted, what prevents future LMs from being trained on generations of SYNTHWORLDS-style corpora?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- SYNTHWORLDS uniquely separates reasoning complexity from parametric knowledge through parallel corpora.\n- The framework is fully automatic and scalable, leveraging knowledge graphs (e.g., Wikidata) to generate large, interconnected corpora without manual curation.", "weaknesses": "- This paper proposes the challenges in distinguishing reasoning from reciting for controlled evaluation. However, we don't know whether the paper really solve this problem. I mean, if your scores can precisely reflect the real reasoning abilities of LLMs, then you should observe a correlation between human preference (e.g. LM Arena Rankings on Reasoning) and your scores.\n- This paper mentions two kinds previous approaches on controlled evaluation: (1) curation of “clean” evaluation sets and (2) synthetic\ndataset generation, but the authors did not compare their method with these methods in experiments. I know you may argue you did not know how to measure the effectiveness of evaluation methodologies on controlled evaluation. You may refer to this work's settings for comparison experiments [1].\n- Typo: Line 102: \" (§5).Across\" has no space between \")\" and \"Across\".\n\n\n[1]  Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysis. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3809–3822, Vienna, Austria. Association for Computational Linguistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VXV6bkbI5m", "reviewer": "Reviewer_HETT", "summary": "SynthWorlds proposes a fully automatic, scalable framework to disentangle language-model reasoning from memorized factual knowledge by constructing parallel corpora and tasks in two mirrored worlds—real-mapped (RM) and synth-mapped (SM)—and measuring a knowledge advantage (KA) gap, i.e., the performance difference between RM and SM conditions.", "questions": "1. Do the findings persist across model sizes within a family? Please discuss whether scaling reduces the gap via better retrieval or stronger reasoning.\n2. Can you include at least one additional domain (e.g., biomedical/software) and report the results?\n3. What are the dominant RM vs. SM failure modes?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper offers a novel and well-motivated formulation of disentangling reasoning from memorized factual knowledge by constructing paired “real-mapped” and “synth-mapped” worlds with matched structure, and by introducing a formal knowledge-advantage metric to quantify the contribution of parametric knowledge.\n2. The data-generation pipeline is fully automated; difficulty is explicitly controlled; the evaluations span parallel multi-hop QA and page navigation tasks; and the comparisons cover closed-book, single-step RAG, and iterative reasoning–retrieval (IRCoT + RAG) across multiple models with appropriate metrics, yielding credible, interpretable results.", "weaknesses": "1. Conclusions are based on two property models; it is unclear how KA scales with capacity or how post-training techniques affect KA.\n2. The corpus and questions derived largely from one source (Wikidata) may yield relation distributions and writing style that favor certain generalization paths, which may disrupt the evaluation.\n3. Although the benchmark proposed in the paper can quantify KA, the quantification results do not seem surprising. Moreover, it is not yet shown that improvements on SynthWorlds translate to gains on standard web or scientific tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xkyktw4gBL", "reviewer": "Reviewer_Ra5g", "summary": "This paper investigates the real capability of reasoning of LLMs when isolating parametric knowledge. They design a simulated world from the Wiki so that both simulated world and real world share the same degree of difficulty. After that, they conduct detailed experiments and analyze the effect of reasoning and knowledge.", "questions": "- For page navigation, is the content simulated or from the real world? I wonder if you could report the difference between the minimum steps needed to reach the final page and the steps LLMs take to reach the final page to see whether there is a potential shortcut.\n- Could you explain why the F1-Score of Multi-hop QA of SM is higher in reading comprehension, since it is counterfactual (maybe) and should not be higher than RM.\n- Coule you train an embedding model from scratch using you real-world corpora and simulated-world corpora separately to ablate the effect of LLM retrievals since they are constructed from the same source?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper provides a detailed construction pipeline for a simulated world from the real world, including entity selection, question construction, and human annotation.\n- This paper rigorously analyzes the effect of reasoning disentangled from parametric knowledge, using commonsense/multi-hop QA and page navigation.", "weaknesses": "- All the experiments are based on closed-source models. Some open-sourced models should be included, e.g., DeepSeek-V3.1 and K2, and various model sizes should be considered for generability, e.g., Qwen3 series.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)