PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation
Authors: Albert Gong
Kamilė Stankevičiūtė
Chao Wan
Anmol Kabra
Raphael Thesmar
Johann Lee
Julius Klenke
Carla P Gomes
Kilian Q Weinberger
Venue: ICML 2025 poster (2025)
Keywords: logical reasoning
information retrieval
data memorization
data leakage
LLM evaluation
retrieval-augmented generation
agentic workflows
LLM agents
multi-hop reasoning
data contamination
large language models
Primary area: deep_learning->large_language_models

Abstract: High-quality benchmarks are essential for evaluating reasoning and retrieval capabilities of large language models (LLMs). However, curating datasets for this purpose is not a permanent solution as they are prone to data leakage and inflated performance results. 
To address these challenges, we propose PhantomWiki: a pipeline to generate unique, factually consistent document corpora with diverse question-answer pairs. Unlike prior work, PhantomWiki is neither a fixed dataset, nor is it based on any existing data. Instead, a new PhantomWiki instance is generated on demand for each evaluation. We vary the question difficulty and corpus size to disentangle reasoning and retrieval capabilities, respectively, and find that PhantomWiki datasets are surprisingly challenging for frontier LLMs. Thus, we contribute a scalable and data leakage-resistant framework for disentangled evaluation of reasoning, retrieval, and tool-use abilities.

Phantom Wiki: On-Demand Datasets for Reasoning and Retrieval Evaluation
Albert Gong" Kamilé Stankevičiüte 12 Chao Wan" 1 Anmol Kabra
Raphael Thesmar' Johann Lee!
Julius Klenke!
Carla P. Gomes"
Kilian Q. Weinberger'
Abstract
tive questions. On benchmarks like DROP (Dua et al., 2019) and MMLU (Hendrycks et al., 2020), LLMs demonstrate advanced reading comprehension and general reasoning capabilities, both necessary for intelligent agents. When augmented with retrievers (Muennighoff et al., 2022) and tools (Patil et al., 2023), language models seem to already possess a strong ability for accessing external datastores and knowledge bases.
High-quality benchmarks are essential for evalu- ating reasoning and retrieval capabilities of large language models (LLMs). However, curating datasets for this purpose is not a permanent solu- tion as they are prone to data leakage and inflated performance results. To address these challenges, propose Phantom Wiki: pipeline to generate unique and factually consistent document corpora with diverse question-ansee answerr pairs. Unlike prior work, Phantom Wiki neither a fixed dataset, nor is it based on any existing data. Instead, a new Phantom Wiki instance is generated on demand for each evaluation. We vary the question dif- ficulty and corpus size to disentangle reasoning and retrieval capabilities respectively, and find that Phantom Wiki datasets are surprisingly chal- lenging for frontier LLMs. Thus, we contribute a scalable and data leakage-resistam framework for disentangled evaluation of reasoning, retrieval, and tool-use abilities.
However, it is unclear to what extent these models rely on their internal knowledge, which can easily become out- dated, versus their reasoning and retrieval abilities. For example, consider the question "What is the date of birth of Wolfgang Amadeus Mozart?". Since this fact is contained within LLM pre-training data, asking LLMs this question cannot provide reliable insight whether the answer was deduced, retrieved or recalled. At the same time, existing approaches that perturb Wikipedia facts (Cohen et al., 2024; Meng et al., 2022; Elazar et al., 2021; Ho et al., 2020) to con- struct new question- answer pairs face challenges of ensuring factual consistency across articles. For example, changing Mozart's date of birth to 2025 would also require modifyyng Beethoven's 's article to erase the fact that Beethoven might have met Mozart in 1787!
1. Introduction
One could hope to isolate reasoning from factual knowl- edge using mathematical or logical reasoning benchmarks. Unfortunately, such benchmarks are not entirely reliable as indicators of reasoning performance either. On GSM8K, dataset of grade school math problems, Mirzadeh al. (2024) report that frontier models perform significantly worse with minor or even meaningless alterations to the test data indicating these models are vulnerable to overfit- ting at best and exact memorization at worst. To ensure fair comparison, LLMs need to be evaluated a way that does not depend on any particular dataset instance.
Designing agents that can perform complex reasoning while interfacing with large-scale, dynamic corpus- like Wikipedia-is a long-standing goal in the field of natural language processing (Feldman & EI- El-Yaniv, 2019; Min et al., 2019). Such goal may be within reach given the impressive capabilities of recent language models, which are all trained on internet-scale data. For example, the ability of L.MSs to solve math problems on GSM8K (Cobbe et al., 2021) and AlphaGeometry,
mathematical olympiads (AlphaProo) 2024) could bode well for agents answer highly quantita-
Following this philosophy, we develop Phantom Wiki. At the click of button, PhantomWiki generates synthetic, fictional universe of characters along with set of facts about them. We reflect these facts in large-scale corpus of templated articles, mimicking the style of fan-wiki websites. Then we generate question-answer pairs about the universe, encapsulating the types of multi-hop questions commonly considered in the question-answering (QA) literature.
Equal contribution 'Department Computer Science, Cornell University, Ithaca, New York, USA 'Department of Computer Science and Technology, University of Cambridge, Cambridge, UK. Correspondence Albert Gong <agong @cw. wow.e..e........................ Kamile Stankevičiūté <ks830@ cam.ac.uk>. Chao Wan www862@conell.cduus. Anmol Kabra <anmol ccc.o.ell.....
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

PhantomWiki: On-Demand Datascts for Reasoning and Retrieval Evaluation
In-Context
RAG
Agentic
(Ho et al., 2020), HybridQA (Chen et al., 2020), MuSiQue (Trivedi et al., 2022), RepLiqA (Monteiro al., 2024) among many others. These benchmarks are typically cu- rated by crowdsourcing questions based on excerpts from sources like Wikipedia, crowdsourcing the text corpus itself, or leveraging structured knowledge bases like Wikidata to partially automate the question-generatinn process (Agar- wal et al., 2020; Ye al., 2022). Such benchmark cura- tion is limited it is time-consuming and expensive, and the released static benchmarks are prone to eventual data contamination and memorization through internet-scraping (Trivedi et al., 2022; Monteiro et al., 2024). This has been partially addressed by dynamic frameworks such as Real- Time QA (Kasai al., 2024) and FreshQA (Vu et al., 2023); however, those frameworks are focused on real-time fac- tual knowledge, while Phantom Wiki's focus is evaluate bowwledge-indepedeere reasoning and retrieval.
Reasoning steps
15
10
Fl Score
5-
10a
10
10
10a
10 10
10
10
Number of documents
Figure 1. Evaluating LLM capabilities with Phantom Wiki, We tune the reasoning and retrieval difficulty by the number of rea- soning steps and documents, respectively. In this representative example Llama-3.3 3.3 (other LLMs being similar), the top- right regions in all settings grow darker, indicating that the F1 score plummets as both reasoning and retrieval complexity in- crease. Note that co-context prompting has sharp cut-off' along the x-axis at Llama-3. 3.B-70B's 's context length limit.
Retriever- and Tool-Enabled LLM Agents. Modern LLM benchmarking places increasing importance on innovations like longer supported context lengths (Hsich et al., 2024; Wang et al., 2024; Zhang et al., 2024; An et al., 2023; Bai et al., 2023; Li al., 2023), the ability to reference external documents through retrieval-augmeetedd generation (RAG) (Lewis et al., 2020; Karpukhin et al., 2020; Guu et al., 2020;
We design PhantomWite as data contamination-rrsistaan tool for controlled stress-testing of LLM reasoning and re- trieval capabilities. (i) On-demand, fully algorithmic genera- tion of previously unseen universes- complete with articles and question-answer
the need of expensive data curation and annotation, encourages the use of fresh dataset instances, and therefore reduces the risk of data leakage. (ii) Controlling the difficulty and structure of the questions and using logic programming methods to find an exhaustive list of solutions allows for comprehensive testing of both multi-step and multi- -halti-bancchae reasoning, pinpointing the limits of I.I. reasoning capabilities. (iii) Adjusting the universe size-and therefore the size of the reference document corpus-facilitan testing in a range of LLM settings, such as in-context learning in both short and long context windows, Attrivall-uggment
Petroni et al., 2020; Saad-Falcon et al., 2023; Jin et al., 2024;
Hsia et al., 2024; Mao et al., 2024; Rau et al., 2024; Shi
et al., 2023; Borgeaud et al., 2022; Tang & Yang, 2024; Su et al., 2024), and the use of other external tools like search. Many of these benchmarks similarly require at least partial manual curation, do not provide unique corpus, or, in case they are fully synthetic and customizable (Hsieh et al., 2024), are limited in their scope.
Many LLM agent benchmarks (Yao et al., 2024; Lattimer et al., 2024; Shridhar et al., 2020; Zhou et al., 2023) focus on binary-reward tasks (such as booking a flight, making purchase), tasks that require navigation across multiple pages, or meeting user preferences. Closer to work are tool-augmented question-answerigg benchmarks. For example, ToolQ (Zhuang et al., 2023) introduces number of tasks in a range of domains and (combinations of) pre- defined tools (e.g. calculators and databases) to evaluate whether a model answers questions using tools or recalls a memorized answer. Compared to this work, Phantom Wiki is more focused long-lasting methods to stress-test long- context logical reasoning, uses limited tools, and resists data contamination without requiring dataset curation or manual verification of generated questions.
generation (RAG), and external tool use (as is common in agentic workflows).
Our evaluation on Phantom Wiki confirms that the bench- mark presents significant challenges for all state-of-the-ar LLMs that we used. As a representative example, in Fig- ure we plot F1 scores for L.ama-3.3-70B for in-context, RAG, and agentic settings at varying reasoning and retrieval cecrcuutes-demmossiii
the usefulness of Phantom Wiki despite the simplicity of its text corpus and otherwise triv- ially solvable questions.
2. Related Work
Question-Answerinn Benchmarks. A standard technique to evaluate machine reading comprehension is through question-answering benchmarks such as SQuAD (Ra- jpurkar, 2016), HotpotQA (Yang et al., 2018), ComplexWe We- bQuestions (Talmor & Berant, 2018), QAngaroo (Welbl et al., 2018), DROP (Dua et al., 2019), 2WikiMultiHopAA
Logical Reasoning Benchmarks. Logical reasoning tasks have become central to LLM evaluation and have garnered significant attention in recent time (Zhu et al., 2023). How- ever, many existing benchmarks do not disentangle the eval- uation on logical reasoning with other abilities such as nat-
2

---

PhantomWize On-Demand Datascts for Reasoning and Retrieval Evaluation
Who is <...> ? the of <...> the of <...>
==
o
looo
the person whose is
1M
+ {nephew> + {friend} + {hobby}, (*)
(3) Generate questions using a context-free grammar
(1) Generate random universe of size
Q:
Who is the nephew of the friend of the person whose hobby is birdwatching?
David Smith
?- nephew(X2, Y), fxiend(X1, x2),
The friend David John Harper.
A: Y=
The hobby of David is birdwatching.
hobby(X1, x))
(2) Create the document corpus for the universe
(4) Use a logic program to deduce the answers
Figure 2. Overview of the Phantom Wiki pipeline.
ural language inference and commonsense reasoning (Sak- aguchi et al., 2021; Zellers et al., 2019; Sprague et al., 2023). Another line of work focuses on the synthesis of datasets containing a variety of logic reasoning tasks (Tafjord et al., 2020; Saparov He, 2022; Liu et al., 2020; Han et al., 2022; Weston et al., 2015). Closer to our work, CLUTRRR (Sinha al., 2019) crowd-source short stories about individuals related through family graph and ask models deter- mine the relationship between two individuals. We extend this idea to a fully-synthetic on-demand dataset generation pipeline with more diverse question types of controllable difficulty, and enable evaluation in the modern LLM era at long-context and multi mult--oocument scale.
3.1. Generating PhantomWike Universe
The first stage of the PhantomWiki pipeline generates random universe of characters as well as the document corpus describing it, as illustrated in Figure 2, (1-2).
Each character a Phantom Wiki universe is described through its social relationships and personal facts as shown in Figure 2 (I). For the social relationships, we first generate family trees, following the family tree generator of Hohenecker & Lukasiewicz (2020). We iteratively pick person and generate their parent or child based on various constraints, until the user-specified universe size of n people is reached. The user can also specify other hyperparameters like the number of trees, their maximal depth, and the maximal number of offspring for each person. In addition the family trees, we generate friendship graph using the Erdös-Rees model (making two people friends with some fixed probability, typically controlled by the desired average number of friendships.)
Generating Characters.
3. Phantom Wiki Construction
PhantomWik is at its core on-demand random generator of fictional worlds. Similarly to the wiki hosting services popular film, video games, and literature, we represent these fictional worlds through Wikipedia-like biographical entries about their characters. We then test the model's retrieval skills and its understanding of the fictional world through accompanying set of automatically generated question-answer pairs.
Generating Facts. Next, we generate personal facts for each person in the PhantomWil universe. Names are as- signed during the family generation procedure, with the first name sampled based on the character's gender and the name based the family tree, resulting in 15M full names in total. We also add dates of birth in a way that is con-
For example, stardewvalley canddwvalley.faddom.comm fandom.com harrypot- ter. fer.fandom.com com.
Forr example, the number of offspring of person has be smaller than some threshold, parents of the people the maximal tree level will not be generated, etc. experiments, but Phantom Wiki
"We unique names also supports repeated names.
3

---

PhantomWike On-Demand Datascts for Reasoning and Retrieval Evaluation
sistent with the existing family relations, and assign each person a job and a hobby that we uniformly sample from over 300 and 600 options respectively.
does David have?"), and combinations of all three ("How many friends does the brother of the person whose hobby is birdwatching have?") (See Appendix B for the full CFG.)
Generating Articles. Given all relevant facts for each per- son, we convert them into articles using pre-defined tem- plates, e.g. "The job of David is a farmer. The hobby of David is birdwatching." (see Figure 2. (2)). This construc- tion conveys the necessary information while keeping the articles short (about 160 tokens average) and thus sup- ports larger effective universe size. The resulting articles are the only component of Phantom Wiki available to the model during its evaluation (see Appendix E for an example of a small PhantomWiki corpus).
Generating Answers. To ensure that the answers to the sampled questions are verifiably correct, we represent our generated universe in Prolog, a logic programming language (Sterling & Shapiro, 1994). Each Prolog pro- gram consists of a set of facts known about the world such as hobby ( ("David", "birdwatching") and set of rules defining how facts are related to each other, such as nephew (X, Y) :- sibling (X, A), son (A, Y). The Prolog program uses these facts and rules to deduce the exhaustive set of answers to its queries (i.e., the UFG-generated questions). For example, question "Who the nephew of the friend of the person whose hobby is birdwatching?" ?" corresponds to the three-statement Pro- log query ?- nephew (X2, Y), friend (X1, x2). hobby (x1, , "bi seirddatton ch nnin",, which returns all people satisfying these constraints in the Phantom Wiki uni- verse (see Figure 2 (4)).
While is possible to improve on the minimalistic tem- plated articles using LLM-based rephrasing (see e.g. Shao et al. 2024), this poses the challenge of guaranteeing factual correctness without additional costs or external supervision. This has been supported by our experiments on articles rephrased using Alamma-3.3-7BB (see Appendix D.2) while we did not observe any noticeable differences in perfor- mance trends, we identified factual errors (hallucinations) in the rephrased articles. Using templates therefore keeps PhantomWik, dataset cheaper (no computational or API cost), faster (no latency from LLM queries), and fully factu- ally consistent; however we do see seesistency-presenvee LLM rephrasing methods as an exciting future direction.
To construct the Prolog queries automatically, we modify the CFG algorithm to generate both the question and query templates in parallel. We note, however, that the queries are separate from the final Phantom corpus and question- answer pairs, and the answers returned by the Prolog pro- gram should be held out as part of the evaluation procedure.
3.2. Generating Question-Answer Pairs
3.3. Phantom Wiki Complexity
In the second half of the Phantom Wiki pipeline, we gener- ate a set of questions with verifiable answers, as shown in Figure 2, (3-4).
The goal of Phantom is to generate memorization- resistant evaluation datasets that are challenging in both reasoning and retrieval aspects. In this section, we discuss our conceptual and practical design choices that help us achieve this goal.
Generating Questions. We implement automatic question generation through a context-free grammar (CFG, Hopcroft et al. 2001) of question templates, which we then use to sample complete questions. For example, the question tem- plate "Who is the <relation>> of <name>?" can be used to sample the question "Who is the friend of David?" (see Figure 2. (3)). The main advantage of using a CFG is that it efficiently and systematically obtains all possible composi- tions of questions for some recursion depth d. For instance, the following subset of our context-free grammar:
Universe Space Complexity. To ensure that our evalua- tion with PhantomWiki is memorization and data leakage- resistant, we first show that the space of possible universes is sufficiently large to generate enough unique instances. Observe that the number of possible friendship assignments grows at the rate of O(2n²) ) (Flajolet & Sedgewick, 2009, Ex. II.5) as the number of individuals n in the universe in- creases. Similarly, assuming each individual is assigned one fact from each category (job, hobby, etc.), the num- ber of possible fact assignments grows at the rate O(c"), where is the total number of choices across the categories. Phantom Wiki thus samples corpus from O(2"² c²) possi- ble universes, which leads to diverse datascts optimal for data leakage-resistant evaluation. We note that as future work Phantom Wiki could be extended to increase this diver- sity, e.g. by adding a temporal dimension of events.
S Who is R
R > the <reltionn> of R'
R' - R <name>
can lead to questions ranging from "Who is the friend of David?" to "Who is the nephew of the friend of the brother of David?" as d increases. In addition to these nested com- positions, our CFG also supports questions about personal attributes (c.g. "Who is the person whose hobby is bird- watching?") ?"), aggregation questions ("How many brothers
Reasoning Complexity. The CFG enables us to recursively compose templates that lead to complex reasoning ques-

---

PhantomWiki On-Demand Datascts for Reasoning and Retrieval Evaluation
tions. Observe that our CFG in Appendix produces (dd) question templates as the recursion depth d increases. More-
4.1. Evaluation Setup
We generate PhantomWiki instances with n ranging from 50 to 10K-a universe size for which the total length of articles exceed context lengths of all existing LLMs. For evaluation, we only provide the articles to the LLMs, not the Prolog database or the generated graphs. To ensure that our findings are not tied to any specific Phantom Wiki instance, we use 3 random dataset seeds for each configuration. Cre- ating Phantom Wiki instances with different random seeds leads entirely different combinations of names, relations, and personal facts. In each instance, we generate question templates with maximum recursion depth d = 20, for a total of 50 templates. We sample 10 questions for each tem- plate, yielding a total of 500 questions per Phantom Wiki instance. As shown in Figures 5 and 6 (Appendix B), these questions have varying difficulty and number of answers. Accordingly, we prompt the L.MMs to predict all answers as comma-separated list and measure correctness with the answer-level F1 score.
over, we can increase the difficulty of each template by increasing the number of reasoning steps. For example, substituting <relation> with nephew in a template adds two reasoning steps (nephew (X, Y) :- sibling (X, A) , son (A, Y) ). since Phantom Wiki articles only con- tain immediate family relationships like sibling and son. In contrast, substituting <relation> with second cousin would lead to five reasoning steps. As we will show in Section 4, Phantom Wiki questions are sufficiently complex to evaluate reasoning capabilities of state-of-the-ar LLMs. We further note that Phantom Wiki's CFG can be easily extended to support more question types like comparison and multiple- constraint questions.
Retrieval Complexity. To assess model's retrieval capa- bilities, we increase the universe size so that the document corpus exceeds the model's context length-the this makes retriever necessary to answer questions correctly. For state- of-the-art LLMs with a context length of 128K, such as Ope- nAl's GPT-4o and Meta's Llama-3.3-708: this corresponds to PhantomWiki universes of n 2 1K. This increases to z 3K for Google's Gemini-1.5-Flast with context length IM. Further scaling n leads to further increase in retrieval difficulty. In Table 1, we show that Phantom Wiki is well- suited for generating universes of this size on standard CPU hardware: generating questions with recursion depth d = 10 for size n = 100K-well beyond any existing LLM's con- text length takes just minutes on 8 Intel Cascade Lake CPU cores. Moreover, we conveniently generate in- stances of 1M, which is the scale of Wikipedia's corpus of 2 million biographical entries.*
4.2. Models and Prompting Techniques
We test both open- and closed-source LLMs, namely OpenAl's GPT-4o (OpenAI, 2024), Google's Gemini-1.5- Flash (Gemini Team, Google, 2024), and the instruction- tuned version of Meta's Llama-3.3-700 model (Llama Team AI at Meta, 2024). We also evaluate DeepSeckAl's DeepSeek-Re PepSeekk-1-3255 (DeepSeek AI, 2025) distilled with Qwen-2.5-32B (Qwen Team, 2024), which an open- weights LLM trained on reasoning trace datasets. We prompt each LLM with the following techniques, broadly grouped in three ways:
In-Context Prompting. This technique includes the full document corpus as part of the prompt. We implement this technique with two strategies: ZFROSHOT, where the docu- ment corpus is immediately followed by the question, and Chain-of-Thought (CoT) (Wei et al., 2022), where we addi- tionally include a few examples of step-by-step reasoning leading to the correct answer. See Appendix C. for the prompt details.
Table 1. Runtime breakdown of generating a Phantom Wiki in- stance for facts, articles and questions for universe sizes n.
Total Runtime
Facts
Articles
Questions
10a
0.97 s
0.46
0.07 s
0.44 s
103
2.86
0.90
0.59 s
1.37 s
1
20.91 s
5.38
5.87 s
9.66 s
10"
5.57 m
0.81
0.97 m
3.79 m
RAG Prompting. This setting augments generation with retriever as pioneered by Lewis et al. (2020). Due to its synthetic nature, the documents from PhantomWiki do not necessarily match real- world corpora, making neural retrievers a poor fit for evaluation. Instead, we use the BM25 retriever, which uses keyword matching, and search for the top-4 most relevant documents for each question. Next, we incorporate these retrieved documents into the model's prompt. Finally, we add in the same ZEROSHOT and CoT prompts as In-Context Prompting. See Appendix C.2 for details about our retrieval setup.
106
3.86 h
9.47 m
11.77 m
3.51 h
4. Experimental Validation
We evaluate reasoning and retrieval capabilities of several frontier LLMs using PhantomWike by decomposing their performance over questions of varying difficulty and uni- verses of varying sizes.
'hups://en. as of January 30, 2025.
Agentic Prompting, REACT (Yao et al., 2022) is prompt-
5

---

ChantomWiki: On-Demand Datascts for Reasoning and Retrieval Evaluation
Table F1 scores (in %) for various LLMS and prompting techniques. report mean = standard error across 3 dataset generation seeds (except for GPT-4o with CoT and REACT due to cost constraints), and indicate the highest Fl score for each in bold. In-Context prompting infeasible for n 5K the corpus cannot be fully included in the context.
Universe Size
Model
In-Context
RAG
Agentic REACT
ZEROSHOT
COT
ZEROSHIOT-RAG
COT-RAG
DeepSeek-R1-322 R1-2BB
42.42 + 1.69
52.42 + 2.64
19.93 + 0.49
21.51 # 1.31
5.47 + 1.36 38.70
GPT-4o
27.20 = 0.76
50.66
28.05 2.48
20.49 = 1.07
50
28.49 = 1.15
34.61 2.41
# 2.60
20.12 1.69
30.92 # 1.41
Llama-3.3-700
25.64 + 0.56
48.37 1.75
25.18 1.91
27.63 #
35,83 + 1.00
DeepSeck-RLe -1-32B
18.33 + 2.33
19.65 + 3.00
16.70 + 0.85
1.32 17.87 1.32
3,57 + 0.01
GPT-4o
16.76 = 0.87
41.02
22.32 + 1.99
16.39 + 0.85
37.39
500
Gemini-L.5-Flaat
17.39 = 1.45
25.17 1.77
21.47 = 1.44
15.09 # 1.28
26.99 # 1.84
Llama-3.3-70B
11.59 # 1.19
25.99 + 2.09
19.45 + 0.93
21.60 # 2.06
35.56 # 0.49
DeepSeek-R1-328
15.64 + 0.88
14.81 + 1.44
4.74 + 0.04
GPT-4o
N/A (exceeds
18.13 = 0.66
14.25 1.54
36.85
5000
Cemini--.5--Fae
maximum context)
17.94 = 0.93
12.51 0.94
23.47 = 1.53
Llama-3.3-70B
15.07 = 0.55
17.89 = 0.45
30.89 + 2.24
ing technique that enables LLMs interleave reason- ing steps with tool interactions, to solve complex tasks. For PhantomWiki QA task, the LLMs are provided with keyword-basse tools RetrieveArticle and Search to retrieve relevant documents. See Appendix C.5 for tool details. These settings materialize the limitations of in-context prompting and necessitate the use of advanced RAG prompting and agentic prompting approaches.
(n 500). Here the full document corpus can still be in- cluded in all LLMs' contexts, but we find that ZEROSHOO starts to perform poorly for all LLMs, and DeepSeek-R) 32B especially struggles. FI scores of CoT for all LLMs degrade as well compared to n = 50, but perform com- parably to REACT workflow. Finally, in the setting of large universes (n 5000), none of the LLMs we eval- uate can accommodate the full document corpus. As the in- context techniques are longer sufficient, we must rely on RAG prompting and agentic prompting. RAG prompting attains poor Fl scores because the retriever fails to retrieve documents relevant for answering complex questions. On the other hand, agentic prompting technique shines in com- parison to other techniques, indicating that LLMS are better suited dynamically retrieve documents while reasoning on question. We attribute the poor performance of DeepSeek- R1-32B with agentic prompting to its inferior tool-calling abilities compared to the other LLMs.
In the CoT and REACT prompts, we include 10 QA exem- plars and hand-written reasoning traces. We choose these exemplars from a dataset instance of size 25 that is not used for evaluation. In REACT, we limit LLMs to interact with the text corpus for up to 50 steps, which is sufficient to answer almost all questions in PhantomWik instances.
We cap all LLM outputs to 4096 tokens and use greedy de- coding (temperature = 0). For DeepSeek-R. 1atsSeek-R133221 1-32B, we use temperature = 0.6 and top-p 0.95, following DeepSeek Al (2025, Section 3).
5. Evaluating Reasoning
4.3. Results
To isolate LLM reasoning capabilities on PhantomWike we investigate model performance on small universes (n = 50) in Figure 3. Note that the context windows of all LLMS can fully include small universe document corpora. Each PhantomWike dataset contains questions covering a wide range of difficulty. We evaluate three approaches: in-context prompting, RAG prompting, and agentic prompting. For each we plot the Fl scores as a function of question difficulty, measured by the number of reasoning steps necessary to answer the question. As mentioned in Section 3.3, this is determined by the type of question templates and the sampled relationships. For all
In Table 2, we report the mean Fl score across various universe sizes, LLMs, and prompting techniques. We first average Fl scores over all questions in a Phantom Wiki in- stance, then compute the mean and standard error across the dataset generation seeds.
We first consider the small-universe setting (n 50) in Table 2, which corresponds to roughly 16K tokens for the LLMs we test. In-Context prompting techniques outper- form other techniques: COT with BbepSeek-R1-322 at- tains the highest performance, followed by CoT with GPT- 4o. Next, we consider the setting of medium universes
6

---

PhantomWike On-Demand Datascts for Reasoning and Retrieval Evaluation
In-Context
RAG
Agentic
ZeroShot
ZeroShot-RAG
ReAct
CoT
CoT-RAG
0.75
0.75
0.75
F1
0.5
0.5
0.25
0.25
0.25
0-
10
15
5
10
15
5
10
15
Reasoning steps
Reasoning steps
Reasoning steps
DeepSeek-R1-22
Gemini-1.5- ---hal
GPT-4o
Llama-3.3-708
Figure 3. F1 scores versus question difficulty, measured by reasoning steps. We plot LLM performance universe size 50, and report scores averaged over generation seeds. Increasing question difficulty in Phantom reveals a clear decline across all state-of-the-ar the LLMs and prompting techniques. showing their struggle with reasoning.
LLMs and prompting techniques, we verify empirically that questions with larger number of reasoning steps are indeed more challenging to answer.
then retrieving their friend's 's document find the nephew. Since RAG prompting techniques retrieve documents only once by matching vector embeddings of questions and docu- ments, they are unlikely to retrieve all necessary documents required to answer such questions. Recent multi-hop RAG prompting methods exhibit slightly better performance on Phantom Wiki, but similarly struggle on questions requiring many reasoning steps, as shown in Appendix D.1.
ZEROSHOT performance declines sharply as the num- ber of reasoning steps increases for all LLMs, except for DeepSeck-R1-32BB, which deteriorates more gradually. LLMS perform better with COT than with ZEROSHOT, but each additional reasoning step remains increasingly chal- lenging. Another failure mode balti- reasoning, i.e. in keeping track of multiple reasoning traces that could lead to valid solutions. For example, model might fail to find all possible solutions to a question "Who is the great- grandchild of David?" by forgetting to check some of the grandchildren; this compounds further as the number of possible constraints increases. This suggests that even in the absence of retrifval-specffic constraints, LLMs struggle to navigate logical reasoning sequences.
Finally, the agentic prompting technique REACT allows LLMs to avoid the steep initial performance drop in RAG prompting. On given a question, REACT prompt- ing requires LLMs to retrieve documents dynamically in a conversation and justify why they are relevant. Concretely, before using a tool conversation turn, the LLM is asked describe how the tool will help using a "Thought" step (Yao et al., 2022), analogous to the CoT prompting approach. This approach shows promise in answering questions correctly. Even so, REACT struggles as the question difficulty increases.
or Search) in
RAG prompting techniques (ZEROSHOT- RAG and CoT- RAG) stunt reasoning performance across the board-F1 scores are near zero on questions with 5 or more reasoning steps as opposed to 15 steps for in-context prompting. We attribute this core problem with RAG prompting: re- trieving documents in the initial prompt before starting to answer the question, as opposed to reasoning through the question and retrieving documents dynamically.
Figure thus decomposes LLM performance along the lines of reasoning capabilities. reveals that all in-context prompting and agentic prompting achieve near- perfect FI scores low-difficulty questions. Therefore, the stratification between them in Table 2 can be attributed to varying performance on high difficulty questions. To further isolate the impact of question difficulty, in Figure 7 we plot Fl scores as a function of reasoning steps for questions with only one solution.
We find that RAG prompting techniques can only answer questions that require single reasoning step (e.g., "Who the friend of David?"). Thus, answering questions that require information from multiple reasoning steps is ex- tremely challenging for ZEROSHOT-RA RAG and COT-RAG. Consider the question, "Who is the nephew of the friend of David?." which requires retrieving David's document first,
7

---

PhantomWike On-Demand Datascts for Reasoning and Retrieval Evaluation
In-Context
RAG
Agentic
1
1
ZeroShot
ZeroShot-RAG CoT-RAG
ReAct
CoT
0.75
0.75
0.75
0.5
0.5
0.25
0.25
0.25
0
128K
1M
10a
10
10
10
10
10
10
10
10
Universe size n
Universe size n
Universe size n
DeepSeek-R1-228
Gemini-1.5-Flaas.
GPT-4o
Llama-3.3-708
Figure 4. F1 scores as function of universe size We evaluate LLM performance questions with < 10 reasoning steps, and report F1 scores averaged over dataset generation seeds. As we increase universe size Phantom Wiki, F1 scores for all LLMs and prompting techniques deteriorate-alle at different rates.
6. Evaluating Retrieval
that ZEROSHOT-RAG outperforms ZEROSHOT, but CoT- RAG underperforms CoT (assuming the corpus can fit into context). Agentic prompting techniques like REACT show immense promise by avoiding a steep downward trend. This suggests that agentic workflows can be effective in dy- namically retrieving documents at scale.
Next, to evaluate LLM retrieval capabilities, we use PhantomWiki to contrast two settings: (1) small universes where the document corpus can comfortably fit in LLM con- text, and (2) large universes where the full corpus exceeds context lengths. To this end, we increase the universe size up to n = 10K, which corresponds to document corpora well beyond the context lengths of state-of-the-art LLMs, and display the results in Figure 4.
7. Fine-tuning on Phantom Wiki
Phantom Wiki generates on-demand datasets with random facts and relationships, ensuring that each dataset instance is unique and evaluation is resistant to fact memorization. Even so, it is possible that fine-tuning on Phantom Wiki- generated datasets could improve performance by leverag- ing linguistic structure in Phantom articles and ques- tions. We fine-tune LLMs on PhantomWiki datasets, si- multaneously (1) assessing the viability of PhantomWiki for training LLMs and (2) testing the robustness of Phantom Wiki to memorization when evaluating LLMs.
For small universes ROSHOT for all LLMs. However, Fl scores noticeably worsen as more documents are included in models' con- texts, with DeepSeck-R1-32D suffering a dramatic perfor- mance drop.Thi analysis regime indicates that state-of-the- art L.LMs struggle at in-context retrieval for complex tasks.
50), CoT outperforms ZE-
At the large universe scale, in-context prompting techniques become nonviable as the document corpus exceeds model context lengths. Therefore the use of out-of-context re- trieval, such as RAG prompting and agentic prompting tech- niques, is necessary for obtaining the answers. Here we ob- serve that RAG prompting techniques, whose performance is heavily determined by the quality of retrieved documents, deliver sub-par F1 scores across all universe sizes. As ex- pected, the performance declines with increasing universe size. Interestingly, chain-of-thought does not always im- prove performance over zeroshot in the RAG setting. In fact, Geemini-..5-Flask and GPT-4o with COT-RAG un- derperforms their ZEROSHOOT-RAC counterparts, When ofuin-of-though does provide an improvement, however, the gap between CoT and zeroshot is narrower the RAG setting than in the in-context setting. Thus, we observe
We fine-tune Qwen2.50.5B and 3B LLMs (Qwen Team, 2024) on 10 new PhantomWiki dataset instances with two popular fine-tuning algorithms: group relative policy op- timization (GRPO) (DeepSeek AI, 2024) and supervised fine-tuning (SFT), and evaluate on the three PhantomWiki datasets of size n 50 used in Table 2 (refer to Ap- pendix D.3 for details). We find that GRPO improves FI scores over in-context prompting techniques (Table 3) and SFT does not. The performance is still far from optimal, however, and the scores decline as the number of reason- ing steps increases (Figure 10), imitating the phenomenon in Figure 3. This demonstrates that PhantomWin evaluation is robust to LLM memorization, and paves the way for further research in improving LLM reasoning capabilities.
8

---

PhantomWiki On-Demand Datascts for Reasoning and Retrieval Evaluation
helpful feedback on this work.
Table Evaluation scores (in %) of LLMs fine-tuned on PhantomWilk As in Table 2, we report the mean + standard error across dataset generation seeds for universe size n 50. ZEROSHOT and CoT are prompting methods; SFT and GRPO are fine-tuning methods. Qwen2.5-0.5B CoT performance is notably worse than ZEROSHOT since the small model fails to follow the desired answer template.
Impact Statement
By leveraging context-free grammars and Prolog, PhantomWike is able to generate large, durable and challenging datasets without using LLMs. The datasets have low computational, monetary, and environmental cost and open-source framework is accessible to any user.
Method
Qwen2.5-0.5B
Qwen2.5-3B (LoRA)
ZEROSHOT
11.78 + 0.94
16.82 + 2.37
COT
2.68 + 0.22
13.71 + 0.81
Since Phantom Wiki randomly generates datasets that do not reference any existing data, the evaluation benchmark is resistant to data leakage and memorization while training.
SFT
11.71 + 1.10
16.89 + 2.22
GRPO
13.25 + 0.93
31.38 + 0.86
The approach of publishing a dataset generation procedure rather than a fixed dataset also encourages better research practices (by using fresh datasets instead of overfitting to a single instance), and enables more accurate evaluation of model performance. Since we do not use any personal data, use of Phantom does not have any privacy concerns.
8. Conclusion and Future Work
We introduce Phantom Wiki framework for benchmark- ing the reasoning and retrieval capabilities of language mod- els. As we increase the question complexity and universe size, we observe that current state-of-the-ard LLMs struggle in terms of both reasoning and retrieval. Phantom Wiki is scalable and memorization-resistee hence well-suited to evaluate future generations of language models.
References
Agarwal, O., Ge, H., Shakeri, S., and AI-Rfou, R. Knowledge graph based synthetic corpus generation for knowledge-enhancee language model pre-training. arXiv preprint arXiv:2010. 12688, 2020.
Our work brings forth several research directions. One of them could be to extend PhantomWik: from its limited setting and form to support more complex types of facts and relations. The simplicity and minimalism of the text corpus could be improved using LLM-based paraphrasing methods, which, as noted before, would require innovation in preserving consistency of and preventing hallucinations in over
AlphaProof & AlphaCieometry. AI achieves silver-medal standard solving International Mathematical Olympiad problems. https//ddepmmnnddoogglgleveeeeee
irves-imo-problmms at sellver-medal-leveee 2024. Ac- cessed: 2025-01-25.
the data. this work we focus on question-answer text corpora; this leaves potential extend Phantom Wiki to other knowledge bases and modalities such as vision and audio, enabling analogous test suites in multimodal settings.
An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., and Qiu, X. L-Eval: Instituting standard- ized evaluation for long context language models. arXiv preprint arXiv:2307. 1088, 2023.
Software and Data
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. LongBench: A bilingual, multitask benchmark for long context under- standing. arXiv preprint arXiv:2308. 14508, 2023.
The source code for this work can be found at and via pip install phantom-wi ki, and the sample Hugging- Face datasets are available at killan-groupphaanoo-wwkky wikk...
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Van Den Driessche, G. B., Lespiau, J.-B., Damoc, B., Clark, A., et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2206-2240. PMLR, 2022.
Acknowledgements
KS gratefully acknowledges support from AstraZeneca. CW is supported by the National Science Foundation (NSF) 0AC-2118310 and NSF-1934714 14 grant. This work was partially supported by funding from New York-Presbyterinn for the NYP-Corea Cardiovascular AI Collaboration, the National Institute of Food and Agriculture (USDA/NIFA), the Air Force Office of Scientific Research (AFOSR), and a Schmidt AI2050 Senior Fellowship, a Schmidt Sciences program. The authors thank anonymous reviewers for their
Chen, W., Zha, H., Chen, Z., Xiong, W., Wang, H., and Wang, W. HybridQA: dataset of multi-hop question answering over tabular and textual data. arXiv preprint arXiv:2004.07347, 2020.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano,

---

ThantomWikii On-Demand Datascts for Reasoning and Retrieval Evaluation
R., Hesse, C., and Schulman, J. Training verifiers solve math word problems. arXiv preprint 10.4466,, 2021.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring sive multitask language understanding. arXiv preprint WxXi:2009.033300 2020.
Cohen, R., Biran, E., Yoran, O., Globerson, A., and Geva, M. Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics, 12:283-299 2024.
Ho, X., Nguyen, A.-K. D., Sugawara, S., and Aizawa, A. Constructing multi-hop QA dataset for compre- hensive evaluation of reasoning steps. arXiv preprint arXi::201 2020.
DeepSeek AI. DeepSeekMath: Pushing the limits of math- ematical reasoning in open language models. arXiv preprint 4rX1v:2402.03300, 2024.
Hohenecker, P. and Lukasiewicz, T. Ontology reasoning with deep neural networks. Journal of Artificial Intelli- gence Research, 68:503-544 --500-540, 2020.
DeepSeek AI. DeepSeek-RI:
Incentivizing reasoning capa- bility in LLMs via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Hoperoft, J. E.,, Motwani, R., and J. Introduction to automata theory, languages, and computation. Acm Sigact News, (2(1):60-65. 2001.
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. DROP: A reading comprehension bench- mark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00140 2019.
Hsia, J., Shaikh, A., Wang, Z.. and Neubig, G. RAGGED: Towards informed design of retrieval augmented genera- tion systems. arXiv preprint arXiv:2403.09044 2024.
Hsich, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. RULER: What's the real context size of your long-context language models? COLM, 2024.
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., and Goldberg, Y. Measuring and improv- ing consistency in pretrained language models. Transac- tions of the Association for Computational Linguistics, 9: 1012-1031, 2021.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.,, and Chen, W. LoRA: Low-rank adaptation of large language models. ICLR, 1(2)3, 2022.
Feldman, Y. and El- Yaniv, R. Multi-hop paragraph retrieval for open-domain question answering. arXiv preprint (o0p::99660000 2019.
Jin, J., Zhu, Y., Yang, X., Zhang, C., and Dou, Z. FlashRAG: A modular toolkit for efficient retrieval-augmented gener- ation research. arXiv preprint arXiv:2405.13577 2024.
Flajolet, P. and Sedgewick, R. Analytic Combina- torics. Cambridge University Press, 2009. ISBN 9781139477161. URL https://ookss.ooglee.comm
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906 2020.
lokk%/d--n--QcAlclCC.
Gemini Team, Google. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint ar arXiv:2403.05530 2024.
Kasai, J., Sakaguchi, K., Takahashi, Y., Bras, R. L., Asai, A.. Yu, X., Radev, D., Smith, N. A., Choi, Y., and Inui, K. REALTIME QA: what's the answer right now? Advances in Neural Information Processing Systems, 36, 2024.
Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. Retrieval augmented language model pre-training. In International conference machine learning, pp. 3929- 3938. PMLR. 2020.
Lattimer, B. M., Gangal, V., McDonald, R., and Yang, Y. Sparse rewards can self-train dialogue agents. arXiv preprint arXiv:2409.046/ 2024.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., tau Yih, W., Rocktäschel, T.. Riedel, S., and Kiela, D. Retrieval- augmented generation for knowledge-intensing nlp tasks. Advances in Neural Information Processing Systems, 33: 9459 9474, 2020.
Han, S., Schoelkopf, H., Zhao, Y., Qi, Z., Riddell, M., Zhou, W., Coady, J., Peng, D., Qiao, Y., Benson, L. Sun. L., Wardle-Solano, A., Szabo, H., Zubova, E., Burtell, M., Fan, J., Liu, Y., Wong, B., Sailor, M., Ni, A., Nan, L., Kasai, J., Yu, T., Zhang, R., Fabbri, A. R., Kryscinski, W., Yavuz, S., Liu, Y., Lin, X. V., Joty, S., Zhou, Y., Xiong, C., Ying, R., Cohan, A., and Radev, D. FOLIO: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209. 00840, 2022.
Li, J., Wang, M., Zheng, Z., and Zhang, M. LooGLE: Can long-context language models understand long contexts? arXiv preprint 1.XXx:3311.049)) 2023.
10

---

PhantomWiki: On-Demand Datascts for Reasoning and Retrieval Evaluation
Liu, J.. Cui, L., Liu, H., Huang. D., Wang, Y., and Zhang, Y. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124, 2020.
Qwen Team. Qwen 2.5 Technical Report. arXiv preprint arXiv:24/2.55/ 2024.
Rajpurkar, P. SQuAD: 100,000+ questions for machine comprehension text. arXiv preprint 1oXiv:66060052500 2016.
Llama Team - Al at Meta. The Llama 3 herd of models. arXiv preprini arXiv:2407.21788. 2024.
Rau, D., Déjean, H., Chirkova, N., Formal, T., Wang, S., Nikoulina, V., and Clinchant, S. BERGEN: A bench- marking library for aeuienell-uggmented generation. preprint arXiv:2407.01102, 2024.
Mao, Q.. Luo, Y., Zhang, Q.. Luo, Y., Cao, Z., Zhang, J., Hao, H., Chen, Z.,, Jiang, W., Liu, J., Wang, X., Huang, Z., Tan, Z., Jie, S., Li, B., Liu, X., Zhang, R., and Li, XRAG: eXamining the core- be-ehmarknhmnngggg foundational components in advanced retrieval-augne augmented generation. arXiv preprint 2tx.ix:2412./5522 2024.
generaaatioooooono arXiv
Saad-Falcon, J., Khattab, O., Potts, C., and Zaharia, M. ARES: An automated evaluation framework for eetrievl--uugmente generation systems. arXiv preprint WrXV:2231.009476 2023.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. Advances in Neu- ral Information Processing Systems, 35:17359-17372, 2022.
Sakaguchi, K., Bras, L., Bhagavatula, C., and Choi, Y. WinoGrande: An adversarial Winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Min, S., Chen, D., Hajishirzi, H., and Zettlemoyer, I.. A discrete hard em approach for weakly supervised question answering. arXiv preprint arXiv: 1009:099990090 9999949499 2019.
Saparov, and He, H. Language models are greedy rea- soners: A systematic formal analysis of chain-of-thought. arXiv preprint arXiv:2210.012400 2022.
Mirzadeh, I,, Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., and Farajtabar, M. GSM-Symbolie: Understanding the limitations of mathematical reasoning in large language models. Xiviv preprint arXiv:2410.05220 2024.
Shao, Y., Jiang, Y., Kanell, T. A., Xu, P., Khattab, O., and Lam, M. S. Assisting in writing Wikipedia- articles from scratch with large language models. arXiv preprint arXiv: 14207, 2024.
Monteiro, J., Noel, P.-A., Marcotte, E., Mudumba, S. R., Zantedeschi, V., Vazquez, D., Chapados, N., Pal, C., and dataset for benchmarking LLMs unseen reference content. Advances in Neural Information Processing Systems, 37: 24242-24276, 2024.
Taslakian, P. RepLiQA: A question-answer
Shi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., and Yih, W.-t. REPLUG: Retrieval- augmented black-box language models. arXiv preprint 1XXi:2301.122550 2023.
Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. MTEB: Massive text embedding benchmark. arXiv preprint arXiv:22/0.073/66 2022.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal rein- forcement learning. Advances in Neural Information
Processing Systems, 36, 2024.
OpenAl. Gpt-4o system card. arXiv:2410.21276, 2024.
arXiv preprint
Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03766. 2020.
Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. Go- rilla: Large language model connected with massive APIs. arXiv preprint arXiv:2305.15334 2023.
Sinha, K., Sodhani. S., Dong, J., Pineau, J., and Hamilton, W. L. CLUTRR: A diagnostic benchmark for inductive reasoning from text. arXiv preprini urXiv:1908. 0008006177 2019.
Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., Cao,
N. D., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., Plachouras, V., Rocktäschel, T., and Riedel, S. KILT: a benchmark for knowledge intensive language tasks. arXiv preprint arXiv:2009. 02252, 2020.
Sprague, 7.,, Ye, X., Bostrom, K., Chaudhuri, S., and Durrett, G. MuSR: Testing the limits of chain-of-thought with mul- tistep soft reasoning. arXiv preprint arXiv:23 13.10.1044999 2023.
Press, O., Zhang, M., Min, S., Schmidt, I.., Smith, N. A., and Lewis, M. Measuring and narrowing the composi- tionality gap in language models. In Findings of the As- sociation for Computational Linguistics: EMNLP 2023, pp. 5555677771,,, 2023. 2023.
Sterling, L. and Shapiro, E. Y. The art of Prolog: advanced programming techniques. MIT press, 1994.

---

PhantomWiki: On-Demand Datascts for Reasoning and Retrieval Evaluation
Su, H., Yen, H., Xia, M., Shi, W. Muennighoff, N., yu Wang. H., Liu, H., Shi, Q., Siegel, Z.. S., Tang, M., Sun, R., Yoon, J., Arik, S. O., Chen, D., and Yu, T. BRIGHT: A realistic and challenging benchmark for reasoning- intensive retrieval. arXiv preprint arXiv:2407. 12883, 2024.
Yang. Z., Qi. P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D. HotpotQA: A dataset for diverse, explainable multi-hop question an- swering, arXiv preprint arXiv 2018.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. ReAct: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.055: 2022.
Tafjord, O., Mishra, D., and Clark, P. Proofwriter: Gen- erating implications, proofs, and abductive statements over natural language. arXiv preprint arXiv:2012.13045 2020.
Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. r-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.1200 2024.
Talmor, A. and Berant, The web as a knowledge- base for answering complex questions. arXiv preprint 1rXiv:1803.06643 2018.
Ye, H., Zhang, N., Chen, H., and Chen, H. Generative knowledge graph construction: A review, arXiv preprint arXiv:2210.1277 2022.
Tang. Y. and Yang, Y. MultiHop-RAs RAG: Benchmarking retrieval-augee augmented generation for multi-hop queries. arXiv preprini arXiv:2401.15391 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sentence? arXiv preprint arXiv: 1905. 07830, 2019.
Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. MuSiQue: Multihop questions via single-hop ques- tion composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022.
Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M. K.,
Han, X., Thai, Z. L.., Wang. S., Liu, Z., et al. ooBecch: Extending long context evaluation beyond 100K tokens. ACL, 2024.
Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. Interleaving retrieval with chain of thoutht reasoning for snowledge-intensive multi-step questions. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 10014-10037, 2023.
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., and Neu- big, G. WebArena: A realistic web environment for build- ing autonomous agents. arXiv preprint arXiv:2307.13854 2023.
Vu, T., Iyyer, M., Wang. X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. FreshLLMs: Refreshing large language models with search engine augmentation. arXiv preprint 23X.2:231.003214, 2023.
Zhu, Z., Xue, Y., Chen, X., Zhou, D., Tang, J., Schuurmans, D.,, and Dai, H. Large language models can learn rules. arXiv preprint arXiv:23 2023.
Zhuang, Y., Yu, Y., Wang, K., Sun, H., and Zhang, C. ToolQA: A dataset for LLM question answering with external tools. Advances in Neural Information Process- ing Systems, 36:501 17-50143,, 2023.
Wang, M., Chen, L., Fu, C., Liao, S., Zhang, X., Wu, B., Yu, H., Xu, N., Zhang, L., Luo, R., et al. Leave no document behind: Benchmarking long-context LLMs with extended multi-doc QA. EMNLP, 2024.
Wei, J., Wang. X.,, Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E.,, Le, Q., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 24824-24837, 2022.
WelbI, J., Stenetorp, P., and Riedel, S. Constructing datasets for multi-hop hop reading comprehension across documents. Transactions of the Association for Computational Lin- guistics, 6:287-302, 2018
Weston, J., Bordes, A., Chopra, S., Rush, A. M., Van Merriënboer, B., Joulin, A., and Mikolov, T. Towards Al-complete question answering: A set of prerequisite toy tasks. arXiv preprint 150. 2015.
12

---

PhantomWiki On-Demand Datascts for Reasoning and Retrieval Evaluation
A. Background
A.1. Context-Free Grammars
Context-free grammar (CFG) is type of formal grammar where the productions rules govern how to generate text from non-terminals and terminals. context-free grammar is defined by = (V. Σ, R, S) where V and denotes nonterminal and terminal respectively. R is finite relation in V x (V which specifies the production rules of the grammar. S E V the start symbol. A production rule in R has the form
(I)
where a E V. 3 E (VUE), conventional to list rules with the same left-hand side on the same line and separate the right-hand side with "I" like a 31 82.
B. Question Template Generation
B.1. Context-Free Grammar
We use the following CFG to generate question templates:
s Who R? What is How many _N does R have ?
R -> the RN of R_c I the person whose AN is AV
R_c -> R N
A -> the AN of R
RN -> <relation>
RN_P -> <relation_plural>
AN -> <attribute__ame>
AV
ietribbut_vaaueee
N -> <name>
B.2. CFG-generated question templates
Our CFG produces the following 50 question templates recursion depth d 20, Note how the recursive production rule
R_c -> R leads chained productions.
1. Who is the <relation>_3 relation>_9 of the <relation>_ll of the <relation>_13 of the <relation>_i of the <relation>_17 of the person whose <attribute_name> is attribute_value> 2. Who is the <relation> 3 of the <relation> of the <relation>_> of the < relation>_9 of the <relation>_l1 of the <relation>_13 of the <relation> of the <relation>_ll of <name>_ir __8?
the <relation>_5 of the <relation>_7 of the
3. Who is the <relation>_3 of the <relation>_5 of the <relation>_7 of the relation>_$ of the <relation> the person whose <attribute, Aittribute_name__11 is <attribute_value>_1] 4. Who is the <relation>_3 of the <relation>_5 of the <relation>_7 of the relation>_g of the <relation>_ll of the 13 of the <relation>_Ls _1 of <name>_16>
the <relation>_is of the <relation> _1 of
5. Who is the <relation> of the <relation>_5 of the <relation>_7 of the < relation>_9 of the <relation>_l1 of the <relation>_13 of the person whose < attribute_namex 6. Who is the <relation> 3 the <relation>_5 of the <relation>_] of the relation>_9 of the <relation>
is <attribute <attribute_value> value>_151
of the <relation>_ls of <name>_ 14? 7. Who is the <relation>_3 of the <relation>_5 of the <relation>_? of the relation>_g of the <relation> 11 of the person whose aattribut__namer 13 is < attribute_valle>_133
8. Who is the <relation> 3 of the <relation>_ of the <relation>_ of the relation>_9 of the <relation>_11 <name>_12
13

---

PhantomWiki: On-Demand Datascts for Reasoning and Retrieval Evaluation
9. Who is the <relation>_3 of the <relation>_5 of the <relation>_7 of the relation>_9 of the person whose <attribute_name__ll is vnttribute_valuess 11? 10. Who is the <relation>_3 of the <relation>_5 of the <relation>_7 of the relation>_$ of <name>_10?
11. who is the <relation>_3 of the <relation>_5 of the <relation>_? of the person whose <attribute_name>,
is <attribute_valu>___?
12. Who is the <relation>_3 of the <relation>_5 of the <relation>_7 of <name>_8?
13. Who is the <relation> is <attribute_valu>>]]]
the <relation>_5 of the person whose
attribute_name>_ 14. Who is the <relation>_3 of the <relation>_5 of <name>_6?
15. who is the <relation>_3 of the person whose <attribute_nane _amb>>_5_am__55 is < eetribute_value>_
16. who is the <relation>_> of <name>_47
17. who is the person whose <attribute_name> is <attribute_valae__33
18. What is the <attribute cetrribute_nam>>__ of the <relation>_ the <relation>_ 6 of the
<relation>_ of the <relation>_10 the <relation>_12 of the <relation>_14 of the <relation>_L
of the <relation>_Ls of <name>_19) 19. what is the Sittribuue_name>___ <relation>__ of the <relation>_10 of the <relation>_i2 of the <relation>_1e of the <relation>_le _6 of the person whose <attribute_name> 18 is serribute_value>___
of the <relation>_4 of the <relation>_6 of the
20. what is the <attribute_names name>__ of the <relation>_4 the <relation>_6 of the <relation>_ of the <relation> _10 of the <relation>_12 of the <relation>_14 of the <relation> 16 of <name> 17?
21. what is the centribbut__aam>>>
of the <relation>_4 of the <relation>_6 of the <relation>_B of the <relation> _10 of the <relation>_i2 of the <relation>_ 14 is catrribuut__aluu>__11
of the person whose 22. What is the <attribute Cestribute_name>___ of the <relation> of the < 6 of the <relation>_B B of the <relation> of <name>_15
Sittrribute_name____
the <relation>_12 of the <relation>_14
23. what is the <attribute <attribute_name> _ame__ of the <relation>_4 of the <relation>_6 of the <relation>_8 of the <relation>_10 of the <relation>_i2 of the person whose < perribute_name>___ 24. what is the <attribute__ame>>
is <attribute lattribute_value>_14. value>_14?
of the <relation>_4 of the <relation>_6 of the <relation>_B of the <relation> of the <relation>_Ls of <name>_13?
25. What is the Sattribute_name>_ of the <relation>_4 of the <relation>_6 of the <relation>__ of the <relation> <attribute_value>__] 12?
the person whose <attribute names 12 is
26. What is the cnstribute_name: of the <relation>_4 of the <relation>_6 of the <relation>_@ of the trelation>__ of <name>_ 11?
27. What is the setribuute_nmme>> of the <relation>_ _4 of the <relation>_6 of the <relation>_B of the person whose <attribute_name>> 10 is @etribute__valu>__uu ?
28. What is the <attribute 3 name>_3 of the <relation>_4 the <relation>_6 of the <relation>_8 of <name>_9?
29. What is the <attribute [ of the <relation>_4 of the <relation>_ 6 of the person whose <attribute_name>_ is <attribute_value>_P
30. What is the <attribute_name> of the <relation>_4 of the <relation>_6 of < name>_72
31. What is the <attribute fattribute_name> of the <relation> _4 of the person whose is btttrribute_value>.ee 32. What is the <attribute_nam>>_3 of the <relation>_4 of <name>_5?
attribute_namex.
33. What is the <attribute name>> of the person whose <atiribute name>_ 4 is attribute_value___
14

---

PhantomWiki: On-Demand Datascts for Reasoning and Retrieval Evaluation
34. How many <relation_plurall22 does the <relation>_4 of the <relation>_6 of the <relation>_6 of the <relation>_10 of the <relation>_12 of the <relation>_14 of the <relation>_16 of the <relation>_18 of <name>_19 have?
35. How many velattion_puuaal>, <relation>_8 of the <relation>_ll of the <relation>_12 of the <relation>_14 of the <relation>_16 of the person whose <attribute_name>> is attribute_value__it have?
does the <relation>_ of the <relation>_e _ of the
36. How many drelation_plurall 2 does the <relation>_ the <relation> 6 of the <relation>_6 of the <relation> _11 of the <relation>_1s 12 of the <relation>_l_ 14 of the <relation>_16 of <name>_ have?
37. How many <relatton_pluralle
does the <relation>_4 of the <relation>_ of the
<relation>_ of the <relation> _10 of the <relation>_ia of the <relation>_1< have?
of the person whose <attribute _atttee__mme______ is <attribute Letpbbbtt_vauuee__e 38. How many <relation__lurall>2 does the <relation>_4 of the <relation>_6 of the <relation>__ 8 of the <relation> 10 the <relation>_ls of the <relation>_14 of <name>_15 have?
39. How many <relation erelation_plural> plurpl>_rr__1 2 does the <relation>_4 of the <relation>_6 of the <relation>_ß of the <relation>_10 of the <relation>_12 of the person whose < attribute_name>_1
is <attribute_valle>___ 40. How many <telation_plural>> does the <relation>_4 of the <relation>_6 of the <relation>_F 8 of the <relation> 10 of the <relation> __ of <name>_i> have? the <relation>_6 of the <relation>_ of the <relation>_10 of the person whose <attribute_name> is Eictribute_valuee 12 have?
have?
41. How many <relation_plural>2 does the <relation>_4
42. How many <relation_pluraly> does the <relation> 4 of the <relation>_6 of the <relation>_B of the <relation> 10 of <name>_ 11 have?
43. How many srelattonnplural>22 does the <relation>_4 of the <relation>_6 of the <relation>__ of the person whose aattribute_namee 10 is cetrribute_value>___e have?
44. How many <relation sxelation_plural>> plural> _2 does the <relation> the <relation>_6 6 of the <relation>_ß of <name>_9 have?
45. How many <relation_plural> person whose <attribute name>_ 8 is <attribute <attribute_value> 8 have?
does the <relation>_4 of the <relation>_6 of the
46. How many <relation_plurad
does the <relation>_4 of the <relation>_6 of <
name>_7 have?
47. How many <relatiinn_purral>22 does the <relation>_4 of the person whose attribute_name> is bttrriibutevaalueeee have?
48. How many <celatiio__purra>> does the <relation>_4 of <name>_5 have?
49. How many <relation__lurabb__ does the person whose <attribute_name>> is
attribute_value> have?
50. How many crelation_plurat does <name>_ have?
B.3. Question-Answer Characteristies
C. Baseline Details
C.1. ZEROSHOT-SIMPLE
We use the following prompt for all models, where evidence is the concatenation of documents in the Phantom Wiki instance.
You are given the following evidence: (BEGIN EVIDENCE) ([evidence)) (END EVIDENCE)
15

---

PhantomWiki On-Demand Datascts for Reasoning and Retrieval Evaluation
0.2
Depth= 10
0.15
Depth= 20
0.05
0
1
6
16
Reasoning steps
Figure 5. Histogram of question difficulties (measured by reasoning steps) for universe size = 50 at two CFG recursion depths d {10, 20). We average the frequencies dataset generation seeds.
Distribution of Solution Count the Dataset
0.15
0.075
0.10
0.2
0.025
Number Quastions
20
50
100
100
0.04
0.02
0.00
0.00
20
100
200
400
0.4
0.2
0.02
0.0
0.0
0.00
10
20
20
40
100
200
Solutions
Figure 6. Distribution of number of answers sizes (50, 500, 50003, seeds (1,2,3), and CFG depth 20.
You will be provided question. Your task is provide an answer according to these instructions:
The output must be one of the following: name (if there is only one correct answer); or a list of names separated by (constants, answer_sep)' (if there are multiple correct answers).
Do NOT include any additional information in your answer.
Question: ((question))
Answer:
For DeepSeek-R1-32BB, we additionally parse the output to separate the model's reasoning process from its final answer using the </think> tag.
C.2. ZEROSHOT-RAG
The prompt is exactly the same as ZEROSHOT, except we replace evidence with documents retrieved using the BM25 retriever. Upon generation, we search for similar documents for quest ion according to maximum inner product search on document and question embeddings.
16

---

PhantomWiki: On-Demand Datascts for Reasoning and Retrieval Evaluation
C.3. CHAIN-OF-THOCCHT-SSME
We the following prompt for all models, where evidence is replaced with a list of all documents. We use regular expression to parse the output.
You are given the following evidence: (BEGIN EVIDENCE) ([evidence)]
(END EVIDENCE)
You will be provided a question. Your response must end in the following sentence : The answer is <answer>.
Here, <answer> must be of the following:
name (if there is only correct answer); ;
-
list of names separated by "cconstants. answer_sep)' (if there are multiple correct answers).
Here are some examples:
(START OF EXAMPLES) Example 1:
Question: Who is the brother of Dino Beltran?
Answer: Based the evidence, the brother of Dino Beltran is Orlando Beltran, The answer is Orlando Beltran.
Example 2:
Question: Who the sibling of Barabara Beltran?
Answer: Based on the evidence, the siblings of Barabara Beltran are Aida Wang, Vicki Hackworth. The answer is Aida Wang lannsttattante wnw.ttsssasssweeeeeeeeee ser___seiiiie Vicki Hackworth
Example 3:
Question: Who is the child of the sibling of Stacia Toombs?
Answer: First need to find the sibling stacia Toombs. Based on the evidence, the sibling of Stacia Toombs is Shelli Beltran. Now need find the child of Shelli Beltran. Based on the evidence, the children of Shelli Beltran are Aida Wang, Barabara Beltran, Vicki Hackworth. The answer is Aida Wang! constants answer sep) Barabara Beltran tttttraacconttattsssatt vicks Hackworth.
Example 4:
Question: Who is the uncle of William Smock?
Answer: An uncle is the brother of a parent. Based on the evidence, the parents of William Smock
Dominique Smock, Gene Smock. find the uncle of William Smock, I need to find the brother of Dominique Smock and Gene Smock. Based on the evidence, Dominique Smock has no brother, and the brother of Gene Smock is Eli Smock. So the uncle of Milliam Smock is Eli Smock. The answer is Eli Smock.
Example
Question:
What is the occupation of the sister of the grandmother of Virgil Hackworth?
Answer: A grandmother is the mother of parent. Based the evidence, the parents of Virgil Hackworth are Ricardo Hackworth, Vicki Hackworth. To find the grandmother of Virgil Hackworth, Hackworth and Vicki Hackworth. Based on the evidence, Ricardo Hackworth has no mother, and the mother of Vicki Hackworth is Shelli Beltran. Now need to
need to find the mother of Ricardo
17

---

PhantomWikii On-Demand Datascts for Reasoning and Retrieval Evaluation
find the sister of Shelli Beltran. Based on the evidence, the sister of Shelli Beltran is stacia Toombs. Based on the evidence, the occupation of Stacia Toombs is actuary. The answer is actuary.
Example 6:
Question: Who is the brother of the person whose occupation is associate professor?
Answer: need to search for people whose occupation is associate professor. Based the evidence, the person whose occupation is associate professor is Dino Beltran. And the brother of Dino Beltran is Orlando Beltran. The answer is Orlando Beltran.
Example 7:
Question: What the date of birth of the person whose hobby is meteorology? Answer: I need to search for people whose hobby is meteorology. Based on the evidence, the people whose hobby is meteorology are Alison Smock, Barabara Beltran. The date of birth of Alison Smock is 0929-10-28, and the date of birth of Barabara Beltran is 0989-06-11, The answer is 0929-10-28 (constants. 09990-009p0099-0666
Example 8:
Question: Who the cousin the person whose occupation broadcast engineer? Answer: I need search for people whose occupation is broadcast engineer. Based the evidence, the person whose occupation is broadcast engineer is Barabara Beltran. cousin is the child of the sibling of the parent. Based on the evidence, the parents of Barabara Beltran are Dino Beltran, Shelli Beltran. The sibling of Dino Beltran is Orlando Reltran, and the sibling of Shelli Beltran is Stacia Toombs, Based on the evidence, Orlando Beltran has no child, and the child of Stacia Toombs is Leslee Toombs. So the cousin of Barabara Beltran is Leslee Toombs. The answer is Leslee Toombs.
Example 9:
Question: Who is the great-granddaughtte of the person whose hobby is biology? Answer: I need to search for people whose hobby is biology. Rased on the ev dence the person whose hobby is biology is Alvaro Smock. To find the great- granddaughter of Alvaro Smock, I need to find the daughter of the child of the child of Alvaro Snock. Based on the evidence, the children Alvaro Smock are Eli Smock, Gene Smock. Eli Smock has child, and the child of Gene Smock is Williams Smock. The daughters of Williams Smock are Shelli Belt Beltran, Stacia Toombs. So the great-graanddughtters of Alvaro Smock, whose hobby is biology, are Shelli Beltran, Stacia Toombs. The answer is Shelli Toombs.
Beltran (constannss anwwweeerttt............. (END OF EXAMPLES)
Question: ((question)) Answer:
C.4. CHAIN-OF-TTHOLGHH--AA
The prompt is exactly the same as CoT, except we replace evidence with documents retrieved using BM25, We use the same retriever setup as described in Appendix C.2.
18

---

ThantomWiki On-Demand Datascts for Reasoning and Retrieval Evaluation
C.5. REACT
We modify the ReAct agent implementation of Shinn al. (2024), which can be found http:://gthub..cm//// com/noahshi and Search. As in the following
nn/reflexion, to support keyword base- search tools RetrieveArtion examples, the RetrieveArticle tool fetches the documents by title, and the Search tool fetches all document titles whose bodies contain the argument. We instruct the LLM to output predict the answer using third tool Finish.
Solve a question answering task with interleaving Thought, Action, Observation steps.
Thought can reason about the current situation, and Action can be 3 types: (1) R ((entttty(( This action retrieves the article about (H entityll)) if it exists. If the article does not exist, the action will say so.
( (1 (2) ( (attributelllll. This action searches the database for {H{
attribute)]]) and retrieves all articles that contain ([f(attribute)))] If no article contains [(flattribute)))), the action will say so. (3) Finish[[[(annver]]]]
This action answers the question with ([[(answrr))) ))) If you cannot find the answer, output the empty answer like: Finish[]. there are multiple answers B, answer with a list like: Finish (Aconsaants hwc.ssettsccoostttsssssssoo
You may take many steps necessary. Here are some examples: (START OF EXAMPLES)
Example 1: Question: who the brother Dino Beltran?
Thought 1: need to retrieve article about Dino Beltran and find who his brother is.
Action 1: RetrieveArticcl
Beltran].
Observation 1: # Dino Beltran ## Family The parents of Dino Beltran are Brian Beltran, Daisy Beltran. Dino Beltran's s sibling is Orlando Beltran. The brother of Dino Beltran is Orlando Beltran. The mother of Dino Beltran is Daisy Beltran. The father of Dino Beltran is Brian Beltran. The children Dino Beltran are Aida Wang, Barabara Beltran, Vicki Hackworth. The daughters of Dino Beltran are Aida Wang, Barabara Beltran, Vicki Hackworth, The wife of Dino Beltran is Shelli Beltran. ## Friends The friend of Dino Beltran is Alvaro Smock. ## Attributes The date of birth of Dino Beltran is 0958-08-09, The occupation of Dino Beltran is associate professor. The hobby of Dino Beltran is shogi.
Thought 2: The brother of Dino Beltran is Orlando Beltran, so the answer is Orlando Beltran.
Action 2: Finish [Orlando Beltran].
Example 2:
Question: Who is the mother of Ivana Smith?
Thought 1: I need to retrieve article about Ivana Smith and find who her mother is.
Action RetrieveArticle [Ivana Smith].
Observation No article exists for the requested entity. Please try retrieving article for another entity.
Thought 2: Article about Ivana Smith does not exist, so I cannot find the mother of Ivana Smith. I will output empty answer. Action 2: Finish[].
Example
19

---

PhantomWiki: On-Demand Datascts for Reasoning and Retrieval Evaluation
Question: Who is the sibling of Barabara Beltran?
Thought 1: need to retrieve article about: Barabara Beltran and find who her sibling is.
Action 1: RetrieveArticle (Barabara Beltran].
Observation 1: # Barabara Beltran ## Family The perents of Barabara Beltran are Dino Beltran, Shelli Beltran. Barabara Beltran's siblings are Aida Wang, Vicki Hackworth, The sisters of Barabara Beltran are Aida Wang, Vicki Hackworth. The mother of Barabara Beltran is Shelli Beltran. The father of Barabara Beltran is Dino Beltran. ## Friends ## Attributes The date of birth of Barabara Beltran is 0989-06-11. The occupation of Barabara Beltran is broadcast engineer. The hobby of Barabara Beltran is meteorology.
Thought 2: The siblings of Barabara Beltran are Aida Wang, Vicki Hackworth. So the answer is Aida Wang, Vicki Hackworth.
Action 2: Finish [Aida Wang(constants. http:/ttttttt...sse_ssse..
Hackworth).
Example
Question: Who is the child of the sibling of Stacia Toombs? Thought 1: First, I need to retrieve article about Stacia Toombs and find who her sibling is.
Action 1: RetrieveArticle (Stacia Toombs]
Observation 1: Stacia Toombs ## Family The parents of Stacia Toombs are Alison Smock, Williams Smock. Stacia Toombs's s sibling Shelli Beltran. The sister of Stacia Toombs is Shelli Beltran. The mother of Stacia Toombs is Alison Smock. The father of Stacia Toombs is Williams Smock. The child of stacia Toombs is Leslee Toombs. The daughter of stacia Toombs is Leslee Toombs. The husband of Stacia Toombs is Wilbert Toombs. ## Friends The friends of Stacia Toombs are Brian Beltran, Tsiah lutz, Leeann Hackworth, Lesley Lutz, Ryan Rang. ## Attributes The date of birth of Stacia Toombs is 0959-03-22. The occupation of Stacia Toombs is actuary. The hobby of Stacia Toombs is finance.
Thought 2: The sibling of stacia Teombs is Shelli Beltran. Now I need to retrieve article about Shelli Beltran and find who her child is.
Action 2: RetrieveArticle (Shelli Beltran].
Observation 2: Shelli Beltran ## Family The parents of Shelli Beltran are Alison Smock, Williams Smock. Shelli Beltran's sibling is Stacia Toombs. The sister of Shelli Beltran is Stacia Toombs. The mother of Shelli Beltran is Alison Smock. The father of Shelli Beltran is Williams Smock. The children of Shelli Beltran are Aida Wang, Barabara Beltran, Vicki Hackworth. The daughters of Shelli Beltran are Aida Wang, Barabara Beltran, Vicki Hackworth. The husband of Shelli Beltran is Dino Beltran. ## Friends The friends of Shelli Beltran are Brian Beltran, Eli Smock, Isiah Lutz, Leslee Toombs, Lesley Lutz, Ryan Wang. ## Attributes The date of birth of Shelli Beltran is 0958-03-08. The occupation of Shelli Beltran is occupational therapist. The hobby Shelli Beltran is sociology.
Thought 3: The child of Shelli Beltran is Aida Wang, Barabara Beltran, Vicki Hackworth. So the answer is Aida Wang, Barabara Beltran, Vicki Hackworth. Action 3: Finish [Aida Wang (oonggtntstts...mmmm answerssssswsssseeee Barabara Beltran Beltran (constants. answer_sep/Vice Hackworth).
Example 5:
Question: Who the unele William Smock?
Thought 1: First, need to retrieve article about William Smock. To find the uncle of William Smock, I need to find the brother of his parent.
Action 1: RetrieveArticle (William Smock)
20

---

PhantomWiki: On-Demand Datascts for Reasoning and Retrieval Evaluation
Observation 1:
Milliams Smock # Family The parents of Williams Smock are Dominique Smock, Gene Smock. The mother of Williams Smock is Dominique Smock The father of Williams Smock is Gene Smock. The children of Williams Smock are Shelli Beltran, Stacia Toombs. The daughters of Williams Smock are Shelli Beltran, Stacia Toombs. The wife of Williams Smock is Alison Smock. +# Friends The friend of Williams Smock is Lannie Smock. ## Attributes The date of birth of Williams Smock is 0926-04-04. The occupation of Williams Smock is clinical biochemist. The hobby of Williams Smock is social studies. Dominique Smock, Gene Smock. To find the uncle of William Smock, I need to find the brother of Dominique Smock and Gene Smock.
Thought 2: The parents of William Smock
Action 2: RetrieveArticle (ueminqquu Smock]
Observation 2: Dominique Smock ## Family The child of Dominique Smock is Williams Smock. The son of Dominique Smock is Williams Smock. The husband of Dominique Smock is Gene Smock. # Friends The friends of Dominique Smock are Gene Smock, Isiah Lutz, Orlando Beltran, Vicki Hackworth, ## Attributes The date of birth of Dominique Smock is 0897-09-08. The occupation of Dominique, Smock is sports therapist. The hobby of Dominique Smock is dominoes.
Thought 3: Dominique Smock has no brother. Now I need to zetrieve article about Gene Smock and find his brother.
Action 3: RetrieveArticle [Gene Smock]
Observation 3: Gene Smock ## Family The parents of Gene Smock are Alvaro Smock, Lannie Smock. Gene Smock's sibling is Eli Smock. The brother of Gene Smock is Eli Smock. The mother of Gene Smock is Lannie Smock. The father of Gene Smock is Alvaro Smock. The child of Gene Smock is Williams Smock. The son of Gene Smock is Williams Smock. The wife of Gene Smock is Dominique Smock.
## Friends The friends of Gene Smock are Leeann Hackworth, Leisa Lutz, Ricardo Hackworth, Alvaro Smock, Dominique Smock. ## Attributes The date of birth Gene Smock is 0898-08-16. The occupation of Gene Smock is immunologist. The hobby of Gene Smock is architecture.
Thought 4: The brother of Gene Smock is Eli Smock, and Dominique Smock has no brother. Since Gene Snock and Dominique Smock are the parents of William Smock, the uncle of William Smock is Eli Smock. So the answer is Eli Smock. Action 4: Finish [Eli Smock]
Example
Question: What is the occupation of the sister of the grandmother of Virgil Hackworth?
Thought 1: First, I need to retrieve article about Virgil Hackworth. To find the grandmother of Virgil Hackworth, I need to find the mother of his parent. Action 1: RetrieveArticle [Virgil Hackworth).
Observation 1: Virgil Hackworth ## Family The parents of Virgil Hackworth are Ricardo Hackworth, Vicki Hackworth. Virgil Hackworth's s siblings are Leeann Hackworth, Leisa Lutz. The sisters of Virgil Hackworth are Leeann Hackworth, Leisa Lutz. The mother of Virgil Hackworth is Vicki Hackworth. The father of Virgil Hackworth is Ricardo Hackworth. ## Friends The friends of Virgil Hackworth are Alison Smock, Eli Smock, Orlando Beltran, Ryan Wang. ## Attributes The date of birth of Virgil Hackworth is 1009-06-23. The occupation of Virgil Hackworth is sound technician. The hobby of Virgil Hackworth is wikipedia editing.
Thought 2: The parents of Virgil Hackworth are Ricardo Hackworth, Vicki Hackworth To find the grandmother of Virgil Hackworth, need to find the mother of Ricardo Hackworth and Vicki Hackworth.
21

---

PhantomWiki: On-Demand Datascts for Reasoning and Retrieval Evaluation
Action 2: RetrieveArticle (Ricardo [Ricardo Hackworth).
Observation 2: # Ricardo Hackworth ## Family The children of Ricardo Hackworth are Leeann Hackworth, Leisa Lutz, Virgil Hackworth. The son of Ricardo Hackworth is Virgil Hackworth. The daughters of Ricardo Hackworth are Leeann Hackworth, Leisa Lutz. The wife of Ricardo Hackworth is Vicki Hackworth. ## Friends The friends of Ricardo Hackworth are Gene Smock, Isiah Lutz, Johnetta Wang. ## Attributes The date of birth of Ricardo Hackworth is 0983-02-24. The occupation of Ricardo Hackworth is clinical research associate. The hobby of Ricardo Hackworth is dairy farming.
Thought 3: Ricardo Hackworth has no mother. So I need to retrieve article about Vicki Hackworth and find her mother.
Action 3: RetrieveArticle (Vicki Hackworth].
Observation 3: # Vicki Hackworth ## Family The parents of Vicki Hackworth are Dino Beltran, Shelli Beltran. Vicki Hackworth's s siblings are Aida Wang, Barabara Beltran. The sisters of Vicki Hackworth are Aida Wang, Barabara Beltran. The mother of Vicki Hackworth is Shelli Beltran. The father of Vicki Hackworth is Dino Beltran. The children of Vicki Hackworth are Leeann Hackworth, Leisa Lutz, Virgil Hackworth. The son of Vicki Hackworth is Virgil Hackworth. The daughters of Vicki Hackworth are Leeann Hackworth, Leisa Lutz. The husband of Vicki Hackworth is Ricardo Hackworth. ## Friends The friends of Vicki Hackworth are Brian Beltran, Dominique Smock, Eli Smock
## Attributes The date of birth Vicki Hackworth is 0985-05-30. The occupation of Vicki Hackworth is police officer. The hobby of Vicki Hackworth is meditation.
Thought 4: The mother of Vicki Hackworth is Shelli Beltran. So the only grandmother of Virgil Hackworth is Shelli Beltran. Now I need to retrieve article about Shelli Beltran and find her sister. Action 4: RetrieveArticle (Shell (Shelli Beltran].
Observation Shelli Beltran ## Family The parents of Shelli Beltran are Alison Smock, Williams Smock. Shelli Beltran's sibling is Stacia Toombs. The sister of Shelli Beltran is stacia Toombs. The mother of Shelli Beltran is Alison Smock. The father of Shelli Beltran is Williams Smock. The children of Shelli Beltran are Aida Wang, Barabara Beltran, Vicki Hackworth. The daughters of Shelli Beltran are Aida Wang, Barabara Beltran, Vicki Hackworth. The husband of Shelli Beltran is Dino Beltran. ## Friends The friends of Shelli Beltran are Brian Beltran, Eli Smock, Isiah Lutz, Leslee Toombs, Lesley Lutz, Ryan Wang.
Attributes The date of birth of Shelli Beltran is 0958-03-08. The occupation of Shelli Beltran is occupational therapist. The hobby of Shelli Beltran is sociology.
Thought 5: The sister of Shelli Beltran is Stacia Toombs. Now I need to retrieve article about Stacia Toombs and find her occupation.
Action 5: RetrieveArticle (Strea (Stacia Toombs].
Observation 5: Stacia Toombs ## Family The parents of Stacia Toombs are Alison Smock, Williams Smock. Stacia Toombs's sibling is Shelli Beltran. The sister of Stacia Toombs is Shelli Beltran. The mother of Stacia Toombs is Alison Smock. The father of Stacia Toombs is Williams Smock. The child of Stacia Toombs is Leslee Toombs. The daughter of Stacia Toombs is Leslee Toombs. The husband of Stacia Toombs is Wilbert Toombs. ## Friends The friends of Stacia Toombs are Brian Beltran, Isiah Lutz, Leeann Hackworth, Lesley Lutz, Ryan Wang. ## Attributes The date of birth of stacia Toombs is 0959-03-22. The occupation of stacia Toombs is actuary. The hobby of stacia Toombs is finance.
Thought 6: The occupation of stacia Toombs is actuary. So the answer is actuary. Action 6: Finish/actuary]
22

---

PhantomWiki: On-Demand Datascts for Reasoning and Retrieval Evaluation
Example 7:
Question: Who is the brother of the person whose occupation is associate professor?
Thought 1: First, I need to search for people whose occupation is associate professor.
Action 1: Search lassociate professor].
Observation (1) Dino Beltran
Thought 2: Person whose hobby is associate professor is Dino Beltran. Now need to retrieve article about Dine Beltran and find his brother. Action 2: RetrieveArticlee (Dino Beltran].
Observation 2: Dino Beltran ## Family The parents of Dino Beltran are Brian Beltran, Daisy Beltran. Dino Beltran's s sibling is Orlando Beltran. The brother of Dino Beltran is Orlando Beltran. The mother of Dino Beltran is Daisy Beltran. The father of Dino Beltran is Brian Beltran.
children Dino Beltran are Aida Wang, Barabara Beltran, Vicki Hackworth. The daughters of Dino Beltran are Aida Wang, Barabara Beltran, Vicki Hackworth. The wife of Dino Beltran is Shelli Beltran. ## Friends The friend of Dino Beltran is Alvaro Smock. ## Attributes The date of birth of Dino Beltran is 0958-08-09. The occupation of Dino Beltran is associate professor. The hobby of Dino Beltran is shogi.
Thought 3: The brother of Dino Beltran is Orlando Beltran, so the answer Orlando Beltran.
Action 3: Finish (Orlando Beltran].
Example 8:
Question: What is the date of birth of the person whose hobby is meteorology? Thought 1: First, I need to search for people whose hobby is meteorclogy.
Action 1: Search [meteorology].
Observation (1) Alison Smock (2) Barabara Beltran
Thought 2: People whose hobby is meteorology loy Alison Smock, Barabara Beltran. Now I need to retrieve article about Alison Smock and Barabara Beltran and find their date of birth.
Action 2: Retrieverrticlell (Alison Smock]
Observation 2: Alison Smock ## Family The children of Alison Smock are Shelli Beltran, Stacia Toombs. The daughters of Alison Smock are Shelli Beltran, Stacia Toombs. The husband of Alison Smock is Williams Smock. #i Friends The friends of Alison Smock are Eli Smock, Lannie Smock, Virgil Hackworth. ## Attributes The date of birth of Alison Smock is 0929-10-28. The occupation of Alison Smock is health promotion specialist. The hobby of Alison Smock is meteorology.
Thought 3: The date of birth of Alison Smock is 0929-10-28. Now I need to retrieve article about Barabara Beltran and find her date of birth. [Barabara Observation 3: Barabara Beltran ## Family The parents Barabara Beltran are Dino Beltran, Shelli Beltran. Barabara Beltran's s siblings Vicki Hackworth. The sisters of Barabara Beltran are Aida Wang, Vicki Hackworth. The mother of Barabara Beltran is Shelli Beltran. The father of Barabara Beltran is Dino Beltran. 4# Friends ## Attributes The date of birth of Barabara Beltran is 0989-06-11. The occupation of Barabara Beltran is broadcast engineer. The hobby Barabara Beltran meteorology.
Action 3: RetrieveArticle BBarabar Beltran].
Aida Wang,
Thought 4: The date of birth of Barabara Beltran is 0989-06-11. So the answer is 0929-10-28, 0989-06-11.
Action 4: Finish (0929-11 10-2811 constants,... www.eerssspssseeee900
23

---

ChantomWiki, On-Demand Datascts for Reasoning and Retrieval Evaluation
Example 9:
Question: Who is the cousin of the person whose occupation is broadcast engineer? Thought 1: Pirst, I need to search for people whose occupation is broadcast engineer.
Action 1: Search (broadcast engineer).
Observation 1: Barabara Beltran ## Family The parents of Barabara Beltran are Dino Beltran, Shelli Beltran. Barabara Beltran's siblings are Aida Wang, Vicki Hackworth. The sisters of Barabara Beltran are Aida Wang, Vicki Hackworth. The mother of Barabara Beltran is Shelli Beltran. The father of Barabara Beltran is Dino Beltran. ## Friends ## Attributes The date of birth of Barabara Beltran is 0989-06-11. The occupation of Barabara Beltran is broadcast engineer. The hobby of Barabara Beltran is meteorology.
Thought 2: Person whose hobby broadcast engineer is Barabara Beltran. The cousin Barabara Beltran is the child of sibling her parent. Barabara Beltran's parents are Dino Beltran and Shelly Beltran. Now I need find the sibling of Dino Beltran and Shelly Beltran. Action 2: SetrieverrticleeDiinn
Beltran].
Observation 2: # Dino Beltran ## Family The parents of Dino Beltran are Brian Beltran, Daisy Beltran. Dino Beltran's s sibling is Orlando Beltran. The brother of Dino Beltran is Orlando Beltran. The mother of Dino Beltran is Daisy Beltran. The father of Dino Beltran is Brian Beltran. The children Dino Beltran are Aida Wang, Barabara Beltran, Vicki Hackworth. The daughters of Dino Beltran are Aida Wang, Barabara Beltran, Vicki Hackworth. The wife of Dino Beltran is Shelli Beltran. ## Friends The friend of Dino Beltran is Alvaro Smock. ## Attributes The date of birth of Dino Beltran is 0958-08-09. The occupation of Dino Beltran is associate professor. The hobby of Dino Beltran is shogi.
Thought 3: The sibling of Dino Beltran is Orlando Beltran. Now need retrieve article about Orlando Beltran and find his child.
Action 3:
Beltran].
Observation 3: Orlando Beltran ## Family The parents of Orlando Beltran are Brian Beltran, Daisy Beltran. Orlando Beltran's sibling is Dino Beltran. The brother of Orlando Beltran is Dino Beltran. The mother of Orlando Beltran is Daisy Beltran. The father of Orlando Beltran is Brian Beltran. ## Friends The friends of Orlando Beltran are Virgil Hackworth, Dominique Smock. # Attributes The date of birth of Orlando Beltran is 0953-10-23. The occupation of Orlando Beltran is petroleum engineer. The hobby of Orlando Beltran is learning.
Thought 4: Orlando Beltran has no child, Beltran. Now I need to retrieve article about Shelli Beltran and find her sibling.
cousin from the side of Dino
Action 4: RetrieveArticle
Beltran].
Observation Shelli Beltran # Family The parents of Shelli Beltran are Alison Smock, Williams Smock. Shelli Beltran's sibling is Stacia Toombs. The sister of Shelli Beltran is Stacia Toombs. The mother of Shelli Beltran is Alison Smock. The father of Shelli Beltran is Williams Smock. The children of Shelli Beltran are Aida Wang, Barabara Beltran, Vicki Hackworth. The daughters of Shelli Beltran are Aida Wang, Barabara Beltran, Vicki Hackworth. The husband of Shelli Beltran is Dino Beltran. # Friends The friends of Shelli Beltran are Brian Beltran, Eli Smock, Isiah Lutz, Leslee Toombs, Lesley Lutz, Ryan Wang. ## Attributes The date of birth of Shelli Beltran is 0958-03-08. The occupation of Shelli Beltran is occupational therapist. The hobby of Shelli Beltran is sociology.
24

---

PhantomWiki: On-Demand Datascts for Reasoning and Retrieval Evaluation
Thought 5: The sibling of Shelli Beltran is Stacia Toombs. Now I need retrieve article about Stacia Toombs and find her child, get the cousin of Barabara Beltran.
Action 5: Ertzievearticlesttaee
Toombs]
Observation 5: # Stacia Toombs ## Family The parents of Stacia Toombs are Alison Smock, Williams Smock. Stacia Toombs's s sibling is Shelli Beltran. The sister of Stacia Toombs is Shelli Beltran. The mother of Stacia Toombs is Alison Smock. The father of Stacia Toombs is Williams Smock. The child of Stacia Toombs is Leslee Toombs. The daughter of Stacia Toombs is Leslee Toombs. The husband of Stacia Toombs is Wilbert Toombs. ## Friends The friends of Stacia Toombs are Brian Beltran, Tsiah Lutz, Leeann Hackworth, Lesley Lutz, Ryan Wang. ## Attributes The date of birth of Stacia Toombs is 0959-03-22. The occupation of Stacia Toombs is actuary. The hobby of Stacia Toombs is finance.
Thought The child of Stacia Toombs is Leslee Toombs. the cousin Barabara Beltran is Leslee Toombs. The answer is Leslee Toombs.
Action 6: Finish[Lesles Toombs].
Example 10:
Question: Who is the graat-grandaughter of the person whose hobby is biology? Thought 1: First, I need to search for people whose hobby is biology.
Action 1: Search[biolog]].
Observation (1) Alvaro Smock
Thought 2: Person whose hobby is biology is Alvaro Smock. New I need to retrieve
article about Alvaro Smock and find his great-grarddaughterr A great- granddaughter is the daughter of the child of the child of the person. Action 2: @etrieverrttilleAAlee
Smock]
Observation 2: Alvaro Smock ## Family The children of Alvaro Smock are Eli Smock, Gene Smock. The sons of Alvaro Smock are Eli Smock, Gene Smock. The wife Alvaro Smock Lannie Smock. ## Friends The friends Alvaro Smock Dino Beltran, Gene Smock, Aida Wang. ## Attributes The date of birth of Alvaro Smock is 0867-07-12. The occupation of Alvaro Smock is osteopath. The hobby of Alvaro Snock is biology.
Thought 3: The children of Alvaro Smock are Eli Smock, Gene Smock. Now need to retrieve article about Eli Smock and Gene Smock and find their child. Action 3: RetrieveArticle [Eli Smock].
Observation
Eli Smock Family The parents of Eli Smock are Alvaro Smock, Lannie Smock. Eli Smock's sibling is Gene Smock. The brother of Eli Smock is Gene Smock. The mother of Eli Smock is Lannie Smock. The father of Eli Smock is Alvaro Smock. ## Friends The friends of Eli Smock are Leisa Lutz, Shelli Beltran, Vicki Hackworth, Virgil Hackworth, Alison Smock, Brian Beltran. ## Attributes The date of birth of Eli Smock is 0901-01-18. The occupation of Eli Smock is retail manager. The hobby of Eli Smock is tether car.
Thought Eli Smock has no child. Now need retrieve article about Gene Smock and find his child. Action 4: RetrieveArticlee (Gene Smock]
Observation 4: # Gene Smock #t Family The parents of Gene Smock are Alvaro Smock, Lannie Smock. Gene Smock's s sibling is Eli Smock. The brother of Gene Smock is Eli Smock. The mother of Gene Smock is Lannie Smock. The father of Gene Smock Alvaro Smock. The child Gene Smock Williams Smock. The son of Gene Smock is Williams Smock. The wife of Gene Smock is Dominique Smock. # Friends The friends of Gene Smock are Leeann Eackworth, Leisa Lutz, Ricardo Hackworth, Alvaro Smock, Dominique Smock. ## Attributes The date of
25

---

PhantomWike On-Demand Datascts for Reasoning and Retrieval Evaluation
In-Context
RAG
Agentic
ZeroShot CoT
ZeroShot- RAG CoTRAG
ReAct
0.75
0.75
0.75
F1
0.5
0.5
0.5
0.25
0.25
0.25
10
15
Reasoning steps
Reasoning steps
Reasoning steps
DeepSeek-R1-3
Gemini-1.5 Fl
GPT-4o
Llama-3.3-708
Figure 7. F1 scores versus question difficulty, measured by reasoning steps, for questions with exactly solution, We observe similar trends as in Figure 3, demonstrating that the number of solutions is not solely responsible for the drop in performance.
birth Gene Smock is 0898-08-16. The occupation Gene Smock is immunologist. The hobby of Gene Smock is architecture. Thought 5: The child of Gene Smock is Williams Smock. Now need to retrieve article about Williams Smock and find his daughter, to get the great- granddaughter of Alvaro Smock.
Action 5: RetrieveArticle [Williams Smock]
Observation Williams Smock ## Family The parents of Williams Smock are
Dominique Smock, Gene Smock. The mother of Williams Smock is Dominique Smock The father of Williams Smock is Gene Smock. The children of Williams Smock are Shelli Beltran, Stacia Toombs. The daughters of Williams Smock Shelli Beltran, Stacia Toombs. The wife of Williams Smock is Alison Smock. ## Friends The friend of Williams Smock is Lannie Smock. ## Attributes The date of birth of Williams Smock is 0926-04-04. The occupation of Williams Smock clinical biochemist. The hobby of Williams Smock is social studies. Thought The daughters of Williams Smock are Shelli Beltran, Stacia Toombs. So the great-grandduugheess of Alvaro Smock, whose hobby is biology, are Shelli Beltran, Stacia Toombs. The answer is Shelli Beltran, Stacia Toombs. Action 5: Finish [Shelli cotttaancoontantss..o antwer_snssserssee Stacia Toomba]. (END OF EXAMPLES)
Now answer the following question: Question: { ((question) ([scratchpad)
D. Additional Results
D.1. Multi-Hop RAG Baselines
We include two RAG baselines that interleave reasoning with retrieval: Self-Ask (Press et al., 2023) and IRCoT (Trivedi et al., 2023). We use the implementation from FlashRAG examples suited to PhantomWiki. Table includes the results of these baselines on the same Phantom Wiki instances as in Table 2. Notably, IRCoT improves over ZEROSHOT-B RAG and COT-RAG, but struggles on questions requiring many reasoning steps, as revealed by Figure 8.
and write few-shot
26

---

PhantomWike On-Demand Datascts for Reasoning and Retrieval Evaluation
Table Fl scores (in %) for Llama-3.3-70B with RAG prompting. ZEROSHOT-RAG and COT-RAC use UAE-LARCE retriever model (top-k=< 4); SELF-ASK and IRCoT use the BM25 retriever (top-k= 5). We report mean + standard error across dataset generation seeds.
Universe Size
ZeROSHOT-RAG
COT-RAG
SELF-ASK
IRCoT
50
17.55 + 2.20
20.01 # 1.81 15.77 + 3.22 23.93 + 0.93
D.2. Article Rephrasing
To explore how LLMs can be used to improve the realism of our templated articles, we instruct Llama-3.3-70B rephrase articles in corpora of size 50 from Figure 3. Our first, "short" prompt instructs the LLM condense the templated articles, while still retaining all factual information:
Shuffle and rephrase the following wikipedia-like article. Keep ALL facts like name, relation, date, occupation, hobby and gender exactly as stated. Do not add ANY new information.
Our second, "long" prompt permits the LLM introduce new facts, without contradicting existing facts:
Transform the following factual article into engaging harrative profile:
1.
Add colorful descriptions and personality traits that might be inferred
2.
Create vivid backstory about how relationships formed
3. Elaborate on the person's career path and achievements
4. Describe their hobbies rich detail, including when they might have started them
5. Include hypothetical quotes from family members
6. Imagine and describe the person's daily routine
7.
Add details about where they might live and their home environment
8.
Write in warm, personal tone as if you' ve known the subject for years Maintain a natural flow
9.
When transforming the article, ensure the following:
1.
All factual information remains 1008 accurate (names, relationships, dates,
ZeroShot-RAG
CoT-RAG
0.75
SelfAAk
IRCoT
E1
0.5
0.25
15
Reasoning steps
Figure 8. F1 scores versus question difficulty, measured by reasoning steps. ZEROSHOT-RAN RAG and COT-RAG use the UAE-LARGE VI retriever model (top-k= 4); SELF-ASK and IRCoT use the BM25 retriever (top-k= For all methods, use Llama-3.3-70BB as the generator model. We report mean standard error across 3 datasct generations seeds with universe size n = 50.
27

---

PhantomWike On-Demand Datascts for Reasoning and Retrieval Evaluation
Templated Rephrased (Short)
Templated Rephrased (Short)
Rephrased (Long)
Rephrased (Long)
F1
0.25
0.25
0
10 15
10
15
Reasoning steps
Reasoning steps
(a) ZEROSHOT Prompting
(b) COT Prompting
Figure F1 scores versus question difficulty, measured by reasoning steps, for various article styles. We rephrase templated articles generated by PhantomWik, with Llama-3.3-70B. Then for evaluation, replace templated articles with M.Meprhhraee articles in ZEROSHOT and CoT prompts from Appendix C.I and Appendix C.3 respectively. As Figure 3, we report mean +: standard error across 3 dataset generation seeds for universe size 50. We use evaluation sampling typer-parameee from Section 4.
occupation, hobby, gender)
2. No new information is added or implied
3. The content is presented in natural, flowing narrative rather than just shuffling bullet points
4. Different sections are noorganized in way that still makes logical sense
5. Sentence structures and vocabulary are varied from the original
This is the article to transform:
To generate articles of either styles, we prompt Llama-3.3-70B with temperature = 0.7, top-p = 0.9, and max output tokens 1024. We include an example short article Appendix D.2.1 and an example long article in Appendix D.2.3. Figure shows plots of Fl scores vs question difficulty of Llama-3.3-70B using ZEROSHOT and CoT prompting on these LLM-rephrased articles. We find that regardless of the article style-temlee templated LLM-rephrased F1 scores decline the number of reasoning steps increases.
D.2.1. EXAMPLE OF TEMPLATED ARTICLE
Alison Smock
## Family
The daughters Alison Smoak are Cythia Smock, Shelli Beltran, Stacia Toombs. The husband of Alison Smock is Williams Smock.
# Friends
The friends of Alison Smock are Cortney Parmer, Jamison Baptiste, Wilbert Toombs.
## Attributes
The date of birth of Alison Smock is 0929-10-28. Alison Smock is broadcast engineer. The hobby of Alison Smock is meteorology.
The occupation
The gender of Alison Smock is female.
28

---

PhantomWikk Wiki: On-Demand Datascts for Reasoning and Retrieval Evaluation
D.2.2. EXAMPLE SHORT REPHRASED ARTICLE
The following article was generated with Llama-3.3-708B using 1 118 output tokens.
Alison Smock, female born on 0929-10-28 is broadcast engineer with hobby of meteorology. She is married to Williams Smock and they have daughters named Cythia Smock, Shelli Beltran, and Stacia Toombs. Alison Smock's social circle includes friends such as Cortney Parmer, Jamison Baptiste, and Wilbert Toombs.
D.2.3. EXAMPLE OF LONG REPHRASED ARTICLE
The following article was generated with Llama-3.3- 3.B-70B using 669 output tokens. 1lama--.3-700 incorrectly stated the date-of-birth (red) and omitted the last names of the friends (orange).
Alison Smock, a brilliant and charismatic broadcast engineer, has been fascinated by the intricacies of technology and the wonders of the atmosphere for as long as anyone can remember. Born on October 28, 1929, Alison grew up with an insatiable curiosity and a passion for learning that would serve her well throughout her life. As a young girl, she always tinkering with gadgets and machinery, trying to understand how they worked, and this innate curiosity eventually led her to pursue a career in broadcast engineering.
As she navigated her professional path, Alison met her husband, Williams Smock, and together they built loving family. They had three beautiful daughters, Cythia, Shelli, and Stacia, who brought immense joy and fulfillment to their lives. According to her daughter Cythia, "Mom was always there for us, supporting us and encouraging us to pursue our dreams, no matter how big or small they seemed." Alison's family was her rock, and she cherished every moment she spent with them.
Alison's 's career broadcast engineer was marked by numerous achievements and accomplishments. She was known for her exceptional cobblem-solving skills and her ability to stay calm under pressure. Her colleagues admired her dedication and work ethic, and she quickly became an invaluable member of her team. When she wasn't working, Alison loved to indulge in her hobby of meteorology. She found the study of weather patterns and atmospheric conditions absolutely captivating, and she spent hours poring over books and watching documentaries on the subject. As her friend Cortney Parmer recalls, "Alison would often call me up and say, "Did you see that storm system moving in? I've been tracking it all day!" She was like a kid in a candy store when it came to weather."
In her free time, Alison enjoyed spending time with her friends, including Cortney, Jamison Baptiste, and Wilbert Toombs. They would often meet up for coffee or go on hikes together, discussing everything from the latest news to their personal lives. Wilbert, who is also the husband of Alison's daughter Stacia, says, "Alison is like a mother to me. She's ' always been so kind and supportive, and I admire her intelligence and passion for life."
A typical day for Alison begins early, with a quiet moment of reflection and cup of coffee on her back porch, surrounded by the beauty of nature. She lives in a cozy home filled with books, gadgets, and mementos from her travels, and she loves to spend time in her garden, tending to her plants and watching the birds. As she gets ready for work, Alison feels grateful for the life she's built and the people who bring joy and love into her world. As her husband Williams says, "Alison is an amazing woman, and I feel so lucky to have her by my side. She's intelligent, kind, and beautiful, and admire her every day." With her quick wit, sharp mind, and generous spirit, Alison Smock is a true treasure, and those who know her feel fortunate to have her in their lives.
D.3. Fine-tuning on Phantom Wiki Exirrimext Details
PhantomWikt creates datasets with relationships, names, attributes populated randomly on-demand, ensuring that each dataset instance is unique and robust to LLMs memorizing dataset facts. Even so, it is possible that fine-tuning on PhantomWike Wiki-generated datasets could improve performance by leveraging language and template structure in PhantomWike Wiki articles and questions. Accordingly, we fine-tune LLMs on PhantomWile datasets to (1) assess the viability of Phantom Wiki for training L.MSs and (2) test the robustness of Phantom on-demand datasets from memorization.
We generate 10 new Phantom Wiki dataset instances (question depth 20 and universe size 50) amounting to 5K training dnesttio-answer pairs. We then perform full fine-tuning of Wwen2.550.5B-Innttrue (Qwen Team, 2024) and parameter- efficient fine-tuning of Qwen2.5-3B 3-Btttrruutttt with LoRA (Hu et al., 2022) applied to all linear layers. We use node of
29

---

PhantomWike On-Demand Datascts for Reasoning and Retrieval Evaluation
A 100 GPUs each with 80GB GPU VRAM for each experiment. For each base model, we employ popular fine-tuning algorithms:
1. The first is Group Relative Policy Optimization (GRPO) (DeepSeek AI, 2024), where we elicit chain-of of-thought response from the LIM using COT template (Appendix C.3) and return reward the Fi-score of the generated answer list against the ground-truth answer list.
We use the GRPO implementation from Huggingface's TRL library. We set max prompt length 4096, sufficient to include our prompt of the 50 articles of the universe, and limit the max completion length to 128. We reserve GPU for generation with vLLM, and use the other 3 GPUs for training to get 6 total generations per step of GRPO update, For full fine-tuning, we set per-device training batch size = 8 and gradient accumulation steps = 8, and for LoRA fine-tuning we set 2 and 4 respectively. We fine-tune for epochs using the AdamW optimizer with initial learning rate set 10-6 for full fine-tuning and 10-4 for LoRA fine-tuning.
2. The second Supervised Fine-tuning (SFT), where we provide all articles and the question using our ZEROSHOT prompt in Appendix C.1 and train on the ground-truth answer list.
Again we use the SFT implementation from Huggingface's 's TRL library and set the max prompt length to 4096. For both full fine- tne-tunnng and LoRA fine-tuning, we set per-device training batch size and gradient accumulation steps 1. We fine-tune for 3 epochs using the AdamW optimizer with initial learning rate set x 10-5 for full fine-tuning and 10 for LoRA fine-tuning.
We evaluate these models on the three Phantom Wiki dataset instances of size 50 and maximum recursion depth 20 (500 questions per dataset instance), the used in Table 2 and Figure 3. Figure 10 shows F1 scores function of reasoning steps, comparing base LLMs with fine-tuned ones, and Table 3 shows the aggregate Fl scores. To adhere to the fine-tuning experiment setup, SFT-trained LLMs are evaluated with ZEROSHOT prompt, and GRPO-trained with CoT prompt.
For both full fine-tuned Qwen2.5-0.. 10055--Instruc and I LoRA fine-tuned -wee2.5-3B-Instreee we see that GRPO improves performance for all question difficulties compared to COT and ZEROSHOT prompting. We find that SFT does not improve performance over ZEROSHOT prompting. Notably, performance of fine-tuned LLMs still far from optimal and the Fl scores struggle as question difficulty increases.
ZeroShot
ZeroShot
CoT
CoT
0.75
SFT
0.75
SFT
GRPO
GRPO
0.5
0.25
0.25
15
Reasoning steps
Reasoning steps
(a) S0wen.55-0.5B-Instrue (full fine-tuning)
(b) 1wwe2.5--3B-Insrrue (LoRA fine-tuning)
Figure 10. F1 scores versus question difficultyy, measured by reasoning steps. We report mean + 1 standard error across 3 dataset generation seeds for universe size 50. ZEROSHOT and CoT are prompting methods; SFT and GRPO fine-tuning methods.
E. Example of Small Corpus
We generate a universe of size n 4, setting the number of family trees to be one. We include the articles below.
30

---

PhantomWiki On-Demand Datascts for Reasoning and Retrieval Evaluation
E.I. Article
Claud Colin
## Family The brother of Claud Colin is Mckinley Colin. The mother of Claud Colin is Ramona Colin.
The father of Claud Colin is Danilo Colin.
Friends
The friend of Claud Colin is Danilo Colin.
## Attributes
The date of birth of Claud Colin is 0241-12-06. The occupation of Claud Colin is academic librarian. The hobby of Claud Colin is amateur astronomy. The gender of Claud Colin is male.
E.2. Article
Danilo Colin
## Family
The sons of Danilo Colin are Claud Colin, Mckinley Colin. The wife of Danilo Colin is Ramona Colin.
## Friends
The friends of Danilo Colin are Mckinley Colin, Ramona Colin, Claud Colin.
## Attributes
The date of birth of Danilo Colin is 0219-08-09.
The occupation of Danilo Colin is clinical research associate. The hobby of Danilo Colin is crystals.
The gender of Danilo Colin is male.
31

---

Friends PhantomWiki: On-Demand Datascts for Reasoning and Retrieval Evaluation
E.3. Article
Mckinley Colin
# Family The brother of Mckinley Colin is Claud Colin. The mother of Mckinley Colin is Ramona Colin. The father of Mckinley Colin is Danilo Colin.
The friends of Mckinley Colin are Ramona Colin, Danilo Colin.
## Attributes
The date of birth of Mckinley Colin is 0246-10-18. The occupation of Mckinley Colin is museum curator. The hobby of Mckinley Colin is stamp collecting.
The gender Mckinley Colin is male.
E.4. Article
Ramona Colin
## Family
The sons of Ramona Colin are Claud Colin, Mckinley Colin. The husband of Ramona Colin is Danilo Colin.
## Friends
The friends of Ramona Colin are Danilo Colin, Mckinley Colin.
## Attributes
The date of birth of Ramona Colin is 0219-09-08. The occupation of Ramona Colin is technical sales engineer. The hobby of Ramona Colin is trainspotting.
The gender of Ramona Colin is female.
32


Reviews:
[{"review_id": "efhznPuPX6", "reviewer": "Reviewer_YauR", "summary": "The paper presents PhantomWiki, a pipeline that generates synthetic large-scale document corpora and question-answer pairs. PhantomWiki works by generating articles containing facts about characters (e.g., “The job of x is y.”), and generating question-answer pairs from templates. PhantomWiki allows generating large corpora, exceeding the context length of current LMs. The paper experiments with in-context prompting, RAG, and agentic RAG models on PhantomWiki, and demonstrates that current models struggle with large corpora and complex questions generated using the PhantomWiki pipeline.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and suggestions, which we believe improve our manuscript significantly. We are pleased that you find PhantomWiki a **promising alternative to the popular “needle-in-a-haystack” test**. We address your comments below and will add results for suggested experiments to our manuscript.\n\n**1. Use of templated articles**\n\nThank you for pointing this out. **We have added an experiment where we use Llama-3.3-70B to rephrase our templated articles** (see prompts: https://imgur.com/a/ffvugvg and example generations: https://imgur.com/a/wFvjAFP) and evaluated on these rephrased PhantomWiki corpora: see https://imgur.com/a/DQuvQIA. We find the **drop in F1 scores versus question difficulty to be similar** to our original experiments (Table 2 of the manuscript). On the other hand, templated articles are cheap, fast, and free of LLM hallucinations, while enabling compelling evaluations for LLM reasoning capabilities. Importantly, PhantomWiki is designed to be modular to incorporate LLM-generated articles, as future research in reducing LLM hallucination advances.\n\n**2. Evaluating LLMs finetuned on PhantomWiki data**\n\n> Second, as the data was generated via templates, I am concerned it will be easy to game with additional training (for example by using the same templates to generate training data).\n\n> Additionally, as the paper proposed a new evaluation method for RAG, it will be helpful to evaluate the gains from fine-tuning on the synthetic distribution (see Claims and Evidence).\n\nThank you for suggesting this experiment. As the relationships, names, and attributes in each PhantomWiki universe are generated randomly on-demand, we believe that naive training on PhantomWiki data can only yield limited improvements. To support this claim, we have a **new experiment** where **we generate 10 new PhantomWiki dataset instances** (question depth 20 and universe size 50) amounting to 5K training question-answer pairs. We then **perform full fine-tuning** of Qwen2.5-0.5B-Instruct and parameter-efficient fine-tuning of Qwen2.5-3B-Instruct with LoRA on all linear layers. \n\nFor each base model, we employ two popular training algorithms. The first is Group Relative Policy Optimization (GRPO) from [Shao et al.](https://arxiv.org/abs/2402.03300) with an F1-score reward between 0 and 1. We use the CoT prompt template from App. C.4, a batch size of 32, and 8 generations per prompt to sample. The second is supervised fine-tuning (SFT) with answer(s) as the ground-truth label. We use the zeroshot prompt in App. C.2 and a batch size of 4. For all training experiments, we train for 3 epochs (or until convergence) using the AdamW optimizer with initial learning rate set to $5\\times 10^{-6}$ for full fine-tuning and $10^{-4}$ for LoRA fine-tuning. \n\nWe then evaluated the fine-tuned models using the PhantomWiki instances of size $n=50$ from Table 2: see https://imgur.com/a/QHftpCM. For Qwen2.5-0.5B, we find that GRPO and SFT both improve F1 compared to prompting-based methods, likely due to improved ability to output the proper answer format. For Qwen2.5-3B, we find that GRPO improves F1 slightly, whereas SFT worsens F1, likely due to overfitting on the training samples. These experiments show that **further advances beyond fine-tuning are needed to truly close the gap on PhantomWiki**; we hope that PhantomWiki will serve as a valuable tool for future research on LLM reasoning and retrieval.\n\n**3. Additional multi-hop RAG baselines e.g. IRCoT and Self-Ask, discussion on synthetic data from KG**\n\nFollowing your suggestion, **we have performed additional evaluation using IR-CoT and Self-Ask** with BM25 as the retriever and Llama-3.3-70B-Instruct as the generator model: see https://imgur.com/a/OMjtf3t. Specifically, we used the implementations from FlashRAG ([Jin et al. 2024](https://github.com/RUC-NLPIR/FlashRAG)) and re-wrote the few-shot examples where needed to match the formatting of PhantomWiki questions. We find that both IR-CoT and Self-Ask are competitive with ZeroShot-RAG and CoT-RAG. However, when decomposing model performance on question difficulty (like Figure 3), **both IR-CoT and Self-Ask struggle like other prompting methods**. In fact, this is a key contribution of PhantomWiki: quantitatively decomposing performance of prompting methods and LLMs on axes of reasoning and retrieval.\n\nNotably, IR-CoT and Self-Ask alternate between reasoning and retrieval like agentic prompting (e.g., ReAct). A key difference: ReAct uses LLM-driven retrieval via tools, while multi-hop RAG methods delegate retrieval to an external retriever—decoupling reasoning and retrieval. We’ll highlight this nuance in the revised manuscript. We’ll also add a brief discussion about generating data from knowledge graphs.\n\nGiven these **new experiments** support the paper's message and **strengthen our empirical results**, we would like to politely ask you to consider increasing your score. Thank you again!", "claims_and_evidence": "While the paper has merits, and I believe the proposed method can be a promising alternative to the popular “needle-in-a-haystack” experiments, I believe that the claim that the current method is reliable in evaluation of RAG models is problematic.\n\nMy main concern regards the templated data generation approach. Specifically, the distribution of the generated documents (e.g., “The job of x is y.”) and questions (templates are presented in B.2), seems to be very different from realistic RAG settings. Second, as the data was generated via templates, I am concerned it will be easy to game with additional training (for example by using the same templates to generate training data).", "methods_and_evaluation_criteria": "I believe the paper could benefit from additional multi-hop RAG basslines (e.g., IR-CoT, Self-Ask). \n\nAdditionally, as the paper proposed a new evaluation method for RAG, it will be helpful to evaluate the gains from fine-tuning on the synthetic distribution (see Claims and Evidence).", "theoretical_claims": "None.", "experimental_designs_or_analyses": "I checked the soundness of the experiments in Sections 4-6.", "relation_to_broader_scientific_literature": "By generating a large corpora with matching question-answer pairs that is robust to data contamination teh paper proposes an interesting evaluation method for long-context reasoning, which has several benefits over previous approaches (e.g., “needle-in-a-haystack”). The paper also compares in-context prompting, RAG, and agentic-RAG models, showing all approaches struggle as the number of reasoning steps increases.", "essential_references_not_discussed": "I believe that the paper can benefit from a discussion regarding multi-hop RAG (e.g., IR-CoT), and synthetic data generation from KG (many methods use a KG, e.g., Wikidata, and then generate questions synthetically using the relations).", "other_strengths_and_weaknesses": "The paper is overall well-written and easy to follow.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lMJeivKlJs", "reviewer": "Reviewer_8LpG", "summary": "The manuscript introduces PhantomWiki, a generator for fictional universe of characters in the form of a fandom wiki. The knowledge graph for characters, their relations, and facts about them, are generated by sampling simple distributions. Articles are generated from these facts using templates. Questions are generated using templates, and answers are generated using Prolog solver on the original facts. Different size of fictional wiki can be generated using different numbers of characters, and easier/harder questions may be considered in terms of the number of \"hops\" required to answer them. PhantomWiki thus generates benchmarks involving novel \"facts\" to evaluate different language models and techniques in different regimes.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Question 1 / Evaluating language models fine-tuned on PhantomWiki data**\n\n> Another avenue is to try to actually fine-tune models on multiple instances of PhantomWiki, and assess how this affects performances on fresh instances.\n\nWe add a **new experiment** and find that **fine-tuning on PhantomWiki data helps** performance on held-out instances, but only **to a limited extent**: see https://imgur.com/a/QHftpCM (please see also our rebuttal to reviewer YaUR for full experiment details). Both prompting and fine-tuning approaches struggle as question difficulty increases. Thank you for suggesting this experiment idea! We are excited to see how PhantomWiki inspires research in LLM reasoning.\n\n**2. Response to Claim 3 and Question 2**\n\n> PhantomWiki itself is not subjected to any direct evaluation of fitness for a purpose, nor are its scores related to existing benchmarks for consistency and/or redundancy assessment, nor is the score variability between two instances of PhantomWiki assessed. \n\nWe appreciate your framing of “what would negative results look like?”, which we find especially helpful for clarifying our benchmark’s utility. Below, we outline how PhantomWiki provides actionable assessments and how we plan to revise the manuscript.\n\nA poorly designed benchmark might fail to appropriately differentiate models and/or distinguish between reasoning paradigms. Our experiments suggest the opposite: **PhantomWiki differentiates performance** based on both model architecture and prompting techniques, and **elucidates failure modes** (e.g., hallucinated intermediate hops or context retrieval mismatches) that align with known model weaknesses. \n\nBased on our evaluation results, PhantomWiki is _not_ a good benchmark for reading comprehension, as evidenced by the near perfect F1 scores of all LLMs/prompting techniques for questions that require 1 reasoning step. Indeed, PhantomWiki is meant to complement more complex reading comprehension benchmarks like DROP and SQuAD. Nonetheless, it is a good benchmark for evaluating **multi-step and multi-branch reasoning**. Quantitatively, we can see this from the rapid drop in performance in Figure 3 as the number of reasoning steps increases. \n\nOne area where LLMs struggle is with complex relations. For example, Llama-3.3-70B-Instruct struggles with “Who is the great-grandchild of X?”, not because it doesn’t know what great-grandchild means, but rather because of a high step count (child of child of child) and a high branching factor (having to find the children of each grandchild). One actionable insight is to improve language models’ ability to keep track of not only multiple steps (this is corroborated by the Self-Ask paper, which highlights a compositionality gap), but also multiple branches. PhantomWiki thus **sheds light on both of these challenges** in LLM reasoning. We will include the discussion of failure modes of different models in the revision.\n\nIn terms of benchmark stability, we run all the evaluations with the same hyperparameters on the first three seeds of generated PhantomWiki instances and **measure variability** through standard deviation across runs. This provides evidence that **PhantomWiki is robust** to sampling variance and not overly sensitive to individual instance characteristics.\n\n> The use of a single template to generate all entries is very disputable (and part of my concerns in Claim 3), especially considering that this template basically amounts to concatenating sentences of the form \"The X of Y is Z.\"\n\n**In a new experiment, we use Llama-3.3-70B-Instruct to rephrase our templated articles** (see prompts: https://imgur.com/a/ffvugvg and example generations: https://imgur.com/a/wFvjAFP), which we then use for downstream evaluation: see https://imgur.com/a/DQuvQIA (please see also rebuttal w1Q5 for further details). We report similar performance trends as in Figure 3 from our manuscript, and in fact, using rephrased articles only makes PhantomWiki more challenging. Thus templated articles have the benefit of being cheaper, faster, and free of LLM hallucinations, while still providing valuable insight into LLM reasoning capabilities. Importantly, PhantomWiki is designed to be modular to incorporate LLM-generated articles, as future research in reducing LLM hallucination matures.\n\n**3. Additional references on long-context and friendship graphs**\n\nWe greatly appreciate the reviewer’s suggestion of connecting to the long-context literature. We will revise the Related Works section to include them. We also plan to support more realistic friendship graphs, especially heavy-tailed degree distributions, in future versions of PhantomWiki.\n \n**4. Example of a small universe**\n\nPlease see https://imgur.com/a/QLhC54R.\n\nWe hope the new experiments and discussion addressed your concerns and strengthened our results, and if so we would like to politely ask you to consider raising your score. Thank you again!", "claims_and_evidence": "The manuscript does not provide an explicit list of claims, so I identified some myself.\n\n### Claim 1: PhantomWiki can be tuned to benchmark different aspects/regimes\n\nThe benchmark can be made smaller or larger than a model's context length, and the questions can be made to require more or less documents to be answered. Varying these parameters yield consistent results on In-Context, RAG and Agentic strategies. Ignoring eventual Claim 3 issues (below), I judge that Claim 1 is supported by evidence.\n\n\n### Claim 2: PhantomWiki is resistant to data contamination\n\nBy construction, models evaluated on PhantomWiki cannot memorize the exact answer to a question, nor any of the key information required in the intermediate steps, because this data is generated on-the-fly, immediately before evaluation.\n\nHowever, once PhantomWiki is release in the wild, models can be trained, fine-tuned, etc. to perform better on PhantomWiki. Indeed, the probability distribution over the different generated datum can be learned, and learning how to best extract key information pieces from template-generated documents is easy for LLMs.\n\nMy assessment is that, while there exists a weaker claim that is supported, the claims made in the current manuscript are too broad. This is the topic of my Question 1 below.\n\n\n### Claim 3: (Implicit) PhantomWiki is a good/useful benchmark that assesses aspects that other benchmarks don't assess\n\nAssuming that there is no bug in the code, there should, by construction, be valid answers(s) planted in the dataset for each question (and the manuscript mentions that the case where there are multiple correct answers is handled properly). In a sense, the benchmark should be \"more valid\" than one that relies on human or LLM annotations.\n\nHowever, although the manuscript presents experimental results for combinations of LLMs and reasoning/retrieval paradigms (In-Context, RAG, Agentic), PhantomWiki **itself** is not subjected to any direct evaluation of fitness for a purpose, nor are its scores related to existing benchmarks for consistency and/or redundancy assessment, nor is the score variability between two instances of PhantomWiki assessed. \n\nMy Question 2 below pertains to this topic.", "methods_and_evaluation_criteria": "The results presented make sense on their own. However, the purpose of the manuscript is to introduce a benchmarking method, and that benchmarking method itself is not directly evaluated. See my comments on Claim 3 and Question 2.", "theoretical_claims": "The description of how the original graph/data is generated makes sense, though is quite simplistic. The use of a single template to generate all entries is very disputable (and part of my concerns in Claim 3), especially considering that this template basically amounts to concatenate sentences of the form \"The X of Y is Z.\". The generation of questions and answers make sense, though I did not verify in depth the grammars and templates.", "experimental_designs_or_analyses": "The experimental design uses the introduced benchmark to assess different LLMs in different retrieval/reasoning modes. For that purpose, the experimental design appears decent to me. However, as mentioned before, there are no direct experiments evaluating the benchmark itself.", "relation_to_broader_scientific_literature": "The related work section focuses on agent and/or tool-use evaluation benchmarks, but neglects \"long-context\" ones. Some suggestions are provided in the next section.\n\n> Importantly, none of these benchmarks creates the underlying corpus, a limitation which we bridge in this work.\n\n∞Bench (Zhang et al., 2024) uses a key entity replacement strategy to this end. However, it is subjected to the \"Beethoven might have met Mozart\" failure mode mentioned by the authors. RepLiQA (Monteiro et al. 2024) have humans generate the corpus. However, the questions can all be answered using a single document.\n\nThis work could be framed as the generation and verbalization of a knowledge graph. Some relevant works are listed below.", "essential_references_not_discussed": "Some long-context benchmarks:\n\n- Hsieh et al. RULER: What’s the Real Context Size of Your Long-Context Language Models? COLM 2024. https://openreview.net/pdf?id=kIoBbc76Sy\n- Wang et al. Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA. EMNLP 2024. https://aclanthology.org/2024.emnlp-main.322.pdf\n- Zhang et al. ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens. ACL 2024. https://aclanthology.org/2024.acl-long.814.pdf\n\nOn creating the corpus:\n\n- Monteiro et al. RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content. NeurIPS 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/2b23626015b6311369e95a70735cbb72-Paper-Datasets_and_Benchmarks_Track.pdf\n\nKnowledge graphs:\n\n- Agarwal et al. Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training. ACL:HLT 2021. https://aclanthology.org/2021.naacl-main.278.pdf\n- Ye et al. Generative Knowledge Graph Construction: A Review. EMNLP 2022. https://aclanthology.org/2022.emnlp-main.1.pdf", "other_strengths_and_weaknesses": "The core idea of automatically generating a benchmarking corpus, with associated questions and answers, parametrized in terms of corpus size and question complexity, is a promising one.", "other_comments_or_suggestions": "> friendship graph using the Erdős–Rényi model\n\nFriendship (social) networks, real and fictional, diverge in many way from this model. In my personal opinion, the two main aspects worth capturing are transitivity (\"I am more likely to be friend with the friend of my friend than with with a random person\") and heavy-tailed degree distribution (\"most people have few friends, few people have lots of friends\"). If the authors are interested to learn more, they may start with https://en.wikipedia.org/wiki/Triadic_closure and https://en.wikipedia.org/wiki/Scale-free_network for these two aspects, respectively, and https://en.wikipedia.org/wiki/Social_network_analysis for a more general discussion. No actual changes are requested on this manner: these issues are very weakly coupled with the manuscript's concrete goals.", "questions_for_authors": "### Question 1: Do you have a solution to propose for my concerns with Claim 2?\n\nOne possible avenue is to tone down the resistance to data contamination claims. Another avenue is to try to actually fine-tune models on multiple instances of PhantomWiki, and assess how this affects performances on fresh instances. In any case, please state what edits would be made to the manuscript.\n\n### Question 2: Can you provide more direct evidences that PhantomWiki provides actionable assessments of language models?\n\nOne possible avenue is to relate the results on PhantomWiki to pre-existing benchmarks and/or human evaluations. Other avenues involve designing and running new experiments, but I understand that time is short. Finally, perhaps I've missed something obvious, and all you have to do is to explain it better to me.\n\nIn any case, please keep in mind the following question \"**What would negative results look like?**\". In the (potentially counterfactual) scenario where PhantomWiki turned out to not be a useful benchmark, what different observations would have been made? I acknowledge that the narrative consistency of results mentioned in Claim 1 is weak evidence in PhantomWiki's favour, but I'm looking for something more direct.\n\n### Question 3: Could you provide examples of a small corpus?\n\nSay, generate $n=4$ articles in an appendix, with some associated question/answer pairs? Figure 2 (2) is very minimal...", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fbJGipLXZX", "reviewer": "Reviewer_GF7T", "summary": "This paper presents a solution for creating a high-quality benchmark to evaluate the RAG and reasoning abilities of LLMs. Specifically, the proposed method, PhantomWiki, introduces a novel pipeline for generating unique, factually consistent document corpora with diverse question-answer pairs for evaluation. PhantomWiki generates a new instance on demand for each evaluation, effectively mitigating data leakage and inflated performance issues. By varying the difficulty of questions and the size of the corpus, the framework disentangles reasoning from retrieval capabilities.\n\nThrough experiments involving various document sizes, question difficulties, and frontier long-context LLMs, PhantomWiki demonstrates itself as a challenging benchmark for state-of-the-art models. It offers a scalable and leakage-resistant evaluation method for assessing reasoning, retrieval, and tool-use abilities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and encouraging assessment of PhantomWiki. We are pleased that you find PhantomWiki a high-quality benchmark for fine-grained evaluation of reasoning and retrieval. We are also delighted to hear that our work is **well-documented**, and presents **solid experimentation and analysis**. We address your concerns below and outline how we plan to revise the paper accordingly.\n\n**1. Multi-hop RAG baselines**\n\n> This paper could be improved by incorporating more recent baselines. e.g. IRCoT\n\nWe add a **new experiment** with results of IRCoT from Trivedi et al. (2022) and Self-Ask from Press et al. (2022) in comparison to ZeroShot-RAG and CoT-RAG: see https://imgur.com/a/OMjtf3t. We would like to point out that **IRCoT and Self-Ask continue to struggle on questions with high difficulty**, measured by number of reasoning steps. We will include a more detailed comparison of multi-hop RAG to Agentic prompting techniques in our main paper.\n\n**2. Comparison to other existing benchmarks**\n\n> While using a context-free grammar and Prolog to generate question-answer pairs is intriguing, the concept of generating questions beyond the pre-training set of LLMs is not novel. \n\nWe appreciate the reviewer’s observation regarding prior benchmarks that also generate dynamic or leakage-resistant question-answer pairs. However, we respectfully argue that **PhantomWiki is fundamentally different** in motivation, construction, and evaluative capabilities compared to these works:\n- ToolQA [4] focuses on tool-use evaluation, involving curated tools and questions that explicitly require external tools (e.g., calculators, databases). While it minimizes data overlap, its scope is about measuring LLMs’ ability to use predefined tools. In contrast, PhantomWiki evaluates **reasoning and retrieval in a self-contained world** and no external tools are assumed. The choice of external tools is left to the LLM and the prompting method.\n- REALTIME QA [5] and FreshQA [6] both target dynamic, real-world, up-to-date question answering, emphasizing temporal freshness and world knowledge. These benchmarks evaluate how well LLMs adapt to new information by querying real events or fast-changing facts. Our work takes the opposite approach: we **intentionally avoid real-world knowledge to enable clean disentanglement of reasoning and retrieval**. \n\nThe core novelty of PhantomWiki lies **not just in generating unseen questions**, but in generating **entire fictional universes**, complete with articles, knowledge graphs, and questions. As you also point out, our datasets are **coherent**, and allow for **fine-grained control** over question difficulty and retrieval complexity—these functionalities distinguish PhantomWiki from [4,5,6]. LLMs can thus be evaluated on-demand on their ability to navigate and reason over an unseen yet structured universe.\n\nWe appreciate the reviewer’s framing—you're right that our main contribution lies in the fine-grained controlled evaluation that fictional synthetic universes offer. We’ll make this positioning clearer in the revised manuscript and expand the related work section to reflect this comparison. \n\n**3. On Question Diversity and Domain Scope**\n\n> The generated questions tend to be less diverse, primarily focusing on reasoning between personal relationships and objects. \n\nWe appreciate the reviewer’s point about the limited scope of the current question types, which primarily focus on relationships and attributes within a fictional universe. This design was intentional for the initial version of PhantomWiki, as it allowed us to **carefully control the reasoning complexity** and ensure data consistency and leakage resistance.\n\nThat said, we completely agree that increasing the diversity of question types—especially to include temporal, causal, and dynamic aspects of the universe—would significantly expand the benchmark’s utility and better reflect real-world reasoning demands. \n\nDue to space constraints in this paper we are planning future work on extending PhantomWiki to support temporal components, such as characters aging, changing jobs, or forming new relationships over time. This will enable questions involving temporal reasoning, event sequences, and dynamic changes in the knowledge graph. These additions would naturally move PhantomWiki closer to the evolving and dynamic nature of benchmarks like REALTIME QA [5] and FreshQA [6], while still retaining the **key benefits of synthetic control and contamination resistance**.\n\nWe’ll highlight these future directions more clearly in the discussion section of the revised manuscript, and we appreciate the reviewer for raising this valuable point.\n\nThank you again for your detailed review, and please let us know if you have any more questions or concerns that could help improve the paper further!", "claims_and_evidence": "Please see the “Other Strengths And Weaknesses” section below.", "methods_and_evaluation_criteria": "Please see the “Other Strengths And Weaknesses” section below.", "theoretical_claims": "Please see the “Other Strengths And Weaknesses” section below.", "experimental_designs_or_analyses": "Please see the “Other Strengths And Weaknesses” section below.", "relation_to_broader_scientific_literature": "This paper relates to retrieval-augmented generation and reasoning abiltiies of long-context LLMs. It also relates to dataset and benchmark construction and evaluation.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. The proposed benchmark framework, PhantomWiki, is free from data leakage and can be adjusted to control reasoning complexity and retrieval difficulty. The authors also ensure that the generated documents are accurate and error-free, making the dataset high quality.\n2. The details of the dataset construction are well-documented, accompanied by solid experiments with various complexity and context lengths, and analysis for evaluating reasoning and retrieval abilities under different conditions.\n3. If publicly released, the benchmark framework could benefit the community in evaluating future generations of language models.\n\n**Weaknesses:**\n\n1. This paper could be improved by incorporating more recent baselines. The reason RAG performs poorly in Figure 3 is that it retrieves documents only once. However, several recent methods ([1][2][3], etc.) allow for multiple retrievals, significantly enhancing answer accuracy.\n2. While using a context-free grammar and Prolog to generate question-answer pairs is intriguing, the concept of generating questions beyond the pre-training set of LLMs is not novel. For instance, [4][5][6] also generate questions that are resistant to data leakage and are dynamic rather than fixed. The innovation of this paper lies primarily in proposing an unreal universe setting and the ability to dynamically generate questions and corpus sizes of varying difficulty, which may be limited.\n3. The domain of the generated questions is limited to the universe. While this helps prevent data leakage, it also restricts the comprehensive evaluation of LLMs. The generated questions tend to be less diverse, primarily focusing on reasoning between personal relationships and objects. In contrast, the questions in [4] are more diverse, covering temporal, spatial, mathematical, social, scientific, and personal contexts.\n\n[1] Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy\n\n[2] Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions\n\n[3] Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity\n\n[4] ToolQA: A Dataset for LLM Question Answering with External Tools\n\n[5] RealTime QA: What's the Answer Right Now?\n\n[6] FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation", "other_comments_or_suggestions": "Please see the “Other Strengths And Weaknesses” section.", "questions_for_authors": "Please see the “Other Strengths And Weaknesses” section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bboLeiuW7p", "reviewer": "Reviewer_w1Q5", "summary": "The paper proposes PhantomWiki, a benchmark that dynamically generates a fictional universe for evaluating retrieval and multi-hop reasoning. The proposed benchmark can be generated on-demand and is free of data leakage because the fictional events are independent with the real-world.\n\nThe PhantomWiki data generation pipeline first generates a random universe of n characters, as well as their social relationships and personal facts, e.g., date of birth, job, and hobby. The facts are then converted into articles by filling pre-defined templates. The questions are also generated by templates and answers derived using Prolog by inferring from the rules of the universe.\n\nExperiments are carried out with varying difficulty levels and universe sizes: 1/ in-context prompting that the entire universe is placed in the context window, 2/ RAG prompting where a neural retriever is used to search for top-4 most relevant documents, 3/ Agentic prompting where tool use is enabled.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback on the design of PhantomWiki and for acknowledging its novelty in providing a scalable and data leakage-resistant evaluation framework. We address your concerns in turn.\n\n**1. Using templated articles**\n\n> Article content and questions lack realism. The articles are generated using templates, are very short and lack realism for real-world retrieval augmented generation applications.\n\nWe agree that the current articles and questions are intentionally stylized and minimalistic due to their templated construction. To explore how LLMs can be leveraged to improve the realism of PhantomWiki articles, we **add new experiments using Llama-3.3-70B-Instruct to rephrase the articles**. We prompt the model in two ways (see https://imgur.com/a/ffvugvg): the “short” prompt instructs the LLM to paraphrase the templated articles, while still retaining all factual information; the “long” prompt permits the LLM to expand on the articles, without contradicting existing facts. We experimented with multiple temperature and top-p settings to mitigate hallucinations while encouraging creative outputs (see e.g., https://imgur.com/a/wFvjAFP).\n\nAdditionally, we quantify the effect of using these rephrased articles by reporting F1 scores of Llama-3.3-70B-Instruct with ZeroShot and CoT prompting: see https://imgur.com/a/DQuvQIA.\nRemarkably, we find **similar trends in performance regardless of whether we use LLM-generated or templated articles**. Thus, **templated articles** provide **four key benefits**: (1) they are cheap—no GPU or API costs, (2) fast—no latency from querying LLMs, (3) 100% factually consistent and (4) they allow for larger universe sizes in limited context windows. We leave as future work how to ensure that question-answer pairs are consistent with rephrased articles without relying on human intervention.\n\n**2. Use case of PhantomWiki and evaluating LLMs fine-tuned on PhantomWiki data**\n\n> If the dataset is focused on context free grammar, is it easy to learn after fine-tuning on the same distribution because CFG is well defined?\n\n> Is this dataset intended for any training or evaluation only?\n\nThese are great questions! To assess the viability of PhantomWiki for training language models, we **add new fine-tuning experiments**: see https://imgur.com/a/QHftpCM. Specifically, we perform full fine-tuning of Qwen2.5-0.5B-Instruct and parameter-efficient fine-tuning of Qwen2.5-3B-Instruct (via LoRA applied to all linear layers). We also experiment with two popular training algorithms: Group Relative Policy Optimization (GRPO) and supervised fine-tuning (SFT). For Qwen2.5-0.5B, we find that GRPO and SFT both improve F1 compared to prompting-based methods, likely due to improved ability to output the proper answer format. For Qwen2.5-3B, we find that GRPO improves F1 slightly, whereas SFT worsens F1, likely due to overfitting on the training samples. These experiments **show that further advances beyond fine-tuning are needed to truly close the gap on PhantomWiki**. Please see also our rebuttal to reviewer YauR for full experiment details. \n\n**3. Evaluating retrieval**\n\n> the questions and articles are not appropriate for neural retrieval. \n\n> a question like \"What is the job of the father of the uncle of the sister of {personA}?\" requires iterative bridge entity resolution, making neural retrieval ineffective.\n\n> RAG prompting only retrieved 4 most relevant documents, which are likely insufficient for many multi-hop questions.\n\nTo address all three concerns, we **include new multi-hop RAG baselines** (namely, IRCoT and Self-Ask) using BM25 (top-k=5) as the retriever and Llama-3.3-70B-Instruct as the generator model: see https://imgur.com/a/OMjtf3t. We find that these baselines **outperform** the original ZeroShot-RAG and CoT-RAG baselines on **low-difficulty** questions, but **similarly struggle** on questions with **high difficulty**.\n\n**4. Ambiguous questions & Calculation of F1 score**\n\nPhantomWiki questions always have at least one correct answer and often have multiple correct answers, e.g., “Who is the friend of X?” if X has several friends, or “What is the hobby of the sister of X?” if X has multiple sisters. Furthermore, we explicitly instruct the model to return all possible answers (see prompts in App. C). We compare the LLMs’ predicted answer list against the ground-truth list to compute precision, recall, and F1 per question. Final F1 scores (e.g., in Table 2) are averages over all 500 questions in a PhantomWiki instance. **We will include this clarification in Section 4.3.**\n\n**5. Additional references**\n\nWe will include your suggested references on synthetic data generation in our final manuscript.\n\nWe hope that these **new experiments** and clarifications have significantly strengthened our message and empirical results, and if so we would like to politely ask you to consider raising your score. Thank you!", "claims_and_evidence": "This is a dataset paper. I think the main claim could be viewed as \"PhantomWiki is a scalable and data leakage-resistant framework for disentangled evaluation of reasoning, retrieval, and tool-use abilities.\" I think this claim is supported because the dataset is generated from a fictional and controlled environment.", "methods_and_evaluation_criteria": "### Strength\nOverall, I liked the considerate design of the PhantomWiki data generation pipeline.\n1. It generates a knowledge graph of a universe and uses logical programming to provide answers to multihop questions.\n2. The dataset is fictional and free of data leakage.\n\n### Weakness\nArticle content and questions lack realism. The articles are generated using templates, are very short and lack realism for real-world retrieval augmented generation applications. The multi-hop questions are highly contrived; they are unlikely to appears in real-world situations.", "theoretical_claims": "The paper does not make theoretical claims.", "experimental_designs_or_analyses": "The experiments are technically sound. My main concern is on bias in retrieval evaluation:\n\nThe dataset is not well-suited for evaluating retrieval. Although the paper claims to evaluate retrieval, the questions and articles are not appropriate for neural retrieval. For instance, a question like \"What is the job of the father of the uncle of the sister of {personA}?\" requires iterative bridge entity resolution, making neural retrieval ineffective. In addition, in the retrieval experiment, RAG prompting only retrieved 4 most relevant documents, which are likely insufficient for many multi-hop questions.", "relation_to_broader_scientific_literature": "Prior work on RAG evaluation often confounds with parametric knowledge. This work create fictional world that decouples the impact of data leakage.\n\nPrior work on reasoning with synthetic data does not consider the retrieval setup where the content may not fit in context window as task complexity grows.", "essential_references_not_discussed": "This is another related work that evaluates reasoning using synthetic data.\n\nLevy, M., Jacoby, A., & Goldberg, Y. (2024, August). Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 15339-15353).", "other_strengths_and_weaknesses": "No. Please refer to my comments in the above sections.", "other_comments_or_suggestions": "It would be helpful to define how F1 is calculated in the appendix.", "questions_for_authors": "1. Is there any mechanism to resolve conflicts or ambiguous questions? Does all templates lead to answerable questions?\n2. If the dataset is focused on context free grammar, is it easy to learn after fine-tuning on the same distribution because CFG is well defined?\n3. Is this dataset intended for any training or evaluation only?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a synthetic benchmark for evaluating multi-hop RAG. Reviewers appreciated the scale of the dataset and the control over the difficulty of examples generated. During the rebuttal, addition of new baselines and a paraphrased version was also useful to understand the utility of the benchmark. However, the fact remains that the dataset constructed may not be realistic and contain template-like relationships. As Rev. 8LpG suggests, it will be good to highlight in the intro the kind of tasks for which the benchmark is well-suited for. 
On balance, the paper presents a useful addition to evaluating RAG systems.
Decision: Accept (poster)