ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in Game RAG Benchmarks
Authors: Liyang He
Yuren Zhang
Ziwei Zhu
lizhenghui
Shiwei Tong
Venue: ICLR 2026 Poster (2026)
Keywords: Retrieval Augmented Generation
Dynamic Benchmarking
Dual Dynamics
User Interest Drift
Gaming
Player-Centric Evaluation
Primary area: datasets and benchmarks

Abstract: Retrieval Augmented Generation (RAG) systems are increasingly vital in dynamic domains like online gaming, yet the lack of a dedicated benchmark has impeded standardized evaluation in this area. The core difficulty lies in Dual Dynamics: the constant interplay between game content updates and the shifting focus of the player community. Furthermore, the necessity of automating such a benchmark introduces a critical requirement for player-centric authenticity to ensure generated questions are realistic. To address this integrated challenge, we introduce ChronoPlay, a novel framework for the automated and continuous generation of game RAG benchmarks. ChronoPlay utilizes a dual-dynamic update mechanism to track both forms of change, and a dual-source synthesis engine that draws from official sources and player community to ensure both factual correctness and authentic query patterns. We instantiate our framework on three distinct games to create the first dynamic RAG benchmark for the gaming domain, offering new insights into model performance under these complex and realistic conditions. Our code is available at: https://github.com/hly1998/ChronoPlay.

Published as a conference paper at ICLR 2026
CHRONOPLAY: A FRAMEWORK FOR MODELING DUAL DYNAMICS AND AUTHENTICITY IN GAME RAG BENCHMARKS
Liyang He* Yuren Zhang"), Ziwei Zhu', Zhenghui Li", Shiwei Tong
'Tencent, The Chinese University of Hong Kong, $Independent Researcher [liyanghe, yurenzzhang @tencent com, 22104902801ine cuhk. edw.edda.... cn,
4173418798qq.com com, shiveitongétencett: com
ABSTRACT
Retrieval Augmented Generation (RAG) systems are increasingly vital in dynamic domains like online gaming, yet the lack of a dedicated benchmark has impeded standardized evaluation in this area. The core difficulty lies in Dual Dynamics: the constant interplay between game content updates and the shifting focus of the player community. Furthermore, the necessity of automating such benchmark introduces critical requirement for player-centric authenticity ensure generated questions are realistic. To address this integrated challenge, we introduce ChronoPlay, a novel framework for the automated and continuous generation of game RAG benchmarks. ChronoPlay utilizes a dual-dynamic update mechanism to track both forms of change, and dual-source synthesis engine that draws from official sources and player community to ensure both factual correctness and authentic query patterns. We instantiate our framework on three distinct games create the first dynamic RAG benchmark for the gaming domain, offering new insights into model performance under these complex and realistic conditions. Our code is available at: lttps::/githbb.comt/ 1ttm///lyy99//Chroooolaa
INTRODUCTION
The advancement of Retrieval Augmented Generation (RAG) has been largely driven by its bench-
marks (Yu et al.] 2024; Gan et al. (025). From foundational benchmarks (Yang al.,
2018
Kwiatkowski et al. 2019) and specialized domain evaluations (Li et al. 2025 Wang et al. 2024
Zhong et al. 2025| Jon et al. 2025) to sophisticated diagnostic frameworks (Ru et al. 2024). these
resources provide the essential, standardized means measure progress. Recent efforts have further pushed the frontier from static snapshots to dynamic benchmarks that evolve over time (Ouyang
al. 2025 Ko et al., 2024), reflecting the currency of information in the real world. The global gaming industry, a vast and highly dynamic digital frontier, represents a critical domain for such advancements (Goh al. 2023). Within this ecosystem, RAG is emerging enhance player experiences, from intelligent assistants to automated support bots (Liu et al. 2025 Feng et al. 2025). However, a significant gap emerges: there are currently no RAG benchmarks for the gaming domain, leaving the application of RAG systems in this area without standardized evaluation.
key technology to
This absence stems from the unique nature of the gaming ecosystem, which is composed of two constantly evolving entities: the game itself and its player community. This structure gives rise to core challenge we term Dual Dynamics, as illustrated in Figure [ On one hand, Knowledge Evolution occurs as the game's s content and rules are in a constant state of flux due to frequent patches and version updates Wang et al. (2023). This is different from some stable domains, where comprehensive static benchmark is enough g to evaluate a general- purpose model. In the world of live- service games, such static evaluation would become obsolete. On the other hand, a benchmark must also track User Interest Drift, which represents the systematic evolution of the player community's
Equual contribution. Corresponding author.

---

Published as a conference paper at ICLR 2026
Knowledge Evolution
Game Version
Game Patch
Game Patch
Game Version 2.0
Weapons Update
Balance
New DLC Available
User Interest Drift
Performance Issue: is computer powerful enough run this game? Initial gameplay: What the best starting character for a beginnar?
rad--amm game achievements?
Access: Do need to finish the main story to play the DIC7
High-level strategy: How do obtain the best in siot gear?
New Content: What new weapons and armer are being added?
Player onboarding
Late game
Game major version update
Figure I: An illustration of Dual Dynamics in Game, including Knowledge Evolution, which traces the game's knowledge updates, and User Interest Drift, which maps the changing interests of players.
focus, from initial onboarding questions late game content Kummer et al. (017): Yilmaz (2025)
Capturing this shifting focus is crucial, as a benchmark that fails to do so would lead to models being optimized on distribution of problems that no longer reflects what the community cares about, resulting in biased and irrelevant evaluation.
Given the sheer velocity of these dual dynamics, manually curating a benchmark that remains consistently up-to-date is practically impossible. This challenge is compounded by the vast diversity across different games. Consequently, automated synthesis has emerged as the only viable path forward for creating dynamic benchmarks in this domain. However, existing synthesis approaches Kasai al. (203)); Ko et (2024) Kim
al. (2025): (Shabtay et al. (2025) have focused primarily on single dimension of change: the evolution of knowledge. By concentrating only on tracking knowledge updates, they have overlooked the critical need for authenticity generation process. In user-centered domain, benchmark is fundamentally invalid if it is filled with unrealistic questions that no real player would ask, regardless of their grammatical correctness.
their
To tackle the challenges mentioned above, introduce ChronoPlay novel framework for modeling both dual dynamics and authenticity in the generation game RAG benchmarks. Its core features include: (1) a dual-source synthesis engine that ensures factual correctness by drawing from official sources, while capturing authentic player question patterns and interest preferences by mining player community information; and (2) a dual-dynamic update mechanism that precisely refreshes question- answer pairs by identifying entities in game updates and dynamically adjusts the distribution of evaluation questions response to shifts in community interest.
Using this framework, we instantiate three distinct games. Our subsequent analysis provides an initial baseline for classical RAG systems, and the results demonstrate that our benchmark effectively captures the unique challenges of these dually dynamic conditions, highlighting aspects of model performance that existing evaluation methods cannot measure. Notably, while we focus on the gaming domain, methodology is applicable to other domains characterized by evolving knowledge base and an active user community, such as e-commerce and social media platforms. Our contributions follows:
1. We identify dual dynamics as a fundamental challenge for RAG systems operating in dynamic domains such gaming, and argue that addressing requires an automated synthesis process with player-centric authenticity as critical requirement.
2. We propose ChronoPlay, the first framework designed to automatically generate dynamic benchmark that integrally addresses the challenges of temporal relevance and player-centric authenticity.
3. We instantiate framework three distinct games create the first dynamic RAG benchmarks for the gaming domain. Our analysis offers new insights into the performance of existing RAG systems under these complex, realistic conditions.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
The development of RAG systems benefited significantly from question-answerrigg benchmarks like Natural Questions (NQ) (K Kwittkowski et al. 2019), HotpotQA (Yang et al., 2018), PopQA (Mallen al. 2023), and CRAG Yang et al. (024). They provided standardized environment for evaluating a model's 's retrieval and reasoning capabilities within a closed, static knowledge world. However, their static nature makes them incapable of assessing system's adaptability to the continuously evolving knowledge of the real world.
To address this limitation, some works have focused creating dynamic benchmarks (Shirali et al., 2022). These efforts can be broadly categorized by their update trigger: being either period-driven or fact-driven. The period-driven approach updates at fixed intervals, often leveraging retrospective
snapshots of sources like Wikipedia. For instance, HOH (Ouyang et al. 22025) GrowOVER (Ko et al.) 2024), EvolvingQA (Kim al., 2024), and DynaQuest (Lin al. (025a) use monthly snapshots to generate question-answe -nnwer pairs that test model's ability to handle outdated information. Besides, DRAGON (Chernogorski et al., 2025) is built upon a periodically updated news corpus. The fact- driven approach aims for lower latency by triggering updates upon the emergence of new information. While not all are designed strictly for RAG, methods like REALTIMEQA (Kasai et al.] 2003), which uses live news feeds, and LIVEXIV (Shabtay et al. 2025). which scrapes new preprints, exemplify this paradigm.
However, this exclusive focus on knowledge evo- lution overlooks the user interest drift. The dy-
100
namic evolution of user interests is significant phe- nomenon in many fields like social computing and recommendation systems (Singer et al.] [014 Yin ct al. 2016; Sun & Dong 2017: Ding et al.) [2022]
Ganapky Mechaner Pertemrano Lores
GollEinns
Lin et al. 2025b). This challenge is also mirrored in granular tasks such as aspect-leved sentiment analysis (Zhang al., 202I). This is particularly pronounced in gaming, trend confirmed by our preliminary anal- ysis. As shown Figure 2 discussion hotspots the Dying Light 2 player community clearly drifted from carly topics like System Requirements to later con- cerns about Gameplay Mechanics. By focusing exclu- sively on knowledge updates, all current temporally- aware benchmarks ignore the evolution of user interest, which is critical for building applications that serve genuine user needs.
20
Time Perial
Figure 2: Distribution of main topics over time in Dying Light 2.
THE CHRONOPLAY FRAMEWORK
The ChronoPlay framework designed to transform the challenge of dual dynamics and authenticity in the gaming domain into an executable and automated solution. Its core objective is to continuously generate a dynamically evolving benchmark, denoted as B = {Bi, B2,..., B,,..., slice B, at specific time point t is pair (Ke,, Dz), which consists of a corpus knowledge base IC, for retrieval and a corresponding evaluation dataset D:. This dataset is composed of a series of evaluation tuples, with each tuple defined as d (Q, A, Cref, 8, 0,T, O)). These components represent the question Q, answer A, reference knowledge snippets Cref E K,,, question topic 0, timestamp т, and associated in-game entities a, respectively.
Each benchmark
To achieve this, we have designed two closely integrated core components: a data synthesis pipeline that builds upon dual-source data assets, and dual-dynamic update mechanism responsible for the continuous evolution of the benchmark. We describe the overall framework and pipeline in the main text, while the technical details are provided in Appendix D
3.1 DUAL- DATA SYNTHESIS FRAMEWORK
To ensure the benchmark is both factually accurate and representative of authentic player questions, have designed dual-source data synthesis framework. Instead of relying on noisy questions
3

---

Published as a conference paper at ICLR 2026
Question Type 9:
betache SpeaiooAAanstr
Question: quide
Agent
for my (a% buld la the
at (LOCATINNT
Hepethettial
Co Wewww (octted
Rest moll
Flbrring
Cantrol
Pass
2cis Abpunal detioling
Annowe: or the
(p.
(Mlaypon ALyas)
Authority Knowledge
Knowledge snippets Cray
Rallensse Moosladge TD Dur Darguor Sartan pempll
Sample
Kinnth
wearcn is.g Jeen searteod
teeiing
ACE losseel pohine that
Besties (Baignn ldge swird
Mayiaa
Question Template
Base Tormn
User Persona
Base Mursu
Aes-ia ges
Persona
Topic: GAME MYCHANIES
Tamestemy:: 232:
Topic: CAME, MEC-AARIS
Figure 3: An illustration of the Dual-Souree Synthesis Pipeline. It leverages the Authority Knowledge Base Kouth for factual grounding, and uses the Question Template Base Toomm and the User Persona Ucomm for authentic question patterns.
from online communities, our framework employs a multi-stage pipeline that integrates authoritative knowledge with player community information.
3.1.1 AUTHORITY KNOWLEDGE BASE (Nauthh)
To guarantee factual accuracy, we construct an Authority Knowledge Base Kuuth- Each knowledge snippet K; E Kauth is formalized as a tuple (ke: kr kz), representing the content, timestamp, and associated entities. We systematically aggregate game wikis and official patch notes, processing raw HTML pages and tabular data into uniform, retrievable knowledge snippets k; using DOM tree analysis and LLM for formatting, For official data, we precisely extract the publication time as the timestamp ky. Based on the Self-ICL approach Chen et al. (2023), we use NER function E-) on all knowledge snippets to extract in-game entities lig (k.). These entities are crucial for accurately identifying and updating affected knowledge.
3.1.2 QUESTION TEMPLATE BASE (Teomm.) & USER PERSONA BASE (Mromm
To capture the authenticity of user queries, we build two community bases by collecting real questions from player communities. We invite domain experts to develop a hierarchical topic taxonomy θ, based on a large sample of player questions. This taxonomy comprises main categories and 21 sub-categories, covering aspects from technical issues to game content and purchase consultation. Appendix details the taxonomy. Then, we prompt an LLM to mine these question posts, decoupling and extracting two reusable elements: question templates p with their corresponding topics 0 O, and user personas u. This decoupling allows these authentic, game-agnostic query patterns to be reused across different games, which greatly enhances the framework's 's scalability. The resulting template-topic pairs (p, 0) form the Question Template Base Teomm. and the user personas u constitute the User Persona Base Mcomm- We also employ vector-based filtering mechanism deduplicate semantically similar items in both bases, followed by final review by experts.
3.2 MULTI-STAGE SYNTHESIS PIPELINE
Based on the constructed bases, our synthesis pipeline organically combines these two data sources through a sophisticated mtati-stage process as shown in Figure We first vectorize the snippets from the knowledge base Kauth and build an index. To effectively bridge user intent with factual evidence, we draw inspiration from HyDE (Gao al. 2023) and use sampled template-topic pair (p. 0) from Teomm prompt LLM generate hypothetical question-ansee answer pair (Qkypor Ahypo). Although the content of this pair is fictional, its embedding provides a more precise vector to locate relevant knowledge snippets Cref {ki, l.,.... k, within the vector space of Kauth-
Since players often ask context- dependtt questions (e.g., "As a new player, which weapon is easiest to get?"), we incorporate user persona E Ucamm into the generation process. We embed the user persona base Neoum and build a vector index. The hypothetical pair (Ohypo: Apype) is then used to query this index to find the most suitable persona. Only personas whose similarity score surpasses
4

---

Published as a conference paper at ICLR 2026
Cupdate
Stale subset D Patate
Entiles (Dragon Edge sword]
Quemtion: hase a guide seying the Doegon Whet's
The Swerd is no longer direct boss crop and is new ockained by Abyscal that the linal
(()
NER-identry
Edge my
Amswen the the Surker Temple, alter dellesting the bus Airooal Kright.
Monitor
NER
instwed.
Topic
D
Official Website
Auea
Update
Dual Source Data Synthesis Pipline
A
B
Datect
Benchmark
Pr
Pc
Aptn
D
GAME MECHANICS 124%, -a 21.3%
VERSION COMPANSOON 7.2%--J00
Monitor
Topic-ldentify
RERSIONLUPDATES 6.2% +> 7.1%
Player Community
Current Distribution
Topic Distribution Charige
Figure 4:
An illustration of the Dual-Dynamic Update Mechanism, showing both Knowledge Evolution and User Interest Drift pathways.
threshold A are considered candidates, and we select the top-ranked result. no persona meets the threshold, this generation step proceeds without user persona.
The final synthesis stage is orchestrated by specialized data synthesis agent, which designed to autonomously produce high-quality data tuples through an iterative refinement process. For each generation task, the agent begins by sampling question type qi (e.g., extractive, comparative) from predefined set (Wang et al. 2024). It then synthesizes a candidate tuple d = (2, A, Cref, 0 o) by conditioning a generator model on the comprehensive context, which includes the template-topic pair (p, 0), the retrieved factual snippets Cref, the user persona and the question type q.. The in in-game entities a and the timestamp T are derived from the reference context Cref, with 7 being defined as the most recent timestamp available, i.e., max((k,lk E Cref }). If no snippet in Cref contains timestamp, 7 is left undefined.
Another component of this agent is its integrated quality control mechanism. After generating a candidate tuple, the agent immediately assesses against multiple quality dimensions. This assessment is performed by LLM, following the increasingly common LLM-as-Judge paradigm for scalable and automated evaluation (Zheng et al.] 2023 Dubois et af.] (024).. the tuple fails meet a predefined quality threshold, self-correction loop: it re-samples a new question template corresponding to the same topic from the question template base Teomm and re-attempts the synthesis. This iterative process continues until a high-quality tuple is successfully generated and validated.
discarded. Instead of terminating, the agent initiates
3.3 DUAL-DYNAMIC UPDATE MECHANISM
Another core innovation of ChronoPlay is its ability to evolve over time, ensuring the long-term validity of the benchmark. This dynamism is driven by a dual update mechanism that responds to changes in both the game's knowledge and the player's interests as shown Figure
The first mechanism is designed to respond to Knowledge Evolution, guaranteeing the benchmark up-to-date and factually correct with respect to the game content. Game knowledge updates are typically driven by discrete events like official announcements. Our monitoring module detects such events and triggers an update workflow. Let Anew NER function E(-) to identify the affected in-game entities Jxpdate {(Anew)) ). We then identify the stale subset of the dataset Datales where each tuple's entity set intersects with the updated entities:
new announcement time L. We first use the
D Datale (d E D, (d) O Capdate + 0}
The remaining valid tuples are denoted as Dratid = Dj Dstate- The topics from these stale tuples are used to re-initiate the synthesis pipeline described in Section 3.2 This generates a new set of tuples Dnews which reflects the latest information. The dataset is then updated to its next state:
Di+1 = Dratid D new. Besides, the corpus knowledge base is updated by K,-1 = K,, U Anen- Finally, we obtain the new benchmark B441
5

---

Published as a conference paper at ICLR 2026
The second mechanism is designed to respond to User Interest Drift, ensuring the benchmark's s topical relevance. We continuously monitor the topic distribution Pa..M) of questions from the community within sliding time window of size W, where and b represent the start and end timestamps. To quantitatively detect significant shifts, we compare the topic distribution of the current window Pe = P mazz(.... e-- (+), with that of reference period P, P Pp...=-(()) where pr denotes the start time of the previous reference time segment and C, denotes the current time. A collective interest drift is flagged if a wpighteighted Jensen-Shannon Divergence between these distributions exceeds predefined threshold SSSD In this variant, the standard JSD calculation is modified: each topic A assigned a weight wy based on its overall prominence. Assume M 1 (Pc + P.) the mixture distribution in JSD, then the weight wy for each topic is calculated as wg M(0)>/E Zerg M(0)), with y being a hyperparameter. This method focuses the drift detection on significant trends, making it more robust against noise from low-frequency topics.
Upon detecting a drift, the framework initiates a benchmark resampling process to generate B,+1- The goal is to align the benchmark's 's topic distribution with the current topic distribution Pa. This involves down-sampling data from topics with waning interest and synthesizing new data for topics with emerging interest until the overall distribution of B441 reflects that of the active player community.
4 EXPERIMENTS
We conduct experiments to answer four core research questions: RQ1: How do RAG systems perform on our dynamic benchmark, and how does their performance fluctuate across a game's 's lifecycle? RQ2: How do knowledge evolution and user interest drift individually and collectively impact these performance fluctuations? RQ3: How do our key synthesis modules contribute to the authenticity and quality of the generated benchmark data? RQ4: How efficient is our dual-dynamic update mechanism, and what are the respective roles of knowledge and interest as drivers of change?
4.1 EXPERIMENTAL SETUP
4.1.1 DATASET INSTANTIATION
We instantiate our benchmark on three distinct games: Dying Light 2, Dune: Awakening, and PUBG Mobile, which will be abbreviated as DL2, Dune, and PUBG in the following content. These games represent different characteristics. For each game, we collect data from official sources and major player communities. Table summarizes the key statistics for each dataset. For our analysis, partition each game's 's timeline into discrete phases { {B1,... ..., B,, based on significant shifts in user interest. Shifts in user interest directly measure what players deem relevant and often represent the consolidated impact of multiple underlying knowledge updates, thus forming coherent periods for analysis. For more details on the game data, hyperparameter settings, and the specific partitioning of phases, please refer to Appendix A
Table I: Key statistics of the three instantiated game benchmarks.
Game
Time Range
# Comm. Posts
Wiki Tokens
Update Tokens
# Synth.
# Phases
Qs
Dying Light 2 Dune: Awakening
Jan 22 Jul 25
10,478
369,120
297,371
2,000
5
Jun Aug
37,079
18, 199
53,833
3,000
6
PUBG Mobile
24 Jul 25
60.632
86,652
56,703
1,400
7
4.1.2 BASELINES AND METRICS
We construct RAG systems using four retrievers, including BM25 (Robertson et al.
1995 BGE-M3
(Chen et al. 2024). Qwwe3--mbedding Seween--mbedding-.... (Zhang et al. 2025) and text- embedding-3 (eeet-emmbeddng----aallll
souuce and open- source large language models: GPT-4o (GPT-40-2024-11 1-20) (Hurst et al. 2024). Qwen3 (Qwen3-
We used six generators, including
14B) Yang et al. 2025). Ilama-4 (Ilama-4-scon 1 17b) Meta (2025), gemini- 2.5 flash Comanici et al.
(2025) Claude-3.5. Sonnett (claude-3.3 3-suude-----oomeee (0onneet-02406200 Anthroppi (0044) and DeepSeek- PrrpSeek-V3LL.. Liu ct al. 2024). For retrieval, we evaluate using Recall @ K, Fl K, and NDCG K. For generation, we an LL.M-as-Judge assess correctness and faithfulness, with details provided in Appendix C

---

Published as a conference paper at ICLR 2026
4.2 RQI: LIFECYCLE PERFORMANCE EVALUATION
We first evaluate how RAG systems perform each sequential benchmark slice for each game. This phase-by-phase analysis designed to reveal how the performance different RAG systems fluctuates as the game evolves. We present the retrieval and generation results separately.
4.2.1 RETRIEVAL PERFORMANCE
Table[]shooe shows the results when the number of retrieved documents K is set to 3. Appendix shows the results when K is set to and 5. From these results, no single retriever is the best for all situations. For example, text-embedding-3 is the top-performing retriever in most phases, while Qwen3-Fmmeeddin PUBG, Second, the performance of all retrievers changes significantly across different phases, which confirms our main motivation. For example, DL2, all models show a clear performance drop in Phase 4. The topic distribution analysis reveals that questions about GAMEPLAY MECHANICS surged from 17.66 17.64% in Phase making it harder for retrievers to find the precise documents needed. Third, we found that BGE-M3's performance on Dune was very low compared to the other models. This may be because Dune was recently released in June 2025, and it includes more proper nouns (e.g.. terrarium of muad' dib), which severely tests the generalization ability the models.
draw several key conclusions. First,
achieves higher
in several phases of
31.25% in Phase 4. These questions are often more complex,
Table 2: Retrieval performance three games across their phases. We report Recall @3 (R@3), F1 @3, and NDCG@3 (N@3). The best performing result in each row in bold.
BM25
Been3-Embedding
BGE-M3
text-embedding-3
Phase
R@3
F1@3
N@3
R@3
F1@3
N@3
R@3
F1@3
N@3
R@3
F1@3
N@3
D1.2
0.389
0.363
0.498
0.342
0.320
0.454
0.378
0.355
0.507
0.521
0.495
0.698
0.387
0.356
0.509
0.291
0.262
0.372
0.339
0.311
0.437
0.536
0,505
0.683
0.403
0.371
0.526
0.338
0.308
0.449
0.367
0.338
0.497
0.551
0.519
0.716
0.323
0.305
0.428
0.273
0.256
0.398
0.309
0.290
0.429
0.419
0.396
0.604
0.300
0.283
0.388
0.262
0.245
0.367
0.278
0.260
0.385
0.422
0,401
0.593
Dune
0.282
0.268
0.386
0.325
0.612
0.067
0.065
0.056
0.368
0.354
0.645
0,271
0.256
0.370
0.319
0.302
0.570
0.028
0.028
0.027
0.348
0,331
0.635
0.314
0.296
0.446
0.341
0.321
0.678
0.038
0.038
0.036
0.381
0.360
0.738
0.270
0.255
0.362
0.314
0.298
0.563
0.040
0.040
0.038
0.346
0.329
0.612
0.268
0.249
0.355
0.320
0.300
0,580
0.039
0.038
0.038
0.343
0.325
0.622
0.265
0.252
0.355
0.313
0.300
0.549
0.050
0,049
0.046
0.346
0.334
0.604
PUBG
0.415
0.503
0.515
0.515
0.580
0.482
0.482
0.570
0.528
0,528
0.590
0.262
0.262
0.307
0.372
0.372
0.402
0.342
0.342
0.379
0.365
0.365
0.395
0.533
0.533
0.576
0.578
0.578
0.623
0.557
0.557
0.606
0.577
0.577
0.615
0.237
0.237
0.280
0.332
0.332
0.368
0.308
0.308
0,352
0.338
0,338
0.379
0.338
0.338
0.421
0.392
0.392
0.460
0.380
0.380
0.450
0.377
0.377
0.432
0.352
0.352
0.426
0,480
0.480
0.536
0.440
0,440
0.505
0.498
0.498
0.549
0.322
0.322
0.385
0.458
0.458
0,517
0.427
0.427
0.493
0.442
0.442
0.500
4.2.2 GENERATION PERFORMANCE
We use the top retrieved documents from the top perforffonnng retriever, text-embedding-33 as the context for the six generator models. Figure shows the most critical metrie: correctness scores, while the faithfulness scores are discussed in Appendix F
From the results, we can draw several conclusions. First, a key observation is that generator performance not static, similar to the retrieval results. In the game PUBG, the fluctuations each phase are particularly significant. This is because, interest more prone to shifting. Second, as a general trend, the generator performance is consistent with the retrieval results. However, sometimes the correctness score drop with good retrieval. clear example is in PUBG when comparing Phase and Phase 3. Although the retriever performed slightly better in Phase 3 (Table 2 the correctness scores for all generators significantly lower.
highly interactive live-service game, player
7

---

Published as a conference paper at ICLR 2026
DeepSeek Chat
Gemini 25 Elah
GPT 4o
LLAMM 17B
Qwen3 14B
Claude 3.5 Sonnet
Correcince Score
0.4
0.4
0.3
0.3
0.1
0.2
Plase
Pluss
Phase
(a) DL2
(b) Dune
(c) PUBG
Figure 5: Generator correctness scores across the phases each game.
This implies that the questions in Phase 3, while answerable with the provided documents, require more complex reasoning, which challenges the generators.
Overall, these retrieval and generation results confirm that dynamic, phase-based evaluation is essential for the gaming domain. A static benchmark would average out these critical performance differences, failing to identify the specific types of knowledge updates, user interests, and question complexities that challenge modern RAG systems.
4.3 RQ2: DECONSTRUCTING THE IMPACT OF DUAL DYNAMICS
To understand the cause of the fluctuations observed in RQL, isolate the individual effects of knowledge and interest updates using the relatively more volatile PUBG benchmark. We test RAG system (using text-embedding-3 and GPT-4o) on our Dual-Dynamic benchmark against two variants: one that only tracks knowledge updates (Knowledge-Only benchmark) and one that only tracks user interest (Interest-Onl) benchmark).
Figure 6 shows the results. The line chart illustrates the results for the three benchmarks, and the violin plot represents the statistical volatility of these benchmarks. The key finding that both knowledge updates and user interest drift are major factors af- fecting volatility. However, compared our full Dual-Dynamic Benchmark. only considering one type of update leads to some evaluation bias. For ex- bench-
Dual Dynanic
Interest Only
- Knowledge Only
0.45
Careatzes Score Disrrbttin
0.425
sachi
Coneetacss Store
State
0.:0
0:3
0.30
ample, the Knowledge-One mark, which ignores user interest, hides the performance changes in Phase and Phase 7, as these two phases had no knowledge updates. The Interest-Only benchmark, on the other hand, ignores changes caused by game knowledge updates. Therefore, benchmark must track both types of changes.
(a)) Performance over Time
Phase
(b) Volatility Distribution
Figure 6: Comparison of RAG performance three bench- mark variants for PUBG Mobile.
4.4 RQ3: ABLATION STUDY ON SYNTHESIS MODULES
Having demonstrated the importance our benchmark, we now validate its construction with an ablation study. We compare our Full Pipeline against three degraded versions: (1) w/o Hypothetical Q&A, (2) w/o User Persona, and (3) w/o Question Template. Since these synthesis modules specifically designed to make our benchmark player-centrie, our primary evaluation criterion authenticity, which means how much question sounds like was written by a real player. The evaluation uses competitive method where both L.LMMas-judges (we GPT-4o, Gemini 2.5-Pro,
8

---

Published as a conference paper at ICLR 2026
Full Pipline
w/o Hypothoss QKA
wio User Ponena
wño Question Template
19.9
33.8
35.1
27.5
19.7
22.0
22.0
26.4
21.3
24.8
Dune
DL2
PUBG
LLM Average
Iuman Evaluation
Figure 7: Ablation study results for the authenticity criterion. The pie charts show the win rates of our four synthesis settings across the three games, the average score, and the human expert evaluation.
100%
1005
100%
0.90
Process Impoct Ratio
3485
4.%
40%
4079
112
2053
206
206
088
08
082
Plase
Phuse
Phase
(a) DL2
(b) Dune
(c) PUBG
Figure 8: Analysis of the dual-dynamic update process. The composition of each new benchmark phase is broken down by its origin: questions that were inherited, updated due to knowledge changes, or updated due to interest shifts.
and DeepSeek-RI 1 as evaluation model) and human experts select the single best question from the four settings. In addition, we conducted secondary analysis on question clarity. Appendix H details the experimental setup and the clarity results. Figure shows the results for our primary authenticity evaluation. The results are clear and consistent: both the LLM judges and human experts show strong preference for the Full Pipeline. The w/o Question Template setting performs the worst by large margin. This result is expected, as the community-minge realistic phrasing and intent. Without them, the generated questions become generic and lose their authentic feel. This finding validates that our full synthesis pipeline is crucial for creating benchmark that faithfully represents real user questions.
templates are the primary source of
4.5
RQ4: ANALYSIS THE
UPDATE PROCESS
Finally, analyze the efficiency and properties of our update mechanism. We categorize questions in each new phase as either Inherited, updated due to Knowledge, or updated due Interest. Figure visualizes this breakdown. The results show two key findings. First, the process is highly efficient. Across most phases, large portion the benchmark is inherited, meaning full data reconstruction and evaluation unnecessary. Furthermore, the primary driver of change varies dramatically. For example, the update to Phase 3 of PUBG was largely knowledge-driven (34.4%), while the update to Phase was entirely interest-driven (48.2%). This confirms that both dynamics independent and crucial forces that our mechanism successfully captures, ensuring the benchmark remains faithful to the true state of the player-centric game environment.
CONCLUSION AND FUTURE WORK
In this paper, address the critical challenge of evaluating RAG systems in dynamic domains like gaming. We identify the core problem as Dual Dynamics: the constant, co-evolving interplay of factual game knowledge and the shifting interests of the player community. To solve this, proposed

---

Published as a conference paper at ICLR 2026
ChronoPlay, the first framework to automatically generate dynamic and authentic benchmarks that model both of these forces. Our experiments across three games demonstrate that RAG system performance is highly volatile over game's ' lifecycle. We prove that this volatiliit is product of both knowl evolution and user interest drift, and benchmarks that ignore either dimension provide misleadingly stable and unrealistic evaluation. An important future frontier is extending ChronoPlay to personalized user histories, evolving the persona base into a dynamic user state module. This paves the way for stateful RAG architectures that leverage persistent profiles and update mechanisms for personalized retrieval and generation. Ultimately, ChronoPlay offers not just dynamic benchmark for gaming but new paradigm for creating more faithful evaluations in any evolving, user-centric environment, serving as foundation for more adaptive and personalized RAG systems.
ETHICS STATEMENT
We confirm that our work adheres the ICLR Code of Ethics. All data used in this research, including game information and community posts, were collected from publicly accessible online sources such as Fandom wikis, official game websites, and public forums. No private user data was accessed or used in this study. The human annotation and evaluation tasks described in this paper were performed by the authors of this work, who consented to their voluntary participation. The User Persona Base generated by our framework is synthetic description of player types, inferred from public text. These personas were manually reviewed by the authors to ensure they do not contain or infer any personally identifiable information sensitive user data.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our work. The source code for our ChronoPlay framework and the complete, instantiated benchmarks for all three games are included in the sup- plementary materials and will be publicly released upon publication. Detailed descriptions of our data processing pipeline, synthesis prompts, and experimental setup are provided in the main paper and appendices to facilitate replication. We acknowledge that due to the stochastic nature of the Large Language Models used for data synthesis and the random sampling of templates, executing our generation pipeline will not produce bit-for-bi identical benchmark instances on each run. However, we have taken measures to ensure the statistical and conceptual reproducibility of our results. We have conducted multiple generation runs and, as shown in our human evaluation of the synthesized data (Appendix D.5). the resulting benchmarks consistently meet a high-quality standard.
REFERENCES
Anthropic. Introducing Claude 3.5 Sonnet, June 2024. URL sonnet]
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-fuuncionnality. multi granularrnnllrrity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216, 2024.
Wei-Lin Chen, Cheng- Kuang Wu, Yun-Nung Chen, and Hsin-Hsi Chen. Self-icl: Zero-shot in-context learning with self-generated demonstrations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15651-15662, 2023.
Fedor Chernogorskii, Sergei Averkiev, Liliya Kudraleeva, Zaven Martirosian, Maria Tikhonova, Valentin Malykh, and Alena Fenogenova. Dragon: Dynamic rag benchmark on news. arXiv preprint a50iv:2507. 00.00,171 2025.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistcin, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.0626/ 2025.
Hao Ding, Qing Liu, and Guangwei Hu. Tdtmf: A recommendation model based on user temporal interest drift and latent review topic evolution with regularization factor. Information Processing & Management, 59(5): 103037, 2022.
10

---

Published as a conference paper at ICLR 2026
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Lengthh alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024.
Yuan Feng, Yanshuangfei Miao, and Yijie Zhou. Research the humanization design of game npes and user experience optimization based on large language models. In 2025 3rd International Conference Image, Algorithms, and Artificial Intelligence (ICIAAI 2025), pp. 958-970 Atlantis Press, 2025.
John M Ford. Content analysis: An introduction to its methodology. Personnel psychology, 57(4): 1110, 2004.
Aoran Gan, Hao Yu, Kai Zhang, Qi Liu, Wenyu Yan, Zhenya Huang, Shiwei Tong, and Guoping Hu. Retrieval augmented generation evaluation the era of large language models: A comprehensive survey. arXiv preprint 2rXiv:2504.
2025.
Luyu Gao, Xueguang Ma. Jimmy Lin, and Jamie Callan. Precise zero-shot dense retrieval without relevance labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 1762-1777, 2023.
Edward Goh, Omar Al-Tabbaa, and Zaheer Khan. Unravelling the complexity of the video game industry: An integrative framework and future research directions. Telematics and informatics reports, 12:100100 2023.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
Youngseung Jeon, Ziwen Li, Thomas Li, JiaSyuan Chang, Morteza Ziyadi, and Xiang' Anthony' Chen. Ragppi: Rag benchmark for protein-protenn interactions in drug discovery. arXiv preprint arXiv:2505.23883: 2025.
Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, Kentaro Inui, et al. Realtime qa: What's the answer right now? Advances in neural information processing systems, 36:49025-49043, 2023.
Yujin Kim, Jaehong Yoon, Seonghyeon Ye, Sangmin Bae, Namgyu Ho, Sung Ju Hwang, and Se- Young Yun. Carpe diem: On the evaluation of world knowledge in lifelong language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 5401- 5415, 2024.
Dayoon Ko, Jinyoung Kim, Hahyeon Choi, and Gunhee Kim. Growover: How can llms adapt to growing real-world knowledge? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3282-3308, 2024.
Luiz Bernardo Martins Kummer, Júlio César Nievola, and Emerson Cabrera Paraiso. Digital game usage lifecycle: systematic literature review. Anais do XVI Simpósio Brasileiro de Jogos Entretenimento Digital (SBGames' 17), pp. 1 1163-1172, 2017.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Haitao Li, Yifan Chen, Hu YiRan, Qingyao Ai, Junjie Chen, Xiaoyu Yang, Jianhui Yang, Yueyue Wu, Zeyang Liu, and Yiqun Liu. Lexrag: Benchmarking retrieval- augmenel- generation in multi-turn legal consultation conversation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 3606-3615, 2025.
Qian Lin, Junyi Li, and Hwee Tou Ng. Dynaquest: A dynamic question answering dataset reflecting worl-world knowledge updates. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 26918-20 8-69966, 2025a.
11

---

Published as a conference paper at ICLR 2026
Xiaolin Lin, Weike Pan, and Zhong Ming. Towards interest drift-driven user representation learning in sequential recommendation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1541-1551, 2025b.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Xiao Liu, Zhemping Xie, and Senlin Jiang. Personalized non-player characters: A framework for character-consistent dialogue generation. AI, 6(5):93, 2025.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannanch Hajishirzi.
When not to trust language models: Investigating effectiveness of parametric and non-parametris memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 9802-9822, 2023.
Meta. Llama 4: The next generation of open multimodal intelligence, April 2025. URI. https 7/ai. www.com//bbaaaa http://oogglaaaaaa - -ndaal--tteeeeegeeeeee
Jie Ouyang, Tingyue Pan, Mingyue Cheng, Ruiran Yan, Yucong Luo, Jiaying Lin, and Qi Liu. Hoh: A dynamic benchmark for evaluating the impact of outdated information on retrieval- -uugeeeld generation. arXiv preprint arXiv:2503.00 044800 2025.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulien Mike Gatford, et al. Okapi at TREC-3. British Library Research and Development Department, 1995.
Dongyu Ru, Lin Qiu, Xiangkun Hu, Tianhang Zhang, Peng Shi, Shuaichen Chang, Cheng Jiayang, Cunxiang Wang, Shichao Sun, Huanyu Li, al. Ragchecker: A fine-grained framework for diagmosing retrieval-augmented generation. Advances in Neural Information Processing Systems, 17:21999-22022 2024.
Nimrod Shabtay, Felipe Maia Polo, Sivan Doveh, Wei Lin, Jehanzeb Mirza, Leshem Choshen, Mikhail Yurochkin, Yuekai Sun, Assaf Arbelle, Leonid Karlinsky, et al. Livexiv-a multi-modal live bench- mark based on arxiv papers content. In International Conference on Learning Representations, 2025.
Ali Shirali, Rediet Abebe, and Moritz Hardt. A theory of dynamic benchmarks. arXiv preprint arXiv:2210.03165, 2022.
Philipp Singer, Fabian Flöck, Clemens Meinhart, Elias Zeitfogel. and Markus Strohmaier. Evolution of reddit: from the front page of the internet to a self-referential community? In Proceedings of the 23rd international conference on world wide web, pp. 517-522, 2014.
Baoshan Sun and Lingyu Dong. Dynamic model adaptive to user interest drift based on cluster and nearest neighbors. IEEE access, 5:1682-1691, 2017.
Shuting Wang, Jiejun Tan, Zhicheng Dou, and Ji-Rong Wen. Omnieval: An omnidirectional and automatic rag evaluation benchmark in financial domain. arXiv preprint arXiv:2412.13011 2024.
Yimin Wang, Yonglin Dai, Shaokang Chen, Lingxin Wang, and Johan F Hoorn. Multiplayer online battle arena (moba) games: improving negative atmosphere with social robots and ai teammates. Systems, 1(8))425, 2023.
An Yang, Anfeng Ii, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09300 2025.
Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Gui, Ziran Jiang. Ziyu Jiang, et al. Cougecompeeeensiee rag benchmark. Advances in Neural Information Processing Systems, 37: 17:10000-10990 2024.
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio. William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint / 1809009000 2018.
12

---

Published as a conference paper at ICLR 2026
Mustafa Atahan Yilmaz. Player experiences and motivations in the game stages of mmmorgg from the perspective of product life cycle theory. Quality & Quantity, 59(1):599 619, 2025.
Hongzhi Yin, Xiaofang Zhou, Bin Cui, Hao Wang, Kai Zheng, and Quoc Viet Hung Nguyen. Adapting to user interest drift for poi recommendation. IEEE Transactions on Knowledge and Data Engineering, 28( (1)):2556-2281, 2016.
Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu, and Zhaofeng Liu. Evaluation of retrieval- augmented generation: A survey. In CCF Conference on Big Data, pp. 102-120. Springer, 2024.
Kai Zhang, Qi Liu, Hao Qian, Biao Xiang, Qing Cui, Jun Zhou, and Enhong Chen. Eatn: An efficient adaptive transfer network for aepect-level sentiment analysis. IEEE Transactions Knowledge and Data Engineering, 35(1):377-389 2021.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang. Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Erie Xing, et Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595-46623, 2023.
Xianrui Zhong, Bowen Jin, Siru Ouyang, Yanzhen Shen, Qiao Jin, Yin Fang, Zhiyong Lu, and Jiawei Han. Benchmarking retrieval-augmmentd generation for chemistry. arXiv preprint arXiv:2505.07671, 2025.
13

---

Published as a conference paper at ICLR 2026
A DATA COLLECTION AND CONSTRUCTION DETAILS
This section provides additional details on the instantiation of our benchmarks, including the rationale for game selection and the specific hyperparameter settings used for partitioning the game timelines.
A.1 GAME SELECTION AND DESCRIPTION
We chose three games with distinct characteristics to ensure our benchmark covers diverse range of dynamic scenarios. For each game, we construct a authority knowledge base from official game wikis and update announcements. To model user interest, we collect a large corpus of player discussions from major communities
Dying Light 2 was chosen as an example of mature game with long-term support. Its knowledge base grows over a multi-year period through significant patches. Our data collection spans over three and half years for this game (Jan '22 Jul '25), during which we constructed knowledge base from 310 wiki documents and 199 official update articles. Concurrently, we collected 10,478 community posts model long-term user interest drift. For this game, we collected wiki data from its Fandom pago and update information from the official website.
Awakening is our case study for newly launched game. The game is built on the massive pre-existing lore of the Dune universe, creating an enormous knowledge base from day one. Its initial knowledge base is immense, built from 3,377 wiki documents and 43 update articles. To capture the rapid shifts in focus during its volatile launch window (Jun '25 Aug '25), we analyzed 37 37,079 community posts. We sourced its wiki data from the Dune: Awakening Community Wik and gathered update notes from the game's official Steam platform page.
Dune:
PUBG Mobile was selected high-velocity live-service game. Its environment constantly changes due to frequent updates and regular seasonal events. Its high velocity is evidenced by the 60.632 community posts we collected over an 18-month period (Jan '24 Jul '25). The knowledge base for this period is grounded in 142 wiki documents and 27 frequent official updates, providing a basis to evaluate RAG systems in a rapidly changing environment. We sourced wiki data from its Fandom pago and collected update information from the official game website.
A.2 HYPERPARAMETER SETTINGS FOR PHASE PARTITIONING
The partitioning of each game's timeline into distinct phases is governed by our user interest drift detection mechanism, which relies on the following key hyperparameters:
Topic Importance Factor (7): In our topic-weighted JSD calculation, we set y = L.5. This value enhances the weight of more prominent topies in the distribution. This ensures that the drift detection is focused on significant, mainstream shifts in community discussion rather than statistical noise from niche, long-tail topies.
Drift Threshold (ASsD): The threshold for flagging significant drift is uniformly set to ÄJSD = 0.001. This sensitive value was chosen based on conservative principle: for a benchmark designed to study performance volatility, it is more rigorous to risk over- segmenting the timeline with minor fluctuations than to risk overlooking a genuine shift in user interest. This ensures capture the full dynamic nature of the environment.
Sliding Window Size (W): The window size for monitoring the topic distribution of new community questions tailored to the specific dynamics of each game. For DI.2, with its longer update cycle and more gradual shifts in player focus, we use wide window of W months. For Dune, to capture the rapid, day-by-day changes during its critical
Mainly communities including Reddit ht httpp:://///.
com
Discord: fctps://discom
com and Twitch: http:/hht..fannmnccoooo.o httess////imml.h......ooooooo//.
https://awakening
cttes://pubgmmoi1ee..a.mmom
14

---

Published as a conference paper at ICLR 2026
launch period, we use very narrow window of W 5 days. For PUBG, reflecting its regular seasonal updates and events, we use a medium window of W = 2 months.
Applying these tailored hyperparameters, our drift detection mechanism partitioned the timeline for each game into series of distinct phases. We identified 5 phases for Dying Light 2, 6 phases for Dune: Awakening, and 7 phases for PUBG Mobile. Segure@iilussrattee the evolution of the main community discussion topics across these detected phases for each game, visually confirming the significant interest drifts captured by our method.
Diae Foled
T1ae Poled
Tiae Foldd
(a) Dying Light
Dune: Awakening
(c) PUBG Mobile
Figure 9: Evolution of major user interest topics across the detected phases for each of the three games. Each colored area represents the proportion of distinct discussion topic.
B HIERARCHICAL TOPIC TAXONOMY
To systematically categorize the vast number of user questions for analysis and benchmark generation, we developed a detailed hierarchical topic taxonomy. We first collected and sampled thousands of real user questions from the player communities our three target games. Then, team of domain experts manually grouped these questions into thematic clusters. Through several rounds of refinement and consolidation, these clusters were organized into the final hierarchical structure presented Table 3 This taxonomy comprises 6 main categories and 21 sub-categories, covering the full spectrum of player inquiries from pre- purchase consultation to technical support and in-game strategies.
Table 3: The hierarchical topic taxonomy used for question classification.
Main Category
Sub-category
Description
Purchase Consultation
Inquirics about whether to buy, where to buy, and if the game is worth the price.
Purchase Related
Version Comparison
Questions about the differences between game versions.
Selection
Advice seeking on choosing between different platforms.
Preorder Rewards
Inquiries about preorder bonuses, special rewards, keys.
System Requirements
Questions regarding hardware specifications and compatibility.
Technical Support
Performance Issues
Issues related in-game performance, such framerate, lag, and optimization.
Crash & Errors
Problems like game crashes, freezes, black screens, or emur messages.
Network Connection
Issues with multiplayer connectivity, servers, or high latency.
Gameplay Mechanics
Questions ahout basic controls, game systems, and how to play.
Game Content
Content & Features
Inquiries about the game's specific content, modes, world size, and story. Questions seeking tips, guides, advice game progression.
Progress Guide
Version Updates
Questions about patches, updates, fixes, new content.
Team Cooperation
Inquiries related finding teammates and playing with others. Questions about the in-game social features, such as adding friends.
Social Interaction
Friend System
Community Events
Questions ahout official or community run events and competitions.
Refund Policy
Questions regarding the prucess and conditions for getting refund.
After-sales Service
Customer Support
Inquiries about how to contact customer service or report issues.
Account Issues
Problems related to user accounts, login, or activation keys.
Review Questions
Questions about game's ratings and reviews from others.
Review & Discussion
Comparison Discussion
Discussions comparing game similar tilles.
Expectation & Concern
Questions expressing hopes, worries, concerns about the gume's future.
15

---

Published as a conference paper at ICLR 2026
C GENERATOR EVALUATION PROMPTS AND METRICS
This section provides the detailed methodology and LLM prompts used for evaluating the performance of the generator models. We assess the quality of generated answers based on two distinct criteria: correctness and faithfulness.
For both criteria, we employ rigorous, 3-1evel scoring system, instructing the LLM judge to assign a score 0, 1, or based on detailed rubric. Score 2 (Excellent): The answer is perfect or near-perfect according the criterion. Score 1 (Acceptable): The answer is largely correct/faithuul but has minor, non-critical flaws. Score (Defective): The answer contains significant errors or fails to meet the core requirements of the criterion. To ensure consistency in our reporting, the final scores for each metric presented the paper are normalized to a [0. 1] scale by dividing the raw score by the maximum possible score of 2.
C.I CORRECTNESS SCORE
Correctness measures the factual accuracy of the predicted answer when compared against ground- truth answer. To ensure a strict and consistent evaluation, we provided the LLM judge with the following detailed prompt. For the reader's clarity, we have organized the subsequent prompt's structure and bolded key headings and instructions, but this formatting was not present in the operational prompt used by our code.
Prompt for Correctness Evaluation
You are an extremely strict game knowledge correctness evaluator. Your task evaluate the accuracy of a predicted answer against the ground truth answer for game-related question with maximum rigor.
ULTRA-STRICT EVALUATION CRITERIA:
1. FACTUAL ACCURACY (ZERO TOLERANCE), Every single fact about game me- chanics, rules, systems, and lore must be 100% correct. ANY factual error, no matter how minor, significantly impacts the score.
NUMERICAL INFORMATION (EXACT PRECISION); numbers, statistics, val-
ues, quantities, percentages must be exactly correct. Even tiny numerical discrepancies are heavily penalized.
TERMINOLOGY AND NAMES (PERFECT ACCURACY): Character names, cation names, item names, and ALL game-specific terms must be spelled exactly correctly.
4. COMPLETENESS AND COVERAGE (COMPREHENSIVE) The answer must ad- dress EVERY aspect of the question thoroughly. Missing ANY critical information from the ground truth is major defect.
5. ADDITIONAL INFORMATION (STRICT VERIFICATION), Any extra information
beyond the ground truth must be 100% accurate and verifiable. Speculative content hallucinations result in immediate reduction.
ULTRA-STRICT 3-LEVEL SCORING SYSTEM:
2 (Exceptionally Perfect): EVERY fact is 100% accurate. ALL numbers and ter- minology are precise. Comprehensively addresses the question. Truly exemplary answer.
1 (Acceptable with Minor Flaws): Core facts are accurate but contains 1-2 very minor issues (e.g.. missing non-essential details).
(Defective/Inadqquate): Contains ANY significant factual errors, multiple minor errors, notable numerical inaccuracies, or fails to address key aspects of the question.
Question: [Question]
Retrieved Contexts: [Documents]
Predicted Answer: [Answer]
Return your evaluation a JSON object with the "accuracy" field (0, 1, 2).
16

---

Published as a conference paper at ICLR 2026
C.2 FAITHFULNESS SCORE
Faithfulness measures whether the predicted answer is entirely grounded in and supported by the provided retrieved context documents. The prompt for Faithfulness similarly strict, focusing on zero tolerance for hallucinations or any information not present in the provided context.
Prompt for Faithfulness Evaluation
You extremely strict faithfulness evaluator. Your task is to evaluate whether the predicted entirely faithful the provided retrieved context documents with MAXIMUM RIGOR.
ULTRA-STRICT FAITHFULNESS EVALUATION CRITERIA:
1. INFORMATION SOURCE VERIFICATION (ZERO TOLERANCE); EVERY piece information the predicted answer MUST directly supported by the retrieved context. ANY claim not found in the context is violation.
2. FACTUAL CONSISTENCY (EXACT ALIGNMENT); All facts, numbers, and names must match EXACTLY with the context. No paraphrasing that changes meaning.
3. CONTEXT GROUNDING (MANDATORY SUPPORT): Each statement must he traceable to specific parts of the retrieved contexts. No external knowledge is allowed.
4. HALLUCINATION DETECTION (ZERO TOLERANCE). Any information not present in the contexts is considered hallucination, even "common knowledge".
5. OMISSION vs ADDITION PRINCIPLE: Incomplete but faithful answers preferred over complete unfaithful ones.
ULTRA-STRICT 3-LEVEL SCORING SYSTEM:
2 (Perfectly Faithful); EVERY statement is directly supported by the retrieved contexts.
No hallucinations, no external information.
1 (Mostly Faithful with Minor Issues): Core information is supported but contains 1-2 minor unsupported details slight paraphrasing.
0 (Unnaithful/Halllcinneee Contains significant information found in the contexts or any clear hallucination.
Question: [Question]
Retrieved Contexts: [Documents]
Predicted Answer: [Answer]
Return your evaluation as a JSON object with the "faithfulness" field (0, or 2).
C.3 VALIDATION OF L.M--S-JUDGE AGAINST HUMAN EXPERTS
To validate the reliability of our -L-M-as-Judge approach, we conducted an experiment to measure its agreement with human expert evaluations. For this validation, we used GPT-4o as our LLM judge. We randomly sampled 150 question-answer pairs (50 from each of the three games) generated by representative RAG system using text-embeddingg--siii
for retrieval and GPT-4o for generation. Three domain experts were recruited to perform a binary evaluation (0: Fail, 1: Pass) for both correctness and faithfulness, and their majority vote treated the ground truth. To streamline the process, a web-based annotation interface was provided to the experts, as shown in Figure T0
To compare the 3-level scores from the LLM with the binary human scores, applied lenient mapping: L.M scores ("Excellent") and 1 ("Acceptable") were mapped to (Pass), while a score of 0 ("Defective") was mapped to 0 (Fail). We then assessed the alignment using standard classification metrics. The results are presented in Table 4
Table 4: Performance of the LLM-as-Judge against the human expert ground truth. The high precision indicates the model is a reliable, albeit strict, evaluator.
Criterion
Accuracy
Precision
Recall
F1-Score
Correctness
70.67%
98.77%
65.04%
78.43%
Faithfulness
78.00%
96.30%
78.20%
86.31 %
17

---

Published as a conference paper at ICLR 2026
LLM Evaluation Results Human Expert Assessment Tool
LLM Ansewer Evaluation
Corpliteal E/ TTiii 380
Gans pubgin
Prora 253
Epent Exaluation
pubgge
daouae Conpleteenu
Queetion
Conerated Anewer
2 Araoer tat
Nervon Lotine Thersinn
meceria the merk Pire
wnther sn tnight te ale larker
the winte Ju Styrkp Soumaod moke kemba Ideed incodues seent ungs penolle nea The sine cette tine camportal paaaaass Reeons sodve Kappoe tot glvn you chance stan you deri maaed.
(Splnsad)
and was lost
wake leon
1,00
bad sheeiit siite und iest
souttt adition
"Courityy
Sou Cumea
Regurt Soailty
Figure 10: The web interface provided to human experts for the binary evaluation of correctness and faithfulness.
The results reveal key insights into the behavior of our automated judge. The most notable finding the extremely high precision for both correctness (98.77%) and faithfulness (96.30%). This indicates that the LLM judge rarely commits "false positive" error. When LLM classifies an answer as high-quality (Pass), we can be very confident in that judgment. Conversely, the model shows more conservative recall (65.04% for correctness), meaning it is stricter than human experts and sometimes flags acceptable answers as "Fail". This is a deliberate and desirable characteristic for benchmark's 's automated judge. In high-stakes domain like gaming, where incorrect guide faulty information can directly waste a player's 's time and ruin their experience, prioritizing precision over recall is crucial. A strict, conservative judge ensures that only truly high-quality and reliable answers receive passing score. This guarantees the integrity and high standard of our benchmark, ensuring that models that perform well are genuinely robust. Overall, with acceptable accuracy and high Fl Fl-scores, these results validate our use of an LLM, guided by our strict prompts, as scalable and reliable method for evaluating generator performance.
D TECHNICAL DETAILS OF THE CHRONOPLAY FRAMEWORK
This section provides a description of the key components of the ChronoPlay, covering the methods used for NER, the extraction of community-drive data assets, and the based data synthesis pipeline. The entire synthesis pipeline employs the GPT-4o model. Furthermore, we conducted a human expert evaluation the synthesized data to validate the quality of the synthesis.
18

---

Published as a conference paper at ICLR 2026
D.1 NAMED ENTITY RECOGNITION VIA SELF-ICL
A core requirement for our dynamic update mechanism is to accurately identify in-game entities o within the knowledge base. To avoid the need for extensive manual annotation for each new game, we employ a NER strategy based on Self-ICL (Chen et al. 2023). This process works in three main steps:
Pseudo-Input Generation: Given a target text for annotation, we first use an LLM to generate several text samples that are stylistically similar but feature diverse content. These pseudo-inputs are designed to maintain the same domain characteristics and linguistic patterns the original text, effectively creating set of rich, in-domain examples.
Next, we use a zero-shot prompt to perform an initial entity annotation on the pseudo-inputs generated in the previous step. This predicts the likely in- game entities and their types for each pseudo-input, resulting in collection of (pseudo-input, pseudo-label) pairs that serve as noisy but relevant training demonstrations.
Pseudo-Label Prediction:
In-Context Learning Stage: Finally, these (pseudo-input, pseudo-label) pairs are used as demonstrations for in-context learning. We construct new prompt that includes these concrete examples, followed by the original, real target text. Guided by these demonstrations, the LLM performs more accurate and context-aware entity recognition on the real text, having learned from the provided examples.
This Self-ICL method offers significant advantages over traditional zero-shot NER, primarily through superior domain adaptation. By generating pseudo-examples specific to the gaming context, the model effectively learns the unique linguistic patterns and expressions of the domain. This allows for more powerful form of in-context learning, where the model is guided by concrete example-label pairs rather than abstract definitions. The prompts for the three stages described above are detailed below.
Prompt 1: Pseudo-Input Generation
You are tasked with generating pseudo-inputs for game-relate Named Entity Recognition. Given the following game-related text, generate [Num Pscudo_ ] similar but different game-related text examples that would be suitable for entity recognition. The generated texts should: 1. Be similar in style and domain to the input text 2. Contain various types of game entities 3. Be realistic and coherent 4. Have different specific entities but similar context patterns
[Num_Pscudo_
Original text: [Question]
Return the result in the following JSON format only, no other text:
((
"pseudo "pseudo_inputs"t
"example text 1", "example text 2", "example text 3"
H
Prompt 2: Pseudo-Label Prediction
Extract game-related entities from the following text.
Entity Types: [Entity Desc]
Text: [Pseudo_Terte
Return the result in the following ISON format only, other
({
"entities":
11
"text": "entity text",
"type": "ENTITY_TYPE",
19

---

Published as a conference paper at ICLR 2026
"context": "brief context"
F
Prompt 3: In-Context Learning
You are specialized assistant for identifying game-related entities. Learn from following examples and then extract entities from the test input.
Entity Type Definitions: [Entity Desc]
Here
examples: [Demonstrations, Text]
Now, extract entities from the following test input.
Test Input: [Question]
Return the result in the following JSON format only, no other text:
((
" "entities":
"text": "entity text", "type": "ENTITY TYPE", "context"; "brief context"
D.2 EXTRACTION OF QUESTION TEMPLATES (Teomm) AND USER PERSONAS (lcomm)
To capture the authenticity of user questions described in Section 3.1.2
developed pipeline to process data collected from player communitics. The pipeline first filters posts to acquire genuine user questions. Each filtered question is then passed through an LLM classifier, which assigns it topic from our hierarchical taxonomy (see Appendix Bj. Finally, a subsequent LLM prompt is used decouple and extract core, reusable assets from each classified question: question templates p and their associated user personas u. The deduplicated collections of these assets form the Question Template Base Teomm and the User Persona Base Ucomm.
Question Template Extraction: The extraction of the Question Template Base Toomm aims to capture authentic, game-agnostic query patterns. The process orchestrated by an LLM instructed perform generalization task. For each user question, the model identifies and anonymizes specific in- in-game entities (e.g., character names, item names) by replacing them with standardized placeholders (e.g., [CHARACTER __AMEJ) This transformation results in a set of abstract question templates Each template then associated with its corresponding topic 0 from our taxonomy (see Appendix [B). The resulting template-topic pairs (p, 0) are stored in the Question Template Base. This decoupling of query patterns from specific game instances is crucial for the framework' 's scalability.
Prompt for Question Template Extraction
Please analyze the following question and generate 2-3 abstract question templates.
Question Information:
Question: (Question_ Contentt]
Question Topic: [Question_ Topic]
Generation Requirements:
1. Replace specific game names, platform names, character names, etc., with placehold- ers.
2. Maintain the structure and intent of the question.
3. Fach template should be usable for generating similar questions.
4. Use placeholder format: MAACEHCLDER_NANE
Common Placeholders:
20

---

Published as a conference paper at ICLR 2026
GAME NAME Game
PLATFORM] Platform name (Steam, Xbox. Epic, etc.)
[CHARACTER_]] __RRR__AM]]_____ :: Character name
And other relevant placeholders like (ITEM), [LOCACION], [VERSION], etc.
Output Format: Please return in JSON format, containing templates array, with each template including: template text, list of placeholders, and description. Example
format:
"cemplates":
{
"template": "Does pre-ordering [PLATFORM1] also give [REMARD TTPE], and is there any difference from [PLATFORM2]?* "placeholders": ["PLATFORM1", "FLATFORM2", "REWARD_TYPS" "description": "Template for asking about pre-order reward differences between platforms"
"template"; "What's the difference between pre-order rewards on [PLATFORM] and ailachholders"" ["PIATFORM", SMNCTHRR_PLATPORRS "description on": "Template for comparing pre-order rewards across platiorms"
?",
User Persona Extraction: Concurrently with template extraction, the same LLM prompt instructs the model to analyze the user's language, tone, and the context of their query to infer a plausible user persona The output is concise narrative description (e.g., "A new player struggling with the crafting system...") To ensure the quality of the User Persona Base Ucomm the generation process is guided by several constraints. The persona must be inferred solely from the provided text to prevent hallucination. Each generated persona is also accompanied by a model-generan confidence score, and questions lacking sufficient context for nigh-confidence inference are discarded.
Prompt for User Persona Extraction
Please analyze the following gaming player's question and generate a concise player back- ground description.
Question: [Question Content]
Based on the question content, write 50- (0-100 word player hackground description that describes the player's gaming experience, skill level, interests, and preferences. Requirements:
1. The description should be natural and fluent, like brief introduction of person.
2. Only make reasonable inferences based on the question content; do not over-interpret.
3. If the question is too simple or contains personal information, return null. Use second-person (" ("You are player who...") description format.
Output Format: Please return in JSON format. Examplc of successful extraction:
"player_desee dessyer_ddscrrptionn.. "You are player who...", "confiidence_scoree 0.8
If no meaningful player information can be inferred, please return:
gellyye__escrrpption null, "confidence__cee 0.0
21

---

Published as a conference paper at ICLR 2026
Semantic Deduplication: As final quality control step, employ a vector-based filtering mech- anism to deduplicate semantically similar items in both the Teomm and Ucomm bases. We use a sentence-embedding model (text-embedding-3-mme and compute their cosine similarity. Pairs with similarity score exceeding predefined threshold 0.7 flagged duplicates and filtered out. The deduplicated sets then undergo final manual review by domain experts to ensure the high quality and diversity of the final asset bases.
to convert assets into vector representations
D.3 HYPOTHETICAL Q&A GENERATION FOR IMPROVED RETRIEVAL
To bridge the semantic gap between abstract question template and detailed knowledge document, we employ a hypothetical question-answe (Q&A) A) generation step inspired by techniques like HyDE (Gao al. 2023). This process designed create semantically rich query vector that significantly enhances the relevance of documents retrieved for synthesis.
The core of this process is single LLM call that generates a complete, hypothetical Q&A pair directly from a question template. Given template from our Teomm base, its associated topic, and the target game name, we use specialized prompt to instruct LLM. The prompt guides the model to perform two actions in one step. First, it instantiates the abstract template into specific, plausible question by filling its placeholders. Second, generates detailed, hypothetical answer this newly created question. The model fabricates this answer based on its general world knowledge of game-like structures, without access to our specific knowledge base. The embedding of this semantically rich, hypothetical question and answer are then used as the query vector perform search against the knowledge base Kauth-
Prompt for Hypothetical Q&A Generation
Based on the following question template, generate specific question and corresponding hypothetical answer. Please ensure that placeholders in the template are replaced with appropriate content. Pay special attention to the game placeholder [EAMK_NAAM-]] please the correct game name game__ame
Question Template: [Question_ Template]
Question Topic: [Question_Tonee [Topic]
Please provide your response strictly following JSON format. Do not include any other text.
{
"question": "The specific question you generated based the template.", "answer": "The helpful and reasonable hypothetical answer to the question."
Requirements:
1. Questions should he specific and clear, conforming to gaming players' expression habits.
Answers should be reasonable and helpful.
3. there are placeholders in the template (such as [GAME [AME_ ) please replace them with appropriate content.
4. The answer's length should be moderate, both useful and not too lengthy.
D.4 DATA SYNTHESIS AGENT
The entire data synthesis process is orchestrated by a Data Synthesis Agent. This agent executes multi-stage workflow that fuses multiple data assets to generate candidate question-answer pairs, which are then immediately evaluated by real-time quality control mechanism to ensure only high-fidelity data is included in the final benchmark.
D.4.1 MULTI-STAGE SYNTHESIS PROCESS
The agent follows a multi-stage synthesis process that fuses multiple data assets. combines a sampled user persona and question template with pre-processed entities and specific task requirements
22

---

Published as a conference paper at ICLR 2026
(e.g., single-hop vs. multi-hop reasoning) to create comprehensive prompt. This prompt, detailed
below, guides powerful LLM to generate a candidate Q&A pair.
Prompt for Data Synthesis Agent (Data Generation)
SYSTEM PROMPT:
Background: You are an intelligent evaluation data generation assistant with deep role- playing capabilities. I am building multi-task evaluation dataset for retrieval auueeeetd gaming large language models. require you to automatically generate gaming domain evaluation data that is strongly correlated with evaluation tasks. I will provide the following content: [gaming suhtopics of focus for evaluation data, task descriptions and requirements for evaluation, gaming documents from knowledge base, and possible player role backgrounds]. You need to generate high-quality evaluation data based on the following principles:
1. Role Consistency: If a player role background is provided, you must fully immerse yourself that role's identity language style.
2. Authenticity Simulation: Generated questions must reflect real players' questioning habits and expression patterns.
3. Personalized Expression: Questions from different roles should reflect different gaming experience levels and focus points.
4. Template Guidance: If question templates are provided, generated questions should reference the template's 's structure and style, but with reasonable variations.
Quality Requirements for Generated Data
For Documents: Must be high-quality gaming materials (official guides, update logs. etc.). Do not generate from low-quality private content. If irrelevant, return empty list.
For Questions: Must he role-driven, specific, semantically complete, and strongly related to the topic. Must contain phrases like "according the document." Must
strictly comply with task definition (e.g., multi-hop reasoning).
For Answers: Must have high knowledge density, be factually consistent with the document, and contain no hallucinations.
For References: Must accurately extract segments from the original document that support the answer. Extracted content must be informationally complete and not taken out of context.
Data Generation Process: 1. First, determine if the provided document is high-quality and relevant the task. If not, return empty list. 2. If suitable, generate high-quality evaluation samples based
document content, task requirements, and gaming topics. Generated Data Format Requirements First start with #*THOUGHT PROCC_SPOOSS output your thinking process, then put results JSON data list format, surrounded by (ison></json>> format requirements are as follows:
< <json>
{
"question"; "Question raised from player perspective...", "answer": "Direct to the question, concise and clear...", "references": ("Specific content segment 1...", "Quote segment 2"]
</ison>
USER PROMPT:
Gaming Subtopics of Focus for Evaluation Data Descripton))
Evaluation Task Description and Requirements
Query Type: [Dper:_Type]: [Query_Type Descrrpptiiii
Role Context: Goole_Contex]]
Question Template: [Question_Temppee
Question Generation Specificity Guidelines
Important Reminder: Generated questions must be sufficiently specific.
23

---

Published as a conference paper at ICLR 2026
Hardware-related: Must specify exact models (e.g.. "RTX 4070").
Location-related: Must specify exact area names (e.g.. "Harran City Center").
Numerical-related: Must provide specific values or ranges (e.g., "level 30 and above").
Game Content: Must use accurate game terminology.
Version Handling: Do not directly mention version numbers in questions.
Provided Document
[Documents]
D.4.2 REAL-TIME QUALITY CONTROL MECHANISM
Immediately after synthesis, each candidate Q&A pair is passed to real-time quality control module. This module uses an LLM, guided by the prompt below, to score the data's quality on a three-point scale (0: poor, 1: average, 2: excellent) based on criteria like task compliance and answer accuracy. Only data scoring 2 is retained.
Prompt for Quality Control Agent
SYSTEM PROMPT:
Background: You are professional generated data quality assessor. I will provide you evaluation data generated by a large language model. Your task to assess the quality of this data. The quality divided into three levels: (poor), (average), 2 (high quality). Assessment Requirements
1. Determine if generated questions are related to the provided gaming subtopics. Determine questions meet the requirements the evaluation subtask.
Determine answers are correct and can be fully answered by the long document.
4. Determine if the relevant segments complete and sufficient support the answer.
Special Attention Points:
For "yes/no" questions, please mark their quality as O.
For multi-hop reasoning Q&A, please ensure the question requires at least two steps of reasoning. If it is a pseudo multi-hop question, its quality should be or 1.
Output Requirements
Asscssment results should be returned in JSON format: ("evaluation": [,11,
USER PROMPT:
Long Document in Gaming Domain Used to Generate Data: [Documents]
Gaming Subtopics that Generated Data Should Conform To: Descriptoo)))
Description and Requirements of Evaluation Subtasks: [Query_Typel
Evaluation Data Generated by Large Language Model to Assessed: [Generated Data]
D.5 HUMAN EXPERT EVALUATION OF SYNTHESIZED DATA
To validate the quality of the data generated by our synthesis pipeline, we conducted a comprehensive evaluation with human domain experts. We recruited three experts, all of whom are veteran players of the target games. A random sample of 210 instances (70 from each game) was selected for this evaluation.
The experts performed the annotation using a web interface, designed to facilitate clear and efficient evaluation process, as shown Figure [T] To ensure consistent and hugh-quality annotations, we provided all experts with detailed annotation guideline. The guideline instructed annotators to perform binary (Yes/No) evaluation on three dimensions for each data sample: (1) Correctness, assessing if the answer correctly addresses the question; (2) Reference Quality, assessing if the cited documents are relevant; and (3) Entity Accuracy, assessing if the extracted entities are correct. The guideline emphasized a strict evaluation for Correctness and Reference Quality. In contrast, instructed more lenient approach for Entity Accuracy, where only clear and significant errors would result in "No." This lenient standard was chosen because sample entities, and what constitutes a relevant entity can be subjective. A stricter requirement could lead to large annotation variances due differing expert interpretations and preferences.
contain many potential
24

---

Published as a conference paper at ICLR 2026
RAG Evaluation Data Human Expert Assessment Tool
Dota Evaluation
Completed: Tobat
Export Evaluation
dyinglight2_A..o0 D04
dylnglightt
4 230
Quartion
Iyr sultinn In Light withr boong Move any nour pitaho foot
wiant ropldd
Generated Answer
No. vocn updcoo hav rcveace acroul cot ristoons thtt wad not Uoduic Lod succifiall criar or tooro
intics optimted nanery uape duttrni,, whit paoiing wodning ligting selitersie sals vitt- precc,, whlch ouur
Reference
3. up
theore you san wn's fubs uoddall
aduerage
Reference Bocuments
Lucett Sysee zo symtem Eopui Acmetie entaperyy Busiana
mosess nos sentts tholke auciciaaare nudld nutiates
3. Euiiyyy Accurary Assecsment
the entty clbmotnnncoww
bers coses ahore miker the tire
villomes Jnlerise Bilnse
"he 20n,
Entiyy Enfermation
Lightng
Deverect
Nevinu
Sacn
Poport Arotts
Figure 11: The web interface for human expert evaluation. Annotators can view the question, answer, reference documents, and entities one place to provide scores for each dimension.
The average scores from the three experts across the three dimensions presented in Table 5 We also calculated the inter-rater reliability among the three annotators using Krippendorff's Alpha, standard measure of agreement.
Table Human expert evaluation results. Scores represent the average three experts on binary (0/1) scale. Agreement is measured using Krippendorff' s Alpha Ford (2004).
Game
Correctness
Reference Quality
Entity Accuracy
Agreement
Dune: Awakening
0.973
0.956
0.929
0,952
PUBG Mobile
0.932
0.971
0.914
0.934
Dying Light 0.923
0.963
0.914
0.923
The results show a very high level of quality for our synthesized data across all evaluated dimensions, with average scores consistently above 0.91 for all games. The high inter-rater agreement further validates the clarity of our annotation guideline and the reliability the evaluation results, confirming that our data generation pipeline produces high-quality and trustworthy benchmark data.
25

---

Published as a conference paper at ICLR 2026
E SUPPLEMENTAL RETRIEVAL RESULTS FOR K=1 AND K=5
This section provides supplemental results for the retriever evaluation to offer more comprchensive view of model performance. Table and Table present the detailed, thase-by-phaal results when the number of retrieved documents is set to and 5, respectively. These tables complement the K-3 results shown in the main paper.
Table 6: Retrieval performance all three games for K=1. We report Recall @ (R@1), FI and NDCG@1 (N@1), The best performing result in each row bold.
BM25
Owent-Embeddiigg F1@11
BGE-MB F1@1
text-embedding-3
Phase
R
F1
N@1
N@:
R@1
N@1
R@1
FI@
N@
DL2
0.270
0.383
0.718
0.219
0.309
0.578
0.239
0.339
0.637
0.291
0.416
0.787
0.299
0.419
0.777
0.213
0.291
0.522
0.238
0.331
0.608
0.345
0.430
0.802
0.250
0.630
0.367
0.315
0.828
4
0.226
0.322
0.608
0.188
0.264
0.492
0.188
0.265
0.495
0.247
0.352
0.665
0.204
0.290
0.547
0.164
0.228
0.420
0.177
0,250
0.465
0.262
0.375
0.713
Dune
0.194
0.282
0,540
0.237
0.344
0.656
0.000
0.000
0.000
0.240
0.348
0.668
0.188
0.270
0.506
0.211
0.303
0,568
0.000
0,000
0.000
0.236
0.341
0.646
0.238
0.341
0.640
0.257
0.367
0.688
0.000
0.000
0.000
0.272
0.391
0.738
0.188
0.269
0.504
0.215
0.308
0.580
0.000
0.000
0,000
0.230
0.332
0.630
0.179
0.254
0.470
0.219
0.314
0.588
0.000
0.000
0.000
0.232
0.333
0.624
0.181
0.262
0.502
0.201
0.293
0,562
0.000
0.000
0.000
0.225
0.327
0.630
PUBG
0.273
0.410
0.820
0.258
0.388
0.775
0.278
0.417
0.835
0.259
0.393
0.782
0.158
0.237
0.475
0.158
0.237
0.475
0.160
0.240
0.488)
0.157
0.235
0,470
0.245
0.367
0.735
0.255
0.383
0.765
0.257
0.385
0.770
0.245
0.367
0.735
4
0.145
0.217
0.435
0.153
0.230
0.460
0.158
0,237
0.475
0.163
0.245
0.490
0.240
0.360
0.218
0.655
0.222
0.333
0.200
0.600
6
0.233
0.350
0.700
0.232
0.347
0.695
0.235
0.352
0.705
0.233
0.350
0.700
0.202
0.302
0.605
0.212
0.318
0.635
0.220
0.330
0.660
0.208
0.312
0.625
Table Retrieval performance on three games for K=5. We report Recall @5 (R@5), F1 F1 and NDCG@5 (N (). The best performing result in each row is in bold.
BM25
Qwent-Embeddingg F1@5
BGE-M3 F1@5
text-cmbedding-3 F1@5
Phase
R@S
F1@5
N@s
R@5
N@5
R@S
N@5
R@S
N@5
DL2
0.439
0.307
0.528
0.426
0.299
0.524
0.454
0.320
0.571
0.607
0.433
0.807
0.422
0.291
0.529
0.385
0.263
0.435
0.417
0.290
0.615
0.790
0.447
0.309
0.553
0.406
0.278
0,508
0.430
0,299
0.551
0.636
0.450
0.844
0.259
0.323
0.457
0.255
0.502
0.711
0.337
0.237
0.411
0.312
0.219
0.423
0.335
0.236
0.437
0.505
0.359
0.678
Dune
0,334
0.237
0.417
0.399
0.286
0.666
0.167
0.122
0.128
0.434
0.313
0.709
0.628
0.049
0.394
0.280
0.347
0.244
0.465
0.399
0.280
0.729
0.084
0.062
0.073
0.427
0.303
0.785
0.313
0.220
0.387
0.367
0.261
0.615
0.083
0.061
0.078
0.394
0.281
0.664
0.307
0.214
0.378
0.371
0.261
0.634
0.077
0.057
0.074
0.384
0.272
0.675
0.302
0.215
0.377
0.369
0.265
0.605
0.107
0.080
0.094
0.387
0.280
0.652
PUBG
0.478
0.359
0.540
0.595
0.446
0.630
0.555
0.416
0.615
0.595
0.446
0.634
0.308
0.231
0.334
0.445
0.334
0.446
0.420
0.315
0.425
0.440
0.330
0.439
0.582
0.436
0.605
0.680
01.10
0.684
0.642
0.481
0.656
0.678
0.509
0.674
0.287
0.215
0.309
0.390
0.292
0.402
0.365
0.274
0.385
0.390
0.292
0.410
0.380
0.285
0.445
0.453
0.340
0.498
0.437
0.328
0.483
0.423
0.318
0.461
0.306
0.567
0.587
0.383
0.580
0.597
0.393
0.295
0.427
0.537
0.403
0.564
0.483
0.362
0.527
0.512
0.384
0.543
As shown in Table 6 and Table
7 the key trends observed for K=3 in the main analysis are consistent across K=1 and K=5 settings. The summarized conclusions are as follows:
Performance Volatility: The performance of all models continues to exhibit significant volatility across the lifecycle phases. The performance dip in Phase 4 of PUBG is clearly
26

---

and Published as a conference paper at ICLR 2026
Deeplesk v3 Chor
Gemini 2.5 Fash
-8- GPT 4o
LLaMM4 17B
Qwent 14B
Clande 3. Sonner
Paittfilnss Score
0.5
0.5
0.4
DL2
Dune
PUBG
Figure 12: Generator faithfulness scores across the lifecycle phases of each game.
visible in both the K=1 and K=5 results, confirming that major game events impact retrieval regardless of the number of documents returned.
Model Rankings:
The relative ranking of the models also remains largely consistent. text- embedding-3 generally maintains its lead, but other models like BGE-M3 remain competitive in specific phases.
Domain-Specific Challenges: Notably, the extremely low performance of BGE-M3 on the Dune: Awakening benchmark persists across all K values, reinforcing hypothesis that this model struggles with the unique, noun-heavy vocabulary of the Dune universe.
As expected, increasing K from 1 to generally improves Recall and Fl scores for all models, as there a higher chance of retrieving a relevant document. However, the overall performance patterns and the challenges posed by dynamic shifts remain the same. This demonstrates the robustness of our findings and validates that the challenges identified by our benchmark are fundamental and not an artifact of the specific value of K.
F GENERATION PERFORMANCE ON FAITHFULNESS SCORES
This section provides the detailed faithfulness scores for the generator evaluation, complementing the correctness scores presented in the main paper. Faithfulness measures whether the generated answer is grounded in and strictly supported by the provided context documents. The analysis of Faithfulness scores in Figure reveals several unique insights into model behavior that complement the Correctness results from the main paper.
First, we observe shift in the top-performing model. While GPT-4o excelled in correctness, Gemini- 2.5-flash consistently achieves the highest faithfulness scores across all games and phases. This suggests that different models exhibit different strengths. Some are better at producing factually correct answers, while others are better at strictly adhering to the provided documents.
Second, the results show that faithfulness is not always positively correlated with correctness. This divergence highlights fundamental trade-off RAG systems. model can achieve high faithfulness by accurately repeating information from retrieved document, but if that document is incorrect or outdated, the final answer will have low correctness. Conversely, model might use its internal parametric knowledge to produce a correct answer even when the retrieved context is insufficient, leading to high correctness but low faithfulness. Our benchmark's ability to measure both dimensions is crucial for identifying these different model behaviors.
Finally, just like correctness, faithfulness is strongly influenced by the benchmark's dynamic nature. The significant fluctuations, particularly for PUBG, demonstrate that model's tendency to hallucinate or generate ungrounded responses is not fixed trait. Instead. is highly dependent on the challenges of each lifecycle phase, such the quality of retrieved documents. This analysis confirms that comprehensive evaluation requires dynamic benchmark that can surface these complex performance trade-offs over time.
27

---

Published as a conference paper at ICLR 2026
G TOPIC PERFORMANCE ANALYSIS
To provide more fine-gre tne-grained understanding of the RAG system's 's performance in the game domain, we analyzed it topic basis. For this evaluation, we used RAG system with text-embedding- as the retriever and GPT-4o as the generator. Figure 13 presents a heatmap that visualizes the performance across the top six most frequent topics for each of the three games. important to note that the dominant topics vary significantly between games, reflecting the unique focus and lifecycle stage of cach player community. This analysis allows us to identify which types of user interests pose the greatest challenges to modern RAG systems.
Gamupha
Camsply
0.57
Camplaa
Cndi
0.44
0.26
0.40
Porformenc
0.36
0.48
0.56
Accouni
0.24
0.57
0.53
17
Netwokk
0.36
0.77
0.36
Craak
0.48
0.76
0.42
Vasion
0.19
0.81
0.27
L4
0.40
0.41
0.65
0,76
0.75
Pefformance
0.55
0.72
0.45
Vanámn
Porchanc
Comparioo
Prugros
0.35
Newwokk
Cruh
0.09
Cummetsess
FIGSS
Curmecatas
Faialeimos
F1663
Cheracttane
Faihfilnco
F1003
(a) Dune
DL2
(c) PUBG
Figure 13: Heatmap of RAG performance across the top six topics for each game. Lighter colors indicate lower performance, highlighting challenging topies for the RAG system.
The topic performance breakdown reveals several key insights. First, performance is not uniform across topics. Technical, fact-based topics with clear answers generally perform well. For example, the SYSTEM REQUIREMENTS topic in DL2 achieves high scores across retrieval, correctness, and faithfulness. In contrast, more nuanced and complex topics consistently challenge the RAG system. This analysis provides deeper, data-driven explanation for the performance drop observed in our main analysis (Section 4.2.1). For DL2, the GAMEPLAY MECHANICS topic, which became dominant during the challenging Phase 4, shows significantly lower Retrieval FI score (0.44) compared to more straightforward topics like SYSTEM SYSTEM_ REQUIREMENTS
(0.73). This confirms that the performance degradation was caused by a shift in user interest towards an inherently more difficult topic.
Furthermore, the topic performance breakdown highlights the critical divergence between faithfulness and correctness, revealing key failure mode for RAG systems. For instance, the CRASH ERRORS topic in PUBG, the system achieves a very high faithfulness score (0.92) but an extremely low correctness score (0.09). Crucially, this occurs in scenario where the retrieval performance is also exceptionally poor (F1 @ of 0.08). This paradoxical outcome indicates a specific failure cascade: the retriever failing find the correct documents (e.g., official solutions patch notes). Instead, is likely retrieving topically similar but factually incorrect documents. The generator then performs its task perfectly, faithfully summarizing this incorrect information. The result an answer that is well-grounded in the retrieved context but completely wrong. This underscores the importance of evaluating all components of the RAG pipeline, failure in retrieval can directly lead the generator to produce confident but misleading answers.
H ABLATION STUDY DETAILS AND CLARITY ANALYSIS
This section provides a detailed description of the experimental setup for the ablation study presented in RQ3, along with in-depth analysis of the evaluation results for the clarity criterion.
28

---

Published as a conference paper at ICLR 2026
H.1 EVALUATION CRITERIA AND METHODOLOGY
To assess the quality of the questions generated by our different synthesis pipelines, we defined two key criteria:
This metric assesses how closely generated question resembles a question real human player would post in community forum. It considers the tone, phrasing, use of domain-specifer jargon, and the underlying user intent.
Authenticity:
Clarity: This metric evaluates how well-formed, unambiguous, and easy to understand the question is. It assesses grammatical correctness and the straightforwardness of the query.
Our evaluation employs 4-way forced-choice comparison. To conduct this study, we randomly sampled 200 source documents for each of the three games, resulting in a total of 600 evaluation instances and 2,400 questions. For each instance, we generated four questions from each pipeline configuration. Then we presented them in randomized order to the LLM. The LLM was tasked with selecting the single best question for each of the two criteria separately. The final score for each setting is its win rate: the percentage of times it was chosen as the best.
H.2 LLM-AS-JUDGE PROTOCOL
We used a panel of three models as LLM judges: GPT-4o, Gemini 2.5-Pro, and DeepSeek-RI. Each model was provided with unified prompt that included the source document, the four candidate questions, and instructions to evaluate them on both authenticity and clarity. The evaluation was forced-choice selection, where the LLM was instructed to identify the single best question for each criterion. The chosen question receives score of 1, while the others receive O. The final win rate reported the main paper is the average score across the three LLM judges. The comprehensive prompt provided to the models is detailed below.
LLM-as-Judge Prompt for Authenticity and Clarity Evaluation
Instruction: You are an expert in evaluating the quality questions related video games. Please assess the following 4 questions based on two criteria: Authenticity and Clarity.
Evaluation Criteria:
Authenticity: Does the question genuinely reflect the actual needs and confusion of game players? Is based on a real gaming experience? Clarity: Is the question's 's expression clear and explicit? Is it easy understand? Is it specific and not vague?
Task: For each criterion, you must choose only single best question. The chosen question will receive score of 1, and the others 0. This requires you to carefully compare questions select top performer for that specific dimension.
Candidate Questions:
1. [Question 1]
2. [Question 2]
3. [Question 3]
4. [Question 4]
Output Format: Please provide your answer strictly in the following JSON format.
"authenticity"" Chosen question number (integer, e.g., 1), " "clarity": Chosen question number (integer, e.g., 3), "reasoning";
"authenticity" "Detailed reascning for your choice, with comparisons to others.",
"clarity": "Detailed reasoning for your choice, with comparisons others."
Notes: 1. You must select only one question for each criterion (an integer hetween 1-4). 2. Question numbering from 1, You must provide detailed comparative analysis
29

---

Published as a conference paper at ICLR 2026
in your reasoning. Please ensure the JSON format is correct. Return only the JSON object and nothing else.
H.3 HUMAN EXPERT PROTOCOL
We recruited three domain experts who are veteran players of the target games to provide a ground- truth evaluation. To create the human evaluation set, we randomly sampled 50 instances from each of the three games, for total of 150 instances and 600 questions.
To ensure consistent and high-quality annotation process, provided the experts with custom- built web interface as shown in Figure [1 and a detailed annotation guideline. For each instance, their majority vote was used the final decision.
Question Evaluation Expert Tool
1. Upload CSV File
Question Evaluation
Evaluation ID: 11eeb537- 001353773222e 4d7b-af50- wdwp-//550--ee--eeeee 6ac946023cbb
Game: dune
Please evaluate quality following questions:
humen mvakationn deta
Quastion Hox Cantrol Partta
Deaor bost Hozst's
Tooal 150 eviketion neards
Infuence
Strt Exalaation
Queation What makes Cantrol Points Deep Desrrt strategically important
control
Question Cantrol Paints infuence without there constanty??
what strategies
Quertion Whel's the find Cantrdl Deaet after Corbil Dune?
Previous
Noct
Figure 14: The web-based interface provided to human experts for the 4-way forced-choice evaluation.
H.4 DETAILED ANALYSIS OF CLARITY EVALUATION
the main paper, the evaluation of clarity revealed notable discrepancy between the judgments of the LLMs and the human experts.
As shown in Figure 15
The LLM-as-judges consistently rated questions from the w/o User Persona setting as the clearest. We hypothesize this is because removing the user persona strips the question of its subjective context and conversational phrasing (e.g., beginner's tone or an expert's implied knowledge). This results in more direct and objective question. LLMs appear to have strong bias towards this direct style, perceiving it as less ambiguous and therefore higher in clarity, In contrast, the human experts rated the Full Pipeline the best for clarity. This suggests that for domain expert, clear question is not merely the most direct one. It a question that effectively uses appropriate personal context to frame meaningful and unambiguous query. A human might as contextless and harder to understand its true intent. This finding provides valuable insight into the behavior of LLM judges: they highly effective at evaluating objective criteria but may
question without any persona
30

---

Published as a conference paper at ICLR 2026
Full Pipline
w/o Hypothoes QKA
w/o User Ponena
wño Question Template
22.9
19.5
20.4
23.7
19.2
23.9
19.3
35.3
18.1
17.4
16.7
19.4
41.7
31.7
59.2
39.5
Dune
DL2
PUBG
LLM Average
Iuman Evaluation
Figure 15: Ablation study results for the clarity criterion. The pie charts show the win rates of our four synthesis settings across the three games, the average score, and the human expert evaluation.
interpret a subjective metric like clarity as structural simplicity. reinforces the importance of our two-pronged evaluation strategy. By using human experts as the final arbiter for such nuanced criteria, we can confidently validate that our Full Pipeline superior overall, as it produces questions that are genuinely clear and meaningful to a human audience.
I GAMERAG-22 232 LEADERBOARD
In addition to the dynamic lifecycle analysis in the main paper, we constructed a static, high-quality benchmark provide comprehensive comparison of various RAG architectures. This allows to establish clear leaderboard and analyze the performance of different model combinations in the gaming domain.
1.1 THE GAMERAG-233 BENCHMARK
To facilitate a broad and rapid comparison of different RAG architectures, we created high-quality evaluation set, which we term 23m-R2 This dataset consists of 232 expert-verified samples drawn from all three games. Its construction followed a rigorous multi-stage filtering and verification process: (1) We first used our synthesis framework to generate a large pool of 1,000 candidate question-answer pairs for three games. (2) This initial pool was then filtered using embedding-basee similarity calculations to remove redundancy, resulting in a set of 428 diverse samples. (3) Finally, these 428 samples were meticulously reviewed by our domain experts, who selected the final 232 high-quality validated samples.
1.2 GAMERAG-22 232 LEADERBOARD AND ANALYSIS
We benchmarked 50 distinct end-to-end RAG pipeline combinations the CameRAG-232 set. To provide a holistic performance metric, we define Total Score as:
Total Score Correctness + (Faithfulness//3
We designed this composite score to place significantly stronger emphasis on Correctness, this the primary indicator of successful answer for the end-user. In the gaming domain, factually incorrect answer (low Correctness) is highly detrimental to the experience, even if it is perfectly faithful to poorly retrieved document. Faithfulness remains a critical metric, but is weighted as secondary component that assesses the system's 's ability to avoid hallucination. The full leaderboard is presented in Table
ANALYSIS OF LEADERBOARD RESULTS
This leaderboard provides comprehensive insights into RAG design for the gaming domain:
The Critical Role of Reranking: The most significant finding the powerful impact of reranker. The top three ranked pipelines, and seven of the top ten, all utilize the qwen3- reranker-80 8b. Comparing otherwise identical pipelines (e.g., Rank vs. Rank 8, or Rank Rank shows that the reranker provides substantial boost in final performance.
31

---

Published as a conference paper at ICLR 2026
Retriever Choice is Fundamental: The choice of the base retriever is critical. Pipelines using a strong dense retriever like text-embeddingg 3-small vastly outperform their BM25 counterparts.
Hybrid Retrieval is not always better An interesting finding is that hybrid retrieval did not improve performance our tests. all comparable cases, the pipelines using only the dense retriever plus reranker consistently outperformed their hybrid counterparts. This suggests that when a powerful reranker is employed, the additional, dess-preciss, results from BM25 may introduce more noise than signal, slightly hampering the final performance.
The generator LLM is a key component, as different model families are optimized for distinct capabilities. For instance, the GPT series noted for its high general correctness. The Gemini series is recognized for its strong faithfulness. These inherent strengths of the generator play a significant role in the quality the final RAG output.
Generator's Inherent Strengths:
This new experiment successfully provides a broad comparison of different RAG pipline. We believe this leaderboard is valuable contribution that provides the community with clear starting point for selecting RAG pipelines for the gaming domain.
Table 8: RAG Pipeline Leaderboard on the GameRAG-232 dataset.
Rank
Retriever(Topk=3)
Reranker
Generator
Correctness
Faithfulness
Total Score
text-cmbedding---nnm
qwen3-reranker-95
ol gemini-2.5- gemini- p.5-pru
0.5267
0.4353
0,6718
text-ombeddinng---nnn
gwen2-reranker-88
0.3882
0.8362
0.6669
lest eemeediing---smaee
qwen3-reeanker-- 8b
0.3537
0,9095
0.6569
text desteemhedding---mmm.
al ol gemini-2.5-flash
0.5000
0.4504
0.6501
hybrid (rext-emb-3-s bm25)
gwen3-renunke-Sh qwen2-reranker-Bb
0,4800
0,4950
0,6450
6
hybrid (text-emb-3-s + bm25)
0,3700
0,8040
0.6380
text text-embeldingg---mal
gemini-2.5-pm
0.3721
0.7909
0.6357
8
text-ombeddingg---15
gemini-2.5-flash
0.3685
0,7866
0.6307
hybrid Lext-emb--3s- + bm25)
gwen3-rerankes qwen3 reranker 8b
gemini-2.5-pro
0.3400
0.8640
0.6280
10
text embedding 3-small
u3
0.5118
0.2909
0.6088
11
rext-emheldingg---mme
n3
0.5129
0.2737
0.6042
12
text-embeddingg--e I-small
gwen3-renuner-80
gpt-4.1
0.4573
0.3809
0.5843
13
gpt-4.1
0.4440
0.4138
0.5819
text-embedding----nne
gpe-5
0.4332
0.4397
0.5797
15
text-mmbedding---sn..
deepseok-v..2
0.3470
0.6315
0.5575
bmn5
ol
0.4052
0.4418
0.5524
17
text embedding 3-mmal
gwee3-reranker-Se Sb
gpt-
0.4181
0.4030
0.5524
18
text-embedding
gpt-4o
0.3793
0.5151
0.5510
eext-ombedding-3-e 3-small
gwen3-reranee-SSb
gpt-4o
0.3942
0.4698
0.5508
20
bm25
gpt-4.1
0.4181
0.3944
0.5496
bm25
gomini-2.5-flash
0.2974
0.7500
0.5474
22
bm25
gpt-5
0.4203
0.3793
0,5467
text embedding 3-small
claude 4.5
0.3405
0.5819
0.5345
bm25
gemini-2.5-pm
0.2866
0.7371
0.5323
25
texteembalding--ssaal
kimi-42
0.3944
0.4095
0,5309
sat-ombbdding---smaae
glm-4.5
0.3405
0.5280
0.5165
27
3
tur-o turbo
0.3556
0.4612
0,5093
bm25
deepscek-v3.2
0.2931
0.6379
0.5057
29
sat-embbedding---nnn
claude-3.7-somme
0.3017
0.6013
0,5022
30
bm25
0.4095
0.2694
0.4993
gwen3-max
0.3599
0.4181
0.4993
32
bm25
doepscek-v3.1
0,2909
0.6207
0.4978
eext-embeddingg---a 3-mall
claude-sonnet-4
0.3103
0.5539
0.4950
34
bm25
4o
0.3405
0.4483
0.4899
bm25
glm-4.5
0.2996
0.5431
0.4806
36
bm25
kimi-k2
0.3147
0.4569
0.4670
37
text dett-emhelding---oe 3-smal
claude-3.5 soneettt
0.3039
0.4806
0.464)
38
bm25
claude-3.7-sonnet
0.2522
0.6228
0.4598
39
oat-emmbedding--ssma.
deopscek-v3.1
0.2931
0.5000
0.4598
bm25
claude
0.2651
0.5733
0.4562
41
hm25
sonde--onnet44
0.2737
0.5345
0.4519
bm25
gwen3-max
0.3082
0.4138
0.4461
43
lext embedding small
grok
0,2888
0.4375
0,4346
44
bm25
gpt- gpt-3.5-turbo
0.2802
0.3901
0.4102
bm25
claudo-3.5-sonnct
0.2392
0.4978
0.4052
46
bm25
grok-3
0.2150
0.3720
0.3390
embedding small
scout-
0.2371
0.3707
0.3606
48
qwen2.5-72h
0.2694
0.2586
0.3556
bm25
gwen2.5-72b
0.2241
0.2478
0.3068
50
bm25
4-scout- 17h
0.1940
0.3362
0,3060
32

---

Published as a conference paper at ICLR 2026
J DE-CONFOUNDING TOPIC SHIFTS FROM INHERENT QUESTION DIFFICULTY
To strengthen our central claims, we conduct a de-confounding analysis to determine whether the observed performance volatility is driven by shifts in topics or by an inherent shift in question difficulty/type. We analyzed the distribution of question types: Comparative, Extraction-batt baatdd, Long-form Answer, and Multi-hop Reasoning across every phase for all three games. The results, detailed in Tables 9 [0] and [1 show that the distribution of question types remains relatively stable across the entire lifecycle. Crucially, there are no drastic or sudden shifts the proportion of complex question types that correlate with the significant performance drops observed in our main experiments (e.g., the dip in DyingLigh12 Phase 4). For instance, while Extraction-basse QA shows gradual trend in Ligh2 from 62.7% to 73.0%, it lacks any anomalous spike Phase 4 that could otherwise explain that phase's ' poor performance.
Table 9: Question type distribution across phases for DUNE.
Task Type
P2
P3
P5
P6
Comparative
28. 18.0%
27.8%
27.8%
28.4%
25.4%
26.8%
Extraction-based
67. 67.0%
66.6%
67.4%
65.8%
69.2%
67.6%
Long-form Answer
4.6%
4.8%
4.2%
5.0%
4.8%
4.8%
Multi-hop Reasoning
0.4%
0.8%
0.6%
0.8%
0.6%
0.8%
Table 10: Question type distribution across phases for SyyingLight?
Task Type
P1
P2
P3
P4
P5
Comparative 27.0%
24.8%
26.2%
27.8%
19.8%
Extraction-basea
62.7%
65. 55.2%
64.0%
67.8%
73.0%
Long-form Answer
9.2%
9.0%
8.8%
3.8%
5.2%
Multi-hop Reasoning
1.0%
1.0%
1.0%
0.8%
2.0%
Table 11: Question type distribution across phases for PUBGM.
Task Type
P1
P2
P3
P4
P5
P6
P7
Comparative
28.5%
29.0%
23.5%
30.5%
30.0%
29.0%
38.0%
Extraction-basee
69.5%
68.0%
74.0%
67.0%
66.5%
67.5%
59.5%
Long-form Answer
1.5%
2.5%
2.0%
2.0%
3.0%
2.5%
2.5%
Multi-hop Reasoning 0.5%
0.5%
0.5%
0.5%
0.5%
1.0%
0.0%
This stability across phases is an intentional result of our synthesis pipeline's design. As described in Section 3.2.1, our synthesis agent begins by sampling a question type from a predefined set with an initial sampling probability of 25% for all types. This consistent initialization for every phase ensures that the attempt generate each question type remains uniform across the benchmark's entire lifecycle. The final observed distribution deviates from 25% due to our rigorous Quality Control mechanism, which filters out inauthentic unanswerable questions. For instance, multi -opp questions are inherently rare in real-world player communities; thus, most synthetic attempts for this type are discarded as being low-quality or inauthentic, ensuring the benchmark remains faithful to actual player behavior while maintaining structural stability.
In conclusion, this analysis allows us to definitively de-confound the experimental variables. The significant performance volatility observed cannot be attributed to shift in question types, as the distribution remained stable. Instead, the volatility is conclusively driven by topic shifts, proving that specific topies like Gameplay Mechanics inherently more challenging for RAG systems to handle, even for the same basic extraction- based question types. This confirms that Chrono successfully captures genuine, topic-driven challenges that would be overlooked by static benchmarks.
33

---

Published as a conference paper at ICLR 2026
K EVALUATING KNOWLEDGE-ADATTI RAG SYSTEMS
To further demonstrate the utility of ChronoPlay evaluating advanced RAG architectures, we conducted an additional experiment comparing a standard append-only baseline with more so- phisticated knowledge-adaptive RAG system. While standard baselines typically handle knowledge updates by simply appending new documents to the corpus. knowledge-adapting system actively manages its knowledge base to mitigate the impact of stale information.
The whwwledge-adaptivve system incorporates a fund-and-replace" update mechanism: (1) Upon the arrival of new knowledge Kr, the system extracts key entities to retrieve potentially affected snippets Sold from the existing corpus K (2) An LLM agent performs factual validation to identify contradictions between Sold and K₃. (3) Contradicted snippets are updated or rewritten by the agent to reflect the most recent information before being re-indexed alongside the new knowledge. We evaluated the correctness score of both systems on the Dying Light benchmark using GPT-4o and eaatt-emedding---eee -smal.. The results are presented in Table
Table 12: Comparison between Append-only and sooowledge-ddaptiin RAG systems on DL2.
RAG System
P1
P3
P4
P5
Append-only 0.4225
0.4150
0.4687
0,3887
0,3800
Knowledge-adaptive
0.4225
0.4283
0.4752
0.3921
0.4058
The experiment yields two primary findings:
Quantifying Adaptation Benefits: Phase 5, where Knowledge Evolution accounts for 42.2% of the environment change (as shown in Figure 8(a)), the knowledge-adaptive system effectively cleaned stale information, improving correctness from 0.3800 to 0.4058. ChronoPlay successfully quantifies this gain, proving its ability to measure system's capacity for factual ceff-maintenanee.
The Persistent Challenge of Interest Drift: Conversely, Phase 4 is almost entirely driven by User Interest Drift (31.2% of the change) with minimal knowledge updates. In this phase, the knowledge-adaptive mechanism provided negligible gains (0.3887 vs. 0.3921). The performance volatility remains due the surge in complex Gameplay Mechanics questions, an issue that knowledge-cleaning alone cannot solve.
This de-confounded analysis proves that Dual Dynamics represent two independent challenges. By providing the tools to isolate and measure these forces, ChronoPlay enables researchers to identify whether RAG system's 's failure stems from outdated knowledge or an inability to handle shifting user query patterns.
L LLM USAGE STATEMENT
In compliance with the ICLR 2026 policy, disclose the use of a large language model as an assistive tool in the preparation of this manuscript.
The model used Gemini 2.5-Pro. Its role was strictly limited to that of writing assistant for polishing parts the text. Specifically, it was used to improve grammar, clarity, and conciseness for wrrtten content. The LLM was not used for research ideation, experimental design, data analysis, or the formulation of our conclusions.
Our workflow for using the LLM followed strict three-step, human-in-the-loop process:
1. Polish:
We used the model to suggest alternative phrasing or grammatical corrections for existing text drafted by the authors.
2. Review: All suggestions provided by the LLM were critically reviewed by the authors to
verify their accuracy and to ensure they did not alter the original scientific meaning or intent.
3. Manual Revision: We manually integrated and modified any useful suggestions to ensure the final text accurately and precisely reflected our findings and narrative.
34

---

Published as a conference paper at ICLR 2026
The authors take full responsibility for all content presented in this paper, including any text that was revised with the assistance of the LLM.
35


Reviews:
[{"review_id": "wz5O1P7lBk", "reviewer": "Reviewer_YZgc", "summary": "This paper introduces ChronoPlay, a framework for constructing dynamic retrieval-augmented generation (RAG) benchmarks in gaming domains. It models two evolving dimensions: knowledge evolution (continuous updates to game rules and content) and user interest drift (shifts in player community focus over time). The framework integrates an authoritative knowledge base, community-derived question templates, and synthetic player personas to generate realistic, time-evolving question–answer pairs.\n \nThe authors test various retrievers and generators on datasets created with ChronoPlay for three games — Dying Light 2, Dune: Awakening, and PUBG Mobile — and provide detailed ablation studies.", "questions": "1. How are vector databases indexed or re-indexed after each phase update?\n2. Could ChronoPlay be extended to simulate user-specific histories or personalized question evolution, beyond aggregate community changes?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. ChronoPlay is the first framework to incorporate dynamic, evolving environments into RAG evaluation for gaming. By formalizing knowledge evolution and user interest drift, it highlights two important factors often overlooked in static RAG benchmarks.\n\n2. The experiments are extensive, covering several retrievers and generators across three games and including ablations on knowledge and interest dynamics.\n\n3. The proposed datasets and methodology could help the community study how RAG systems behave under temporal and contextual changes.", "weaknesses": "1. While the paper includes a diverse set of retrievers and generators, the RAG experiments themselves are static. It's unclear whether the RAG system re-index as the benchmark evolves. There is no adaptive RAG system (e.g. with finetuning, memory) discussed. It's unclear how the proposed dynamic benchmark would challenge or benefit adaptive RAG systems in practice.\n\n2. The paper also omits details about how the vector databases are indexed or re-indexed after each phase. It is unclear whether embeddings are refreshed, incrementally updated, or kept static.\n\n3. ChronoPlay models a population-level community through generated player personas, but it doesn’t capture personalized or history-dependent questions. I don’t see this as a major weakness, but it does make me wonder how this limitation relates to the idea of dynamic RAG. What would a RAG system need to do to account for personalized shifts in interest over time?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HX2RGGauj0", "reviewer": "Reviewer_bq56", "summary": "This paper introduces a framework for automatically generating dynamic RAG benchmarks for the gaming domain. The key innovation is addressing \"Dual Dynamics\"—the simultaneous evolution of game content and player community interests. The framework combines a dual-source synthesis engine that draws from both authoritative game wikis/patch notes and player community discussions to ensure factual correctness and authentic query patterns. The authors instantiate the framework  on three games spanning different timescales and characteristics. Evaluation of various retrieval models (BM25, BGE-M3, Qwen3-Embedding, text-embedding-3) and generator models (GPT-4o, Claude, Gemini, etc.) reveals significant performance fluctuations across game lifecycle phases, demonstrating that both knowledge updates and interest drift independently contribute to benchmark volatility.", "questions": "While the authors demonstrate that Dual Dynamics are a major driver of performance volatility, the analysis could be strengthened by de-confounding topic shifts from inherent question difficulty. For instance, analyzing the distribution of question types (e.g., single-hop vs. multi-hop) across phases could provide a more direct measure of difficulty.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The concept of \"Dual Dynamics\" is a powerful contribution. The paper addresses a limitation of existing dynamic benchmarks and provides a more realistic evaluation paradigm. \n- The framework this paper proposes is well-designed.. Human expert evaluation and validation of LLM-as-judge against human experts provide credibility. It is also instantiated on multiple games which shows diversity.", "weaknesses": "-  The ChronoPlay pipeline involves multiple LLM-driven stages and several hyperparameters (λ_JSD=0.001, γ=1.5, varying window sizes W). This complexity might pose barriers to easy adoption.\n- The statistical rigor of the evaluation could be strengthened. The paper lacks confidence intervals or significance tests for performance differences. Are the fluctuations statistically significant?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UmUsQolIDN", "reviewer": "Reviewer_2Erh", "summary": "This paper focus on the Retrieval Augmented Generation (RAG) problem in dynamic domains and build a RAG benchmark in the domain of online gaming. The authors propose ChronoPlay, a novel framework for the automated and continuous generation of game RAG benchmarks. Specially, this work introduce dual-dynamic update mechanism that responds to changes in both the game’s knowledge and the player’s interests. The experiments across three games demonstrate that RAG system performance is highly volatile over a game’s lifecycle.", "questions": "Miss related works on other RAG evaluation frameworks, for example, RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation by Ru et al., 2024.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper studies the RAG problems in dynamic domains and build a benchmark in this area with an automated and continuous generation framework. The research topic is interesting and valuable. \n\n2. The data sourced from real games and player communities making the benchmark more realistic.", "weaknesses": "1. Many real-world applications require dynamic RAG systems, including online shopping (where prices and promotional campaigns constantly change) and travel planning (where weather conditions and seasonal attractions vary). However, the benchmark's coverage of only three games limits its scope and makes the data domain insufficiently diverse.\n\n2. The generation performance in this work is evaluated using LLM-as-Judge, as detailed in Appendix C. However, the meta-evaluation results (Section C.3) are concerning. With accuracy ranging from 70% to 78% and F1-scores from 78% to 86%, the reliability of the evaluation methodology itself is questionable, which undermines confidence in the reported results.\n\n3. Additional experiments on end-to-end RAG systems would provide more comprehensive insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)