ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in
  Game RAG Benchmarks
Authors: name: Liyang He; name: Yuren Zhang; name: Ziwei Zhu; name: Zhenghui Li; name: Shiwei Tong
Published: 2025-10-21 09:28:13

Abstract: Retrieval Augmented Generation (RAG) systems are increasingly vital in
dynamic domains like online gaming, yet the lack of a dedicated benchmark has
impeded standardized evaluation in this area. The core difficulty lies in Dual
Dynamics: the constant interplay between game content updates and the shifting
focus of the player community. Furthermore, the necessity of automating such a
benchmark introduces a critical requirement for player-centric authenticity to
ensure generated questions are realistic. To address this integrated challenge,
we introduce ChronoPlay, a novel framework for the automated and continuous
generation of game RAG benchmarks. ChronoPlay utilizes a dual-dynamic update
mechanism to track both forms of change, and a dual-source synthesis engine
that draws from official sources and player community to ensure both factual
correctness and authentic query patterns. We instantiate our framework on three
distinct games to create the first dynamic RAG benchmark for the gaming domain,
offering new insights into model performance under these complex and realistic
conditions. Code is avaliable at: https://github.com/hly1998/ChronoPlay.