Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on
  Diverse Video Games
Authors: name: Dongmin Park; name: Minkyu Kim; name: Beongjun Choi; name: Junhyuck Kim; name: Keon Lee; name: Jonghyun Lee; name: Inkyu Park; name: Byeong-Uk Lee; name: Jaeyoung Hwang; name: Jaewoo Ahn; name: Ameya S. Mahabaleshwarkar; name: Bilal Kartal; name: Pritam Biswas; name: Yoshi Suhara; name: Kangwook Lee; name: Jaewoong Cho
Published: 2025-06-04 06:40:33

Abstract: Large Language Model (LLM) agents are reshaping the game industry,
particularly with more intelligent and human-preferable game characters.
However, existing game benchmarks fall short of practical needs: they lack
evaluations of diverse LLM capabilities across various game genres, studies of
agentic modules crucial for complex gameplay, and fine-tuning datasets for
aligning pre-trained LLMs into gaming agents. To fill these gaps, we present
\benchname{}, a foundational benchmark designed to train and evaluate
LLM agents across diverse real-world video games. Unlike existing benchmarks,
Orak includes 12 popular video games spanning all major genres, enabling
comprehensive studies of LLM capabilities and agentic modules essential for
intricate game scenarios. To support consistent evaluation of LLMs, we
introduce a plug-and-play interface based on Model Context Protocol (MCP) that
enables LLMs to seamlessly connect with games and manipulate agentic modules.
Additionally, we propose a fine-tuning dataset, consisting of LLM gameplay
trajectories across diverse game genres. Orak offers a comprehensive evaluation
framework, encompassing general game score leaderboards, LLM battle arenas, and
in-depth analyses of visual input state, agentic strategies, and fine-tuning
effects, establishing a foundation towards building generic gaming agents. Code
is available at https://github.com/krafton-ai/Orak.