KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation
Authors: Jiajun Shi
Jian Yang
Jiaheng Liu
Xingyuan Bu
Jiangjie Chen
Junting Zhou
Kaijing Ma
Zhoufutu Wen
Bingli Wang
Yancheng He
Liang Song
Hualei Zhu
Shilong Li
Xingjian Wang
Wei Zhang
Ruibin Yuan
Yifan Yao
Wenjun Yang
Yunli Wang
Siyuan Fang
Siyu Yuan
Qianyu He
Xiangru Tang
Yingshui Tan
Wangchunshu Zhou
Zhaoxiang Zhang
Zhoujun Li
Wenhao Huang
Ge Zhang
Venue: NeurIPS 2025 spotlight ()
Keywords: LLM; Evaluation; RL; Game
Primary area: deep_learning
TL;DR: We propose KORGym, a dynamic, game‐based benchmark offering over 50+ interactive tasks with RL support for multi‐turn LLM reasoning evaluation, and validate its effectiveness through extensive experiments, revealing several key insights.

Abstract: Recent advancements in large language models (LLMs) underscore the need for more comprehensive evaluation methods to accurately assess their reasoning capabilities. Existing benchmarks are often domain-specific and thus cannot fully capture an LLM’s general reasoning potential. To address this limitation, we introduce the **Knowledge Orthogonal Reasoning Gymnasium (KORGym)**, a dynamic evaluation platform inspired by KOR-Bench and Gymnasium. KORGym offers over fifty games in either textual or visual formats and supports interactive, multi-turn assessments with reinforcement learning scenarios. Using KORGym, we conduct extensive experiments on 19 LLMs and 8 VLMs, revealing consistent reasoning patterns within model families and demonstrating the superior performance of closed-source models. Further analysis examines the effects of modality, reasoning strategies, reinforcement learning techniques, and response length on model performance. We expect KORGym to become a valuable resource for advancing LLM reasoning research and developing evaluation methodologies suited to complex, interactive environments.

KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation
Jiajun Shi Jaan Yangl.2 ,Jiaheng Liu Xingyuan Bu', Jiangjie Chen',
Junting Zhou? Kaijing Ma Zhoufutu Wen Bingli Wang", Yancheng He? Liang Song². Hualei Zhu?, Shilong LI,, Xingjian Wang", Wei Zhang'. Ruibin Yuan?. Yifan Yao?, Wenjun Yang'. Yunli Wang". Siyuan Fang', Siyu Yuan", Qianyu He',
Xiangru Tang", Yingshui Tan Wangchunshu Zhou, Zhaoxiang Zhang'. Zhoujun Li!
Wenhao Huang* 7t, Ge Zhang
SKLCCSE, Beihang University MM-A-P 'ByteDance Seed *Nanjing University 'CASIA
Equally contributed authors *Corresponding author
Abstract
Recent advancements large language models (LLMs) underscore the need for more comprehensive evaluation methods to accurately assess their reasoning ca- pabilities. Existing benchmarks are often domain-specific and thus cannot fully capture LLM'' general reasoning potential. To address this limitation, we introduce the Knowledge Orthogonal Reasoning Gymnasium (KORGym)" dynamic evaluation platform inspired by KOR-Bench M and Gymnasium [2]. KORGym offers over fifty games in either textual or visual formats and supports interactive, multi-turn assessments with reinforcement learning scenarios. Using KORGym, we conduct extensive experiments on 19 LLMs and 8 VLMs, revealing consistent reasoning patterns within model families and demonstrating the superior performance of closed- source models. Further analysis examines the effects of modality, reasoning strategies, reinforcement learning techniques, and response length on model performance. We expect KORGym to become valuable resource for advancing L.LM reasoning research and developing evaluation methodologies suited to complex, interactive environments.
Introduction
Recent advances in reasoning models have yielded strong performance in tasks such as textual comprehension 3 and logical inference [4. However, most benchmarks remain domain-specific (e.g., AIME [5], PHYBench (6)) and fail to capture general reasoning ability. Even benchmarks intended to evaluate broader reasoning (e.g.. SuperGPQA 77. HLE [8] are heavily influenced by pretraining data, limiting their capacity to measure intrinsic reasoning skills. To address this gap, we propose benchmark designed to evaluate the intrinsic reasoning capabilities of LLMs independent of pretraining knowledge. Games, with their diverse scenarios rarely encountered in pretraining corpora, offer ideal testbed for such evaluation.
While games offer a promising benchmark medium, existing approaches exhibit several shortcomings. LogicGame [9). for example, employs only single-turn scenarios, preventing evaluation of long-term planning in L.Mss. TextArena [10 and SPINBench [I] support multi-turn scenarios but introduce opponent dynamics that generate extraneous variability, confounding pure reasoning assessment and limiting suitability for reinforcement learning (RL) by enabling hacked strategies. Moreover,
OOr codebase and experimental mul milttimdal-arte wtt.-aatproooeeiooooooryy.
results are
available at:
https://github. com/
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

gg-bench [12] relies heavily on generative capacity and lacks robustness in both gameplay fidelity and RL integration.
To overcome these limitations, in Figure [ we introduce the Knowledge Orthogonal Reasoning Gymnasium (KORGym), inspired by the knowledge-orthognn reasoning framework of KOR-
Bench [ (see Appendix
C and built on the eeinforeement-lcariing environment Gymnasium 2
Specifically, KORGym features over fifty games span- ning six reasoning dimensions: mathematical and log- ical reasoning, control interaction reasoning, puzzle reasoning, spatial and geometric reasoning, strategic reasoning, and multimodal reasoning. The platform is organized into four modular tomponents-the in- ference module, game interaction module, evaluation module, and communication module- module-enabling multi- round evaluations, contigurable difficulty levels, and stable reinforcement-learingg support.
Single-epode Game Bench
R
cage chickere robbits. There are
and 94 legal How chickens and how rubbits?
23 chickens, 12 robbits
8
Correct!
KORGym
6ame Configuration
By integrating textual and multimodal challenges, KO- RGym provides a comprehensive assessment of LLMs' adaptability, strategic planning, and decision-male capabilities, thereby offering a more accurate reflection of their intrinsic reasoning abilities. Using KORGym, we conduct extensive experiments, yielding several key insights:
Game Meme:MMineSweppp Seed 1-20 Level
Successfully establish Swespor!
You need to uncover sefe cells and flog hidden mines on board using clues, aiming win the game by flagging mines. Curnent game board.
?
?
? F221
12210
01110
Reasoning abilities display consistent profiles of strengths and weaknesses within the same model series.
My decision is to Uncover (1.1)
Game Over! Final Score:0 46
Modality influences reasoning performance, with distinct patterns observed between open-source and closed-source LLMss
Figure 1: Comparison Between Traditional
Thinking models exhibit behavioral patterns dis- Single-epoch Game Benchmark and KOR- tinct from those of non-thinking models.
Gym.
LLMs often employ explicit reasoning paradigms during problem-solving, which may partially constrain their performance.
Appropriate reinforcement learning enhances reasoning capabilities and yields more balanced performance across different reasoning dimensions.
In summary, our main contributions are follows:
We design a suite of over fifty text- and vision-based games tailored to evaluate the reasoning capabilities of large language models.
We present KORGym, extensible framework supporting incremental development and eeinforeeentt-eeaming integration.
We conduct comprehensive empirical analysis of 19 LLMs and 8 vision-language models and uncover several key insights.
2 Related Work
LLMs for Gaming. Games serve as valuable testbeds for evaluating large language models (LLMs) due to their demands for multi-step reasoning and strategic planning. Early research focused on single-game evaluations in domains like Minecraft [13] or social deduction games [14 [5]. but these narrow settings limited generalizability. Subsequent efforts introduced broader benchmarks with diverse game types and multi-agent frameworks emphasizing coordination or competition, though critical dimensions such open-ended negotiation, dynamic cooperatiionconffic shifts, and rich social dynamics remained underexplored. Some researchers have designed PlanBench [T6] to evaluate the long-term reasoning ability of LLMs and have further conducted studies [7 on reasoning models

---

based on this work. Meanwhile, other researchers have proposed the Game Traversal Benchmark [8 assess LLMs' planning and reasoning ability through the task of traversing 2D game maps. To address these gaps, SPIN-Bench [I] unifies strategic planning and social intelligence by combining formal planning analysis, multi-agent conperation/compettionn and open-ended dialogue. Existing benchmarks vary widely in environment diversity and technical capabilities. Some frameworks offer diverse environments but lack human evaluation, while others focus on specific scenarios yet miss key features. SPIN-Bench stands out with balanced mix of game types, Gym compatibility, and model vs, model evaluation.
Knowledge Orthogonality Based Evaluation. Current Al reasoning benchmarks (e.g.. MMLU [19]. CommonsenseQA (2).. MATH [21]) emphasize factual recall and problem-solving but often conflate memorization with reasoning, limiting insight into underlying cognitive processes. To address this, integration-basee benchmarks (c.g.. ZebraLogic [22]. TravelPlanner (23) test adapt- ability and creativity by requiring pattern recognition, logic, and multi-step reasoning in novel contexts. While these frameworks advance the focus contextual problem-solving, they still risk entanglement with domain-specific knowledge biases, as seen mathematical or logical benchmarks like GSM8K [24] and FOLIO [25]. To address these gaps, the concept of knowledge orthogonality advocates decoupling reasoning assessment from prior knowledge and prioritizing fule-following in cut-of-distributte scenarios to isolate core abilities such systematic generalization and hypothesis testing. This paradigm shift-from memorization-drive fairer framework for measuring cognitive agility, ensuring models demonstrate genuine reasoning rather than reciting learned patterns and fostering Al systems with robust, human-likk adaptability in open-world environments.
metrics to knowledgg-agnoostic. creativity-
focused vvaluations-estable
Approach
3.1
Framework
Gome Interaction Module
Inference Module
asynchronous inferences
reasoning checkpoints
generate
@rint_boardd
verify
API
5. Game information
6.LLM action
4, Game information
7.LLM action
Evaluation Module
Communication Module
I
2.Parometer parsing and
3. Gome environment initialization
communication protocol startup
Parsed Parometres
Game started:
Maze
Process ID: 6836
Type:
Single Epoch
URL:hhtp://llcalhesee
Game Request URL:
https//...
Parometer initialization
8. Score Calculation
Initial Parameters
Evaluation Result
Game Name: Maze
Game Name:
Maze
Model:
GPT-4e
Average
0.56
Seed:
1-50
Model:
GPT-4o
Deploy Port Number:
5000
Game Playing Details: result.
Output Dir:
root/...
Figure 2: Framework of the KORGym system. Our system architecture primarily consists of four modules: the Inference Module, Game Interaction Module, Evaluation Module, and Communication Module. The initialization parameters include: Game Name, Model Information, Seed, Deployment Port Number, and Output Directory.

---

We propose KORGym, an efficient game-based framework for evaluating the complex reasoning capabilities of large language models (LLMs) via both single-turn and multi-turn text-based and multimodal multimodal games. KORGym is organized into three key modules:
Evaluation and Communication Module: the system core, which parses input parameters, establishes inter-module communication protocols, encapsulates and transmits communication packets, and logs final evaluation scores.
Game Interaction Module: encapsulates the game environment and interaction APIs, including:
generate: initializes the game environment.
print board: renders the game board and generates prompts.
verify: updates the game state and computes scores.
Inference Module: manages model inference processes, including asynchronous acceleration and intermediate result checkpointing.
Based on these modules, the primary inference workflow of KORGym proceeds as follows:
Parameter Initialization: Load initial parameters (e.g.. game type, seed, and model).
Parameter Parsing and Communication Protocol Startup: Parse these parameters and encap- sulate them into communication packets.
Game Environment Initialization: Generate the game environment according to the parameters.
Acquisition of Game Information: Invoke the generate and print board APIs in the Game Interaction Module to obtain the current environment state.
Game Information Transmission: Package the retrieved information for transmission.
Generation of LLM Action: Perform model inference to generate the next action.
Transmission of LLM Action: Send the action via the verify API update the game state; the game is not concluded, return to Acquisition of Game Information.
Score Calculation and Result Output: Compute and output the final score.
3.2
Task Introduction
As illustrated Figure 9 (Appendix(H) H.. KORGym supports over fifty novel games, enabling precise and efficient evaluation of the reasoning abilities of large language models (LLMs) across six distinct dimensions (Table TTp: Mathematical and Logical Reasoning. Control Interaction Reasoning. Puzzle Reasoning, Spatial and Geometric Reasoning, Strategic Reasoning and Multimodal Reasoning.
During benchmark development, we selected more than fifty games that effectively capture the reasoning capabilities of LLMs. These games span four categories: traditional puzzles (e.g., Su- doku); adaptations of classic video games (c.g., Plants vs. Zombies; Minesweeper); game-theoretic challenges (e.g., N-point; Evolution of Trust); and multimodal tasks (e.g., Jigsaw; Circle the Cat).
KORGym offers a suite of over fifty games-an with continuous expansion that support multi-turn interactions via standardized APIs (generate, verify, and print board). The platform is tailored for RL, providing environment states and reward signals, and enables users to adjust game difficulty and environmental diversity through scalable parameters. Additionally, it includes nine multimodal games, facilitating comprehensive evaluation in both textual and multimodal contexts. Related platforms include LogicGame 9 AgentBench [26). GameArena [27). SPIN-Bench [28), TEXTARENA (10),
and ReasoningGYM 229]. A detailed comparison appears in Table of Appendix
3.3 Evaluation Method
Score Calculation Rules
To address the limitations of binary (0/1) scoring in reflecting intermediate progress in KORGym, we propose comprehensive scoring scheme comprising three rules:
Binary Scoring: For single-objective games, assign point for success and 0 for failure. For example, in 7-Maze, reaching the exit yields score of 1.

---

For multiple-choiee games, the score equals the number of correct responses divided by the total number of options. For instance, in the 44-Jigsaw Puzzle, the score is the number of correctly placed pieces over the total pieces.
Proportional Scoring:
Cumulative Scoring: For games that award incremental points, accumulate all points earned. For example, in 3-2048, each tile merge contributes to the final score.
Capability Dimension Aggregated Mean Raw game scores KORGym can extend beyond the [0,1] interval and may be skewed by variations in game difficulty or by outlier model behaviors. To mitigate these issues, we introduce Capability Dimension Aggregated Mean, a more robust aggregation metric for evaluating model performance across reasoning dimensions.
Formally, let = {91, .2.....99. IN' } denote the set of all games, M {m1. m2: m } denote the set of models under evaluation, and D = {d1, de,... ......d, dy reepeeset the set of reasoning capability dimensions. Each game g E G is associated with a specific dimension d(g) D. Let Sg. denote the raw score achieved by model m E M game across all models exceeds 1, ie., maxmEM Sg. > 1,we apply logp transformation (i.e., In(1 a(1++)) to compress large score values and reduce skewness; otherwise, we retain the original score:
G. For each game g. if the maximum score
[nn(1 + Sgm,), if max Sg. > mEM
Sy.m
(1)
Sgmm.
otherwise.
To normalize scores across games, further define, for each game
0g min S¹ g.m
bg max 9.m
(2)
If 6, Gg. meaning all models perform identically on game we assign every model normalized score of 0.5 avoid division by zero. Otherwise, we normalize its adjusted score:
S - ag by ag
5gm
Vm E M.
(3)
This normalization ensures that for each game, model performances while preserving relative differences. Subsequently. for each capability dimension d E D, we define the corresponding set of games and aggregated score of model m on dimension d
mapped into the '0, 1] range
Ga tu E G d(g)) = 3,, G
S g.m-
gEGa
The resulting matrix (5 d. ) DE meM provides a normalized, dimension-wise evaluation reason- ing capabilities that is fair across heterogeneous games.
4
Experiments
4.1
Settings
LLMs To comprehensively evaluate LLM performance, we assessed 19 large language mod- els including 11 thinking models and instruction-tunel models 8 vision-language models
(Table
Evaluation Setting During evaluation, we apply distinct protocols for single-epoch and multiple-
epoch games:
Single-epoch Games: Each model is evaluated on 50 independently initialized game instances by varying the "seed" parameter the "generate" API from to 50.
Multiple-epoch Games: For each model, we initialize 20 game environments. Each episode permits up to 100 interaction rounds, and we vary the "seed" parameter in the "generate" API from 50 for reproducibility.
All assessments use zero-shot prompting setup gauge genuine reasoning capabilities, retaining each model's default sampling parameters (temperature and top-p). evaluate closed-source models via their hosted APIs and open-source models on eight NVIDIA A100-80G GPUs.
5

---

Table 1: Overall performances of different models on KORGym. Model capability dimensions include Mathematical and Logical Reasoning (MLR), Control Interaction Reasoning (CIR), Puzzle
Reasoning (PR), Spatial and Geometric Reasoning (SGR) and Strategic (SasoninggRR).
Model
MLR(%)
CIR(S)
PR(%)
SGR(%)
SR(%)
Avg.(%)
O3-mini
77
76
82
Gemm-ni2255pp--002255
63
94
84
79
01-2024-12-17
74
83
79
73
Doubao-1-5-8n --stto----ti
63
74
72
65
72
DeepSeek-RI
66
82
69
56
83
71
Blaude3.7-thinkinng
50
93
$2
53
64
62
Qwen3-32B-thinking
58
55
58
55
71
60
DeepSeek-v3-0722
35
55
27
26
69
42
DeepSeek-R I-Diitilll --iii-i----il---ee-222 Qwen-32B
45
28
35
33
56
39
Gemini-2.0 2.0---.00--ahtiee Flash-thinking
25
53
34
18
58
38
Claude-3.7
25
55
26
17
50
35
Qwen-QwQ
37
39
14
18
33
28
Gemini-2.0-Fii
24
28
17
12
51
26
GPT-4o
12
25
8
11
53
22
Doubao- 1.5-pro
18
16
16
44
20
Swve22.5-725 Instruct
18
10
4
49
18
Owwen.55322B-11 Instructt
13
7
4
46
16
DeepSeek-RI. BitSeekk-R-Diisee RI-Distill L1-Distill-Qwen-- Qwen-7B
10
2
33
11
Owen2.5-7B-IIsttruu
7
29
8
Table 2: Multimodal reasoning abilities of different models on KORGym.
Crosword
Jigsan Pazzla())
Find The Pattera(%)
Circle The
Map Simulation (9)
Sokoban
Bubble Ball Sortiag(5)
Wondle
Square Addiion(S)
Gamini-200-Fuua
Qwen2.5VL 720
10.9
Qwee2.5VL SVL-320
3.5
28
Que-2.5VL-78
U
0
IntoreV1.3-788
11.8
10
38
0
4.2 Main Results
Table reports the performance of LLMs and VLMs on KORGym using the Capability Dimension Aggregated Mean (Section[4.1) Table pressenss VLM performance multimodal tasks and detailed raw scores appear Appendix [ The leaderboard will be updated on GitHub after submission. Across 51 games and six reasoning dimensions, the normalized scores yield the following insights.
Interaction Reasoning
Inter action Beasoning
Reacsning
Beasoning
Douboc: ththiiingg
Reasoning
Reasoning
DeesSeeic Bestll--wee-a 328
Deepbesk-81
Deep/eesc- -e-ee-----ee--0
3.udoe3.7trrrtit kninking
Qwen2.5 5 728 instruct
05 min
Owen22 5 328 instruct
01-2024412-37
Queri2 5-75 5--5ttsiuutt
niz. taashihhnking
Owen-Owa
Spatial and Gesmetiic Ressoning
Mutherattcal and Logical Reasoning
Geometric Reasoning
Logical Reasoning
(a) Model Ability Analysis
(b) Model Scale and Architecture Analysis
Figure 3: Capability Dimension Illustration. Figure (a) showcases the performance of the top- performing models KORGym. Figure (b) showcases the impact of model scale and architecture on reasoning capabilities.

---

Similar Strength-Weaknees Profiles Within Same Model Series Figure 3a shows that O1 and O3-mini excel in spatial reasoning, whereas the Gemini series leads in mathematical and puzzle reasoning.
Closed-Soures Models Demonstrate Superior Reasoning Performance O3-mini achieves the highest overall score on KORGym, particularly in spatial reasoning. Claude-3.7-thinking and Gemini- 2.5-pro top puzzle reasoning, while Doubao-1.55 1.5-o-n..tiiiiiiig--- and DeepSeek-RI deliver balanced performance across dimensions. In contrast, open-source models lag behind.
Impact of Model Scale and Architecture on Reasoning Capabilities Figure 3b demonstrates that model performance scales positively with model size and thinking models outperform size- matched non-thinking variants. For instance, D-sbsee-RR--ssssi---ee--..- though smaller in scale, exceeds the performance Jowen2.5-72B-Instree
5
Discussion
5.1 RQ1: Does Modality Affect Reasoning Performance?
As shown in Figure Figure compares compares the performance closed-source and open-source VLLMs on textual and visual versions of six representative games. Detailed results provided in Figure of Appendix []] Our key findings
Textual-version Game vs. Multimodal-version Game.
Average scores textual versions consistently exceed those visual versions.
Open-source VLMs perform better on text-based reasoning than on visual-based tasks, indicating limited visual grounding underdeveloped multimodal alignment.
Some closed-source VLMs score higher on visual versions than on textual versions, suggest- ing stronger visual reasoning superior multimodal integration.
In mathematiis-rellteed games, models score significantly higher on textual versions, high- lighting the advantage of symbolic representation for numerical reasoning.
5.2 RQ2: Do Different Model Series Exhibit Consistent Behavioral Patterns?
Based on the experimental scores, we com- puted the mean and standard deviation of each dimension's scores (Figure Ba) and per-
Vituall Scaress Each Model
Butbe Sartin
Text Visual
formed principal component analysis (PCA) on the score matrix $ E RMx where M and G denote the numbers of models and games, respectively (Figure(5b) These anal- yses reveal dominant patterns in the models' reasoning bchavior across five dimensions.
0.0
Saketan
Arg
Sauere Wistin
Arg Visual
Word
0.0
n2 3-10 1-1
Gamie2
Gumiei2 5-9ro
Top-Tier Models Exhibit Homogeneous
Behavioral Profiles In PCA space, top- Figure 4: Performance Comparison Between Textual tier models (c.g., 01 and O3-mini) form and Multimodal Game Versions. This figure illustrates given model's performance on both the textual and multimodal versions of the same game. Different games sions. By contrast, secondary models (e.g., are represented by distinct bar colors, and bar shading Claude-3.7-thinkee and Qwen3) display im- balanced performance across reasoning di- mensions.
tight clusters, indicating consistently strong reasoning performance across all dimen-
differentiates text (unshaded) from visual (shaded) ver- sions. Solid and dashed lines correspond to the average textual and visual scores, respectively.
Distinct Behavioral Patterns Between Thinking and Non-Thinking Models PCA reveals that the first two clusters consist exclusively of thinking models, whereas the fourth cluster comprises almost solely non-thinking models.
7

---

Scatter Plet of Model Standarddewialin Average Score
<GPl-4p
PCA Prolection Nosel
0.18
Clsttrr
Gemini2. 5-
Qwen2.5
"Caudee337
Gemini Hommm-2---Taaaeeeee D-Tiasheeiiiiinng
Santartdevistion
0.16-
2ommn-2.00-aap
Gomint Rishh
pm
GPT-t
Dounard.--ype PoosScck v3 0324
DespS
ReepSee--L RL Disil -owen---
Osubmn-15-this
DeesSeec-1
0.03
Qwan2 328 Instruc:
T0aeri:--72B-intttr SespS Retssseet-L--iiii diatil Qwann-22
Owen/qwo
0.10
Owen3-328
01:2026-12-37
Towen 5-1B-tnsirat QwQ
0.08
03-mm:
01-2024-1--17
Owir3 328 thinking
-0.7
03mhi
0.08
0.1
0.2
0.3
0.4
0.5
9.6
0.7
6.8
Mean Score
PCI (715% vert
(a) Stability Analysis of Model Reasoning Capabilities
(b) PCA Projection Models
Figure 5: Analysis of Model Behavioral Characteristics. (a) illustrates the relationship between model stability and reasoning performance, whereas (b) depicts the clustering models based on behavioral traits.
LLMs tend to adopt explicit reasoning paradigms when performing analysis and problem- solving Response-leve case studies reveal that LLMs in KORGym employ four primary reasoning paradigms, each reflecting a distinct cognitive strategy:
Code Paradigm: generating executable code to obtain solutions (e.g., "import math; a = O; for in range(...): ...").
Mathematical Paradigm: applying algebraic equations or arithmetic rules to model and solve problems (e.g., "Let "Let x be the number of creature A and y the number of creature B; construct the system of "quations..."".
Algorithm-Speciie Paradigm: invoking established algorithms (e.g., Dijkstra's 's algorithm, Eulerian path) and adapting them to the task context (e.g., "Use an Eulerian path solve the one-stroke drawing puzzle: first compute...
Natural Language Reasoning Paradigm: conducting spatial, logical, or causal analysis in natural language (e.g.. "If we turn right, we reach (1,2) where springboard lies ahead ...").
To examine reasoning-paradign usage, we employ GPT-4o to annotate model responses for selected KORGym games and compute the mean score for each paradigm (Table(). We then conduct ablation experiments by constraining prompts to disable individual paradigms, with results summarized in Figure 6 detailed information in Appendix Our key findings include:
Table 3: Reasoning Paradigm Proportions and Average Scores for Different Models' Responses
Modell
Propertion (16)
Average Score
Overall Seare(S)
Code
Math
Algocitim
Natural Language
Muthematical
Algorithm
Natural Laaguage
1-5-hinnking--pr
52.6
Douhne- 1.proo
17.1
30
DeepSerk-v-000 v3-0334
0
9.7
90.3
25
22
22.388
Chaate 3.7 3.7-hinking
0.3
13.4
26.3
60
64
37
46
45.907
Gemini-2.5 p00-332-55
46
6.9
12
35.1
83
100
66
63.893
GPT-4o
29.7
67.1
1.936
03-mini
77.56
Models within the same series exhibit distinct preferences Models tend to adopt paradigms aligned with their architec- ture: Gemini-2.5-PP predominantly employs code-based reasoning; 03-mini primarily uti- lizes mathematical and natural language reason- ing; and Doubao-L 1ouuaa-1.5..o 1.5- .5-thinking- pro strongly favors algorithm-specific reasoning strategies.
wooe of no Eart Fxndign Model Berfarmnne
Cramp Bagsermeeng
Doatee Algar
Clsatle Code
Deaate Hith
-20-
Reasoning Paradigms Partially Constrain
140
Serbee & Comaanse sutikking
1o you Seseeeeeeeaa 094322
Model Performance Disabling specifie reason- Figure 6: Impact of Disabling Each Paradigm on ing paradigms via prompt constraints led to in- Model Performance

---

creased average performance across all models. We attribute this improvement to overreliance on large-scale pretraining data rich in mathematical, coding, and algorithmic examples, which can impede generalization and adaptability to novel reason- ing tasks. These findings underscore KORGym's value as a robust benchmark for evaluating genuine reasoning abilities beyond memorized patterns.
Disabling the mathe- Mathematical Reasoning Core Component of the Reasoning Process matical paradigm causes most models to experience a performance decline or exhibit no improvement, indicating that mathematical reasoning is critical to LLM reasoning capabilities.
Stronger Models Exhibit Greater Robustness More capable models are less impacted by disabling individual reasoning paradigms. For example, O3-mini and Doubao- 1obbao-..-ttinkiig 1.5-thinking- pro maintain near- original performance when deprived of their preferred reasoning strategies, demonstrating superior robustness, generalization, and adaptive reasoning under constraint.
5.3 RQ3: What is the Impact of Reinforcement Learning (RL) on Problem Solving Capabilities?
During multi-turn reinforcementtleamiin fine-tuning, Doubao-1 1.5.bba-..--hiniiigg---g- incorporated two specialized algorithmic frameworks-DAPP and VAPO-to address instability reasoning-orientedd model training. In parallel, it trained comprehensive corpus of STEM problems, code- related tasks, logical reasoning challenges, and non-reasoning examples. Additionally, RL training on classic games (e.g., 24-point, mazes, and Sudoku) yielded a marked improvement in its reasoning performance.
In KORGym, RL-driven enhancements yielded substantial gains across reason- ing dimensions. As shown in Table achieved mean score of 0.72 fourth over- all-and excelled in puzzle reasoning with a score of 0.84, surpassing both Ol and O3-mini. Notably, Doubao- 1.5- thinking- pro exhibits minimal perfor- mance degradation under ablation and demonstrates score variance compara- ble to leading models (e.g., O3-mini and 01-2024-12-17), These improve- Figure 7: Correlation Between Response Length and Rea- ments underscore that appropriate re- soning Performance inforcement learning fosters both hanced reasoning and more balanced performance.
Nean Scare Average Tideess
Doub un 1554 2f pikingpoo
legend All Modzls Power-law fit
floudo
Quiin 3-8-tthinnnnng
Mean Score
0.5
Decugčok Bweplzaeeeett--llll---e---22222
Deep
Coude
.ommnii Flash
4p:-tto-2024-13-000
10cubap 1.5 cro
5020
Tilens
5.4 RQ4: There a Correlation Between Response Length and Reasoning Performance?
To examine the effect of response length on reasoning performance, record token counts and reasoning scores for four reasoning models and eight non-reasoning models during gameplay. We then fit curves to the aggregated data identify trends and correlations, as illustrated in Figure From the figure, we derive the following insights:
A strong positive correlation exists between reasoning performance and response length: models with longer responses tend to achieve higher KORGym scores.
Reasoning and non-reasoning models differ markedly in response length distributions: non-reasoning models produce responses within a narrow range, whereas reasoning models exhibit broader and more varied distribution.
The impact of response length on performance exhibits diminishing returns: response length increases, incremental score gains become marginal, suggesting an upper limit to the benefits of verbosity.

---

5.5 RQ5: Does KORGYM contain data leakage?
Many benchmarks inevitably overlap with model training data. To show that KORGYM effectively reduces the extent of data leakage, we propose the Knowledge Impact Coefficient to quantitatively assess the degree of leakage. For a game within KORGYM, MM, the required reasoning information comprises:
K: General background knowledge acquired during training.
T: Single-turn or multi-tum tasks that must be completed during gameplay.
R: Core game-rule information specifically designed for solving task T.
A: The set of actions generated by the LLM to solve task T.
+: The reasoning and interaction process from the game task T to action set A.
S: The final score obtained by the LLM's interaction with the game.
The leakage between K and R. is quantified by the Knowledge Impact Coefficient (3:
8 = 1=1- P(T + A R, K P(T A K)
A P K) (
8 E0 [0.
A smaller /3 indicates less overlap between the LLM's pre-trained knowledge and the game rules; consequently, the LLM cannot achieve optimal scores without explicitly relying on the rules provided in the prompt. Conversely, larger 3 indicates significant overlap, allowing the LLM to complete the game task and achieve optimal scores even without explicit rules.
Model
Snuke
Pipe Gume
Long Cal
Soloobun
Word Transf.
8-puzele
Play Lines
Black-White White Copy
Doubao 1-6-thinking
thinking-pp
0.055
0.085
0
0
0.176
0.166
Deup8oek-81
0.333
0.200
0.208
0.636
0.307
0.133
Quena 329 thinking
0,011
0.000
0.909
0,117
o3-nini
0.966
0.000
0.166
3.aude--.7-chikkii
0.41
0.312
0.461
Table 4: Knowledge Impact Coefficient [3 across selected KORGYM games.
Based on the above definition, we select subset of games from KORGYM, including classic games (Snake, Sokoban, and 8-puzzle) and original/adapted games (Pipe Game, Long Cat, We conduct an ablation study in which the core game-rule information (R) is removed, and then compute the Knowledge Impact Coefficient B. The results are shown Table
Model perspective: Models such as o3-mini and DeepSeek-R1 exhibit relatively notable knowledge leakage, whereas others (e.g., Doubao- 1 1esooo-----thinnn 1-5-thinking pro) demonstrate com- paratively lower leakage. This observation indirectly supports the view that reinforcement learning (RL) processes genuinely enhance a model's reasoning capabilities, rather than merely enabling memorize training questions or regurgitate pre-training knowledge.
Knowledge leakage is more prominent in classic games, while the corresponding /3 values for original/adapted games remain relatively low. Given that more than 80% of the games KORGYM are originally developed or adapted by our engincering team, this strongly supports the validity and reliability of KORGYM.
Game perspective:
6 Conclusion
In this paper, we introduce KORGym-a scalable, game-driven benchmark comprising over fifty tasks spanning six reasoning dimensions. KORGym supports multimodal interactions, reinforcement learning, and parameterized environments, and employs robust evaluation methodology based on dimension-awar score aggregation tailored to game-based reasoning. We evaluate 19 LLMs and 8 VLMs, revealing consistent strength- profiles within model series and demonstrating the impact of model scale and architecture reasoning capabilities. We also conduct ablation studies on modality, model series, and reinforcement learning, providing a detailed analysis of the key factors influencing LLM reasoning performance.
10

---

Acknowledgements
This work is supported by State Key Laboratory of Complex & Critical Software Environment (SKLCCSE). This work is also supported by the Fundamental Research Funds for the Central Universities.
References
[1] K. Ma, X. Du, Y. Wang, H. Zhang, Z. Wen, X. Qu, J. Yang. J. Liu, M. Liu, X. Yue, W. Huang, and G. Zhang, "KOR-Bench: Benchmarking Language Models on Ghowledge-Orthogond Reasoning Tasks," arXiv e-prints, P. arXiv:2410.06526, Oct. 2024.
[2] M. Towers, A. Kwiatkowski, J. Terry, U. Balis, G. De Cola, T. Deleu, M. Goulão, A. Kallinteris, M. Krimmel, A. KG, R. Perez-Vicente, A. Pierré, S. Schulhoff, J. Jet Tai, H. Tan, and O. G. Younis, "Gymnasium: A Standard Interface for Reinforcement Learning Environments," arXiv e-prints, p. mrXiv:2407.17032. July 2024.
[3] C. Ciao, S. X. Xu, K. Zhang, Y. Wang, and L. Xia, "Evaluating reading comprehension exercises generated by LLMs: showcase of ChatGPT in education applications." in Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023) (E. Kochmar, J. Burstein, A. Horbach, R. Laarmann-Quante N. Madnani, A. Tack, V. Yaneva,
Z. Yuan, and T. Zesch, eds.), (Toronto, Canada), pp. 610-625, Association for Computational Linguistics, July 2023.
[4] J. Shi, C. Wei, L. Yang. Z. M. Wang, C. Yang, G. Zhang, S. Huang, T. Peng. J. Yang, and Z. Wen, "Cryptox: Compositional reasoning evaluation of large language models," " 2025.
[5] B. Patel, S. Chakraborty, W. A. Suttle, M. Wang, A. Singh Bedi, and D. Manocha, "AIME: AI System Optimization via Multiple LLM Evaluators," arXiv e-prints, p. arXiv:2410.03111, Oct. 2024.
[6] S. Qiu, S. Guo, Z.-Y. Song, Y. Sun, Z. Cai, J. Wei, T. Luo, Y. Yin, H. Zhang, Y. Hu, C. Wang, C. Tang, H. Chang, Q Liu, Z. Zhou, T. Zhang, J. Zhang, 7. Liu, M. Li, Y. Zhang, B. Jing, X. Yin, Y. Ren, Z. Fu, W. Wang, X. Tian, A. Lv, L. Man, Li, F. Tao, Q. Sun, Z. Liang, Y. Mu,
Z. Li, J.-J. Zhang, S. Zhang, X. Li, X. Xia, J. Lin, Z. Shen, J. Chen, Q. Xiong, B. Wang, F. Wang,
Z. Ni, B. Zhang, F. Cui, C. Shao, Q.-.. Cao, M.-x. Luo, M. Zhang, and H. X. Zhu, "PHYBench: Holistic Evaluation of Physical Perception and Reasoning Large Language Models," arXiv e-prints, P. arXiv:2504. 16074, Apr. 2025.
[7] P Team, X. Du, Y. Yao, K. Ma. B. Wang, T. Zheng, K. Zhu, M. Liu, Y. Liang, X. Jin, Z. Wei, C. Zheng, K. Deng. S. Gavin, S. Jia, S. Jiang, Y. Liao, R. Li, Q. Li, S. Li, Y. Li, Y. Li, D. Ma. Y. Ni, H. Que, Q. Wang. Z. Wen, S. Wu, T. Hsing, M. Xu, Z. Yang, Z. Moore Wang. J. Zhou, Bai, X. Bu, C. Cai, L. Chen, Y. Chen, C. Cheng, T. Cheng, K. Ding, S. Huang. Y. Huang, Y. Li, Y. Li, Z. Li, T. Liang, C. Lin, H. Lin, Y. Ma. T. Pang, Z. Peng, Z. Peng. Q. Qi. S. Qiu, X. Qu, S. Quan, Y. Tan, Z. Wang, C. Wang, H. Wang, Y. Wang. Y. Wang. J. Xu, K. Yang, R. Yuan,
Y. Yue, T. Zhan, C. Zhang. J. Zhang, X. Zhang, X. Zhang, Y. Zhang, Y. Zhao, X. Zheng, C. Zhong, Y. Gao, Z. Li, D. Liu, Q. Liu, T. Liu, S. Ni, J. Peng. Y. Qin, W. Su, G. Wang, S. Wang, J. Yang, M. Yang, M. Cao, X. Yue, Z. Zhang, W. Zhou, J. Liu, Q. Lin, W. Huang, and G. Zhang, "SuperGPQA: Scaling L.MM Evaluation across 285 Graduate Disciplines," arXiv e-prints, p. 11X.:22502.14739 Feb. 2025.
[8] L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. Calvin Zhang, M. Shaaban, J. Ling, S. Shi, M. Choi, A. Agrawal, A. Chopra, A. Khoja, R. Kim, R. Ren, J. Hausenloy, O. Zhang, M. Mazeika, D. Dodonov, Nguyen, J. Lee, D. Anderson, M. Doroshenko, A. Cennyth Stokes, M. Mahmood, O. Pokutnyi, O. Iskra, J. P. Wang, J.-C. Levin, M. Kazakov, F. Feng, S. Y. Feng,
H. Zhao, M. Yu, V. Gangal, C. Zou, Wang, S. Popov, Gerbicz, G. Galgon, Schmitt, W. Yeadon, Y. Lee, S. Sauers, A. Sanchez, F. Giska, M. Roth, S. Ris,, S. Utpala, N. Burns, G. M. Goshu, M. Maheshbhai Naiya, C. Agu, Z. Giboney, A. Cheatom, F. Fournier-Facio, S.-J. Crowson, L. Finke, Z. Cheng. J. Zampese, R. G. Hoerr, M. Nandor, H. Park, T. Gehrunger,
J. Cai, B. McCarty, A. C. Garretson, E. Taylor, D. Sileo, Q. Ren, U. Qazi, L. Li, J. Nam, J. B. Wydallis, P. Arkhipov, J. W. Lun Shi, A. Bacho, C. G. Willcocks, H. Cao, S. Motwani, E. de
11

---

Oliveira Santos, J. Veith, E. Vendrow, D. Cojoc, K. Zenitani, J. Robinson, L. Tang, Y. Li, J. Vendrow, N. Wildner Fraga, V. Kuchkin, A. Pupasov Maksimov, P. Marion, D. Efremov, J. Lynch, K. Liang, A. Mikov, A. Gritsevskiy, J. Guillod, G. Demir, D. Martinez, B. Pageler, K. Zhou, S. Soori, O. Press, H. Tang, P. Rissone, S. R. Green, Brüssel, M. Twayana, A. Dieuleveut, J. Marvin Imperial, A. Prabhu, J. Yang. N. Crispino, A. Rao, D. Zvonkine, G. Loiseau, M. Kalinin, M. Lukas, C. Manolescu, N. Stambaugh, S. Mishra, T. Hogg, C. Bosio, B. P. Coppola, J. Salazar, J. Jin, R. Sayous. S. Ivanov, P. Schwaller, S. Senthilkuma, A. M. Bran, A. Algaba, K. Van den Houte, L. Van Der Sypt, B. Verbeken, D. Noever, A. Kopylov, B. Myklebust, B. Li, L. Schut, E. Zheltonozhskii, Q. Yuan, D. Lim, R. Stanley, T. Yang. Maar, J. Wykowski, M. Oller, A. Sahu, C. Giulio Ardito, Y. Hu, A. Ghislain Kemogne Kamdoum, A. Jin, T. Garcia Vilchis, Y. Zu. M. Lackner. J. Koppel, G. Sun, D. S. Antonenko, S. Chern, B. Zhao, P. Arsene, J. M. Cavanagh, D. Li, J. Shen, D. Crisostomi, W. Zhang, A. Dehghan, S. Ivanov, D. Perrella, N. Kaparov, A. Zang, I. Sucholutsky, A. Kharlamova, D. Orel, V. Porit- ski, S. Ben-David, Z. Berger, P. Whitfill, M. Foster, D. Munro, L. Ho, S. Sivarajan, D. Bar Hava, A. Kuchkin, D. Holmes, A. Modriguez-Roment F. Sommerhage, A. Zhang, R. Moat, K. Schneider, Z. Kazibwe D. Clarke, D. H. Kim, F. Meneguitti Dias. S. Fish, and V. Elser, "Humanity's Last Exam," arXiv e-prints, P. arXiv:2501. 142499 Jan. 2025.
[9] Gui, Y. Liu, Cheng, X. Gu, X. Liu, H. Wang, Y. Dong, Tang, and M. Huang, "LogioGame: Benchmarking Rule- Jule-Based Reasoning Abilities of Large Language Models," arXiv e-prints, p. arXiv:2408.15777 Aug. 2024.
[10] L. Guertler, B. Cheng, S. Yu, B. Liu, L. Choshen, and C. Tan, "TextArena"" arXiv e-prints. arXiv:2504. 1442, Apr. 2025.
[11] J. Yao, K. Wang. R. Hsieh, H. Zhou, T. Zou, Z. Cheng, Z. Wang, and P. Viswanath, "Spin-bench How well do lims plan strategically and reason socially?," arXiv preprint arXiv:2503.1234 2025.
[12] V. Verma, D. Huang, W. Chen, D. Klein, and N. Tomlin, "Measuring General Intelligence with Generated Games," arXiv e-prints, p. WYXv:25005.07215. May 2025.
[13] R. Gong, Q. Huang. X. Ma. Y. Noda, Z. Durante, Z. Zheng, D. Terzopoulos, L. Fei-Fei, J. Gao, and H. Vo, "Mindagent: Emergent gaming interaction." in Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024 (K. Duh, H. Gómez-Adorno, and S. Bethard, eds.), pp. 3154-3183, Association for Computational Linguistics, 2024.
[14] J. Light, M. Cai, S. Shen, and Z. Hu, "Avalonbench: Evaluating Ilms playing the game of avalon," arXiv preprint arXiv:2310.0000 2023.
[15] Y. Xu, S. Wang. P. Li, F. Luo, X. Wang, W. Liu, and Y. Liu, "Exploring large language models for communication games: An empirical study on werewolf," arXiv preprint 2xXi:2239900655. 2023.
[16] K. Valmeekam, M. Marquez, A. Olmo, S. Sreedharan, and S. Kambhampati, "PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change," arXiv e-prints, p. arXiv:2206.1044 June 2022.
[17] K. Valmeekam, K. Stechly, and S. Kambhampati, "LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAl's ol on PlanBench," " arXiv e-prints, p. 1XXi:2240.113333. Sept. 2024.
[18] M. Umair Nasir, S. James, and J. Togelius, "GaameaaversllBeechmaa::::: Evaluating Planning Abilities Of Large Language Models Through Traversing 2D Game Maps," arXiv e-prints, p. arXiv:2410.07765, Oct. 2024.
[19] D. Hendrycks, C. Burns, Basart, Zou, M. Mazeika, D. Song, and J. Steinhardt, "Measuring Massive Multitask Language Understanding." arXiv e-prints, p. arXiv:2009.03300, Sept. 2020.
[20] A. Talmor, J. Herzig, N. Lourie, and Berant, "CommonsenseQA:: A Question Answering Challenge Targeting Commonsense Knowledge," arXiv e-prints, p. arXiv:1811.0093 1.00937, Nov. 2018.
12

---

[21] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang. D. Song, and J. Stein- hardt, "Measuring mathematical problem solving with the math dataset," arXiv preprint arXiv:2103.03874 2021.
[22] Y. C. Bill Yuchen Lin, Ronan Le Bras, "Zebralogic: Benchmarking the logical reasoning ability of language models," 2024.
[23] J. Xie, K. Zhang, J. Chen, T. Zhu, R. Lou, Y. Tian, Y. Xiao, and Y. Su, "Travelplanner: A benchmark for real-world planning with language agents," arXiv preprint arXiv:2402.01622. 2024.
[24] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, "Training Verifiers to Solve Math Word Problems," " arXiv e-prints, p. 1rXiv:2110.14111 Oct. 2021.
[25] S. Han, H. Schoelkopf, Y. Zhao, Qi, M. Riddell, W. Zhou, Coady, Peng, Y. Qiao, L. Benson, L. Sun, A. Wardle- Mardle-Sollno, H. Szabo, E. Zubova, M. Burtell, J. Fan, Y. Liu, B. Wong, Yavuz, Y. Liu, X. V. Lin, S. Joty, Y. Zhou, C. Xiong, R. Ying, A. Cohan, and D. Radev, "FOLIO: Natural Language Reasoning with First-Order Logic," arXiv e-prints, p. arXiv:2209.08440, Sept. 2022.
M. Sailor, A. Ni, L. Nan, J. Kasai, T. Yu, R. Zhang, A. R. Fabbri, W. Kryscinski,
[26] X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang. S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang. S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, and J. Tang, "AgentBench: Evaluating LLMs as Agents," " arXiv e-prints, p. DiXi::3388.03680 Aug. 2023.
[27] L. Hu, Q. Li, A. Xie, N. Jiang, I. Stoica, H. Jin, and H. Zhang, "GameArena: Evaluating LLM Reasoning through Live Computer Games," arXiv e-prints, p. arXiv:2412.063944 Dec. 2024.
[28] J. Yao, K. Wang, R. Hsieh, H. Zhou, T. Zou, Z. Cheng, Z. Wang, and P. Viswanath, "SPIN- Bench- How Well Do LLMs Plan Strategically and Reason Socially?," arXiv e-prints, arXiv:2503. 12349, Mar. 2025.
[29] O.-T. Team, "Reasonggym."
[30] M. Sewvaller-Boiveett B. Dai, M. Towers, R. de Lazcano, L. Willems, S. Lahlou, S. Pal, P. Castro, and J. Terry, "Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks," " CoRR, vol. abs/2306. 13831, 2023.
[31] OpenAI, "Gpt-40 system card," tech. rep., OpenAL, 2024. research/gpt-44
https: http:/wwwwwppii....//////// com/ com/
[32] OpenAl, "Learning reason with Ilms," 2024.
[33] OpenAI, "Learning to reason with Ilms," " 2025.
[34] Anthropic, "Claude 3.7 sonnet and claude code," 2025.
[35] ByteDance, "Doubao- 1.5-pro," 2025.
[36] B. Seed, J. Chen, T. Fan, X. Liu, L. Liu, Z. Lin, M. Wang, C. Wang, X. Wei, W. Xu, Y. Yuan, Y. Yue, I.. Yan, Q. Yu, X. Zno, C. Zhang, R. Zhu, Z. An, 7. Bai, Y. Bao, X. Bin, J. Chen, F. Chen, H. Chen, R. Chen, L. Chen, Z. Chen, Chen, S. Chen, K. Chen, Z. Chen, Chen, J. Chen,
J. Chi, W. Dai, N. Dai, J. Dai, S. Dou, Y. Du, Z. Du J. Duan, C. Dun, T.- Fan, J. Feng. J. Feng, Z. Feng, Y. Fu, W. Fu, H. Fu, H. Ge, H. Guo, M. Han, L. Han, W. Hao. X. Hao, Q. He, J. He,
F. He, W. Heng. Z. Hong, Q Hou, L. Hu, S. Hu, N. Hu, K. Hua, Q. Huang, Z. Huang, H. Huang, Z. Huang, T. Huang, W. Huang, W. Jia, B. Jia, X. Jia, Y. Jiang, H. Jiang, Z. Jiang,
Jiang, C. Jiang, J. Jiao, X. Jin, X. Jin, X. Lai, Z. Li, X. Li, L. Li, H. Li, Z. Li, S. Wan, Y. Wang. Y. Li,
C. Li, N. Li, S. Li, X. Li, X. Li, A. Li, Y. Li, N. Liang, X. Liang, H. Lin, W. Lin, Y. Lin, Z. Liu,
G. Liu, G. Liu, C. Liu, Y. Liu, G. Liu, J. Liu, C. Liu, D. Liu, K. Liu, S. Liu, Q. Liu, Y. Liu,
K. Liu, G. Liu, B. Liu, R. Long, W. Lou, C. Lou, Luo, Y. Luo, C. Lv, Lv, Ma. Q Ma,
H. Ma, Y. Ma. J. Ma, W. Ma. T. Ma. C. Mao, Q. Min, Z. Nan, G. Ning, J. Ou, H. Pan, R. Pang,
13

---

Y. Peng, T. Peng, L. Qian, L. Qian, M. Qiao, M. Qu, C. Ren, H. Ren, Y. Shan, W. Shen, K. Shen,
K. Shen, Sheng. Shi, W. Shi, G. Shi, S. Shuai Cao, Y. Song, Z. Song. J. Su, Y. Sun, T. Sun,
Z. Sun, B. Wan, Z. Wang. X. Wang, X. Wang, S. Wang. J. Wang, Q. Wang. C. Wang, S. Wang. Wang, C. Wang. Wang, Wang, X. Wang, Wang. Y. Wang, W. Wang. T. Wang. C. Wei, H. Wei, Z. Wei, S. Wei, Z. Wu, Y. Wu, Y. Wu, B. Wu, S. Wu, J. Wu, N. Wu, S. Wu, J. Wu, C. Xi,
Xia, Y. Xian, L. Xiang, and B. Xiang, Seeedd-hinnnn-- 1.5: Advancing Superb Reasoning Models with Reinforcement Learning," arXiv e-prints, auXiv:2504.1399 Apr. 2025.
[37] ByteDance, "Doubao- vision-pro" 2025.
[38] A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, al., "Deepseek-v3 technical report," arXiv preprini arXiv:2412. 19437, 2024.
[39] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al., "Deepseek-rl. rl: Incentivizing reasoning capability in llms via reinforcement learning"" arXiv preprint arXiv:2501. 2025.
[40] Q. Team, "Qwen3,"2000
[41] Q. Team, "Qwen2.5: A party of foundation models," September 2024.
[42] Q. Team,
"Qwen-qwa," 2025.
[43] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Yang, H. Xu, and Lin, "Qwen2.5-VL Technical Report," arXiv e-prints, p. arXiv:2502. 13923, Feb. 2025.
[44] G. DeepMind, "Gemini 2.0 flash thinking," 2024.
[45] G. DeepMind, "Gemini 2.5: Our most intelligent ai model." 2025.
[46] J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y. Duan, W. Su, J. Shao, Z. Gao,
E. Cui, X. Wang, Y. Cao. Y. Liu, X. Wei, H. Zhang, H. Wang, W. Xu, H. Li, J. Wang,
N. Deng, S. Li, Y. He, T. Jiang, Luo, Y. Wang, C. He, B. Shi, X. Zhang, W. Shao, J. He,
Y. Xiong, W. Qu, P. Sun, P. Jiao, H. Lv, L. Wu, K. Zhang, H. Deng, J. Ge, K. Chen, L. Wang,
M. Dou, L., Lu, X. Zhu, T. Lu, D. Lin, Y. Qiao, Dai, and W. Wang. VIntern Exploring Recipes for Open- Open-Source Multimodal Models," arXiv e-prints, p. arXiv:2504. 10479, Apr. 2025.
Advanced Training and
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions of the paper, including the design of the KORGym benchmark, the support for multi-tuma evaluation and reinforcement learning, and the empirical analysis across 27 LLMs/VLMs These claims are directly supported by both the theoretical motivation and the experimental results in Sections 3-6.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide detailed discussion of the limitations of KORGym in Appendix
A
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: KORGym's main contributions lic in the dataset construction, framework design, and empirical analysis with corresponding conclusions. The core contributions do not involve theoretical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We publicly release all code, experimental frameworks (Section 3.1). re- sults(Section (.2), and detailed settings(Section 4.1) used in our study. Researchers can easily reproduce the main findings of the paper using the provided materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the complete reproducible codebase and all expeirental results mentioned in the paper htttt:://aaonnnnnii 4open Sccinca////ORRY/-220000
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
poolic/guides// CodeSubmissionooll11
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https http:llccggiieee/eeeeeeeee http://uddeeeeeeeuutiiiiiie www.oodeuuiiiisoonnee..o.
/ //nips.
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All experimental details of KORGym provided in the "Experimental Setting" subsection (Section and further supplemented in our codebase, accessible via the provided URL htttt:////iggg
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not report error bars because evaluating statistical significance across KORGym, which involves a large-scale and diverse of tasks, would incur prohibitively high computational costs.
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details on the compute resources and API usage Section Experiment Setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 'tttt make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httttp////oourrppp...cppppbbbeea
Answer: [Yes]
Justification: The research conducted in this paper fully complies with the NeurIPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
18

---

Answer: [Yes]
Justification: We discuss both the potential positive and negative societal impacts of KO- RGym in Appendix [ including benefits in Al evaluation and interpretability, as well as potential risks such as misuse of advanced reasoning capabilities.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: KORGym does not pose such risks, as it consists of game-based evaluation environments and does involve the release of potentially harmful artifacts.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All existing assets used in this paper are properly credited, and their licenses and terms of use are explicitly stated and fully respected.
Guidelines:
19

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paaassttsspperssaaaes paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: All new assets introduced in KORGym are stored via anonymized URL and are accompanied by detailed documentation provided in the corresponding README file.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
20

---

Answer: [NA]
Justification: This paper does not involve any crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
The core contributions of the paper were entirely developed by the authors. LLMs were only used for minor language editing and polishing, without impacting the methodology, scientific rigor, or originality of the research.
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Limitations
While KORGym already offers a diverse suite of challenges, we recognize that current catalog may not yet fully span the breadth of reasoning styles, difficulty levels, and domain contexts that real-world applications demand.
In particular, certain complex interaction patterns- suchh as multi-agent negotiation, long-horizon strategic planning, and fine- graiggd commonsense inference are over, the balance between textual and visual tasks could be further refined to ensure equitable coverage of each modality
are underrepresented at present. More-
To address these points, we intend to iteratively broaden KORGym's 's game portfolio-adding new thematic categories, adversarial and cooperative modes, and tasks designed probe underexplored reasoning facets-while continuously revisiting our evaluation metrics to capture deeper, more nuanced model behaviors.
B
Potential societal impacts
B.1 Positive impacts
KORGym provides a rigorous, diverse, and extensible platform for evaluating the reasoning ca- pabilities of large language models (LLMs), which accelerate the development of safer, more generalizable, and more reliable AI systems. By analyzing LLM behaviors across varied reasoning tasks, KORGym contributes to improving the transparency, robustness, and alignment of such mod- els positivlly impacting applications in education, scientific discovery, and human-Al collaboration.
B.2 Negative impacts
However, there are potential negative impacts as well. Enhanced reasoning capabilities in LLMs-it if misused-coulle be exploited for harmful purposes, such as reinforcing algorithmic bias in high- stakes decision- decision-making. To mitigate such risks, we emphasize the importance of responsible model deployment and include detailed evaluation metrics that inform future alignment and safety research.
C Formal Definition of "Knowledge Orthogonality"
For a task T, the required reasoning information consists of:
K: General background/dommin--ppeeee knowledge acquired during pre-training, excluding common sense.
R: Core rule information designed to solve 7",
Q: A Rule-Driven question.
A: Answer to the question Q.
Notational Definitions:
+ Represents the cognitive process of deriving A from Q.
P: Represents the belief strength that A is valid answer to Q based on R and/or K.
P(Q A R): Belief in A driven solely by R.
P(Q - A K): Belief in based solely on K.
- P(Q - A R, K): Combined belief A, integrating R and K.
satisfies knowledge orthogonality under the following conditions:
1. Knowledge-Ring Rule Decoupling: Rule Rt is logically self-contained and independent of K.
R 1 K
22

---

2. Knowledge Assistiveness: Background knowledge K may support or interfere with the derivation of A from Q. but does not play a central role in reasoning. The extent of this influence is quantified by the Knowledge Impact Factor (3), defined as:
P(Q + A R, K) P(Q - A R)
P(Q + A R)
3 ranges from (-1, where e is a very small positive number.
When ,3 is positive and close to 0, K has little impact, with R being dominant.
When 3 negative, can range from small negative values to approaching 1,, where K increasingly undermines reasoning.
3. Rule Centrality: Correctness relies understanding and applying R, with R having significantly greater influence than K.
P(Q > R. K) P(Q - R) P(Q + A K)
4. Derivation Adjustment: This formula adjusts the reasoning process based on R, incorpo- rating the influence of K with reflecting its effect.
P(Q A R. P(Q A R) (1 B)
23

---

D Detailed Scores on KORGym
Table 5: Mathematical and logical reasoning abilities of different models on KORGym.
Model
Duse Calcalasiun
Sodtkk
Light Out Come
Squar Additin
Allen
Party Time
Puti Platting Protiem
Consractionn Company
One Sirvic Daswiing
Culorina Isear
Trawller
City
Doubon 1.5
0.060
0.120
0.060
0.000
0,200
0.240
0.140
0.020
0.050
0.050
0.340
01000
0.960
0.300
0.899
0..40
11804
01923)
015300
(.680)
Rhepees--Ri--aalll Di-aill----- 128
0.120
0.300
0.360
0:330
0.403
0.540
0.130
01:00)
0.111
0.200
0.30
DeepSed: Dutlll Qwee
0.120
0.000
0.540
0.530
0.630
0.350
0.300
0.290
0.140
0.200
0.300
0.320
Cladd-337
0.040
0.140
01000
0.30
0.280
0.150
0.130
0.180
112000
1,700
Clane--7-ttnking
11000
0.160
0.580
0.720
01880
01480)
11.50)
(1680)
11:50
01000
GFT-4o
0.000
0.050
0.040
0.000
0.140
0.050
0.020
0.620
0.020
0.630
0.20
(1-miin
0.100
0.430
(1980)
0.380
0.720
0.760
(1.04)
01386)
0.750
01906
01280
01-3044-2--7
0.260
0.640
1,000
0.500
0.840
0.600
01340)
0.250
0.760
0.800
0.900
Gemini 2.0 Flash
0.000
0.160
0.040
0.000
0.160
0.288
0.200)
01.000
0.000
0.080
[1340)
Gemiii 255 pro-413-0 11-25
0.260
11640
0.800
0.700
0.780
01520
11.50)
0.330
11000
01980
(88001
Genini 2.0. Flad-thinking
0.120
0.340
0.220
0.360
0.340
0.040
0.320
0.740
0,500
Qwen-Qued
0.080
0.300
0.340
0.000
0.020
0.000
0.000
0.340
0.000
Qwee2.5 32B
0.050
0.000
0.000
Qwee2.5
0.000
0.000
0.000
0,020
Table 6: Control, interaction, and task reasoning abilities of different models on KORGym.
Minigrid
Snake
Tower Hanoi
Numeric Bricks
Minesweeper
Nullify
PVZ
Long Cat
White Copy
Black
Duubao-1.5 1.5-gno
0.050
1,800
0,000
0.150
0.000
0.000
0.250
17.550
0.020
0.000
Doubao-1: i-S-thining--po
0.300
16,300
0.300
0.650
0.000
0.076
03400
59. 15.990
0.600
0.360
DeepSesk- RI-Distll-Qeen-122
7.480
0.300
0.000
0.300
31,750
0.220
0.200
DeepSeck-K Ri-Distil-Qseen778
0.150
0.000
0.100
11.950
0.000
0.000
DeepSeek-R R1
0.100
16.500
0.650
0.700
0.020
0.185
0.200
68.450
0.480
0.300
DeepSeckk v3-0324
0.250
4,250
0.050
0.200
0.000
0.063
0,300
20.350
0.300
0.080
Clande-3.7
0,150
6.500
0.100
0.100
0.000
0.019
0.150
27. 300
0.140
0,020
2auulo-.7-thinnking
0.350
9,750
0.150
300
0.320
0.180
GPT-4o
0.600
0.000
0.050
0.000
0.150
0.000
0.0X1
03-mini
0.300
15.950
0.900
1,000
0.000
0.555
0,250
41.200
0.700
0.360
01-2022-12--17
0.150
17,500
0.550
0,850
0.000
0.741
033000
$4,850
0.840
0.360
Seemini-220-Flede
0.150
2,050
0.000
0.100
0.000
0.100
0,020
-ermini-2.5-gro-005.
13,500
0.050
0.000
61.300
0.600
0,380
Gemini 2.0 O--lush thiskithinking
0.100
1.850
0.050
0.200
0.000
0.002
0.250
26.000
0,060
0.100
Owen QuQn-QuQ
0.150
3,950
0.000
0.500
0.000
0.052
0.350
30.750
0.020
0.160
Qwen2.5-722 5-B---2--888 Instruct
0.050
0.620
0.000
0.200
0.000
0.000
0.250
21.850
0,040
0,000
Qwen2.5-535 1022.5532---sssuue
0.050
0.000
0.050
0.020
0.020
Quen2..57B--Isttee
0.000
0.450
0.000
0.050
0.000
0.250
0.050
0.000
0.020
Table 7: Language and textual reasoning abilities of different models on KORGym.
Model
Word Problem
Alphaberical sorting
I Letter Connection
Wond Transformation
Wondle
Crypto Word
Douhho- 1.5-pro
0.120
0.360
0.160
0.100
Doahao- 1-5 S-thinnkng-ppr
0.600
0.880
0.720
0.400
0.600
1.000
DeepSeck-RR -isti Q-ee---302
0,340
0,500
0.220
0,140
0,450
0.000
DecpScok-R1-B RL--Disill-C --isiill-QQen-77 QB
0.020
0.020
0.050
DeepSeek-RI
0.820
0.960
0.900
0.420
0.600
0.950
D34p54ek-V3-0324
0.460
0.840
0.42)
0,380
0,500
0,500
Claude-3.7
0.580
0.660
0,250
3.7.he-..
0.820
0.980
0.960
0.560
0.850
1,000
GPT-1o
0.420
0.310
0.160
0.120
0.400
0.100
03-mini
0.880
0,980
0.980
0.400
0,400
1,000
01-21124-22-17
0960
1.000
0,980
0
0.400
0,850
Gemini-2.0-HH 2.0Flahh
0.340
0,560
0.340
0.120
0,250
0.100
Gemini-22 2.5 000-03-25
0.900
0,960
0.940
0,780
0.650
1,000
2 thiihiii
0.620
0.780
0.400
0.180
0.550
0,500
Qwen-QwQ
0.480
0.760
0,400
0.180
0,400
0.050
QwenZ 5-72t-tnttnett
0.880
0,280
0.160
0.020
0.200
0,000
Owen2 S-3tt-nsttucc
0.100
0.080
0.050
Qwen2 Instruct
0.040
0.140
0.020
0.000
0.050
0.000
24

---

Table Spatial and geometric reasoning abilities of different models on KORGym.
Model
Maze
Sokaban
lines
Emoji Connect
Suuzze
Bubble Ball
Sorting
Pipe
Free
Map Simulation
Arrow Palbway
1.5-pro
0.300
Douhoo- 1-S-lhinking-poo
0800
0430
0.580
1.000
0.000
0.380
0.160
0.140
0.221)
11.300
0.900
0.800
0.700
0.040
0.000
DeepSeck-1 RI RI-Distill-uwee
0.120
0.040
0.000
0.000
0.100
0.000
0.000
0.000
0.000
0.000
DeepSea-RR
0.600
0.40
0.360
0.760
0.520
1,000
0.300
0.750
0.140
0.000
DeepSeck-x3-005
0.380
0,080
0.080
0.560
0.220
0.250
0.020
0.650
0,050
0.000
Claate-3.7
0.020
3.7-ae-3.7-chnninn
0.180
GPT--0
0.020
0.040
0.450
0,000
0.000
O3-mini
0.60
0.780
0.720
0.964)
11940
0.950
0.540)
0.850
0.350
0.300
11-2024-12-17
0.360
0.700
0.840
0.540
0,900
0.950
0.840
0.830
0.250
0.000
Gemini-2 O-Flach
0,020
0,000
0.000
0.120
0.840
0.600
0.080
0.050
0.100
0.000
Gemini 2.5 pro-03-2 03-25
0.500
0.100
0.660
0.620
0.720
0.950
0.740
0.850
0,100
0.000
Gemini-2 0-F---2----alh Flass--hinking
0.020
0.050
0,020
Qwes-QwQ
0.400
Qwee2.5 72B- Instruct
0.040
0.020
0.020
0.060
0,040
0.350
0.000
0.100
0,000
0.000
Qwas2.5-328 -----ooe
0,020
0.000
0.040
0.000
0.000
0,500
0.000
0.200
0.020
0.000
Owee255 7B 77-ttttte Instruct
0.000
0.000
0.000
0.020
0.000
0,000
0.000
0.000
0,000
0.000
Table 9: Strategic and game- theorthe reasoning abilities of different models KORGym.
Model
2048
Trust Evolution
point
Spider Solitaire
Circle cat
Soubao--..--pre
648.800
31,650
7.500
0.000
0,000
Doubao-1 1-wiii-ni---i----------
1736.000
36.600
8.000
0,000
0,050
DeepSeek-RI-DDee Distill-Qwenn33
990.000
37.700
8,050
0.000
0,050
DeepSeek-RI-Disse Diw.Dill---w--------
884.200
36.350
3.625
0.000
0.000
DeepSeck-R1
2221.800
63,500
8,250
0.000
0.200
DeepSeck-v3-0324
1101.400
37.750
8.225
0.000
0,200
Claude-3.7
809.000
36,900
8,300
0.000
0.000
Claude-3.7-ttinkinn
1283.600
37.950
8,200
0.000
0.100
GPT-4o
958 6.8.000
54.750
7,650
0.000
0.000
03-mini
1285.600
34.550
7,825
0.000
0.300
01-2024-12-17
1652.200
64.500
8.600
0.000
Gemini-2.0 2.0-ni--
879.800
51.750
7.475
0.000
0.000
Gemini-2.5 2.5pi-2-55-prr---2222 03-22
200
66,500
7.750
0.000
0.250
thinkin
1378.200
37.500
7.925
0.000
0.000
Qwen-QwQ
681.800
58.400
3,950
0.000
0.000
Qwen2.5-722 7772--nsuutt
773.400
47.150
7.600
0.000
0.000
Qwen2.5-32B-o 3322--tsuutt
660.000
51.850
7.550
0.000
0.000
Qwen2.5-7B 7nstttuutt
780.600
0.000
6,200
0.000
0,050
E
PCA Result
The first two principal components capture 96.2% of the total variance (PC1: 91.9%, PC2: 4.3%)), as shown in Figure 5b The model PCA cluster results are follows:
Cluster 0:Doubao1 1-5 5-tthinking-pro DeepSeek-RR R1,, Claude-3.77 3.7-thinking, Gemini-2.5-pra 03-25, Qwen3-32B
Cluster 1:03-mini, 01-2024-12-17
Cluster 2:Doubao-1.5-p DeepSeck-R1-Distiiee -wessill--wen-BBB, GPT-4o, Gemini-2.0-Flaash Qwenn.5-77BB-mm Instruct, -0ven2.5-32B-Inssruue Qwen2.5-7B-Instte
Cluster 3:: DrrSSeck-R--iisill--vveeee DeepScek-v3-0324 Claude-3.7, Gemini-2.0- Flash-thinking, Qwen-QwQ
These results reveal the following insights from the clustering structure:
Cluster 1 predominantly consists of the GPT-o series reasoning models, which exhibit highly similar reasoning patterns, indicating a consistent architectural behavior within the series.
Cluster includes tor-perforning reasoning models, such as Claude-3.7. Gemini-2.5, and DeepSeck-RI, which show strong and mutually consistent performance across dimensions.
25

---

Qwen3 and Doubao--.5-thinkingg pro are located near the boundary between Cluster and Cluster suggesting that these models share reasoning characteristics with both the GPT-o series and the leading reasoning models in Cluster O)
Cluster primarily contains a mix of open-source reasoning models and closed-source non-reasoning models, exhibiting moderate overall reasoning ability.
Cluster 2 consists of mainstream open-source non-reasoning models and closed-source baseline APIs, representing a group of lower-performing or general purposs models.
Detailed Result of Impact of Disabling Each Paradigm on Model Performance
Impact Disabling Each Paradigm on Performance
71.1%
Doalle
Percent Change in AvgScore (%)
48.3%
28.9%
93.6%
8.199.7%
4.0%
0.9%
1.6%
3.7%
2.5%
3.4%
-4.9%
8.3%
-2.3%%
GPT-40
03-mini
Doubao- 1.5ppro 5.p-
Doubao-1-3 1-5 S-tthinking-prr
Claude-3.7 Slaude337-thinkking -.i-kinnnng
Gemini-2.5 11ro--2.--pr----33 03--03--25 03-25
Deep5eek-v3-0324
Figure 8: Detailed Result of Impact of Disabling Each Paradigm on Model Performance.
G Comparison of different game benchmarks.
Table 10: Comparison of different game benchmarks.
Benchmark
Games
Multi-turn
RL
Controllable Difficulty
Multimodal
LogieGame
27
X
x
AgentBench
26
8
/
x
GameArena
3
/
EXX
SPIN-Bench
28
/
X
TEXTARENA
[0]
57
/
RcasoningGY'M
29
100+
X
KORGym
51
V
V
26

---

H Detailed Game Category
Table 11: Game Introduction of KORGym. From top to bottom, the categories are: Mathematical and Logical Reasoning games, Puzzle Reasoning games, Spatial and Geometric Reasoning games, Strategic Reasoning games, Control and Interaction Reasoning games, and Multimodal Reasoning games.
Name
Task Content
Croswoor Puzzle
This game challenges LLM infer current date given future date and number of days berween them.
Sudoku
This game evaluates an LLM's logical reasoning by solving Sudoku puzzle.
Light Out Game
This game tests an L.M's stritoglc reasccing by reguiring to switch off all lights on a 3x3 grid.
Square Addition
This game reguires 11 M to compute column sums hased on symbolic values.
Alien
This game requires LLM count alien species based multiple traits from complex dataset.
Party Time
This game challenges M identify and count stadents who mest specific criteria.
Path Planning Problem
This game requires L.MM to calculate the shortest distance between two cities within a complex, netwark.
Construction Company
This game challenges LLM calculate the minimum time across multiple companies and projects
Tower Hanoi
This game requiies L.MM salve Tower Hanok puzzle, moving dicks between columns reach goal state.
Numeric Bricks
This game challenges an
to grid by expanding each labeled cell according to specifsed count.
One Stoke Drawing
This game roguires 1 M to find an Fulerian path that visits every odge esactly once.
Nullify
This game challeages an LLM to comhinn arithmetic units using operations to achieve a final result of zero. adjacent nodes share the same color. This game roquires M exrmes, filtor, compute city information from a complex city nerwvek.
Coloring Issue
This game challenges LLM to assign colors graph nodes
that
City Traveller
Word Problem
This game challenges an LLM to find specific English word that matches several constraints.
Alphibbtical Sorting
This game roquires LLM rearaang the remmining unomlered letiers form valid word. word
Connection
This game roquires
reconstruct
letter grid.
Word Transformation
This game tequires LLM decode transformed word by reasoning tirough layered transforaations
Wordle
This game taquines LLM lu perform deductive ressonngg through lieralive word guessing bused on structured feodbuck.
Crypto Word
This game requires LLM decode seutence by mapping emojis letters through iterative feedback based guessing.
Maze
This game roquites 1J
a valid path through maze from the start point the destination.
Sokohan
This game requires 1J M to solve a Sokobon puzzle, pushing all boxes onto the targer areas.
Playlines
This game requires LLM fill grids by connecting identical numbers
repeised honizontal vertical putterns grid wilhout leaving empply sprcess.
Emaii Connect
game roguires
grids.
8-puzzle
This game roquires
plan valid moves to reposition torget tile a sliding puzzle grid.
Bubble Bull Sorting
This game toquiiees L. M sori colored balls into unifor tubus under stacking constraints.
Pipe Game
This game roquires 1. M capability to rotate pipes in grid create continunes path.
Key
This game requines LLM move the building blacks
keys This gamee lests an LLM's sbility to simulatc multi-stpp movemnett through a dynamic grid.
exil.
Map Simulation
Aroo--puhwwyy
This game tests an M'''ss ahility to navigare a maze by sequencing directional actions to trigger waypoints.
2048
This game reguires 1 play the 2048 puzzle by choosing the hest hased the identify and exploit opponent behavior pattems through strategic decision-making. This game requires L.M to play expanded 21 Point in dynamic thresholds and an oppotent's is fixed behavioe. This game tests an L shiliry to plan and execure stratepic moves in Spider Solitaire.
game hoard
Trast Evalution
This game roquires
N-point
Spider Solitaire
Circle the Cat
This game requires 11 stritegically
walls bexagonal grid to top moving
Minigrid
This gamae roquires LLM solve a series of lasks based on the Minigrid [30] reinforvement learning system.
Snake
This game requires LLM control growing smake on a bounded grid, aveiding collisions to maximize score. This requires LLM plan by strategieally rotating and placing Tetris blocks cleur and maximize Blg
Totris
Mineswapper
This game requines LLM uncover
mines..
PVZ
This game requires LLM to place plants to counter increasingly strong zombies.
Long Cat
This game requires LLM plan efficient movement sequences by savigating a sliding cat to fill all empty spaces. transform the board into specificl Bouck-nnd--hin target paliern
Black White Copy
This game taquines LLM loggle
Creewwod Puezle
This game requines LLM solve linguistic clues to fill the grid with wonds correclly.
Jiguew Puzzle
This game roquires 1. M to match visual puzzie pieces with mumitered nd slots.
Find The Pattern
This game requires LJ M identify the correct visual piece that completes given partem.
Cincle The (Visual)
and determine optinal wall placements prevent call. This game roquites 1 interact with diverse objects, and accumately calculate the linal position.
game requines LLM amalyze
Map Simulation (Visual)
Sokoban (Visual)
This game tequiiees LLM to interpret a visual Sokoban puzale and generate precise series of moves, genorate valid moves achieve uniform color soeting through multiple This game requires LLM infer inkeer values to compute accurate column sums.
Bubble Ball Sorting (Visual)
This game requiiess L.M
specilied tubes. guessing.
Wordle (Visial)
This game roquires
dellcee
Square Addition (Visal)
27

---

Natheratical and Legical Recsoring
Control Interaction Resseting
Exdeste narencal compristinn lopid rifernce, combinotonil planing graph thery, and watt-shep artheatic adeditias ddit.
Enikatt deckke-rakin naking and openatiooall cxpoblittic with cottects repiiity contieau interacties, cnvreenent feeebock, and stote spdates.
Total Namber of Gamer:4
Tetd Humber foress
Example:Alien
Example:PVZ
There ane searrd clion beinge on dietont planet, Their cotegerice and corresponding features os follows
followirg, you requined participate simplified version
Plants Zombies
Allen()::
Omrivoree, Lege 7, Horne Reproduction
<Rents Informatione
Reprodaction,
crangi:
Zooriiis Information
Alien(2) hhdF2BK, Parasite, Legs Horns Reproduction mommol. Color purple:
<Rules Information
Tumn 25 [un:30 | Soore: 25
Current Bottleriald (X lowe. Wi Peashoster. S Tiree Line Shooter, 2:
Now, there one nAv, I IhhPZBK.... in this area. Please help me calculate the total number of olien onimela that meet the following uniteria alien crinal that 1. legs are ///22/34/556////8// Vioporrus/ Spore Regroducton/ 0/L/2/3 A/B/6/7/8///110 seeecttvrrehhrbiivveee Stwiiiiee//aaatt/eeeeeeeee /eteaat///eeeeeee//eee
Wall-mrt, Torch Shuno, F: Fire Chil, N: Zembie, R: Roodblock Zombie,
reprodaction ore 3. homs are
Buchet Zombre High-Attcck Zambie)
Line8 IX
1o
10
del
Line[XX 13 IR
10
10 10
Seseverger/Cmiiee////rwwwwe oolor are onangnsggr Baingn/grrenn loyon pprpla/yylllo.
Line3 Ix
10
LLne4 (W 1J 12
[JE
W 10 10
Puzzl Resssning
Spotial and Geometric Reasening
legaettic copastines lickadin jazdles noniputtti,, ancrpptior/lccrytiinn
Evelantc puzdes enphancing geametry, nanggetion and path planning including maxes, rasting profeea, and other spatialy oriented taako. Tend Number 9oree-10
Wordle
Exomple:free
Key
You guees specific locotton bosed word according the information for reference. guess your guess
The goms contoris homizontol and vertical blocks (rapresented by letters), e key (1), and on exit (2) Your gool is move the blocks and key to hel the key reach the exit Honzoond blocks can only nove horzonnally vertical blocks can only mave vertically, and key can only move harizentally, Control brocks and the key
recorded the round baxed on the following information, 'Aeswer: happy'. Wordle Gome
moves
the key successfu fuly resches
the
Word langth 4
Attongt: of 30
each furn, output the brock/ and moving direction, For example, moving the means moving the black lobeled "I" one step upward.
History:
"Answer: right
the rightt "Answer: P
1. Guesss rome
Feedback:
Remember the gome will end at 100th epach. Board
The letter located ideeO the word any spot.
spot. The letter located idec3 word the wrong
any spot.
B F
A 8O00EE E 000011
Feedback:
2 Guessi smeil
0 0
0 C0
Strategic Recsoting
Ressoning
Encuutte card theory, and edversattal decisian-mdio scenaria, strategic andlyta
Endaate visad recogeitir, extching suriil perception thrugh photogrohhr, Tehd Namber Gonee:
Tetel Number danes 5
Example Evalution Trust
Example Chril
your When you chouse "calisborste" cheots.
shows, solid crcess represent cats, the represent emply spaces,
"cheat" (put gani thres (-1 (3) Howeserr both choose. to chest, naither gains nor lases anything (0 Oniinnt will play najor rounds against opporennt with a fised dinategy, each major round consisting of miner rounds. You have occess the complete records of all presinus mnjer rounds and the currest major round, while your opconent only has sccess to the records of the major round. Your opponent's a stronogy remaiis unchanged throughout garre
Both collabonste
reprosent blue the with wolls
circles regresient while
the exit esupp.
Major Round: 2/5
3/8
Scooe:
Completed Major Reunds History:
Majer Round
Miner I. You: cheat, Opporeent: chuat
Figure 9: Overview of the KORGym tasks. Our KORGym supports over 50 novel games, enabling precise and efficient evaluation of large language models (LLMs) across six distinct capability dimensions.
28

---

I Models Evaluated in KORGym
Table
Models evaluated in KORGym.
Series
Moxell
GPT-40 [1 mi:mmii 13
Claude
Claude Sonnet 34 Chadde 3.7 Sonnet thinking
Doahan
DeepSrokad [6) Dhulto-1..4tiinkkigge 16 Deuhen-rsian-go
Doulun-1.Sppp [S]
DeepSrek
[10 DeepSreic F1-Deell-QQemm70.22
Q0en -MT--Intrrae(770.087770 tinking Cenmai poo
IeeWVLL-668
Detailed Result of Textual and Multimodal Versions of the Same Game
Text Visual Scores per Game within Each Model
1.0
Game
Type
The Sorting
0.8
Map Simulation
Model Avg
Sokoban
Avg Text
Square Addition
Avg Visual
0.6
Wordle
Score
0.2
0.0
Qwen2.5V SVL-7B
Doutao-vition-25000 250115
Qwen2 SVL.225 52L
Qwen2 5VL-728
IntemVL3-788
GPT4o
Geminiz Gemini2.0-Flash 0-Flash
Gemini2.55PP 5-rro
Figure 10: Performance Comparison the Same Model on Textual and Multimodal Versions of the Same Game. In this visualization, bar colors indicate different games, while shading of the bars distinguishes between text (no shading) and visual (with shading) versions. Solid and dashed lines represent the average scores for the textual and visual versions, respectively.
29


Reviews:
[{"review_id": "ngpC2iUT8O", "reviewer": "Reviewer_npDs", "summary": "This paper introduces KORGym, a game-based LLMs and VLMs evaluation benchmark. The authors design 56 games across 6 reasoning dimensions and supports multi-turn, interactive assessments. The authors conduct large-scale evaluation on 19 LLMs (including 11 thinking models) and 8 VLMs. The results demonstrate KORGym’s utility for evaluating and challenging LLMs on knowledge-orthogonal reasoning abilities.", "questions": "Check the weakness part mentioned above", "limitations": "Check the weakness part mentioned above", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nNovel Benchmark: KORGym offers a diverse set of games to evaluate the reasoning abilities of LLMs and VLMs.\nBroad Scope: KORGym introduces a large suite of games spanning six key reasoning dimensions. Also, the authors design evaluation method beyond binary scoring.\nReady-to-Use Implementation: The authors not only provide the evaluation datasets and metrics but also deliver optimized evaluation code. This significantly lowers the barrier for adoption and ensures reproducibility.\nStrong Empirical Study: The experiments involve a broad collection of current LLMs and VLMs, with both closed- and open-source models. This enables comprehensive comparison and drives key findings on strengths, weaknesses and behavioral tendencies between model series.\n\nWeaknesses:\nPotential Domain and Task Distribution Bias: Although the knowledge-orthogonality concept aims to decouple evaluation from prior knowledge, the game/task design process is not fully transparent regarding avoidance of distributional overlap with LLM pre-training corpora. The paper does not concretely quantify the degree of knowledge leakage, which may influence some of the comparative results or limit conclusions about pure reasoning.", "was_revised": true, "final_justification": "I am satified with the author's repsponse. My final rating is accept.", "rebuttal": "Thank you for your insightful suggestions. We apologize for not clearly addressing this aspect in our KORGYM submission. Below, we will elaborate on the measures taken in the design of KORGYM to minimize overlap with pre-training data. In addition, building on the formal definition of Knowledge Orthogonal Reasoning (KOR) in our paper, we will introduce our quantifiable metrics to assess knowledge leakage and select exemplar games within KORGYM to empirically analyze the extent of such leakage.\n\n**Response to Weakness**\n\n(1) **Question1**:  Potential Domain and Task Distribution Bias: Although the knowledge-orthogonality concept aims to decouple evaluation from prior knowledge, the game/task design process is not fully transparent regarding avoidance of distributional overlap with LLM pre-training corpora. \n\n(1) **Answer1**:\n\nIn the design phase of KORGYM, significant efforts were made to minimize potential knowledge leakage:\n\n> -  Unlike traditional reasoning benchmarks, KORGYM is designed as **dynamically generated games**. The nearly infinite decision space and symbolic diversity significantly reduce the likelihood of overlap with pre-training data.\n> -  Games, especially multi-turn scenarios, require **long-term strategic planning**, which is typically underrepresented in pre-training corpora.\n> -  During the selection and creation of games, we developed numerous **original games and modified traditional games** extensively to further prevent overlap with pre-training data.\n\n---\n\n(2) **Question2**:  The paper does not concretely quantify the degree of knowledge leakage, which may influence some of the comparative results or limit conclusions about pure reasoning.\n\n(2) **Answer2**：\n\nThank you for your valuable suggestion and we will provide a formal definition of the degree of knowledge leakage and supplement it with quantitative experiments:\n\n**2.1 Formal Definition of the Knowledge Leakage in KORGYM**\n\nIn Appendix C of our submitted paper, we formally defined the concept of \"Knowledge Orthogonal Reasoning\" (KOR). However, it does not allow for empirical quantification of the extent of pre-trained knowledge leakage. This limitation arises from the inherent impossibility of fully eliminating the influence of pre-trained knowledge $K$ on the reasoning process. Additionally, the modeling scenario of KORBench[1], structured as a question-and-answer format, differs significantly from that of KORGYM. Consequently, we extend the formal definition originally proposed in KORBench and present an experimental approach to quantitatively measure the degree of pre-trained knowledge leakage.\n\n[1] Ma et al. KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Tasks. arXiv.2410.06526.\n\n---\n\n**2.2 Notational Definitions**\n\nFor a game within KORGYM, the required reasoning information comprises:\n\n- **$K$**: General background or domain‑specific knowledge acquired during pre‑training and post-training.  \n- **$T$**: Single‑turn or multi‑turn tasks that must be completed during gameplay.  \n- **$R$**: Core game‑rule information specifically designed for solving task $T$.  \n- **$A$**: The set of actions generated by the LLM to solve task $T$.\n- **$\\rightarrow$**: The reasoning and interaction process whereby the LLM gradually derives the action set $A$ from the game task $T$.  \n- **$S$**: The final score obtained by the LLM's interaction with the game environment based on reasoning from $R$ and/or $K$:\n  - $S(T \\rightarrow A \\mid K)$: Score from actions derived solely from pre‑trained knowledge $K$.  \n  - $S(T \\rightarrow A \\mid R, K)$: Score from actions derived using both game‑rule knowledge $R$ and pre‑trained knowledge $K$.  \n\n---\n\n**2.3 Analysis of Game Redundancy**\n\n1. **Knowledge‑Rule Decoupling**  \n   Ideally, the game rule information $R$ is logically self‑contained and independent of pre‑trained knowledge $K$:  \n   $$\n   R \\perp K\n   $$\n\n2. **Knowledge Leakage Coefficient**  \n   The Leakage between $K$ and $R$ is quantified by the Knowledge Impact Coefficient $\\beta$:\n   $$\n   \\beta\n   = 1 - \\frac{S(T \\rightarrow A \\mid R, K) - S(T \\rightarrow A \\mid K)}{S(T \\rightarrow A \\mid R, K)}\n   = \\frac{S(T \\rightarrow A \\mid K)}{S(T \\rightarrow A \\mid R, K)},\n   \\quad \\beta \\in [0,1].\n   $$\n- A smaller $\\beta$ indicates less overlap between the LLM’s pre-trained knowledge and the game rules. Consequently, the LLM cannot achieve optimal scores without explicitly relying on the game rules provided in the prompt.\n- Conversely, a larger $\\beta$ indicates significant overlap between the LLM’s pre-trained knowledge and the game rules, allowing the LLM to complete the game task and achieve optimal scores even without explicit game rules provided in the prompt.\n\n---\n\n**2.4 Quantitative Experiment on Knowledge Leakage**\n\nBased on the above definition, we select a subset of games from KORGYM, including some classic games (Snake, Sokoban, and 8-puzzle) and original/adapted games (Pipe Game, Long Cat, ...). We conduct an ablation study in which the core game‑rule information (R) was removed and then compute the Knowledge Impact Coefficient $\\beta$. The results are presented below.\n\n\n|    Model    |  Snake  | Pipe Game | Long Cat | Sokoban | Word Transformation | 8‑puzzle | Play Lines | Black White Copy |\n|:-----------:|:-------:|:---------:|:--------:|:--------:|:-------------------:|:--------:|:----------:|:----------------:|\n| Doubao-1-5-thinking-pro |  0.055  |   0.085   |    0     |  0.904  |         0           |  0.176   |     0      |      0.166       |\n| DeepSeek-R1             |  0.333  |   0.200   |   0.208  |  0.636  |         0           |  0.307   |     0      |      0.133       |\n| Qwen3-32B-thinking      |  0.071  |   0.080   |    0     |  0.909  |         0           |  0.321   |     0      |      0.117       |\n| o3‑mini                 |  0.956  |   0.000   |    0     |  0.948  |         0           |  0.617   |     0      |      0.166       |\n| Claude-3.7-thinking     |  0.441  |   0.200   |   0.312  |  0.461  |         0           |  0.521   |     0      |      0.222       |\n| Average | 0.371 | 0.113 | 0.104 | 0.771 | 0 | 0.388 | 0 | 0.160 |\n\n\n\nFrom the above results, we can draw the following preliminary conclusions:\n\n> - **Model perspective:** Models such as **o3‑mini** and **DeepSeek‑R1** exhibit relatively notable knowledge leakage, whereas other models (e.g., **Doubao‑1‑5‑thinking‑pro**) demonstrate comparatively lower leakage. This observation indirectly supports the view that **reinforcement learning (RL) processes genuinely enhance a model's reasoning capabilities, rather than merely enabling it to memorize training questions or regurgitate knowledge from pretraining**.\n> -  **Game perspective:** Knowledge leakage is more prominent in **classic games**, while the corresponding β values for **original/adapted games** remain at a relatively low level. Given that more than **80%** of the games in KORGYM are originally developed or adapted by our engineering team, this strongly supports the validity and reliability of KORGYM.\n> -  **Methodological perspective:** We believe that the knowledge leakage quantification experiment itself offers an interesting insight and methodology for **analyzing models’ genuine reasoning abilities and uncovering information contained within the pre-training data**. Future work could extend this approach to conduct deeper analyses and evaluations of reasoning capabilities.\n\nOnce again, we sincerely thank you for your valuable feedback on KORGYM. Your comments have provided insightful inspiration and an important reminder. Moving forward, we plan to further increase the proportion of original/adapted games in KORGYM to more accurately capture models' genuine reasoning capabilities."}, {"review_id": "gPryh27h0g", "reviewer": "Reviewer_V7Eo", "summary": "The paper introduces KORGym, a novel interactive benchmark designed to evaluate the reasoning capabilities of large language models (LLMs). KORGym consists of 50 games, most of which do not require external knowledge, hence the term Knowledge Orthogonal. The benchmark includes both textual and visual games. A wide range of LLMs is evaluated on this benchmark, and the results provide valuable insights into the reasoning behavior of modern models.", "questions": "1. It would be important to understand how much training on the same game set influences performance.  \n   For instance, the paper notes that the Duobao model was trained on classical games such as Sudoku.  \n   **Question:** Is it fair to compare Duobao's performance on these games with models that have *not* seen such games during pretraining?\n\n2. In the *Impact of RL* section, could the authors clarify:\n   - What fraction of KORGym games (if any) overlaps with Duobao's RL training set?\n   - Whether there is an ablation isolating the RL impact from other improvements in the \"thinking\" model variant?", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n- The proposed benchmark is highly relevant and timely for evaluating current SOTA LLMs. Unlike many existing benchmarks, KORGym is less susceptible to data contamination and better suited to test strategic thinking and planning.\n\n- The evaluation section is particularly strong. It includes:\n\n-- Analysis of reasoning facets in LLMs\n\n-- Insights into the interplay between textual and visual reasoning\n\n-- Clustering of models based on performance profiles\n\n-- Detailed ablation studies and thinking pattern analysis\n\n**Weaknesses:**\n\n- The section titled “Impact of Reinforcement Learning” lacks clarity:\n\n-- It is unclear how much overlap exists between the KORGym task set and the training tasks used for the Duobao RL training.\n\n-- Without a well-controlled ablation, it's difficult to conclude whether the performance improvement from the non-thinking to thinking variant of the model is due to RL training alone, or to other architectural or training changes.", "was_revised": false, "final_justification": "", "rebuttal": "First and foremost, we sincerely thank you for your recognition of KORGYM and for your valuable feedback on the analysis of game overlap and RL impact. We deeply apologize for our oversight in this regard. To quantitatively analyze the influence of game overlap and RL impact, we formulated the following formal definitions and conducted two sets of ablation experiments:\n\n**Response to Question 1 and Question 2-1**\n\n(1) **Question 1**: \n\n- Is it fair to compare Duobao's performance on these games with models that have not seen such games during pretraining? \n- What fraction of KORGym games (if any) overlaps with Duobao's RL training set?\n\n(1) **Answer 1**:\n\n\tIn Doubao's technical report, only examples of Sudoku, Maze, and 21-Point were provided, without detailed data being included. Therefore, to address these two questions, we need to introduce a parameter capable of quantifying the impact of data leakage on model performance and address the problem through analytical experiments. Although we formally defined the concept of KOR in Appendix C of our submitted paper, it did not permit empirical quantification of pre-trained knowledge leakage. To rectify this, we extend the formal definition and introduce the Knowledge Leakage Coefficient, which can be empirically calculated through experiments, to further substantiate the validity of KORGYM.\n\n---\n\n**1.1 Notational Definitions**\n\nFor a game within KORGYM, the required reasoning information comprises:\n\n- **$K$**: General background or domain‑specific knowledge acquired during pre‑training and post-training.  \n- **$T$**: Single‑turn or multi‑turn tasks that must be completed during gameplay.  \n- **$R$**: Core game‑rule information specifically designed for solving task $T$.  \n- **$A$**: The set of actions generated by the LLM to solve task $T$.\n- **$\\rightarrow$**: The reasoning and interaction process whereby the LLM gradually derives the action set $A$ from the game task $T$.  \n- **$S$**: The final score obtained by the LLM's interaction with the game environment based on reasoning from $R$ and/or $K$:\n  - $S(T \\rightarrow A \\mid K)$: Score from actions derived solely from pre‑trained knowledge $K$.  \n  - $S(T \\rightarrow A \\mid R, K)$: Score from actions derived using both game‑rule knowledge $R$ and pre‑trained knowledge $K$.  \n\n---\n\n**1.2 Analysis of Game Redundancy**\n\n1. **Knowledge‑Rule Decoupling**  \n   Ideally, the game rule information $R$ is logically self‑contained and independent of pre‑trained knowledge $K$:  \n   $$\n   R \\perp K\n   $$\n\n2. **Knowledge Leakage Coefficient**  \n   The Leakage between $K$ and $R$ is quantified by the Knowledge Impact Coefficient $\\beta$:\n   $$\n   \\beta\n   = 1 - \\frac{S(T \\rightarrow A \\mid R, K) - S(T \\rightarrow A \\mid K)}{S(T \\rightarrow A \\mid R, K)}\n   = \\frac{S(T \\rightarrow A \\mid K)}{S(T \\rightarrow A \\mid R, K)},\n   \\quad \\beta \\in [0,1].\n   $$\n- A smaller $\\beta$ indicates less overlap between the LLM’s pre-trained knowledge and the game rules. Consequently, the LLM cannot achieve optimal scores without explicitly relying on the game rules provided in the prompt.\n- Conversely, a larger $\\beta$ indicates significant overlap between the LLM’s pre-trained knowledge and the game rules, allowing the LLM to complete the game task and achieve optimal scores even without explicit game rules provided in the prompt.\n\n---\n\n**1.3 Quantitative Experiment on Knowledge Leakage**\n\nBased on the above definition, we select a subset of games from KORGYM, including the games mentioned in Doubao's technical report(Sudoku and Maze), some classic games (Snake, Sokoban, and 8-puzzle) and original/adapted games (Pipe Game, Long Cat, ...). We conduct an ablation study in which the core game‑rule information (R) was removed and then compute the Knowledge Impact Coefficient $\\beta$. The results are presented below.\n\n\n|    Model    |  Snake  | Pipe Game | Long Cat | Sokoban | Word Transformation | 8‑puzzle | Play Lines | Black White Copy | Sudoku | Maze|\n|:-----------:|:-------:|:---------:|:--------:|:--------:|:-------------------:|:--------:|:----------:|:----------------:|:----------------:|:----------------:|\n| Doubao-1-5-thinking-pro |  0.055  |   0.085   |    0     |  0.904  |         0           |  0.176   |     0      |      0.166       | 0.766 | 0.95 |\n| DeepSeek-R1             |  0.333  |   0.200   |   0.208  |  0.636  |         0           |  0.307   |     0      |      0.133       | 0.666 | 0.966 |\n| Qwen3-32B-thinking      |  0.071  |   0.080   |    0     |  0.909  |         0           |  0.321   |     0      |      0.117       | 0.888 | 0.729 |\n| o3‑mini                 |  0.956  |   0.000   |    0     |  0.948  |         0           |  0.617   |     0      |      0.166       | 0.714 | 0.976 | \n| Claude-3.7-thinking     |  0.441  |   0.200   |   0.312  |  0.461  |         0           |  0.521   |     0      |      0.222       | 1 | 1.05 |\n| Average | 0.371 | 0.113 | 0.104 | 0.771 | 0 | 0.388 | 0 | 0.160 | 0.806 | 0.934 |\n\n>From the above results, we can draw the following preliminary conclusions:\n\n> - For **Question 1**, we observe that: \n\t-  For games explicitly **trained on by Doubao-1-5-thinking-pro** (Sudoku and Maze), the β values are indeed higher but remain at a moderate level among the sampled models, which is within an acceptable range; \n\t-  For **classic games** (Snake and 8-Puzzle), while Doubao-1-5-thinking-pro demonstrates **strong performance** (ranking 3rd on Snake and 4th on 8-Puzzle among all models in the main experiments), its β values remain the **lowest** among the sampled models. This observation supports the view that reinforcement learning (RL) processes genuinely enhance a model’s reasoning capabilities rather than simply enabling it to memorize training questions or regurgitate pretraining knowledge, thus demonstrating **strong generalization**. \n\t-  For games **originally developed or adapted** by our engineering team, all models exhibit very low β values, and given that at least **80%** of KORGYM games are original or adapted, this further validates the effectiveness of KORGYM.\n\n> - For **Question 2-1**, we confirm that the original/adapted games (80%+) enable Doubao-1-5-thinking-pro to maintain relatively low β values. Even for classic games, the β values remain at a low level. Hence, we estimate that the overlap between games and Doubao’s RL training set does **not exceed 15%**. In future work, we will leverage this methodology for further experiments to determine a more precise overlap ratio.\n\n> - From a **methodological perspective**, we believe that this Knowledge Leakage Coefficient evaluation framework can effectively reveal the proportion of data types used in the LLM training process. For instance, although the training data composition is not explicitly disclosed in the reports, our experimental results suggest that **o3-mini likely involved Snake-related data and Doubao-1-5-thinking-pro likely involved Sokoban-related data**. Moving forward, we plan to further develop this methodology to uncover additional interesting insights.\n\n---\n\n**Response to Question 2-2**\n\n(2) **Question 2**: Whether there is an ablation isolating the RL impact from other improvements in the \"thinking\" model variant?\n\n(2) **Answer 2**: \n\nWe sincerely apologize for the oversight regarding the ablation analysis on RL impact from other improvements in the \"thinking\" model variant. Here, we provide additional supplementary analysis. To isolate the effect of RL training from other structural or training environment factors contributing to the performance difference from the non-thinking to the thinking variants of the model, we selected three LLMs that offer both non-thinking and thinking modes. For Gemini-2.0-Flash and Claude-3.7, the results under both modes are already reported in Table 1 and Figure 5 of our paper. The final experimental results are as follows:\n\n\n\n|    Model    |  Snake  |  Long Cat |  Black White Copy |Emoji Connect| Word Transformation | \n|:-----------:|:-------:|:---------:|:--------:|:--------:|:-------------------:|\n| Qwen3-32B-nonthinking    |  14.6  |   0.42   |   0.32  |  0.32  |         0.34           |  \n| Qwen3-32B-thinking |  13.9(-0.7)  |   0.62(+0.2)    |    0.34 (+0.02)    |  0.48 (+0.16) |         0.48 (+0.14)          | \n| Claude3.7-nonthinking         |  6.9  |  0.14   |    0.02     |  0.16  |        0.22           | \n| Claude3.7-thinking      |  9.75 (+2.85) |   0.32 (+0.18)  |    0.18 (+0.16)    |  0.76 (+0.6)  |         0.56  (+0.34)        |  \n| Gemini-2.0-Flash-nonthinking     |  2.05  |   0.1   |  0.02  |  0.12  |         0.12           |  \n| Gemini-2.0-Flash-thinking     |  1.85 (-0.2) |   0.08 (-0.02)  |  0.16(+0.14)  |  0.18 (+0.06) |         0.18   (+0.06)        |  \n\nFrom these results, we observe that the **thinking variants of the same model consistently outperform their non-thinking counterparts**, which supports the **validity of the performance improvements attributed to the transition from the non-thinking to the thinking variant of the model**.\n\n---\n\nFinally, we sincerely thank you for your valuable feedback on KORGYM again, and we look forward to further discussions with you."}, {"review_id": "cC7Pa4R9qa", "reviewer": "Reviewer_3dhT", "summary": "This paper introduces a new benchmark, namely KORGym, which evaluates multiple dimensions of reasoning abilities in an LLM. KORGym is based on a dataset of over fifty games. KORGym consists of separate modules, mainly to parse inputs, initialise game environments and model inference. KORGym uses a detailed scoring scheme for different types of games, such as for single objective games, 1 point is assigned for success and 0 for failure, and for games that award incremental points, they accumulate all points. After this, KORGym uses \"Capability Dimension Aggregated Mean\" which is an aggregation metric that takes into account different ranges of scores and aggregates it semantically. KORGym provides results on many LLMs or all the reasoning tasks. Furthermore, provides some results on multimodal reasoning.", "questions": "None. I have no questions, as the paper is quite nicely presented. It would be nice to simplify the Figure 2. I understand the effort put into it. I also understand that there are a lot of pieces that are important and should be there. It would also be nice for the paper to include the above mentioned citations, plus more citations especially in LLMs for Gaming subsection, as the field has matured to a certain extent and many readers want to explore more papers through your paper. I am willing to change the score to strong accept as I foresee many readers coming to your paper, a lot of them will be skimming and they do need a simplified flow diagram and more papers to read from your paper.", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n- Very neatly written paper.\n- Precise, to the point description of the framework.\n- Brings in over 50 present games to test multiple reasoning fronts: mathematical and logical reasoning, control interaction reasoning, puzzle reasoning, spatial and geometric reasoning, strategic reasoning and multimodal reasoning.\n- The evaluation metric is well-derived. Takes into account that different games have different scoring schemes. Provides fair evaluation scheme.\n\n**Weaknesses:**\n\n- Figure 2 is not straightforward to understand. I appreciate the effort put into it but simplifying it will help the paper.\n- For related work I believe a few citations are a must: PlanBench and the works that followed it [1, 2], as it is one of the first works to evaluate reasoning in LLMs, and GameTraversalBenchmark [3], as it evaluates spatial reasoning and is the first benchmark to evaluate LLMs on games.\n\n[1] Valmeekam, K., Marquez, M., Olmo, A., Sreedharan, S. and Kambhampati, S., 2023. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Neural Information Processing Systems, 36, pp.38975-38987.\n\n[2] Valmeekam, K., Stechly, K. and Kambhampati, S., 2024. LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench. arXiv preprint arXiv:2409.13373.\n\n[3] Nasir, M.U., James, S. and Togelius, J., 2024. GameTraversalBenchmark: Evaluating Planning Abilities Of Large Language Models Through Traversing 2D Game Maps. arXiv preprint arXiv:2410.07765.", "was_revised": true, "final_justification": "I think the paper is a very solid improvement to the field. Hence, I have improved my scores.", "rebuttal": "We sincerely appreciate your recognition and support for our work, and we deeply apologize for the oversight regarding the Figure and related work sections in the paper.\n\n---\n\n**Response to Weakness 1**\n\n(1) **Question 1**: Figure 2 is not straightforward to understand. I appreciate the effort put into it but simplifying it will help the paper.\n\n(1) **Answer 1**:\n\nWe have simplified the content of the figure to enhance readability and provided additional explanations and supplements in the body text. We will update these adjustments once the paper is accepted.\n\n**Response to Weakness 2**\n\n---\n\n(2) **Question 2**: For related work I believe a few citations are a must: PlanBench and the works that followed it [1, 2], as it is one of the first works to evaluate reasoning in LLMs, and GameTraversalBenchmark [3], as it evaluates spatial reasoning and is the first benchmark to evaluate LLMs on games.\n\n(2) **Answer 2**: \n\n\nWe sincerely appreciate your insightful feedback on the shortcomings in our related work section and are grateful for recommending exceptionally engaging and inspiring studies. We found the PlanBench [1] and the accompanying Evaluation work [2], along with the GameTraversalBenchmark [3] that you suggested, to be truly fascinating and substantial contributions to the field. Reading these works significantly enriched our understanding and provided invaluable inspiration during the revision of our paper. We have eagerly integrated the relevant citations into our updated related work section, and we will reflect these enhancements comprehensively in the final version upon acceptance. Moreover, we intend to actively reference these insightful studies in our future research endeavors and surveys, thereby amplifying their visibility and sharing their valuable contributions with a broader scholarly audience.\n\n---\n\nOnce again, thank you very much for your support and constructive suggestions on KORGYM. We look forward to further discussions with you."}, {"review_id": "i9TONYMvJ3", "reviewer": "Reviewer_SShB", "summary": "This paper present KORGym, a new benchmark of over fifty games targeting knowledge-orthogonal evaluation of LLM reasoning abilities. The benchmark categorizes the games along six reasoning dimensions, and supports an aggregated mean metric providing a normalized capability-wise evaluation of reasoning. The paper provides evaluation of many widely used LLMs, and informative analyses are provided.", "questions": "- Could the authors show correlation with other tasks or benchmarks which feature more difficult or specialized tasks? For example control and interaction could measure correlation with alfworld or minecraft.\n- How is the reasoning paradigm disabling achieved?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The benchmark is well motivated and well designed. Due to many existing benchmarks becoming saturated due to pretrained knowledge leakage, it has value in helping assess the raw reasoning capability of models.\n- The benchmark is not saturated, particularly by open-source models, and it could also be interesting to see agent scaffolds evaluated on this benchmark.\n- The analyses of reasoning paradigms is particularly interesting, showing that model series' exhibit paradigm preferences, and some reasoning paradigms like code-based reasoning, can even partially constrain performance. This also enables interesting observations e.g. that the strongest model tested, o3-mini, does not exhibit this bias. \n\nWeaknesses\n- The core claim that the benchmark is knowledge-orthogonal is not validated.\n- The control, interaction, and task reasoning part of the benchmark feels weak, as it consists of relatively simple tasks in terms of control and interaction complexity.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns, which were about 1) the core claim of knowledge-orthogonality of the games, and 2) the lack of popular complex benchmarks for agents like Minecraft. The author response shows that the benchmark is overall sufficiently knowledge-orthogonal and leakage happens in certain recent models like o3-family, which is interesting in itself, and likely to provide new insights into the performance of latest agents. The authors also show a correlation study indicating that, taken together the benchmark could capture capabilities required for more complex benchmarks like Minecraft. Overall the paper makes meaningful contributions in improving the evaluation of reasoning agents, so I have give an accept rating.", "rebuttal": "Thank you very much for your valuable feedback and suggestions on KORGYM. We sincerely apologize for the shortcomings in our work and have conducted a series of experiments and improvements to address these concerns:\n\n---\n\n**Response to Weakness 1**\n\n(1) **Question 1**:The core claim that the benchmark is knowledge-orthogonal is not validated.\n\n(1) **Answer 1**:\n\nAlthough we formally defined the concept of KOR in Appendix C of our submitted paper, it did not permit empirical quantification of pre-trained knowledge leakage, thus failing to validate our core claim of knowledge orthogonality. To rectify this, we have extended the formal definition and introduced the Knowledge Leakage Coefficient, which can be empirically calculated through experiments, to further substantiate the validity of KORGYM.\n\n**1.1 Notational Definitions**\n\nFor a game within KORGYM, the required reasoning information comprises:\n\n- **$K$**: General background or domain‑specific knowledge acquired during pre‑training and post-training.  \n- **$T$**: Single‑turn or multi‑turn tasks that must be completed during gameplay.  \n- **$R$**: Core game‑rule information specifically designed for solving task $T$.  \n- **$A$**: The set of actions generated by the LLM to solve task $T$.\n- **$\\rightarrow$**: The reasoning and interaction process whereby the LLM gradually derives the action set $A$ from the game task $T$.  \n- **$S$**: The final score obtained by the LLM's interaction with the game environment based on reasoning from $R$ and/or $K$:\n  - $S(T \\rightarrow A \\mid K)$: Score from actions derived solely from pre‑trained knowledge $K$.  \n  - $S(T \\rightarrow A \\mid R, K)$: Score from actions derived using both game‑rule knowledge $R$ and pre‑trained knowledge $K$.  \n\n---\n\n**1.2 Analysis of Game Redundancy**\n\n1. **Knowledge‑Rule Decoupling**  \n   Ideally, the game rule information $R$ is logically self‑contained and independent of pre‑trained knowledge $K$:  \n   $$\n   R \\perp K\n   $$\n\n2. **Knowledge Leakage Coefficient**  \n   The Leakage between $K$ and $R$ is quantified by the Knowledge Impact Coefficient $\\beta$:\n   $$\n   \\beta\n   = \\frac{S(T \\rightarrow A \\mid K)}{S(T \\rightarrow A \\mid R, K)},\n   \\quad \\beta \\in [0,1].\n   $$\n  A smaller $\\beta$ indicates less overlap between the LLM’s pre-trained knowledge and the game rules. Consequently, the LLM cannot achieve optimal scores without explicitly relying on the game rules provided in the prompt.\n\n\n---\n\n**1.3 Quantitative Experiment on Knowledge Leakage**\n\nBased on the above definition, we select a subset of games from KORGYM, including some classic games (Snake, Sokoban, and 8-puzzle) and original/adapted games (Pipe Game, Long Cat, ...). We conduct an ablation study in which the core game‑rule information (R) was removed and then compute the $\\beta$:\n\n\n|    Model    |  Snake  | Pipe Game | Long Cat | Sokoban | Word Transformation | 8‑puzzle | Play Lines | Black White Copy |\n|:-----------:|:-------:|:---------:|:--------:|:--------:|:-------------------:|:--------:|:----------:|:----------------:|\n| Doubao-1-5-thinking-pro |  0.055  |   0.085   |    0     |  0.904  |         0           |  0.176   |     0      |      0.166       |\n| DeepSeek-R1             |  0.333  |   0.200   |   0.208  |  0.636  |         0           |  0.307   |     0      |      0.133       |\n| Qwen3-32B-thinking      |  0.071  |   0.080   |    0     |  0.909  |         0           |  0.321   |     0      |      0.117       |\n| o3‑mini                 |  0.956  |   0.000   |    0     |  0.948  |         0           |  0.617   |     0      |      0.166       |\n| Claude-3.7-thinking     |  0.441  |   0.200   |   0.312  |  0.461  |         0           |  0.521   |     0      |      0.222       |\n| Average | 0.371 | 0.113 | 0.104 | 0.771 | 0 | 0.388 | 0 | 0.160 |\n\n\n>From the above results, we can draw the following preliminary conclusions:\n\n> - **Model perspective:** Models such as **o3‑mini** and **DeepSeek‑R1** exhibit relatively notable knowledge leakage, whereas other models (e.g., **Doubao‑1‑5‑thinking‑pro**) demonstrate comparatively lower leakage. This observation supports the view that **RL processes genuinely enhance a model's reasoning capabilities, rather than merely enabling it to memorize training questions**.\n> -  **Game perspective:** Knowledge leakage is more prominent in **classic games**, while the corresponding β values for **original/adapted games** remain at a relatively low level. Given that more than **80%** of the games in KORGYM are originally developed or adapted by our engineering team, this strongly supports **the validity and reliability of KORGYM**.\n> -  **Methodological perspective:** We believe that the knowledge leakage quantification experiment itself offers an interesting insight and methodology for **analyzing models’ genuine reasoning abilities and uncovering information contained within the pre-training data**. \n\nThus, we have demonstrated the effectiveness of KORGYM in achieving knowledge orthogonality. Additionally, we plan to introduce more original and adapted games in the future and exclude games with higher β values to further enhance the degree of knowledge orthogonality in KORGYM.\n\n---\n\n**Response to Weakness 2**\n\n(2) **Question 2**:The control, interaction, and task reasoning part of the benchmark feels weak, as it consists of relatively simple tasks in terms of control and interaction complexity.\n\n(2) **Answer 2**:\n\nDue to time constraints and considerations for overall system coherence, we regretfully did not include highly complex games, such as Minecraft, within the control, interaction, and task reasoning components.  We aimed to address these reasoning dimensions by decomposing them into specific sub-capabilities, each assessed through dedicated games in KORGYM. For example:\n\n> -  **Minigrid** evaluates the model's capability to interact with environments using tools.\n> -  **PVZ** assesses the model's capacity for resource management and allocation (e.g., sunlight) within dynamic and continuous environments, as well as its multi-step strategic planning based on opponent progression.\n> -  **Minesweeper** examines logical reasoning, probabilistic estimation, and risk assessment abilities under incomplete information.\n\nBy decomposing these dimensions into sub-capabilities and designing corresponding games, we provided a comprehensive evaluation framework for control, interaction, and task reasoning. Furthermore, in *Response to Question 1*, we quantitatively demonstrated the effectiveness of this approach through Spearman coefficient analysis.\n\n---\n\n**Response to Question 1**\n\n(3) **Question 3**: Could the authors show correlation with other tasks or benchmarks which feature more difficult or specialized tasks? For example control and interaction could measure correlation with alfworld or minecraft.\n\n(3) **Answer 3**:\n\nTo further analyze the correlation with other tasks or benchmarks featuring more difficult or specialized tasks, we conducted a Spearman correlation analysis with the following domain-specific benchmarks/projects:\n\n> -  MathArena: Specialized assessment of model capabilities in mathematics;\n> -  BBEH: An extended benchmark for BBH proposed by Google;\n> -  MC Bench: Evaluates the model's ability to construct structures described in prompts using Minecraft blocks;\n> -  OmniSpatial: A benchmark designed for comprehensive spatial reasoning evaluation;\n> -  lmARENA-Vision: A comprehensive benchmark assessing visual reasoning capabilities;\n> -  Ace Attorney : A project leveraging the classic game Ace Attorney to test interaction and reasoning capabilities of LLMs;\n> -  BizFinBench: A benchmark for evaluating long-term strategic planning capabilities in financial contexts;\n> -  SnakeBench: A project employing LLM versus LLM scenarios to evaluate competitive gaming capabilities.\n\nThe corresponding meanings of the Capability Dimension fields are as follows:\n> -  **MLR** : Mathematical and Logical Reasoning\n> -  **CIR** : Control Interaction Reasoning\n> -  **PR**: Puzzle Reasoning \n> -  **SGR**: Spatial and Geometric Reasoning\n> -  **SR**: Strategic Reasoning\n> -  **MR**: Multimodal Reasoning\n\nThe results of our analysis are as follows:\n\n\n| Capability Dimension| MLR  |         MLR          | CIR |       CIR        |     PR       |   PR      | SGR |   SGR     | SR     | SR     | MR | \n|-------------|:---------------------------------:|:-------------------------------:|:----------------------------:|:-------------:|:---------------------------:|:-------:|:-----------------------------:|:----------:|:----------:|:----------:|:----------:|\n|   **Related Benchmark**    | MathArena                         | BBEH‑TimeArithmetic            | MC Bench                     | Ace Attorney  | BBEH‑Boolean Expressions    | BBEH‑BoardGameQA | BBEH‑Spatial Reasoning       | OmniSpatial | BizFinBench | SnakeBench | lmARENA-Vision |\n| **Spearman**| 0.866                             | 0.899                           | 0.828                        | 0.666         | 0.700                       | 0.600   | 0.899                         | 0.700      | 0.733 | 0.761 | 0.899|\n\nThus, it can be inferred that KORGYM truly has a correlation with specialized tasks, and can reliably reflect the true reasoning capabilities of models across various dimensions.\n\n---\n\n**Response to Question 2**\n\n(4) **Question 4**: How is the reasoning paradigm disabling achieved?\n\n(4) **Answer 4**:\n\nTo implement the reasoning paradigm disabling, we added explicit instructions in the game rules prohibiting the use of specific reasoning paradigms as auxiliary methods. We also provided illustrative examples, such as:\n```python\n\"Remember, you cannot use the Code analysis method to help you solve the problem, e.g., \"import math; a = 0; for i in range(...): ...\" is forbidden.\"\n```\nFinally, we sincerely thank you once again for your valuable feedback and suggestions on KORGYM, and we look forward to further discussions with you."}]

Author rebuttal:

Meta-review: This paper presents KORGym, a dynamic, game-based benchmark for evaluating reasoning in LLMs and VLMs. The benchmark is well-motivated, offering over 50 interactive tasks across diverse reasoning dimensions, with support for multi-turn and RL settings. Reviewers highlighted its novelty, breadth, and strong empirical evaluation across many models, noting that it fills an important gap in assessing reasoning beyond domain-specific tests. I recommend acceptance.
Decision: Accept (spotlight)