Title: KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM; Evaluation; RL; Game
Abstract: Recent advancements in large language models (LLMs) underscore the need for more comprehensive evaluation methods to accurately assess their reasoning capabilities. Existing benchmarks are often domain-specific and thus cannot fully capture an LLM’s general reasoning potential. To address this limitation, we introduce the **Knowledge Orthogonal Reasoning Gymnasium (KORGym)**, a dynamic evaluation platform inspired by KOR-Bench and Gymnasium. KORGym offers over fifty games in either textual or visual formats and supports interactive, multi-turn assessments with reinforcement learning scenarios. Using KORGym, we conduct extensive experiments on 19 LLMs and 8 VLMs, revealing consistent reasoning patterns within model families and demonstrating the superior performance of closed-source models. Further analysis examines the effects of modality, reasoning strategies, reinforcement learning techniques, and response length on model performance. We expect KORGym to become a valuable resource for advancing LLM reasoning research and developing evaluation methodologies suited to complex, interactive environments.


Peer reviews:
comments: SUMMARY:
This paper present KORGym, a new benchmark of over fifty games targeting knowledge-orthogonal evaluation of LLM reasoning abilities. The benchmark categorizes the games along six reasoning dimensions, and supports an aggregated mean metric providing a normalized capability-wise evaluation of reasoning. The paper provides evaluation of many widely used LLMs, and informative analyses are provided.

STRENGTHS AND WEAKNESSES:
Strengths
- The benchmark is well motivated and well designed. Due to many existing benchmarks becoming saturated due to pretrained knowledge leakage, it has value in helping assess the raw reasoning capability of models.
- The benchmark is not saturated, particularly by open-source models, and it could also be interesting to see agent scaffolds evaluated on this benchmark.
- The analyses of reasoning paradigms is particularly interesting, showing that model series' exhibit paradigm preferences, and some reasoning paradigms like code-based reasoning, can even partially constrain performance. This also enables interesting observations e.g. that the strongest model tested, o3-mini, does not exhibit this bias. 

Weaknesses
- The core claim that the benchmark is knowledge-orthogonal is not validated.
- The control, interaction, and task reasoning part of the benchmark feels weak, as it consists of relatively simple tasks in terms of control and interaction complexity.

QUESTIONS:
- Could the authors show correlation with other tasks or benchmarks which feature more difficult or specialized tasks? For example control and interaction could measure correlation with alfworld or minecraft.
- How is the reasoning paradigm disabling achieved?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new benchmark, namely KORGym, which evaluates multiple dimensions of reasoning abilities in an LLM. KORGym is based on a dataset of over fifty games. KORGym consists of separate modules, mainly to parse inputs, initialise game environments and model inference. KORGym uses a detailed scoring scheme for different types of games, such as for single objective games, 1 point is assigned for success and 0 for failure, and for games that award incremental points, they accumulate all points. After this, KORGym uses \"Capability Dimension Aggregated Mean\" which is an aggregation metric that takes into account different ranges of scores and aggregates it semantically. KORGym provides results on many LLMs or all the reasoning tasks. Furthermore, provides some results on multimodal reasoning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- Very neatly written paper.\n- Precise, to the point description of the framework.\n- Brings in over 50 present games to test multiple reasoning fronts: mathematical and logical reasoning, control interaction reasoning, puzzle reasoning, spatial and geometric reasoning, strategic reasoning and multimodal reasoning.\n- The evaluation metric is well-derived. Takes into account that different games have different scoring schemes. Provides fair evaluation scheme.\n\n**Weaknesses:**\n\n- Figure 2 is not straightforward to understand. I appreciate the effort put into it but simplifying it will help the paper.\n- For related work I believe a few citations are a must: PlanBench and the works that followed it [1, 2], as it is one of the first works to evaluate reasoning in LLMs, and GameTraversalBenchmark [3], as it evaluates spatial reasoning and is the first benchmark to evaluate LLMs on games.\n\n[1] Valmeekam, K., Marquez, M., Olmo, A., Sreedharan, S. and Kambhampati, S., 2023. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Neural Information Processing Systems, 36, pp.38975-38987.\n\n[2] Valmeekam, K., Stechly, K. and Kambhampati, S., 2024. LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench. arXiv preprint arXiv:2409.13373.\n\n[3] Nasir, M.U., James, S. and Togelius, J., 2024. GameTraversalBenchmark: Evaluating Planning Abilities Of Large Language Models Through Traversing 2D Game Maps. arXiv preprint arXiv:2410.07765.\n\nQUESTIONS:\nNone. I have no questions, as the paper is quite nicely presented. It would be nice to simplify the Figure 2. I understand the effort put into it. I also understand that there are a lot of pieces that are important and should be there. It would also be nice for the paper to include the above mentioned citations, plus more citations especially in LLMs for Gaming subsection, as the field has matured to a certain extent and many readers want to explore more papers through your paper. I am willing to change the score to strong accept as I foresee many readers coming to your paper, a lot of them will be skimming and they do need a simplified flow diagram and more papers to read from your paper.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces KORGym, a novel interactive benchmark designed to evaluate the reasoning capabilities of large language models (LLMs). KORGym consists of 50 games, most of which do not require external knowledge, hence the term Knowledge Orthogonal. The benchmark includes both textual and visual games. A wide range of LLMs is evaluated on this benchmark, and the results provide valuable insights into the reasoning behavior of modern models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The proposed benchmark is highly relevant and timely for evaluating current SOTA LLMs. Unlike many existing benchmarks, KORGym is less susceptible to data contamination and better suited to test strategic thinking and planning.\n\n- The evaluation section is particularly strong. It includes:\n\n-- Analysis of reasoning facets in LLMs\n\n-- Insights into the interplay between textual and visual reasoning\n\n-- Clustering of models based on performance profiles\n\n-- Detailed ablation studies and thinking pattern analysis\n\n**Weaknesses:**\n\n- The section titled “Impact of Reinforcement Learning” lacks clarity:\n\n-- It is unclear how much overlap exists between the KORGym task set and the training tasks used for the Duobao RL training.\n\n-- Without a well-controlled ablation, it's difficult to conclude whether the performance improvement from the non-thinking to thinking variant of the model is due to RL training alone, or to other architectural or training changes.\n\nQUESTIONS:\n1. It would be important to understand how much training on the same game set influences performance.  \n   For instance, the paper notes that the Duobao model was trained on classical games such as Sudoku.  \n   **Question:** Is it fair to compare Duobao's performance on these games with models that have *not* seen such games during pretraining?\n\n2. In the *Impact of RL* section, could the authors clarify:\n   - What fraction of KORGym games (if any) overlaps with Duobao's RL training set?\n   - Whether there is an ablation isolating the RL impact from other improvements in the \"thinking\" model variant?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper introduces KORGym, a game-based LLMs and VLMs evaluation benchmark. The authors design 56 games across 6 reasoning dimensions and supports multi-turn, interactive assessments. The authors conduct large-scale evaluation on 19 LLMs (including 11 thinking models) and 8 VLMs. The results demonstrate KORGym’s utility for evaluating and challenging LLMs on knowledge-orthogonal reasoning abilities.

STRENGTHS AND WEAKNESSES:
Strengths:
Novel Benchmark: KORGym offers a diverse set of games to evaluate the reasoning abilities of LLMs and VLMs.
Broad Scope: KORGym introduces a large suite of games spanning six key reasoning dimensions. Also, the authors design evaluation method beyond binary scoring.
Ready-to-Use Implementation: The authors not only provide the evaluation datasets and metrics but also deliver optimized evaluation code. This significantly lowers the barrier for adoption and ensures reproducibility.
Strong Empirical Study: The experiments involve a broad collection of current LLMs and VLMs, with both closed- and open-source models. This enables comprehensive comparison and drives key findings on strengths, weaknesses and behavioral tendencies between model series.

Weaknesses:
Potential Domain and Task Distribution Bias: Although the knowledge-orthogonality concept aims to decouple evaluation from prior knowledge, the game/task design process is not fully transparent regarding avoidance of distributional overlap with LLM pre-training corpora. The paper does not concretely quantify the degree of knowledge leakage, which may influence some of the comparative results or limit conclusions about pure reasoning.

QUESTIONS:
Check the weakness part mentioned above

LIMITATIONS:
Check the weakness part mentioned above
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-06