Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning

## Abstract

Abstract Vision-language reinforcement learning (RL) has primarily focused on narrow domains (e.g. geometry or chart reasoning). This leaves broader training scenarios and resources underexplored, limiting the exploration and learning of Vision Language Models (VLMs) through RL. We find video games inherently provide rich visual elements and mechanics that are easy to verify. To fully use the multimodal and verifiable reward in video games, we propose Game-RL, constructing diverse game tasks for RL training to boost VLMs general reasoning ability. To obtain training data, we propose Code2Logic, a novel approach that adapts game code to synthesize game reasoning task data, thus obtaining the GameQA dataset of 30 games and 158 tasks with controllable difficulty gradation. Unexpectedly, RL training solely on GameQA enables multiple VLMs to achieve performance improvements across 7 diverse vision-language benchmarks, demonstrating the value of Game-RL for enhancing VLMs’ general reasoning. Furthermore, this suggests that video games may serve as valuable scenarios and resources to boost general reasoning abilities.

## 1 Introduction

Vision language models have achieved impressive progress in basic tasks such as image description and vision question answering. However, they still struggle with diverse and complex tasks that require multi-step reasoning in real-world scenarios [18, 33]. One key reason is that vision-language RL primarily focused on narrow domains, mainly including geometry [22, 35] and chart reasoning [8], as shown in Table [1](https://arxiv.org/html/2505.13886v6#S1.T1). This leaves broader training scenarios underexplored, limiting the exploration and learning of VLMs through RL [18, 14, 10, 36].

We recognize that video games inherently have three advantages: First, they can provide rich visual elements and scenes with texts. Second, their mechanics are easy to verify, so we can synthesize reliable tasks with verifiable results. Third, their environments are fully controllable and easy to modify, providing convenience for controlling the difficulty.
Althrough existing works found game is a good arena to evaluate VLMs reasoning ability [33, 21, 32, 13], they did not apply training on it. One possible reason is they did not transform game data into Visual Question and Answer (VQA) format to adapt to training, as shown in Table [1](https://arxiv.org/html/2505.13886v6#S1.T1).

To fully use the multimodal and verifiable rewards in game, we propose Game-RL, constructing game scenarios for RL training to boost VLMs general reasoning ability. Then we propose Code2Logic, a novel approach adapts game code to synthesize game reasoning task data, as shown in Figure [1](https://arxiv.org/html/2505.13886v6#S2.F1). Code2Logic establishes mapping from game code to reasoning logic. Using Code2Logic, we have constructed the GameQA, which is a Visual question Answer (VQA) dataset to train and evaluate the reasoning capabilities of VLMs. GameQA includes 30 games, 158 tasks and 140K questions with controllable difficulty gradation, covering 4 cognitive categories, as shown in Figure [2](https://arxiv.org/html/2505.13886v6#S2.F2) and Figure [6](https://arxiv.org/html/2505.13886v6#S7.F6).
Unexpectedly, despite training solely on game tasks, using Group Relative Policy Optimization (GRPO) [7], several SOTA open-source VLMs demonstrated out of domain generalization, specifically Qwen2.5-VL-7B achieving 2.33% improvement across 7 diverse vision-language benchmarks.

**Table 1: Our GameQA dataset extends RL training scenarios for VLMs to the domain of video games, providing diverse verifiable game tasks along with controllable difficulty. Comparison of existing visual reasoning datasets. Size means sum of the number of VQA pairs in train set and test set. Ration. Annot. means has annotated reasoning process. 3D Scene means some games of it are 3D scene. Detailed introduction can be found in Section [6](https://arxiv.org/html/2505.13886v6#S6).**
| Reasoning | Related Work | Train | Game | Ration. | 3D | Adjustable |
| --- | --- | --- | --- | --- | --- | --- |
| Domain |  | Set | Count | Annot. | Scene | Difficulty |
| Math | MAVIS (Zhang et al. [35]) | ✔ | N/A | ✔ | ✘ | ✘ |
| MultiMath (Peng et al. [22]) | ✔ | N/A | ✔ | ✘ | ✘ |  |
| Geo170k (Gao et al. [6]) | ✔ | N/A | ✔ | ✘ | ✔ |  |
| MathV360K (Jiang et al. [11]) | ✔ | N/A | ✔ | ✘ | ✔ |  |
| Game | ING-VP (Zhang et al. [33]) | ✘ | 6 | ✘ | ✘ | ✔ |
| BALROG (Paglieri et al. [21]) | ✘ | 6 | ✘ | ✘ | ✘ |  |
| VideoGameBench | ✘ | 23 | ✘ | ✔ | ✘ |  |
| (Zhang et al. [32]) |  |  |  |  |  |  |
| VCbench (Li et al. [13]) | ✘ | 10 | ✘ | ✘ | ✔ |  |
| GameQA (Ours) | ✔ | 30 | ✔ | ✔ | ✔ |  |

Our main contributions are as follows:

- •
We propose Game-RL, constructing game tasks for RL training to boost VLMs general reasoning ability. To obtain training data, We propose Code2Logic, a novel approach that adapts game code to synthesize diverse game reasoning task data.
- •
Using Code2Logic, we developed the GameQA dataset, which includes 30 games, 158 tasks and 140K questions with controllable difficulty gradation.
- •
Our experiments show that multiple VLMs achieve performance improvements across 7 diverse vision-language benchmarks through RL training solely on GameQA, demonstrating the value of Game-RL for enhancing VLMs’ general reasoning. This also suggests that video games may serve as valuable scenarios and resources to boost general reasoning abilities.

## 2 Code2Logic: Synthesizing Multimodal Verifiable Game Data

We propose Code2Logic to synthesize verifiable game reasoning tasks data via adapting game code. Code2Logic comprises three core steps: game code construction, task template design, and data engine construction for dataset generation, as shown in Figure [1](https://arxiv.org/html/2505.13886v6#S2.F1). Code2Logic establishes a mapping from game code to reasoning logic.

Figure: Figure 1: Overview of Code2Logic approach. The process involves three main steps: (1) using LLMs to construct game code. (2) LLM-assisted design of the task templates including question and analysis templates based on the generated game code. Each task template condenses one type of reasoning pattern in the game. (3) Using LLMs to construct a data engine that directly reuses the core game code from the first step, including functions like move. (4) After these main steps, the data engine is executed to fill in the task templates developed in Step 2 and generate data samples, as illustrated in the Final Result Section.
Refer to caption: x1.png

### 2.1 Game code construction

The first step is to construct the code of the target video game. Game code defines the state space and contains some core functions encoding the transition rules of the game. These functions can be reused and adapted for data engine construction.

Taking Sokoban as an example, it is simple enough to be easily built with one-line prompt using LLMs such as Claude 3.5 and GPT-4o. Sokoban game state is only composed of wall, player, box and target, while action only includes moving in four directions. The prompt used for game code generation and pseudocode of the game is illustrated in Figure [1](https://arxiv.org/html/2505.13886v6#S2.F1) (Step 1). Meanwhile, the Sokoban move function within Sokoban code can inspire the "State Prediction" questions, such as "Predict where will player end up after these steps", and this function can be reused for data engine construction.

### 2.2 Game task and QA Template design

The second step is to design game task and question-answer templates (QA Templates) according to each video game. Game task design base on the visual element and action space of video game generated in the first step. Taking Sokoban as an example, we can design a task about predicting the position of player after moves according to a game state screen, so we can propose a question and an answer. The question is "If the player moves left, up, down, down, where is the player?" and the answer is "The position of player is (2,2)." Then we can generalize these questions and answers into a QA Template, as shown in figure [1](https://arxiv.org/html/2505.13886v6#S2.F1) (Step 2). Meanwhile, we can fill contents into a QA Template to get many instances. Detailed QA Templates of Sokoban is provided in Appendix [14.1](https://arxiv.org/html/2505.13886v6#S14.SS1).

In GameQA dataset, each task is defined as a group of samples from the same game that share the same QA template. We categorize the reasoning tasks into three types, including "Target Perception Task", "State Prediction Task" and "Strategy Optimization Task", with the detailed descriptions shown in Appendix [12.1](https://arxiv.org/html/2505.13886v6#S12.SS1).

For a task, we use LLMs to assist the design and refinement of the QA template, an example prompt for such refinement is illustrated in Figure [1](https://arxiv.org/html/2505.13886v6#S2.F1) (Step 2). Additionally, LLMs can also design tasks, with example prompt shown in Appendix LABEL:app:supp_prompt.
In summary, designing a QA template is equivalent to extracting one type of reasoning pattern from the video game code obtained in the first step. The third step then further instantiates these reasoning task templates into a dataset.

Figure: Figure 2: Four game examples from GameQA: 3D Reconstruction, Tangram, Sudoku, and Sokoban, each representing distinct cognitive categories. Each game displays two VQA examples consisting of: (a) current game state visualization, (b) a targeted question, and (c) step-by-step reasoning with the answer. GameQA transforms complex game-playing tasks into this structured VQA format. See Appendix LABEL:app:games_example_samples for more VQA examples of some representative games.
Refer to caption: x2.png

### 2.3 Data engine construction for data samples generation

The third step is data engine construction for samples generation. The data engine is a program that, when executed, automatically generates task instances in batch according to task templates. LLMs are used to assist in constructing the data engine’s code based on the game code obtained in the first step. The prompt for guiding LLMs to construct the data engine code is shown in Figure [1](https://arxiv.org/html/2505.13886v6#S2.F1) (step 3). The pseudocode for the data engine is shown in Figure [1](https://arxiv.org/html/2505.13886v6#S2.F1) (step 3). Taking Sokoban’s state prediction task as an example, the data engine program is constructed from four core modules:

- •
Game environment initialization: This module randomly generates Sokoban environment by directly reusing the initialization functions from the Step 1’s game code. Sokoban environment includes the position of player, boxes, goals and walls.
- •
Proposing task instance: This module proposes a state prediction task instance by generating a multi-step random movement sequence.
- •
Solving task instance: This module generates a solution using a corresponding algorithm. For state prediction task, the algorithm simulates each action by reusing the movement logic from game code of Step 1, which handles all situations of movement including collisions and box pushing.
- •
QA data construction: This module fills the task templates. The movement sequence will be filled into the question template to become a question instance. The state transition trajectory will be filled into the answer template to become an answer instance.

### 2.4 Data quality assessment and data augmentation

We implemented manual quality verifications at each step of our method.

Game code verification: We verify the correctness of the game code generated by the models by manually running the game program. If errors are found, including vision issues in the game display or code logic errors, they are fed back to the LLM for regeneration. For complex game features that LLMs cannot generate correctly, we retrieved relevant open-source code and provided it to the LLM. Games created with external code are shown in Appendix [12.4](https://arxiv.org/html/2505.13886v6#S12.SS4).

Data engine validation: During the data engine development process, LLMs generate an initial version based on the prompt. This version is then manually tested. If the generated reasoning questions and answers do not conform to the templates from the second step or contain errors, the LLM will be instructed to regenerate the data engine.

Data augmentation and filtering: Once the data engine is constructed, data samples can be easily generated in batch, yet the answer processes often exhibit repetitive patterns. We therefore employed LLM paraphrasing to perform data augmentation, detailed in Appendix [13.3](https://arxiv.org/html/2505.13886v6#S13.SS3). Additionally, we applied data filtering to ensure the augmented samples were correct, of appropriate length, and without excessive textual repetition. Details of our data quality assurance process can be found in Appendix [13.4](https://arxiv.org/html/2505.13886v6#S13.SS4). Time spent on these main steps for each game in GameQA can be found in Appendix [12.3](https://arxiv.org/html/2505.13886v6#S12.SS3).

## 3 The GameQA dataset

**Table 2: GameQA train and test set statistics summary. The full table is in Appendix [13.1](https://arxiv.org/html/2505.13886v6#S13.SS1).**
| Statistic Category | Train Set | Test Set |
| --- | --- | --- |
| Total Games | 20 | 30 |
| In-Domain | 20 | 20 |
| Out-of-Domain | 0 | 10 |
| Total Tasks | 102 | 158 |
| Total Questions | 126,760 | 15,047 |
| Multiple Choice | 86,520 | 10,518 |
| Avg. Choices | 7.10 | 7.05 |
| Fill-in-the-Blank | 40,240 | 4,529 |
| Unique Images | 74,620 | 8,620 |

The GameQA dataset is a visual-language question answering dataset that transforms game-playing tasks into a Visual Question Answering (VQA) format, as shown in Figure [2](https://arxiv.org/html/2505.13886v6#S2.F2). GameQA includes 30 games, 158 distinct tasks and approximately 140K questions in total.

Dataset composition:
The dataset has 30 different games classified into four categories based on the core abilities required to solve game tasks: "3D Spatial Perception and Understanding", "Pattern Recognition and Matching", "Multi-step Reasoning", and "Strategic Planning", detailed descriptions shown in Appendix [13.2](https://arxiv.org/html/2505.13886v6#S13.SS2). The criteria on choosing these 30 games are listed in Appendix [12.2](https://arxiv.org/html/2505.13886v6#S12.SS2). As illustrated in Figure [6](https://arxiv.org/html/2505.13886v6#S7.F6), the 30 games are divided into two sets: 20 In-Domain games for model training and 10 Out-of-Domain games for testing generalization performance. The Out-of-Domain set was held out during training.

Dataset question types:
To ensure the final answers are verifiable, all questions in the dataset are either multiple-choice or fill-in-the-blank. The multiple-choice questions typically have 7-8 options, while the fill-in-the-blank questions require simple answers such as numbers or coordinates, with detailed statistics shown in Table [2](https://arxiv.org/html/2505.13886v6#S3.T2).

Difficulty levels:
Each task is assigned with one of three difficulty levels of questions ("QA Level"). And data samples are divided into 3 difficulty levels based on image complexity ("Plot Level"), namely game state or grid size, controlled by code parameters, providing three perception and reasoning difficulty levels for the tasks. Examples demonstrating these difficulty levels are in Appendix LABEL:app:games_example_samples. The training and evaluation result about difficulty are shown in Table [12](https://arxiv.org/html/2505.13886v6#S11.T12) and Table [6](https://arxiv.org/html/2505.13886v6#S9.T6).

## 4 Game-RL: RL on game tasks

### 4.1 RL algorithm

We use the Group Relative Policy Optimization (GRPO) [25] as our algorithm. We use the standard formulation from DeepSeek, with the loss function shown below and the hyperparameters shown in Appendix [10.3.2](https://arxiv.org/html/2505.13886v6#S10.SS3.SSS2).

$$ $\displaystyle\mathcal{J}_{\text{GRPO}}(\theta)=\mathbb{E}[q\sim P(Q),\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{old}}(O|q)]\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}$ $\displaystyle\left\{\min\left[\frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}|q,o_{i,<t})}\hat{A}_{i,t},\text{clip}\left(\frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}|q,o_{i,<t})},1-\epsilon,1+\epsilon\right)\hat{A}_{i,t}\right]-\beta\mathbb{D}_{KL}[\pi_{\theta}||\pi_{\text{ref}}]\right\}$ $$

### 4.2 Reward design

LLM as a judge to give reward.
Given the high diversity of responses of our dataset, traditional rule-based judge methods (e.g. text matching) suffer from insufficient accuracy in answer matching. We chose the quantized version of Qwen2.5-32B-AWQ [23] as an evaluator model to assist in scoring the generated outputs.

Outcome reward design of RL
The reward signal used for training was based solely on the correctness of the model’s final answer. Both the reference answer and the answer generated by the policy model are inputted into this evaluator model. The evaluator model then determines whether the generated answer matches the reference answer. The reward allocation adheres to a strict binary principle: a reward of 1 is assigned if they match, and a reward of 0 is assigned otherwise.

## 5 Experiment

**Table 3: Evaluation results on general vision benchmarks. These results show that game-only training enhances general reasoning capability of VLMs. We fine-tuned four VLMs (InternVL2.5-8B [3], InternVL3-8B [4], Qwen2.5-VL-7B [2], and LLaVA-OneVision-7B [12]) on 5K GameQA samples using GRPO, resulting in the models Game-RL-InternVL2.5-8B, Game-RL-InternVL3-8B, Game-RL-Qwen2.5-VL-7B, and Game-RL-LLaVA-OV-7B. The percentage of performance improvements compared to the vanilla model is denoted by ($\uparrow$). Best performance per section is indicated in bold.**
| Models | Avg.<br>($\uparrow$) | MathVista | MathVerse | MMBench | MMMU | CharXiv | MathVision | MMMU-Pro |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline |  |  |  |  |  |  |  |  |
| Random | 14.84 | 17.90 | 12.74 | 26.37 | 24.67 | 0.00 | 10.03 | 12.19 |
| Proprietary Multimodal Large Language Models |  |  |  |  |  |  |  |  |
| GPT-4o | 56.9 | 63.8 | 50.2 | 86.0 | 69.1 | 47.1 | 30.4 | 51.9 |
| Claude-3.5-Sonnet | 59.5 | 67.7 | 56.8 | 78.5 | 68.3 | 60.2 | 33.3 | 51.5 |
| Gemini-2.5-Pro | 73.1 | 77.7 | 65.9 | 88.3 | 79.7 | 62.9 | 66.0 | 71.2 |
| Open-Source Multimodal Large Language Models |  |  |  |  |  |  |  |  |
| Qwen2.5-VL-32B | 60.70 | 77.40 | 60.41 | 88.13 | 63.83 | 47.50 | 33.80 | 53.83 |
| Ovis2-34B | 57.91 | 71.50 | 53.71 | 88.73 | 60.91 | 49.10 | 35.93 | 45.48 |
| InternVL2.5-38B | 55.62 | 68.60 | 48.38 | 86.93 | 57.53 | 41.50 | 39.40 | 46.98 |
| LLaVA-OV-72B | 49.51 | 58.60 | 46.85 | 83.13 | 52.74 | 35.20 | 33.87 | 36.18 |
| Qwen2.5-VL-72B | 60.95 | 75.50 | 56.87 | 86.80 | 65.34 | 48.10 | 40.60 | 53.45 |
| InternVL2.5-78B | 57.96 | 70.20 | 52.34 | 88.33 | 61.84 | 42.70 | 39.93 | 50.38 |
| InternVL2.5-8B | 45.89 | 57.50 | 36.04 | 81.93 | 47.96 | 31.70 | 28.87 | 37.25 |
| Game-RL-InternVL2.5-8B | 47.91 (+2.02) | 61.70 | 37.11 | 83.87 | 50.06 | 32.00 | 31.93 | 38.69 |
| InternVL3-8B | 54.48 | 69.10 | 50.10 | 86.00 | 57.88 | 39.10 | 35.33 | 43.84 |
| Game-RL-InternVL3-8B | 55.88 (+1.40) | 73.00 | 50.71 | 86.20 | 58.34 | 39.90 | 37.93 | 45.10 |
| Qwen2.5-VL-7B | 49.94 | 66.80 | 45.08 | 83.67 | 49.01 | 37.70 | 30.80 | 36.49 |
| Game-RL-Qwen2.5-VL-7B | 52.27 (+2.33) | 68.20 | 47.97 | 83.53 | 50.53 | 42.70 | 33.07 | 39.89 |
| LLaVA-OV-7B | 41.23 | 55.60 | 33.05 | 81.13 | 41.07 | 27.10 | 23.40 | 27.26 |
| Game-RL-LLaVA-OV-7B | 42.27 (+1.04) | 58.20 | 34.92 | 82.53 | 41.31 | 27.30 | 23.07 | 28.58 |

**Table 4: GameQA’s generalization is competitive compared to outstanding geometry visual reasoning datasets. Based on Qwen2.5-VL-7B, we applied the same GRPO method on 5k GameQA samples, 8k samples from MAVIS, 8k Multimodal-Open-R1 samples, 8k samples from MultiMath respectively, a mixture of 5k GameQA and 8k MultiMath samples, to conduct comparative training.**
| Models | Avg.<br>($\uparrow$) | Out of Domain Games | Avg.<br>($\uparrow$) | General Vision Benchmarks |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 3D Spatial<br>Perc. & Under. | Pattern Recog.<br>& Matching | Multi-step<br>Reasoning | Strategic<br>Planning | Math<br>Vista | Math<br>Verse | MMBench | MMMU | CharXiv | Math<br>Vision | MMMU-Pro |  |  |  |
| Qwen2.5-VL-7B | 27.09 | 23.60 | 29.20 | 26.21 | 29.34 | 49.94 | 66.80 | 45.08 | 83.67 | 49.01 | 37.70 | 30.80 | 36.49 |
| + MAVIS (GRPO) | 27.61 | 26.80 | 28.25 | 28.42 | 26.98 | 51.53 | 67.90 | 46.16 | 83.62 | 50.45 | 39.20 | 34.98 | 38.42 |
| (+0.52) | (+1.59) |  |  |  |  |  |  |  |  |  |  |  |  |
| + Multimodal-Open-R1 | 28.33 | 24.87 | 27.86 | 29.93 | 30.64 | 51.86 | 67.63 | 48.09 | 83.78 | 49.78 | 40.20 | 34.89 | 38.65 |
| (GRPO) | (+1.24) | (+1.92) |  |  |  |  |  |  |  |  |  |  |  |
| + MultiMath (GRPO) | 28.38 | 28.10 | 28.45 | 27.45 | 29.53 | 52.81 | 69.36 | 47.99 | 84.13 | 53.44 | 40.83 | 33.92 | 39.91 |
| (+1.29) | (+2.87) |  |  |  |  |  |  |  |  |  |  |  |  |
| + GameQA (GRPO) | 29.87 | 27.00 | 28.52 | 31.49 | 32.46 | 52.31 | 68.70 | 48.72 | 83.16 | 50.21 | 41.40 | 34.27 | 39.74 |
| (+2.78) | (+2.37) |  |  |  |  |  |  |  |  |  |  |  |  |
| +<br>GameQA,<br>MultiMath<br>(GRPO) | 30.93 | 28.20 | 28.28 | 34.26 | 32.99 | 53.23 | 69.20 | 48.02 | 84.73 | 53.21 | 42.10 | 34.47 | 40.89 |
| (+3.84) | (+3.29) |  |  |  |  |  |  |  |  |  |  |  |  |

### 5.1 Experiment settings

The detailed description of data preparation, training processes, and evaluation method can be found in Appendix [10](https://arxiv.org/html/2505.13886v6#S10).

Hyperparamters of GRPO training We rollout 12 samples per questions. The model was trained for one epoch. The learning rate was 2e-7, with a 5% warm-up. The clipping value $\epsilon$ was set to 0.2 and the KL-divergence coefficient $\beta$ was set to 0.04.

Benchmarks We evaluated the models on a set of vison-language reasoning benchmarks consisting of our GameQA test set and public general benchmarks.

- •
GameQA benchmark This test set includes around 500 question-answer pairs for each of 30 games, totaling 15,047 samples.
- •
General benchmarks We used the MMMU validation set [30] for testing general multimodal understanding, and included MMMU-Pro [31], which features 10-option multiple-choice questions. To assess mathematical reasoning in visual contexts, we used MathVista [19] (testmini, 1,000 samples), MathVerse [34] (testmini, 3,940 samples), and MathVision [28] (open subset, 3,040 questions). For general visual understanding, we adopted MMBench [15] (validation set), and for chart-based reasoning, we used CharXiv [29].

### 5.2 Main results

By analyzing the models’ performance on in-domain tasks, related out-of-domain game tasks, and general vision-language benchmarks, we can find that:

Game-only training enhances general reasoning capability of VLMs Training on the GameQA dataset significantly improved performance on the GameQA test set. Moreover, models trained on GameQA exhibited strong generalization to unseen games, with accuracy improvements ranging from 1.16% to 3.82% (Table [5](https://arxiv.org/html/2505.13886v6#S5.T5)). On broader general vision benchmarks, these models showed robust generalization, achieving consistent performance gains across all general vision reasoning tasks. These results suggest that the models successfully learned transferable visual understanding and reasoning abilities from the GameQA dataset (see Table [3](https://arxiv.org/html/2505.13886v6#S5.T3)).

GameQA’s out-of-domain generalization is competitive compared to outstanding geometry visual reasoning datasets
To better understand the GameQA dataset’s advantages, we compared it with the outstanding geometry visual reasoning datasets, include MAVIS [35], Multimodal-Open-R1 [16] and MultiMath [22]. MAVIS includes various geometry and function problems. Multimodal-Open-R1 is a geometry-centered dataset. MultiMath is a comprehensive and diverse multimodal mathematical dataset. The results (Table [4](https://arxiv.org/html/2505.13886v6#S5.T4)) show that despite having fewer training samples (5k vs. 8k) that are also out-of-domain for geometry and function tasks, the GameQA-trained model is competitive compared to its counterparts trained on geometry or function data, where general vision benchmarks would be considered in-domain. These results suggest that GameQA enables stronger out-of-domain generalization, even when using less data from a mismatched domain.

GameQA enhances model performance in mixed training.
We trained Qwen2.5-VL-7B using GRPO on a mixture of 5k GameQA and 8k MultiMath samples, as shown in Table [4](https://arxiv.org/html/2505.13886v6#S5.T4). This results suggest that GameQA can bring extra benefits when mix training with other dataset.

Figure: Figure 3: The scaling effect of training data quantity on general vision benchmarks for Qwen2.5-VL-7B-Instruct. The model was trained on a total of 20k samples (20 games) and evaluated every 1,000 samples. To clearly demonstrate the upward trend, the results are divided into three stages and presented using bin averaging (as described in Section [11.3](https://arxiv.org/html/2505.13886v6#S11.SS3)).
Refer to caption: x3.png

Figure: Figure 4: The scaling effect of GameQA. As VLMs are trained on an increasing number of distinct games, their performance on general visual benchmarks improves. Game selection is shown in Table [7](https://arxiv.org/html/2505.13886v6#S10.T7).
Refer to caption: x4.png

### 5.3 Scaling effect of game samples on generalization

We trained the Qwen2.5-VL-7B model on a GameQA subset of 20,000 samples from 20 games using the GRPO method. As shown in Figure [3](https://arxiv.org/html/2505.13886v6#S5.F3), the model’s performance score demonstrates a overall upward trend on 7 general vision benchmarks as the amount of training data increases. This indicates that scaling up training on GameQA data effectively enhances the VLM’s general reasoning abilities.

### 5.4 Scaling effect of game diversity on generalization

With around 5,000 total training samples, we trained the Qwen2.5-VL-3B on GameQA subsets with 4, 10, and 20 distinct games (Table [7](https://arxiv.org/html/2505.13886v6#S10.T7)). The results (Figure [5](https://arxiv.org/html/2505.13886v6#S5.F5)) show a positive correlation between game diversity and generalization ability. This suggests that scaling game diversity makes better generalization, enabling the model to acquire more robust visual understanding and reasoning abilities.

### 5.5 Qualitative analysis

To confirm that GRPO substantially enhances visual perception and text reasoning abilities of models, we manually analyzed 790 cases randomly sampled from the results of InternVL2.5-8B, containing responses before and after GRPO. The results (Figure [5](https://arxiv.org/html/2505.13886v6#S5.F5)) confirm that after GRPO, the model demonstrates improved visual recognition of image elements and performs more precise reasoning. More statistics and cases are in Appendix [9.2](https://arxiv.org/html/2505.13886v6#S9.SS2) and [9.3](https://arxiv.org/html/2505.13886v6#S9.SS3).

In addition, our qualitative analysis of model performances across four game categories reveals common behaviors and challenges, detailed in Appendix LABEL:app_sec:gpt4o_case_study.

### 5.6 Advanced vision-language models perform notably worse than humans on the GameQA benchmark

Both leading open-source and proprietary models achieve average accuracy levels considerably lower than those of human (Table [5](https://arxiv.org/html/2505.13886v6#S5.T5)). This clear difference highlights the difficulty and high requirements of the GameQA benchmark, requiring not only accurate visual comprehension of game scenes but also the ability to carry out multi-step logical reasoning. The training and evaluation result about difficulty are shown in Table [12](https://arxiv.org/html/2505.13886v6#S11.T12) and Table [6](https://arxiv.org/html/2505.13886v6#S9.T6). Furthermore, our qualitative analysis and case study in Appendix LABEL:app_sec:gpt4o_case_study reveal that even the most advanced models currently struggle to match human-level understanding, particularly in tasks demanding deep reasoning. More experiments can be found in Appendix [11](https://arxiv.org/html/2505.13886v6#S11).

**Table 5: Evaluation results on GameQA benchmark. The experimental setup is consistent with Table [3](https://arxiv.org/html/2505.13886v6#S5.T3): four VLMs were fine-tuned on 5K GameQA samples using GRPO. We evaluate the models on both in-domain and out-of-domain game categories. The percentage improvement over the vanilla model is marked with a green upward arrow ($\uparrow$). The best performance for each category is presented in bold.**
| Models | Avg.<br>($\uparrow$) | In Domain Games | Avg.<br>($\uparrow$) | Out of Domain Games |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 3D Spatial<br>Perc. & Under. | Pattern Recog.<br>& Matching | Multi-step<br>Reasoning | Strategic<br>Planning | 3D Spatial<br>Perc. & Under. | Pattern Recog.<br>& Matching | Multi-step<br>Reasoning | Strategic<br>Planning |  |  |  |
| Baselines |  |  |  |  |  |  |  |  |  |  |
| Human | 84.75 | 85.18 | 80.74 | 84.46 | 88.62 | 81.61 | 79.17 | 73.81 | 81.27 | 92.19 |
| Random | 11.90 | 11.69 | 12.04 | 10.24 | 13.61 | 9.68 | 7.11 | 9.50 | 11.83 | 10.27 |
| Proprietary Multimodal Large Language Models |  |  |  |  |  |  |  |  |  |  |
| GPT-4o | 40.52 | 32.01 | 34.81 | 50.67 | 44.59 | 43.81 | 48.90 | 36.91 | 48.58 | 40.87 |
| Claude-3.5-Sonnet | 47.69 | 37.41 | 43.16 | 56.09 | 54.11 | 50.34 | 51.30 | 43.62 | 60.42 | 46.01 |
| Claude-4-Sonnet | 46.58 | 31.12 | 39.73 | 66.90 | 48.57 | 55.16 | 45.60 | 56.58 | 63.28 | 55.17 |
| Gemini-2.5-Pro | 58.95 | 46.93 | 52.79 | 74.62 | 61.46 | 67.60 | 57.60 | 77.37 | 77.62 | 57.80 |
| Open-Source Multimodal Large Language Models |  |  |  |  |  |  |  |  |  |  |
| Ovis2-8B | 24.98 | 19.92 | 24.43 | 27.21 | 28.37 | 26.99 | 29.70 | 20.70 | 34.14 | 23.41 |
| InternVL3-9B | 26.89 | 21.86 | 22.53 | 32.54 | 30.65 | 26.73 | 25.20 | 30.38 | 32.14 | 19.18 |
| LLaMA3.2-11B-Vision | 19.69 | 19.12 | 16.48 | 21.30 | 21.86 | 18.04 | 18.40 | 14.92 | 17.73 | 21.11 |
| Qwen2.5-VL-32B | 34.09 | 28.26 | 30.99 | 40.27 | 36.83 | 35.97 | 32.90 | 33.02 | 44.03 | 33.94 |
| Ovis2-34B | 34.53 | 27.92 | 32.72 | 39.46 | 38.03 | 35.29 | 35.50 | 34.20 | 38.71 | 32.75 |
| InternVL2.5-38B | 30.04 | 23.39 | 25.86 | 36.82 | 34.08 | 32.42 | 30.60 | 33.96 | 39.35 | 25.79 |
| InternVL3-38B | 35.23 | 28.33 | 31.76 | 41.89 | 38.96 | 38.69 | 33.30 | 43.62 | 50.09 | 27.75 |
| LLaVA-OV-72B | 24.87 | 19.92 | 24.88 | 27.72 | 26.95 | 28.32 | 26.80 | 23.52 | 32.87 | 30.11 |
| Qwen2.5-VL-72B | 37.63 | 29.47 | 32.85 | 45.76 | 42.42 | 39.22 | 35.90 | 37.38 | 46.86 | 36.75 |
| InternVL2.5-78B | 32.35 | 27.15 | 28.84 | 39.53 | 33.90 | 35.30 | 32.80 | 37.26 | 42.41 | 28.75 |
| InternVL3-78B | 38.00 | 33.15 | 33.03 | 46.60 | 39.20 | 39.74 | 34.90 | 40.70 | 50.95 | 32.43 |
| InternVL2.5-8B | 22.22 | 20.39 | 17.18 | 25.34 | 25.97 | 20.05 | 20.80 | 22.45 | 18.88 | 18.07 |
| Game-RL-InternVL2.5-8B | 29.44 (+7.22) | 26.74 | 26.05 | 29.51 | 35.44 | 23.87 (+3.82) | 25.00 | 25.12 | 24.91 | 20.45 |
| InternVL3-8B | 26.51 | 22.53 | 21.91 | 30.18 | 31.43 | 27.64 | 29.60 | 27.44 | 29.62 | 23.91 |
| Game-RL-InternVL3-8B | 33.09 (+6.58) | 27.94 | 28.52 | 36.81 | 39.07 | 28.80 (+1.16) | 29.20 | 25.31 | 34.59 | 26.09 |
| Qwen2.5-VL-7B | 25.78 | 22.58 | 21.92 | 25.21 | 33.40 | 27.09 | 23.60 | 29.20 | 26.21 | 29.34 |
| Game-RL-Qwen2.5-VL-7B | 32.12 (+6.34) | 26.80 | 26.88 | 33.34 | 41.45 | 30.51 (+3.42) | 27.10 | 31.56 | 31.24 | 32.13 |
| LLaVA-OV-7B | 21.79 | 18.84 | 19.69 | 21.24 | 27.38 | 20.39 | 21.00 | 20.19 | 20.13 | 20.27 |
| Game-RL-LLaVA-OV-7B | 33.49 (+11.70) | 29.87 | 31.10 | 30.96 | 42.03 | 23.34 (+2.95) | 27.20 | 20.05 | 23.55 | 22.57 |

## 6 Related work

Multimodal reasoning data construction
Currently, the data construction methods are mainly divided into two categories: human expert supervision and automated synthesis. Peng et al. [22] and Lu et al. [17] collect visual reasoning problems from textbooks, Lu et al. [18] constructs datasets through labeling by STEM students, but they are limited by the scarcity of high-quality data sources and the high cost of manual annotation. Lu et al. [17], He et al. [8], Gao et al. [6], Shi et al. [26] uses expert models to generate reasoning processes, but the results are limited by the performance of the expert model. Trinh et al. [27] and Zhang et al. [35] synthesize geometric reasoning data through procedural methods, but these methods are often designed for specific domains and have high transfer costs. Table [1](https://arxiv.org/html/2505.13886v6#S1.T1) provides a comparison of existing vision language reasoning datasets.

Using game data to enhance VLMs reasoning capabilities
Game environments provide well-defined rules and mechanics that are easy to verify. However, existing work has not fully leveraged the potential of game environments in visual reasoning data construction. Reed et al. [24] trains a general agent by tokenizing game images and action sequences, but it is difficult to generalize on out-of-domain game tasks and this method relies on expensive expert trajectory data; Paglieri et al. [21], Zhang et al. [33, 32], Li et al. [13] all established gaming environments for Vision-Language Models, but these were used exclusively for evaluation purposes. These limitations indicate that how to effectively use game data to enhance the reasoning ability of visual language models remains a critical problem that needs to be addressed. Table [1](https://arxiv.org/html/2505.13886v6#S1.T1) provides a comparison of existing game reasoning benchmarks.

## 7 Conclusion

To explore broader training scenarios and resources for vision-language RL, we propose Game-RL to construct game tasks for VLMs’ RL training. We also propose the novel Code2Logic approach adapting game code to synthesize diverse game reasoning task data, thus obtaining the GameQA dataset for Game-RL. Multiple VLMs trained through RL solely on GameQA achieved performance improvements across diverse general vision-language reasoning benchmarks. This not only demonstrates the value of Game-RL for enhancing VLMs’ general reasoning abilities, but also suggests that video games may serve as valuable scenarios and resources to boost VLMs’ general reasoning.

## References

- Anthropic [2024]
Anthropic.
Claude 3.5 sonnet model card addendum, 2024.
URL
[https://www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf](https://www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf).
- Bai et al. [2025]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai
Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun
Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng
Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang,
Haiyang Xu, and Junyang Lin.
Qwen2.5-vl technical report, 2025.
URL [https://arxiv.org/abs/2502.13923](https://arxiv.org/abs/2502.13923).
- Chen et al. [2024]
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo
Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al.
Expanding performance boundaries of open-source multimodal models
with model, data, and test-time scaling.
*arXiv preprint arXiv:2412.05271*, 2024.
- Chen et al. [2025]
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo
Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al.
Internvl3: Exploring advanced training and test-time recipes for
open-source multimodal models.
*arXiv preprint arXiv:2504.10479*, 2025.
- Chu et al. [2025]
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale
Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma.
Sft memorizes, rl generalizes: A comparative study of foundation
model post-training, 2025.
URL [https://arxiv.org/abs/2501.17161](https://arxiv.org/abs/2501.17161).
- Gao et al. [2023]
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang,
Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al.
G-llava: Solving geometric problem with multi-modal large language
model.
*arXiv preprint arXiv:2312.11370*, 2023.
- Guo et al. [2025]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu,
Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via
reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- He et al. [2024]
Wei He, Zhiheng Xi, Wanxu Zhao, Xiaoran Fan, Yiwen Ding, Zifei Shan, Tao Gui,
Qi Zhang, and Xuanjing Huang.
Distill visual chart reasoning ability from llms to mllms.
*arXiv preprint arXiv:2410.18798*, 2024.
- Hu et al. [2021]
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean
Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models, 2021.
URL [https://arxiv.org/abs/2106.09685](https://arxiv.org/abs/2106.09685).
- Jiang et al. [2024a]
Jin Jiang, Yuchen Yan, Yang Liu, Yonggang Jin, Shuai Peng, Mengdi Zhang,
Xunliang Cai, Yixin Cao, Liangcai Gao, and Zhi Tang.
Logicpro: Improving complex logical reasoning via program-guided
learning.
*arXiv preprint arXiv:2409.12929*, 2024a.
- Jiang et al. [2024b]
Zhiqiang Jiang, Yihuai Lan, Minggao Zhang, Haozhe Feng, Wenyu Liu, Junwei Dong,
Ze Ji, Xin Zhao, and Xing Xu.
Math-llava: Bootstrapping mathematical reasoning for multimodal large
language models, 2024b.
- Li et al. [2024a]
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen
Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li.
Llava-onevision: Easy visual task transfer, 2024a.
URL [https://arxiv.org/abs/2408.03326](https://arxiv.org/abs/2408.03326).
- Li et al. [2024b]
Chenglin Li, Qianglong Chen, Zhi Li, Feng Tao, and Yin Zhang.
Vcbench: A controllable benchmark for symbolic and abstract
challenges in video cognition.
*arXiv preprint arXiv:2411.09105*, 2024b.
- Liu et al. [2025]
Junteng Liu, Yuanxiang Fan, Zhuo Jiang, Han Ding, Yongyi Hu, Chi Zhang, Yiqi
Shi, Shitong Weng, Aili Chen, Shiqi Chen, et al.
Synlogic: Synthesizing verifiable reasoning data at scale for
learning logical reasoning and beyond.
*arXiv preprint arXiv:2505.19641*, 2025.
- Liu et al. [2023]
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike
Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin.
Mmbench: Is your multi-modal model an all-around player?
*arXiv preprint arXiv:2307.06281*, 2023.
- lmms lab [2025]
lmms lab.
Multimodal-open-r1-8k-verified.
[https://huggingface.co/datasets/lmms-lab/multimodal-open-r1-8k-verified](https://huggingface.co/datasets/lmms-lab/multimodal-open-r1-8k-verified),
2025.
- Lu et al. [2021]
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and
Song-Chun Zhu.
Inter-gps: Interpretable geometry problem solving with formal
language and symbolic reasoning.
*arXiv preprint arXiv:2105.04165*, 2021.
- Lu et al. [2023]
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh
Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao.
Mathvista: Evaluating mathematical reasoning of foundation models in
visual contexts.
*arXiv preprint arXiv:2310.02255*, 2023.
- Lu et al. [2024]
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh
Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao.
Mathvista: Evaluating mathematical reasoning of foundation models in
visual contexts.
In *The Twelfth International Conference on Learning
Representations*, 2024.
- OpenAI [2024]
OpenAI.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- Paglieri et al. [2024]
Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg,
Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński,
Lerrel Pinto, Rob Fergus, et al.
Balrog: Benchmarking agentic llm and vlm reasoning on games.
*arXiv preprint arXiv:2411.13543*, 2024.
- Peng et al. [2024]
Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang.
Multimath: Bridging visual and mathematical reasoning for large
language models.
*arXiv preprint arXiv:2409.00147*, 2024.
- Qwen et al. [2025]
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu,
Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang,
Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang
Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue,
Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia,
Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan,
Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 technical report, 2025.
URL [https://arxiv.org/abs/2412.15115](https://arxiv.org/abs/2412.15115).
- Reed et al. [2022]
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander
Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay,
Jost Tobias Springenberg, et al.
A generalist agent.
*arXiv preprint arXiv:2205.06175*, 2022.
- Shao et al. [2024]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei
Zhang, Mingchuan Zhang, YK Li, Y Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open
language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Shi et al. [2024]
Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong
Bing, and Roy Ka-Wei Lee.
Math-llava: Bootstrapping mathematical reasoning for multimodal large
language models.
*arXiv preprint arXiv:2406.17294*, 2024.
- Trinh et al. [2024]
Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong.
Solving olympiad geometry without human demonstrations.
*Nature*, 625(7995):476–482, 2024.
- Wang et al. [2024a]
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie
Zhan, and Hongsheng Li.
Measuring multimodal mathematical reasoning with math-vision dataset.
*arXiv preprint arXiv:2402.14804*, 2024a.
- Wang et al. [2024b]
Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu
Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev
Arora, and Danqi Chen.
Charxiv: Charting gaps in realistic chart understanding in multimodal
llms.
*arXiv preprint arXiv:2406.18521*, 2024b.
- Yue et al. [2024a]
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel
Stevens, Dongfu Jiang, Chen Cui, Tanmay Rao, Human-Machine interfacing,
et al.
Mmmu: A massive multi-discipline multimodal understanding and
reasoning benchmark for expert agi.
In *Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition*, pages 28039–28064, 2024a.
- Yue et al. [2024b]
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong,
Yuxuan Sun, Meiqi Yin, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and
Graham Neubig.
Mmmu-pro: A more robust multi-discipline multimodal understanding
benchmark.
*arXiv preprint arXiv:2409.02813*, 2024b.
- Zhang et al. [2025]
Alex L Zhang, Thomas L Griffiths, Karthik R Narasimhan, and Ofir Press.
Videogamebench: Can vision-language models complete popular video
games?
*arXiv preprint arXiv:2505.18134*, 2025.
- Zhang et al. [2024a]
Haoran Zhang, Hangyu Guo, Shuyue Guo, Meng Cao, Wenhao Huang, Jiaheng Liu, and
Ge Zhang.
Ing-vp: Mllms cannot play easy vision-based games yet.
*arXiv preprint arXiv:2410.06555*, 2024a.
- Zhang et al. [2024b]
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu,
Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li.
Mathverse: Does your multi-modal llm truly see the diagrams in visual
math problems?
*arXiv preprint arXiv:2403.14624*, 2024b.
- Zhang et al. [2024c]
Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong,
Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, et al.
Mavis: Mathematical visual instruction tuning.
*arXiv e-prints*, pages arXiv–2407, 2024c.
- Zhao et al. [2024]
Jun Zhao, Jingqi Tong, Yurong Mou, Ming Zhang, Qi Zhang, and Xuanjing Huang.
Exploring the compositional deficiency of large language models in
mathematical reasoning through trap problems.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,
*Proceedings of the 2024 Conference on Empirical Methods in Natural
Language Processing*, pages 16361–16376, Miami, Florida, USA, November 2024.
Association for Computational Linguistics.
[10.18653/v1/2024.emnlp-main.915](https:/doi.org/10.18653/v1/2024.emnlp-main.915).
URL [https://aclanthology.org/2024.emnlp-main.915/](https://aclanthology.org/2024.emnlp-main.915/).
- Zheng et al. [2023]
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao
Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E.
Gonzalez, and Ion Stoica.
Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
URL [https://arxiv.org/abs/2306.05685](https://arxiv.org/abs/2306.05685).

## Appendix Contents

## 8 Limitations and future work

Using reasoning processes to conduct Supervised Fine-Tuning (SFT) has not achieved satisfactory out-of-domain generalization. Therefore, future work could explore methods to better leverage these reasoning processes to enhance the model’s general capabilities, such as employing reinforcement learning based on process supervision.
In addition, GameQA currently involves single-turn game question answering. Future work could focus on developing training and evaluation methods for multi-turn interactions in gaming scenarios.

Figure: (a) Number of distinct games
Refer to caption: x7.png

## 9 More analysis

### 9.1 GameQA evaluation results by difficulty

**Table 6: Evaluation results on GameQA benchmark by difficulty. Scores are broken down by question difficulty (QA Level) and image complexity (Plot Level) within in-domain and out-of-domain game sets. The percentage of performance improvements compared to the vanilla model is denoted by ($\uparrow$). Best performance per section is indicated in bold.**
| Models | Avg.<br>($\uparrow$) | In Domain Games by Difficulty | Avg.<br>($\uparrow$) | Out of Domain Games by Difficulty |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| QA Level | Plot Level | QA Level | Plot Level |  |  |  |  |  |  |  |  |  |  |  |
| Easy | Medium | Hard | Easy | Medium | Hard | Easy | Medium | Hard | Easy | Medium | Hard |  |  |  |
| Proprietary Multimodal Large Language Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-4o | 41.56 | 48.10 | 40.63 | 32.95 | 47.69 | 38.60 | 37.80 | 44.03 | 55.63 | 44.45 | 32.05 | 54.49 | 42.98 | 33.71 |
| Claude-3.5-Sonnet | 48.90 | 59.26 | 46.07 | 36.70 | 55.49 | 45.97 | 43.80 | 50.94 | 62.80 | 46.15 | 43.80 | 60.58 | 49.19 | 42.26 |
| Claude-4-Sonnet | 48.80 | 58.36 | 50.14 | 42.23 | 56.45 | 48.99 | 47.81 | 56.03 | 66.71 | 55.91 | 45.45 | 62.80 | 54.31 | 50.37 |
| Gemini-2.5-Pro | 60.62 | 67.43 | 62.36 | 56.21 | 67.48 | 62.42 | 57.72 | 67.68 | 74.47 | 67.56 | 61.10 | 74.23 | 66.43 | 61.90 |
| Open-Source Multimodal Large Language Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Ovis2-8B | 25.56 | 33.67 | 22.80 | 20.21 | 28.30 | 26.12 | 23.69 | 27.37 | 32.41 | 28.87 | 20.90 | 33.96 | 25.56 | 22.06 |
| InternVL3-9B | 27.45 | 34.92 | 24.20 | 21.15 | 30.42 | 26.94 | 24.43 | 26.54 | 30.45 | 23.23 | 25.86 | 34.41 | 23.93 | 20.63 |
| LLaMA3.2-11B-Vision | 19.67 | 24.51 | 18.54 | 15.96 | 20.71 | 20.22 | 19.16 | 18.35 | 19.37 | 20.32 | 15.41 | 20.99 | 18.14 | 15.68 |
| Qwen2.5-VL-32B | 34.80 | 41.52 | 33.43 | 27.18 | 38.07 | 34.93 | 31.09 | 36.59 | 47.99 | 33.60 | 28.16 | 46.87 | 32.25 | 29.86 |
| Ovis2-34B | 35.32 | 46.36 | 31.81 | 25.11 | 39.57 | 34.70 | 31.70 | 35.40 | 46.86 | 30.02 | 29.22 | 42.95 | 33.76 | 28.87 |
| InternVL2.5-38B | 30.83 | 40.08 | 28.21 | 22.68 | 35.55 | 30.36 | 27.23 | 32.50 | 38.80 | 29.71 | 28.93 | 41.52 | 30.74 | 24.47 |
| InternVL3-38B | 36.07 | 44.05 | 31.73 | 30.60 | 40.70 | 36.47 | 30.20 | 38.78 | 50.00 | 36.14 | 30.17 | 49.03 | 35.62 | 30.86 |
| LLaVA-OV-72B | 25.49 | 35.50 | 21.21 | 17.57 | 27.81 | 24.66 | 23.87 | 28.98 | 38.80 | 23.83 | 24.20 | 34.07 | 27.00 | 25.46 |
| Qwen2.5-VL-72B | 38.58 | 47.61 | 36.70 | 27.38 | 43.50 | 37.67 | 33.43 | 39.77 | 52.90 | 37.90 | 28.51 | 49.15 | 37.13 | 32.28 |
| InternVL2.5-78B | 33.05 | 41.90 | 30.58 | 24.00 | 36.58 | 32.30 | 30.00 | 35.40 | 43.96 | 34.57 | 27.69 | 45.22 | 32.97 | 27.20 |
| InternVL3-78B | 38.64 | 47.64 | 35.60 | 30.47 | 43.91 | 37.34 | 34.57 | 40.17 | 52.37 | 35.84 | 32.23 | 50.34 | 39.36 | 29.93 |
| InternVL2.5-8B | 22.31 | 27.12 | 21.98 | 18.35 | 24.65 | 22.66 | 21.37 | 19.78 | 18.36 | 21.29 | 19.72 | 24.52 | 17.78 | 16.67 |
| + GameQA (SFT) | 47.33 (+25.02) | 56.59 | 45.36 | 39.79 | 53.73 | 45.50 | 44.76 | 26.10 (+6.32) | 27.84 | 21.95 | 28.39 | 30.43 | 27.37 | 20.07 |
| + GameQA (GRPO) | 29.52 (+7.21) | 36.37 | 26.95 | 25.90 | 34.23 | 28.14 | 27.91 | 23.67 (+3.90) | 24.41 | 22.92 | 23.67 | 28.44 | 22.85 | 19.33 |
| InternVL3-8B | 26.85 | 33.82 | 26.37 | 19.79 | 28.91 | 28.20 | 24.90 | 27.49 | 32.29 | 26.62 | 23.55 | 32.59 | 26.46 | 22.99 |
| + GameQA (SFT) | 51.08 (+24.23) | 63.08 | 47.66 | 43.22 | 58.93 | 48.69 | 48.74 | 27.35 (-0.14) | 37.56 | 22.07 | 22.31 | 35.38 | 24.83 | 21.19 |
| + GameQA (GRPO) | 33.53 (+6.68) | 39.73 | 32.17 | 28.99 | 37.16 | 32.37 | 32.80 | 29.14 (+1.65) | 34.36 | 29.41 | 23.67 | 34.81 | 27.31 | 24.85 |
| Qwen2.5-VL-7B | 26.02 | 36.25 | 23.11 | 17.75 | 28.29 | 25.69 | 25.67 | 27.25 | 31.87 | 25.83 | 24.03 | 33.73 | 25.38 | 22.12 |
| + GameQA (SFT) | 49.23 (+23.21) | 60.19 | 47.47 | 39.13 | 55.56 | 47.65 | 46.60 | 30.33 (+3.08) | 42.00 | 25.35 | 23.55 | 36.86 | 29.29 | 24.29 |
| + GameQA (GRPO) | 32.41 (+6.39) | 42.51 | 30.25 | 23.96 | 35.05 | 32.40 | 31.79 | 30.77 (+3.52) | 38.68 | 25.96 | 27.57 | 38.17 | 28.87 | 24.66 |
| LLaVA-OV-7B | 21.65 | 29.28 | 19.24 | 17.67 | 23.19 | 22.03 | 22.20 | 20.37 | 22.99 | 17.04 | 21.02 | 24.40 | 20.31 | 16.05 |
| + GameQA (SFT) | 46.25 (+24.59) | 54.62 | 46.81 | 36.70 | 51.65 | 44.55 | 45.03 | 23.39 (+3.02) | 29.56 | 20.86 | 19.72 | 29.35 | 23.51 | 16.79 |
| + GameQA (GRPO) | 33.61 (+11.95) | 41.41 | 31.26 | 29.94 | 38.83 | 31.57 | 33.53 | 23.32 (+2.94) | 30.15 | 18.25 | 21.43 | 28.90 | 23.51 | 17.04 |

The fine-grained difficulty gradation in the GameQA dataset enables a more systematic evaluation of the models. As shown in Table [6](https://arxiv.org/html/2505.13886v6#S9.T6), when either QA Level or Plot Level increases, the models’ accuracy scores generally show a noticeable decrease.

Figure: Figure 8: Impact of GRPO on text reasoning performance, with two specific improvement examples.
Refer to caption: x9.png

Figure: Figure 9: Impact of GRPO on visual reasoning performance on GameQA.
Refer to caption: x10.png

### 9.2 Error types analysis

We manually checked the reasoning process of 650 GameQA cases randomly sampled from InternVL2.5-8B before and after GRPO to identify the error types. We categorize errors as 3 types: Visual Perception Error, Text Reasoning Error and both. The results are presented in Figure [10](https://arxiv.org/html/2505.13886v6#S9.F10). Besides, in Figure [8](https://arxiv.org/html/2505.13886v6#S9.F8) and Figure[9](https://arxiv.org/html/2505.13886v6#S9.F9) we supplement to Figure [5](https://arxiv.org/html/2505.13886v6#S5.F5) with two examples of text reasoning improvement and another example of visual reasoning improvement after GRPO training.

Figure: Figure 10: InternVL2.5-8B error types before and after GRPO show that GRPO increases correct ratio and reduces visual and reasoning errors. Two cases show the two types of error in detail. Solid and dash lines connected to two cases means both are before GRPO.
Refer to caption: x11.png

### 9.3 More results and analysis on the impact of training game diversity and task instance diversity on model generalization

Game diversity While the model trained on 20 games achieved the highest performance gain on both unseen out-of-domain games and general visual benchmarks, interestingly, we observed that the model trained on 4 games (out-of-domain games +1.20%, general +0.66%) slightly outperformed the one trained on 10 games (out-of-domain games +0.96%, general +0.63%) on both benchmarks. We speculate that this non-linear improvement may result from cases where a smaller but well-chosen set of games provides a comparable amount of key generalization signals as a larger set, or that certain random factors affected the performance of specific game combinations.

Task instance diversity

Figure: Figure 11: The scaling effect of GameQA with an increasing number of task instances.
Refer to caption: x12.png

We further investigated how the task instance diversity influence model generalization using the Qwen2.5-VL-3B model. We compared three configurations: (a) 5k samples for 1 epoch, (b) 1k samples for 5 epochs, and (c) 0.5k samples for 10 epochs, ensuring consistent total training steps and batch sizes across all settings. On general visual benchmarks, the model trained with 5k samples for 1 epoch performed best, with a +1.2% improvement (Figure [11](https://arxiv.org/html/2505.13886v6#S9.F11)). This suggests that scaling game task instances diversity makes better generalization. Extended data scaling experiment is in Appendix [11.3](https://arxiv.org/html/2505.13886v6#S11.SS3).

On out-of-domain games in GameQA, unlike on general visual benchmarks, we observed a counterintuitive trend: as the number of training samples per epoch decreased and the number of training epochs increased, the model’s generalization performance improved, as shown in Figure [7](https://arxiv.org/html/2505.13886v6#S8.F7). The best out-of-domain gain (+2.5%) was achieved by the model trained on 0.5k samples for 10 epochs, outperforming both the 1k $\times$ 5 epochs (+2.2%) and 5k $\times$ 1 epoch (+1.8%) configurations. This suggests that for near-domain tasks closely related to the training games, repeatedly training on a smaller but diverse dataset may help the model better capture task-specific patterns and nuances, leading to improved generalization.

## 10 Experiment details

### 10.1 Human and random baselines

We included two baselines for comparison: human and random.

- •
Human Baseline: We selected approximately 20 questions from each of the 30 games, resulting in 623 questions. These were grouped into 30 sets and assigned to STEM undergraduates unfamiliar with the games. Each question was presented in a PowerPoint slide (Figure [12](https://arxiv.org/html/2505.13886v6#S10.F12)) using python-pptx(^1^11https://python-pptx.readthedocs.io/en/latest/), and responses were collected via an online questionnaire.(^2^22https://www.wjx.cn/)
- •
Random Baseline: This represents the lower performance bound, calculated as the expected score from random guessing on multiple-choice questions, with fill-in-the-blank tasks contributing zero.

Figure: Figure 12: Two example PowerPoint slides demonstrating how the student participants, during the human baseline evaluation, took the tests and might write or draw on the slides.
Refer to caption: images/human_participant_do_on_ppt.png

### 10.2 Data preparation

#### 10.2.1 SFT data preparation

- •
Game data preparation.
First, we used our data engine to generate 5000 task instances from 20 in-domain games, obtaining 100000 samples in total. Then we performed data augmentation, as shown in Appendix [13.3](https://arxiv.org/html/2505.13886v6#S13.SS3). Finally, we filtered these samples, as shown in Appendix [13.4](https://arxiv.org/html/2505.13886v6#S13.SS4). 40000 samples were selected for SFT.
- •
Other dataset preparation.
We prepared the Geo-Multi dataset for comparing the data quality. Geo-Multi consists of 40,000 samples, with 20,000 randomly sampled from MultiMath300K [22] and another 20,000 from Geo170K [6].

#### 10.2.2 Reinforcement Learning Data preparation

- •
Game data preparation.
We sampled 5000 samples from 20 in-domain games. This sample size is smaller than that used for SFT to balance performance and computational cost, as the GRPO training process is relatively resource-intensive.
- •
Other dataset preparation.
We sampled 4k examples from each of the geometry and function splits of MAVIS [35]. For multimodal-open-r1-8k-verified [16], we used all the
samples.

**Table 7: Selection of games for the game diversity scaling experiment across four cognitive categories.**
| Game Set | 3D Spatial<br>Perception and<br>Understanding | Pattern Recognition<br>and Matching | Multi-step<br>Reasoning | Strategic<br>Planning |
| --- | --- | --- | --- | --- |
| 4 Games | 3D Reconstruction | Tangram | Word-Search | TicTacToe |
| 10 Games | 3D Maze | Spider Solitaire | Tents,<br>2D Turing Machine | Sokoban,<br>Space Invaders |
| 20 Games | Rubik’s Cube | Freecell, Tetris,<br>Zuma, Color Hue | Langton’s Ant,<br>Rhythm Game,<br>Star Battle | Maze,<br>Ultra TicTacToe |

To investigate the impact of game diversity on model generalization, we constructed subsets of 4, 10, and the full 20 in-domain training games. The selection process aimed to maintain representation across our four defined cognitive categories by randomly selecting games from each category to form the smaller subsets. Table [7](https://arxiv.org/html/2505.13886v6#S10.T7) details the specific games included in each experimental set for this scaling analysis.

### 10.3 Training details

#### 10.3.1 Models

We trained four VLMs, InternVL2.5-8B [3], InternVL3-8B [4], Qwen2.5-VL-7B [2], and LLaVA-OneVision-7B [12] on our data.

#### 10.3.2 Training hyperparameters

LoRA-based supervised fine-tuning hyperparameters In this setup, Low-Rank Adaptation (LoRA) [9] was applied to all linear layers of the language model. We trained the model for one epoch. The rank was set to 16, with alpha set to 32. The learning rate was 5e-5, including a 3% warm-up period followed by a constant rate.

GRPO-based reinforcement learning In this setup, we conducted full-parameter fine-tuning of the language model while freezing the visual encoder and projection layers. We rollout 12 samples per questions. We trained the model for one epoch. The learning rate was 2e-7, with a 5% warm-up. The clipping value $\epsilon$ was set to 0.2 and the KL-divergence coefficient $\beta$ was set to 0.04. More hyperparameters are listed in Table [8](https://arxiv.org/html/2505.13886v6#S10.T8).

**Table 8: GRPO Hyperparameters.**
| Hyperparameter | Value |
| --- | --- |
| Learning Rate | 2e-7 |
| Batch Size | 3 |
| KL-divergence Coefficient ($\beta$) | 0.04 |
| Number of Generations | 12 |
| Temperature | 1.0 |
| Top-p | 0.85 |
| Top-k | 50 |
| Optimizer | AdamW |
| Warm-up Ratio | 0.05 |
| Weight Decay | 0.1 |

#### 10.3.3 Compute resources

LoRA-based supervised fine-tuning
For the models in the 7-8 billion parameter range, this LoRA-based SFT training was conducted on a single A800 GPU and the training duration lasted around 15 hours or less for 40k samples (1 epoch).

GRPO-based reinforcement learning

- •
7-8 billion parameter model Training on 5k samples (1 epoch) required approximately 22 hours, utilizing five A800 GPUs, including resources for the deployment of the evaluator model.
- •
3 billion parameter model The training process lasted approximately 18 hours for 5k samples (1 epoch) on four A800 GPUs, also including the evaluator model’s deployment.

GPU usage
Training a 7-8 billion parameter model with GRPO-based RL required approximately 22 hours, utilizing five A800 GPUs. This GPU allocation included resources for the deployment of the evaluator model. For a 3 billion parameter model, the training process lasted approximately 18 hours on four A800 GPUs, with this count also inclusive of the evaluator model’s deployment. For the 32 billion parameter model, the LoRA-based training took approximately 22 hours using eight H20 GPUs with 141GB memory each, while the reward model was deployed on four A100 GPUs.

### 10.4 Inference and evaluation

Besides trained models, we also evaluated proprietary large-scale models such as GPT-4o (20240806) [20] and Claude 3.5 Sonnet (20241022) [1], and open-source models that represent the current state of the art, including InternVL3-78B and Qwen2.5-VL-72B.

First, for evaluation on both GameQA and general vision benchmarks, the inference and evaluation configurations were unified across the original models and our trained models, detailed below.

For inference, the inference temperature parameter is set to 0.2, and the prompt is shown in Appendix LABEL:app_sec:prompt_for_inference. We evaluated the generated answers using the LLM-as-a-judge approach [37], with Qwen2.5-72B-AWQ acting as the evaluator and the prompt for evaluation shown in LABEL:app_sec:prompt_for_evaluation. To improve the reliability of the evaluation results, we introduced a series of engineering optimizations that ensured consistent and accurate assessments.

Second, We adopted different model evaluation strategies for different types of experiments to balance efficiency, accuracy, and the reliability of our conclusions, detailed below.

- •
Selecting the model checkpoint with best performance on the validation set for evaluation This approach is employed for the GRPO-based training experiments shown in Table [3](https://arxiv.org/html/2505.13886v6#S5.T3), [5](https://arxiv.org/html/2505.13886v6#S5.T5), [6](https://arxiv.org/html/2505.13886v6#S9.T6) and the SFT experiments detailed in Appendix [11.1](https://arxiv.org/html/2505.13886v6#S11.SS1). We saved 10 evenly spaced model checkpoints and evaluated the model that achieved the best performance on the validation set. The validation set was created by splitting 1% of the training data. It was found that the selected model was generally the final model in the training process. This approach efficiently and directly reflects the ultimate performance of our methods in practical scenarios.
- •
Evaluating the top three model checkpoints based on the validation set performance and averaging the evaluation results This is employed for the comparative training shown in Table [4](https://arxiv.org/html/2505.13886v6#S5.T4) and the analytic experiments of Section [5.4](https://arxiv.org/html/2505.13886v6#S5.SS4), Appendix [9.3](https://arxiv.org/html/2505.13886v6#S9.SS3) and Appendix [11.2](https://arxiv.org/html/2505.13886v6#S11.SS2). This approach was chosen because the performance differences are often subtle and susceptible to randomness in the training process. Averaging across multiple top checkpoints helps smooth out training noise, enhances the robustness and sensitivity of the evaluation, and more accurately reflects the differences under various experimental settings. This method also reduces the impact of chance results, making our experimental conclusions more reliable and representative.

## 11 More Experiments

### 11.1 SFT experiments

**Table 9: Evaluation results on GameQA benchmark. In-domain and out-of-domain game category results are shown. The percentage of performance improvements compared to the vanilla model is denoted by ($\uparrow$). Best performance per section is indicated in bold.**
| Models | Avg.<br>($\uparrow$) | In Domain Games | Avg.<br>($\uparrow$) | Out of Domain Games |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 3D Spatial<br>Perc. & Under. | Pattern Recog.<br>& Matching | Multi-step<br>Reasoning | Strategic<br>Planning | 3D Spatial<br>Perc. & Under. | Pattern Recog.<br>& Matching | Multi-step<br>Reasoning | Strategic<br>Planning |  |  |  |
| InternVL2.5-8B | 22.22 | 20.39 | 17.18 | 25.34 | 25.97 | 20.05 | 20.80 | 22.45 | 18.88 | 18.07 |
| + GameQA (SFT) | 46.73 (+24.51) | 42.38 | 45.55 | 51.56 | 47.44 | 25.81 (+5.76) | 24.40 | 24.69 | 25.90 | 28.25 |
| + GameQA (GRPO) | 29.44 (+7.21) | 26.74 | 26.05 | 29.51 | 35.44 | 23.87 (+3.82) | 25.00 | 25.12 | 24.91 | 20.45 |
| InternVL3-8B | 26.51 | 22.53 | 21.91 | 30.18 | 31.43 | 27.64 | 29.60 | 27.44 | 29.62 | 23.91 |
| + GameQA (SFT) | 50.72 (+24.21) | 46.91 | 45.42 | 55.91 | 54.63 | 26.02 (-1.62) | 24.20 | 14.80 | 32.03 | 33.05 |
| + GameQA (GRPO) | 33.09 (+6.58) | 27.94 | 28.52 | 36.81 | 39.07 | 28.80 (+1.15) | 29.20 | 25.31 | 34.59 | 26.09 |
| Qwen2.5-VL-7B | 25.78 | 22.58 | 21.92 | 25.21 | 33.40 | 27.09 | 23.60 | 29.20 | 26.21 | 29.34 |
| + GameQA (SFT) | 48.37 (+22.59) | 42.05 | 42.82 | 58.66 | 49.96 | 29.27 (+2.18) | 26.80 | 21.16 | 31.79 | 37.32 |
| + GameQA (GRPO) | 32.12 (+6.34) | 26.80 | 26.88 | 33.34 | 41.45 | 30.51 (+3.42) | 27.10 | 31.56 | 31.24 | 32.13 |
| LLaVA-OV-7B | 21.79 | 18.84 | 19.69 | 21.24 | 27.38 | 20.39 | 21.00 | 20.19 | 20.13 | 20.27 |
| + GameQA (SFT) | 45.40 (+23.61) | 38.50 | 39.47 | 54.78 | 48.84 | 22.74 (+2.34) | 22.00 | 17.17 | 25.16 | 26.63 |
| + GameQA (GRPO) | 33.49 (+11.70) | 29.87 | 31.10 | 30.96 | 42.03 | 23.34 (+2.95) | 27.20 | 20.05 | 23.55 | 22.57 |

**Table 10: Evaluation results on general vision benchmarks. The percentage of performance improvements compared to the vanilla model is denoted by ($\uparrow$). Best performance per section is indicated in bold.**
| Models | Avg.<br>($\uparrow$) | MathVista | MathVerse | MMBench | MMMU | CharXiv | MathVision | MMMU-Pro |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| InternVL2.5-8B | 45.89 | 57.50 | 36.04 | 81.93 | 47.96 | 31.70 | 28.87 | 37.25 |
| + GameQA (SFT) | 45.06 (-0.83) | 58.10 | 35.79 | 82.87 | 48.07 | 31.20 | 22.80 | 36.56 |
| + Geo-Multi (SFT) | 43.84(-2.05) | 53.60 | 36.90 | 82.80 | 43.17 | 30.40 | 27.80 | 32.22 |
| + GameQA (GRPO) | 47.91 (+2.02) | 61.70 | 37.11 | 83.87 | 50.06 | 32.00 | 31.93 | 38.69 |
| InternVL3-8B | 54.48 | 69.10 | 50.10 | 86.00 | 57.88 | 39.10 | 35.33 | 43.84 |
| + GameQA (SFT) | 49.66(-4.82) | 63.20 | 43.30 | 84.53 | 53.44 | 32.90 | 29.60 | 40.64 |
| + Geo-Multi (SFT) | 48.52 (-1.42) | 62.60 | 40.96 | 82.87 | 48.54 | 37.80 | 29.80 | 37.06 |
| + GameQA (GRPO) | 55.88 (+1.40) | 73.00 | 50.71 | 86.20 | 58.34 | 39.90 | 37.93 | 45.10 |
| Qwen2.5-VL-7B | 49.94 | 66.80 | 45.08 | 83.67 | 49.01 | 37.70 | 30.80 | 36.49 |
| + GameQA (SFT) | 47.26 (-2.72) | 63.00 | 37.31 | 83.07 | 47.49 | 38.60 | 25.73 | 35.62 |
| + Geo-Multi (SFT) | 48.52 (-1.42) | 62.60 | 40.96 | 82.87 | 48.54 | 37.80 | 29.80 | 37.06 |
| + GameQA (GRPO) | 52.27 (+2.33) | 68.20 | 47.97 | 83.53 | 50.53 | 42.70 | 33.07 | 39.89 |
| LLaVA-OV-7B | 41.23 | 55.60 | 33.05 | 81.13 | 41.07 | 27.10 | 23.40 | 27.26 |
| + GameQA (SFT) | 35.83 (-5.40) | 45.90 | 25.99 | 80.27 | 32.21 | 20.50 | 20.47 | 25.44 |
| + Geo-Multi (SFT) | 40.35 (-0.88) | 52.50 | 31.57 | 82.40 | 41.31 | 25.90 | 22.60 | 26.19 |
| + GameQA (GRPO) | 42.27 (+1.04) | 58.20 | 34.92 | 82.53 | 41.31 | 27.30 | 23.07 | 28.58 |

As described in Appendix [10](https://arxiv.org/html/2505.13886v6#S10), we performed additional supervised fine-tuning (SFT) experiments on four models. The results of these experiments are presented in Tables [9](https://arxiv.org/html/2505.13886v6#S11.T9) and [10](https://arxiv.org/html/2505.13886v6#S11.T10), and we found that:

The GRPO demonstrates an advantage over SFT in out-of-domain generalization, though SFT yields strong in-domain gains. When evaluating model performance, the effects of SFT and GRPO training methods showed clear differences across domains. Specifically, SFT training with the GameQA dataset led to substantial in-domain improvements: the InternVL2.5-8B and Qwen2.5-VL-7B models improved their average accuracy by 24.51% and 22.59% respectively on a test set covering 20 games, reaching final scores of 46.73% and 48.37% (Table [9](https://arxiv.org/html/2505.13886v6#S11.T9)). However, while SFT excels at improving in-domain performance, it can sometimes lead to performance degradation on general tasks as seen in [9](https://arxiv.org/html/2505.13886v6#S11.T9), a phenomenon known as "catastrophic forgetting". Crucially, for training sets that belong to the mathematics domain, such as Geo-Multi (see Appendix [10.2](https://arxiv.org/html/2505.13886v6#S10.SS2)), models trained on them can still exhibit a decline in general capabilities. This may reflect the tendency of the SFT method to overfit the model to specific domain data [5].

In contrast, when training on GameQA using the GRPO, models not only successfully avoided performance degradation on general tasks but also generally achieved performance improvements across multiple general visual benchmarks (Table [3](https://arxiv.org/html/2505.13886v6#S5.T3)). A typical example is the Qwen2.5-VL-7B model, which, after GRPO training, showed enhanced performance on challenging benchmarks such as MathVista, MathVerse, and CharXiv, reaching a level comparable to larger-scale models like InternVL2.5-38B.

Pure RL outperforms SFT-then-RL in out-of-domain generalization. To validate our choice of directly applying RL, we compare our approach to a two-stage SFT-then-RL pipeline on Qwen2.5-VL-7B. As shown in Table [11](https://arxiv.org/html/2505.13886v6#S11.T11), while the two-stage pipeline yields strong performance on in-domain tasks, it leads to a significant performance drop of -2.5% on general vision benchmarks. This suggests that full-parameter SFT on a narrow, specialized domain like GameQA can cause catastrophic forgetting. In contrast, pure RL better preserves and enhances out-of-domain generalization capabilities, achieving a +2.33% improvement.

**Table 11: Comparison of training pipelines on Qwen2.5-VL-7B. SFT is performed on 20k samples, followed by GRPO on 5k. Performance on general benchmarks is shown as a relative change from the baseline.**
| Training Stage | In-domain | Out-of-domain | General Benchmarks |
| --- | --- | --- | --- |
| Games | Games | (Avg. Change) |  |
| Baseline (Before SFT) | 25.89 | 26.92 | 0.0% |
| SFT | 56.21 | 30.74 | -3.9% |
| SFT-then-RL | 58.08 | 31.96 | -2.5% |
| Pure RL (Our Method) | 32.12 | 30.51 | +2.33% |

### 11.2 Training data difficulty experiment

To analyze how reasoning difficulty impacts generalization, we categorized tasks into Easy (55-85% baseline accuracy), Medium (30-55%), and Hard (5-30%). We trained Qwen2.5-VL-7B on 5k samples from different difficulty combinations. Table [12](https://arxiv.org/html/2505.13886v6#S11.T12) shows that training on a diverse mix of difficulties yields the best generalization. The model trained on "Easy+Medium+Hard" samples achieves the highest average score (52.74), outperforming models trained on simpler subsets. This confirms that the complex reasoning patterns in GameQA are crucial for improving general reasoning.

**Table 12: Impact of training data difficulty on generalization. Results are averaged over the top three checkpoints. The baseline score is 49.94.**
| Training Data (5k) | Average | MathVista | MathVerse | MMBench | MMMU | CharXiv | MathVision | MMMU-Pro |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline | 49.94 | 66.80 | 45.08 | 83.67 | 49.01 | 37.70 | 30.80 | 36.49 |
| Easy | 52.40 | 67.87 | 47.93 | 83.91 | 51.58 | 41.77 | 33.73 | 40.03 |
| Medium | 51.95 | 67.30 | 48.10 | 83.64 | 50.53 | 40.83 | 33.96 | 39.30 |
| Hard | 52.07 | 67.57 | 47.53 | 83.58 | 51.03 | 40.97 | 34.58 | 39.26 |
| Easy+Medium | 52.45 | 68.60 | 48.07 | 83.42 | 51.30 | 41.40 | 34.86 | 39.49 |
| Easy+Medium+Hard | 52.74 | 68.33 | 48.68 | 83.93 | 52.36 | 40.73 | 34.31 | 40.81 |

### 11.3 Data scale experiment

We extended our data scaling experiment up to 20k samples. While individual checkpoints exhibit fluctuations common in RL, a "binned averaging" analysis (Table [13](https://arxiv.org/html/2505.13886v6#S11.T13)) reveals a clear, monotonic positive trend. By averaging performance across training stages, we smooth out short-term noise and observe a stable improvement trajectory. The average performance on general benchmarks steadily increases from +2.31 in the early stage to +3.02 in the late stage, confirming that model performance continues to improve with more data from GameQA.

**Table 13: Binned averaging analysis of data scaling effect on Qwen2.5-VL-7B. Performance gains over baseline are shown in parentheses.**
| Training Stage | Sample Range | General Vision Benchmarks | In-domain Games |
| --- | --- | --- | --- |
| Baseline Model | 0k | 49.94 | 25.78 |
| Early Stage Training | 1k–5k | 52.25 (+2.31) | 27.68 (+1.90) |
| Mid Stage Training | 6k–15k | 52.57 (+2.63) | 34.97 (+9.19) |
| Late Stage Training | 16k–20k | 52.96 (+3.02) | 38.69 (+12.91) |

## 12 Approach details

### 12.1 Task category

Target Perception Task focuses on visual perception and basic state awareness. State Prediction Task, building directly on the perceptual capacities, needs predictions about state transitions. Strategy Optimization Task then needs both perceptual and predictive capacities to find optimal solutions. This progressive structure helps organize reasoning skills from simple to complex.

The conceptual outline for each category, using Sokoban as an example, is as follows:

- •
Target Perception Task: Queries static information within the game state. For instance, questions ask about the position or number of boxes, and the answers list the specific positions of each box by directly inspecting the current state.
- •
State Prediction Task: Infers state changes following actions. For instance, questions predict the player’s position after a sequence of moves. Answers are derived by analyzing the initial state, simulating the execution of each step, recording the resulting state changes, and thus determining the final player position.
- •
Strategy Optimization Task: Aims to find optimal solutions. For instance, questions ask for the shortest path to push a specific box to its designated target. The answer is derived by first analyzing the initial state to determine the optimal route for moving the box to its target, and then simulating the execution of this optimal action sequence.

### 12.2 Selection criteria of 30 games in GameQA

We selected these 30 games based on the following criteria.

- •
Ability coverage The games need to cover a diverse range of reasoning abilities, including 3D Spatial Perception and Understanding, Pattern Recognition and Matching, Multi-step Reasoning, and Strategic Planning.
- •
Code simplicity The code should be easy to construct, meaning they are simple enough to be programmed by an LLM, or are open-sourced.
- •
Static game They should be static or can be transformed into a static state, so that problems can be solved from a static image.

### 12.3 Time spent on the main steps of Code2Logic for each game

Figure [13](https://arxiv.org/html/2505.13886v6#S12.F13) illustrates the estimated time spent on implementing the main steps of the Code2Logic approach across all the 30 games in the GameQA dataset. The time ranges from a minimum of 4 hours to a maximum of 12 hours, with an average of 7.5 hours per game.

This average time investment is relatively cost-effective and appears highly acceptable, especially considering that once the code data engine is built, it can generate an unlimited number of data samples for training and evaluation purposes.

Figure: Figure 13: Estimated time (in hours) spent on implementing the main steps of Code2Logic across different games in the GameQA dataset, with an overall average of 7.5 hours per game.
Refer to caption: x13.png

### 12.4 Games generated using external open-source code

This section lists the games whose code is generated based on open-source code, as referenced in Section [2.1](https://arxiv.org/html/2505.13886v6#S2.SS1).

Spider Solitaire (Open-source code URL: [https://github.com/rdasxy/spider_solitaire](https://github.com/rdasxy/spider_solitaire))

Based on an original Python implementation of Spider Solitaire, our code reused its core rules and GUI. And we simplified the game to a single suit (Spades) to reduce complexity, enriched initial setups through LLM-implemented random deals, and adapted the original game rules into detailed instructions.

Klondike (Open-source code URL: [https://github.com/milorb/klondike](https://github.com/milorb/klondike))

Based on the original open-source Klondike Solitaire project built on Pygame(^3^33https://www.pygame.org/news), we adapted the code using an LLM into an automated dataset generation tool. It reused the core game engine and introduced a method for generating diverse random initializations.

Space Invaders (Open-source code URL: [https://github.com/leerob/space-invaders](https://github.com/leerob/space-invaders))

Based on the original Space Invaders game built with Pygame, we utilized its core elements and visual assets to generate static game scenes. Using an LLM, we converted the dynamic game into static scene snapshots for dataset generation.

## 13 More information about the GameQA Dataset

### 13.1 The GameQA dataset statistics

The full GameQA train and test set statistics table is shown in Table [14](https://arxiv.org/html/2505.13886v6#S13.T14).

### 13.2 Definition of each category

3D Spatial Perception and Reasoning Game of this type involves the ability to perceive, plan, and reason in 3D space to complete tasks such as navigation and spatial transformation. For example, in 3D Reconstruction, the task is to reconstruct the stacking arrangement of 3D cubes from a side view. Solving this task requires 3D spatial reasoning to establish the relationship between the 2D view and the 3D view.

**Table 14: GameQA train and test set statistics summary. All lengths are calculated by words.**
| Statistic Category | Train Set | Test Set |
| --- | --- | --- |
| Overall Counts |  |  |
| Total Games | 20 | 30 |
| Total Tasks | 102 | 158 |
| Total Questions | 126,760 | 15,047 |
| Image Statistics |  |  |
| Unique Images | 74,620 | 8,620 |
| Avg. Image Width (px) | 511.00 | 504.10 |
| Avg. Image Height (px) | 475.73 | 468.98 |
| Question Characteristics |  |  |
| Avg. Question Length | 275.27 | 272.43 |
| Avg. Analysis Length | 106.85 | 144.89 |
| - After Augmentation | 300.79 | - |
| Multiple Choice Questions | 86,520 | 10,518 |
| Avg. Choices for MCQs | 7.10 | 7.05 |
| Fill-in-the-Blank | 40,240 | 4,529 |

Pattern Recognition and Matching Game of this type requires capability on discerning and matching visual patterns related to object shapes, colors, combinations, and other regularities. For example, in Tangram, the task is to identify which piece can fill the empty space. Solving this task requires recognizing the shape of void and matching with given pieces.

Multi-step Reasoning Game of this type features multi-step reasoning and iteratively applying rules to reach the solution. For example, in Sudoku, the task is to infer which color should fill the empty space to ensure that no colors are repeated in the same row, column, or 3x3 grid. The "no repetition" rule needs to be applied repeatedly to deduct the correct color of a cell.

Strategic Planning Game of this type requires planning the optimal solution in optimization problems. For example, in Sokoban, the task is to plan the shortest path for pushing a box from the starting point to the target location.

### 13.3 Data augmentation

To prevent model overfitting to specific reasoning patterns, we employed an LLM-based reasoning paraphrase strategy using InternVL2.5-78B. Based on initial experiments showing that visual input could lead to errors due to the model’s insufficient visual capabilities, we provided it only with textual information, namely the question, original answer process and final answer, to rewrite the answer process. This approach enriches linguistic style and logical expression diversity while maintaining semantic consistency.

### 13.4 Data quality assurance

To ensure the high quality and reliability of our synthetic dataset, we implemented a quality assurance process, consisting of four stages:

- 1.
Human Inspection: STEM students inspected initial samples to ensure logical correctness, clarity and completeness of questions, images and reasoning steps.
- 2.
LLMs Check: Fed samples to GPT-4o and Claude-3.5 Sonnet to ensure model comprehensibility, identifying necessary refinements in the samples.
- 3.
Post-Augmentation Verification: Manually verified reasoning accuracy in a random subset of augmented data.
- 4.
Automated Data Filtering: Removed samples based on length, high repetition (>70% 4-gram overlap), or wrong answers, reducing the set from 1̃50k to 126,760.

## 14 Details on Sokoban task systhesis

### 14.1 Sokoban QA templates

Here we provide templates used for generating Sokoban puzzle samples, including the three problem types: Target Perception, State Prediction, and Strategic Optimization.

Target perception QA template The Target Perception template (Table [14.1](https://arxiv.org/html/2505.13886v6#S14.SS1)) is used to generate questions about identifying the current position of game elements.

Table: Table 15: Target perception QA template

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - What proportion of games required manual correction or external code reuse?
- What is the importance of task templates? 
- What happens if you just SFT on GameQA?
soundness: 3
strengths: - Scalable data-generation pipeline: Code2Logic programmatically maps game code to reasoning logic and auto-generates verifiable multimodal QA data.
- Demonstrates that purely synthetic, self-verifiable environments can modestly improve general VLM reasoning—important for RL reproducibility.
- Diverse benchmark coverage (3D perception, pattern matching, planning, reasoning).
- Clear, reproducible methodology; good visualizations and qualitative examples.
summary: This paper introduces Game-RL, a framework for vision-language reinforcement learning that uses video game environments as multimodal, verifiable training data. Using the proposed Code2Logic pipeline, game code is transformed into reasoning-oriented visual question–answering (VQA) tasks, producing the GameQA.
weaknesses: - Small gains on external benchmarks are statistically and practically modest; no significance tests or efficiency comparisons.
- Lack of ablation isolating contributions of Code2Logic data vs RL itself (no SFT vs RL comparison on the same data).
- Evaluator bias: rewards rely on QwQ-32B, potentially aligning to its own style and inflating self-consistency.
- Unclear verification metrics: “verifiable” is claimed, but no automated correctness guarantees are quantified.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: * In Appendix B.3, you note that training on a subset of 4 games led to better generalization than training on 10 games. You speculate this is due to "random factors" or a "well-chosen set." Could you elaborate on this?

* The Code2Logic pipeline works well for the 30 games presented, which are largely deterministic, static, or turn-based. How do you see this framework extending to more complex, real-time games with non-deterministic physics (e.g., Angry Birds) or continuous action spaces (e.g., a racing game)? Does the reliance on LLMs to generate the game code fundamentally limit the complexity of the games that can be synthesized?
soundness: 3
strengths: * This paper is clear writing and easy to follow.

* Dataset contribution: GameQA spans 30 games / 158 tasks with explicit difficulty control and verifiable answers.

* GRPO on GameQA yields consistent improvements on diverse general benchmarks
summary: This paper addresses the challenge of improving the general reasoning capabilities of VLMs, arguing that current RL training is overly focused on narrow domains like geometry and charts. The authors propose Game-RL, a framework that leverages video games as a rich, verifiable, and scalable source of training tasks. They introduce Code2Logic, a novel pipeline that uses LLMs to synthesize a large-scale dataset, GameQA, by first generating game code and then using that code to create verifiable VQA tasks. The central finding is that training VLMs on GameQA using GRPO with a simple outcome-based reward signal unexpectedly improves performance not only on unseen games but also across a suite of 7 diverse, out-of-domain general reasoning benchmarks.
weaknesses: * The Code2Logic pipeline is presented as highly scalable, but Section 2.4 and Appendix F.4 reveal a significant reliance on manual verification at every step (code, data engine, and augmented samples). Furthermore, the data augmentation relies on paraphrasing from InternVL2.5-78B, and data quality checks use commercial LLMs. This "human-in-the-loop" and "proprietary-LLM-in-the-loop" requirement makes the process less automated and scalable than implied.

* The work only generates massive amounts of game QA data; I don't believe this is the primary task that a 'game agentic' model should be excelling at. Instead, the model should be interacting directly within the code-driven game to generate trajectories for SFT or RFT, or to perform RLVR.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The paper highlights the Code2Logic pipeline as a key contribution for generating the GameQA dataset. However, Appendix E.3 notes an average of 7.5 hours of human effort was required per game. This suggests a significant reliance on manual oversight. Therefore, 
a) What are the most common failure modes of the LLMs in this pipeline? What kinds of errors (e.g., logical inconsistencies in game rules, un-compilable code, nonsensical QA pairs) required the most frequent manual intervention?
b) Given this reliance on expert human-in-the-loop validation, how do you envision this approach scaling to a much larger and more diverse set of games (e.g., hundreds or thousands) without becoming prohibitively expensive and time-consuming?
2. Justification of the Reward Design. The RL training utilizes a sparse, binary reward based on the final answer's correctness. However, your GameQA dataset commendably contains detailed, step-by-step reasoning chains in the "analysis" field for each question. This rich information seems ideal for process-based supervision or a denser reward signal that could more effectively guide the model toward correct reasoning patterns.
a) Did you experiment with using these ground-truth reasoning steps to shape the reward signal (e.g., via sequence matching or a separate reward model trained on the analysis text)?
b) If not, could you elaborate on the rationale for choosing a sparse, outcome-based reward? Especially for complex reasoning, why is this preferable to process supervision, which is often considered more effective for teaching reasoning?
3. Reliability of the LLM-as-a-Judge. The entire Game-RL training loop hinges on the accuracy of the Qwen2.5-72B model acting as the judge for reward allocation. The validity of your experimental results is therefore contingent on the reliability of this judge.
Could you please provide an evaluation of the judge model's performance on the GameQA test set? Specifically, what are its accuracy, precision, and recall when compared against the ground-truth answers? An agreement score against human evaluators on a representative sample would be particularly insightful.
soundness: 2
strengths: - The motivation is clear and sound. The authors correctly identify a critical limitation in the current VLM training paradigm, the over-reliance on narrow, static domains like geometry or chart reasoning. The proposal to use video games as a more dynamic, verifiable, and diverse training environment for fostering general reasoning is an interesting idea. 
- The GameQA dataset could be a substantial contribution to the community. Its scale (30 games, 158 tasks, 140K samples), diversity across four distinct cognitive categories, and controllable difficulty levels make it a valuable resource for both training and benchmarking VLMs. The effort invested in its creation is evident, and it provides a tangible asset that can spur further research in this area.
- The paper presents a comprehensive and rigorous experimental evaluation. The authors test their approach not just on a few models but on a range of popular open-source VLMs. Crucially, evaluation is not confined to their own dataset; they demonstrate generalization by testing on seven diverse external benchmarks. The inclusion of ablations on data quantity and game diversity further strengthens the quality and credibility of the empirical findings.
- The paper is well-written, logically structured, and easy to follow. The Code2Logic pipeline is explained with a clear diagram (Figure 1), and the examples provided throughout the paper and appendix effectively illustrate the nature of the tasks in GameQA. This clarity makes the work accessible and its contributions understandable.
summary: This paper addresses the narrow domain focus of current reinforcement learning (RL) methods for Vision-Language Models (VLMs), arguing that such limitations hinder the development of general reasoning. To this end, the authors propose Game-RL, an approach that leverages diverse and verifiable video game environments for training. To generate the necessary training data, they introduce Code2Logic, an LLM-assisted, three-stage pipeline for synthesizing game code, task templates, and ultimately, game-based VQA samples. The primary tangible contribution is the resulting GameQA dataset, a large-scale collection of 140K question-answer pairs spanning 30 distinct games and 158 tasks, categorized by cognitive skills. The authors demonstrate that training VLMs solely on GameQA via an RL algorithm (GRPO) yields consistent performance improvements across a suite of seven external, general-purpose vision-language benchmarks, thereby providing evidence for the transferability of reasoning skills learned within these game environments.
weaknesses: 1. The novelty is somewhat incremental and limited. The Code2Logic pipeline, while well-executed, is fundamentally a specific instance of LLM-based synthetic data generation. This approach is becoming increasingly common, and the paper does not sufficiently differentiate its technical contribution from existing work in program-aided or LLM-driven data synthesis. It seems to be more of an extensive engineering protocol than a novel, generalizable method. Similarly, Game-RL is an application of an existing RL algorithm (GRPO) to a new dataset. The name might imply a novel RL framework tailored for games, which is not the case. The contribution is thus more of an empirical finding ("RL on game data generalizes") rather than a new RL technique.
2. There is a lack of reproducibility in data generation. The Code2Logic pipeline is described as "LLM-assisted," but the degree of human intervention appears substantial, undermining its scalability and reproducibility. The appendix notes an average of 7.5 hours of human effort per game. This suggests a highly iterative and manual process of prompt engineering, code verification, and template refinement. For the work to be truly reproducible, the authors should provide a much more detailed account of this human-in-the-loop process, including the specific prompts, failure cases encountered with the LLM, and the nature of the manual corrections required.
Insufficient Justification for the RL Setup: The design of the reinforcement learning component is not adequately justified and appears suboptimal for the stated goal of improving reasoning.
3. The use of a sparse, binary (0/1) reward based only on the final answer is a weak signal for complex, multi-step reasoning tasks. It provides no credit for partially correct reasoning chains and fails to guide the model on how to arrive at the correct answer. The analysis field already present in the GameQA dataset seems perfectly suited for developing a more informative, process-based reward signal. The paper would be significantly stronger if it explored this or at least provided a detailed justification for why this simpler reward scheme was chosen.
4. The use of a powerful VLM (Qwen2.5-72B) as a judge introduces a potential ceiling effect and a source of bias. The paper provides no evaluation of this judge's accuracy on the GameQA tasks. Without knowing the reliability of the reward model, it is difficult to assess the quality of the training signal the agent receives. The authors should report the judge's agreement rate with human annotations on a sample of the data.
5. While the paper claims to boost "general reasoning," the connection is supported by relatively small performance gains on the external benchmarks. Furthermore, the selection of the 30 games appears somewhat arbitrary. The claim would be more convincing if the game selection process can demonstrate diversity and wide coverage.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. How does performance change on sequential decision making tasks requiring vision? Perhaps on some of the game benchmarks you mentioned? It feels weird mentioning other game environments and then not testing performance on them.

2. Would it be possible to test more seeds to get error bars? Are the improvements significant or within error range? I understand that these experiments are likely quite expensive, but at least 3 seeds on some of the trainings and tasks could be useful, especially given the tight margin of improvement.
soundness: 2
strengths: 1. The verifiable synthetic data generation is a timely and important problem given the landscape of LLM/VLM trainings with RL.
2. Using games to improve vision capabilities in a verifiable way is an interesting contribution.
summary: The paper proposes a method for verifiable synthetic vision-language data generation with video games, by training VLMs on this synthetic data with GRPO, they get improvements in other benchmarks for vision language models.
weaknesses: 1. The games used for data generation are all grid-like games with very simple interfaces.
2. Games are inherently sequential-decision making tasks, so it feels strange that the authors did not then test performance on any of the game benchmark they mention. This seems like a natural extension to the project so far.
3. The performance improvements of 1 to 2% are quite small, and using a single seed might not be enough to properly test this.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: •	As mentioned in Section 2.2, the QA templates are either (1) human-designed and refined by LLMs, or (2) generated by LLMs and manually validated by humans. What is the proportion between these two approaches, and what are the respective time costs involved in each process?
•	The *Code2Logic* pipeline employs LLMs in multiple stages while keeping humans in the loop. As I understand it, this process still requires considerable manual effort. Have the authors considered adopting a more agentic approach to further reduce human labor by leveraging LLM agents more extensively?
soundness: 4
strengths: •	Data Quality: A wide range of video games are selected, ensuring diversity and strong generalization potential in the training data. The synthesis process leverages LLMs across multiple stages and includes manual validation, which guarantees both the efficiency and reliability of the data generation pipeline.
•	Clarity: The approach is clearly presented, supported by a well-illustrated pipeline diagram and numerous examples in the appendix. The experiments are reported in a detailed and organized manner. 
•	Significance: The experiments cover a diverse set of models and benchmarks, outperforming most comparable datasets. Comprehensive ablation studies, such as scaling effect analyses, further strengthen the empirical evidence. Together, these results provide strong support for the overall effectiveness of GameQA.
summary: To address the lack of large-scale, high-quality data for RL training in video game contexts, this work:
(1) Proposes *Code2Logic*, a pipeline that leverages LLMs to synthesize reasoning data based on game code; and
(2) Constructs *GameQA* a large dataset of diverse tasks and 140K VQA examples. After post-training MLLMs on this dataset, the authors report improved reasoning performance across multiple general-domain benchmarks, achieving better out-of-domain generalization compared to other comparable datasets.
weaknesses: •	Game-RL can to construct large-scale datasets for diverse games at relatively low cost. Nevertheless, the idea of synthesizing game-based benchmarks to enhance MLLMs’ reasoning ability via RL is a well-established and commonly explored direction. There already exist many high-quality, large-scale benchmarks—such as VisualSphinx (https://arxiv.org/abs/2505.23977) VGRPBench (https://arxiv.org/abs/2503.23064) and other recent game-based reasoning datasets—that share similar objectives, and some of them also asserts that RL training on their dataset can improve MLLMs’ reasoning ability in general-domain. Therefore, the overall objective of this paper is not highly novel.


Decision: Accept (Poster)