GridAgent: A 2D Grid-Based Game Framework And Benchmark For Multimodal Large Language Models

Authors: Hengwei Ye, Yuanting Guan, Yuxuan Ge, Zheng Tian
Primary area: datasets and benchmarks
Keywords: MLLM; benchmark; game

Abstract:
Multimodal Large Language Models (MLLMs) integrate the linguistic capabilities of LLMs with the ability to process multimodal data, enabling them to address a wider array of tasks. However, a comprehensive and standardized benchmark for evaluating MLLMs' complex visual reasoning performance in multimodal tasks has yet to be established. We introduce GridAgent, a versatile 2D grid-based framework that serves as a benchmark for assessing five essential capabilities of MLLMs: execution, perception reasoning, memory, learning, and planning. The framework includes twelve unique game tasks specifically designed to avoid overlap with the model's pre-training corpus. Each task targets at least one core competency and is enriched with diverse semantic information. Additionally, the game layouts are randomly generated, ensuring a more rigorous and authentic assessment of the MLLMs' capabilities. Experimental results indicate that although certain MLLMs excel in specific capabilities, none exhibit a comprehensive skill set comparable to the human baseline. Our work can be seen at: https://iclr2025gridagent.github.io/GridAgent-website.

Decision: reject
Reviewer scores: 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730367199349

thread_type:
official_review

<thread_object>
creator: reviewer_sehm

<content>
summary:
The paper introduces GridAgent, a 2D grid-based benchmark framework designed to evaluate Multimodal Large Language Models (MLLMs) across five core capabilities: execution, perception reasoning, memory, learning, and planning. This is accomplished through twelve distinct game tasks. The results indicate that while models like GPT-4o perform well in specific tasks, none achieve human-level performance across all categories.

The article currently still has some obvious issues and writing structure that need to be addressed. Nonetheless, this contribution is significant, and with further clarification during the rebuttal phase, it could be accepted for the conference.

strengths:
- The benchmark is well-structured, effectively targeting key competencies in MLLMs and providing a robust evaluation framework.
- The introduction of randomized game layouts enhances the test's generalization and robustness by minimizing overfitting to training data.
- The paper presents detailed empirical evaluations of multiple MLLMs, offering clear comparative insights.
- The use of diverse tasks ensures a comprehensive assessment of the models’ abilities, extending beyond simple task-solving to encompass more complex multimodal reasoning, while minimizing overlap with pre-training data, thus enhancing validity.

weaknesses:
- There is a limited discussion regarding the selection of specific game tasks and their comprehensive coverage of the intended capabilities.
- For some tasks, such as Puzzle and Sorting, model performance results close to random, raising questions about the alignment of task complexity with the current capabilities of models.
- Although the paper mentions the randomness in game layouts to avoid overfitting, it lacks sufficient analysis on how varying levels of task complexity affect model performance or whether simpler versions of the benchmark could serve as a more effective baseline. This introduces potential biases due to the artificial nature of the tasks.
- The comparison between MLLMs and human benchmarks appears overly simplistic, lacking a detailed exploration of potential improvements to models in order to bridge the performance gap.

questions:
- The explanation of certain tasks, such as Perception Reasoning, lacks clarity regarding how performance metrics are derived and how these align with human cognition benchmarks.
- Section 5 exhibits some structural confusion, indicating significant issues in the writing structure of the Methodology section. The structure could be better organized to flow logically into the methodology and experimental analysis.
- The current comparative approach between MLLMs and humans is somewhat simplistic; exploring alternative comparison methods could provide deeper insights.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731085374373

thread_type:
official_review

<thread_object>
creator: reviewer_gbmp

<content>
summary:
The paper presents GridAgent, a benchmark inspired by Wechsler Intelligence Test to  evaluate five MLLM capabilities: execution, perception reasoning, memory, learning, and planning. To do so, the paper proposes 12 games in 2D grid based environment which are diverse semantic environments, with randomized layouts and varying difficulty to test generalization and robustness of MLLM models. Proposed tasks evaluate multimodal reasoning and understanding abilities of existing MLLMs. Finally, the paper presents quantitative comparison of state-of-the-art proprietary and open-source MLLMs on the GridAgent benchmark and find that on majority of these tasks current MLLMs achieve performance closer to random chance performance. Additionally, the paper presents some preliminary analysis that suggests current MLLMs lack training for image-only perception tasks and it is difficult for these models to solve tasks that require contradictory knowledge from what is learned by these models.

strengths:
1. The paper is well written and easy to follow
2. The tasks in the benchmark are thought through and carefully designed to test the five key capabilities the paper is focused on. I also like how authors created a set of tasks that require composing these capabilities to solve some of the tasks.
3. The experiments section covers most popular state-of-the-art MLLMs apart from Gemini. The zero-shot evaluation results on the tasks clearly demonstrate that these models are lacking zero-shot generalization ability to solve the proposed tasks.

weaknesses:
1. For completeness I’d recommend authors to add results for latest Gemini model in comparison as well.
2. The results demonstrated in the paper are under zero-shot setting. I’d be curious to see how well all of these models ( or a subset of these  for which it is possible given constraints on context length) perform on these tasks with 1-2 in-context examples. We have seen remarkable results with In-Context learning for these long-context models and it’d be good to validate if these tasks can be solved with in-context examples. I believe it should help for SO, FI, and PL tasks. Adding these will also make paper stronger and help highlight the difficulty of tasks in the benchmark
3. Some details about the resolution of image input and how it is fed to each of these models is missing in the paper and supplementary. My understanding is - for all tasks where agent needed to take multiple actions to complete the task the input was fed to the model step by step. Is that the case? If yes, I’d recommend authors to add more details about how the evaluation was done, the resolution of image observations used, etc to the paper.
4. Did the authors ever run experiments with higher resolution image inputs for some of the tasks like Filing? I’d imagine these models should perform better on the task if input image is much higher resolution. If not, I think it would be a good experiment to add to the paper. This would clearly highlight the lack of perception reasoning ability if the input resolution is not a bottleneck.

questions:
I’d also recommend authors to add evaluation examples where some of these models succeeded in supplementary. Examples from the best model should be good to add

I believe the benchmark is interesting and valuable to the community if authors address my concerns I'd be happy to increase my rating.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731195062024

thread_type:
official_review

<thread_object>
creator: reviewer_bi4b

<content>
summary:
The authors introduce GridAgent, a new benchmark for evaluating "execution", "perception reasoning", "memory", "learning" and "planning" abilities of Multimodal Large Language Models. The authors base this taxonomy on the Wechsler Intelligence Test. The benchmark includes 15 different tasks setups targeting one or more of the abilities.

strengths:
S1: The authors propose a taxonomy for evaluating visual abilities of MLLMs and ground it in existing intelligence test (Wechsler Intelligence Test) setup which has been used to evaluate human cognition

S2: The task setups and scenarios are reproducible and demonstrate the shortcoming of current MLLMs on visual reasoning

weaknesses:
W1: This work provides limited insight compared to previous works. [1] Provides a more in-depth analysis of memory capabilities of MLLMs. Embodied and Computer control benchmarks like [2], [3], [4] provide more real-world insight into the planning and execution abilities of MLLMs in visual setups. [5] provides a detailed analysis of perception reasoning abilities of MLLMs including analysis of Visual, Text and Joint reasoning abilities and failure modes. 

W2: Reasoning strategies like Chain-of-Thought [6], Self-consistency [7] are necessary for invoking reasoning abilities of LLMs. Similarly multi-turn tasks (like the ones introduced in this environment) are more suited to agentic formulation of LLMs like ReAct [8], and Reflexion [9]. The results sections lacks these details (did you use COT?) and evaluations of agentic frameworks. 

W3: Human evaluation settings are unclear and not well-motivated. Firstly, “Five players (including two authors) participated in the games through the GridAgent interface, with each completing five rounds across all tasks. " It is not well motivated to have authors who are intimately familiar with the task setups to serve as testers. Secondly, "The results show that any adult, once familiar with the rules and putting in serious effort, is fully capable of completing all the games." Where are these results? Lastly, "Consequently, we set the human baseline to 1 for all tasks.” This implies that the human baselines were set to 1.0 without actually completing the same set of tasks that the MLLMs solved. These facts need to be clarified and more details about the human evaluation need to be mentioned. 

W4: Lack of Variability studies: Does the performance of MLLMs across multiple trials remain consistent? Authors could evaluate this on a subset of tasks. 

W5: Lack of studies about scaling with model sizes: Does the performance of MLLMs on these tasks improve with more model parameters? All considered models are 7B parameters or less providing no information about whether larger model sizes correlated with better performance. 

[1] Wang, H., Shi, H., Tan, S., Qin, W., Wang, W., Zhang, T., Nambi, A., Ganu, T., & Wang, H. (2024). Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models. arXiv preprint arXiv:2406.11230. https://arxiv.org/abs/2406.11230

[2] Gan, C., Zhou, S., Schwartz, J., Alter, S., Bhandwaldar, A., Gutfreund, D., Yamins, D. L. K., DiCarlo, J. J., McDermott, J., Torralba, A., & Tenenbaum, J. B. (2021). The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI. arXiv preprint arXiv:2103.14025. https://arxiv.org/abs/2103.14025

[3] Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., & Yu, T. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arXiv preprint arXiv:2404.07972.

[4] Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P. Y., Neubig, G., Zhou, S., Salakhutdinov, R., & Fried, D. (2024). VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. arXiv preprint arXiv:2401.13649.

[5] Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., Manocha, D., & Zhou, T. (2024). HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models. arXiv preprint arXiv:2310.14566. https://arxiv.org/abs/2310.14566

[6] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2023). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv preprint arXiv:2201.11903. https://arxiv.org/abs/2201.11903

[7] Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv preprint arXiv:2203.11171. https://arxiv.org/abs/2203.11171

[8] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv preprint arXiv:2210.03629. https://arxiv.org/abs/2210.03629

[9] Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv preprint arXiv:2303.11366. https://arxiv.org/abs/2303.11366

questions:
Q1: GPT-4o performs worse than random and other, much smaller MLLMs on PU, PL, and SO tasks. Can you provide a discussion on why this is the case? Can you also check if other models which are larger than 7B also similarly perform worse on these tasks? 

Q2: How does the Level of difficulty affect LLMs performance? Could you add results across different difficult levels for those MLLMs? 

Q3: Do agentic workflows like ReAct and Reflexion improve the performance of MLLMs in these multi-turn tasks? 

Q4: Could you provide more details about how those specific abilities are connected to the Wechsler Intelligence Test?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732359755820

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response 1

comment:
We sincerely thank all reviewers for their thoughtful feedback. We are encouraged that our work is recognized as valuable (gbMp) and significant (SeHM), our tasks are considered thoughtful (gbMp), and our scenarios are deemed reproducible (bi4B). In this rebuttal, we provide a point-by-point response to all comments and include additional experiments conducted on the GridAgent benchmark.

Below, we summarize the major additions and adjustments made to the paper:

In the main paper:

- We incorporated the Gemini-1.5-Flash model into our benchmark evaluations to enhance completeness.
- We rephrased Section 5 (Tasks in GridAgent) to align with the sequence of capabilities introduced in Section 3. The revised section begins with tasks evaluating individual capabilities and progresses to tasks assessing composite capabilities, ensuring a clear hierarchy and logical flow.
- We updated Table 3 in Section 6 to replace the original success rate data with scores normalized to the human baseline. This adjustment enables readers to better compare the performance of different models with that of humans.

In the appendix:

- We introduced a new Section A to describe the connection between our benchmark and the Wechsler Intelligence Test, explaining how we designed the five core capabilities (Execution, Perception Reasoning, Memory, Learning, Planning) to evaluate MLLMs.
- A new Section B was added to provide additional experimental details, including the resolution setup and parameter selection during the inference stage of MLLMs.
- Section C was reorganized to specify the evaluation procedure of our benchmark, detailing the code structure, prompt design, action choices, and the decoding process for extracting answers from MLLMs.
- In Section E, we added new experiments to further support our work, including analyses of input resolution, model size, and the impact of in-context learning.
- We updated Section G (Task Information) by replacing the original human examples with successful examples from the best-performing model for each task.

These additions and adjustments strengthen the clarity, rigor, and reproducibility of our work and address the insightful feedback provided by the reviewers.

Furthermore, we would like to highlight some important changes and additional experiments:



C1 **(Difficulty Level)** Initially, we observed that most MLLMs performed reliably only at Level 1, with performance dropping significantly, often approaching the random baseline at Levels 2 and 3. Based on these findings, we focused on Level 1 results in the main paper to highlight scenarios where the models demonstrated meaningful capabilities. Additionally, some tasks originally did not differentiate between difficulty levels. In our latest version, every task includes three difficulty levels: Level 1 represents the simplest version of the task, while Levels 2 and 3 introduce progressively greater complexity through additional challenges. For tasks where MLLMs generally performed poorly, we also adjusted the difficulty settings to better align with the models’ current capabilities, ensuring a more balanced evaluation. Specific changes are detailed below, with additional information on difficulty levels provided in Section G.

- PU: Considering that the original PU was difficult (need to complete 4 puzzles), the new difficulty level 1/2/3 was adjusted to complete 1/2/3 puzzle(s) separately.
- SO:  Considering that the original SO was difficult (need to sort 3 animals according to the rule), the new difficulty level 1/2/3 was adjusted to rank 2/3/4 kinds of animals.
- PL: Considering that the original PL had only one difficulty (need to place an item according to the opposite instruction), the new difficulty level was adjusted to  perform operations according to simple/medium/hard rule. 
- FI & FI*: Considering that the original FI and FI\* had only one difficulty (need to complete 1 missing piece of an animal image), the new difficulty level 1/2/3 was adjusted to complete 1/2/3 missing piece(s) separately. Besides, we removed some of confusing options with only marginal information.
- MA & MA*: Considering that the original MA and MA\* was difficult (always appear 3 pairs of doors and keys), the new difficulty level 1/2/3 was adjusted to appear 1/2/3 pairs of doors and keys in the scene.

We recognize the importance of presenting the complete range of results to illustrate the impact of increasing task difficulty. Accordingly, we have now included the performance results across all difficulty levels (levels 1, 2, and 3) in Section B.4 of the appendix.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a synthetic 2D grid based multimodal benchmark to evaluate the different abilities like planning, learning new knowledge during interaction, memory, task execution. These abilities seem to be inspired from the  Wechsler Intelligence Test. The authors evaluated multiple closed-source and open-weights model including GPT4o, Gemini, QwenVL2-72b. 

The task setups and scenarios are reproducible and demonstrate shortcoming of current MLLMs. Some findings (for example, learning new knowledge which conflicts with the models' prior knowledge is difficult to learn) are insightful and could be beneficial to the community. 

However, several concerns raised during the review and during a closer inspection of the paper remain. 
- Reviewer bi4B pointed that the paper uses vanilla models without using Chain-of-Thought, ReACT and Reflexion style models. While these strategies are quickly evolving, they have found to be useful in many benchmarks, and will provide insights on how to more effectively use MLLMs. When presenting a new benchmark, it's important to show results on well-established / popular techniques to extract the most out of existing MLLMs. This provides a reasonable baseline for the benchmark, and will encourage the community to innovate beyond current techniques. 
- The work provides limited insights compared to previous works. While no work tests all the described capabilities in a single benchmark, there are several existing benchmarks (specially in the RL community) which test similar or related capabilities like perceptual understanding, action grounding and planning (Reivewer bi4B). 
- The paper will benefit from improved writing. While the paper writing improved during the rebuttal period, the paper still doesn't concretely show the agent's trajectory (prompt, task, responses) in one single place either in the main manuscript or supplementary