Empowering Multimodal Large Language Model as A Small Rule-Abiding World Simulator
Authors: Xingjin Wang, Hao Sun, Jiahao Zhao, Yinhe Zheng, Linjing Li, Daniel Dajun Zeng


Reviewer paper summaries: This paper introduces the small rule-abiding 2048 benchmark for evaluating the model's ability to learn world rules accurately. The paper further introduces a method to train a multimodal large language model whose performance is superior to other baselines on this 2048 benchmark.
Summary of strengths: - The paper is well-written.
- The proposed 2048 benchmark may be useful to researchers who are interested in world models.
- The proposed method provides insights on how different kinds of transition text can affect MLLMs' performance on the reasoning in a world simulator environment.
Summary of weaknesses: - One main motivation of this paper is current world simulators are quite complex thus this paper wants to build a simpler one with well-defined and limited numbers of rules with a 2D grid environment. But there already exist some simulation environments based on 2D grid format such as gym-minigrid [1], Bomberland [2]. Each environment also has strict rules. So the paper should also discuss the relationship and comparison with these existing 2D simulation environments though the paper regards their proposed benchmark as a world simulator.
- The paper replaces the number in the block with the special pattern cupcakes of different color to investigate the model's ability to learn visual rules. I am concerned about whether this kind of special cupcake pattern would bring bias to the recognition and reasoning abilities of VLMs or video models. The paper should also discuss this point.
- The introduced training method injects additional transition text for reasoning. However, all other baselines do not use this additional information. Is this setting unfair for comparison? 

References:
- [1] Chevalier-Boisvert, Maxime, et al. "Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks." Advances in Neural Information Processing Systems 36 (2024).
- [2] https://www.gocoder.one/bomberland/
Comments, suggestions and typos: Questions:
- The 2048 benchmark requires predicting the random new appearance. It is not predictable if I understood correctly, right?

 
Typos:
- reagrad -> regard on line 414?
Overall assessment scores: 3