Matrix-Game: Interactive World Foundation Model
Authors: name: Yifan Zhang; name: Chunli Peng; name: Boyang Wang; name: Puyi Wang; name: Qingcheng Zhu; name: Fei Kang; name: Biao Jiang; name: Zedong Gao; name: Eric Li; name: Yang Liu; name: Yahui Zhou
Published: 2025-06-23 14:40:49

Abstract: We introduce Matrix-Game, an interactive world foundation model for
controllable game world generation. Matrix-Game is trained using a two-stage
pipeline that first performs large-scale unlabeled pretraining for environment
understanding, followed by action-labeled training for interactive video
generation. To support this, we curate Matrix-Game-MC, a comprehensive
Minecraft dataset comprising over 2,700 hours of unlabeled gameplay video clips
and over 1,000 hours of high-quality labeled clips with fine-grained keyboard
and mouse action annotations. Our model adopts a controllable image-to-world
generation paradigm, conditioned on a reference image, motion context, and user
actions. With over 17 billion parameters, Matrix-Game enables precise control
over character actions and camera movements, while maintaining high visual
quality and temporal coherence. To evaluate performance, we develop GameWorld
Score, a unified benchmark measuring visual quality, temporal quality, action
controllability, and physical rule understanding for Minecraft world
generation. Extensive experiments show that Matrix-Game consistently
outperforms prior open-source Minecraft world models (including Oasis and
MineWorld) across all metrics, with particularly strong gains in
controllability and physical consistency. Double-blind human evaluations
further confirm the superiority of Matrix-Game, highlighting its ability to
generate perceptually realistic and precisely controllable videos across
diverse game scenarios. To facilitate future research on interactive
image-to-world generation, we will open-source the Matrix-Game model weights
and the GameWorld Score benchmark at https://github.com/SkyworkAI/Matrix-Game.