RLVR-World: Training World Models with Reinforcement Learning
Authors: name: Jialong Wu; name: Shaofeng Yin; name: Ningya Feng; name: Mingsheng Long
Published: 2025-05-20 05:02:53

Abstract: World models predict state transitions in response to actions and are
increasingly developed across diverse modalities. However, standard training
objectives such as maximum likelihood estimation (MLE) often misalign with
task-specific goals of world models, i.e., transition prediction metrics like
accuracy or perceptual quality. In this paper, we present RLVR-World, a unified
framework that leverages reinforcement learning with verifiable rewards (RLVR)
to directly optimize world models for such metrics. Despite formulating world
modeling as autoregressive prediction of tokenized sequences, RLVR-World
evaluates metrics of decoded predictions as verifiable rewards. We demonstrate
substantial performance gains on both language- and video-based world models
across domains, including text games, web navigation, and robot manipulation.
Our work indicates that, beyond recent advances in reasoning language models,
RLVR offers a promising post-training paradigm for enhancing the utility of
generative models more broadly.