Title: RLVR-World: Training World Models with Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: world models

reinforcement learning with verifiable rewards
Abstract: World models predict state transitions in response to actions and are increasingly developed across diverse modalities. However, standard training objectives such as maximum likelihood estimation (MLE) often misalign with task-specific goals of world models, i.e., transition prediction metrics like accuracy or perceptual quality. In this paper, we present RLVR-World, a unified framework that leverages reinforcement learning with verifiable rewards (RLVR) to directly optimize world models for such metrics. Despite formulating world modeling as autoregressive prediction of tokenized sequences, RLVR-World evaluates metrics of decoded predictions as verifiable rewards. We demonstrate substantial performance gains on both language- and video-based world models across domains, including text games, web navigation, and robot manipulation. Our work indicates that, beyond recent advances in reasoning language models, RLVR offers a promising post-training paradigm for enhancing the utility of generative models more broadly. Code, datasets, models, and video samples are available at the project website: https://thuml.github.io/RLVR-World.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWorld models are becoming increasingly important for robotics, and the current approach of training them through MAE/MSE regression can yield bad results, like blurry outputs. Related works in LLMs have found that RLVR increases performance of sequence modeling, which is related to the objectives of world models. This paper brings RLVR to these world models by defining verifiable rewards in a text exploration environment and using similarity metrics to ground truth video frames in the video world model setup. Results show meaningful improvement of both text and video models after RLVR. World models trained with RLVR have downstream benefits, including planning (MPC) and real-to-sim evaluations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- Paper proposes a reasonable application of RLVR to world models: like proofs and science, there are certain facts about the world that are very rule-based, like physics and other causal chains.\n- Verified that RLVR works not only on text world models, but also on video world models with meaningful qualitative differences between pre-RLVR and post-RLVR models\n- verified that the model can work with MPC in the text environment, showing meaningful downstream improvements (table 2)\n- showed that the video model can be used for real-to-sim evaluation and quantiative results show RLVR as closest to corresponding to real success rate.\n- good inclusion of repetition rejection as a baseline for multi-step prediction\n\nWeaknesses\n\n- rule-based rewards make sense in the text environment but the connection is weaker for the video model. The objective just becomes another distance metric to a ground truth. A closer analogy to rule-based rewards would be understanding if the laws of physics is being obeyed, if the task is being represented conceptually correctly, etc.\n- Related to this past point: GRPO gradient and the objective gradients are different, although on this bandit setting, it’s not immediately obvious what GRPO can do that directly optimizing the reward objective can’t do.\n- For video world model, it could be interesting to see other objectives for RLVR to showcase how it can work beyond one type of loss function. In particular, it could be interesting to show how it can work with non-differentiable objectives that can’t be directly optimized upon.\n- Summarizing some of the points above: the most notable weakness is with the video model and separating the effects of reinforcement from the choice of objective. I am willing to improve my marks on “significance” and “originality” if a more suitable explanation and/or experiments are done on this point.\n\nQUESTIONS:\nQuestions\n\n- From Figure 4 Right, it seems like training on LPIPS objective directly can get comparable performance (0.124) to the convergence value of RLVR (Figure 3 Left). Is there an explanation and/or experiment that could differentiate the 0.124 achieved by directly optimizing on LPIPS vs. running RLVR? Or, is the difference significant already?\n- Related to the question above, what happens when you train the video world model directly on the reward objective? It could be interesting to see this comparison to gauge the impact of the reinforcement approach over just finding a better objective. If I get these results and there is indeed a significant difference, I’m willing to increase my scores\n- The paper mentioned that there are no off-the-shelf video world models, although there is a recent world model (UVA, https://unified-video-action-model.github.io/) that could potentially be relevant. It could be interesting to show results on this or another similar off-the-shelf model to verify your findings, although this is a heavy lift and I don’t expect this to be done during the review process.\n- How is repetition defined? In pixel space or in a different representation?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce RLVR-World, a novel unified framework that leverages reinforcement learning with verifiable rewards to directly optimize world models for downstream task objectives. Through extensive evaluations on both language- and video-based benchmarks, RLVR-World achieves significant and consistent performance improvements across diverse domains.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- Extends the RLVR framework—originally formulated for pure LLM settings—to vision-based world models, thereby broadening its applicability.\n- The authors conduct extensive experiments on both language- and video-based world models across multiple domains, demonstrating the robustness and generality of their approach.\n- Overall, the paper is well written and easy to follow.\n\n**Weaknesses:**\n\n- The paper’s central paradigm—leveraging a pretrained model followed by task-specific fine-tuning (whether via supervised learning or reinforcement learning)—is by now a well-worn recipe, particularly in NLP. Extending this two-stage approach to autoregressive video world models is sensible, but it does not in itself constitute a surprising or fundamentally new insight, especially a number of prior works have already framed world-model learning as autoregressive sequence prediction over discretized multimodal tokens as mentioned in section 2.\n\nQUESTIONS:\n- The scope of the evaluation is currently limited to narrowly-defined, task-specific scenarios (e.g., RT1). By contrast, DeepSeek-R1 applies the RLVR framework across a diverse set of domains—math, code, reasoning—and demonstrates consistent gains, suggesting that RLVR can endow models with broader “general” capabilities. Could the authors extend their analysis to assess whether RLVR likewise enhances the general-purpose predictive power of the world model? Without such analyses, the reported results remain consistent with the well-known benefits of the pretrain–finetune paradigm rather than demonstrating novel, task-agnostic advances.\n- About Figure 5, a smaller measured discrepancy does not necessarily correspond to a reduced sim-to-real gap. For example, imagine evaluating 100 trajectories: the first 50 succeed only in the real environment and the remaining 50 succeed only in simulation. In this case, the discrepancy metric would be zero, yet the sim-to-real performance gap is maximal.\n\n**Minor Comments**\n\n- (Line 220) “observation conditioed on” -> “observation conditioned on”\n\nLIMITATIONS:\nThe authors discuss the limitations of their work in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

comments: SUMMARY:
This paper presents RLVR-World, a framework that applies reinforcement learning with verifiable rewards (RLVR) to optimize world models across different modalities, including language and video. It investigates the use of RLVR for fine-tuning language models in text games and web navigation tasks, as well as video models for robot manipulation trajectory prediction. The results show performance improvements in prediction accuracy and visual metrics, and the framework is also applied to downstream tasks such as model predictive control and policy evaluation.

STRENGTHS AND WEAKNESSES:
Strength：
1. The idea of using RLVR to optimize world models is novel and represents a significant departure from traditional training objectives.
2. The paper pioneers the application of RLVR in video world models, demonstrating its potential beyond language models. This opens up new avenues for research in generative models for visual data.

Weakness:
1.  The paper does not provide a detailed comparison with other recent advances in video world models

QUESTIONS:
1. " Why does the model's accuracy improve by +34.7% for 'unchanged' cases and by +8.9% for 'changed' cases in Text Game State Prediction?   Does this difference reflect an inherent mechanism within the model for dealing with different situation?"

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to train world model on text or video based transition datasets with reinforcement learning instead of the conventional maximum likelihood estimation. Specifically, the inputs of states and actions are tokenized and the model is trained to predict the next state via reinforcement learning. The authors evaluate the proposed method on text-based transition dataset and a manipulation task, and show that the proposed method can achieve better performance than the supervised fine-tuning (SFT) baseline.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n* The paper is well-written and clearly presents the motivation, method, and results of the proposed approach. The clarity of the writing makes it easy to follow the logic and understand the contributions.\n\n## Weaknesses\n\n* The overall technical contribution of the paper is limited. The tokenization, model architecture, and training method are not novel. This paper does not provide a significant insight or advancement in this field.\n\n* The proposed method of training a world model is very general, while the evaluations are mostly limited to either text form under a language model or a single visual dataset of manipulation. There are diverse datasets in the field of world model, and experiments in other popular domains of robotic or gaming simulations are worthwhile to conduct [r1].\n\n* In experiments, there are very few baselines for comparison with the proposed world model training. The main comparisons are executed based on vanilla pre-trained models and SFT. Many other methods of world model like recurrent latent state models are also important baselines [r1].\n\n* The size of the language model used is much smaller than the state-of-the-art models, which may limit the scalability of the proposed method over larger models.\n\n* It seems that there is no experiments over model predictive control (MPC) in the manipulation task. One major application of world models is to enable MPC in robotic manipulation tasks without interaction with the true environment. The paper would benefit from including experiments on MPC with the proposed world model.\n\n[r1] Zhou G, Pan H, LeCun Y, Pinto L. Dino-wm: World models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983. 2024 Nov 7.\n\nQUESTIONS:\nSee strengths and weaknesses part.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}