RLVR-World: Training World Models with Reinforcement Learning
Authors: Jialong Wu
Shaofeng Yin
Ningya Feng
Mingsheng Long
Venue: NeurIPS 2025 poster ()
Keywords: world models
reinforcement learning with verifiable rewards
Primary area: deep_learning
TL;DR: We pioneer training world models through reinforcement learning with verifiable rewards (RLVR), demonstrating substantial performance gains on both language- and video-based world models.

Abstract: World models predict state transitions in response to actions and are increasingly developed across diverse modalities. However, standard training objectives such as maximum likelihood estimation (MLE) often misalign with task-specific goals of world models, i.e., transition prediction metrics like accuracy or perceptual quality. In this paper, we present RLVR-World, a unified framework that leverages reinforcement learning with verifiable rewards (RLVR) to directly optimize world models for such metrics. Despite formulating world modeling as autoregressive prediction of tokenized sequences, RLVR-World evaluates metrics of decoded predictions as verifiable rewards. We demonstrate substantial performance gains on both language- and video-based world models across domains, including text games, web navigation, and robot manipulation. Our work indicates that, beyond recent advances in reasoning language models, RLVR offers a promising post-training paradigm for enhancing the utility of generative models more broadly. Code, datasets, models, and video samples are available at the project website: https://thuml.github.io/RLVR-World.

RLVR-World, Training World Models with Reinforcement Learning
Jialong Wu', Shaofeng Yin 1.2 Ningya Feng', Mingsheng Long Ibs
School of Software, BNRist, Tsinghua University "Zhili College, Tsinghua University
vujiallong229gmmil.ccom com, mingsheng@tsinggua.eeuu cm
Abstract
World models predict state transitions in response to actions and are increasingly developed across diverse modalities. However, standard training objectives such as maximum likelihood estimation (MLE) often misalign with task-specific goals world models, i.e., transition prediction metries like accuracy or perceptual quality. In this paper, we present RLVR-World, unified framework that leverages reinforcement learning with verifiable rewards (RLVR) directly optimize world models for such metrics. Despite formulating world modeling as autoregressive prediction of tokenized sequences, RLVR-World evaluates metries of decoded predictions as verifiable rewards. We demonstrate substantial performance gains on both language- and video-based world models across domains, including text games, web navigation, and robot manipulation. Our work indicates that, beyond recent advances in reasoning language models, RLVR offers promising post- training paradigm for enhancing the utility of generative models more broadly. Code, datasets, models, and video samples are available at the project website: https: /ttpp://thuml.. gi thub. Me/RRRR-WWold..
Maximum Likelihoad Estimation (MLE)
s'
Reinforcement Learning from Verifiable Rewards (RLVR)
Samples
Decode
World Model
World Model
a
Samples
v Scalable x Surrogate Optimization
Task-aligned x Compute-heavy
Pre-train
SFT
World Model Training
Figure Training world models with reinforcement learning. (Left) As world models adopt increasingly advanced and scalable architectures, they are typically pre-trained or supervised fine- tuned using surrogate objectives such as maximum likelihood estimation (MLE), which misalign with task-specific prediction metrics. (Right) We propose post-training world models via reinforcement learning with verifiable rewards (RLVR) to directly optimize for these metrics.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
World models 17 34
which predict state transitions under action interventions, offer opportunities
for autonomously evaluating and optimizing the behavior policies of intelligent agents [20] 21]. By
scaling world models across various modalities such text [65 15]. video [6] D. and sensory
data [54] [4] -significntt progress has been made in a range of applications, including games 77,30,,
robotics
73 and autonomous driving [25]]]]]]] [2].
Effectively scaling world models involves training highly expressive models, e.g., Transformers, on massive data. This critically depends on differentiable, rich, and stable training signals. In practice, world models are typically trained using surrogate objectives such as maximum likelihood estimation (MLE). For instance, language models are trained via next-token prediction, which supports reasoning through chain-of-thought generation prior to producing final answers [67], while diffusion models optimize variational lower bound of the log-likelihood [23]. Moreover, non-end- to-end architectures that incorporate separately trained components like visual tokenizers [1]
enhance training efficiency and stability, especially for large-scale models. This paradigm has built powerful probabilistic world models, as well as many other foundation models [4].
However, the ultimate objectives of world models are beyond capturing complex distributions of data, but to meet the usage requirements of transition prediction metrics, such as high accuracy or perceptual quality. Surrogate non-end-to-en and optimizations are often infcasible for, agnostic to, or even diverge from this. In fact, even with differentiable objectives beyond likelihood, non-end-to-ene architectures like autoregressive models based discrete tokenizers or diffusion models cannot directly optimize them. Typical likelihood objectives are not well aligned with the world modeling task: in language models, likelihood-base objectives have been linked to issues like repetition and hallucination [24] 335 29, 644, and training video models with standard mean squared error is known produce blurry predictions
43). Moreover, the widely adopted training paradigm of teacher-forcing next-step prediction is also unaware of accumulation errors over multi-step horizons. promising emerging approach is tuning pre-trained models to directly optimize toward the target task via reinforcement learning with verifiable rewards (RLVR) 33 16), which replaces the learned reward model in reinforcement learning from human feedback (RLHF) 48 with a faithful, rule-based reward function. Using this approach, the language model community is now producing impressive advances in tackling complex math reasoning and code generation problems.
In this work, we explore the RLVR paradigm for training world models, referred to as RLVR- World. We first propose to unify world modeling across diverse modalities into general autoregressive generation framework. Concretely, current states and actions are encoded as a sequence of question tokens using modality-specifie tokenization schemes, while next states are encoded as response tokens, mirroring the language model formulation. Within this unified framework, we then comprehensively investigate RLVR for world models on two representative modalities:
Language world models: Beyond the success of large language models (LLMs) in math and code domains, introduce the world modeling task new testbed for RLVR in LLMs. This task, predicting the transition of verbal world states, naturally lends itself to using prediction accuracy as verifiable reward. Our experiments show that RLVR can effectively fine-tune LLMs language world models, yielding significant improvements, including +30.7% 7% accuracy on text-based game state prediction (65] and 15.1% FI score on web page state prediction (8].
Video world models: We pioneer the RLVR fine-tuning of autoregressive video world models [69] by directly measuring and optimizing perceptual metrics of decoded predicted frames against ground-truth observations. Notably, our method achieves substantial gains, e.g., +9.2% relative improvement on LPIPS (78), on robot manipulation trajectory prediction (5). with merely a few hundred RLVR gradient steps, in contrast to the hundreds of thousands required by MLE training to achieve. It also yields state-of-the-art performance compared with advanced world models [7]]3]] 31]
We further prove the effectiveness of RLVR in bridging the gap between pre-trained models and the target world modeling task by showing that it mitigates the repetition issue, a phenomenon commonly seen in LLMs [35] and also observed in our pre-trained video world model.
Finally, we demonstrate the utility of reinforced world models in downstream applications, including policy evaluation 37 and model-predictio control 8 We hope our method, experiments, and analysis will inspire future research to apply RLVR as a general post-training paradigm to significantly boost the usefulness of world models, and more broadly, generative models.

---

2
Related Work
World models. Learning accurate world models to predict environment state transitions in response to actions is fundamental to model-based planning and reinforcement learning [60]. Due to the inherent complexity and uncertainty of real-worle environments, world models more commonly implemented as generative models next states conditioned on current states and actions [10
177, rather than as deterministic models [53] 2T]. For visual observations, sequential variational while recent advances have seen the which offer superior visual fidelity. Another important line of work formulates world models autoregressively by discretizing visual inputs into This approach naturally extends modeling additional modalities through unified token-based state representations. For example, previous work explores language- based state representation [65]. which has been applied for building world models of the Internet in web agents [8] [5]. Similarly. trajectories of proprioceptive sensors and actuators can also be quantized into token sequences [54 74). Motivated by the prospect of building multimodal world models [32] [3] and the potential to leverage the large language model (LLM) ecosystem, our work explores the RLVR paradigm for training world models under the unified autoregressive formulation.
autoencoders have been widely adopted 128
19
18
68
emergence of diffusion-based world models 73
sequences of tokens 45 25 69 N.
RL for generative models. Reinforcement learning has emerged as critical paradigm for post- training generative models to better align with human preferences or task-specific objectives. In language models, InstructGPT [48] employs reinforcement learning from human feedback (RLHF) to enhance harmlessness, helpfulness, and honesty. However, RLHF is susceptible to reward model overoptimization [14]. In contrast, DeepSeek-R1 [56] [6] adopts reinforcement learning with verifiable rewards (RLVR), achieving significant advances in math, code, and logical reasoning domains. For visual generative models, text-to- text-to-image diffusion models have also been via reinforcement learning [2] to optimize measurable metrics (e.g.. compressibility) or human evaluations (7).. Our work identifies world modeling an underexplored yet natural fit for RLVR, where prediction accuracy serves as task-aligned, verifiable reward, enabling direct optimization of generative models across various modalities as world models.
3 Preliminaries
This section provides brief background on visual tokenization for unifying visual and verbal state representations, along with the reinforcement learning algorithm used in our work.
Visual tokenization. Given an image x E HR x x3 the encoder of discrete visual tokenizer [63 maps to its latent representation h ghxwxx This latent is then quantized by performing a nearest neighbors lookup in a codebook of embeddings C = { yielding a discrete representation z E [K]4xu, which is passed through decoder to reconstruct the original image x. The token map can be flattened into ID sequence of length h x w and subsequently modeled by autoregressive models such as decoder-only Transformers [T]. For videos in R x x3, a straightforward
approach is to tokenize each frame independently using an image tokenizer 45
39 However, this
leads to excessively long token sequences. To mitigate this, Wu es al. 169 propose a compressive tokenization method that exploits temporal redundancy in videos by tokenizing each frame into reduced number of n tokens 2 E [K,]". conditioned on shared N context tokens ze E [K2] N
is originally developed for post-training LLMs with reinforcement learning. Compared to PPO (55] GRPO eliminates the need for a value function and estimates advantages in a group-relative manner. Specifically, given a question g, GRPO samples a group of responses {o.) from the behavior policy Pew:: and computes the advantage of each
Group relative policy optimization (GRPO) (56)
response by normalizing its reward R₃ within the group: A = R, R2-mean([RR). (R.))(
std([R,
Similar to PPO, GRPO uses a clipped objective with KL divergence penalty:
TeRPO (0) E BDrD (Pea
loel
2. Pg 3tt
1.t e,
-3DKi. (po [ppeee )}.
1
min
A clip
Pa 3, Pe
1 (
Poas
(1)
where Pa 3.1 denotes po(or. q, (1,e)) for simpilicity. Refer to Shao al. (56) for more details.
3

---

(a) Language world model
GRPD
("hame": "sink", "ison": False)
Token- ization
Sample group
False...
"Lson"s
(Acourary Aecuuray/Flsstcnn Fl-score)
Verifiblle Roward
["name": "cup"
"contaiis": (1)
Tram" "sink"
Language World Model
Detakon ization
[VVtte"")
Chaeee "eap",
Lson True)
containn" "]
turn on the sink
Token- Iration
Tconta Ins [ [.]...
ground-
Extracted predictions
truth
(h) Video world model
GRPO
Sample group
Verifible Reward
Visual Encoder
(MSS/LLIF//SSIM....
Video World Model
Visual
Decader
(x, V. roll, pitch, yaww, gripper opennese)
Quant-
ground
zetion
Decoded predictions
Figure 2: Illustration of BLVR-Worlle framework. World models across various modalities are unified under sequence modeling formulation, and task-specific prediction metrics serve verifiable rewards. (Top) Language-based world models predict verbal state transitions in response to verbal actions. (Bottom) Video-based world models, equipped with visual tokenizer, predict future visual observations conditioned on action vectors.
4 RLVR-World: Training World Models with RLVR
As illustrated in Figure 2 this section introduces RLVR-Wort World, a unified framework for training world models across various modalities by reinforcement learning with verifiable rewards.
4.1 Problem Formulation
Environments simulated by world models are typically formulated as Markov decision process (MDP) M = (S, A. p. r, 7, Depending on the target task, the state space S can flexibly include various modalities, such as verbal, visual, proprioceptive signals. At each timestep. the agent observes a state 8; E S, takes an action a, E A, then transits to a new state according to the distribution p(s(+1 $e, az), and receives immediate reward rt ~ r(st, az). More generally, the states can be partially observable and the transitions can modeled as a k-order Markov process p(s(+1 $t-k+1:6, 0)-++1:)) World models need to learn to approximate the state transition p and the reward function r accurately. Since rewards can be considered as an extended dimension of the state space 110 focus is modeling the transition distribution p(8++1 9t-ki Ot 0t-k+12)-
4.2 World Models as Sequence Modeling
While different architectures have been proposed for world models on top of different modalities, next-token prediction by decoder-only Transformers has emerged as a general-purpose formulation applicable tasks across various modalities. We unify world models into this general sequence mod- cling framework. To transform states and actions into tokens, different modalities have unique, com- monly used tokenization schemes: languages are processed by standard text tokenization techniques like BPE [3]; images and videos are encoded by learned visual tokenizers; and low-dimensional continuous values, e.g., proprioceptive signals,
be quantized to uniform bins over fixed range.
Then, analogous to language models, we use manually designed templates to construct input token sequences g(s, a) as "questions" and output sequences o(s') as "responses". For simplicity and clarity, here we take the first-order Markov case p(s' 8, a) as an example, but the sequence modeling formulation can naturally extend to higher-order cases.
While partial observations are commonly denoted o, in classical RL literature, we keep using $t to avoid confusion with the generated outputs {o. in our RLVR context.

---

We assume an existing world model pre- pre-trained via maximum likelihood estimation (MLE):
|o(x')
(MLE(() log po (o(s') 1 g(s,
= logpo(o((s))
4.3 Prediction Metrics as Verifiable Rewards
We then post-train the world model using RLVR directly optimize verifiable metries for state transition prediction. Specifically, given an input g(ss a). the pre-trained model generates a group of samples {or) from which the predicted next states 8 are extracted using modality-speciffe decoding schemes, such as rule-based extractor for language and visual decoder for videos. The reward is computed by comparing each prediction in the group to the ground-truth next state s':
RS sign(D): s'),
where sign( (D) 1 if lower values of the metric indicate better predictions (e.g., mean squared error or perceptual loss for visual observations), and sign D) = 1 otherwise. Using this task-oriented reward, we can fine-tune the world model according to the RI. objective in Eq.
Remarks, (1) Instead of solving the original environment MDP in Section 4.1 we focus on learning its transition distribution by formulating the next-state prediction process as another MDP, optimized using RLVR. (2) RLVR-World is general framework, where input/output sequences and reward functions can be domain- -aenii-specifically designed. We describe them in the experimental sections. (3)
Our framework is compatible with various RL algorithms [55 75],
not limited to GRPO.
5 Evaluating Language World Models with RLVR
In the following sections, we evaluate the effectiveness our framework for training world models across different modalities, particularly language (Section 5 and video (Section 6), using RLVR. Inspired by the success in domains like math and code generation, we begin by evaluating our framework on world modeling as a new verifiable task for LLMs, focusing on two domains: text games (Section 5.1 and web navigation (Section 5.2)) Experimental details can be found in Appendix A
5.1 Text Game State Prediction
Dataset and task. We use ByteSized32-State- Preei-Prrditiion 65 a dataset of text game state transitions for evaluating how well LLMs can as text-based world simulators. The dataset contains 76,369 transitions from 31 distinct text games, with 2954 high-quality transitions selected for testing. In this task, LLM models the world simulator function F C S x A S natural language contexts describing the task and action semantics, S is the state space encoded as ISON objects, A is the action space, R is the task reward, and T = {0,1) indicates task completion.
T, where C represents
World model. We use DeepSeek-R -iw.--iilll-we---II to their limited capability, first apply supervised fine-tuning (SFT) using responses generated by DeepSeek-R1 [16]. and then fine-tune with RLVR using either a binary accuracy reward I = Il ((8', ff, w) (s, w)), task-specific one reflect the problem structure (see Appendix A.I)
and 7B [16] 72 as our base model. Due
Results. The results are shown in Table I Since predicting state transitions is inherently more difficult when action changes the state, all models perform significantly better unchanged cases. For the 1.5B base model, our RLVR- World with a minimalist binary reward can substantially improve performance over SFT, achieving +34.7% accuracy for unchanged and +8.9% for changed cases. Incorporating human knowledge through a tailored reward yields even larger gains (+44.8% for unchanged, +9.6% for changed). When scaled the 7B base model, our method achieves overall performance surpassing that of GPT-4, although the accuracy on changed cases still falls short due to the limited base model's capacity.
5.2
Web Page State Prediction
Dataset and task. We further evaluate our approach on more realistic web navigation scenarios, using web page state transition dataset collected by WMA [8] from the WebArena benchmark (80).
5

---

Table I: Language world model:
text game state prediction. The test set is divided into unchanged and changed subsets, depending on whether the ground-truth next state differs from the current state.
Accuracy Changed
Model
Unchanged
Overall
Text Game State Prediction
Base (RI Distill Qwen 1.5B)
1 11.98% %
0.08%
7.11%
0.8
SFT
38.88% %
24.21% %
32.87%
R-Woo (Ours, binary)
73.57% %
33.14%
57.01%
Acoute
0.6
Test Acc. (Unchanged)
Triining Rowards
RLVR- ELVR-World (Ours, task-spocific)
83.66 %
33.80%
63.24%
0.5
Test (Changed)
Base (R1- (RI-Distill )
46.90%
5.53%
29.92%
SFT
65.94%
31.32%
51.76%
0.5-
RI.RRWorll (Ours, hinary)
83.08
40.33%
65.53%
0.2
100
200
GPT-4 665
73.90%
51.60%
64.76%
Training Steps
Table 2: Language world model: web page state prediction and model predictive control for web agents. A: relative performance gains from RLVR.
Web Page Stage Prediction
Web Page State Prediction Recall
Web Agent
Model
Precision
F/
Success Rate
0.60
Base (RI-Dist.Qwen.Ja. (wen--Qnen11.B)
15.59%
15.70%
11.83%
n/a
0.55
SFT
48.99%
56.05%
49.94%
12.06%
RLVR-Wordd (Ouns)
72.77%
64.55%
65.11%
14.29%
Training
0.46
+48.5%
+15.1%
+30.3%
+18.4%
0.40
250
500
750
1000
1250
1500
Training Steps
For training and testing, we select 7K-samlle subset consisting of shorter-length samples to avoid out-of-memory issues during training. In this task, a website's state is represented by accessibility tree, which is simplified from its Document Object Model (DOM) tree, and an LLM is used to predict state transitions after user actions such as clicking. Item changes in the accessibility tree caused by actions are extracted using the Hungarian algorithm, enabling the model to predict these changes directly. Unlike WMA, which generates natural language descriptions state changes, this design choice facilitates clear verification during the RLVR stage.
World model. Following the previous setup, we adopt DeepSeek-RI-iisill---eeee ---iissilll---en--.55 as our base model. We first perform supervised fine-tuning (SFT) using chain-of-thought (CoTT) data provided by WMA [8]. Subsequently. we apply RLVR, using the Fl score between predicted item changes As and ground truth As as the reward function: harmonic mean of precision and recall, is computed by treating precision as the proportion of correctly predicted item changes among all generated ones, and recall as the proportion of correct predictions relative to the ground-truth item changes. An item change is considered correct only if it exactly matches the corresponding ground truth.
F1(Aš, As). The Fl score, defined as the
Results. As shown in Table 2 the world model of the Internet can also be enhanced substantially by RLVR leading to the first key finding our experiments:
Finding on language world models: Beyond its success in math and coding, RLVR can also improve LLMs' performance on world modeling tasks involving verbal state transitions.
5.3 Application: Model Predictive Control for Web Agents
Lastly, we show that the reinforced language world models enable more powerful web agents.
we build web agents for the WebArena benchmark 880 composed of three components: a policy model, world model, and a value model. The model predictive control pipeline proceeds follows: the policy model first proposes multiple candidate actions; the world
Setup. Following WMA g
'Due to our novel setup for predicting precise item changes instead of natural language descriptions. direct comparison with established methods like WMA is not feasible. We therefore compare only our base models.

---

Table Video world model: robot manipulation trajectory prediction on RT-1. We report the mean and standard deviation for each metric calculated over three sampling runs, MSE, LPIPS, and SSIM scores are scaled by 100 for better readability. A: relative performance gains from RLVR.
Task
Model
Repetition Rate
MSE +
PSNR1
SSIMI
LPIPS!
Base
n/a 0.336
25.3
81.7#0.00
13.0
Single-step Prediction
RLVR-World (Ours
n/a 0.287 =0.00
25.9:0m
83.1
12.2:001
A
n/a +14.3%
+2.6%
+1.6%
+6.0%
Base
48.6%
0.659 a0.06
23.1+00l
80.9
14.8
Base (w/ repetition rejection)
0.0%
a.593
23,3.001
81.0se.0
14.42001
Multi-step Prediction
RLVR-World (Ours)
9.9%
0,486
24.1.000
82.4
13,4
+79.6%
+26,1 %
+4.5%
+1.9%
49.2%
RLVR- R-Woldd (w/ rep. penalty reward)
0.0%
0.506
24.0a0.0
82.2.o..
13.7
Single-Ster Step Prediction
Multi-Step Prediction
145
0.17
Pre-train (MLE)
Post-train (RLVR)
Pre-train
Post-train
(MLE)
(RLVR)
0.16
LPIPS
LPIPS
Pre-train [extended 600K)
0.14
125
20
40
60
80
30
60
90 120150
10
20
30
40
0
30
80
90
120 150
Training Steps (x10*)
Training Steps (x1)
Training Steps (x109)
Training Steps (x1)
Figure 3: Learning curves of video world models RT-1. Note the significant difference in the
x-axis scale between the pre-training and post-training stages.
model then predicts the outcomes of these actions; finally, the value model scores each predicted outcome based on the task goal. The action with the highest score is selected for execution. For both
the policy and value models, we use DeepSeek-V3 (38), while the world model is taken from trained models in the previous section. Additional details are provided in Appendix A.3]
In Table 2 we compare web agents on top of SFT- and RLVR-trained world models and observe significant improvements. We expect that further gains can be achieved by incorporating stronger policy and value models and extending maximum context length during world model training.
Results.
6
Evaluating Video World Models with RLVR
We then take a pioneering step in evaluating RLVR to train autoregressive video world models, offering analyses and insights into broader generative models beyond the scope of reasoning models.
6.1
Sctup
Dataset and task. We primarily use the RT-1 robotic manipulation dataset [5] for experiments, which contains 87,212 tabletop teleoperation trajectories collected from a Google Robot, with 1 1% left for testing. Each frame of visual observation has resolution of 256 320, and the action space consists of 13 dimensions, including arm and base movement. To compare with state-of- the-art models, we also include tabletop pushing (PushT) (9] and deformable object manipulation (Rope and Granular) [77) datasets from DINO-WM WM (79.. We evaluate world models on two task settings: (1) Single-step prediction, formulated as p( p(sr+1 St-T+1:2, d;-T+1a): predicting the next observation given the past T'-sep observations and actions; (2) Multi-step prediction, formulated
predicting next T-step observation conditioed on the current observation and future action sequence. Predictions are assessed against ground-truth observations using frame-level metrics: MSE, PSNR (27). SSIM (66), and LPIPS [78).
as p( 51+ 11477 St, 0r,+++7111 = [ p(s: 8t:i-1; ; 1)::
(+2'
7

---

Table Video world model: comparisons with state-of-the--rt on PushT, Rope, and Granular. We follow the single-step prediction setting and include the baseline results reported in DINO- ANN 79.. In addition, we report our reproduced evaluation results using the public DINO-WM checkpoint
on PushTT LPIPS and SSIM scores are scaled by 100.
PushT
Rope
Granular
Model
LPIPS
SSIM
LPIPS
SSIMT
LPIPS,
SSIM
Recurrent latent space models
R3M 46
79
4.5
95.6
2.3
98.2
8.0
91.7
ResNet 22
79
6.3
95.0
2.5
98.0
8.0
91.5
DINO CLS
+
3.9
97.3
2.9
98.0
8.6
91.2
DINO WM (Reported)
79
98.5
0.9
98.5
94.0
DINO- WM (Public checkpoint)
79
3.39
96.38
Diffusion models
AVDC
4.6
95.9
6.0
97.9
10.6
90.9
Autoregressive models
Base (Ours)
0.83
98.28
3.03
97.86
3.14
94.79
RLVR-Wool R-Wll (Ours)
0.70
98.46
2.08
98.14
2.42
95.42
World model. Since no off- aff-the-shelf general-purpee purpose video world models are available, we pre- train variants of VViieoGPT [69] target datasets base models by ourselves. For each trajectory segment, observations and actions are tokenized and concatenated into a unified token sequence. We train image tokenizer to independently tokenize video frames for single-step prediction, but train a compressive tokenizer from iVideoGPT to mitigate sequence length explosion for multi-step prediction. Each action dimension discretized into 256 uniform bins, determined based its value range across the entire dataset. During RLVR fine-tuning, we define the reward function the sum of LI and perceptual loss between decoded predicted and ground-truth frames:
E'+r [I, (8,, 87) + LPIPS($,, 67)]. commonly used in visual tokenizer training [1]. See implementation details in Appendix A.4
6.2 Main Results
As shown in Table ELVR-Worle significantly improves the base model across all visual metries on RT-1, demonstrating more accurate and perceptually better video predictions, showcased in Figure 6 We highlight that these gains are achieved with only hundreds of RLVR gradient steps, compared to hundreds of thousands required for MLE pre-training (see the training curves in Figure Even after continuing pre-training for multi-step prediction with 150k additional steps-nearly 1000 more than
RLVR fine-tuning the resulting LPIPS score remains at 14.5, still substantially lagging behind.
Comparison with state-of-the-art. We then compare our models against advanced world models, including all recurrent latent state models introduced by DINO-WI MINO-WM [79], which use different latent spaces from pre-trained encoders such as R3M, ResNet, and DINOv2, and an action-condittoned diffusion model, AVDC [31]. As shown in Table 4 after RLVR, model achieves overall perfor- mance comparable to the strongest baseline, DINO-WM, and surpasses by a significant margin on the most challenging particle-based dataset, Granular. Additional model predictive control results on P are provided Appendix A.4.2.
6.3 Model Analysis
Mitigating repetition. Prior studies (35 have identified the likelihood objective as primary cause of repetition in LLM generation. We observe similar phenomenon in multi-step video prediction, as showcased in Figure 6 This likely stems from the fact that approximately 20% of tokens in each frame remain unchanged from the previous frame, encouraging the model to exploit this shortcut. By directly optimizing video-level prediction metrics rather than next-token likelihood, RLVR effectively mitigates this issue, reducing the repetition rate from 48.6% to 9.9%. To ensure our improvements are not merely due to reducing repetition, in Table 3 we include additional baseline that repeatedly queries the base model for predictions until a non-repetitive output is sampled.

---

(b)
0 9:130
Base Mode RovR
0.128
G=12
G= 16
Test LPIPS
G=8
5001
08000
50000
LPIPS (Best
G=4
0.125
26.5
Test
0.124
28
3012
6.07
20.5
0.120
3107
6:33
5:151
100
100
200
150
Number of Samples
Training Steps
Training Metric
Figure 4: Model analysis on RT-1. (a) Test-time scaling: best performance among different numbers of generated samples. (b) RL training scaling: learning curves using different group sizes GRPO. (c) Metric-oriented optimization: W-r-d trained and tested on different visual metrics.
Metric-oriented optimization. To further demonstrate the effect of direct metric optimization, we post-train five variants of the base model using five different metries as reward functions: MAE, MSE, PSNR. SSIM, and LPIPS. As shown in Figure 4 models fine-tuned with specific metric generally achieve the best performance when evaluated on that same metric. Additionally, show the effectiveness of our method for non-differentiable rewards and achieve zero repetition rate, we introduce an additional repetition penalty reward, defined as the negative rate of consecutive identical frames. In Table 3 after training the model with this extended reward, we observe that it maintains comparable prediction performance while effectively eliminating repetition artifacts.
Test-time scaling, We evaluate the test-time scaling behavior of our base and RLVR-trained models by reporting the best metric achieved across N samples 42 in Figure RLVR-World improves one- shot performance, even outperforming the base model's best-of-5 results. This is particularly valuable in practical scenarios where generating large numbers of samples is computationally expensive and ground-trutt comparisons are unavailable. However, as N increases to 100, the base model catches up and eventually surpasses aLVaetrained models, echoing findings from Yue al. [76]. This suggests limitations of current RLVR methods and ample opportunities for future research.
RL training scaling, While generating more samples at test time is expensive, Figure 4 shows that is essential for training. Specifically, increasing the group size in GRPO improves both convergence speed and final performance by enhancing sample diversity and expanding the exploration space.
Finding on video world models: RLVR bridges the gap between pre- pre-training objectives and visual prediction metrics, leading to more accurate predictions, improved training efficiency, and reduced artifacts such as repetition.
6.4 Application: Real2Sim Policy Evaluation
We finally show that our models can serve as real-world simulators for im- proved policy evaluation.
Open Drawer
Close Drawer
SIMPLERIA
BINPLER-VR
0.8
Dase Model
Dace Model
RLVR-Word
RLVR-Word
Setup. Following SIMPLER 377]
0.6
we evaluate four policy checkpoints from RT-1 [5] and RI-1-X 449 on
Evalueted
six tasks involving opening and clos- ing top, middle, and bottom drawers. Starting from real- wolld observation frame, policies can interact with video world models to roll out neural simu-
0.0
0.2
0.6
1.0
0.2
0.6
1.0
Real Success Rate
Real Success Rate
Figure 5: Real2Sim policy evaluation.
lated trajectories, allowing for policy evaluation without real-world deployment. Since our preliminary attempts on VL.M-based automatic evaluation (61] fail to provide reliable judgments, we rely on human annotators to assess the success of simulated trajectories. Besides our base model, we compare against the simulators developed in SIMPLER. Experimental details can be found in Appendix A.5
Results. As shown in Figure 5 compared to handcrafted SIMPLER simulators, video world models yield smaller discrepancies between real and simulated success rates, suggesting world models

---

Video World Modeling
t=1
t=2
t=3
t 4
t=
t=6
t=7
Ground-truth
(input)
RLVR-World
Base model
Ground-truth
(input)
RLVR-World (No repetitio)
Base model
(Repettion)
Policy Eval
t=
1= 10
t= 20
1= 30
40
t= 50
60
Open Drawer Success
(RT-1 Converged)
Failure (RT-1 Begin)
Figure 6: Qualitative analysis: multi-step video prediction and policy evaluation on RT-1.
scalable approach to bridging the sim-to-real gap. Among the video world models, RLVR-World further improves upon the base model, achieving more accurate policy evaluation.
Finding on model- based applications: RLVR-trained world models can improve downstream tasks, including policy evaluation and model predictive control.
7 Discussion and Limitations
In this work, we pioneer RLVR for training world models across language and video modalities, with practical applications in web navigation and robotic manipulation. We believe RLVR has the potential to become general post-training paradigm for generative models. To this end, several challenges remain for future exploration. From the algorithmic perspective: (1) Breaking performance barriers: Although RLVR yields significant gains, training typically converges within hundreds of steps. Unlocking continual improvements calls for deeper analysis of bottlenecks models, data, and algorithms: (2) Quu-of-distribuuinn (OOD) generalization: Inspired by RLVR's 's success in enabling LLMs to generalize beyond training domains [57], it important to study whether similar benefits extend world models, particularly for counterfactual reasoning OOD actions in sequential decision- makiin.. For developing foundation world models: (1) Post bost-training general-purpose purpose world models: Our current video world models adopt two-stage training process the same dataset. We believe that the full potential of RLVR will be further unlocked once the community develops general- pprepoee video world models [ 36) This would enable the complete paradigm of supervised pre- tre-training on diverse domains - supervised fine- tne-tunnng- - reinforcement tine-tuning, which has already proven effective in our language world model experiments. (2) Broader application across model classes: While the core insight behind RLVR-Wond is model-agnostic, GRPO algorithms for diffusion models [71 40) concurrently developed and thus fall outside the scope of our current study. (3) Vask-aligned rewards: While classical visual metrics align better with the world modeling task than MLE, they still fail fully capture user-intended qualities. Incorporating constraints such as physical rules and temporal consistency will require more sophisticated reward designs.
10

---

Acknowledgements
We would like to thank many colleagues, in particular Chaoyi Deng and Haixu Wu, for their valuable discussion. This work was supported by the National Natural Science Foundation of China (U2342217 and (2021002), the BNRist Innovation Fund (BNR2024RC01010), and the National Engineering Research Center for Big Data Software.
References
[1] Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arX1::2501.0357, 2025.
[2] Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. In NeurlPS, 2024.
[3] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In ICLR, 2024.
[4] Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On opportunities and risks of foundation models. arXiv preprint arXiv:210S.072555 2021.
[5] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, al. RT-1: Robotics transformer for real-world control scale. In RSS, 2023.
[6] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
[7] Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder. Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps. et al. Genie: Generative interactive environments. In ICML, 2024.
[8] Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sungh- wan Kim, Dongha Lee, and Jinyoung Yeo. Web agents with world models: Learning and leveraging environment dynamics in web navigation. In ICLR, 2025.
[9] Cheng Chi, Zhenjia Xu, Siyuan Feng. Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song, Diffusion policy: Visuomotor policy learning via action diffusion. 1RRR, 2023.
[10] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine Deep reinforcement learning in handful of trials using probabilistic dynamies models. In NeurlPS, 2018.
[11] Patrick Esser. Robin Rombach, and Bjorn Ommer. Taming transformers for high resouutooo image synthesis. In CVPR, 2021.
[12] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mo- hammad Ghavamzadeh, Kangwook Lec, and Kimin Loc. Dpok: Reinforcement learning for fine-tuning next-to-image diffusion models. In NewrlPS, 2023.
[13] Philip Gage. A new algorithm for data compression. The C Users Journal, 12(2):23-38, 1994.
[14] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In ICML, 2023.
[15] Yu Gu, Kai Zhang, Yuting Ning. Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava. Yanan Xie, Peng Qi al. your Ilm secretly world model of the internet? model-based planning for web agents. TMLR, 2025.
[16] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseck-rl incentivizes reasoning in ums through reinforcement learning. Nature, (:58808))6633-44 638, 2025.
[17] David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In NewrIPS, 2018.
11

---

[18] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2020.
[19] Danijar Hafner, Timothy Lillicrap, Ian Fischer. Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In ICMI, 2019.
[20] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, pages 1-7, 2025.
[21] Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. In ICLR, 2024.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[23] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NewrlPS, 2020.
[24] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degencration. arXiv preprint arXiv:/944.0975/ 2019.
[25] Anthony Hu, Lloyd Russell, Hudson Yeo, 7ak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint 1XXi:23399170080 2023.
[26] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation large language models. In ICLR, 2022.
[27] Quan Huynh-Thu Thu and Mohammed Ghanbari. Scope of validity of psnr in image/video quality assessment. Electronics letters, (00800000- 2008
[28] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model-based rein- forcement learning for atari. In ICLR, 2020.
[29] Adam Tauman Kalai and Santosh S Vempala. Calibrated language models must hallucinate. In STOC, 2024.
[30] Anssi Kanervisto, Dave Bignell, Linda Yilin Wen, Martin Grayson, Raluca Georgescu, Sergio Valcar- cel Macua, Shan Zheng Tan, Tabish Rashid, Tim Pearce, Yuhan Cao, al. World and human action models towards gameplay ideation. Nature, (38(8051):656 663, 2025.
[31] Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B Tenenbaum. Learning act from actionless vidcos through dense correspondences. In ICLR, 2024.
[32] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. In JCML, 2024.
[33] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Facze Brahman, Lester James Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. Tulu3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124 2024.
[34] Yann LeCun. A path towards autonomous machine intelligence. preprint posted on openreview, 2022.
[35] Huayang Li. Tian Lan, Zihao Fu, Deng Cai. Lemao Liu, Nigel Collier, Taro Watanabe, and Yixuan Su. Repetition in repetition out: Towards understanding neural text degeneration from the data perspective. In NeurlPS, 2023.
[36] Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified video action model. In RSS, 2025.
[37] Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sich, Scan Kirmani, et al. Evaluating real-world robot manipulation policies in simulation. In CoRL, 2024.
[38] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 194477 2024.
[39] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on Dallion--enngh video and language with blockwise ringaltention. In ICLR, 2025.
12

---

[40] Jie Liu, Gongye Liu, Jiajun Liang. Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan. Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.05470. 2025.
[41] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2019.
[42] Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang. Yandong Tommi Jaakkola, Xuhui Jia, et al. Scaling inference time compute for diffusion models. CVPR, 2025.
[43] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square emror. In ICLR, 2016.
[44] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. In ICLR, 2024.
[45] Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample efficient world models. In ICLR, 2023.
[46] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsca Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. In CoRL, 2022.
[47] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2024.
[48] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
[49] Abhishek Padalkar, Acom Poolcy, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, al. Open x-embodiment: Robotic learning datasets and rt-x models. In JCRA, 2024.
[50] Alec Radford, Jeff Wu, Rewon Child. David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[51] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-rcsolution image synthesis with latent diffusion models. In CVPR, 2022.
[52] Lloyd Russell, Anthony Hu, Lorenzo Bertoni, George Fedoseev, Jamie Shotton, Elahe Arani, and Gianluca Corrado. Gaia-2: A controllable multi-view generative world model for autonomous driving. arXiv preprint arXiv:2503. 20523, 2025.
[53] Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-69) 2020.
[54] Ingmar Schubert, Jingwei Zhang. Jake Bruce Sarah Bechtle, Emilio Parisotto, Martin Riedmiller, Jost To- bias Springenberg, Arunkumar Byravan, Leonard Hasenclever, and Nicolas Heess. A generalist dynamics model for control. arXiv preprint 230.:2331..11112 2023.
[55] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1700:06477 06347, 2017.
[56] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint Sit2:22402.030000 2024.
[57] Maohao Shen, Guangtao Zeng, Zhenting Qi Zhang Wei Hong, Zhenfang Chen, Wei Lu, Gregory Wornell, Subhro Das, David Cox, and Chuang Gan. Satori: Reinforcement learning with saaii--f--ctioo-thougge enhances Ilm reasoning via autoregressive search. In ICMI. 2025.
[58] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. In EuroSys, 2025.
[59] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127066 2024.
13

---

[60] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction. MIT press Cambridge. 1998.
[61] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang. et al. Gemini 1.5: Unlocking multimodal understanding across millions tokens of context. arXiv preprint urXiv:2403.05530,2 2024.
[62] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[63] Aaron Van Den Oord. Oriol Vinyals, et al. Neural discrete representation learning, In NewrlPS, 2017.
[64] Chaojun Wang and Rico Sennrich. On exposure bias, hallucination and domain shift in neural machine translation. In ACL, 2020.
[65] Ruoyao Wang, Graham Todd, Ziang Xiao, Xingdi Yuan, Marc-Alexandre Côté, Peter Clark, and Peter Jansen. Can language models serve as text-based world simulators? In ACL, 2024.
[66] Zhou Wang, Alan C. Bovik, Hamid R Sheikh, and Fero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4)x600-61 612, 2004.
[67] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le. Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
[68] Jialong Wu, Haoyu Ma. Chaoyi Deng, and Mingsheng Long. Pre-training contextualized world models with in-the-wild videos for reinforcement learning. In NeurlPS, 2023.
[69] Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. ivideogpt: Interactive videogpts are scalable world models. In NeurlPS, 2024.
[70] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for texl-to-image generation. In NeurlPS, 2023.
[71] Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.078/ 2025.
[72] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint 2rXiv:2412.15/11 2024.
[73] Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbecl. Learning interactive real-world simulators. In ICLR, 2024.
[74] Shaofeng Yin, Jialong Wu, Siqiao Huang, Xingjian Su, Xu He, Jianye Hao, and Mingsheng Long. Trajectory world models for heterogeneous environments. 2025.
[75] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale, arXiv preprint 250:: 14476, 2025.
[76] Yang Yuc, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in lims beyond the base model? arXiv preprint arXiv:2504. 13837, 2025.
[77] Kaifeng Zhang, Baoyu Li, Kris Hauser, and Yunzhu Li. Adaptigraph: Material-adaptive graph- neural dynamics for robotic manipulation. In RSS, 2024.
[78] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018
[79] Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable shoo planning. In ICML, 2025.
[80] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyuc Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In ICLR, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our experiments in Section 5 and 6 well support our claims and contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed limitation Section 7 and analyzed failure cases in Appendix A.5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
We do not contribute theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have described clearly and fully the experiment details the main text and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: Our code, newly collected data, and trained models are available at https //thuml.gite github. 1o/RLVR-World
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
public/guides/Codes httpeee//oodsuuuiiisiioooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssicnlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Table we report the mean and standard deviation for each metric calculated over three sampling runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp/ss///pccec..cc//pp//////t
Answer: [Yes]
Justification: We have carefully read the code of ethics and believe that our work poses no harm to society.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: In work, the models trained in specific domains have relatively small number of parameters and limited capability to generate content that is indistinguishable from realistic data. Therefore, believe the risk of misuse is low.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
we explicitly mentioned and respected the license. The only exceptions are the datasets used for text game and web page state prediction, whose authors do not specify license information.
Justification: We have cited all assets used in the paper. In Appendix A
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code, newly collected data, and trained models available at https / //thuml. github. io/RLio-RRRi--- World They are documented with detailed instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The only experiment involving human subjects is annotated by the authors themselves, without the use of crowdsourcing, Details of the annotation process are provided in Appendix A.5
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
This work does not involve crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The usage LLMs for both data annotation and as post-training base models is clearly described in the main text and detailed in Appendix A
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

A Implementation Details and Extended Experiment Results
A.1 Text Game State Prediction
Dataset detail. Sreeizzed22-Stat--PPeiichi
665 comprises 76,369 transitions collected from 31
distinct text-based games. A curated subset of 2,954 high-quality transitions forms the official test. set, which we use for both our SFT and RLVR experiments.
The dataset includes task rules written either by human experts or large language models (LLMs). To minimize potential inaccuracies, we adopt the expert-written rules our experiments.
Each state is represented as a JSON object, describing the environment objects and their associated properties (e.g., dish with a "clean" "dirty" status). Actions are described in natural language (e.g., "clean the dish"), and the world simulator aims to predict the resulting state, the task reward based on predefined rules (e.g.. +1 for each clean dish), and task completion (e.g., when all dishes are clean). We categorize samples into "unchanged cases" when the action leaves the state unchanged, i.e. the ground truth next state exactly the same the current state, and "changed cases" otherwise.
To reduce output complexity, as suggested by the original paper [65]. we train our world models for state-difference prediction, generating updates solely for objects whose properties have changed. This design mitigates challenges in generating long, correctly formatted outputs with our 1.5B small base model. Prompt examples are provided in Appendix
Supervised fine-tuning. To overcome the poor perfor- mance of base model, DeepSeek-R1-Disdee Qwen- 1 especially its 0.08% accuracy on changed cases (see Table 1. first apply supervised fine-tuning (SFT) with LoRA [26] to enable effective RL training.
0.4
Accuracy (%)
0.3
The ByreSSized32-Sttee-Peeeee Predicton dataset lacks suitable SFT data, as contains only final answers without in- termediate of-thought reasoning annotations. To remedy this, we prompt DeepSeck-R. on a subset of changed cases, apply reject sampling to collect the correct responses, and build a training dataset of 4,237 samples within $40 budget. We fine-tune the model and report test accuracy across epochs Figure 7 The checkpoint from epoch 10 is selected the final SFT result.
0.1
Overall
Unchanged
Changed
0.0
2.5
5.0
7.5
10.0
12.5
Epoch
Figure 7: The learning curves SFT for text game state prediction.
The reward is defined based on the difference between the model's prediction and the ground truth. We explore two reward schemes:
RLVR.
Binary reward assigns a reward of only if the prediction is completely correct; otherwise, the reward is O.
Task-specific reward considers three components: (1) accuracy over all properties accall+ (2) accuracy over properties that supposed change aCCchunged- and (3) the binary reward. The final reward is computed as
R = ai accall + a22 BCCChanged + ag I(correct)
our experiments, we use weights 01 = 0.1, 02 = 1,3g = 0.2, which are chosen based heuristic intuition without further tuning.
To address data imbalance where over 85% training samples are unchanged cases-we we downsample unchanged cases while retaining all changed cases. For binary reward experiments, set the changed- to-unchanged ratio at 100 40, and for task-specific reward experiments, we use 100 5, since the task-specific reward more effectively teaches the model output "no change" when appropriate. Empirically, increasing unchanged samples improves performance on unchanged cases but degrades changed case accuracy, aligning with expectations. Though this ratio tuned casually, current setup sufficiently demonstrates RLVR's 's effectiveness in improving language-basee world models. Hyperparameters for training are provided in Table
DeepSeek- RI-Distill- --istill-Qwen- 1.5B follows the MIT License.
22

---

Table 5: Hyperparameters for language world model training.
Hyperparameter
Text Game
Web Page
Batch size
16
8
LoRA rank 26
32
LoRA
16
16
SFT
Epoch
10
Learning rate
1 x 10-5
1 x 10-5
Weight decay
0.01
0.01
Max response length
4096
7288
Batch size
128
64
PPO (GRPO) mini batch size
64
16
RLVR
KL loss coefficient
1 x 10
10-3
Group size
5
5
Learning rate
1 x 10-6
1 x 10
1.0 for prediction
Top-p
1.0
Sampling
0.95 for MPC
Temperature
1.0
1.0
A.2
Web Page State Prediction
Dataset detail. We use the dataset provided by the WMA repository" As the dataset does not explicitly annotate precise changes the accessibility tree in response user actions-i.e., which items are added, removed, or updated-we extract this information using the official script released by the authors An example of an item [1220] textbox *ue60Oc' focused: True required: False, which includes item ID, type, content, and additional attributes. We can then formulate the next-state prediction task as prompting the LLM generate all changed items within the accessibility tree. To avoid out-of-memory (OOM) issues during supervised fine-tuning (SFT), we discard samples in which the total length of the prompt and the ground- truth response exceeds 5,000 tokens. After this filtering process, approximately 7,000 trajectories remain from the original 14,000. We allocate 99% of this subset for training and reserve the remaining 1% for testing.
g
Supervised fine-tuning. For our new task formulation, we construct target responses for each sample in the WMA dataset by concatenating the Shain-of-thoughh (CoT) annotations, which are generated by GPT-4o-mini and released by the authors, and our extracted next-state changes. We then perform supervised fine-tuning (SFT) with LoRA (26) on our base model, DeepSeek-R1-Disse Qwen-1.5B. Training hyperparameters are summarized Table 5
We define the reward function using the F1 score, based on the predicted and ground- truth changed items. Let As {é1, ....... é Cm represent the set of predicted changed items, and As (c1, 02. .,, ] the set ground-truth changed items. We define:
RLVR.
True Positives (TP) | 112 A CE As)
(5)
(6)
A >
A
Precision (P)
1.
if
A
0 and
As 0
0,
if
A
0 and
As > 0
dataset
ammotation_for_tae under the MIT License
23

---

TP
if
A A8
> 0
As/
Recall (R)
1,
0 and
A 0
0,
if
A8
0 and
As = 0
2.P.R P+R (0
iPP+R>0
F1 Score (Reward R)
otherwise
item change is considered correct (i.e., true positive) only if exactly matches ground-truth item change across all fields. For example, in the item change [1273] StaticText 'Vortex Running Shoes', the model must produce the exact same string-including all characters and formatting-for the prediction to be considered correct. In cases where change occurs, the world model is trained to output special item None, which is treated as a valid item and included the F1 computation.
RLVR hyperparameters are also listed in T We select the checkpoint that achieves the highest reward on the test set for final evaluation in the subsequent model predictive control experiments.
A.3 Model Predictive Control for Web Agents
To ensure consistency with prior work, all experiments are conducted using the official WebArena environment, deployed on an Amazon Web Services (AWS) EC2 instance pre- configured with Docker. WebArena covers five task domains: Shopping, Content Management Systems (CMS), Reddit, GitLab, and Mapping.
Environments.
Model predictive control. Following the method introduced in WMA 8 we use a policy model sample 20 candidate actions and select the three most frequently sampled for further evaluation. For each selected action, our trained world model performs next-state prediction. A summarization model is then used to (1) identify the top 10 most salient state changes in the predicted next state, and (2) generate natural language summary describing those changes. This transition summary, along with the task objective, is provided
value model, which assigns score between I and 5. This scoring query is repeated 20 times, and the final score for each action is computed as the average of these 20 responses. The action with the highest average score selected for execution. Except for the world model, all models, including policy, summarization, and value, are implemented by prompting DeepSeek- V3 with top-p sampling (p 0.95). All prompts used in model predictive control are detailed in Appendix
Domain-specific results.
We report web agent performance on different domains in Table 6
Table Nomain-speciffe model predictive control performance for web agents. A: relative perfor- mance gains from RLVR.
Methods / Domains
Shopping
CMS
Reddit
Gitlab
Map
Overall
SFT
18.23%
11.54%
4.39%
6.63%
19.53%
12.07%
RLVR-World (Ours)
21.88%
10.99%
6.14%
10.71%
20.31%
14.29%
+20.0%
-4.8%
+40.0%
+61.5%
+4.0%
+18.4%
A.4 Robot Manipulation Trajectory Prediction
A.4.1 RT-1
Hyperparameters of architectures and training process for robot manipulation trajectory prediction are listed in Table and
24

---

Table Hyperparameters for visual tokenizers robot manipulation trajectory prediction. Unless specified, the compressive tokenizer shares the same hyperparameter values with the per-frame
tokenizer.
Hyperparameter
Value
Input resolution
256 x 320
Layers per block
Channels
[128, 256, 256, 512, 768] 7 x 5 x x x 4375
FSQ levels
Token number N
16 x 20 320
Training steps
5 x 10
Batch size
16
Per-frame Tokenizer
Segment length Optimizer
8
Adam
41
Learning rate
5 10
L1 loss weight
1.0
Perceptual loss weight
1.0
Adversarial loss weight
0.1
Discriminator layers
6
Discriminator training start step
10000
Channels
[128, 256, 256, 7 x
FSQ levels K1
4375
Token number
8 10 = 80
Context FSQ levels K₂
7 5 x 5 4375
Compressive
Context token number N
32 x 40 = 1280
Tokenizer (69
Maximum cross-attention resolution
Training steps Batch size
6 105
16
Segment length
32
Number of sampled frames
7
Dataset. We use the RT-1 dataset released from Open X-Embodiment 149 which follows the Apache license and contains 87,212 trajectories with 256 x 320 visual observations and 13- dimensional actions. Following Wu et al. [69], 99% of trajectories are used the training set and 1% are left as the test set. During testing, a fixed segment from each trajectory is used for prediction.
Visual tokenizer. For single-step and multi-step prediction settings, train per-frame tokenizer and a compressive tokenizer [69] respectively on RT-1 from scratch. The per-frame tokenizer is essentially VQGAN [11] with convolutional encoder-decoder architecture, which tokenizes each frame 8; in the trajectory independently into tokens 2 E [K] where K is the codebook size. The compressive tokenizer is implemented as conditional VQGAN with dual encoder-decoder pairs {(E,, Do), (Ep }. The context encoder E, first tokenizes a context frame s. independently into context tokens z, E [22]%. Subsequently, each frame is tokenized by Ep, conditioned on the context encoder's feature maps through cross attention, into tokens z, E [Ki]a, Since the context features can capture rich shared information across the trajectory, the number of tokens per frame can be significantly reduced compared independent tokenization (n < N). We refer Wu et al. (69 for further details on the compressive tokenizer.
Architectural details of these tokenizers are provided in Table
Notably, we adopt finite scalar quantization (FSQ) 44 instead vector quantization (VQ) 65 within our VQGANs, due to its superior codebook utilization.
For per-frame tokenizer training, we sample batches of trajectory segments and independently reconstruct each frame to compute VQGAN losses. For compressive tokenizer training, the first frames of trajectory segments are used as context frames, and from the remaining frames, we randomly sample subset to reconstruct for training reduce memory requirements.
25

---

Sequence modeling formulation. We consider two task settings for video prediction and describe how token sequences are constructed for autoregressive Transformers:
Single-step prediction p(5++1 81-34, ag ax-3:): : By denoting the tokenized representation of 8t and ae as žt and be respectively, the token sequence is constructed:
I = concat(21- 33 2t- b2-2,
bt, [bos], 2t-1, [eos]).
where [bos] and [eos] are two special token Each dimension of actions is discretized into 256 uniform bins, with all dimensions sharing the same 256 codes. Visual and action codes are offset to avoid overlapping, resulting in total codebook size of 4633. The final sequence length is 4 x (320- + 13) + 1 + 320 - 1 = 1654, where the underlined part of length 321 corresponds to output tokens used for computing the cross-entropy loss.
Multi-step prediction p($++1:0+7 St, 02.46: Sc) Similarily, the token sequences is constructed as follows
1+7
1= p(s; 9ta-1;00t-1,:0
x = concat( Ze, zt, bt, Zt+1, b,+1, Zt+2, be42, Fe47, bet7),
(8)
where 2, represents the tokenized context frame. Codes for context tokens, per-frame tokens, and actions are offset to avoid index overlapping, resulting in total codebook size of 9006. The total sequence length is 1280 + (80 + 13) = 2024. Only tokens of frames that need to be predicted contribute to the loss. During generation, predicted frame tokens and action tokens are appended to the sequence alternately.
Autoregressive transformer. We adopt the standard architecture of LLaMA 62] instantiated smaller models with 138M parameters, matching GPT-2 small (50). Separate Transformers are pre-trained from scratch for single-step and multi-step prediction, respectively. During multi-step prediction training, context frames sampled from earlier frames preceding the trajectory segment to prevent information leakage for prediction. Architecture and training are detailed in Table S
The pre-trained transformers are fine-tuned with GRPO. The reward function is defined as a negative loss function of all predicted frames and ground truth. Specifically, for single-step prediction:
RLVR.
R(S 8t41) = - (8 LPIPS((41
(9)
and for multi-step prediction:
1+7
(44444447 =
Σ rmt+1
LPIPS(&,,
(10)
We do not update visual tokenizers during fine-tuning.
Model analysis. All experiments presented in Figure are conducted in the single-step prediction setting using a group size of 16, unless specified. Although we find that using group size of 32 yields slightly better performance, we report performance with group size of 16 in our main results (Table to maintain consistency with the majority of our experiments.
Training curves. We present the curves of training rewards during single-step prediction RLVR training Figure
Qualitative analysis. Additional showcases of video prediction and policy evaluation are presented in Figure
'We note that these two tokens necessary in the sequence formulation but included due to historical reasons.
"b.i7 is used here as a placeholder for implementation convenience. Due to the causal architecture and the absence of loss terms bt+7, it completely has effect the training process.
26

---

Table 8: Hyperparameters for transformers in robot manipulation trajectory prediction. Unless other- wise specified, transformers for single-step and multi-step prediction share the same hyperparameter values.
Hyperparameter
Value
Layers
12
Hidden size
768
FFN intermediate size
3072
Architecture
Number of attention heads
12
RoPE
[59]
10000.0
Vocabulary size
4633 for single-step prediction
9008 for multi-step prediction
Training steps
9.9 x 10 for single-step prediction
4.5 x 10 for multi-step prediction
Optimizer Batch size
Adam W 41
Pre-training
32
Segment length
5 for single-step prediction
8 for multi-step prediction
Learning rate
5 x 10-5
Sequence length
1654 for single-step prediction
2024 for multi-step prediction
Optimizer
AdamW [1
Batch size
128
RLVR
PPO (GRPO) mini batch size
32
KL loss coefficient
1 10
Group size
16
Learning rate
5 x 10-5
Weight decay
0.01
Top-k Temperature
100
Sampling
1.0
-0 165
0.031
Raw
Raw Smooth
Smooth
0.1900
-0.150
Loos
0.030
0.1278
0.029
-0.55
0.088
0 1295
19
-0.10-
Raw Smooth
0.027
0.1200
0.117
-0.66
0.028
100
300
100
Training Sleps
Training Steps
Training Steps
Figure 8: Training curves of RLVR-World for single-step prediction: rewards, L losses, and perceptual losses.
27

---

Video World Modeling
t=1
[=
t=3
1=4
t=
1=6
t=7
Ground-truth
(input)
RLVR-World
Base model
Ground-truth
(input)
RLVR-WWord repetition)
Base model
(Rspetition)
Policy Eval
8
12
= 16
20
Close Drawer Success (RT-1 Converged)
1
1-
10
20
30
40
1 50
60
1
Fallure (RT-1 Begin)
Figure 9: Additional qualitative analysis: multi-step video prediction and policy evaluation on RT-1.
A.4.2 PushT, Rope and Granular
Datasets. We the datasets provided by DINO-WM [79]. PushT is environment introduced by Chi et al. (9] that features a pusher agent interacting with a T-shaped block. The collected dataset contains 18,685 training trajectories and 21 test trajectories. Each step has frame resolution of 224 x 224 and includes proprioceptive information. The task of the dataset is to predict the next step based on the previous three steps. Rope and granular manipulation tasks introduced by Zhang al. (77), simulated with Nvidia Flex. Each environment collects 900 training trajectories and 100 testing trajectories, in the resolution of 224 x 224. The task of these datasets is predict the next step based on the previous one step.
Implementation details. Unless otherwise specified, we adopt the same hyperparameters as in Appendix 1.4.1 for RT-1. For PushT, proprioceptive information is quantized into uniform bins, consistent with the treatment of action values, and its mean squared error is included in the RLVR reward. For Rope and Granular, given the limited amount of training data, employ smaller model with Transformer layers. A group size of 32 is used in GRPO for all three datasets.
Learning curves. The learning curves our models on three datasets are shown Table 9 Across all datasets, RLVR consistently improves upon our base model. Notably, when training with next- token prediction on smaller datasets (Rope and Granular), the models are prone to overfitting and require early stopping. In contrast, RLVR continues significantly improve prediction performance on the validation set, demonstrating greater robustness and generalization.
Extended analysis. Our model underperforms DINO-WM on Rope dataset. We hypothesize that this is due to our model being prone to overfitting on this relatively small dataset. In contrast, recurrent latent state models that leverage large-scale pretrained visual encoders demonstrate greater robustness. To validate this hypothesis, explore training base model jointly the Rope and
28

---

Table 9: Learning curves of video world models on PushT, Rope, and Granular datasets. LPIPS scores are scaled by 100.
Pre-train
RLVR
Training steps
1 x 10
2 x 10
3 x 10
60
100
200
300
400
PushT LPIPS}
1.19
0.90
0.83
0.78
0.75
0.74
0.72
0.70
Pre-train
RLVR
Training steps
2 x 10
4 x 10*
6 x 10*
7 x 104
100
200
400
720
Rope LPIPS}
3.73
3.23
3.13
3.03
2.60
2.49
2.25
2.08
Pre-train
RLVR
Training steps
x 10*
10*
10*
40
100
200
300
500
Granular LPIPS..
3.82
3.22
3.14
2.68
2.55
2.56
2.46
2.42
Table 10: Performance of video world models jointly trained on deformable object manipulation tasks. LPIPS and SSIM scores scaled by 100.
Rope
Granular
Model
LPIPS
SSIM
LPIPS!
SSIM
Base (individually trained)
3.03
97.86
3.14
94.79
RLVR-Word (then individually tuned)
2.08
98.14
2.42
95.42
Base (jointly trained)
2.59
97.95
3.41
94.54
RLVR-Wood World (then individually tuned)
1.65
98.29
2.44
95.40
Table 11: Cross-task evaluation of video world models jointly trained on deformable object manipu- lation tasks. LPIPS and SSIM scores scaled by 100.
Rope
Granular
Model
LPIPS
SSIM
LPIPS
SSIM
Base (jinlyy trained)
2.59
97.95
3.41
94.54
RLVR- World (tuned on Rope)
1.65
98.29
3.35
94.60
RLVR-WA World (tuned Granular)
2.54
97.97
2.44
95.40
Granular datasets (both collected from PyFlex simulators), followed by fine-tuning with RLVR on each individual dataset. The results are shown in Table [0] We observe that the LPIPS on Rope improves from 0.0208 to 0.0165. Looking forward, we believe the potential of RLVR can be further unlocked when applied to large-scale pre-traine pre-trained world models, mirroring the remarkable success of LLMs.
We further investigate whether improvements in one domain can benefit the other. As shown in the Table TT] we observe that when the model is RL-trained exclusively on one dataset, performance on the other can improve slightly. This result aligns with discoveries the LLM field that improvements in one domain, such as math, can generalize to other domains. However, we also find that performance on the held-out dataset later degrades due overfitting. We attribute this to the limitations of the base model, which is not pre-trained on large-scale, diverse data and thus lacks truly generalizable representations. We hope these preliminary results can encourage the community to further explore RLVR in the context of large-scale world models.
Qualitative analysis.
Showcases of our RLVR-World models are presented in Figure 10
Model-predictive control. We strictly follow the same planner configuration as DINO-WM. including goal construction, number of samples, and optimization steps. Since our model predicts discrete tokens and raw pixels without compact latent space, the publicly available DINOv2
29

---

Ground-Iruth
(input)
(inpur)
(ingut)
(input)
(input)
(irput)
RLVR-World
Ground-truth
(Input)
(Input)
(Input)
(Input)
RLVR-World
Ground-truth
(input)
(input)
(input)
(input)
RLVR-World
Figure 10: Qualitative analysis: video prediction on PushT, Rope, and Granular.
Table 12: Model-predictive control with different world models on PushT.
CEM w/
CEM w/
CEM w/
Model
IRIS
DreamerV3
TD-MPC2
DINO-WM
Our Base
Our RLVR-Worl
PushT
0.32
0.30
0.00
0.86
0.80
0.86
Success Rate
encoder to embed predicted frames and compare them to the goal observation during planning. We find that the DINO latent space is more effective for planning than pixel-level MSE or LPIPS, consistent with the findings in DINO-WM. We emphasize that our MPC experiments are intended to compare world models, and the choice of distance metric is orthogonal
our main contributions.
We do not conduct experiments on Rope and Granular due to the lack of MPC configurations and details in the official DINO-WM code, making difficult to replicate the setup.
The results Table 12 show that RLVR also enhances the control performance of our base model, achieving results comparable to DINO-WM.
A.5 Real2Sim Policy Evaluation
Evaluated policies and baselines. We aim evaluate popular open-source generalist policies for the Google Robot, including series of RT- [5] checkpoints at different training stages: RT-1 trained convergence (RT-1 (Converged)), RT-1 at 15% of training steps (RT-1 (15%)). and RT-1 at the beginning of training (RT-1 (Begin)), as well as RT-1-X 49 SIMPLER BY is collection of simulated environments for manipulation policy evaluation on common real robot setups, including different methods to align with real-world environments: "SIMPLER Visual Matching" (short as SIMPLER- VM in our paper) and "SIMPLER- Variant Aggregation" (short as SIMPLER-VV). We compare our world models against SIMPLER the representation for policy evaluation methods
30

---

based on physical simulation, which requires substantial human effort to develop task-specific environments.
Task selection. We experiment with three task types: "pick coke can," "move near," and "open/close drawers," each further divided into subcategories (e.g., "open top/middle/bottom drawer"). ). In tasks involving the "pick" action, observe a high rate of false positive evaluations where models incorrectly predict successful trajectories despite poor action quality. This is likely due to the limitation in the expert-level training data, which lacks failed grasp attemptss. As result, the model tends to predict successful grasps even when the robot arm remains some distance from the object. In contrast, the "open/close drawers" task offers more diverse training data, including several failure cases (e.g., drawer slipping from grasp), making it more suitable evaluation setting for both our base model and RLVR-Wordd
Trajectory generation. For each task, we randomly sample 30 initial frames from the real-world RT-1 dataset serve as initial observations for world model simulations. Trajectories are then generated iteratively: (1) the policy selects the next action based on the latest frame; (2) the world model predicts the subsequent frame given action. Following SIMPLER 337 we limit each trajectory to maximum of 112 frames. Specifically, we use our trained world models for multi-step prediction, able to predict seven future frames conditioned on the current and initial context frame. To predict long trajectories, becomes the input first frame in the next generation round. To conclude, Our evaluation covers two world models (the base model and RLVR- RLVR-World), four policies (RT-1 (Begin), RT-1 (15%), RT-1 (Converged), and RT-1-X) 5 49 six tasks (open/close top/middleebootoom drawer), and 30 trajectories per task, resulting in total of 2 4 x 6 30 1440 generated trajectories.
apply sliding window approach, where the last predicted frame
Success judgment. We initially explored automatic evaluation of trajectory success using vision- language models (VLMs), including GPT-4o and Gemini 2.0 Flash API. However, both failed to provide reliable evaluations due to two main reasons: (1) results varied significantly with minor changes the prompt, and (2) the models lack consistent criteria-for example, how much the drawer must be opened to be considered success varied across scenes and backgrounds. Therefore, we resort human annotation for evaluation. tasked with labeling success or failure, provided only with the final frame of each trajectory and the task description, without any information about the model or policy used.
ensure consistency and rigor, single annotator is
Quantitative results.
The evaluated success rates of different methods are reported in Table 13
Table 13: Quantitative results for real2sim policy evaluation, corresponding to Figure
Task
Model
RT-1 (Begin)
RT-1-X
RT-1 (15%)
RT-1 (Converged)
Real
0.0%
51.9%
70.4%
81.5%
SIMPLER-VE
0.5%
6.9%
21.2%
27.0%
Open
IIMPLER-VM.
0.0%
29.6%
46.3%
60.1%
Drawer
Base model
4.4%
18.8%
50.0%
48.9%
-LrRdWorld
3.3%
33.3%
62.2%
62.2%
Real
0.0%
74.1%
88.9%
92.6%
SIMPLER-VA
13.2%
51.9%
32.3%
37.6%
Close
SIMPLER VM
27.8%
89.1%
66.7%
86,1
Drawer
Base model
20.0%
56.7%
76.7%
81.1%
RLrldWorld
18.9%
71.1%
78.9%
88.9%
B
Computational Resources
Text game state prediction The SFT phase uses 4x 80G A100 GPUs for 6.5 hours of training. The RLVR phase is conducted 8x 80G A100 GPUs over 22.5 hours. Both SFT and RLVR implemented using the verl framewor!
58
htttg:///gthabbb.om/oolloogggggs following the Apache License.
31

---

Web page state prediction SFT is performed on 8x 40G A100 GPUs over 17 hours. while the RLVR training is conducted on 8x 80G H100 GPUs for 25 hours. Both SFT and RLVR are implemented using the verl framework.
Robotic manipulation trajectory prediction. All experiments in this domain are conducted on 40G A100 GPU cluster. For single-step prediction, we pre-train the tokenizer using approximately 360 GPU hours. Transformer pre-training takes 530 GPU hours, and RLVR post-training for 200 steps requires 3.5 hours with 4x 40G A100 GPUs. For multi-step prediction, we pre-train the tokenizer with 480 GPU hours. Transformer pre-training takes 500 GPU hours, and RLVR post-training for 200 steps requires 10 hours with 4x 40G A100. Both settings implement tokenizer and transformer pre-training with accelerate and RLVR post-training using verl.
C Broader Impact
Academic research. In the of foundation models, our work provides proof of concept that the task performance of generative models can be directly optimized using RLVR, demonstrated through the world modeling task. These results may inspire further research into this paradigm. For instance, task-aligned metrics rather than human preferences could be used to reinforce the visual understanding capabilities of multimodal LLMs tasks such visual counting, object detection, and optical character recognition. Additionally, the effectiveness of RLVR as a post-training strategy may further solidify the community's 's preference for autoregressive generative models over alternatives like masked or diffusion models, as it allows leveraging the well-established LLM ecosystem with minimal modifications.
Practical applications. Enhancing the accuracy of world models through RLVR has clear benefits for real-world autonomous agents, both in digital domains (e-g., web navigation) and physical settings (e.g., robot manipulation), as showcased in our experiments. By enabling agents to simulate the outcomes of actions before execution more accurately. RLVR-trained world models can improve agentic task performance and reduce the risk of harmful behaviors. However, RLVR is relatively lightweight post-training stage, effectiveness ultimately bounded by the capabilities of the underlying base model. Continued efforts toward building more powerful foundation world models remain essential.
D Prompt Examples
In this section, present detailed prompts used in our LLM experiments to offer more intuitive understanding of language world models. These prompts are either directly taken from or adapted from prior work 65 8.
Text game state prediction: We provide examples of the prompts in blue boxes, including the JSON-forma game state, rules for action/ohject/score, as well as the complete prompt. Additionally, provide example answer for the stafe-difference prediction task. They are all kept identical to those in the original dataset.
Web page state prediction and web agents We present the prompts used in our study in orange boxes. Specifically, we include four types of prompts: Web State Prediction Prompt Example, Top-10 Salient Change Extraction Prompt, Next-State Change Summarization Prompt, and Web State Evaluation Prompt Example. The web state prediction prompt and next-change summarization prompt are modified versions of the original prompts proposed by WMA [8), while the top-10 salient change extraction prompt and value prediction (evaluation) prompt are used unchanged, adopted directly from WMA.
http:://gggtabbccoohggggghoooooo following the Apache License.
32

---

Text Game: State Difference Prediction Prompt Expample
You are simulator of text game. Read the task description and the current environment observation description. Given the current game state in JSON, you need to decide the new game state after taking action.
Your response should be in the JSON format. It should have three keys: 'modified', 'removed', and 'score'. The 'modified" key stores list of all the object states that added changed after taking the action. Keep an empty list if no object is added or modified. The 'removed" key stores list of uuids of the objects that are removed. Keep it an empty list if object is removed. The 'score' key stores a dictionary with three keys: 'score' is the current game score, 'gameOver' is boolean of whether the game is over, and 'gameWon' the game. If a player earns score or wins/loses the game, you should reflect that change in the dictionary saved under the 'score' key. Otherwise, you should set value of the "score' key to an empty dictionary.Noe Noto that while game states can be changed by actions, some game states may change over the time, which is described in the tick function each object class. Note that while game states can be changed by actions. some game states may change over the time, which is described in the tick function of each object class.
boolean of whether the agent
Here are two examples of both cases. Both examples are from the same example game. Example game task description: Your task is to wash the dirty dishes.
Here are the descriptions of all game objects properties example game:
Here are descriptions of all game actions in the example game:
LACTION_RULESS
Here a description of the score function the example game:
{SCORE RULES)
In the first example, the game state is changed by an action:
Current observation:
{GAME OBSERVATION]
Here is the game state:
{GAME_STAAE STATE]
The action to take is put dirty plate (ID: in mug (ID: 6)
The expected response is:
(GAME_STATEE STATTEDIFFERENCE
In the second example from the same example game, the game state is changed over the time. Note that while this example the game state changed by time only, is possible that a game state is changed by both an action and time.
Current observation:
[Example_ 2 observation
Here the game state:
('AME_STATE]
The action to take eat dishwasher (ID: 2) with dirty plate (ID: 5)
The expected response is:
SLAME_STATEE STTATE_DIFFRRNCC
Here is the game that you need to simulate:
Task Description: Your task is to boil water.
Here are the descriptions of all game objects properties:
{OBJECT RULES)
Here are the descriptions of all game actions:
{ACTION _CCIIN__RLLES]
Here a description of the score function of the game:
[CCOEERRULES]
Current observation:
{GAME OBSERVATION)
Here is the game state:
{GAME [
The current game UUID base is 12
The action to take is:
look
33

---

Text Game: Game State Example
I I'game_ state': [[ n'amem: "agent (ID: 0)', 'uuid': 0, 'type': 'Agent', 'properties': ('isContainer',
True, 'isMoveable': True, 'isOpenable": False, 'iSOpen:: True, 'containerPrefix:: 'in ). 'contains'. ['plate (ID: 5)', "mug (ID: 6)' 'knife (ID: 7)]}, ('name": "llate (ID: 5y, 'uuid': 'type": 'Dish', "properties': : ('isContainer': True, 'iMooeablle, True, 'sOOeenalle' False, 'isOpen': True, 'con- tainerPrefix': 'on', 'dishType': 'plate', 'isDirty': True, 'foodMessName" 'orange") 'contains':
[}. : mug (ID: 6)', 'uuid': 6, 'type': 'Dish', 'properties': { !'isContainer': True, iis- Moveable': True. 'sSOpnable': False, 'isOpen': True, 'containerPrefix": 'in'. 'dishType': 'mug', 'isDirty": True, MoooMMessName'.. 'sandwhich', J. "contains': [}). ('name": 'knife (ID: 7)', 'uuid': 'type'' 'Dish', 'properties': ('isContainer': True, 'isMoveable': True, 'isOpenable': False, 'isOpen': True, 'containerPrefix': 'in', "dishType": 'knife', "isDirty": True, "aoodMessNamme"
"apple (ID: 11)'), 'contains': (]). I'name': 'dishwahher (ID: 2)', 'uuid': 'type": Washer',.'', "properties': I'ssoonainer': True, 'ssoveabl'' False, 'isOpenable': True, 'isOpen': True, 'containerPrefix': 'in', 'isDevice': True, 'isActivatable": True, 'isOn': False, "cycleStage': O, 'finishedCycle' False), 'contains": l'cup (ID: 4)'l). ('name': ccup (ID: 4y', 'uuid": 4, 'type': 'Dish', 'properties:: ('isContainer': True, 'isMoveable': True, 'isOpenable'. False, 'isOpen': True. 'containetPrefix:: 'in', 'dishType': 'cup', 'isDirty': True, 'foodMessName" 'peanut but- ter'), 'contains': []}. I'name': 'bottle of dish soap (ID: 3)', 'uuid': 3, 'type': 'DishSoapBottle" 'properties': ('isContainer's False, 'sMoveable' True, 'isDevice': True. 'ssActivatable' True, 'isOn': False), 'contains': []. { 'name': : glass (ID: 8)', 'uuid': 8, 'type': 'Dish', 'properties': ('is- Container': : True, 'sMoveable'. True, 'sOpenable', False, 'isOpen': True, 'containerPrefix' 'in', 'dishType': "glass', , 'iDiitty:: False). 'contains:: [].. ( I'name': : 'bowl (ID: 9)', 'uuid': 9, 'type': 'Dish', properties': ('isContainer': True, "isooeabll' True, 'isOpenable': False, 'isOpen'. True, 'containerPrefix':: 'in', 'dish'yye': 'bowl', 'isDirty': False), 'contains': []. ( ("name": 'banana (ID: 10)', 'uuid': 10, 'type': 'Food', 'properties': ("isContainer': False, 'isMoveable':
True, 'isFood": True), "contains': (]). { ('score': -1, "gameOver': False, 'gameWon Won': False)])
Text Game: Game State Difference Example
('modified': (('amee: 'agent (ID: 0)', 'uuid': O, 'type": 'Agent', 'properties': ('isContainer':
True, 'isooveblee': True, 'isOpenable': False, 'isOpen': True, 'containerPrefix": 'in'), 'contains':
('mug (ID: 6)', "knife (ID: 7)']), ('name": "mug (ID: 6)', 'uuid': 6, 'type": "Dish", 'properties':
('isContainer': True, 'isMoveable': True, 'sSOpenalle:: False, 'isOpen': True, 'containerPrefix"
'in', 'dishType': 'mug'. 'isDirty": True, 'aoodMessName:: 'sandwhich's J, 'contains': ['plate (ID:
5)'] 'removed': [], 'score': (I)
34

---

Text Game: Action Rule Example
put:
Description: put an object into target container
Rules:
The target must be a container (Container)
2. The target container must be open
The object must be in the inventory
4. The object must be moveable (isMoveable)
Text Game: Object Rule Example
Object: Container
Description: Abstract class for things that can be considered 'containers' (e.g. drawer, box, table, a shelf, etc.)
Properties:
A Container a container.
A Container could be opened (e.g., e.g. drawer, door, box, etc.), always "open" (e.g. a table, shelf, etc.).
-A Container has property indicating if it opened.
-A Container has property indicating the prefix to use when referring to the container (e.g. "in the drawer", "on the table", etc.). By default, the prefix is "in"
Text Game:
Score Rule Example
The player wins the game by getting all dishes clean.
The player gets one point for each dish that is cleaned.
The player loses point for each dish that is made dirty.
35

---

Web Page: Web State Prediction Prompt Example
You intelligent agent that predicts the state from given current action using your logical reasoning. You will be given web-based tasks.
Here's the information you'll have:
The user's objective: This is the task you'r trying complete.
The current web page's accessibility tree: A simplified representation of the webpage, providing key information.
The current web page's URL:
This is the page you're 're currently navigating. The previous action: The action you just performed. It may help you track your progress.
The current action: The action that you will perform next to achieve the user's ss objective in the current accessibility tree.
The format of previous and current actions can fall into several categories:
Page Operation Actions:
click [id]: Click element with specific id.
type [id] [content]: Type the content into the field with the specified id. By default, the "Enter" key is pressed after typing unless [o] is appended.
hover [id]: Hover over an element.
press komb] Simulate pressing keyboard combination (e.g., Ctrl+v).
scroll [down] or scroll [up]: Scroll the page.
Tab Management Actions: new_tab: Open new browser tab.
tab_focus :aa_index]]: Switch focus to specific tab. close_tab: Close the currently active tab.
URL Navigation Actions:
goto [url]: Navigate to specific URL.
go_back: Go to the previous page.
go_farwdrd: Go forward (if previous gack was executed).
Completion Action:
stop [answer]: Use this when you believe task complete. Provide the final answer in brackets if applicable.
To succeed, it's essential to understand how the current action impacts the next state of the webpage. You must verify whether the current action successfully produces the intended effect.
Reasoning Guidelines:
1. Generate your answer using logical REASONING.
Directly predict the next state based on the current action.
3. Format your prediction exactly described below.
Format Description: The output divided into three sections:
New items:
Deleted items:
Updated items:
Each section should be separated by empty line. If section has no content, write None.
36

---

Web Page:
Web State Prediction Prompt Example (continued)
Each item must follow this format:
[ID] type 'content' [additional attributes]
Details:
[ID]: Numeric ID in square brackets.
type: Element type (e.g., button, textbox, StaticText, etc.).
'content': Element's 's displayed content in single quotes. Can be empty: ", [additional attributes]: Optional key-value pairs such required: False, True, etc.
focused:
Example New and Updated Items:
New items:
[1273] StaticText 'Vortex Running Shoes
[1272] button 'Search'
Deleted items:
None
Updated items:
[1220] textbox 'Que60c' focused: True required:
False
Now, let's start the task. Here's 's the task for you:
URL: {URL]
OBJECTIVE: (OBJECTIVE)
PREVIOUS ACTION: (PREVIOUS ACTION)
CURRENT OBSERVATION: [CURRENT OBSERVATION]
CURRENT ACTION: (CURRENT ACTION)
Please give the result directly in the format of the state prediction.
Web Agent: Top- Salient Change Extraction Prompt
Summarize the key changes in the web page based on the following information.
New items: (new_iteme)
Updated items: {updated_ items)
Deleted items: thelletedittems
When summarizing, follow this output format: [First key change]
2. [Second key change]
[Third key change]
10. [Tenth key change]
37

---

Web Agent: Next-State Change Summarization Prompt
You are an intelligent agent that predicts the next state based on the current action using your own logical reasoning. You will be given web-based task.
You will have access to the following information:
The user's objective: The task you re trying to complete.
The current observation: simplified representation of the current webpage.
The current URL: The webpage you're re currently on.
Previous actions: The action(s) performed just prior the current one.
Current action: The action being evaluated for its effect on the webpage.
Key changes in next state observation: summary of differences between the current and next state observations.
Action types may include:
Page Operation Actions:
click [id]: Click an element with the given ID.
type [id] [content]: Type content into field with the given ID (append [o] to suppress Enter).
hover [id]: Hover
element.
press [kmb]comb]: Simulate key combination presses (e.g., Ctrl+v).
scroll [down] or scroll [up]: Scroll the page.
Tab Management Actions:
new_tab, tab_focus [index], close_tab
URL Navigation Actions:
goto [url], back, gorward
Completion Action:
stop [answer]: Use this when the task is complete. Provide the answer in the brackets if required.
Your task: Predict the next state with proper reasoning. Follow these rules:
1. Begin your answer with: [Rationale] Let's think step by step...
2. Your reasoning must mention the key changes in the next state observation.
3. Then, deseribe the next state based on those key changes.
4. Output the prediction in this format: [Next State] The expected effect is that
Input fields:
URL: [url}
OBJECTIVE: {objective)
PREVIOUS ACTION: {prev_action}
CURRENT OBSERVATION: (cur_observation)
CURRENT ACTION: {cur_action)
KEY CHANGES IN NEXT STATE OBSERVATION: {tao}
38

---

Web Agent: Web State Evaluation Prompt Example
You are an expert in evaluating and guiding web navigation agent. Your task is to help the agent effectively complete given mission website based on the user's intent. The agent's 's goal is to navigate through the website to reach the desired state that aligns with the user's objective.
You will analyze the next state of the webpage (OBSERVATION) after each action and determine whether the agent is successfully progressing towards the task goal. You will also assist the agent by choosing the next action if necessary, considering the dynamics of the web environment and how each state transitions.
Key Points:
1. Understand the Intent:
Identify the user's goal (e.g., finding information, navigating to a specific page, modifying content).
Ensure the next state of the webpage aligns with achieving that goal based on the current state and user's 's' intent.
2. Evaluate the Next State:
Assess how the next state contributes to reaching the intended goal.
If the next state moves the agent closer to the user's goal, evaluate it positively.
If the next state does not progress towards the goal or leads to error, suggest alternative actions.
3. State Guidance:
If the agent is the right track but hasn't completed the task yet, recommend further actions.
Guide the agent toward a state that reflects clear progress towards the goal.
4. Types of Tasks:
Information Seeking: The next state must provide specific information (e.g., product price, reviews).
Site Navigation: The next
reflect navigation the exact page item. The next state must show that content has been successfully
Content Modification: modified.
General Task: The final state should reflect that the task is completed. Only issue stop action when the objective is met.
5. Common Pitfalls:
Avoid corrupted input from repeated typing actions.
Ensure the agent navigates the specific item or content, not just to general page.
Output Format with Likert Scale:
Each next state will be evaluated on a fine-grained scale from 1 to 5:
1: The next state is failure or leads away from the task.
2: The next state does not contribute meaningfully.
3: The next state neutral; the agent maintaining position.
4: The next state is helpful; progress is being made.
5: The next state is optimal; directly aligns with the task goal.
Response Instructions:
Write a rationale providing a detailed analysis of the next state and justify your chosen score. Output Format:
[Rationale]:
<your thought> [Score]
<1-5>
39


Reviews:
[{"review_id": "r5kVg7pd3K", "reviewer": "Reviewer_BbTo", "summary": "This paper proposes to train world model on text or video based transition datasets with reinforcement learning instead of the conventional maximum likelihood estimation. Specifically, the inputs of states and actions are tokenized and the model is trained to predict the next state via reinforcement learning. The authors evaluate the proposed method on text-based transition dataset and a manipulation task, and show that the proposed method can achieve better performance than the supervised fine-tuning (SFT) baseline.", "questions": "See strengths and weaknesses part.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n* The paper is well-written and clearly presents the motivation, method, and results of the proposed approach. The clarity of the writing makes it easy to follow the logic and understand the contributions.\n\n## Weaknesses\n\n* The overall technical contribution of the paper is limited. The tokenization, model architecture, and training method are not novel. This paper does not provide a significant insight or advancement in this field.\n\n* The proposed method of training a world model is very general, while the evaluations are mostly limited to either text form under a language model or a single visual dataset of manipulation. There are diverse datasets in the field of world model, and experiments in other popular domains of robotic or gaming simulations are worthwhile to conduct [r1].\n\n* In experiments, there are very few baselines for comparison with the proposed world model training. The main comparisons are executed based on vanilla pre-trained models and SFT. Many other methods of world model like recurrent latent state models are also important baselines [r1].\n\n* The size of the language model used is much smaller than the state-of-the-art models, which may limit the scalability of the proposed method over larger models.\n\n* It seems that there is no experiments over model predictive control (MPC) in the manipulation task. One major application of world models is to enable MPC in robotic manipulation tasks without interaction with the true environment. The paper would benefit from including experiments on MPC with the proposed world model.\n\n[r1] Zhou G, Pan H, LeCun Y, Pinto L. Dino-wm: World models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983. 2024 Nov 7.", "was_revised": true, "final_justification": "Additional experiments addressed part of my concerns. While the overall technical novelty remains limited and new experiments on MPC in PushT do not show improved performance over previous work.", "rebuttal": "We sincerely thank Reviewer BbTo for the valuable and comprehensive feedback. We have made our best effort to provide additional clarifications and experimental results to address your concerns.\n\n### Q1: Novelty and Insights\n\nWe respectfully disagree with the comment that our work lacks significant insight or advancement in this field, and we apologize for not articulating our contributions clearly enough.\n\nAs illustrated in Figure 1 and the Introduction, world models are becoming increasingly capable but also more complicated, often involving non-end-to-end architectures and training objectives. To our knowledge, our work is **the first to identify the following key insight**:\n\n> As world models are built with more advanced models, their training methods diverge further from the actual task objective of world modeling.\n\nThis fundamental observation has not been discovered or addressed in prior work on autoregressive world models [1,2] or diffusion-based world models.\n\nThis necessitates new and more aligned training methods, especially after large-scale pre-training with traditional surrogate objectives. In this context, **identifying RLVR as a direct and effective training method to meet the above insight, is a novel and meaningful contribution**. Fully leveraging this insight leads to substantial improvements: as shown in Figure 3, the gains achieved in just a few hundred RL steps cannot even be achieved by 1000x more steps of standard next-token prediction training. We also consider **this first successful application of RLVR to video world models** as a minor but still novel technical contribution.\n\nAlthough our work may not introduce entirely new techniques, we believe this paradigm-shifting insight is both valuable and impactful. We are encouraged that other reviewers recognized this contribution—for instance, Reviewer ndrj described the idea as \"novel and a significant departure from traditional training objectives,\" while Reviewer d8tC noted that we \"broaden the applicability\" of RLVR.\n\n### Q2: Additional Datasets & Baselines\n\nTo provide a more comprehensive evaluation, we include additional datasets and advanced baselines, following the reviewer's suggestion.\n\n**Datasets**: We adopt **three datasets (Push-T, Rope, and Granular)**, from DINO-WM [3]. We strictly follow the evaluation protocol described in the original paper and official code, including image resolution, training-validation split, number of history frames, and frame skip.\n\n**Baselines**: We compare our models against all recurrent latent state models provided in [3], which use different latent spaces from pretrained encoders such as R3M, ResNet, and DINO. We also include an action-conditioned diffusion model, AVDC. \n\n**Results**: The learning curves of our models on three datasets are shown below (Due to NeurIPS policy, we cannot provide figures in rebuttal). \n\n|Training step|100000|200000|300000 (Pretrain)|(RLVR) 60|100|200|300|430|\n|-|-|-|-|-|-|-|-|-|\n|PushT LPIPS|0.0119|0.0090|0.0083|0.0078|0.0075|0.0074|0.0072|**0.0070**|\n\n|Training step|20000|40000|60000|70000 (Pretrain)|(RLVR) 100|200|400|720|\n|-|-|-|-|-|-|-|-|-|\n|Rope LPIPS|0.0373|0.0323|0.0313|0.0303|0.0260|0.0249|0.0225|**0.0208**|\n\n|Training step|10000|20000|30000 (Pretrain)|(RLVR) 40|100|200|300|500|\n|-|-|-|-|-|-|-|-|-|\n|Granular LPIPS|0.0382|0.0322|0.0314|0.0268|0.0255|0.0256|0.0246|**0.0242**|\n\nAcross all datasets, **RLVR consistently improves upon our base model**. Notably, when training with next-token prediction on smaller datasets (Rope and Granular, each containing only 1,000 episodes of length 20), the models are prone to overfitting and require early stopping. In contrast, RLVR continues to significantly enhance prediction performance on the validation set, demonstrating greater robustness and generalization.\n\nWe then compare the prediction capabilities of different world models. Since DINO-WM only releases model checkpoints for the Push-T dataset among the three, we additionally include our own evaluation results using the publicly available Push-T checkpoint. \n\n|Method|LPIPS ↓|LPIPS ↓|LPIPS ↓|SSIM ↑|SSIM ↑|SSIM ↑|\n|-|-|-|-|-|-|-|\n| |PushT|Rope|Granular|PushT|Rope|Granular|\n|R3M|0.045|0.023|0.080|0.956|0.982|0.917|\n|ResNet|0.063|0.025|0.080|0.950|0.980|0.915|\n|DINO CLS|0.039|0.029|0.086|0.973|0.980|0.912|\n|AVDC|0.046|0.060|0.106|0.959|0.979|0.909|\n|DINO-WM (Reported)|**0.007**|**0.009**|0.035|**0.985**|**0.985**|0.940|\n|DINO-WM (Public checkpoint)|0.0339|-|-|0.9638|-|-|\n|Base (Ours)|0.0083|0.0303|0.0314|0.9828|0.9786|0.9479|\n|RLVR-World (Ours)|**0.0070**|0.0208|**0.0242**|**0.9846**|0.9814|**0.9542**|\n\n**Our model after RLVR achieves overall performance comparable to the strongest baseline**, DINO-WM, and outperforms it by a significant margin on the most challenging particle dataset, Granular.\n\nOur model underperforms DINO-WM on the Rope dataset. We hypothesize that this is due to our model being prone to overfitting on this relatively small dataset. In contrast, recurrent latent state models that leverage large-scale pretrained visual encoders demonstrate greater robustness. To validate this hypothesis, we explore training a base model jointly on the Rope and Granular datasets (both collected from PyFlex simulators), followed by fine-tuning with RLVR on each individual dataset:\n\n|Method|LPIPS ↓|LPIPS ↓|SSIM ↑|SSIM ↑|\n|-|-|-|-|-|\n| |Rope|Granular|Rope|Granular|\n|Base (Ours, jointly trained)|0.0259|0.0341|0.9795|0.9454|\n|RLVR-World (Ours, individually fine-tuned)|**0.0165**|**0.0244**|**0.9829**|**0.9540**| \n\nWe observe that the LPIPS on Rope improves from 0.0208 to 0.0165. Looking forward, we believe the potential of RLVR can be further unlocked when applied to large-scale pre-trained world models, mirroring the remarkable success of LLMs.\n\n### Q3: Scaling Language Models\n\nWe appreciate the reviewers' rigor and agree that, in the modern era, the effectiveness of LLM algorithms should ultimately be validated in truly large-scale settings. At the same time, we believe that scientific progress is inherently incremental. The foundational algorithm behind RLVR, GRPO, was initially validated on a relatively small-scale language model, DeepSeekMath 7B. We believe our current results remain valuable and can inspire further exploration within the community.\n\nNonetheless, we have made our best effort to scale our text game state prediction experiments to 7B-parameter language models. Specifically, we use DeepSeek-R1-Distill-Qwen-7B as our base model.\n\n|Model|Unchanged Acc.|Changed Acc.|Overall Acc.|\n|-|-|-|-|\n|Base (1.5B)|11.98%|0.08%|7.11%|\n|SFT (1.5B)|38.88%|24.21%|32.87%|\n|RLVR-World (1.5B, Ours)|73.57% (+34.69%)|33.14% (+8.93%)|57.01% (+24.14%)|\n|Base (7B)|46.90%|5.53%|29.92%|\n|SFT (7B)|65.94%|31.32%|51.76%|\n|RLVR-World (7B, Ours)|**83.08%** (+17.14%)|40.33% (+9.01%)|**65.53%** (+13.77%)|\n|GPT-4|73.90%|**51.60%**|64.76%|\n\nThe results demonstrate that the effectiveness of RLVR-World scales to 7B models, with our final overall accuracy matching that of GPT-4. We hope these new results further strengthen your confidence in the scalability of our method.\n\n### Q4: Model Predictive Control\n\nWe first note that our original submission already includes experiments on policy evaluation, a critical subtask of model predictive control (MPC). Conducted in the realistic RT-1 domain, this experiment is both challenging and well-suited to demonstrate the benefits of our method for robotic manipulation. \n\nDue to time constraints, it is not feasible for us to set up real robot control experiments. Instead, we conduct additional MPC experiments in the simulated Push-T environment, following the setup used in DINO-WM.\n\n**Setup**: We strictly follow the same planner configuration as DINO-WM, including goal construction, number of samples, and optimization steps. Since our model predicts discrete tokens and raw pixels without a compact latent space, we use the publicly available DINOv2 encoder to embed predicted frames and compare them to the goal observation during planning. We find that the DINO latent space is more effective for planning than pixel-level MSE or LPIPS, consistent with the findings in DINO-WM. We emphasize that our MPC experiments are intended to compare world models, and the choice of distance metric is orthogonal to our main contributions.\n\nWe do not conduct experiments on Rope and Granular due to the lack of MPC configurations and details in the official DINO-WM code, making it difficult to replicate the setup.\n\n**Results**: The results below show that RLVR also enhances the control performance of our base model, achieving results comparable to DINO-WM.\n\n|Model|IRIS|DreamerV3|TD-MPC2|CEM with DINO-WM|CEM with Our Base|CEM with Our RLVR-World|\n|-|-|-|-|-|-|-|\n|PushT Success Rate|0.32|0.30|0.00|**0.86**|0.80|**0.86**| \n\n### In Summary\n\nWe sincerely thank the reviewer once again, especially for pointing us to the relevant work, DINO-WM. The additional experiments on **three new datasets and two task setups, with comparisons against several advanced world models**, further strengthen our contributions. We will include all experimental settings, quantitative results, and visualizations of both prediction and planning in the next version of our paper.\n\n[1] Transformers are Sample-Efficient World Models. ICLR 2023. \n\n[2] iVideoGPT: Interactive VideoGPTs are Scalable World Models. NeurIPS 2024.\n\n[3] DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning. ICML 2025."}, {"review_id": "QNZqMSjEVD", "reviewer": "Reviewer_ndrJ", "summary": "This paper presents RLVR-World, a framework that applies reinforcement learning with verifiable rewards (RLVR) to optimize world models across different modalities, including language and video. It investigates the use of RLVR for fine-tuning language models in text games and web navigation tasks, as well as video models for robot manipulation trajectory prediction. The results show performance improvements in prediction accuracy and visual metrics, and the framework is also applied to downstream tasks such as model predictive control and policy evaluation.", "questions": "1. \" Why does the model's accuracy improve by +34.7% for 'unchanged' cases and by +8.9% for 'changed' cases in Text Game State Prediction?   Does this difference reflect an inherent mechanism within the model for dealing with different situation?\"", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength：\n1. The idea of using RLVR to optimize world models is novel and represents a significant departure from traditional training objectives.\n2. The paper pioneers the application of RLVR in video world models, demonstrating its potential beyond language models. This opens up new avenues for research in generative models for visual data.\n\nWeakness:\n1.  The paper does not provide a detailed comparison with other recent advances in video world models", "was_revised": true, "final_justification": "My main criticism was the lack of SOTA comparisons. The authors gave a sufficient reaspn and completely rectified this with extensive and convincing new experiments. This new evidence fundamentally elevates the paper's contribution.", "rebuttal": "We sincerely thank Reviewer ndrJ for the thorough review and valuable questions. We hope our following response can resolve your concerns and strengthen your confidence in the value of our work.\n\n### Q1: Comparison with Advanced Video World Models\n\nFirst of all, we would like to clarify that our work does not aim to introduce a new state-of-the-art world model. Instead, we reveal a new training paradigm (RLVR) that can improve existing world models more effectively than traditional training approaches. This paradigm, along with the underlying insight, is general and applicable to a wide range of models. For demonstration, our paper applies it to the recent state-of-the-art world model, iVideoGPT (NeurIPS 2024).\n\nNevertheless, in response to Reviewer BbTo's suggestion, we have added **comparisons with a number of advanced video world models across three new datasets and two tasks**: next-frame prediction and model predictive control.\n\nPlease refer to our response to Q2 of Reviewer BbTo for full experimental details and extended results. Key results are presented as follows.\n\n**Next-Frame Prediction**\n\nWe have conducted experiments on three additional datasets (PushT, Rope, Granular), comparing against advanced recurrent latent space models, including DINO-WM (ICML 2025) [1], as well as an action-conditioned diffusion model, AVDC (ICLR 2024) [2]. The results are presented below.\n\n|Method|LPIPS ↓|LPIPS ↓|LPIPS ↓|SSIM ↑|SSIM ↑|SSIM ↑|\n|-|-|-|-|-|-|-|\n| |PushT|Rope|Granular|PushT|Rope|Granular|\n|R3M|0.045|0.023|0.080|0.956|0.982|0.917|\n|ResNet|0.063|0.025|0.080|0.950|0.980|0.915|\n|DINO CLS|0.039|0.029|0.086|0.973|0.980|0.912|\n|AVDC|0.046|0.060|0.106|0.959|0.979|0.909|\n|DINO-WM (Reported)|**0.007**|**0.009**|0.035|**0.985**|**0.985**|0.940|\n|DINO-WM (Public checkpoint)|0.0339|-|-|0.9638|-|-|\n|Base (Ours)|0.0083|0.0303|0.0314|0.9828|0.9786|0.9479|\n|RLVR-World (Ours)|**0.0070**|0.0208|**0.0242**|**0.9846**|0.9814|**0.9542**|\n\nOverall, our model, after applying RLVR, achieves performance comparable to the strongest baseline, DINO-WM, and significantly outperforms all other baselines. Notably, it surpasses DINO-WM by a substantial margin on the most challenging particle-based dataset, Granular.\n\n**Model Predictive Control (MPC)**\n\nWe also conduct MPC experiments, comparing with state-of-the-art world models, including DreamerV3 (Nature 2025), TD-MPC2 (ICLR 2024), IRIS (ICLR 2023), and DINO-WM.\n\n| Model              | IRIS | DreamerV3 | TD-MPC2 | DINO-WM  | Base (Ours) | RLVR-World (Ours) |\n| ------------------ | ---- | --------- |-|-| ----------- | ----------------- |\n| PushT Success Rate | 0.32 | 0.30|0.00| **0.86** | 0.80        | **0.86**          | \n\nThe results show that RLVR also improves the control performance of our base model, achieving the strongest results and outperforming popular world model methods such as DreamerV3 and TD-MPC2.\n\n### Q2: Discussion on Text Game State Prediction Accuracy\n\nFor the accuracy difference in unchanged and changed cases:\n\n1. **The difference in accuracy and RL gains stems from differences in task difficulty**. Changed cases are inherently more challenging than Unchanged ones: for Unchanged cases, the model only needs to correctly predict that no objects have changed; for Changed cases, the model must not only identify which objects will change but also accurately predict the outcomes of those changes. Therefore, Changed cases are harder to answer, and improvements are more difficult to achieve.\n2. Improvement in both cases indicates the model can handle opposing tasks. Unchanged cases favor more conservative prediction (\"no changes\" is sufficient), while Changed cases require more aggressive predictions to capture object changes. Our approach avoids collapsing into a single scenario, and instead can accommodate both types of tasks and truly enhance the learned relationship between actions and changes.\n\nIn summary, these results show that RLVR is applicable to complex and diverse world model settings, **improving performance across multiple task types**, with more substantial gains observed in the simpler task.\n\n[1] DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning. ICML 2025.\n\n[2] Learning to Act from Actionless Videos through Dense Correspondences. ICLR 2024."}, {"review_id": "BHqCc6jzur", "reviewer": "Reviewer_d8tC", "summary": "The authors introduce RLVR-World, a novel unified framework that leverages reinforcement learning with verifiable rewards to directly optimize world models for downstream task objectives. Through extensive evaluations on both language- and video-based benchmarks, RLVR-World achieves significant and consistent performance improvements across diverse domains.", "questions": "- The scope of the evaluation is currently limited to narrowly-defined, task-specific scenarios (e.g., RT1). By contrast, DeepSeek-R1 applies the RLVR framework across a diverse set of domains—math, code, reasoning—and demonstrates consistent gains, suggesting that RLVR can endow models with broader “general” capabilities. Could the authors extend their analysis to assess whether RLVR likewise enhances the general-purpose predictive power of the world model? Without such analyses, the reported results remain consistent with the well-known benefits of the pretrain–finetune paradigm rather than demonstrating novel, task-agnostic advances.\n- About Figure 5, a smaller measured discrepancy does not necessarily correspond to a reduced sim-to-real gap. For example, imagine evaluating 100 trajectories: the first 50 succeed only in the real environment and the remaining 50 succeed only in simulation. In this case, the discrepancy metric would be zero, yet the sim-to-real performance gap is maximal.\n\n**Minor Comments**\n\n- (Line 220) “observation conditioed on” -> “observation conditioned on”", "limitations": "The authors discuss the limitations of their work in the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- Extends the RLVR framework—originally formulated for pure LLM settings—to vision-based world models, thereby broadening its applicability.\n- The authors conduct extensive experiments on both language- and video-based world models across multiple domains, demonstrating the robustness and generality of their approach.\n- Overall, the paper is well written and easy to follow.\n\n**Weaknesses:**\n\n- The paper’s central paradigm—leveraging a pretrained model followed by task-specific fine-tuning (whether via supervised learning or reinforcement learning)—is by now a well-worn recipe, particularly in NLP. Extending this two-stage approach to autoregressive video world models is sensible, but it does not in itself constitute a surprising or fundamentally new insight, especially a number of prior works have already framed world-model learning as autoregressive sequence prediction over discretized multimodal tokens as mentioned in section 2.", "was_revised": true, "final_justification": "The authors propose RLVR, an effective method for directly optimizing task objectives in world models. While RLVR has demonstrated success in the context of large language models, its successful adaptation to world modeling—along with comprehensive empirical validation—represents a meaningful contribution.", "rebuttal": "We would like to sincerely appreciate Reviewer d8tC for the comprehensive review and insightful questions. \n\n### Q1: Clarification on Our Insight and Contribution\n\nWe would like to clarify that our central paradigm contribution is **not simply performing task-specific fine-tuning after pre-training**, but rather **revealing RLVR as a more task-aligned fine-tuning paradigm for world models**, compared to conventional supervised fine-tuning (referred to as maximum likelihood estimation, or MLE, in our paper). In other words, our key insight addresses *how* fine-tuning should be done, rather than *whether* fine-tuning should be done at all.\n\nAs we discussed in the Introduction, advanced world models, whether autoregressive or diffusion-based, often employ non-end-to-end architecture and surrogate training objectives. While such approaches are scalable and well-suited for pre-training, we point out that **continuing to use the same training methods for task-specific fine-tuning is not effective enough**. This is because these surrogate training methods misalign with the actual task objectives of world models. Our insight is that switching to a more aligned training paradigm, RLVR, can lead to significantly faster and more effective post-training. For example, as shown in Figure 3, **training speed of RLVR can be more than 1000x faster**. Such results are **not already well-known**, sufficiently surprising, and worth sharing with the community.\n\nWe acknowledge that the current form of Figure 1 may have caused confusion regarding our paradigm contribution. We will revise it in the next version to more clearly emphasize the differences between supervised and RLVR fine-tuning.\n\n### Q2: Enhancing General-Purpose Capability\n\nThe mechanism by which RLVR improves general capabilities in LLMs such as DeepSeek-R1 remains an open and active research question. A common belief is that the base model must be sufficiently strong for task-specific improvements (e.g., in math or code) to transfer to broader domains. Unfortunately, training LLMs with hundreds of billions of parameters exceeds our available computational resources. Moreover, at present, no video world model exists with similarly strong general capabilities as LLMs. For this reason, we focus our experiments on a specific scenario (RT-1). Nevertheless, we provide additional experiments below using a \"general\" video world model across two relevant domains.\n\n**Preliminary experiments**: We adopt two datasets, Rope and Granular,  from DINO-WM [1], a related work suggested by Reviewer BbTo. Both are collected from the same simulator, PyFlex, but involve different tasks. A shared base model is jointly trained on the two datasets, and then fine-tuned by RLVR on each dataset individually, to investigate whether improvements in one domain can benefit the other.\n\n| Method                   | LPIPS ↓ | LPIPS ↓  | SSIM ↑ | SSIM ↑   |\n| ------------------------ | ------- | -------- | ------ | -------- |\n|                          | Rope    | Granular | Rope   | Granular |\n| Base (jointly trained)   | 0.0259  | 0.0341   | 0.9795 | 0.9454   |\n| RLVR-World (on Rope)     | 0.0165  | 0.0335   | 0.9829 | 0.9460   |\n| RLVR-World (on Granular) | 0.0254  | 0.0244   | 0.9797 | 0.9540   |\n\nAs shown in the table above, we observe that when the model is RL-trained exclusively on one dataset, performance on the other can improve slightly. This result aligns with discoveries in the LLM field. However, we also find that performance on the held-out dataset later degrades due to overfitting. We attribute this to the limitations of the base model, which is not pre-trained on large-scale, diverse data and thus lacks truly generalizable representations. We hope these preliminary results can encourage the community to further explore RLVR in the context of large-scale world models.\n\n### Q3: Discussion on Policy Evaluation Metrics\n\nThe goal of real2sim policy evaluation is to **identify effective policies using only rollouts from simulated environments**. In this setting, our focus is not on reducing the \"sim-to-real gap\", a topic already addressed in detail in our prediction experiments (Sections 6.2 and 6.3). Instead, we aim to assess whether the model can accurately estimate the success rate of a given policy. In this context, a smaller measured discrepancy is sufficient to demonstrate the model’s effectiveness.\n\nThis formulation aligns with the standard setup in most existing policy evaluation studies. For example, [2] introduced a benchmark using metrics like rank correlation to compare policy rankings derived from success rates in simulators and the real world. Similarly, [3] and [4] employed metrics like Pearson correlation and regret@1, both of which are computed over aggregated success rates rather than individual trajectories. SIMPLER [5], a collection of simulated environments on common real robot setups, also takes its accurately estimated success rates as evidence to demonstrate that simulation-based evaluation can be a scalable, reproducible, and reliable proxy for real-world evaluation.\n\n[1] DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning. ICML 2025.\n\n[2] Benchmarks for Deep Off-Policy Evaluation. ICLR 2021.\n\n[3] Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization. ICLR 2021.\n\n[4] Variational Latent Branching Model for Off-Policy Evaluation. ICLR 2023.\n\n[5] Evaluating Real-World Robot Manipulation Policies in Simulation. CoRL, 2024."}, {"review_id": "Wxg9WXZI7I", "reviewer": "Reviewer_NuUF", "summary": "World models are becoming increasingly important for robotics, and the current approach of training them through MAE/MSE regression can yield bad results, like blurry outputs. Related works in LLMs have found that RLVR increases performance of sequence modeling, which is related to the objectives of world models. This paper brings RLVR to these world models by defining verifiable rewards in a text exploration environment and using similarity metrics to ground truth video frames in the video world model setup. Results show meaningful improvement of both text and video models after RLVR. World models trained with RLVR have downstream benefits, including planning (MPC) and real-to-sim evaluations.", "questions": "Questions\n\n- From Figure 4 Right, it seems like training on LPIPS objective directly can get comparable performance (0.124) to the convergence value of RLVR (Figure 3 Left). Is there an explanation and/or experiment that could differentiate the 0.124 achieved by directly optimizing on LPIPS vs. running RLVR? Or, is the difference significant already?\n- Related to the question above, what happens when you train the video world model directly on the reward objective? It could be interesting to see this comparison to gauge the impact of the reinforcement approach over just finding a better objective. If I get these results and there is indeed a significant difference, I’m willing to increase my scores\n- The paper mentioned that there are no off-the-shelf video world models, although there is a recent world model (UVA, https://unified-video-action-model.github.io/) that could potentially be relevant. It could be interesting to show results on this or another similar off-the-shelf model to verify your findings, although this is a heavy lift and I don’t expect this to be done during the review process.\n- How is repetition defined? In pixel space or in a different representation?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- Paper proposes a reasonable application of RLVR to world models: like proofs and science, there are certain facts about the world that are very rule-based, like physics and other causal chains.\n- Verified that RLVR works not only on text world models, but also on video world models with meaningful qualitative differences between pre-RLVR and post-RLVR models\n- verified that the model can work with MPC in the text environment, showing meaningful downstream improvements (table 2)\n- showed that the video model can be used for real-to-sim evaluation and quantiative results show RLVR as closest to corresponding to real success rate.\n- good inclusion of repetition rejection as a baseline for multi-step prediction\n\nWeaknesses\n\n- rule-based rewards make sense in the text environment but the connection is weaker for the video model. The objective just becomes another distance metric to a ground truth. A closer analogy to rule-based rewards would be understanding if the laws of physics is being obeyed, if the task is being represented conceptually correctly, etc.\n- Related to this past point: GRPO gradient and the objective gradients are different, although on this bandit setting, it’s not immediately obvious what GRPO can do that directly optimizing the reward objective can’t do.\n- For video world model, it could be interesting to see other objectives for RLVR to showcase how it can work beyond one type of loss function. In particular, it could be interesting to show how it can work with non-differentiable objectives that can’t be directly optimized upon.\n- Summarizing some of the points above: the most notable weakness is with the video model and separating the effects of reinforcement from the choice of objective. I am willing to improve my marks on “significance” and “originality” if a more suitable explanation and/or experiments are done on this point.", "was_revised": true, "final_justification": "See comment added to the review; In summary, the additional result and clarification removed the main concerns I had about this paper. Although this idea has existed outside of robotics, I believe that this implementation can be a meaningful contribution to the robotics field.", "rebuttal": "We sincerely thank Reviewer NuUF for the thoughtful and insightful comments. We hope our response addresses any concerns or misunderstandings and more clearly conveys the value of our work.\n\n### Q1: Can We Directly Optimize Reward Objectives?\n\nWe respectfully suppose that there might be a slight misunderstanding regarding the motivation of our work, which we clarify below:\n\n1. **Advanced world models (including ours) typically cannot directly optimize task objectives via gradients, due to their non-end-to-end architectures**, as discussed in the Introduction. \n    - Our autoregressive world models consist of a discrete visual tokenizer and a GPT-style transformer. The standard training pipeline involves encoding ground-truth frames into discrete tokens using a pre-trained tokenizer and training the transformer for next-token prediction via a cross-entropy loss. It is not feasible to directly optimize objectives such as pixel MSE or LPIPS by gradients, because **the intermediate steps between transformer and decoder, i.e., sampling predicted tokens and retrieving their corresponding visual embeddings from the tokenizer codebook, are not differentiable**.\n    - Another popular class of diffusion-based world models also face similar problems: they are trained by step-wise denoising objectives, rather than end-to-end target objectives.\n2. These surrogate training paradigms make **task-specific prediction objectives in video world models well analogous to rule-based reward functions** in LLMs, just as an LLM supervised trained to mimic the chain-of-thought reasoning for math problems does not directly optimize for answer correctness.\n2. With this insight, our contribution is **not to compete with or enhance direct objective gradient optimization**, but to **enable direct optimization of task objectives in complicated world model architectures where this is previously infeasible**, by leveraging RL. This is why we believe the comparison with direct gradient optimization, as suggested by the reviewer, is not necessary (and actually not feasible) in our context.\n3. **Clarification on metric-oriented optimization (Figure 4)**: We would like to clarify that in Figure 4, all metrics are optimized using RLVR (instead of end-to-end gradients which is infeasible here), same as our main results. The purpose of this experiment is to show that RLVR can indeed target optimizing desired objectives, similar in spirit to direct gradient optimization in end-to-end architectures. The difference in test LPIPS between training solely for LPIPS (0.124 in Figure 4) and our main result (0.122 in Figure 3, optimized jointly for MAE and LPIPS) is not statistically significant and is expected, given that both setups emphasize LPIPS.\n\n### Q2: Non-Differentiable Rewards\n\nWe indeed agree with the reviewer that performing experiments on non-differentiable rewards, which can not even be optimized by gradients in an end-to-end architecture, will strengthen our experiments, which we conduct as follows. \n\n**Repetition penalty reward (RPR)**: In Section 6.3, we show that RLVR can reduce repetition artifacts in multi-step video prediction (for the formal definition of repetition, see Q4). However, the repetition rate remains non-zero. To address this, we introduce an additional reward term, repetition penalty reward (RPR), defined as the negative rate of consecutive identical frames. We note that this term is non-differentiable. After training the model with this extended reward using RL, we observe that it maintains comparable prediction performance while effectively eliminating repetition artifacts.\n\n| Model                 | Repetition Rate↓ | MSE↓  | PSNR↑ | SSIM↑ | LPIPS↓ |\n| --------------------- | ---------------- | ----- | ----- | ----- | ------ |\n| Base                  | 48.6%            | 0.659 | 23.1  | 80.9  | 14.8   |\n| RLVR-World (w/o RPR)  | 9.9%             | 0.486 | 24.1  | 82.4  | 13.4   |\n| RLVR-World (with PRR) | **0.0%**         | 0.503 | 24.0  | 82.1  | 13.6   |\n\n**On physical law**: We have discussed the use of physical laws as reward functions in the Discussion section. Unfortunately, we currently lack a comprehensive method to detect whether generated videos adhere to physical laws—a challenging problem that still requires further advancements from the research community.\n\n### Q3: Recent World Model UVA\n\nWe greatly appreciate the reviewer bringing up the advanced world model, UVA. While the core insight behind RLVR-World is general and not tied to a specific model class, we note that UVA is implemented as a diffusion model, and its associated GRPO algorithm [2] was developed concurrently with our submission. As such, it falls outside the scope of our current work. We are excited to explore this and will add discussions on applying RLVR for diffusion world models in the next revision of our paper.\n\n### Q4: Repetition Definition\n\nAs our model uses a visual tokenizer to encode pixels into discrete tokens, we define a case of \"repetition\" when the predicted tokens of the last two frames have exactly the same token IDs, which are also decoded into identical pixel values. Empirically, we find that when the last two predicted frames are the same, it often indicates repetition across many consecutive frames, as showcased in Figure 6.\n\n[1] DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning. ICML 2025.\n\n[2] DanceGRPO: Unleashing GRPO on Visual Generation. arXiv 2025.05."}]

Author rebuttal:

Meta-review: This paper proposes to train world models for downstream task objectives, by tokenizing the states and actions and asking the model to predict next state via reinforcement learning, instead of the conventional MLE approach. This idea has been explored in LLM research but not in robotics/worldmodels until this work. The results are strong in multiple domains. The paper received a consensus of positive reviews: 4, 4, 5, 5. Initially the reviewers pointed out that the paper was missing some SOTA comparisons, but sufficiently convincing experiments were added in the rebuttal to turn all reviewers positive. The AC sides with the consensus and recommends acceptance.
Decision: Accept (poster)