Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
Authors: Minting Pan
Yitao Zheng
Jiajian Li
Yunbo Wang
Xiaokang Yang
Venue: ICML 2025 poster (2025)
Keywords: Offline Reinforcement Learning
Model-based Reinforcement Learning
Visuomotor Control
Primary area: reinforcement_learning->batchoffline

Abstract: Offline reinforcement learning (RL) enables policy optimization using static datasets, avoiding the risks and costs of extensive real-world exploration. However, it struggles with suboptimal offline behaviors and inaccurate value estimation due to the lack of environmental interaction. We present Video-Enhanced Offline RL (VeoRL), a model-based method that constructs an interactive world model from diverse, unlabeled video data readily available online. Leveraging model-based behavior guidance, our approach transfers commonsense knowledge of control policy and physical dynamics from natural videos to the RL agent within the target domain. VeoRL achieves substantial performance gains (over 100% in some cases) across visual control tasks in robotic manipulation, autonomous driving, and open-world video games. Project page: https://panmt.github.io/VeoRL.github.io.

Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
Minting Pan Yitao Zheng'
Jiajian Li
Yunbo Wang' Xiaokang Yang
Abstract
2020; Kostrikov al., 2022). However, key challenge in offline RL lies in the inherent incompleteness of fixed datasets, which can result in distributional shifts from the target environment. This issue, compounded by the well- documented overestimation bias value approximations, creates significant difficulties for current offline RL meth- ods (Yu et al., 2022: Zang et al., 2023; Lu et al., 2023).
Offline reinforcement learning (RL) enables pol- icy optimization using static datasets, avoiding the risks and costs of extensive real-world explo- ration. However, it struggles with suboptimal offline behaviors and inaccurate value estimation due to the lack of environmental interaction. We present Video-Enhanced Offline RL (VeoRL), a model-based method that constructs interactive world model from diverse, unlabeled video data readily available online. Leveraging model-based behavior guidance, our approach transfers com- monsense knowledge of control policy and physi- cal dynamics from natural videos to the RI. agent within the target domain. VeoRL achieves sub- stantial performance gains (over 100% in some cases) across visual control tasks in robotic ma- nipulation, autonomous driving, and open-world video games. Project page: https: /os:://peee /ppamtt. github. io /eooL.. taub.ii.....
The core insight of VeoRI. is to leverage unlabeled Inter- net videos to broaden the agent's 's understanding of human behaviors and the real physical world, going beyond the limited scope of fixed, predefined datasets. Notably, the vast volume of online video data possesses several key ad- vantages: (i) It can be acquired at a significantly lower cost compared to data collected through real interactions with the target environments. (ii) It implicitly contains demon- stration policies that, while potentially suboptimal, can still be beneficial for current visual control tasks.
Nonetheless, the online videos typically lack action labels or reward function annotations and may present remark- able data distributional shifts compared to the target RI. domain. These properties introduce unique challenges when using the auxiliary videos for policy optimization. Existing RL pretraining methods generally operate within the raw action space (e.g., through behavior cloning) perform action-free model warmup (e.g., R3M (Nair et al., 2022) and APV (Seo et al., 2022)), making them ill-suited for transferring underlying behaviors from unlabeled videos.
1. Introduction
When humans tackle long-term control tasks they are unfa- miliar with, they typically start by watching instructional videos on platforms like YouTube to develop an abstract understanding of the task, before transitioning to hands-on on practice in a real environment to refine their skills. Can machines replicate this learning process? In this work, we introduce video-enhanced offline reinforcement learning (VcoRL) to explore this problem.
To address these challenges, we begin by constructing a discrete latent action space recover the underlying, unob- servable control policies from raw video data (see Figure la). Specifically, we model the temporal events in the video as Markov decision process with a continuous state space and a discrete action space. To optimize finite set of "behavior abstractions", we apply vector quantization techniques and use the learned behavior vectors drive the training of a video prediction model. Empirical results demonstrate that the latent behaviors learned through this approach exhibit strong semantic alignment with the actual actions observed in the target environment.
Offline reinforcement learning refers to the scenario where an RL system is restricted to learning from a limited, static dataset. This approach offers several benefits, as it allows agents to learn from existing data, reducing the need for costly and potentially unsafe real-world engagement (Fuji- moto al., 2019; Kumar et al., 2019; 2020; Levine al.,
'MoE Key Lab of Artificial Intelligence, Al Institute, Shanghai Jiao Tong University. Correspondence to: Yunbo Wang <yun- bow @sw.t.cdu.cn....
VeoRL is essentially a model-based RL algorithm, where the agent interacts with a world model that generates state transitions and reward feedback. We train a world model comprising two state transition branches: One branch pre-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

physical Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
b
Meta-Worde
Place
Episode Retur
Pick
Generate
Grip
Rotate
1000-
Push
Pull
500
Press
CARLA
Real-world Videos
Behavior Abstraction
Episode Return
40
Pick + Place
20
Train
Guided toward the underlying control behaviors behind the videos
MineDojo
0.6
Success Rate
0.4
DreamerV2
0.2
LOMPO
Offline Datasets
Agent
VeoRL
0.0
Figure 1. Overview of the training setup of VeoRL that leads to improved offfine RL performance. We extract latent behavior abstractions from task-agnostic, unannotated natural video data to enrich the world model's commonsense understanding world. By interacting with this world model, the agent performs policy optimization guided explicitly by the latent policies learned from the natural videos. b, Overall performance of VeoRL. It demonstrates signiticant improvements over existing offine RL methods across different visual control benchmarks, including Meta- World robotic manipulation, CARLA autonomous driving, and Minecraft open open- world video games. We present the average performance on all tasks on each benchmark.
diets future state evolution based on the agent's 's real actions (trunk net), while the other branch predicts longer-term envi- ronmental feedback derived from latent behaviors (plan net). We further introduce an intrinsic reward that encourages the state-action rollouts along the trunk net to progressively align with the long-term future state predicted by the plan net. This approach connects the target policy with the be- havior abstractions learned from auxiliary videos, while mitigating the overestimation bias that arises from solely relying environmental rewards to train the value network.
havioral outcomes, model-based RL approaches (Ha & Schmidhuber, 2018; Hafner et al., 2019; 2020) typically of- fer better sample efficiency than model-free methods (Yarats et al., 202 Kostrikov et al., 2021; Laskin et al., 2020). World Models (Ha & Schmidhuber, 2018) were introduced to first learn compressed latent representations of the en- vironment in a self-supervised manner, and then use these latent states train the agent. Dreamer (Hafner et al., 2020) and DreamerV2 (Hafner al., 2021) optimize behavior by predicting and then acting upon expected values over the latent states generated by the Recurrent State-Space Space Model. Recently, research on goal-conditioned RL using hierarchi- cal world model architectures (Veeriah et al., 2021; Rao et al., 2022; Hafner al., 2022; Gumbsch et al., 2023; Park et al., 2024) has shown how decomposing tasks into sub- goals can facilitate efficient decision-making making in complex environments. However, none of these methods address the challenges of offline RI or facilitate unsupervised policy transfer from enriched out-of-domain video data.
We evaluate VeoRL on a diverse set of visual control benchmarks. For the Meta- World robotic manipulation tasks, we use the BridgeData-V2 dataset (Walke et al., 2023) as the source of auxiliary real-world videos. For the CARLA autonomous driving tasks, we employ the NuScenes dataset (Caesar et al., 2019), collection of 1,000 diverse real-world driving scenes, the auxiliary source domain. For the MineDojo open-world gaming environ- ment, where the agent must navigate a vast state space, we use online Minecraft videos created by human players to provide unlabeled video demonstrations. As summarized Figure 1b, VeoRL achieves substantial performance im- provements over existing RL approaches.
Offline RL. Offline RL offers a cost-effective alternative by reducing interactions with online environments. Previous research has primarily focused on addressing the challenge of value overestimation, falling into two broad categories: model-free methods (Cen et al., 2024; Wu et al., 2019; Ku- mar et al., 2020; Osa & Harada, 2024; Li et al., 2023; Fu- jimoto & Gu, 2021; Hong et al., 2024) and model-based approaches (Chen et al., 2023; Yu et al., 2021; Hatch et al.,
2. Related Work
Model-based visual RI. For visual control tasks, agents learn control policies from high- nigh-dimensional visual inputs.
2023; Mazoure et al., 2023; Hatch et al., 2022; Sun et al.,
By explicitly modeling state transitions and predicting be-
2023; Rigter et al., 2022; Yang al., 2024b). For example,
2

---

Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
CQL (Kumar et al., 2020) introduces a model-free offline RL method that mitigates the value overestimation bias by learning a conservative state-action value function. This method can be seamlessly integrated with existing visual RL. approaches. In the context of offline visual RI, an ad- ditional challenge is effectively handling high-dimensional visual inputs while preventing overfitting during the repre- sentation learning of latent states (Shah et al.. 2022; Tian et al., 2020; Yadav et al., 2023; Nair et al., 2022; Wang et al., 2024). Specifically, LOMPO (Rafailov et al., 2021) presents model-based offline visual RI. approach that ex- plicitly manages the dynamics uncertainty by incorporating a reward penalty during world model training. In this work, we compare our approach with the model-based LOMPO method and the model-free CQL method integrated into the visual RL backbone of DrQ-V2 (Yarats et al., 2021a).
challenges arise in this context: (i) The auxiliary videos generally lack action labels and reward function annotations; (ii) There exist inherent distributional shifts between B and D To address these challenges, VeoRL employs a three-stage approach (as detailed in Algorithm 1):
1. We introduce a behavior abstraction network designed to extract underlying control behaviors directly from the raw observations of unlabeled videos.
2. We design a hierarchical world model to capture state transitions in two ways: through real actions using a trunk net and through latent behaviors using a plan net.
3. We propose a model-based RL method that optimizes the target policy over the trunk net transitions while being guided by latent behavior rollouts from the plan net.
Video-enhanced RL. In recent years, there has been grow- ing interest in harnessing auxiliary video data to enhance RL. such data is more accessible and affordable in real- world settings. We can broadly categorize these approaches into two groups The first group focuses on using videos to improve agent performance through representation learn- ing (Seo et al., 2022; Ma et al., 2023; Wu et al., 2023). For instance, APV (Seo et al., 2022) pre-trains an action- free video prediction model using videos and then finetunes the agent with these pretrained representations. VIP (Ma et al., 2023) enhances generalization to downstream offline tasks by leveraging self-supervised visual representations. The second group focuses on learning transferable behavior policies from the source video domain (Torabi et al., 2018: Chang et al., 2022; Baker et al., 2022; Schmeckpeper et al., 2021; Ye et al., 2022). For example, the VPT model (Baker et al., 2022) trains an inverse dynamies model with small set of labeled auxiliary videos and then assigns pscudo- actions to unlabeled video data. This method requires strong consistency across the source to target domains. Another line of work focuses on learning interactive world mod- els from the unlabeled target-domain videos to generate playable virtual environments (Valevski et al., 2024; Yang et al., 2024a). While promising, these methods highlight the potential of interactive video generation, without explicitly using them for downstream control tasks. Besides, these approaches have yet to be formally published. In summary, unlike VeoRI., previous video-enhanced approaches typi- cally require the source video set to be either annotated with action/reward labels or derived from the same domain as the target environment for policy deployment, or not directly designed for sequential dicision-making tasks.
3.1. Latent Behavior Abstraction
To address the challenge of video data lacking action la- bels, previous methods (Baker et al., 2022; Zhang et al., 2022) typically focus on using target domain observations with real action labels (i.e., the offline RL dataset) to train an inverse dynamics model, which is then used to assign pseudo-action labels to the source domain auxiliary videos. However, this approach assumes that the source and target domains share the same action space, leading to two poten- tial issues: First, action predictions for source video data may suffer from domain distribution discrepancies. Second, the learned actions are limited in their ability to provide high-level behavior guidance for subsequent policy learn- ing. Instead, we propose a fully unsupervised approach for behavior inference, leveraging a compact categorical latent action space significantly smaller than the raw con- tinuous action space. This approach avoids the challenge of real action prediction, generating higher-level behavior representations that are distinct from the original actions for low-level visual control.
To obtain a discretized latent action space, we design a behavior abstraction network (BAN) based on vector quan- tization (Gray, 1984). As shown in Figure 2a, this module is trained within a one-step video prediction framework, where its output is used to drive recurrent state-space model (RSSM) (Hafner et al., 2019) for forward dynamics modeling. The learning process begins by taking two obser- vations 1o from the target dataset. We extract image encodings from these observations using CNN model (Eg) and concatenate them into latent vector, e which captures the temporal difference between consecu- tive video frames. We maintain learnable latent codebook R k D consisting of K continuous vectors Ck=l....KK E RD, each representing a distinct behavior abstraction. Given [e eerr BAN identifies the nearest neighbor e; from the set { and outputs this vector as the chosen latent
etar]
E RRD
3. Method
We aim to enhance the performance of offline visual RL trained on an offline target dataset a by leveraging task- agnostic video data from a source domain 1es0 Two key
3

---

Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
dar t-1
Latent codebook
8t
(cx)k-1
Do
Dg
8 B
Policy network T
Value network Vi+
ct-1
FC
Quantizer
81-1
81
BAN
BAN
MMD
Trunk Net
Plan Net
Eg
Eg
Fgg
F.g
S1+1
St+1
8+1
C-1
05
far
O2 TTC
Intrinsic
Pc+1
T++1
reward
Figure 2. Model architecture. We construct discrete, high-level latent action space by training the BAN, enabling forward dynamics modeling independent of real actions. b, The visualization of model-basee actor-critic learning at single rollout step. We leverage FBc replay the video-informed latent hehaviors, serving the inputs of the and critic for producing goal-conditioned policies and value estimations, as well as the plan net for generating a long-term state rollout.
behavior Q corresponding to the transition from o to ofth Similar to VQ-VAE (Van Den Oord et al., 2017), the optimization objective of the vector quantization process
the outputs of the image encoder to align visual embeddings across different domains.
can be formulated as
3.2. Two-Stream World Model
lvo
|
uu o 15 c: 112
sg -g(()
We design a two-stream interactive world model consist- ing of: (i) a Trunk net that captures future state transitions driven by real actions and the associated envirommental re- wards, (s1, a1, 71. $2, a2 T2....) Trunk(o1: TT)) and (ii) Plan net that learns to predict future trajectories based high-level behavior abstractions, (5), ãi, 52, a2,...) Plan (01: BAN). Here, = [h,, 21] and 3, = [hg Ed). where (/4, ha) and (z,, že) represent the deterministic and stochastic components of the predicted states, respectively. Specifically, the plan net includes a behavior cloning mod- ule FBC that predicts āe based solely on 3,. The learning target for FBC is the latent behavior inferred by BAN from the state transition pair (5t, 5141). The world model can be formulated as
codebook loss
commitment loss
where sg(.) denotes gradient stopping. The codebook loss encourages the selected code to closer to the input, while the commitment loss facilitates gradient backpropagation into the encoder. This helps stabilize the training process by preventing the image embedding from switching indiscrimi- nately between different codes in the codebook.
As illustrated in Figure 2a, the selected latent behavior serves as input to a state transition model referred to as the plan net. We backpropagate the video prediction error to the BAN to refine the selection of latent behaviors, ensuring that they better predict subsequent frames. The overall objective function of BAN can be formulated fbn εvg + al plan. where fpam denotes the loss function associated with the plan net, which will be detailed subsequently.
Trunk Net:
Plan Net:
h4 CRU($L-1,,1-12
h CRU(S--1,
~ p(he;
5. p(he; 0)
To leverage the BAN trained on B as an estimator of latent behaviors for OOD videos from gare we adopt the maximum mean discrepancy (MMD) loss (Borgwardt al., 2006), method widely used in domain adaptation (Huang et al., 2006; Li et al., 2013; Gong et al., 2013). The MMD loss is defined as LMDD
2 ~ g(he, ee;
o ~ qlhe, ec;
ô, ~ p(s(; 0)
6, ~ p( B)
nr ~ p(s:; 0)
a. = (Bc
(2)
where (z/, x) indicate the posteriors inferred from the cur- rent image observation and (ôt, ō.) are the reconstructed
e e tor 1 and is applied to

---

Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
images. 0 represents the combined parameters of the trunk net, while 0 represents the combined parameters of the plan net. The trunk net and plan net share the same model param- eters for the image encoder and decoder, but have separate parameters for the remaining parts of the model. The trunk net is trained exclusively on the offline RL dataset B The objective function as follows:
the value network are additionally conditioned the es- timated latent behavior to incorporate high-level control guidance, such that (T | St, @e) and vol (st, az), where ae = Fsc( This modification allows for more informed decision-making by integrating abstract behavioral patterns from the auxiliary videos into the learning process.
The learning approach in Eq. (4) enables the plan net to predict long-term states that can serve as decision-making subgoals. Accordingly, we propose goal-conditioned in- trinsic reward that quantifies the similarity between short- term state transitions induced by real actions and long-term state transitions driven by high-level latent behaviors:
frrnk (0) g
Iu s.)) ll | (t)
image recumstructoon
reward prediction
(3)
+ KT[[(=1 he, 64) p(=e h+)].
KL divergence
The plan net is trained using trajectories from the tar- get dataset 8 and unlabeled videos from the source dataset B We adopt a specialized training strategy to ensure that the plan net models meaningful long-term state transitions. Instead of focusing on one-step future predictions, we consider the state transition "shortcuts" that occur when BAN predicts different a compared to the previous step āz-1- Specifically, starting with an original trajectory (01, 02, a...... 0r-1, or) sampled from the dataset and precomputed by BAN, we retain only those state-action pairs (ā,, 01+1) where a change in latent behavior is detected, i.e., 0g + āz-1- This process generates a refined trajectory that captures significant behavioral changes over time, represented (o),. Oj O/2: O221... 1 1: where Oj 01. These refined trajectories are subsequently used to optimize the plan network through the following objective function:
Ft- - || 31112
8, ~ Trunk
(5)
8, ~ Plan(s (
The policy network is trained to output actions that lead to states that maximize the value network's 's output, while the value network seeks to accurately estimate the discounted future returns achieved by the policy network starting from each imagined state. Specifically, the value target V/A for the value network incorporates a weighted average of reward information over n-step future horizon, which is defined recursively as follows:
V = (fr + wF))
+y (1- A) vo + AVA1, if t< L, (6) (x()))
if t t= L.
where À is set to 0.95 for considering more on long horizon targets. y is a discount factor set to 0.99 in practice. Given a trajectory of model states and rewards, the value network be trained to regress V/A and the policy network Tg is optimized with entropy regularization:
pplan (0) ETEBur B
In p(o, 5t) - In | st)
31 image recornstruction
beluvior cloning
+ [L he,ea) || he].
KL divergence
(4)
This approach enables the plan net to capture significant behavior shifts, enhancing its ability to predict impactful long-term transitions. It provides supplementary informa- tion that complements short-term, real action-based state transitions during the subsequent policy optimization stage.
((b) = Epo.re Po
(Uo (sr, -
2
(()) = Epo po
(1-p) mll aa |3,, at.
(7)
Dynamics BP
Fntmpy
3.3. Model-Based Policy Learning
-p In -pInpo (āt
I 1t, as )sV(ss - up (8),
We perform temporal difference learning using a prob- abilistic policy network TT(+) and a deterministic value network (-). As shown in Figure 2b, these model components are optimized over two-stream imagined trajectories generated by the trunk net and the plan net, i.e., (81, 51, âi, ā1, P1, 82, 82, 02, ā2, f2...., .1,, 81, āz, ā., fL), where 31 is derived from an observation 01 randomly sampled from the target set Bar.
REINFORCE
For the first term in the actor loss, we employ the straight- through gradients (Bengio et al., 2013) backpropagate the value targets of the sampled actions and state sequences di- rectly through the learned dynamics. In continuous control tasks like Meta- World and CARLA, we set = 0. In dis- crete control tasks, we set = 1 and observe that integrating REINFORCE gradients yields improved results.
Unlike previous methods, both the policy network and
5

---

Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
Drawer Open
Hand Press
7666
Episade Return
Episode Return
Episode Return
1200
1550-
250-
-1000
Handle Pull
Plate Slide
Source domain
Target domain
COL DreamerVz
LOMPO VeeRL
2750
Figure 4. Experiments of autonomous driving. a. Showcases of the source NuScenes and target CARLA datasets. b, Performance comparison CARLA, measured by averaged episode returns.
Episode Retumn
Epicode Return
1250-
2ced-
tzee-
3380
150-
1040
VPT (Baker et al., 2022) particularly designed for Minecraft tasks, so we only use as baseline model on the Mojo environment. Further details of the com- pared models can be found Appendix Section D.
Coffee Push
Button Press
800
-00
Episode Retuin
Episode Return
009
400
4.1. Main Results
233
Meta-World robotic manipulation. For the Meta-World offline RL tasks, we use the BridgeData- V2 dataset (Walke et al., 2023) with real-world videos as the unlabeled source domain. We conduct all tasks with 3 random seeds and report the mean results and standard deviations over 50 episodes. As shown in Figure 3, our approach achieves state- of-the-art performance in episodic returns over all six tasks in this domain. Specifically, VeoRI. outperforms Dream- erV2 by large margin in Drawer Open (1168 + 1953), Handle Press (1201 2650) and Handle Pull (203 + 1365). Compared APV and VIP, which also employ video-enhanced pretraining, VeoRI. demonstrates signifi- cant advantage by effectively exploiting and transferring the underlying control behaviors behind these videos. Further- more, we provide additional model comparisons measured by success rate Appendix Table 3, where VeoRL consis- tently achieves the best performance.
COL
DreamerV2
APV
MI
LOMPO
VeoRL
Figure 3. Performance on Meta Woold robotic manipulation tasks in episode return. Error bars indicate standard deviation across the 50 evaluation episodes, with 3 random training seeds.
4. Experiments
In this section, we present (i) quantitative comparisons with existing RL methods on a diverse set of visual control bench- marks, (ii) offline-to-online transfer learning results on novel interactive control tasks, (iii) ablation studies for each pro- posed model component in VeoRI,, and (iv) hyperparameter analyses along with visualizations of the learned latent be- havior abstractions. The hyperparameters for implementing VeoRL are provided Appendix Table 2.
CARLA autonomous driving. We employ the large-scale NuScenes dataset (Caesar et al., 2019) with 1,000 diverse real-world driving scenes to improve the autonomous driv- ing performance on the CARLA benchmark. Figure shows the performance of our model compared to different base- lines across three random training seeds. As we can see, VeoRI. achieves notably the best mean results with the small- est standard deviations, outperforming DreamerV2 by as much as 350%. Notably, although the VIP method also in- volves pretraining on the same NuScenes dataset, its policy transfer approach (based on a self-supervised value-function objective) struggles to achieve satisfactory results in this driving task. We exclude the results from APV in Figure to provide clearer comparison of the other approaches, as it yields average episode return of approximately -500.
To demonstrate the capability VeoRL, we conduct ex- periments on three visual RL environments, including Meta-World (Yu et al., 2019), CARLA (Dosovitskiy al., 2017), and MineDojo (Fan et al., 2022) Like D4RL (Fu et al., 2020), collect the offline RL datasets of medium- quality trajectories using a partially-traine -raineed DreamerV2 agent (Hafner et al., 2020). Please refer to Appendix Section for more details of the benchmarks.
We compare VeoRL with: (1) DrQ+CQL (Kumar et al., 2020) and LOMPO (Rafailov et al., 2021) are specifically designed for offline RL tasks. (2) DreamerV2 (Hafner et al., 2020) serves as our primary baseline. It follows a similar model-based RL framework but lacks the video- enhancement method. (3) Like our approach, APV (Seo et al., 2022) and VIP (Ma et al., 2023) exploit auxil- iary videos to improve model performance. (4) Finally,
MineDojo open-world games. MineDojo an open- world 3D environment with a vast state space for the agent
6

---

Video-Enhanced Offlin Reinforcement Learning: A Model-Based Approach
Source domain
Target domain
b
Harvest log plains
Harvest water with bucket
Harvest sand
Success Rate
Success Rate
0.5
0.3
Success
0.4
04
0.3
0.0
0.2
0.1-
0.0
0.0
DreamerV2
LOMPO
VPT
VeoRL
Figure Experiments of the MineDojo 3D navigation and control tasks. a, Showcases of source online vidcos and target offline datasets. b, Performance comparison in success rate.
Meta-World (Soccer)
CARLA (Night)
MineDojo (Cobblestone)
1.0
DreamerV2
75
DreamerV2
Episode Return
250
VIP
Ours
Episode Return
50
Success Rate
0.8
VPT
Ours
200
150
0.4
100
-25
DreamerV2
-50
LOMPO
0.2
-75
Ours
100
60
50
100
150
Environment Steps (x10*)
Environment Steps (x10²)
Environment Steps (x10*)
Figure 6. Performance of oflline-to-online finetuning novel tasks. By conducting offline pretraining. VcoRI. demonstrates improved training efficiency and final performance when adapting to unseen, more challenging visual control tasks. Notably, we train the DreamerV2 model under same offline-to-one setup. The results are averaged random training seeds.
to explore, making effective video demonstrations particu- larly essential the target offline learning scenarios. As shown in Figure 5a, we use online videos of human play- the auxiliary source video set. These videos contain random Minecraft tasks that may differ significantly in gam- ing tactics from those for the target tasks. The results in Figure 5b show that VeoRL outperforms the other baseline models. Specifically, it improves upon DreamerV2 by 250 % (0.10 - 0.35) Harvest log in plains, by approximately 130% (0.36 - 0.83) in Harvest water with bucket, and by around 100% (0.27 0.55) in Harvest sand.
a two-stage Dacision-making process where the agent must fetch ball and then push it to goal position. For CARLA, the offline RL agents are pretrained using the Day-mode dataset and then finetuned in the Night mode driving scenarios. For MineDojo, the agents pretrained fixed dataset for the Harvest sand task and finetuned on the Harvest cobblestone with wooden pickaxe task. As shown Figure 6, VeoRL demonstrates superior training efficiency compared DreamerV2, which follows the same offline-to-online domain adaptation setup, highlighting the effectiveness of our offline training approach.
Offinne-toonllne transfer learning. To evaluate the gen- eralizability of our model, we finetune the pretrained offline RL agents in wnseen visual control tasks through online interactions with the environment. We select a more chal- lenging Soccer task as our offline-to-onlin transfer target on the Meta- World benchmark. This task typically involves
4.2. A Ablation Studies
we will discuss in Section VeoRI. introduces three key innovations to the model-based actor-critic learning frame- work of DreamerV2 (Hafner et al., 2021): (i) It uses latent behaviors as inputs to the policy network (iee, actor); (ii) It

---

Vidco-Enhanced Offline Reinforcement Learning: A Model-Based Approach
Meta-Worid
CARLA
MineDojo
3000
0.9-
Episode Return
2500
Episode Return
Success Rate
9.6
2000
20-
1500
0.4
1000
02
0.0
Ours
Ours w/o latent behavior
Ours w/o intrinsic reward
Ours w/o videos
DreamerV2
Figure 7. Ablation studies of VeoRL. We show the impact of latent behavior guidance (yellow) and the intrinsic reward (green). We also compare VeoRI, with baseline model that does not use any source video data throughout training process (blue).
Meta-World World robotic manipulation tasks in success rate and episode return. We compute the mean and standard deviation of 50 episodes over 3 random training seeds.
Table Performance comparison with V3eamerV3 Meta-
Methods
DreamerV2
DreamerV3
VeoRL(DV2)
VeoRL(DV3)
Success Rate
Drawer Open Handle Press
0.18 + 0.04 0.33 + 0.11
0,00 + 0.00 0.05 + 0.05
0.70 + 0.07
0.55 + 0.15 0.35 + 0.15
0.60 = 0.12
Episode Return
Drawer Open
1168.35 + 59.55
674.55 + 79.04
1953.60 + 121.48 2650.90 + 619.60
1393.50 = 122.50 1360.15 = 547.85
Handle Press
1201.75 + 422.10
257.85 +: 247.05
employs latent behaviors to compute an intrinsic reward for training the value network (i.e., critic), and (iii) It trains the plan net in the world model using the auxiliary video set and the offline RL dataset jointly. Accordingly, we conduct ablation studies to evaluate the effectiveness of individual techniques. Figure 7 provides corresponding results on Merl-World (Handle Press), CARLA and MineDojo (Har- vest water with bucket). As illustrated by the yellow and green bars, excluding the latent behavior input in the policy network or removing its influence from the value network's training objective both result in significant performance drops in VcoRL. These results highlight the crucial role of latent behaviors as effective high-level guidance in policy learning. The blue bar illustrates that training all network components solely on the target offline dataset (without ac- cess to auxiliary videos) leads a significant performance decline. This highlights that our eiienned method effectively extracts valuable prior knowledge of underlying expert behaviors from the unlabeled videos.
our experiment results in Table demonstrate that un- derperforms in offline RI settings without hyperparameter tuning compared DreamerV2. (ūi) Backbone consistency, LAMPO, key baseline in our work, uses Dreamer V2 as its backbone. To ensure direct and fair comparisons be- tween our method and LAMPO, we maintain consistency by adopting the same backbone. Notably, we also conduct additional experiments on Meta- Meta-World by integrating Ve- oRL with DreamerV3. The results demonstrate that approach outperforms vanilla DreamerV3, showcasing its ability to seamlessly integrate with different architectures.
4.4. Analyses of Latent Actions
To illustrate the correspondence between latent behaviors and real actions, evaluate the distribution of the learned latent behavior abstractions in Meta-World Coffee Push in Figure 8. VeoRI,, we roll out the trunk net and plan net simultaneously for model-based policy learning. Start- ing from the same input state 8t, we obtain two future tra- jectories of real actions a1:+ and corresponding latent behaviors 0u++H. To show the correspondence between them, we specifically select trajectories of four consecu- tive real actions ar:+4 with time-invariant latent behaviors ā 0 and then use the t-SNE algorithm (Van der Maaten & Hinton, 2008) to project a1s+4 into a 2D plane.
4.3. Comparison with DreamerV3
The decision to use DreamerV2 as the backbone is driven by two key considerations: (i) Offline setting performance. While DreamerV3 (Hafner et al., 2023) achieves strong per- formance across diverse tasks with fixed hyperparameters,
8

---

Video-Enhanced Offlie Reinforcement Learning: A Model-Based Approach
Meta- Meta-World World (target domain)
BridgeData
'reach'
"push'
BridgeData
Figure Correspondence between latent behaviors and real actions in Meta-World. During model-based state rollouts, starting from the same state 8, of the target domain, VeoRL simultaneously generates real action trajectories 4 and latent behaviors a We consider trajectories of four consccutive real actions 04:444, where the latent behaviors are time-invariant, i.e., āe 0t:4444 We then use t-SNE to project ae:++4 into 2D plane. To distinguish between different latent behaviors, we assign distinct colors to the points, visualizing 10 latent hchaviors for clarity. We present BridgeData examples where VcoRI. generates the corresponding latent hchaviors.
Different colors assigned the points indicate distinct la- tent behaviors. From Figure 8, we observe that the latent behaviors align well with distinct clusters of real action se- quences, demonstrating the high representational fidelity of the learned latent action space. These latent behaviors effectively capture semantic information from online videos and can interpreted as fundamental operational skills, which can then be used as policy guidance for the RL agent. Furthermore, we provide the ablation results of varying the number of discrete latent behavior in Appendix Figure 9.
offline visual RL. VeoRL. constructs world model with two state transition branches: one conditioned on latent behav- iors extracted from video data and the other on the agent's s real actions. This dual-branch structure allows the agent to effectively learn control policies by aligning state rollouts from two branches, thereby transferring knowledge from the diverse, out-of-domain video data to the RL agents. Our results demonstrate that VeoRL. significantly outperforms existing offline RL methods across wide range of chal- lenging visual control tasks, including robotic manipulation, autonomous driving, and open-world video games. The in- tegration of video data not only boosts the agent's ability to generalize across different domains but also accelerates the learning process by incorporating a broader understanding of the physical world and control policies.
4.5. Further Results and Discussion
Please refer to Appendices F-I for additional experimental results, including the quality analysis of source videos and target domain trajectories. the hyperparameter analysis of latent action space, the generalizability across tasks, and the impact MMD domain adaptation.
The limitation of this work is the computational overhead. The training process of VeoRI. involves extracting latent behaviors and optimizing a world model with two state transition branches. Despite the performance gain, the dual- branch setup could increase the computational complex- ity and memory requirements, making the approach more resource-intensive than simpler RL algorithms.
5. Conclusions and Limitations
In this paper, we introduced VeoRL, a novel approach that leverages freely available, unlabeled video data to enhance

---

Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
Acknowledgments
Dosovitskiy, A., Ros, G., Codevilla, E., Lopez, A., and Koltun, V. CARLA: An open urban driving simulator. In CoRL, pp. 1-16, 2017.
This work was supported by the National Natural Science Foundation of China (62250062), the Smart Grid National Science and Technology Major Project (2024ZD0801200) the Shanghai Municipal Science and Technology Major Project 2221SHZDZX0100 and the Fundamental Re- search Funds for the Central Universities.
Fan, L., Wang. G., Jiang, Y., Mandlekar, A., Yang. Y., Zhu, H., Tang. A., Huang, D.-A., Zhu, Y., and Anandkumar, A. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In NeurlPS, volume 35, pp. 18343-18368 2022.
Impact Statement
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
VeoRL represents significant advancement in offline RL by leveraging unlabeled video data to enhance agent policy learning and generalization. It enables agents to be trained on vast amounts of real-world data without the need for labor-intensive annotations, making it more accessible for researchers, developers, and practitioners. VeoRI. brings RL closer to how humans learn by observing and interacting with the world. Additionally, by aligning RL with human learning processes, enables the development of more intuitive AI systems.
Fujimoto, S. and Gu, S. minimalist approach to offline reinforcement learning. NeurlPS, 34:20132-20145 2021.
Fujimoto, S., Meger, D., and Precup. D. Off-policy deep reinforcement learning without exploration. In ICML, pp. 2052-2062, 2019.
Gong, B., Grauman, K., and Sha, F. Connecting the dots with landmarks: Discriminatively learning domain- invariant features for unsupervised domain adaptation. In ICML, pp. 222-230, 2013.
References
Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, J. Video pretraining (VPTy: Learning to act by watching unlabeled online videos. In NeurlPS, volume 35, pp- 24639-24654, 2022.
Gray, R. Vector quantization. IEEE Assp Magazine, 1(2): 4-29, 1984.
Gumbsch, C., Sajid, N., Martius, G., and Butz, M. V. Learn ing hierarchical world models with adaptive temporal abstractions from discrete latent dynamics. In ICLR, 2023.
Bengio, Y., Léonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for con- ditional computation. arXiv preprint arXiv:1308.3432. 2013.
Ha, D. and Schmidhuber, J. Recurrent world models facili- tate policy evolution. In NeurIPS, 2018.
Borgwardt, K. M., Gretton, A., Rasch, M. J., Kriegel, H.-P., Schölkopf, B., and Smola, J. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 22(14):e49-e5 e57, 2006.
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D.,
Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels. In ICMI., pp. 2555-2565. 2019.
Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination. In ICLR, 2020.
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q. Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. nuscenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv: 1903. 1027, 2019.
Hafner, D., Lillicrap, T., Norouzi, M., and Ba, J. Mastering atari with discrete world models. In ICLR, 2021.
Cen, Z., Liu, Z., Wang, Z., Yao, Y., Lam, H., and Zhao, D. Learning from sparse offline datasets via conservative density estimation. arXiv preprint arXiv:2401.06 108819, 2024.
Hafner, D.. Lee, K.-.,, Fischer, I., and Abbeel, Deep hierarchical planning from pixels. In NeurlPS, volume 35, pp. 26091-26100 2022.
Chang, M., Gupta, A., and Gupta, S. Learning value func- tions from undirected state-only experience. In ICLR, 2022.
Hafner, D., Pasukonis. J., Ba, J., and Lillicrap. T. Mastering diverse domains through world models. arXiv preprint Ttl::2301.41000 2023.
Chen, C., Zhang, Y., Liu, X., and Coates, M. Bidirectional learning for offline model-based based biological sequence de- sign. In ICML., pp. 5351-5366 2023.
Hatch, K., Yu, T., Rafailov, R., and Finn, C. Example-based offline reinforcement learning without rewards. Proceed- ings of Machine Learning Research vol, 1 144:1-1 2022.
10

---

Vidco-Enhanced Offlne Reinforcement Learning: A Model-Based Approach
Hatch, K. B., Eysenbach, B., Rafailov, R., Yu, T., Salakhut- dinov, R., Levine, S., and Finn, C. Contrastive example- based control. In L4DC, pp. 155-169, 2023.
Mazoure, B., Eysenbach, B., Nachum, O., and Tompson, J. Contrastive value learning: Implicit models for simple offline RL. In CoRL, pp. 1257-1267, 2023.
Hong, J., Dragan, A., and Levine, Offline rl with ob- servation histories: Analyzing and improving sample complexity. In ICLR, 2024.
Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta, A. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203 /2601, 2022.
Huang. J., Gretton, A., Borgwardt, K., Schölkopf, B., and Smola, A. Correcting sample selection bias by unlabeled data. In NeurIPS, volume 19, 2006.
Nottingham, K., Ammanabrolu, P., Suhr, A., Choi, Y., Ha- jishirzi, H., Singh, S., and Fox, R. Do embodied agents dream of pixelated sheep: Embodied decision making using language guided world modelling. In ICML, pp. 26311 2023.
Kostrikov, I,, Yarats, D., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In ICLR, 2021.
Osa, T. and Harada, T. Discovering multiple solutions from a single task in offline reinforcement learning. In ICML, 2024.
Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit Q-learning. ICLR, 2022.
Kumar, A., Fu, J. Soh, M., Tucker, G., and Levine, S. Stabilizing off-policy q-learning via bootstrapping error reduction. In NeurlPS, volume 32, 2019.
Park, S., Ghosh, D., Eysenbach, B., and Levine, S. HIQL: Offline goal-conditioned RL with latent states as actions. NeurIPS, volume 36, 2024.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conser- vative q-learning for offline reinforcement learning. In NeurlPS, volume 33, pp. 1179-112 1191, 2020.
Peng, X. B., Kumar, A., Zhang, G., and Levine, Advantage-weighie regression: Simple and scalable off off-policy reinforcement learning. arXiv preprint arXiv: 191001/900.0077,00 2019.
Langley, P. Crafting papers machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (1CM), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Rafailov, R., Yu, T., Rajeswaran, A., and Finn, Offline reinforcement learning from images with latent space models. In L4DC, pp. 1154-1168, 2021.
Laskin, M., Lee, K., Stooke, A., Pinto, L.., Abbeel, P., and Srinivas, A. Reinforcement learning with augmented data. In NeurlPS, 2020.
Rao, D., Sadeghi, F., Hasenclever, L., Wulfmeier, M., Zam- belli, M., Vezzani, G., Tirumala, D., Aytar, Y., Merel, J., Heess, N., al. Learning transferable motor skills with hierarchical latent mixture policies. In ICLR, 2022.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.066 2020.
Rigter, M., Lacerda, B., and Hawes, N. Rambo-RLe Robust adversarial model-based offline reinforcement learning. In NeurIPS, volume 35, pp. 16082-16097. 2022
Li, A., Boots, B., and Cheng, C.-A. Mahalo: Unifying offline reinforcement learning and imitation learning from observations. In ICML, pp. 19360-19384, 2023.
Schmeckpeper, K., Rybkin, O., Danilidis, K., Levine, S., and Finn, C. Reinforcement learning with videos: Com- bining offline observations with interaction. In CoRL, volume 155, pp. 339-355, 2021.
Li, W., Duan, L., Xu, D., and Tsang, I. W. Learning with augmented features for supervised and semi-supervised heterogeneous domain adaptation. IEEE Transactions on Pattern analysis and machine intelligence, 36(6): 1134 1148, 2013.
Schulman, J., Wolski, F,, Dhariwal, P., Radford, A., and
Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Lu, C., Ball, P. J., Rudner, T. G., Parker-Holder, J., Osborne, M. A., and Teh, Y. W. Challenges and opportunities in offline reinforcement learning from visual observations. Transactions on Machine Learning Research, 2023.
Seo, Y., Lee, K., James, S. L., and Abbeel, P. Reinforcement learning with action-free pre-training from videos. In ICML, pp. 19561-19579. PMLR, 2022.
Ma, Y. J., Sodhani, S., Jayaraman, D., Bastani, O.,, Kumar, V., and Zhang, A. Vip: Towards universal visual reward and representation via value- iipllci pre-trate vre-training. In ICIR, 2023.
Shah, D., Bhorkar, A., Leen, H., Kostrikov, I., Rhinehart, N., and Levine, S. Offline reinforcement learning for visual navigation, arXiv preprint arXiv:22 1a..22///..82444 2022.

---

Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
Sun, Y., Zhang, J., Jia, C., Lin, H., Ye, J. and Yu, Y. Model- bellman inconsistency for model-based offline reinforce- ment learning. In ICML, pp. 33177-33194, 2023.
Yang, R., Zhong, H., Xu, J., Zhang, A., Zhang, C., Han, L., and Zhang, T. Towards robust offline reinforcement learning under diverse data corruption. In ICLR, 2024b,
Tian, S., Nair, S., Ebert, F., Dasari, S., Eysenbach, B., Finn, C., and Levine, Model-based visual planning with self- supervieee functional distances. arXiv preprint arXiv:2012.15373, 2020.
Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Mastering visual continuous control: Improved data-augmented re- inforcement learning. arXiv preprint vrxiv:2107.0644
2021a.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In IROS, pp. 5026-5033, 2012.
Yarats, D., Zhang, A., Kostrikov, I., Amos, B., Pineau, J., and Fergus, R. Improving sample efficiency in model- free reinforcement learning from images. In AAAI, pp. 10674-10681, 2021b.
Torabi, F., Warnell, G., and Stone, P. Behavioral cloning from observation. In I/CAI, pp. 195504957, 2018.
Ye, W., Zhang, Y., Abbeel, P., and Gao, Y. Become a proficient player with limited data through watching pure videos. In ICLR, 2022.
Valevski, D., Leviathan, Y., Arar, M., and Fruchter, S. Dif- fusion models are real-time game engines. arXiv preprint arXiv:2408. 14837, 2024.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. CoRL, 2019.
Van Den Oord, A., Vinyals, O., al. Neural discrete repre- sentation learning. In NeurlPS, volume 30, 2017.
Van der Maaten, L., and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., and Finn, C. Combo: Conservative offline model- based policy optimization. In NeurIPS, volume 34, pp. 28954-28967, 2021.
Veeriah, V., Zahavy, T., Hessel, M., Xu, Z., Oh, J., Kemaev, I. van Hasselt, H. P., Silver, D., and Singh, S. Discov- ery of options via meta-learned subgoals. In NeurIPS, volume 34, pp. 29861-29877 2021.
Yu, T., Kumar, A., Chebotar, Y., Hausman, K., Finn, C., and Levine, S. How to leverage unlabeled data in of- fline reinforcement learning. ICML, pp. 25611 1 1-6335, 2022.
Walke, H., Black, K., Lee, A., Kim, M. J., Du, M., Zheng, C., Zhao, T., Hansen-Estruch, P., Vuong, Q. He, A., Myers, V., Fang, K., Finn, C., and Levine, S. Bridgedata v2: A dataset for robot learning at scale. In CoRI,, 2023.
Zang, H., X., Yu, Liu, C., Islam, R., Combes, R. T. D., and Laroche, R. Behavior prior representation learning for offline reinforcement learning. ICLR, 2023.
Wang, Q., Yang, J., Wang, Y., Jin, X., Zeng, W., and Yang, X. Making offline online: Collaborative world models for offline visual reinforcement learning. In NeurIPS, 2024.
Zhang, Q. Peng, Z., and Zhou, B. Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining. In ECCV, pp. 111-128. Springer, 2022.
Wu, J., Ma, H., Deng, C., and Long. M. Pre-training con- textualized world models with in-the-wild videos for re- inforcement learning. In NeurIPS, 2023.
Wu, Y., Tucker, G., and Nachum, O. Behavior regu- larized offline reinforcement learning. arXiv preprint 1 13611 2019.
Yadav, K., Ramrakhya, R., Majumdar, A., Berges, V.PP., Kuhar, S., Batra, D., Baevski, A., and Maksymets, O. Offline visual representation learning for embodied nav- igation. In Workshop Reincarnating Reinforcement Learning at ICLR 2023, 2023.
Yang, M., Li, J., Fang, Z., Chen, S., Yu, Y., Fu, Q., Yang, W., and Ye, D. Playable game generation. arXiv preprint MxXiv:24/2.0056 2024a.
12

---

Video-Enhanced Offlie Reinforcement Learning: A Model-Based Approach
Appendix
A. Algorithm of VeoRL
provide the overall training scheme of VeoRL in Algorithm
Algorithm The training scheme of VeoRL
Require: Target offline RL dataset 8 and source video set B Initialize: Random policy network ng and value network Ui--
while not converged do
2: // Behavior abstraction learning
3: for update m steps do
4: Draw observation sequences 0.T Btr and 1.T B
5: Train BAN using εvo + pl on o.r--
6: Train the image encoder using MMDD on 0.7 and o.r
7:
end for
8:
// World model learning
9:
for update m2 steps do
10:
) Draw trajectories -اu { at r J==
Bu and o Bur
11:
Train the low-level trunk net using Objective (3) on 7l
12:
Predict latent behaviors -i-r-1 (AN(oTTT) and 01:T-1 BA
13:
Train the high-level plan net using Objective (4) on [1T 41T-11 and (oT 1TT-1 star Train the behavior cloning network FBC using "1. -1 and s0e
14:
1:7-11
15:
end for end while
// Model-based policy learning
18: while not converged do
19:
Draw [[ot"r atth , ((rr)) 8tr.
20:
for each do
Generate {s,, ae r,)]+++ using T and the low-level trunk net. 22: Generate {3), a a) 1 using FBC and the high-level plan net. Compute the intrinsic rewards (Fr)t I+L using Eq. (5).
21:
24: Update vy and F using Objective (7).
25:
end for
26:
end while
B. Implementation Details
The model configurations and hyperparameter settings are detailed in Table 2.
C. Benchmark Details
We evaluate the performance of VeoRL in the following three visual RL environments:
Meta-World (Yu et al., 2019): The Meta-World benchmark simulates 50 manipulation tasks with complex visual dynamics, all executed by the same robotic arm. These tasks involve variety of actions such as reaching, pushing, and grasping, implemented with the MuloCo physics engine (Todorov et al., 2012). The action space is represented as 2-tuple: the 3D movement of the end-effector and a normalized torque for the gripper fingers, with values ranging from -1 to 1. We collect six offline datasets from the tasks button press, drawer open, handle pull, handle press, plate slide, and coffee push. Each dataset consists of 200 trajectories, each with 500 time steps.
CARLA (Dosovitskiy et al., 2017): CARLA provides realistic visual observations for autonomous driving research, including varied weather conditions, road layouts, and traffic signs. The agent controls the vehicle using steering, acceleration, and braking commands. The goal to maximize driving distance over 1,000 time steps while avoiding
13

---

Video-Enhanced Offlie Reinforcement Learning: A Model-Based Approach
Table An overview of layers and hyperparameters used in VeoRI. in the three environments.
Item
Meta-Worke
CARLA
MineDojo
World Model
Image encoder
Conv3-32
Conv3-32
Conv3-96
GRU hidden size
200
200
4096
RSSM number of units
200
200
1024
Stochastic latent dimension
50
50
32
Discrete latent classes Weighting factor o World model learning rate
0
0
32
1
1
1
3 10
3 10
1 10
BAN's update iterations
40K
40K
30K
Behavior Learning
Imagination horizon
15
15
15
Attarget Discount y Eq. (6)
0.95
0.95
0.95
0.99
0.99
0.99
in Eq. (6)
0.05
0.1
1 10105
in Eq. (7)
0
0
1
?) in Eq. (7)
1 10
10-1
3 10-4
MLP number of Policy network MLP number of Value network Policy network learning rate
4
4
5
3
3
5
10-5
10-5
10-5
Value network learning rate
8 10-5
10-5
3 10-5
Environment Setting
Time limit
500
1000
1000
Action repeat
1
4
1
Image size
64 x 64
64 x 64
64 x 64
collisions with 30 vehicles or barriers. The reward function incorporates velocity, collision impact, and steering effort, encouraging safe and efficient driving. The offline dataset includes approximately 1,000 episodes.
MineDojo (Fan et al., 2022): This platform provides convenient APIs built on Minecraft, standardizing task specifications, world settings, and observation and action spaces for agents. In study, focus three challenging tasks: harvest log in plains, harvest water with bucket, and harvest sand. We set the maximum time steps of each episode 1,000 and adopt a binary reward indicating task completion, supplemented by the MineCLIP reward (Fan et al., 2022).
For the auxiliary video datasets, we select two datasets, BridgeDData-V V2 and NuScenes, source domains due to their relevance to Meta-World and CARLA, as well as publicly available Minecraft videos for MineDojo:
BridgeData-V (Walke al., 2023): This large and diverse robot manipulation dataset featuring environments and 60.096 trajectories. Most trajectories are video recordings from the Toy Kitchen and Toy Tabletops environments. For our experiments, use trajectories longer than 20 steps as the source domain data for Mela-World, resulting in approximately 50,000 trajectories.
NuScenes (Caesar et al., 2019): This scale autonomous driving dataset contains 1,000 diverse scenes with detailed 3D object annotations. For our experiments, we use only the video recordings, excluding annotations, as the source domain data for tasks in the CARLA environment. total, we collect 850 episodes with varying time step lengths.
Minecraft videos: For the source domain of MineDojo, we use publicly available videos of humans playing Minecraft. We collect a dataset of 5,000 episodes with varying lengths.
14

---

Video-Enhanced Offlie Reinforcement Learning: A Model-Based Approach
Table Performance on Neta-World robotic manipulation tasks in success rate. compute mean and standard deviation 50 episodes over 3 random training seeds.
Methods
DrQ CQL
Dreamer V2
APV
VIP
LOMPO
Ours
Drawer Open
0
0.18 + 0.04
0.02 + 0.05
0.02 + 0.04
0 0
0.70 + 0.07
Handle Press
0.60 = 0.48
0.33 = 0.11
0.10 + 0.09
0.56 0.17
0
0.60 + 0.12
Handle Pull
0
0.04 + 0.01
0+0
0.36 + 0.15
0+0
0.30 0.10
Plate Slide
0.23 0.41
0.48 = 0.02
0
0.34 - 0.12
0
0.67 0.03
Coffee Push
0 0
0.29 = 0.01
0.11
0.16 = 0.15
0
0.31 + 0.05
Button Press
0 0
0.58 + 0.08
0.01 + 0.01
0.18 + 0.08
0
0.62 + 0.02
Average
0.14
0.32
0.04
0.27
0.53
Table 4. The ablation study of the quality of source videos on Meta-World (Handle Press), As the number of source domain videos increases, model's performance improves accordingly.
Handle Press
All videos
1/2 videos
1/4 videos
Dreamer
Episode return
2651 + 620
2477 441
1859 + 423
1202 +: 422
D. Compared Methods
We compare VeoRL against existing model-based and model-free offline RL methods:
CQL (Kumar al., 2020): A model-free method that conservatively learns value functions to address value overestimation. Following (Fu al., 2020), we integrate CQL regularizers into the DrQ-V2 algorithm (Yarats et al., 2021a) adapt it for pixel-based inputs.
Dreamer (Hafner et al., 2019): model-based RL approach that learns policies directly from latent states within the world model. We adapt it to the offline setting and refer to this version as Offline DV2.
APV (Seo et al., 2022): This method builds an cittionconditionall RSSM model on top of an action-free RSSM model pretrained on video datasets. We adapt APV to the offline scenario for a fair comparison.
VIP (Ma et al., 2023): A pretraining approach that focuses on learning universal visual representations from video datasets, enabling generalization to downstream offline tasks. For this comparison, we pair VIP with the AWR algorithm (Peng et al., 2019) for offline policy learning.
LOMPO (Rafailov et al., 2021): A model-based offline visual RL method that addresses model uncertainty in the latent space while incorporating explicit reward estimation.
VPT (Baker et al., 2022): A foundational model for Minecraft trained using standard behavior cloning. VPT uses an inverse dynamics model trained on limited labeled data to annotate vast set of unlabeled online videos. For task mask-specific performance, we finetune VPT on the offline dataset using PPO (Schulman et al., 2017), ()17, following the methodology outlined in DECKARD (Nottingham et al., 2023).
E. Further Results of Performance Comparison
Table provides detailed comparison of VeoRL and existing methods on the Meta-World benchmark, evaluated by success rate. VeoRL outperforms DreamerV2 with remarkable advantage in Drawer Open (0.18 + 0.70), Handle Pull (0.04 + 0.30) and Handle Press (0.33 0.60) in terms of success rate.
F. Analyses of the Quality of Source Videos and Target Domain Trajectories
To partially investigate the impact of using fewer high-tuality videos, we have conducted an ablation study analyzing how the quantity of unlabeled source videos affects policy performance. The results are summarized Table the number of
15

---

Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
source domain videos increases, the model's performance improves accordingly, demonstrating that method effectively leverages information from source domain videos and exhibits strong scalability. Even when using only quarter of the videos, VeoRL's results still significantly outperform DreamerV2 (which does not utilize any source domain videos for training), indicating that our method can effectively extract useful skills from the videos.
Our framework trains the Plan Net and the corresponding BC model using data from both the source domain (BridgeData) and the target domain
This design leverages the diversity of BridgeData to enable the BC model to infer high-level behavioral abstractions (e.g.. "reach" or "grasp") that are applicable the target task (e.g.. "Button press"), even when the offline data of low quality and lacks successful trajectories. To validate this, we conduct new experiment in which the offlin Meta-World data consists of random trajectories, potentially containing extremely poor demonstrations, As shown in Table 5, the results indicate that our method could still benefit from the latent skills (estimated by BC) from the auxiliary videos, demonstrating the robustness of our approach in challenging offline RL settings.
Table The ablation study the quality of target domain trajectories on Meta-World (Button Press) episode return. demonstrates the robustness of approach in challenging offline RL settings.
Button Press
Medium offline data
Random offline data
Ours
850 + 169
638 + 150
DreamerV2
765 + 120
505 + 150
VIP
545 + 94
276 + 99
G. Hyperparameter Analysis of Latent Action Space
Figure 9 provides the results of varying the number of discrete latent behaviors in the Meta-World (Handle Press) and CARLA benchmarks. We observe that setting num 50 consistently yields the best performance across all tested environments, striking a good balance between the complexity the learned latent action space and its stability of policy learning. Furthermore, VeoRL demonstrates robust performance across varying numbers of latent behaviors, compared with DreamerV2 without training with natural videos from the BridgeData- V2 and NuScenes datasets (dashed lines).
Meta-World
CARLA
80
3000
Episode Return
2500
Episode Return
40
2000
20
1500
1000
num=5
num=20
num=50
num=100
Figure Hyperparameter analyses of the dimension of the latent action space. We observe that (i) VeoRL. demonstrates robust performance across varying numbers of latent actions, compared with DreamerV2 without training with natural videos from the BridgeData- and NuScenes datasets; (ii) setting latent actions consistently yields performance across different tasks.
H. Generalizability across Tasks
We conduct the experiments on Meta-World and MineDojo to validate the transferability of latent behaviors across tasks. In the training phase, the behavior abstraction network is frozen after training on Task A, with fixed latent codebook. For Task B, this pre-trained network is directly deployed without undergoing task-spec sone-tunngg on source video data. As presented in Table 6, the minimal performance degradation confirms that our method's latent behaviors trained on Task
16

---

Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
remain effective for Task B, consistently outperforming other baselines trained specifically on Task B. This transferability significantly reduces training costs for downstream tasks, it spares the time for re-training the auxiliary videos.
Table The study of the generalization ability the latent behaviors across tasks on Meta-Worke and MineDojo. We compute the mean and standard deviation over 3 random training seeds.
Methods
Codebook construction
Downstream task
Success rate
Meta- World
VeoRL
Button Press
Button Press
0.62 + 0.02
VeoRL
Handle Press
Button Press
0.63 + 0.12
DreamerV2
N/A
Button Press
0.58 + 0.08
VIP
N/A
Button Press
0.18 + 0.08
MineDojo
VeoRL.
Harvest sand Harvest water with bucket
Harvest sand
0.55 + 0.05
VcoRL
Harvest sand
0.50 + 0.10
DreamerV2
N/A
Harvest sand
0.25 + 0.06
VPT
N/A
Harvest sand
0.20 + 0.04
Analyses of MMD in Domain Adaptation
We performed an ablation study in the Meta-World environment to evaluate the role of MMD loss. To quantify its impact, we compared performance under conditions with and without MMD loss, accompanied by feature distribution visualizations derived from principal component analysis (PCA). The results in Table 7 validate that MMD loss critically bridges domain discrepancies. By aligning feature distributions, it enables the target domain to effectively transfer and adapt knowledge from the source domain, leading 33% improvement in success rates (0.60 vs. 0.45). Furthermore, as shown in the Figure 10, incorporating MMD significantly aligns the feature distributions between the source and target domains. This alignment correlates with policy generalization.
Table 7. The ablation study the MMD loss in the Meta-
Meta-World World euvironment.
Handle Press
w/ MMD loss
w/o MMD loss
DreamerV2
Episode return
2651 620
1961 + 585
1202 + 422 0.33 + 0.11
Success rate
0.60 0.12
0.45 + 0.15
W/ MMD loss
MMD loss
Source Target
Source
Target
Dimension
-1t
-1
-22
Dimension 1
Dimension 1
Figure 10. Distribution visualization using PCA. We observe that incorporating MMD significantly aligns the feature distributions between the source and target domains.
17


Reviews:
[{"review_id": "V585b1PHTH", "reviewer": "Reviewer_e9tP", "summary": "This paper presents a model-based RL method, Video-Enhanced Offline RL (VeoRL), which leverages unlabeled Internet videos to enrich the world model. The world model comprises two state transition branches: through real actions using a trunk net and through latent behaviors using a plan net. And the policy is trained by two-stream imagined trajectories generated by the trunk net and the plan net.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments.\n\n> Q1. Complete code documentation and environment configuration files.\n\nCode and full configuration files will be open-sourced upon acceptance.\n\n> Q2. Computational complexity. \n\nBelow, we present the training time and GPU usage until model convergence. While the latent behavior extraction and dual-branch architecture in VeoRL introduce additional computations, our experiments demonstrate a favorable efficiency-performance trade-off:\n- **Compared with DreamerV2:** VeoRL requires 1.3× GPU memory and 2× training time. This modest cost results in about 360% improved episode return in CARLA and 140% higher success rates in MineDojo, as shown in Fig 1.\n- **Compared with APV, VIP, and LOMPO:** VeoRL shows comparable training time and offers advantages in lower memory usage.\n\n||VeoRL|DreamerV2|APV|VIP|LOMPO|\n|-|-|-|-|-|-|\n|Training time|19h|9h|19h|20h|25h|\n|GPU memory|5126M|3892M|6039M|12292M|8841M|\n\n> Q3. Interpretability of latent behaviors.\n\n**(1) Semantic interpretation of latent behaviors:**\n\nLatent behaviors align with real actions through our model, guided by the reward function and optimization objectives. As visualized in Fig 7, for example, in BridgeData, distinct latent behaviors correspond to atomic actions like \"grasp\", \"reach\", or \"push\". \n\n**(2) How do the latent behaviors guide policy optimization?**\n\nLatent behaviors are abstract and high-level, requiring alignment with low-level actions in the target task. To enable latent behaviors to guide target policies, we modify the MBRL algorithm as follows:\n- The policy network conditions on both the current state and latent actions from the behavior cloning module, enabling high-level intentions to guide low-level actions.\n- State transitions are modeled using a Trunk Net (driven by real low-level actions) and a Plan Net (guided by high-level latent behaviors) for long-horizon planning.\n- An intrinsic reward encourages the rollouts from the Trunk Net to progressively align with long-term states predicted by the Plan Net, bridging the target policy with behaviors extracted from auxiliary videos.\n\n**(3) Generalizability across tasks:**\n\nWe conduct additional experiments on Meta-World and MineDojo to validate the transferability of latent behaviors across tasks. The results are presented below.\n\nTraining schemes: The behavior abstraction network is frozen after training on Task A, with a fixed latent codebook. For Task B, this pre-trained network is directly deployed without undergoing task-specific fine-tuning on source video data.\n\nThe minimal performance degradation confirms that our method's latent behaviors trained on Task A remain effective for Task B, consistently outperforming other baselines trained specifically on Task B. This transferability significantly reduces training costs for downstream tasks, as it spares the time for re-training on the auxiliary videos.\n\n|Meta-World|Codebook construction|Downstream task|Success rate|\n|-|-|-|-|\n|VeoRL|Button Press|Button Press|0.62 $\\pm$ 0.02|\n|VeoRL|Handle Press|Button Press|0.63 $\\pm$ 0.12|\n|DreamerV2|NA|Button Press|0.58 $\\pm$ 0.08|\n|VIP|NA|Button Press|0.18 $\\pm$ 0.08|\n\n|MineDojo|Codebook construction|Downstream task|Success rate|\n|-|-|-|-|\n|VeoRL|Harvest sand|Harvest sand|0.55 $\\pm$ 0.05|\n|VeoRL|Harvest water with bucket|Harvest sand|0.50 $\\pm$ 0.10| \n|DreamerV2|N/A|Harvest sand|0.25 $\\pm$ 0.06|\n|VPT|NA|Harvest sand|0.20 $\\pm$ 0.04| \n\n> Q4. More seeds. \n\nAs suggested, we have conducted additional experiments on MineDojo using five different random seeds. The updated results below show consistent performance of our model. We'll include full results in the revised paper.\n\n||VeoRL|DreamerV2|LOMPO|VPT|\n|-|-|-|-|-|\n|Harvest log in plains|0.40 $\\pm$ 0.07|0.13 $\\pm$ 0.08|0.03 $\\pm$ 0.04|0.02 $\\pm$ 0.01| \n|Harvest sand|0.55 $\\pm$ 0.05|0.25 $\\pm$ 0.06|0.05 $\\pm$ 0.05|0.20 $\\pm$ 0.04|\n\n> Q5. The distribution differences between video data and the target tasks.\n\nWe summarize the differences between the source and target domains below. These distinctions highlight VeoRL’s ability to effectively leverage real-world video data, even with notable distributional discrepancies, to enhance target domain task performance.\n\n||Source: BridgeData-V2 | Target: Meta-World|\n|-|-|-|\n|Robot arm|WidowX 250 6DOF|Simulated Sawyer|\n|Data source|Collected from real robots|Generated in simulated environments|\n|Task design|Real-world tasks (e.g., kitchen activities, tool use)| Standardized tasks (50 predefined tasks)|\n|Camera view|Random-view|Right-view|\n|Episode length|Random|500|\n\n||Source: NuScenes| Target: CARLA|\n|-|-|-|\n|Data source|Real-world data collected in Boston and Singapore using sensor-equipped vehicles|Synthetic data generated via an open-source simulation platform|\n|Scenarios|1000 real-world scenarios (e.g., lane change, unprotected turn, jaywalker)|Customizable scenarios (e.g., dynamic weather, traffic density)|\n|Camera number|6|1|\n|Episode length|Random|1000|", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The paper does not contain any proofs or theoretical developments; hence, it is unnecessary to perform a check.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "This contribution is closely related to existing offline RL methods (e.g., CQL, DreamerV2) and video-enhanced RL approaches (e.g., APV, VIP).", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "**Strength**：\n\n1. The paper is well-written and easy to follow.\n\n2. The method leverages unlabeled internet video data to enhance offline RL performance and addresses cross-domain distribution shifts.\n\n3. The paper provides extensive experimental validation across multiple visuomotor control tasks (e.g., robotic manipulation, autonomous driving, and open-world games), demonstrating that VeoRL significantly outperforms existing offline RL methods on multiple tasks.\n\n**Weaknesses**:\n1. **Computational Complexity**: The paper mentions that VeoRL's training process involves extracting latent behaviors and optimizing a dual-branch world model, which may lead to high computational complexity and memory requirements. However, the paper does not provide a detailed quantitative analysis of the computational overhead or compare it with other methods.\n\n2. **Interpretability of Latent Behaviors**: While the paper shows the correspondence between latent behaviors and real actions, the semantic interpretation of these latent behaviors remains unclear. For example, the paper does not discuss in detail how these latent behaviors specifically guide policy optimization or their generalizability across different tasks.", "other_comments_or_suggestions": "1. The paper could include a quantitative analysis of computational complexity in the experimental section, particularly in comparison with other offline RL methods.\n\n2. In the experimental section, the authors use only 3 seeds. However, I believe that this is not enough. And I recommend the use of more than five seeds to ensure robustness.", "questions_for_authors": "The paper uses datasets like BridgeData-V2 and NuScenes as auxiliary video sources but does not provide an in-depth analysis of their diversity and suitability. Did the authors perform an analysis of the distribution differences between these datasets and the target tasks? If so, could more details be provided in the revised version?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BoNiN1hhui", "reviewer": "Reviewer_LSCr", "summary": "This paper introduces a method for offline reinforcement learning through using unlabeled video data to train a world model and doing model based policy optimization. The paper include experimental results from Meta-World, CARLA, and MineDojo.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s insightful comments and provide our responses below.\n\n> Q1. Comparison with DreamerV3. \n\nThe decision to use DreamerV2 as the backbone stems from two key considerations:\n- **Offline setting performance:** While DreamerV3 achieves strong performance across diverse tasks with fixed hyperparameters, our experiments (as shown below) demonstrate that it underperforms in offline RL settings without hyperparameter tuning compared to DreamerV2. \n- **Backbone consistency:** LAMPO, a key baseline in our work, employs DreamerV2 as its backbone. To enable direct and equitable comparisons between our method and LAMPO, we maintained consistency by using the same backbone. \n\nNotably, we have conducted additional experiments on Meta-world by integrating VeoRL with DreamerV3. The results show that our approach outperforms vanilla DreamerV3, showing its ability to seamlessly integrate with different model architectures. \n\n|Episode return|DreamerV2|DreamerV3|VeoRL(DV2)|VeoRL(DV3)|\n|-|-|-|-|-|\n|Drawer Open|1168.35 $\\pm$ 59.55|674.55 $\\pm$ 79.04|1953.6 $\\pm$ 121.48|1393.5 $\\pm$ 122.5|\n|Handle Press|1201.75 $\\pm$ 422.10|257.85 $\\pm$ 247.05|2650.90 $\\pm$ 619.60|1360.15 $\\pm$ 547.85|\n\n|Success rate|DreamerV2|DreamerV3|VeoRL(DV2)|VeoRL(DV3)|\n|-|-|-|-|-|\n|Drawer Open|0.18 $\\pm$ 0.04|0.00 $\\pm$ 0.00|0.70 $\\pm$ 0.07|0.55 $\\pm$ 0.15|\n|Handle Press|0.33 $\\pm$ 0.11|0.05 $\\pm$ 0.05|0.60 $\\pm$ 0.12|0.35 $\\pm$ 0.15|\n\n> Q2. Offline to online transfer. \n\nIn the offline-to-online transfer setup, we used DreamerV2 as the comparison model, as it has demonstrated stable performance across multiple environments. In this rebuttal, we conduct further experiments with the second-best comparison models in each environment (VIP, LOMPO, and VPT, respectively). The results presented below consistently demonstrate that our approach outperforms existing methods. The updated Fig 6 is available at https://anonymous.4open.science/r/image-765.\n\n|Meta-World return (Soccer)|20K|60K|100K|\n|-|-|-|-|\n|DreamerV2|37.22|81.41|106.92|\n|VIP|18.52|29.03|117.31|\n|VeoRL|150.09|213.34|234.80|\n\n|CARLA return (Night)|20K|60K|100K|\n|-|-|-|-|\n|DreamerV2|2.01|15.17|42.95|\n|LOMPO|−20.45 |-12.05|-1.26|\n|VeoRL|27.24|37.61|79.07|\n\n|MineDojo success rate (Cobblestone)|50K|100K|150K|\n|-|-|-|-|\n|DreamerV2|0.27|0.41|0.70|\n|VPT|0|0|0|\n|VeoRL|0.58|0.86|1.00|\n\n> Q3. Value overestimation. \n\n**(1) Does VeoRL alleviate value overestimation?**\n\nOverestimation bias is indeed a key challenge in offline RL, and our method mitigates this indirectly through intrinsic rewards derived from auxiliary videos. To further validate its impact, we compare estimated values with true values (i.e., the discounted sum of actual returns) on the Meta-World Handle Press task. \n- We analyze the distribution of estimated and true values using histograms. The results below show that VeoRL's predicted values align more closely with the true values, whereas DreamerV2 exhibits significant overestimation. Specifically, in 69.12% of states, DreamerV2 predicts values exceeding 1000, despite 57.71% of true values being below 200. \n\nNotably,\n- Since our model incorporates both environmental and intrinsic rewards in value estimation, we also include the intrinsic reward when computing the true value. The negative intrinsic reward leads to a lower overall highest true value in VeoRL. As a result, directly comparing the true values of the two models is not meaningful.\n- Additionally, value computation follows each model’s successful trajectories. \n\nCorresponding figures are available at https://anonymous.4open.science/r/image-765. Our approach is shown to produce value estimates that are closer to the true values compared to DreamerV2, indicating reduced overestimation bias.\n\n|Histogram of 50 episodes|[0,200)|[200,400)|[400,600)|[600,800)|[800,1000)|[1000,1200]|\n|-|-|-|-|-|-|-|\n|DreamerV2 estimate|0.44%|3.12%|5.20%|6.45%|15.68%|69.12%|\n|True value|57.71%|5.08%|5.73%|6.34%|25.14%|0%|\n\n|Histogram of 50 episodes|[0,200)|[200,400)|[400,600)|[600,800)|[800,1000)|[1000,1200]|\n|-|-|-|-|-|-|-|\n|VeoRL estimate|1.98%|11.79%|30.69%|55.54%|0%|0%|\n|True value|15.93%|24.51%|49.98%|9.58%|0%|0%|\n\n**(2) How does VeoRL specifically handle the value estimation bias?**\n\nVeoRL leverages **intrinsic rewards** to mitigate value bias in offline RL by **grounding value estimates in realistic behaviors**:\n- Traditional value functions in offline RL can extrapolate poorly for unseen states because they optimize based only on limited environmental feedback.\n- Intrinsic rewards encourage high-value states to align with realistic source behaviors. By incorporating intrinsic rewards from video priors, our approach provides an alternative supervision signal, reducing reliance on extrapolated Q-values.\n- This approach acts as a regularization mechanism, constraining the model from assigning arbitrarily high values to unfamiliar states, preventing unrealistic value spikes.", "claims_and_evidence": "This paper makes several claims:\n1. The proposed method has a performance improvement compared to other offline RL baselines. \n\nThis claim is generally supported with evidence from Meta-World, CARLA, and MineDojo environments. However, the paper states that the DreamerV2 algorithm is a main baseline without acknowledging DreamerV3, which is a newer method in the same area. \n\n2. Better offline to online transfer. \n\nWhile it does show this compared to DreamerV2, it could use more empirical evidence comparing to other methods. \n\n3. Overestimation bias.\n\nThe claim is made that overestimation bias is the key challenge in offline reinforcement learning, but does not show if the proposed method mitigates overestimation in the value function.", "methods_and_evaluation_criteria": "The benchmark datasets of Meta-World, CARLA, and MineDojo make sense for the problem or application at hand.", "theoretical_claims": "There are no substantial theoretical claims made in the paper.", "experimental_designs_or_analyses": "The experimental design is generally sound. The paper compares VeoRL against both model-based methods such as DreamerV2 and LOMPO and model-free methods such as CQL.", "relation_to_broader_scientific_literature": "The key contributions of the paper build on several important research threads in offline reinforcement learning namely leveraging unlabeled videos for RL and model based offline RL.", "essential_references_not_discussed": "The key contribution is a method for model based offline RL from unlabeled videos. It does not compare with DreamerV3, which is a model based RL algorithm that builds on DreamerV2, a main baseline in this paper.", "other_strengths_and_weaknesses": "The paper is original and significant. The main weakness is it does not compare with an important baseline which is DreamerV3.", "other_comments_or_suggestions": "I do not have other suggestions.", "questions_for_authors": "1. Value overestimation: in the paper it is claimed overestimation bias the key issue in offline RL, but don't provide measurements for value function estimates in the proposed method. Did you compare estimated vs actual values in the experiments?\n\n2. Baseline comparisons: the main baseline for the paper is DreamerV2. Is there a reason DreamerV3 is not compared to in the paper?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wH0bvwwo7h", "reviewer": "Reviewer_W3sg", "summary": "This work proposes a novel method called VeoRL (Video-enhances offline RL). This method allows to use additional pre-training video data, without the need for reward or action annotations. That data is used to train a behavior abstraction network and the planning network. The in-domain data is used to train the planning network, as well as the trunk network. Then, inside the world model, a policy is trained to optimize the rewards, as well as to optimize suboal reaching objective, with subgoals specified by the planning network. The method is evaluated on metaworld, Carla and minecraft, showing impressive performance compared to existing methods such as APV, VIP, and DreamerV2.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's valuable comments and have addressed each point below.\n\n> Q1. The method is compared to DreamerV2 and not V3.\n\nPlease refer to our response to **Reviewer LSCr Q1**.\n\n> Q2. This paper doesn't cite FICC, which is another method that tackles a very similar problem.\n\nThank you for pointing out the FICC paper. We will add the citation and provide a detailed comparison in the revised manuscript. Although FICC and VeoRL both employ VQ-VAE and a forward model to extract latent actions in an unsupervised manner, they are distinct in the following aspects:\n\n**(1) We focus on offline RL.** \n\nWhile FICC focuses on pre-training a world model with offline data and then transferring it to downstream **online tasks**, our work addresses **offline RL**, where policies must be learned purely from static datasets without online interactions. These are fundamentally distinct problem settings with different technical challenges. For instance, offline RL primarily grapples with distributional shift due to the mismatch between the fixed dataset and real environment dynamics, while FICC employs online fine-tuning to achieve fast adaptation to downstream tasks.\n\n**(2) We handle significant domain gaps.**\n\nFICC operates under the assumption that the source domain (offline data) and target domain (downstream tasks) share the **same** environment. In contrast, our method tackles a more challenging **cross-domain** transfer scenario: extracting knowledge from real-world videos and employing it in a simulation environment. The significant visual and dynamics gaps between real-world and simulated domains substantially increase the difficulty of knowledge transfer.\n\n**(3) We leverage hybrid action spaces for behavioral alignment**\n\nFICC is explicitly designed for **discrete actions** (tested only on Atari) and acknowledges its limitation in handling continuous control tasks (\"difficult to handle the continuous action space\" as stated in their paper). Our method, however, focuses on the tasks with **continuous actions**. The dual-path policy optimization process we designed associates discrete actions extracted from videos with the continuous action space of the target environment, thereby enhancing offline RL policies with high-level behavioral abstractions (e.g., 'reach', 'push') from video data.\n\n> Q3. Experiments on more difficult, multi-step tasks.\n\nAs suggested by the reviewer, we compare our method with DreamerV2 and VIP in **Soccer** from Meta-World. This task typically involves a two-stage decision-making process where the agent must first fetch a ball and then push it to a goal position.\n\n|Soccer|DreamerV2|VIP|Ours| \n|-|-|-|-|\n|Episode return|112.53 $\\pm$ 37.90|109.84 $\\pm$ 17.32| 231.10 $\\pm$ 20.68|\n\n> Q4. The natural videos this method leverages contain, for the most part, high-quality trajectories. This enables the planner to simply use a behavior cloning policy. This however is a quite limiting assumption.\n\n**(1) Do the auxiliary videos contain only high-quality trajectories?**\n \nNo, the videos we leverage are **not curated or task-specific**. Instead, they are **diverse, easily accessible, and collected without explicit task filtering**. While these videos may not always align closely with the target tasks (meaning they **should not** be considered high-quality trajectories). Even when the tasks in the videos differ from the target tasks, our method can extract useful priors for policy learning.\n\n**(2) Does the method reduce to simple behavior cloning?**\n\nNo, our approach goes beyond naive behavior cloning, as real action labels are unavailable in passive videos. Instead, we learn discrete latent actions that capture abstract behavioral patterns. This allows our method to **generalize across environments**, as it clones high-level behavioral concepts rather than relying on low-level action sequences. By leveraging these latent structures, our method can transfer useful priors even when direct action replication is impossible.\n\n> Q5. 247 right - considering more on -> did you mean \"focusing more on\"?\n\nYes.", "claims_and_evidence": "The main claim:  VeoRL can transfer common sense knowledge of control behaviors from natural videos.\n\nThe claim is supported by the experiments.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I checked the design of the main expeirments. One issue I have is the choice of DreamerV2 as opposed to DreamerV3. DreamerV3 appears to be a more natural baseline in this case, especially for minecraft, as it has demonstrated good performance there. Can the authors justify their choice of DreamerV2 over V3? Is that due to the fact that the world models you trained have the DreamerV2-style architecture?", "relation_to_broader_scientific_literature": "This paper addresses the important question of extracting knowledge from videos. Natural videos are widely available and contain enormous amounts of information, yet most modern approaches do not tap into that resource. This particular work focuses on videos showing similar tasks but in different environments.", "essential_references_not_discussed": "This paper doesn't cite FICC, which is another method that tackles a very similar problem:\n\nYe, Weirui, et al. \"Become a proficient player with limited data through watching pure videos.\" _The Eleventh International Conference on Learning Representations_. 2022.", "other_strengths_and_weaknesses": "###### Strengths\n- The paper investigates an important yet underexplored direction of research, an introduces a novel method\n- The proposed method shows great improvement over the baselines on the proposed experiments\n- The paper is well-written\n\n###### Weaknesses\n- The method is compared to DreamerV2 and not V3\n- The tasks selected for minecraft and metaworld are quite simple, and do not involve multi-step decision making, e.g. pick and place for metaworld, or crafting an metal sword for minecraft\n- The natural videos this method leverages contain, for the most part, high-quality trajectories. This enables the planner to simply use a behavior cloning policy. This however is a quite limiting assumption.", "other_comments_or_suggestions": "247 right - considering more on -> did you mean \"focusing more on\"?", "questions_for_authors": "Q1 Why did you opt for DreamerV2 instead of DreamerV3?\nQ2 Do you have any intuition regarding how the quality of the action-free data affects performance? Since the downstream policy learning relies on behavior cloning, my intuition is that you may need something more intricate than behavior cloning if the data is lower quality or is more diverse.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DNvxPiDwLs", "reviewer": "Reviewer_dg3G", "summary": "The paper proposes a method that leverages unlabeled internet videos to enhance the performance of offline RL. A discretized latent action space is learned on unlabeled video data is learned by the behavior abstraction network (BAN) using vector quantization. The paper proposes using a two-stream world model, one conditioned on latent behaviors extracted from video data and the other on the agent’s real actions. This dual-branch structure allows the agent to effectively learn control policies by aligning state rollouts from two branches, thereby transferring knowledge from the diverse, out-of-domain video data to the RL agents. The authors compare with several prior works to show the efficacy of the proposed method. The paper also provides an ablation study to justify the design choices.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. Below, we address each comment point-by-point.\n\n> Q1. The authors use Dreamer V2 as their primary baseline. However, this is an outdated baseline and it would be great if the authors could provide comparisons with Dreamer V3.\n\nPlease refer to our response to **Reviewer LSCr Q1**.\n\n> Q2. It would be great if the authors could include details about how Dreamer V2, which is an online RL algorithm, is adapted to the offline setting.\n\nThe core adaptation of DreamerV2 to the offline setting lies in how the replay buffer is initialized and utilized. In the original online DreamerV2, the agent interacts with the environment to collect new trajectories incrementally. These trajectories are stored in a replay buffer, and the world model and policy are trained by sampling batches from this dynamically updated buffer. In the offline setting, we instead preload the entire static dataset into the buffer once at the start of training. No new data is collected during training, and all models are trained exclusively on this fixed dataset.\n\n> Q3. It would be great if the authors could include an ablation study of the effect of different amounts of unlabeled data on policy performance. It would be interesting to see if reducing the amount of data also reduces the policy performance.\n\nAs suggested, we have conducted a new ablation study to analyze how the amount of unlabeled data impacts policy performance. The results are summarized below. As the amount of source domain data increases, the model's performance improves accordingly, demonstrating that our method effectively leverages information from source domain videos and exhibits strong scalability. Full results across different datasets will be included in the revised paper. \n\n|Handle Press   | All videos  | 1/2 videos | 1/4 videos | w/o videos (results from Fig 7) | DreamerV2 |\n| ----- | --- | --- |--- | --- |--- |\n|Episode return  |  2651 $\\pm$ 620      | 2477 $\\pm$ 441      |  1859 $\\pm$ 423 | 1814 $\\pm$ 212.06 |  1201.75 $\\pm$ 422.10|", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Overall, the experiments and analyses seem comprehensive.", "relation_to_broader_scientific_literature": "The paper proposes a method for enhancing offline RL with off-domain video data. This is an important direction and baking in task priors from internet videos could potentially improve the robustness as well as sample efficiency of such policies.", "essential_references_not_discussed": "Being a researcher in a closely related but not the same field, the references seem adequate to me.", "other_strengths_and_weaknesses": "Strengths\n- The paper tackles an important problem of enhancing offline learning with unlabeled video data and shows the efficacy of the proposed approach in improving offline RL.\n- The proposed method significantly improves upon prior work and also enables finetuning online on a new scenario with better sample efficiency than DreamerV2.\n- The authors also provide an ablation study justifying design choices.\n\nWeaknesses\n- The authors use Dreamer V2 as their primary baseline. However, this is an outdated baseline and it would be great if the authors could provide comparisons with Dreamer V3.\n- It would be great if the authors could include details about how Dreamer V2, which is an online RL algorithm, is adapted to the offline setting.", "other_comments_or_suggestions": "- It would be great if the authors could include an ablation study of the effect of different amounts of unlabeled data on policy performance. It would be interesting to see if reducing the amount of data also reduces the policy performance.", "questions_for_authors": "It would be great if the authors could address the questions from the previous section and in Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper is a borderline case, and as such I conducted a short review myself. 

The paper proposes an approach to leverage unlabelled video data for offline RL with a world model. It connects the two modalities by training a latent action model from unlabelled videos and a regular world model from labelled, and then training the agent across both. The rebuttal addressed several concerns, notably 1) comparison with DreamerV3, 2) computational cost 3) ablation on the quantity of unlabelled video.

Overall I believe this could be an interesting contribution to ICML 2025. My main issue with the work is the lack of relation to the Genie paper, which won a Best Paper award at ICML 2024. I think the authors could *strengthen* the impact of their work by making this connection clear. For instance, Genie showed the power of pre-training with latent actions at scale, but did not provide an explicit path to using this for transferring policies to real environments. This paper could position itself as one path for this, potentially unlocking large-scale video world models.
Decision: Accept (poster)